Conversation #3ll4fja

Recent tws in this reply to #3ll4fja

(#3ll4fja) I feel like most feeds I would want to subscribe to are updated pretty infrequently so once a day would be sufficient. But I’ve seen other feeds that are updated more frequently. Perhaps some sort of heuristic based off the feed itself? (For example, average time between entries with a max of 1 day.) If you combined that with a feature to manually trigger a feed recrawl, that might be good enough.


(#3ll4fja) This is a good suggestion! Right now the crawler is a “one-short” thing (just to be nice while I develop/improve it). From your suggestion it sounds like I can possibly remove the “one-shot” restriction and just setup a daily job that re-crawls the entire Twtxt space from a seed feed. The best seed so far is probably my own followed by ’s – Then I can focus on the refetch/rescrape parts based in heuristics. What do you think?


(#3ll4fja) As a user, I think 1-3 times a day would be fine.
As someone who pays to host a twtxt feed, I don’t mind what you set it to as long as it’s not unreasonably often. It won’t really make a difference to me. As of about 6 months ago, 3 different pods each request /twtxt.txt (with an If-Modified-Since header) every 5 minutes, 24 / 7. I think I also had a different twtxt client requesting it every 10 minutes.

You must be Logged in to join the conversation.