@brasshopper@twtxt.net This is a good suggestion! Right now the crawler is a “one-short” thing (just to be nice while I develop/improve it). From your suggestion it sounds like I can possibly remove the “one-shot” restriction and just setup a daily job that re-crawls the entire Twtxt space from a seed feed. The best seed so far is probably my own followed by @jlj@twt.nfld.uk ’s – Then I can focus on the refetch/rescrape parts based in heuristics. What do you think?