In yarnd one can block certain URLs. Is there a way to do the same in yarns, @prologic@twtxt.net? If so, can you please add https://lyse.isobeef.org/user/lyse/twtxt.txt, http://lyse.isobeef.org and https://lyse.isobeef.org to the yarns blacklist? Especially the first URL spams my error log every hour. It might also be useful to do some housekeeping with other crap URLs: https://search.twtxt.net/stats/feeds/discovered?l=30&q=lyse&s=-failures

Also, @darch@neotxt.dk’s yarnd is requesting my /twtxt-2022-08.txt every few minutes (interval varies between ~2-5 minutes). Nothing seems to be cached, though, because 200 is sent all the time. Any idea what’s going on here and how to fix it, @prologic@twtxt.net?

And another weird thing is that yarnd’s User-Agent header now contains all sorts of build information: yarnd/0.15.1@7dd5a93e 2023-03-27T00:53:59+10:00 go1.20.3 (~https://… I reckon this is a regression from the version change on the info page I proposed on IRC months ago. Maybe even last year.

What a joy to look in the logs once in a while. :-D

@lyse@lyse.isobeef.org Yup this is on you 😆

@lyse@lyse.isobeef.org Yeah we can do someyhing similar for the crawler 👌

Now there is a way to block feeds in yarns: https://git.mills.io/yarnsocial/search/pulls/16