@prologic@twtxt.net, does this rings a bell to you? 159-196-9-199.9fc409.mel.nbn.aussiebb.net
@bender@twtxt.net Yes, why? 🤔
@prologic@twtxt.net it is kind of hammering my VPS, specifically netbros.com, every single day.
Also hammering ferengi.one.
@bender@twtxt.net define hammering?
@prologic@twtxt.net thousands of daily requests.
The thing is, I don’t have a twtxt.txt file on netbros.com.
@bender@twtxt.net I’ve blocked it from this pod for now 🤞Not sure which users are still trying to fetch the non-existent feeds sorry!
@prologic@twtxt.net no worries mate, and thanks! I wonder if something could be done for feeds rendering 404, so that they get automatically “unfollowed”, and removed.
A twtxt.txt file should never spit out a 404, unless it’s no more.
@bender@twtxt.net Unfoetunately that isn’t actually true as it depend on the ingress architecture and networking.
@prologic@twtxt.net if a twtxt.txt is not found, under which conditions will it be found again, and can something be done if say, it isn’t found for X amount of hours, days, months?
@prologic@twtxt.net @bender@twtxt.net Exponential backoff? Seems like the right thing to do when a server isn’t accepting your connections at all, and might also be a reasonable compromise if you consider 404 to be a temporary failure.
@falsifian@www.falsifian.org that sounds like a good compromise. Regardless of what @prologic@twtxt.net wrote, a 404 is a 404.
@bender@twtxt.net 404 could be indeed a temporary error if the file resides on a mounted remote filesystem and then the mount point fails for some reason. With a symlink from the web root to the file on the mount, the web server probably will not recognize the mount point failure as such. Thus, it might not reply with a 503 Service Unavailable (or something like that), but 404 Not Found instead. (I could be wrong on that, though.)
The right™ way is to signal 410 Gone if the feed does not exist anymore and will not come back to life again. But that’s hard to come by in the wild. Somebody has to manually configure that in almost all situations.
But yes, as @falsifian@www.falsifian.org points out, exponential backoff looks like a good strategy. Probably even report a failure to users somehow, so they can check and potentially unsubscribe.
@lyse@lyse.isobeef.org right, now, on this:
“The right™ way is to signal 410 Gone if the feed does not exist anymore and will not come back to life again. But that’s hard to come by in the wild. Somebody has to manually configure that in almost all situations.”
Even so, what does Yarn do if a 410 is sent? I don’t think it does anything at the moment, but I could be wrong.
@bender@twtxt.net You could be right. Grepping the yarnd code for 410 and Gone did not reveal anything. Maybe, maybe it is handled by another library. But I kinda doubt it.
I’m wrong! Both 404 and 410, among others, are considered dead feeds: https://git.mills.io/yarnsocial/yarn/src/branch/main/internal/cache.go#L1343 Whatever that actually means.
Yeah, the ErrDeadFeed is never actually checked anywhere. It’s only set and that’s it.
Righto, I cobbled something together here: https://git.mills.io/yarnsocial/yarn/pulls/1172 It needs a bunch more work, though. Screen time is up for today.
This has become quite a large thread. 😅