@stigatle@yarn.stigatle.no / @abucci@anthony.buc.ci My current working theory is that there is an asshole out there that has a feed that both your pods are fetching with a multi-GB avatar URL advertised in their feed’s preamble (metadata). I’d love for you both to review this PR, and once merged, re-roll your pods and dump your respective caches and share with me using https://gist.mills.io/
Or if y’all trust my monkey-ass coding skillz I’ll just merge and you can do a git pull and rebuild 😅
I’m going to merge this…
@abucci@anthony.buc.ci / @stigatle@yarn.stigatle.no Please git pull, rebuild and redeploy.
There is also a shell script in ./tools called dump_cache.sh. Please run this, dump your cache and share it with me. 🙏
@prologic@twtxt.net I’m running it now. I’ll keep an eye out for the tmp folder now (I built the branch you have made). I’ll let you know shortly if it helped on my end.
@stigatle@yarn.stigatle.no The problem is it’ll only cause the attack to stop and error out. It won’t stop your pod from trying to do this over and over again. That’s why I need some help inspecting both your pods for “bad feeds”.
if we can figure out wtf is going on here and my theory is right, we can blacklist that feed, hell even add it to the codebase as an “asshole”.
Just thinking out loud here… With that PR merged (or if you built off that branch), you might hopefully see new errors popup and we might catch this problematic bad feed in the act? Hmmm 🧐
@prologic@twtxt.net so, if I’m correct the dump tool made a pods.txt and a stats.txt file, those are the ones you want? or do you want the output that it spits out in the console window?
I’m seeing GETs like this over and over again:
"GET /external?nick=lovetocode999&uri=https://vuf.minagricultura.gov.co/Lists/Informacin%20Servicios%20Web/DispForm.aspx?ID=8375144 HTTP/1.1" 200 35861 17.077914ms
always to nick=lovetocode999, but with different uris. What are these calls?
@stigatle@yarn.stigatle.no You want to run backup_db.sh and dump_cache.sh They pipe JSON to stdout and prompt for your admin password. Example:
URL=<your_pod_url> ADMIN=<your_admin_user> ./tools/dump_cache.sh > cache.json
But just have a look at the yarnd server logs too. Any new interesting errors? 🤔 No more multi-GB tmp files? 🤔
@prologic@twtxt.net thank you. I run it now as you said, I’ll get the files put somewhere shortly.
@prologic@twtxt.net Hitting that URL returns a bunch of HTML even though there is no user named lovetocode999 on my pod. I think it should 404, and maybe with a delay, to discourage whatever this abuse is. Basically this can be used to DDoS a pod by forcing it to generate a hunch of HTML just by doing a bogus GET like this.
@stigatle@yarn.stigatle.no Ta. I hope my theory is right 😅
@prologic@twtxt.net Try hitting this URL:
https://twtxt.net/external?nick=nosuchuser&uri=https://foo.com
Change nosuchuser to any phrase at all.
If you hit https://twtxt.net/external?nick=nosuchuser , you’re given an error. If you hit that URL above with the uri parameter, you can a legitimate-looking page. I think that is a bug.
@stigatle@yarn.stigatle.no Thank you! 🙏
@prologic@twtxt.net No worries, thanks for working on the fix for it so fast :)
./tools/dump_cache.sh: line 8: bat: command not found
No Token Provided
I don’t have bat on my VPS and there is no package for installing it. Is cat a reasonable alternate?
Ooof
$ jq '.Feeds | keys[]' cache.json | wc -l
4402
If you both don’t mind dropping your caches. I would recommend it. Settings -> Poderator Settings -> Refresh cache.
That was also a source of abuse that also got plugged (being able to fill up the cache with garbage data)
@prologic@twtxt.net you want a new cache from me - or was the one I sent OK for what you needed?
@stigatle@yarn.stigatle.no The one you sent is fine. I’m inspecting it now. I’m just saying, do yourself a favor and nuke your pod’s garbage cache 🤣 It’ll rebuild automatically in a much more prestine state.
@prologic@twtxt.net will do, thanks for the tip!