@stigatle@yarn.stigatle.no / @abucci@anthony.buc.ci My current working theory is that there is an asshole out there that has a feed that both your pods are fetching with a multi-GB avatar URL advertised in their feed’s preamble (metadata). I’d love for you both to review this PR, and once merged, re-roll your pods and dump your respective caches and share with me using https://gist.mills.io/
Or if y’all trust my monkey-ass coding skillz I’ll just merge and you can do a git pull and rebuild 😅
I’m going to merge this…
@abucci@anthony.buc.ci / @stigatle@yarn.stigatle.no Please git pull, rebuild and redeploy.
There is also a shell script in ./tools called dump_cache.sh. Please run this, dump your cache and share it with me. 🙏
@prologic@twtxt.net I’m running it now. I’ll keep an eye out for the tmp folder now (I built the branch you have made). I’ll let you know shortly if it helped on my end.
@stigatle@yarn.stigatle.no The problem is it’ll only cause the attack to stop and error out. It won’t stop your pod from trying to do this over and over again. That’s why I need some help inspecting both your pods for “bad feeds”.
if we can figure out wtf is going on here and my theory is right, we can blacklist that feed, hell even add it to the codebase as an “asshole”.
Just thinking out loud here… With that PR merged (or if you built off that branch), you might hopefully see new errors popup and we might catch this problematic bad feed in the act? Hmmm 🧐
@prologic@twtxt.net so, if I’m correct the dump tool made a pods.txt and a stats.txt file, those are the ones you want? or do you want the output that it spits out in the console window?
@stigatle@yarn.stigatle.no You want to run backup_db.sh and dump_cache.sh They pipe JSON to stdout and prompt for your admin password. Example:
URL=<your_pod_url> ADMIN=<your_admin_user> ./tools/dump_cache.sh > cache.json
But just have a look at the yarnd server logs too. Any new interesting errors? 🤔 No more multi-GB tmp files? 🤔
@prologic@twtxt.net thank you. I run it now as you said, I’ll get the files put somewhere shortly.
@stigatle@yarn.stigatle.no Ta. I hope my theory is right 😅
@stigatle@yarn.stigatle.no Thank you! 🙏
@prologic@twtxt.net No worries, thanks for working on the fix for it so fast :)
Ooof
$ jq '.Feeds | keys[]' cache.json | wc -l
4402
If you both don’t mind dropping your caches. I would recommend it. Settings -> Poderator Settings -> Refresh cache.
That was also a source of abuse that also got plugged (being able to fill up the cache with garbage data)
@prologic@twtxt.net you want a new cache from me - or was the one I sent OK for what you needed?
@stigatle@yarn.stigatle.no The one you sent is fine. I’m inspecting it now. I’m just saying, do yourself a favor and nuke your pod’s garbage cache 🤣 It’ll rebuild automatically in a much more prestine state.
@prologic@twtxt.net will do, thanks for the tip!