Hack of the day: running watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
@abucci@anthony.buc.ci Wer that’s a bug!
@abucci@anthony.buc.ci What revision are you running?
@abucci@anthony.buc.ci Please update!
@abucci@anthony.buc.ci I will have a look but I suspect it has something to do with the open nature of the external endpoint. I closed this loophole recently due to other reasons myself.
At work right now so will have more concrete details in a few hours from now
I also think you may be running a version that had a bug and lacked cleanup of those temp files
For example this one that got fixed this year:
commit 4304ec7ea3c5df95e0ed82bfa292c9330e342f61
Author: James Mills <james@mills.io>
Date: Mon Jan 24 00:10:33 2022 +0000
Fix bug in DownloadImage() leaking termporary files for external avatar downloads (#746)
@abucci@anthony.buc.ci Fuck that script 🤣 you’re good! Just follow the Build from Source docs 😅
@abucci@anthony.buc.ci Hopefully it shouldn’t 🤞
Hopefully you should see traffic die off a bit too as the /external endpoint is no longer externally abusable (get it) without being an authenticated user – which became problematic 🤦♂️ – The web is so fucking hostile 🤬
@abucci@anthony.buc.ci Hmm that’s a bit weird then. Lemme have a poke.
Hah 😈
prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
(main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666: tf, err := receiveFile(res.Body, "yarnd-avatar-*")
@abucci@anthony.buc.ci Don’t suppose you can inspect one of those files could you? Kinda wondering if there’s some other abuse going on here that I need to plug? 🔌
These should be getting cleaned up, but I’m very concerned about the sizes of these 🤔
Do you happen to have the activitypub feature turned on btw? In fact could you just list out what features you have enabled please? 🙏
@abucci@anthony.buc.ci sift is a tool I use for grep/find, etc.
What would you like to know about the files?
Roughly what their contents are. I’ve been reviewing the code paths responsible and have found a flaw that needs to be fixed ASAP.
Here’s the PR: https://git.mills.io/yarnsocial/yarn/pulls/1169
@abucci@anthony.buc.ci So… The only way I see this happening at all is if your pod is fetching feeds which have multi-GB sized avatar(s) in their feed metadata. So the PR I linked earlier will plug that flaw. But now I want to confirm that theory. Can I get you to dump your cache to JSON for me and share it with me?