Hack of the day: running watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
@abucci@anthony.buc.ci that’s so weird! I wonder what in the world is causing it. Hopefully @prologic@twtxt.net has an insight.
@bender@twtxt.net I hope so too. I’ve never seen anything like this before. Whatever it is, it’s strange.
@abucci@anthony.buc.ci Wer that’s a bug!
@abucci@anthony.buc.ci What revision are you running?
@prologic@twtxt.net 0.15.1, looks like.
https://anthony.buc.ci/info has the deets!
@abucci@anthony.buc.ci Please update!
@prologic@twtxt.net Sure, but why would this start happening all of a sudden today? Nothing like this has happened before. Is this a known bug?
@abucci@anthony.buc.ci I will have a look but I suspect it has something to do with the open nature of the external endpoint. I closed this loophole recently due to other reasons myself.
At work right now so will have more concrete details in a few hours from now
I also think you may be running a version that had a bug and lacked cleanup of those temp files
For example this one that got fixed this year:
commit 4304ec7ea3c5df95e0ed82bfa292c9330e342f61
Author: James Mills <james@mills.io>
Date: Mon Jan 24 00:10:33 2022 +0000
Fix bug in DownloadImage() leaking termporary files for external avatar downloads (#746)
@prologic@twtxt.net Aha, got it. Thanks for looking into it. I’m updating now and we’ll see if that stops it.
abucci@buc:~/yarnd/yarn$ make preflight
Checking Go version ... [ ERR ]
Go 1.16+ is required, found go1.22.5
FATAL: 🙁 preflight failed
make: *** [Makefile:33: preflight] Error 1
🤔
@abucci@anthony.buc.ci Fuck that script 🤣 you’re good! Just follow the Build from Source docs 😅
@prologic@twtxt.net Alright, running yarnd 0.15.1 now. I stopped my hack so we’ll see if the VPS gets clogged with junk 😆
@abucci@anthony.buc.ci Hopefully it shouldn’t 🤞
Hopefully you should see traffic die off a bit too as the /external endpoint is no longer externally abusable (get it) without being an authenticated user – which became problematic 🤦♂️ – The web is so fucking hostile 🤬
@prologic@twtxt.net I’m still getting this crap:
abucci@buc:~/yarnd/yarn$ ls -lh /tmp/yarnd-avatar-*
-rw------- 1 abucci abucci 863M Jul 25 14:19 /tmp/yarnd-avatar-1594499680
-rw------- 1 abucci abucci 7.8G Jul 25 14:19 /tmp/yarnd-avatar-2144295337
-rw------- 1 abucci abucci 9.8G Jul 25 14:19 /tmp/yarnd-avatar-2334738193
-rw------- 1 abucci abucci 10G Jul 25 14:14 /tmp/yarnd-avatar-2494107777
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-2619243454
-rw------- 1 abucci abucci 11G Jul 25 14:04 /tmp/yarnd-avatar-2922187513
-rw------- 1 abucci abucci 7.5G Jul 25 14:14 /tmp/yarnd-avatar-349775570
-rw------- 1 abucci abucci 10G Jul 25 14:09 /tmp/yarnd-avatar-3640724243
-rw------- 1 abucci abucci 901M Jul 25 14:19 /tmp/yarnd-avatar-3921595598
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-609094539
-rw------- 1 abucci abucci 9.3G Jul 25 14:04 /tmp/yarnd-avatar-755173392
-rw------- 1 abucci abucci 7.9G Jul 25 14:09 /tmp/yarnd-avatar-984061000
Something like 100 Gbytes of this junk has accumulated since I updated and re-started the server. I’m now running the latest version of yarnd, so the update did not fix the problem. Something else is going wrong.
How are temporary files growing to 10 Gbytes in size? The name of the file is “yarn-avatar”, but why would avatars be so large?
@abucci@anthony.buc.ci Hmm that’s a bit weird then. Lemme have a poke.
Hah 😈
prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
(main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666: tf, err := receiveFile(res.Body, "yarnd-avatar-*")
@abucci@anthony.buc.ci Don’t suppose you can inspect one of those files could you? Kinda wondering if there’s some other abuse going on here that I need to plug? 🔌
These should be getting cleaned up, but I’m very concerned about the sizes of these 🤔
@prologic@twtxt.net 10 Gbytes has accumulated since I made that last post. It’s coming in at a rate of 55 Mbits/second !
Do you happen to have the activitypub feature turned on btw? In fact could you just list out what features you have enabled please? 🙏
@prologic@twtxt.net Inspect? What’s sift? What would you like to know about the files?
@abucci@anthony.buc.ci sift is a tool I use for grep/find, etc.
What would you like to know about the files?
Roughly what their contents are. I’ve been reviewing the code paths responsible and have found a flaw that needs to be fixed ASAP.
Here’s the PR: https://git.mills.io/yarnsocial/yarn/pulls/1169
@abucci@anthony.buc.ci So… The only way I see this happening at all is if your pod is fetching feeds which have multi-GB sized avatar(s) in their feed metadata. So the PR I linked earlier will plug that flaw. But now I want to confirm that theory. Can I get you to dump your cache to JSON for me and share it with me?