@stigatle@yarn.stigatle.no Ta. I hope my theory is right 😅
But just have a look at the yarnd server logs too. Any new interesting errors? 🤔 No more multi-GB tmp files? 🤔
@stigatle@yarn.stigatle.no You want to run backup_db.sh and dump_cache.sh They pipe JSON to stdout and prompt for your admin password. Example:
URL=<your_pod_url> ADMIN=<your_admin_user> ./tools/dump_cache.sh > cache.json
Just thinking out loud here… With that PR merged (or if you built off that branch), you might hopefully see new errors popup and we might catch this problematic bad feed in the act? Hmmm 🧐
@slashdot@feeds.twtxt.net I thought Sunday was the hottest day on Earth 🤦♂️ wtf is wrong with Slashdot these days?! 🤣
if we can figure out wtf is going on here and my theory is right, we can blacklist that feed, hell even add it to the codebase as an “asshole”.
@stigatle@yarn.stigatle.no The problem is it’ll only cause the attack to stop and error out. It won’t stop your pod from trying to do this over and over again. That’s why I need some help inspecting both your pods for “bad feeds”.
@abucci@anthony.buc.ci / @stigatle@yarn.stigatle.no Please git pull, rebuild and redeploy.
There is also a shell script in ./tools called dump_cache.sh. Please run this, dump your cache and share it with me. 🙏
I’m going to merge this…
@abucci@anthony.buc.ci Yeah I’ve had to block entire ASN(s) recently myself from bad actors, mostly bad AI bots actually from Facebook and Caude AI
Or if y’all trust my monkey-ass coding skillz I’ll just merge and you can do a git pull and rebuild 😅
@stigatle@yarn.stigatle.no / @abucci@anthony.buc.ci My current working theory is that there is an asshole out there that has a feed that both your pods are fetching with a multi-GB avatar URL advertised in their feed’s preamble (metadata). I’d love for you both to review this PR, and once merged, re-roll your pods and dump your respective caches and share with me using https://gist.mills.io/
@stigatle@yarn.stigatle.no I’m wondering whether you’re having the same issue as @abucci@anthony.buc.ci still? mulit-GB yarnd-avatar-*1 files piling up in /tmp/? 🤔
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
@abucci@anthony.buc.ci So… The only way I see this happening at all is if your pod is fetching feeds which have multi-GB sized avatar(s) in their feed metadata. So the PR I linked earlier will plug that flaw. But now I want to confirm that theory. Can I get you to dump your cache to JSON for me and share it with me?
@abucci@anthony.buc.ci Yeah that should be okay, you get so much crap on the web 🤦♂️
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
@abucci@anthony.buc.ci sift is a tool I use for grep/find, etc.
What would you like to know about the files?
Roughly what their contents are. I’ve been reviewing the code paths responsible and have found a flaw that needs to be fixed ASAP.
Here’s the PR: https://git.mills.io/yarnsocial/yarn/pulls/1169
@abucci@anthony.buc.ci I believe you are correct.
@abucci@anthony.buc.ci That’s fucking insane 😱 I know what code-paths is triggering this, but need to confirm a few other things… Some correlation with logs would also help…
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
Do you happen to have the activitypub feature turned on btw? In fact could you just list out what features you have enabled please? 🙏
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
These should be getting cleaned up, but I’m very concerned about the sizes of these 🤔
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
Hah 😈
prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
(main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666: tf, err := receiveFile(res.Body, "yarnd-avatar-*")
@abucci@anthony.buc.ci Don’t suppose you can inspect one of those files could you? Kinda wondering if there’s some other abuse going on here that I need to plug? 🔌
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
@abucci@anthony.buc.ci Hmm that’s a bit weird then. Lemme have a poke.
Hmm remove the cpu limits on this pod, not even sure why I had ‘em set tbh, we decided at my day job that setting cpu limits on containers is a bit of a silly idea too. Anyway, pod should be much snappier now 😅
@movq@www.uninformativ.de Oh nothing much 🤣 Just a bunch of folks running really old versions of yarnd that were susceptible to abuse on the open web 🤣
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
Hopefully you should see traffic die off a bit too as the /external endpoint is no longer externally abusable (get it) without being an authenticated user – which became problematic 🤦♂️ – The web is so fucking hostile 🤬
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
@abucci@anthony.buc.ci Hopefully it shouldn’t 🤞
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
@abucci@anthony.buc.ci Fuck that script 🤣 you’re good! Just follow the Build from Source docs 😅
Thinking we need to adapt the UI a little bit to something like this
@bender@twtxt.net I can see the same errors again hmmm 🧐 @stigatle@yarn.stigatle.no Did you run out of disk again? 😅
I had a play with LiveKit Agents Playground: KITT and I have to say it’s pretty impressive. Not the ChatGPT part of course, but the speech recognition and text to speech synthesis.
KITT is an AI voice assistant powered by LiveKit Agents, Deepgram, Eleven Labs, and ChatGPT. It is running on LiveKit Playground.
It’s too bad it relies on three cloud services, none of which can be run locally (with the exception of Ollama that you could replace the OpenAI component with).
@lyse@lyse.isobeef.org Man gotta love that sunset !!! So nice 😊
You should have the fancy new SPA-like UI too 😅 (just checked!)
@stigatle@yarn.stigatle.no No worries at all! 👌
@stigatle@yarn.stigatle.no Note that “Building From Source” is covered in the docs
You are reminding me that I should cut a release soon™ so there are binaires you can just “download” and use for the platform of choice 😅
@stigatle@yarn.stigatle.no So make deps would have installed some tools in either $GOPATH/bin or $GOBIN. See which with go env. Chuck that in your $PATH and you’re good to run make server. Normally this would be something like:
GOBIN=$HOME/go/bin
GOPATH=$HOME/go
export GOPATH GOBIN
...
@stigatle@yarn.stigatle.no Run make deps. I use a non-standard (written in Go) minify tool
@stigatle@yarn.stigatle.no Take a backup of the data dir in case I screwed something up 🙏
@stigatle@yarn.stigatle.no Sweet 👌
@stigatle@yarn.stigatle.no I think pods have become exploited over time so I’ve had to tighten up some feature like the external handler 😢
@stigatle@yarn.stigatle.no Ahh! Please update to the latest main 🙏
@stigatle@yarn.stigatle.no Works now! 🥳
@stigatle@yarn.stigatle.no It looks like your some kind of problem with the reverse proxy in front of yarnd? 🤔I ’m seeing this error: incomplete chunked encoding error(s) 
I don’t know anything about this though, tbh I’ve never seen this before myself 🤔”
@stigatle@yarn.stigatle.no I’m doing okay 👌 Busy with work as you can imagine, and still tinkering of course whenever I can spare a moment or two! 😅
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
For example this one that got fixed this year:
commit 4304ec7ea3c5df95e0ed82bfa292c9330e342f61
Author: James Mills <james@mills.io>
Date: Mon Jan 24 00:10:33 2022 +0000
Fix bug in DownloadImage() leaking termporary files for external avatar downloads (#746)
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
I also think you may be running a version that had a bug and lacked cleanup of those temp files
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
At work right now so will have more concrete details in a few hours from now
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
@abucci@anthony.buc.ci I will have a look but I suspect it has something to do with the open nature of the external endpoint. I closed this loophole recently due to other reasons myself.
@stigatle@yarn.stigatle.no Nice to see you still around (even if occasionally) 😅
Same hwre
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
@abucci@anthony.buc.ci Please update!
@abucci@anthony.buc.ci can you update to the latest revision in the main branch?
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
@abucci@anthony.buc.ci What revision are you running?
watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
@abucci@anthony.buc.ci Wer that’s a bug!
Are we over Crowdstrike yet? 🤔🤣 Have We forgotten about it? 😅
@bender@twtxt.net This is sadly all too true 🤣
@hacker-news-newest@feeds.twtxt.net this reminds me I need to seriously consider exploring the idea of replacing my Plex set up with Jellyfin
@lyse@lyse.isobeef.org so in other words, their own entire sections of global industries that are using this rubbish crowd, strike antivirus/endpoint detection, piece of crap that are infection effectively in violation of the terms of conditions of the service? 🤔
That’s some good sleuth thing that @lyse@lyse.isobeef.org 🙇♂️
@bender@twtxt.net sometimes having the open honest and transparent conversation and discussion is more valuable than anything to be individually gained. 😅 it helps us learn!
@movq@www.uninformativ.de Who says it’s bad stuff or negativity? These are lessons to be learned and things to gain experiences from right?
@movq@www.uninformativ.de Yeah pretty much ooops 🤦♂️
@movq@www.uninformativ.de Haha 🤣 I might have liked a few, hell even agreed with you on some 🤣
Some bad code just broke a billion Windows machines - YouTube – This is a really good accurate and comical take on what happened with this whole Crowdstrike global fuck up.
@bender@twtxt.net i’m thinking it’s probably worth to try and implement both features as use for configurable preferences just so that we can see what it’s like. 🤣 The default, of course would be the regular pagination that exists now.
@lyse@lyse.isobeef.org Heh 😀 This is why we’ve always implemented features that have differing opinions as either user options, configuration or feature gated 🤣
@lyse@lyse.isobeef.org So you neither like “Click to Load” nor “Infinite Scroll” eh? I mean I sort of get it, infinite scroll is kind of annoying as you never really know wtf is going on. Click to Load misses out on “how many pages”, “what page you’re on”, etc. But is it really that bad? 🤔
Hmmm according to this you are right! I wonder what kind of crypto they use?
@bender@twtxt.net Messages (formally known as iMessages) has always lacked e2e encryption though and often falls back to using SMS which is worse for security and privacy. – OTOH this might be all changing for the better with Google lounging for a new standard? 🤔 (Although I do have to wonder: “What’s in it for Google?”)
@bender@twtxt.net Pretty much. One post per feed.
@lyse@lyse.isobeef.org My god ! 🤦♂️
@lyse@lyse.isobeef.org What about “Click to Load”?
I haven’t figured out a way of integrating the couch with the search archive yet…
@bender@twtxt.net to be clear it’ll be infinite scroll up until you reach the end of the cache 🤣
Thinking about building infinite scroll for yarn.
@slashdot@feeds.twtxt.net it’s amazing that anyone thinks that these so-called instructions in large language models are anything close to what you would consider instructions or even remotely intelligible.
📣 NEW: Added a new feature for pod operators to optionally configure. Compact Front Page.
When enabled will display only one post per feed on the unauthenticated Discover view (the front page).
@cncf@feeds.twtxt.net Yes yes, but so what?! What’s like saying the importance of having unit tests…
@xuu@txt.sour.is Your pod is behaving much better now right? Any other issues aside from the Edit problem? 🤔
@abucci@anthony.buc.ci yeah, according to this article, it stems from what appears to be a global monopoly in the so-called cyber security industry.
⨁ Follow button on their profile page or use the Follow form and enter a Twtxt URL. You may also find other feeds of interest via Feeds. Welcome! 🤗
@abucci@anthony.buc.ci do it now. 😅
@movq@www.uninformativ.de The first time I tried matrix I didn’t like it either. I tried to join the public room and waited and waited and waited and waited for like 20+ minutes before I could even see anything in the room or even talk in the room! I just find the protocol to be so fucking overcomplicated for what it actually is meant to do.
I assume not necessarily to fit an entire program in a 3 x 5 card though.
@abucci@anthony.buc.ci straight up in the first minute I was talking he describes basically the concept of being able to describe your idea a 3 x 5 card. I think that’s rather elegant.
@xuu@txt.sour.is what the fuck is Krack anywa? 🤣 sounds like a scam.
@movq@www.uninformativ.de isn’t this why management should never be in charge of fucking technical decisions?
@lyse@lyse.isobeef.org out of interest what is the self description of crowd strike? What the fuck is it even?
docker build without any --build-arg VERSION= or --build-arg COMMIT= there was no version information in the built binary and bundled assets. Therefore cache busting would not work as expected. When introducing htmx and hyperscript to create a UI/UX SPA-like experience, this is when things fell apart a bit for you. I think....
@xuu@txt.sour.is well, it does right but only with special LD flags?
@movq@www.uninformativ.de Yeah didn’t affect us either
@slashdot@feeds.twtxt.net So this is this “Crowdstrike Outage” my wife told me about a few mins ago hmmm 🧐 wtf is going on here? 😅
I re-blocked this feed (if anyone really wants it, please let me know!) and nuked it from the cache. It’s full of garbage AFICT 🤦♂️
@movq@www.uninformativ.de I couldn’t agree more!I have many projects and tools like this that are just “done”
Hmm I thought we blocked this rubbish anonymous gopher feed before? Has it change its location? 🤔
@bender@twtxt.net Yeah I dunno wtf this feed is 🤣 Looks like a gopher feed. Either their client is fucked up, or this is just pure spam 🤔 Anyway found some bugs in yarnd that I should fix because of this feed.
@bender@twtxt.net hmmm