I think I know how to solve this. I just have two sessions per week (my Saturday). One at around 0400-0500 UTC and another one a few hours later.

@markwylde@twtxt.net Sounds about right. I think there are some toggles to make it go a bit faster (but less data durability) and I’m sure we can improve this too 👌

@markwylde@twtxt.net Yeah we have to be clear on what you behaviour you actually want here… So for example Read Solomon Erasure Encoding basically describe what you’re describing here whereby you can recover data from N/M nodes. I feel like this is something we could built that sits in front of a Bitraft cluster (or single node)? 🤔

@mckinley@twtxt.net No problems 👌 Timezones are hard 🤣

@mckinley@twtxt.net is right. There are three different things going on here.

• The Cache limits what it hold “in-memory” by Size per Feed
  
• The Cache limits what it holds “in-memory” by TTL
  
• Old Twts are Archived to disk.
  

At some point soon™ we will have integrated search on all Yarn.social pods (running yarnd) that will give us the capability of two things:

• Full Text / Keyword / Tag search (searching the archive)
  
• Pulling up older Twts for a Conversation view (from the archive)

Actually @darch@neotxt.dk has gone to bed 😅 So usual time of 0500 UTC? 🤔

Shall we do the Yarn/social weekly call now since it appears most people are awake and available?

@markwylde@twtxt.net If you could benchmark this that would be wonderful! 👌 – Also reading your thought son “Sharding”, I think you might be slightly confused, because what you just described is essentially “High Availability”, and not Sharding.

In fact Bitraft already has this anyway. It fully supports forming a High Availability Cluster.

Does this make sense? 🤔 In fact I probably wouldn’t have any Metadata on your Github feed side at all, just delete it and re-sync (after fixing the dupes), and then help me add some additional features to add additional KV pairs to your feed’s Metadata in yarnd (the backend for Yarn.social) 🙏

@rsdoiel@twtxt.net Where is your Github feed you’re mirroring? 🤔 The only reason for “dupes” is the way hashes are calculated. They are based on what’s called a “Hashing URI”, essentially the first # url = field in your feed’s metadata. I think (but would have to confirm) that if you’re trying to sync your Github hosted twtxt.txt feed to your twtxt.net feed (on my pod), if you have a # url = field in your Github side that isn’t the same value as your feed on your twtxt.net feed, then you’ll end up with dupes. You can have multiple # url = fields, but the first one is used as the “Hashing URI” and if you’re syncing, it must be the same on both sides.

@rsdoiel@twtxt.net Hmmm I think you might have run into a bug 🤣 I might need your help figuring this out 👌

@rsdoiel@twtxt.net Hmmm you shouldn’t be getting duplicates. Can you share how you’re doing this? I use the tool myself to backup my feeds 👌

@markwylde@twtxt.net No but I could do some testing and publish the results 👌

As for the sharding though… Let’s discuss this?

@lyse@lyse.isobeef.org Your cover pic seriously looks like a nice oil painting 😅

@lyse@lyse.isobeef.org No it’s not right but as I said on IRC I haven’t figured out a way to handle bad Following yet 🤣

@marado@twtxt.net Yes! 😅 Coming soon 🤗

@carsten@yarn.zn80.net very nice pics 👌

@jlj@twt.nfld.uk I think I’ve managed to fix bugs there recently 🤗

@off_grid_living@twtxt.net It’s being moved to a filter. It’s a current feature being developed and enables on my pod only 🤗

@off_grid_living@twtxt.net This is the poem:

Title: in the ocean waves

in the ocean waves I can see a dolphin swimming next to me.
and if listen I can here fish swimming in the currents near.
I can taste the salty air and fish jumping there.
there’s so much to see here and taste in this ocaen wave’s community.

@jlj@twt.nfld.uk It’s known to work for dozens of users actually 😆 What’s the error? 🤔

@lyse@lyse.isobeef.org I pronounce the same as @lyse@lyse.isobeef.org 👌

@movq@www.uninformativ.de I can’t live without my nice Bose 🤣

@hashrock@twtxt.net This is really cool! 👌 What feed is this pulling from? 🤔

Thays a bug 😅

@hashrock@twtxt.net LOL 🤣 Probably ripped it off our JavaScript library 🤦‍♂️

@hashrock@twtxt.net The fact that we can search and pull up media a Yarn user shares is probably all we need to pull this off whether it’s a view on the ‘yarndUI or a separate external app 👌

@rsdoiel@twtxt.net Hello! 👋 Welcome to Yarn.social 🤗

@markwylde@twtxt.net Bitraft is close 👌 but not sharded 😢

@carsten@yarn.zn80.net Good night 😴

@jlj@twt.nfld.uk I’ll try to put a fix in the next version 👌

Hmmm I’m open to whatever time/day works for more of the community 🤗

@mckinley@twtxt.net Shall we try for 0300 UTC? 🤔 @darch@neotxt.dk Anyone else interested in coming? 👋

@justamoment@twtxt.net D’oh! I do too 😅

Who’s coming to tomorrow’s Yarn.social weekly call? 🤔 There seems to be some desire to change the time, so which time?

• 0500 UTC (original time)
  
• 0000 UTC
  
• 0300 UTC
  
• Something else?

@darch@neotxt.dk Yeah maybe… Keen to hear some other points of view too 👌

yeah I haven’t committed it anyway because I realized that it wouldn’t work for Mobile so I think it’s a bad implementation 😢

@lyse@lyse.isobeef.org After this commit:

* 0d751800 2022-11-11 | Add FeatureFixCachedTwtersForArchivedFeeds (fix_cached_twters_for_archived_feeds) to avoid caching Twter objects for Archived Feeds (HEAD -> filter_and_lists, origin/filter_and_lists) [James Mills]

And blacklisting ^lyse\.isobeef\.org$ (because I couldn’t figure out how to go fix broken/bad Followings of Users on my pod easily) there should only be two Twter objects for you now, @lyse@lyse.isobeef.org and @lyse-backup@twtxt.net (as expected).

Hopefully we don’t run into any other weird edge cases 😅

@justamoment@twtxt.net I think so, but also your paste above is now empty for some reason? 🤔

@marado@tilde.pt To be honest, I normally just say to my wife (whenever I can encourage her to):

That’s a good one to post on Yarn

So I use the term post and Yarn (to refer to Yarn.social)

@stigatle@yarn.stigatle.no Hey 👋 So I guess after thinking about this on/off all day, I’d like to go back to what the problem is we’re trying to solve here? 🤔 Is the problem that you sometimes see Twts from feeds you’re not sure that you even follow? 🤔 There are only 2 cases where this can happen:

• You are following someone that replies to a feed that you don’t follow and whose Root Twt is missing from your Pod, so your Pod asks its peers if it has that Twt.
  
• You are following someone clearly having a discussion with one ore more folks you don’t follow and so you only see one side. TBD (this is not implemented yet)

@stigatle@yarn.stigatle.no @darch@neotxt.dk What about this? 🤔

I hate this 🤣

Oh but there is one problem with that… We don’t show an icon for a normal human / user 😅 On purpose too as per a comment from @mckinley@twtxt.net which I agreed with 👌

@darch@neotxt.dk That could work. Wanna give it a try and let us know? 🤔

@darch@neotxt.dk What about an icon?

@lyse@lyse.isobeef.org Yeah I dumped my cache to have a look 🤦‍♂️ Somehow someone or another has followed / fetched your archived feeds and those Twters are now cached 😢

I fixed one case already but this is different 😅

Looking into it… 👀

@abucci@anthony.buc.ci Toot Honk Tweet Twt Post Blog 🤣

@stigatle@twtxt.net Yeah I often have to do the same 😆 Soimds good to me 👌

@axodys@octobloc.xyz No worries 👌