For those curious, the archive on this pod had reached around ~22GB in size. I had to suck it down to my more powerful Mac Studio to clean it up and remove a bunch of junk. Then copy all the data back. This is what my local network traffic looked like for the last few hours 😱

And we’re back. Sorry about that 😅

@lyse@lyse.isobeef.org Hmmm I’m not sure sure I get what you’re getting at here. In order for this to be true, yarnd would have to be maliciously fabricating a Twt with the Hash D.

i.e: there must be two versions of the Twt in the feed.

@lyse@lyse.isobeef.org This is true. But the client MUST supply the original too! Or this doesn’t work 😢

If OTOH your client doesn’t store individual Twts in a cache/archive or some kind of database, then verification becomes quite hard and tedious. However I think of this as an implementation details. The spec should just call out that clients must validate/verify the edit request and the matching hash actually exists in that feed, not how the client should implement that.

@lyse@lyse.isobeef.org Yes you do. You keep both versions in your cache. They have different hashes. So you have Twt A, a client indicates Twt B is an edit of A, your client has already seen A and cached and archived it, now your client fetches B which is indicated of editing A. You cache/archive B as well, but now indicate in your display that B replaces A (maybe display, link both) or just display B or whatever. But essentially you now have both, but an indicator of one being an edit of the other.

The right thing to do here of course is to keep A in the “thread” but display B. Why? So the thread/chain doesn’t actually break or fork (forking is a natural consequence of editing, or is it the other way around? 🤔).

@lyse@lyse.isobeef.org I’m all for dropping delete btw, Or at least not making it mandatory, as-in “clients should” rather than “clients must”. But yes I agree, let’s explore all the possible ways this can be exploited (if at all).

@movq@www.uninformativ.de I think not.

What about edits of edits? Do we want to “chain” edits or does the latest edit simply win?

This gets too complicated if we start to support this kind of nonsense 🤣

@movq@www.uninformativ.de Thank you! 🙏

@lyse@lyse.isobeef.org Walk me through this? 🤔 I get what you’re saying, but I’m too stupid to be a “hacker” 🤣

But yes, at the end of the day if the edit request is invalid or cannot be verified, it should be ignored as treated as “malicious”.

@lyse@lyse.isobeef.org @movq@www.uninformativ.de So a client that has the idea of a cache/archive wouldn’t necessarily have to re-check that the Twt being marked as “edited” belongs to that feed or not, the client would already know that for sure. At least this is how yarnd works and I’m sure jenny can make similar assertions too.

@lyse@lyse.isobeef.org @falsifian@www.falsifian.org Contributions to search.twtxt.net, which runs yarns (not to be confused with yarnd) are always welcome 🤗 – I don’t have as much “spare time” as I used to due to the nature of my job (Staff Engineer); but I try to make improvements every now and again 💪

@falsifian@www.falsifian.org You make good points though, I made similar arguments about this too back in the day. Twtxt v2 / Yarn.social being at least ~4 years old now 😅

@falsifian@www.falsifian.org Do you have specifics about the GRPD law about this?

Would the GDPR would apply to a one-person client like jenny? I seriously hope not. If someone asks me to delete an email they sent me, I don’t think I have to honour that request, no matter how European they are.

I’m not sure myself now. So let’s find out whether parts of the GDPR actually apply to a truly decentralised system? 🤔

LOL 😂 This:

anyone could claim that some feed contained a certain message which was then removed again by just creating the hash over the fake message in said feed and invented timestamp themselves

I’d like to see a step-by-step reproduction of this. I don’t buy it 🤣

Admittedly yarnd had a few implementation security bugs, but I’m not sure this is actually possible, unless I’m missing something? 🤔

@david@collantes.us Very nice! 👍

@movq@www.uninformativ.de Ok 😅

@movq@www.uninformativ.de Hmmm not sure what I was thinking sorry 🤦‍♂️been a long day 😂

@movq@www.uninformativ.de Am I missing something? 😅

@movq@www.uninformativ.de Precisely 👌

@movq@www.uninformativ.de Is t it? You read each Twt and compute its hash. It’s a simple O(1) lookup of the hash in that feed or your cache/archive right?

👋 Reminder that next Saturday 28th September will be out monthly online meetup! Hope to see some/all of you there 👌

I’ll try to reproduce locally later tonight

@lyse@lyse.isobeef.org I don’t think this is true.

@lyse@lyse.isobeef.org No that’s never a problem because we really only want to “navigate” the web anyway not form threads of xonversation 🤣

@movq@www.uninformativ.de this approach also wouldn’t work and when that Feed gets archived so you’ll be forced to crawl archived feeds at that point.

The important bits missing from this summary (devil is in the details) are two requirements:

• Clients should order Twts by their timestamp.
  
• Clients must validate all edit and delete requests that the hash being indicated belongs to and came from that feed.
  
• Client should honour delete requests and delete Twts from their cache/archive.

@lyse@lyse.isobeef.org This is why hashes provide that level of integrity. The hash can be verified in the cache or archive as belonging to said feed.

@movq@www.uninformativ.de I think the order of the lines in a feed don’t matter as long as we can guarantee the order of Twts. Clients should already be ordering by Timestamp anyway.

@movq@www.uninformativ.de Pretry much 👌

@lyse@lyse.isobeef.org Sorry could you explain this sifferently?

Do you k ow what you clicked on before going back?

@eldersnake@we.loveprivacy.club Sweet thank you! 🙇‍♂️ I’ll merge this PR tonight I think.

@david@collantes.us I think we can!

e.g: Shutdown yarnd and cp -a yarn.db yarn.db.bak before testing this PR/branch.

Can I get someone like maybe @xuu@txt.sour.is or @abucci@anthony.buc.ci or even @eldersnake@we.loveprivacy.club – If you have some spare time – to test this yarnd PR that upgrades the Bitcask dependency for its internal database to v2? 🙏

VERY IMPORTANT If you do; Please Please Please backup your yarn.db database first! 😅 Heaven knows I don’t want to be responsible for fucking up a production database here or there 🤣

nevermind; I think this might be some changes internally in Go 1.23 and a dependency I needed to update 🤞

Can someone much smarter than me help me figure out a couple of newly discovered deadlocks in yarnd that I think have always been there, but only recently uncovered by the Go 1.23 compiler.

https://git.mills.io/yarnsocial/yarn/issues/1175

Location Addressing is fine in smaller or single systems. But when you’re talking about large decentralised systems with no single point of control (kind of the point) things like independable variable integrity become quite important.

What is being proposed as a counter to content-addressing is called location-addressing. Two very different approaches, both with pros/cons of course. But a local cannot be verified, the content cannot be be guaranteed to be authenticate in any way, you just have to implicitly trust that the location points to the right thing.

For example, without content-addressing, you’d never have been able to find let alone pull up that ~3yr old Twt of me (my very first), hell I’d even though I lost my first feed file or it became corrupted or something 🤣 – If that were the case, it would actually be possible to reconstruct the feed and verify every single Twt against the caches of all of you 🤣

@david@collantes.us I really thinks articles like this explain the benefits far better than I can.

@david@collantes.us Oh ! 🤦‍♂️

@david@collantes.us Witout including the content, it’s no longer really “content addressing” now is it? You’re essentially only addressing say nick+timestamp or url+timestamp.

Speaking of AI tech (sorry!); Just came across this really cool tool built by some engineers at Google™ (currently completely free to use without any signup) called NotebookLM 👌 Looks really good for summarizing and talking to document 📃

@eldersnake@we.loveprivacy.club Yeah I’m looking forward to that myself 🤣 It’ll be great to see where technology grow to a level of maturity and efficiency where you can run the tools on your own PC or Device and use it for what, so far, I’ve found it to be somewhat decent at; Auto-Complete, Search and Q&A.

@sorenpeter@darch.dk I really don’t think we can ignore the last ~3 years and a bit of this threading model working quite well for us as a community across a very diverse set of clients and platforms. We cannot just drop something that “mostly works just fine” for the sake of “simplicity”. We have to weight up all the options. There are very real benefits to using content addressing here that really IMO shouldn’t be disregarded so lightly that actually provide a lot of implicit value that users of various clients just don’t get to see. I’d recommend reading up on the ideas behind content addressing before simply dismissing the Twt Hash spec entirely, it wasn’t even written or formalised by me, but I understand how it works quite well 😅 The guy that wrote the spec was (is?) way smarter than I was back then, probably still is now 🤣

@falsifian@www.falsifian.org Right I see. Yeah maybe we want to avoid that 🤣 I do kind of tend to agree with @xuu@txt.sour.is in another thread that there isn’t actually anything wrong with our use of Blake2 at all really, but we may want to consider all our options.

@xuu@txt.sour.is I don’t think this is a lextwt problem tbh. Just the Markdown aprser that yarnd currently uses. twtxt2html uses Goldmark and appears to behave better 🤣

@xuu@txt.sour.is Long while back, I experimented with using similarity algorithms to detect if two Twts were similar enough to be considered an “Edit”.

Right I see what you mean @xuu@txt.sour.is – Can you maybe come up with a fully fleshed out proposal for this? 🤔 This will help solve the problem of hash collision that result from the Twt/hash space growing larger over time without us having to change anything about the way we construct hashes in the first place. We just assume spec compliant clients will just dynamically handle this as the space grows.

@xuu@txt.sour.is I think we never progressed this idea further because we weren’t sure how to tell if a hash collision would occur in the first place right? In other words, how does Client A know to expand a hash vs. Client B in a 100% decentralised way? 🤔

Plus these so-called “LLM”(s) have a pretty good grasp of the “shape” of language, so they appear to be quite intelligent or produce intelligible response (when they’re actually quite stupid really).

@eldersnake@we.loveprivacy.club You don’t get left behind at all 🤣 It’s hyped up so much, it’s not even funny anymore. Basically at this point (so far at least) I’ve concluded that all this GenAI / LLM stuff is just a fancy auto-complete and indexing + search reinvented 🤣

@bender@twtxt.net This is the different Markdown parsers being used. Goldmark vs. gomarkdown. We need to switch to Goldmark 😅

@quark@ferengi.one i’m guessing the quotas text should’ve been emphasized?

@slashdot@feeds.twtxt.net NahahahahHa 🤣 So glad I don’t use LinkedIn 🤦‍♂️

@falsifian@www.falsifian.org No u don’t sorry. But I tend to agree with you and I think if we continue to use hashes we should keep the remainder in mind as we choose truncation values of N

@falsifian@www.falsifian.org Mostly because Git uses it 🤣 Known attacks that would affect our use? 🤔

@xuu@txt.sour.is I don’t recall where that discussion ended up being though?

@bender@twtxt.net wut da fuq?! 🤣

@xuu@txt.sour.is you mean my original idea of basically just automatically detecting Twt edits from the client side?

@xuu@txt.sour.is this is where you would need to prove that the editor delete request actually came from that feed author. Hence why integrity is much more important here.

@falsifian@www.falsifian.org without supporting dudes properly though you’re running into GDP issues and the right to forget. 🤣 we’ve had pretty lengthy discussions about this in the past years ago as well, but we never came to a conclusion. We’re all happy with.

@movq@www.uninformativ.de it would work, you are right, however, it has drawbacks, and I think in the long term would create a new set of problems that we would also then have to solve.

@david@collantes.us Hah 🤣

@david@collantes.us We’ll get there soon™ 🔜

@david@collantes.us Hah Welcome back! 😅

Finally @lyse@lyse.isobeef.org ’s idea of updating metadata changes in a feed “inline” where the change happened (with respect to other Twts in whatever order the file is written in) is used to drive things like “Oh this feed now has a new URI, let’s use that from now on as the feed’s identity for the purposes of computing Twt hashes”. This could extend to # nick = as preferential indicators to clients as well as even other updates such as # description = – Not just # url =

Likewise we could also support delete:229d24612a2, which would indicate to clients that fetch the feed to delete any cached Twt matching the hash 229d24612a2 if the author wishes to “unpublish” that Twt permanently, rather than just deleting the line from the feed (which does nothing for clients really).

An alternate idea for supporting (properly) Twt Edits is to denoate as such and extend the meaning of a Twt Subject (which would need to be called something better?); For example, let’s say I produced the following Twt:

2024-09-18T23:08:00+10:00	Hllo World

And my feed’s URI is https://example.com/twtxt.txt. The hash for this Twt is therefore 229d24612a2:

$ echo -n "https://example.com/twtxt.txt\n2024-09-18T23:08:00+10:00\nHllo World" | sha1sum | head -c 11
229d24612a2

You wish to correct your mistake, so you make an amendment to that Twt like so:

2024-09-18T23:10:43+10:00	(edit:#229d24612a2) Hello World

Which would then have a new Twt hash value of 026d77e03fa:

$ echo -n "https://example.com/twtxt.txt\n2024-09-18T23:10:43+10:00\nHello World" | sha1sum | head -c 11
026d77e03fa

Clients would then take this edit:#229d24612a2 to mean, this Twt is an edit of 229d24612a2 and should be replaced in the client’s cache, or indicated as such to the user that this is the intended content.

@bender@twtxt.net Just replace the echo with something like pbpaste or similar. You’d just need to shell escape things like " and such. That’s all. Alternatives you can shove the 3 lines into a small file and cat file.txt | ...

With a SHA1 encoding the probability of a hash collision becomes, at various k (number of twts):

>>> import math
>>>
>>> def collision_probability(k, bits):
...     n = 2 ** bits  # Total unique hash values based on the number of bits
...     probability = 1 - math.exp(- (k ** 2) / (2 * n))
...     return probability * 100  # Return as percentage
...
>>> # Example usage:
>>> k_values = [100000, 1000000, 10000000]
>>> bits = 44  # Number of bits for the hash
>>>
>>> for k in k_values:
...     print(f"Probability of collision for {k} hashes with {bits} bits: {collision_probability(k, bits):.4f}%")
...
Probability of collision for 100000 hashes with 44 bits: 0.0284%
Probability of collision for 1000000 hashes with 44 bits: 2.8022%
Probability of collision for 10000000 hashes with 44 bits: 94.1701%
>>> bits = 48
>>> for k in k_values:
...     print(f"Probability of collision for {k} hashes with {bits} bits: {collision_probability(k, bits):.4f}%")
...
Probability of collision for 100000 hashes with 48 bits: 0.0018%
Probability of collision for 1000000 hashes with 48 bits: 0.1775%
Probability of collision for 10000000 hashes with 48 bits: 16.2753%
>>> bits = 52
>>> for k in k_values:
...     print(f"Probability of collision for {k} hashes with {bits} bits: {collision_probability(k, bits):.4f}%")
...
Probability of collision for 100000 hashes with 52 bits: 0.0001%
Probability of collision for 1000000 hashes with 52 bits: 0.0111%
Probability of collision for 10000000 hashes with 52 bits: 1.1041%
>>>

If we adopted this scheme, we could have to increase the no. of characters (first N) from 11 to 12 and finally 13 as we approach globally larger enough Twts across the space. I think at least full crawl/scrape it was around ~500k (maybe)? https://search.twtxt.net/ says only ~99k

@quark@ferengi.one My money is on a SHA1SUM hash encoding to keep things much simpler:

$ echo -n "https://twtxt.net/user/prologic/twtxt.txt\n2020-07-18T12:39:52Z\nHello World! 😊" | sha1sum | head -c 11
87fd9b0ae4e

I think it was a mistake to take the last n base32 encoded characters of the blake2b 256bit encoded hash value. It should have been the first n. where n is >= 7

Taking the last n characters of a base32 encoded hash instead of the first n can be problematic for several reasons:

1. Hash Structure: Hashes are typically designed so that their outputs have specific statistical properties. The first few characters often have more entropy or variability, meaning they are less likely to have patterns. The last characters may not maintain this randomness, especially if the encoding method has a tendency to produce less varied endings.

2. Collision Resistance: When using hashes, the goal is to minimize the risk of collisions (different inputs producing the same output). By using the first few characters, you leverage the full distribution of the hash. The last characters may not distribute in the same way, potentially increasing the likelihood of collisions.

3. Encoding Characteristics: Base32 encoding has a specific structure and padding that might influence the last characters more than the first. If the data being hashed is similar, the last characters may be more similar across different hashes.

4. Use Cases: In many applications (like generating unique identifiers), the beginning of the hash is often the most informative and varied. Relying on the end might reduce the uniqueness of generated identifiers, especially if a prefix has a specific context or meaning.

In summary, using the first n characters generally preserves the intended randomness and collision resistance of the hash, making it a safer choice in most cases.

@quark@ferengi.one Bloody good question 🙋 God only knows 🤣

@bender@twtxt.net Welcome! 🤗

@movq@www.uninformativ.de Haha 😝

What I was referring to in the OP: Sometimes I check the workphone simply out of curiosity. 😂

@movq@www.uninformativ.de Fair 👌

Current Twt Hash spec and probability of hash collision:

The probability of a Twt Hash collision depends on the size of the hash and the number of possible values it can take. For the Twt Hash, which uses a Blake2b 256-bit hash, Base32 encoding, and takes the last 7 characters, the space of possible hash values is significantly reduced.

1. Base32 encoding: Each character in the Base32 encoding represents 5 bits of information (since ( 2^5 = 32 )).
   
2. 7 characters: With 7 characters, the total number of possible hashes is:
   [
 32^7 = 3,518,437,208
 ]
   This gives about 3.5 billion possible hash values.
   

The probability of a hash collision depends on the number of hashes generated and can be estimated using the Birthday Paradox. The paradox tells us that collisions are more likely than expected when hashing a large number of items.

The approximate formula for the probability of at least one collision after generating n hashes is:
[
P(\text{collision}) \approx 1 - e^{-\frac{n^2}{2M}}
]
Where:

• (n) is the number of generated Twt Hashes.
  
• (M = 32^7 = 3,518,437,208) is the total number of possible hash values.
  

For practical purposes, here are some example probabilities for different numbers of hashes (n):

• For 1,000 hashes:
  [
 P(\text{collision}) \approx 1 - e^{-\frac{1000^2}{2 \cdot 3,518,437,208}} \approx 0.00014 \, \text{(0.014%)}
  ]
  
• For 10,000 hashes:
  [
 P(\text{collision}) \approx 1 - e^{-\frac{10000^2}{2 \cdot 3,518,437,208}} \approx 0.14 \, \text{(14%)}
  ]
  
• For 100,000 hashes:
  [
 P(\text{collision}) \approx 1 - e^{-\frac{100000^2}{2 \cdot 3,518,437,208}} \approx 0.999 \, \text{(99.9%)}
  ]
  

• For small to moderate numbers of hashes (up to around 1,000–10,000), the collision probability is quite low.
  
• However, as the number of Twts grows (above 100,000), the likelihood of a collision increases significantly due to the relatively small hash space (3.5 billion).

@quark@ferengi.one Add here:

* a0826a65 - Add debug sub-command to yarnc (7 weeks ago) <James Mills>

I’d recommend a git pull && make build

$ echo -n "https://twtxt.net/user/prologic/twtxt.txt\n2020-07-18T12:39:52Z\nHello World! 😊" | sha1sum | head -c 11
87fd9b0ae4e

$ echo -n "https://twtxt.net/user/prologic/twtxt.txt\n2020-07-18T12:39:52Z\nHello World! 😊" | sha256sum | base32 | tr -d '=' | tr 'A-Z' 'a-z' | tail -c 12
tdqmjaeawqu

Just experimenting…

$ echo -n "https://twtxt.net/user/prologic/twtxt.txt\n2020-07-18T12:39:52Z\nHello World! 😊" | sha256sum | base64 | tr -d '=' | tail -c 12
NWY4MSAgLQo

It would appear that the blake2b 256bit digest algorithm is no longer supported by the openssl tool, however blake2s256 is; I’m not sure why 🤔

$ echo -n "https://twtxt.net/user/prologic/twtxt.txt\n2020-07-18T12:39:52Z\nHello World! 😊" | openssl dgst -blake2s256 -binary | base32 | tr -d '=' | tr 'A-Z' 'a-z' | tail -c 7
zq4fgq

Obviously produce the wrong hash, which should be o6dsrga as indicated by the yarnc hash utility:

$ yarnc hash -u https://twtxt.net/user/prologic/twtxt.txt -t 2020-07-18T12:39:52Z "Hello World! 😊"
o6dsrga

But at least the shell pipeline is “correct”.

FWIW the standard UNIX tools for Blake2b are openssl and b2sum – Just trying to figure out how to make a shell pipeline again (if you really want that); as tools keep changing god damnit 🤣

@quark@ferengi.one Do you mean something like this?

$ ./yarnc debug ~/Public/twtxt.txt | tail -n 1
kp4zitq 2024-09-08T02:08:45Z	(#wsdbfna) @<aelaraji https://aelaraji.com/twtxt.txt> My work has this thing called "compressed work", where you can **buy** extra time off (_as much as 4 additional weeks_) per year. It comes out of your pay though, so it's not exactly a 4-day work week but it could be useful, just haven't tired it yet as I'm not entirely sure how it'll affect my net pay

@quark@ferengi.one Watch TV at the moment so unsure about “out of the box tools” but there are 3 implementations on the spec page https://dev.twtxt.net/doc/twthashextension.html

@aelaraji@aelaraji.com Sometimes that’s the most valuable though! Feedback!

I’ve been using Codeium too the last week or so ! It’s pretty good and like @xuu@txt.sour.is said is a pretty desent Junior assistant, it helps me write good docs and the tab completion is amazing!

It of course completely sucks at doing anything “intelligent” or complex, but if you just use it as a fancier auto complete it’s actually half way decent 👌

@quark@ferengi.one I admit I find the general “click here to share blah” generally wasteful, useless and unengaging really. Not just Wordle.

I admittedly however, I’ve been guilty of doing this sometimes myself. 🤦‍♂️ sometimes though I think it’s OK to show your achievements. 👌

@quark@ferengi.one LOL 😂 No worries! 😉

@quark@ferengi.one All fixed! 🥳 pleasure doing business with y’all. 🤣

@lyse@lyse.isobeef.org Just as an aside, shouldn’t you assume utf-8 anyway these days if not specified? 🤔 I mean basically everything almost always uses utf-8 encoding right? 😅

@quark@ferengi.one Okay fair 👌

@movq@www.uninformativ.de So basically on-call, but you don’t get paid for it? 🤔

@quark@ferengi.one aye aye captain 🤣