@eldersnake@we.loveprivacy.club Yeah I’m looking forward to that myself 🤣 It’ll be great to see where technology grow to a level of maturity and efficiency where you can run the tools on your own PC or Device and use it for what, so far, I’ve found it to be somewhat decent at; Auto-Complete, Search and Q&A.

@prologic@twtxt.net That’s definitely a little less depressing, when thinking of it that way 🤣 Be interesting when the hype dies down.

I’m not the biggest Apple fan around, but that is pretty awesome.

i found this site by searching “thing-fish”

@sorenpeter@darch.dk I really don’t think we can ignore the last ~3 years and a bit of this threading model working quite well for us as a community across a very diverse set of clients and platforms. We cannot just drop something that “mostly works just fine” for the sake of “simplicity”. We have to weight up all the options. There are very real benefits to using content addressing here that really IMO shouldn’t be disregarded so lightly that actually provide a lot of implicit value that users of various clients just don’t get to see. I’d recommend reading up on the ideas behind content addressing before simply dismissing the Twt Hash spec entirely, it wasn’t even written or formalised by me, but I understand how it works quite well 😅 The guy that wrote the spec was (is?) way smarter than I was back then, probably still is now 🤣

Estava tentado em escolher outra coisa para esta #musiquinta , mas seria enganar-me. Não consigo pensar em #covers sem me vir esta logo imediatamente à cabeça, é como se estivesse enfeitiçado…

Marilyn Manson - I Put A Spell On You
https://youtu.be/4qkrsodwJGY

@quark@ferengi.one It does not. That is why I’m advocating for not using hashes for treads, but a simpler link-back scheme.

@falsifian@www.falsifian.org Right I see. Yeah maybe we want to avoid that 🤣 I do kind of tend to agree with @xuu@txt.sour.is in another thread that there isn’t actually anything wrong with our use of Blake2 at all really, but we may want to consider all our options.

@xuu@txt.sour.is I don’t think this is a lextwt problem tbh. Just the Markdown aprser that yarnd currently uses. twtxt2html uses Goldmark and appears to behave better 🤣

@xuu@txt.sour.is Long while back, I experimented with using similarity algorithms to detect if two Twts were similar enough to be considered an “Edit”.

Right I see what you mean @xuu@txt.sour.is – Can you maybe come up with a fully fleshed out proposal for this? 🤔 This will help solve the problem of hash collision that result from the Twt/hash space growing larger over time without us having to change anything about the way we construct hashes in the first place. We just assume spec compliant clients will just dynamically handle this as the space grows.

@xuu@txt.sour.is I think we never progressed this idea further because we weren’t sure how to tell if a hash collision would occur in the first place right? In other words, how does Client A know to expand a hash vs. Client B in a 100% decentralised way? 🤔

Plus these so-called “LLM”(s) have a pretty good grasp of the “shape” of language, so they appear to be quite intelligent or produce intelligible response (when they’re actually quite stupid really).

@eldersnake@we.loveprivacy.club You don’t get left behind at all 🤣 It’s hyped up so much, it’s not even funny anymore. Basically at this point (so far at least) I’ve concluded that all this GenAI / LLM stuff is just a fancy auto-complete and indexing + search reinvented 🤣

Getting a little sick of AI this, AI that. Yes I’ll be left behind while everyone else jumps on the latest thing, but I’m not sure I care.

Oh. looks like its 4 chars. git show 64bf

@prologic@twtxt.net where was that idea?

i feel like we should isolate a subset of markdown that makes sense and built it into lextwt. it already has support for links and images. maybe basic formatting bold, italic. possibly block quote and bullet lists. no tables or footnotes

the stem matching is the same as how GIT does its branch hashes. i think you can stem it down to 2 or 3 sha bytes.

if a client sees someone in a yarn using a byte longer hash it can lengthen to match since it can assume that maybe the other client has a collision that it doesnt know about.

@prologic@twtxt.net the basic idea was to stem the hash.. so you have a hash abcdef0123456789... any sub string of that hash after the first 6 will match. so abcdef, abcdef012, abcdef0123456 all match the same. on the case of a collision i think we decided on matching the newest since we archive off older threads anyway. the third rule was about growing the minimum hash size after some threshold of collisions were detected.

@prologic@twtxt.net Wikipedia claims sha1 is vulnerable to a “chosen-prefix attack”, which I gather means I can write any two twts I like, and then cause them to have the exact same sha1 hash by appending something. I guess a twt ending in random junk might look suspcious, but perhaps the junk could be worked into an image URL like

Image

git only uses sha1 because they’re stuck with it: migrating is very hard. There was an effort to move git to sha256 but I don’t know its status. I think there is progress being made with Game Of Trees, a git clone that uses the same on-disk format.

I can’t imagine any benefit to using sha1, except that maybe some very old software might support sha1 but not sha256.

@bender@twtxt.net This is the different Markdown parsers being used. Goldmark vs. gomarkdown. We need to switch to Goldmark 😅

Apple A16 SoC Now Manufactured In Arizona
“Apple has begun manufacturing its A16 SoC at the newly-opened TSCM Fab 21 in Arizona,” writes Slashdot reader NoMoreACs. AppleInsider reports: According to sources of Tim Culpan, Phase 1 of TSMC’s Fab 21 in Arizona is making the A16 SoC of the iPhone 14 Pro in “small, but significant, numbers. The production is largely a test for the facility at this stage, but more production is expected … ⌘ Read more

@quark@ferengi.one i’m guessing the quotas text should’ve been emphasized?

@slashdot@feeds.twtxt.net NahahahahHa 🤣 So glad I don’t use LinkedIn 🤦‍♂️

@falsifian@www.falsifian.org No u don’t sorry. But I tend to agree with you and I think if we continue to use hashes we should keep the remainder in mind as we choose truncation values of N

@falsifian@www.falsifian.org Mostly because Git uses it 🤣 Known attacks that would affect our use? 🤔

@xuu@txt.sour.is I don’t recall where that discussion ended up being though?

@bender@twtxt.net wut da fuq?! 🤣

@xuu@txt.sour.is you mean my original idea of basically just automatically detecting Twt edits from the client side?

@xuu@txt.sour.is this is where you would need to prove that the editor delete request actually came from that feed author. Hence why integrity is much more important here.

@falsifian@www.falsifian.org without supporting dudes properly though you’re running into GDP issues and the right to forget. 🤣 we’ve had pretty lengthy discussions about this in the past years ago as well, but we never came to a conclusion. We’re all happy with.

@movq@www.uninformativ.de it would work, you are right, however, it has drawbacks, and I think in the long term would create a new set of problems that we would also then have to solve.

@david@collantes.us Hah 🤣

@prologic@twtxt.net :-D Thanks! Things can come in cycles, right? This is simply another one. Another cycle, more personal than the other “alter egos”.

@david@collantes.us We’ll get there soon™ 🔜

@david@collantes.us Hah Welcome back! 😅

@aelaraji@aelaraji.com hey, hey! You are my very first reply! 👋🏻 Cheers!

@david@collantes.us “Hello back” from the other corner of the world! 🫡

Incredibly upset—more than you could imagine—because I already made the first mistake, and corrected it (but twtxt.net got it on it’s cache, ugh!) :‘-( . Can’t wait for editing to become a reality!

Alright. My first mentions—which were picked not so randomly, LOL—are @prologic@twtxt.net, @lyse@lyse.isobeef.org, and @movq@www.uninformativ.de. I am also posting my first image too, which you see below. That’s my neighbourhood, in a “winter” day. Hopefully @prologic@twtxt.net will add my domain to his allowed list, so that the image (and any other further) renders.

Image

Alright, announce_me set to true. Now, who do I pick to be my first mention? Decisions, decisions. Next twtxt will have my first mention(s). :-)

I have configured my twtxt.txt as simple as possible. I have setup a publish_command on jenny. Hopefully all works fine, and I am good to go. Next will be setting the announce_me to true. Here we go!

Everything starts at a “hello world”. At least around these parts; the nerdy parts.

@sorenpeter@darch.dk hmm, how does your client handles “a little editing”? I am sure threads would break just as well. 😉

@prologic@twtxt.net, there is a parser bug on parent. Specifically on this portion:

"*If twtxt/Yarn was to grow bigger, then this would become a concern again. *But even Mastodon allows editing*, so how
+much of a problem can it really be? 😅*"

@movq@www.uninformativ.de going a little sideways on this, “*If twtxt/Yarn was to grow bigger, then this would become a concern again. But even Mastodon allows editing, so how much of a problem can it really be? 😅*”, wouldn’t it preparing for a potential (even if very, very, veeeeery remote) growth be a good thing? Mastodon signs all messages, keeps a history of edits, and it doesn’t break threads. It isn’t a problem there.😉 It is here.

I think keeping hashes is a must. If anything for that “feels good” feeling.

@movq@www.uninformativ.de Agreed that hashes have a benefit. I came up with a similar example where when I twted about an 11-character hash collision. Perhaps hashes could be made optional somehow. Like, you could use the “replyto” idea and then additionally put a hash somewhere if you want to lock in which version of the twt you are replying to.

There is nothing wrong with how we currently run a diff to see what has been removed. if i build a merkle tree off all the twt hashes in a feed i can use that to verify a twt should be in a feed or not. and gossip that to my peers.

So.. basically a rehash of the email “unsend” requests? What if i was to make a (delete: 5vbi2ea) .. would it delete someone elses twt?

isn’t the benefit of blake2b that it is a more efficient algo than sha1 and has the same or similar entropy to sha3? i thought we had partially solved this with some type of expanding hash size? additionally we could increase bit density by using base36 or base64/url-safe…

Hey, @movq@www.uninformativ.de, a tiny thing to add to jenny, a -v switch. That way when you twtxt “That’s an older format that was used before jenny version v23.04”, I can go and run jenny -v, and “duh!” myself on the way to a git pull. :-D

@movq@www.uninformativ.de ooooh, nice! commit 62a2b7735749f2ff3c9306dd984ad28f853595c5:

Crawl archived feeds in –fetch-context

Like, very much! :-)

@movq@www.uninformativ.de to paraphrase US Presidents speech on each State of the Union, “the State of the Jenny is strong!” :-D As for the potential upcoming changes, there has to be a knowledgeable head honcho that will agglomerate and coalesce, and guide onto the direction that will be taken. All that with the strong input from the developers that will be implementing the changes, and a lesser (but not less valuable) input from users.

@lyse@lyse.isobeef.org I call upon the services of the @yarn_police@twtxt.net to further investigate this oddness!

@quark@ferengi.one Oh, sure, it would be nice if edits didn’t break threads. I was just pondering the circumstances under which I get annoyed about data being irrecoverably deleted or otherwise lost.

@falsifian@www.falsifian.org “I don’t really mind if the twt gets edited before I even fetch it.”, right, that’s never the problem. Editing a twtxt before anyone fetches it isn’t even editing, right? :-P The problem we are trying to fix is the havoc is causes editing twtxts that have already been replied to, often ad nauseam. That’s the real problem.

@quark@ferengi.one I don’t really mind if the twt gets edited before I even fetch it. I think it’s the idea of my computer discarding old versions it’s fetched, especially if it’s shown them to me, that bugs me.

But I do like @movq@www.uninformativ.de’s suggestion on this thread that feeds could contain both the original and the edited twt. I guess it would be up to the author.

@lyse@lyse.isobeef.org now, how am I not surprised at that reply?! Hahahahaha!

@falsifian@www.falsifian.org that would be problematic to do on a fully decentralised system. I am not disagreeing, though. That’s the reason I have stopped editing twtxts. I strive to own mistakes, as minor as they might be. Now, if trail editing can be accomplished, I am all for it!

@quark@ferengi.one None. I like being able to see edit history for the same reason.

@movq@www.uninformativ.de You’re right! switching from zsh to bash gave me the same result zq4fgq Thanks!

@falsifian@www.falsifian.org what would the difference be between an edit the changes everything on the original twtxt, and a delete?

@prologic@twtxt.net Why sha1 in particular? There are known attacks on it. sha256 seems pretty widely supported if you’re worried about support.

@prologic@twtxt.net I wouldn’t want my client to honour delete requests. I like my computer’s memory to be better than mine, not worse, so it would bug me if I remember seeing something and my computer can’t find it.

@prologic@twtxt.net

There’s a simple reason all the current hashes end in a or q: the hash is 256 bits, the base32 encoding chops that into groups of 5 bits, and 256 isn’t divisible by 5. The last character of the base32 encoding just has that left-over single bit (256 mod 5 = 1).

So I agree with #3 below, but do you have a source for #1, #2 or #4? I would expect any lack of variability in any part of a hash function’s output would make it more vulnerable to attacks, so designers of hash functions would want to make the whole output vary as much as possible.

Other than the divisible-by-5 thing, my current intuition is it doesn’t matter what part you take.

1. Hash Structure: Hashes are typically designed so that their outputs have specific statistical properties. The first few characters often have more entropy or variability, meaning they are less likely to have patterns. The last characters may not maintain this randomness, especially if the encoding method has a tendency to produce less varied endings.

2. Collision Resistance: When using hashes, the goal is to minimize the risk of collisions (different inputs producing the same output). By using the first few characters, you leverage the full distribution of the hash. The last characters may not distribute in the same way, potentially increasing the likelihood of collisions.

3. Encoding Characteristics: Base32 encoding has a specific structure and padding that might influence the last characters more than the first. If the data being hashed is similar, the last characters may be more similar across different hashes.

4. Use Cases: In many applications (like generating unique identifiers), the beginning of the hash is often the most informative and varied. Relying on the end might reduce the uniqueness of generated identifiers, especially if a prefix has a specific context or meaning.

@aelaraji@aelaraji.com odd, I ran it under Ubuntu 24.04, and got the same result as @prologic@twtxt.net (which is on macOS), zq4fgq.

@prologic@twtxt.net I ran the same command and got an even different result xD

~ » echo -n "https://twtxt.net/user/prologic/twtxt.txt\n2020-07-18T12:39:52Z\nHello World! 😊" | openssl dgst -blake2s256 -binary | base32 | tr -d '=' | tr 'A-Z' 'a-z' | tail -c 7
p44j3q

LinkedIn Is Training AI on User Data Before Updating Its Terms of Service
An anonymous reader shares a report: LinkedIn is using its users’ data for improving the social network’s generative AI products, but has not yet updated its terms of service to reflect this data processing, according to posts from various LinkedIn users and a statement from the company to 404 Media. Instead, the company says it … ⌘ Read more

@prologic@twtxt.net I just realised the jenny also does what I want, as of latest commit. Simply use jenny --debug-feed <feed url>, and it will do what I wanted too!

@movq@www.uninformativ.de alright, fair, and interesting. I was expecting them to be all the same (format wise), but it doesn’t matter, for sure, as it works just fine. Thanks!

I have noticed that twtxt timestamps differ. For example:

• @prologic@twtxt.net (and I assume any Yarn user)
  2024-09-18T13:16:17Z
  
• @lyse@lyse.isobeef.org
  2024-09-17T21:15:00+02:00
  
• @aelaraji@aelaraji.com (and @movq@www.uninformativ.de, and me)
  2024-09-18T05:43:13+00:00
  

So, which is right, or best?

I came across this Gallery Theme for Hugo, and @lyse@lyse.isobeef.org immediately came to mind. I think it would be a very fitting theme to use for all your photos, Lyse!

Finally @lyse@lyse.isobeef.org ’s idea of updating metadata changes in a feed “inline” where the change happened (with respect to other Twts in whatever order the file is written in) is used to drive things like “Oh this feed now has a new URI, let’s use that from now on as the feed’s identity for the purposes of computing Twt hashes”. This could extend to # nick = as preferential indicators to clients as well as even other updates such as # description = – Not just # url =

Likewise we could also support delete:229d24612a2, which would indicate to clients that fetch the feed to delete any cached Twt matching the hash 229d24612a2 if the author wishes to “unpublish” that Twt permanently, rather than just deleting the line from the feed (which does nothing for clients really).

An alternate idea for supporting (properly) Twt Edits is to denoate as such and extend the meaning of a Twt Subject (which would need to be called something better?); For example, let’s say I produced the following Twt:

2024-09-18T23:08:00+10:00	Hllo World

And my feed’s URI is https://example.com/twtxt.txt. The hash for this Twt is therefore 229d24612a2:

$ echo -n "https://example.com/twtxt.txt\n2024-09-18T23:08:00+10:00\nHllo World" | sha1sum | head -c 11
229d24612a2

You wish to correct your mistake, so you make an amendment to that Twt like so:

2024-09-18T23:10:43+10:00	(edit:#229d24612a2) Hello World

Which would then have a new Twt hash value of 026d77e03fa:

$ echo -n "https://example.com/twtxt.txt\n2024-09-18T23:10:43+10:00\nHello World" | sha1sum | head -c 11
026d77e03fa

Clients would then take this edit:#229d24612a2 to mean, this Twt is an edit of 229d24612a2 and should be replaced in the client’s cache, or indicated as such to the user that this is the intended content.

@bender@twtxt.net Just replace the echo with something like pbpaste or similar. You’d just need to shell escape things like " and such. That’s all. Alternatives you can shove the 3 lines into a small file and cat file.txt | ...

With a SHA1 encoding the probability of a hash collision becomes, at various k (number of twts):

>>> import math
>>>
>>> def collision_probability(k, bits):
...     n = 2 ** bits  # Total unique hash values based on the number of bits
...     probability = 1 - math.exp(- (k ** 2) / (2 * n))
...     return probability * 100  # Return as percentage
...
>>> # Example usage:
>>> k_values = [100000, 1000000, 10000000]
>>> bits = 44  # Number of bits for the hash
>>>
>>> for k in k_values:
...     print(f"Probability of collision for {k} hashes with {bits} bits: {collision_probability(k, bits):.4f}%")
...
Probability of collision for 100000 hashes with 44 bits: 0.0284%
Probability of collision for 1000000 hashes with 44 bits: 2.8022%
Probability of collision for 10000000 hashes with 44 bits: 94.1701%
>>> bits = 48
>>> for k in k_values:
...     print(f"Probability of collision for {k} hashes with {bits} bits: {collision_probability(k, bits):.4f}%")
...
Probability of collision for 100000 hashes with 48 bits: 0.0018%
Probability of collision for 1000000 hashes with 48 bits: 0.1775%
Probability of collision for 10000000 hashes with 48 bits: 16.2753%
>>> bits = 52
>>> for k in k_values:
...     print(f"Probability of collision for {k} hashes with {bits} bits: {collision_probability(k, bits):.4f}%")
...
Probability of collision for 100000 hashes with 52 bits: 0.0001%
Probability of collision for 1000000 hashes with 52 bits: 0.0111%
Probability of collision for 10000000 hashes with 52 bits: 1.1041%
>>>

If we adopted this scheme, we could have to increase the no. of characters (first N) from 11 to 12 and finally 13 as we approach globally larger enough Twts across the space. I think at least full crawl/scrape it was around ~500k (maybe)? https://search.twtxt.net/ says only ~99k

@quark@ferengi.one My money is on a SHA1SUM hash encoding to keep things much simpler:

$ echo -n "https://twtxt.net/user/prologic/twtxt.txt\n2020-07-18T12:39:52Z\nHello World! 😊" | sha1sum | head -c 11
87fd9b0ae4e

I think it was a mistake to take the last n base32 encoded characters of the blake2b 256bit encoded hash value. It should have been the first n. where n is >= 7

Taking the last n characters of a base32 encoded hash instead of the first n can be problematic for several reasons:

1. Hash Structure: Hashes are typically designed so that their outputs have specific statistical properties. The first few characters often have more entropy or variability, meaning they are less likely to have patterns. The last characters may not maintain this randomness, especially if the encoding method has a tendency to produce less varied endings.

2. Collision Resistance: When using hashes, the goal is to minimize the risk of collisions (different inputs producing the same output). By using the first few characters, you leverage the full distribution of the hash. The last characters may not distribute in the same way, potentially increasing the likelihood of collisions.

3. Encoding Characteristics: Base32 encoding has a specific structure and padding that might influence the last characters more than the first. If the data being hashed is similar, the last characters may be more similar across different hashes.

4. Use Cases: In many applications (like generating unique identifiers), the beginning of the hash is often the most informative and varied. Relying on the end might reduce the uniqueness of generated identifiers, especially if a prefix has a specific context or meaning.

In summary, using the first n characters generally preserves the intended randomness and collision resistance of the hash, making it a safer choice in most cases.

@quark@ferengi.one Bloody good question 🙋 God only knows 🤣

@bender@twtxt.net Welcome! 🤗

@prologic@twtxt.net the real conclusion is, is it going to change, to what, and when? :-P

An event, in Portugal, in 2024, with a COVID-19 health and safety policy? It can be done!

I just found out about https://wiki.digitalrights.community/index.php?title=Global_Gathering_COVID-19_Policy , the COVID-19 policy for the #digitalrights “Global Gathering 2024” event that is going to happen in September 27-29 in #Estoril and I must say I’m quite happy about it.

#phpledge

@movq@www.uninformativ.de Haha 😝

What I was referring to in the OP: Sometimes I check the workphone simply out of curiosity. 😂

@movq@www.uninformativ.de Fair 👌

Current Twt Hash spec and probability of hash collision:

The probability of a Twt Hash collision depends on the size of the hash and the number of possible values it can take. For the Twt Hash, which uses a Blake2b 256-bit hash, Base32 encoding, and takes the last 7 characters, the space of possible hash values is significantly reduced.

1. Base32 encoding: Each character in the Base32 encoding represents 5 bits of information (since ( 2^5 = 32 )).
   
2. 7 characters: With 7 characters, the total number of possible hashes is:
   

@quark@ferengi.one Add here:

* a0826a65 - Add debug sub-command to yarnc (7 weeks ago) <James Mills>

I’d recommend a git pull && make build

$ echo -n "https://twtxt.net/user/prologic/twtxt.txt\n2020-07-18T12:39:52Z\nHello World! 😊" | sha1sum | head -c 11
87fd9b0ae4e

$ echo -n "https://twtxt.net/user/prologic/twtxt.txt\n2020-07-18T12:39:52Z\nHello World! 😊" | sha256sum | base32 | tr -d '=' | tr 'A-Z' 'a-z' | tail -c 12
tdqmjaeawqu

@prologic@twtxt.net yes, that would work, except there is no debug command on my local yarnc. Are you talking about a potential future implementation here?

Just experimenting…

$ echo -n "https://twtxt.net/user/prologic/twtxt.txt\n2020-07-18T12:39:52Z\nHello World! 😊" | sha256sum | base64 | tr -d '=' | tail -c 12
NWY4MSAgLQo

It would appear that the blake2b 256bit digest algorithm is no longer supported by the openssl tool, however blake2s256 is; I’m not sure why 🤔

$ echo -n "https://twtxt.net/user/prologic/twtxt.txt\n2020-07-18T12:39:52Z\nHello World! 😊" | openssl dgst -blake2s256 -binary | base32 | tr -d '=' | tr 'A-Z' 'a-z' | tail -c 7
zq4fgq

Obviously produce the wrong hash, which should be o6dsrga as indicated by the yarnc hash utility:

$ yarnc hash -u https://twtxt.net/user/prologic/twtxt.txt -t 2020-07-18T12:39:52Z "Hello World! 😊"
o6dsrga

But at least the shell pipeline is “correct”.

FWIW the standard UNIX tools for Blake2b are openssl and b2sum – Just trying to figure out how to make a shell pipeline again (if you really want that); as tools keep changing god damnit 🤣