@cuaxolotl@sunshinegardens.org

i hope we don’t go chasing mass-appeal

I don’t think this is going to happen any time soon.

What makes twtxt unique is its radical technical simplicity. And that means you have to be a tech-savvy person to appreciate twtxt and that means mass-appeal is pretty much out of the question to begin with. 😅

Yarn adds a lot of user-friendliness and, in a way, does try to appeal to the masses. And it almost worked. When Twitter died, we saw a substantial influx of new users, didn’t we? Only problem is, Mastodon had a massive headstart. The Fediverse was already huge, so what did Yarn/twtxt have to offer? It is way simpler and way easier to self-host, but who cares about that? Right, the tech-savvy people, not the masses. Many of the tech-savvy people were already using Mastodon, though, and, frankly, “simplicity” is not something that a lot of folks even care about. Thus Yarn/twtxt never took off.

A nice afternoon. Mild weather (~23°C), sitting on the balcony, working a bit on jenny, and spamming twt’s. 😅

@aelaraji@aelaraji.com I just added support for passing a custom template file via -T/--template in case you need a custom template 👌

prologic@JamessMacStudio
Wed Sep 18 01:27:29
~/Projects/yarnsocial/twtxt2html
 (main) 130
$ ./twtxt2html --help
Usage: twtxt2html [options] FILE|URL

twtxt2html converts a twtxt feed to a static HTML page
  -d, --debug             enable debug logging
  -l, --limit int         limit number ot twts (default all) (default -1)
  -n, --noreldate         do now show twt relative dates
  -r, --reverse           reverse the order of twts (oldest first)
  -T, --template string   path to template file
  -t, --title string      title of generated page (default "Twtxt Feed")
  -v, --version           display version information
pflag: help requested

@movq@www.uninformativ.de Oh! 🤣 For some reason I thought my first feed was lost and gone 🤣 Hahahaha it’s been there the whole time 😅 Fuck we write good specs and software 🥳

@falsifian@www.falsifian.org I like this idea too as a completing solution to the “identify problem”:

Or maybe url changes could somehow be combined with the archive feeds extension? Could the url metadata field be local to each archive file, so that to switch to a new url all you need to do is archive everything you’ve got and start a new file at the new url?

👌

@quark@ferengi.one At the moment, the twt in question exists in the sixth archive:

$ jenny -D https://twtxt.net/user/prologic/twtxt.txt/6 | head
[o6dsrga] [2020-07-18 12:39:52+00:00] [Hello World! 😊]

Does that work for you? 🤔

One thing that’s on my mind over the last few days about all this Twt editing and identity stuff we’ve been having hot debates over is this…

I don’t really have a problem with editing twts, or someone changing their feed’s URL.

Personally I think the folks that do are rightfully pedantic and like a good user experience, which I don’t blame ‘em. I would expect the same too. Anyway, just wanted to get that out there, I believe we can support editing and identity in a way that is still simple, as long as we bring clients along for the ride with us. The old/legacy original client though will have to remain well, ya know 😅

@movq@www.uninformativ.de Well at this point I think I’m going to try to combine @lyse@lyse.isobeef.org’s idea for supporting moving your feed to a different URL and this idea for supporting editing. I’ll spec it up and see if what we think from there…

@prologic@twtxt.net Correct. :-D

@prologic@twtxt.net Yeah, that thing with (#hash;#originalHash) would also work.

Maybe I’m being a bit too purist/minimalistic here. As I said before (in one of the 1372739 posts on this topic – or maybe I didn’t even send that twt, I don’t remember 😅), I never really liked hashes to begin with. They aren’t super hard to implement but they are kind of against the beauty of the original twtxt – because you need special client support for them. It’s not something that you could write manually in your twtxt.txt file. With @sorenpeter@darch.dk’s proposal, though, that would be possible.

I don’t know … maybe it’s just me. 🥴

I’m also being a bit selfish, to be honest: Implementing (#hash;#originalHash) in jenny for editing your own feed would not be a no-brainer. (Editing is already kind of unsupported, actually.) It wouldn’t be a problem to implement it for fetching other people’s feeds, though.

@prologic@twtxt.net I’m basically with @movq@www.uninformativ.de, but in contrast to him, I’m not looking forward to implement something like that. :-)

A feed URL is plenty good enough for me. Since I only fetch feeds that I explicity follow, there is some basic trust in those feeds already. Spoofing, impersonation and what not are no issues for me. If I were to find out otherwise, I just unsubscribe from the evil feed. Done.

To retrieve public feeds, I just rely on TLS. Most are served via HTTPS. If a feed is down, I’m not trying to fetch it from some other source, I just wait and try again later. So signed messages/feeds are not a use case I’m particularly benefitting from.

To me, it’s just not worth at all adding this crypto complexity on top.

@aelaraji@aelaraji.com Btw, I’m also open to ideas for this tool and welcome any contributions 👌

@bender@twtxt.net It’s just a simple twtxt2html and scp … it goes like:

twtxt2html $HOME/path/to/local_twtxt_dir/twtxt.txt > $HOME/path/to/local_twtxt_dir/log.html && \
scp $HOME/path/to/local_twtxt_dir/log.html user@remotehost:/path/to/static_files_dir/

I’ve been lazy to add it to my publish_command script, now I can just copy/pasta from the twt 😅

@movq@www.uninformativ.de I did the same. jenny fetches archives, yes, but that twtxt I am referring about is no longer. If you fetch it, but I don’t, there is certainly something going on…

It would mean clients that support the TwtSubject and TwtHash extension, should also indicate the previous version of their Twt when editing.

Ultimately I think we just need to agree on a way to represent an edit and the previous version of a Twt in a way that makes sense. I like one of the ideas presented earlier in some other thread (god only knows which one haha 😝); That is: <timestamp> (#hash;#originalHash) <content> For example.

@quark@ferengi.one Hmm. I cannot reproduce this. 🫤 I just removed all files in ~/.cache/jenny and ~/Mail/twt/cur, and a subsequent jenny -f properly fetches everything.

Do you see all the “Fetching archived feed …” messages?

@movq@www.uninformativ.de I think if Git can solve the same problem of branching, forking, patching and merging, so can we 🤣

@movq@www.uninformativ.de No that’s okay. I happen to agree with you really, I just wanted to get a bit of a vibe on using cryptography in general and the idea of signing feeds. It’s not particularly about a problem being solved, just gauging your opinions/thoughts on this 👌

@prologic@twtxt.net

Your propose scheme while simple doesn’t do this.

It doesn’t do that because it’s not taking the content of a twt into account (only its timestamp). Okay. But the mere fact that we’re talking about “how to solve the edit problem” stems from using content addressing – so maybe content addressing isn’t the best thing to use here? 🤔

@prologic@twtxt.net I’m very torn on this.

It’s a cool idea and it’s cool technology. It would (probably) even be fun to implement.

But do we need it? Or rather, does twtxt need it? What problem are you trying to solve – are people migrating their feeds to new URLs all the time? 🤔 That’s rather rare in my experience. The URL as the primary identifier of a feed works fine for me.

Maybe I just don’t understand the problem well enough yet? 🤔

@movq@www.uninformativ.de I did started from scratch, today. I using am commit 6e8ce5afdabd5eac22eae4275407b3bd2a167daf (HEAD -> main, origin/main, origin/HEAD), I keep myself up-to-date, LOL. Still, that specific twtxt (o6dsrga) is no longer.

@quark@ferengi.one

jenny does fetch archived feeds during the normal jenny -f operation […]

… but you need to use the current Git version which includes this commit:

https://www.uninformativ.de/git/jenny/commit/6e8ce5afdabd5eac22eae4275407b3bd2a167daf.html

There was a bug that broke on @prologic@twtxt.net’s feed. 🥴

@quark@ferengi.one

Since jenny can’t fetch archived twtxts

I wiped my entire maildir and re-fetched everything. I did that recently because @aelaraji@aelaraji.com asked me to 😅, but I guess I also did this back in 2023.

What did you do to make yours work?

jenny does fetch archived feeds during the normal jenny -f operation. Only when using the recently implemented --fetch-context, archived feeds are not fetched (yet). That was an oversight and I intend to fix that.

@aelaraji@aelaraji.com can you share the workflow you are using on jenny to convert twtxt.txt to HTML using @prologic@twtxt.net’s code?

@aelaraji@aelaraji.com We just have to write better clients 🤣 I have figured out how to detect edits FWIW, but haven’t gone from R&D phase to an actual design and implementation. But it is possible to detect an edit as well as a similarity score and the matching Twts.

@sorenpeter@darch.dk It’s nobody’s fault! 😇 It’s all part of the fun with them Ones and Zeros

@quark@ferengi.one I think I may have lost my original feed because it was written with the old software before Yarn social became a thing.

@sorenpeter@darch.dk not really you’re really forming a cryptographic chain of twts, that are cryptographically provable by anyone, at least in one direction ). It’s called content addressing. Your propose scheme while simple doesn’t do this.

After re-fetching feeds, the earliest twtxt I have from you is n7gavoa.

@prologic@twtxt.net, your first twtxt ever was o6dsrga, but I can’t find the source for it (the raw file). I reset everything, and re-fetched fresh feeds (allegedly including archives). Where is it?

@movq@www.uninformativ.de I figured it will be something like this, yet, you were able to reply just fine, and I wasn’t. Looking at your twtxt.txt I see this line:

2024-09-16T17:37:14+00:00	(#o6dsrga) @<prologic https://twtxt.net/user/prologic/twtxt.txt>

@<quark https://ferengi.one/twtxt.txt> This is what I get. 🤔

Which is using the right hash. Mine, on the other hand, when I replied to the original, old style message (Message-Id: <o6dsrga>), looks like this:

2024-09-16T16:42:27+00:00	(#o) @<prologic https://twtxt.net/user/prologic/twtxt.txt> this was your first twtxt. Cool! :-P

What did you do to make yours work? I simply went to the oldest @prologic@twtxt.net’s entry on my Maildir, and replied to it (jenny set the reply-to hash to #o, even though the Message-Id is o6dsrga). Since jenny can’t fetch archived twtxts, how could I go to re-fetch everything? And, most importantly, would re-fetching fix the Message-Id:?

@quark@ferengi.one Ahh, I see:

Message-Id: <o6dsrga>

That’s an older format that was used before jenny version v23.04. It should look like this nowadays:

Message-Id: <o6dsrga@twtxt>

Changelog entry from back then:

v23.04  2023-04-19
  [Changed]
  - The format of the "Message-Id" and "In-Reply-To" headers has
    changed. They now need an "@twtxt" suffix to be more compliant with
    RFC(2)822. This fixes issues when using aerc
    (https://aerc-mail.org/) as a frontend instead of mutt.

    If you want to retain compatibility with existing files in your
    maildir, you must manually add this suffix to these headers. (Or go
    ahead and re-sync everything.)

I guess I could have added backwards compatibility to the code. Maybe I’ll fix that later. 🤔

@prologic@twtxt.net you will always be replying to OP - that is what the twthash is a shorthand for, it it not?!

no my fault your client can’t handle a little editing ;)

@movq@www.uninformativ.de I’m glad you like it. A mention (@<movq https://www.uninformativ.de/twtxt.txt>) is also long, but we live with it anyway. In a way a replyto: is just a mention of a twt instead of a feed/person. Maybe we chould even model the syntax for replies on mentions: (#<2024-09-17T08:39:18Z https://www.eksempel.dk/twtxt.txt>) ?!

@lyse@lyse.isobeef.org and @movq@www.uninformativ.de and possibly @aelaraji@aelaraji.com and even @cuaxolotl – I’m very curious to understand and hear thoughts, pros and cons or other feelings about introducing the notation of a feed’s identify using cryptography? If we were to keep things simple, and use what’s commonly available, for example SSH ED25519 keys? using the ssh-keygen -Y sign or ssh-keygen -Y verify tools already available? Maybe in combination with @xuu ’s idea of generating a random unique ID for your feed, say # id = and signing it with your ED25519 key? 🔑

@falsifian@www.falsifian.org You mean the idea of being able to inline # url = changes in your feed?

Whatever gets used, it would be nice to be able to rotate identities. I like @lyse@lyse.isobeef.org’s idea for that.

This scheme also only support threading off a specific Twt of someone’s feed. What if you’re not replying to anyone in particular?

Noting that this scheme cannot support disjoint threads that should be merged together once either party discovers each other 😅

@movq@www.uninformativ.de It does now as of several weeks ago or so 👌

@lyse@lyse.isobeef.org Yea I think your idea of inclining url changes in the feed works perfectly as long as folks remember to do so I guess? 🤔

@bender@twtxt.net Seems to have used the hash correctly here 🤣

Holy shot! what an old thread 🤣

@cuaxolotl@sunshinegardens.org Gitea is plenty fast for me 👌 Even with a SQLite database 🤣

I dunno whether any of this is actually true 🤷‍♂️ The LLM 🤖could have made (hallucinated) this shit up 🤣

WiscKey’s approach to handling key-value pairs in SSDs offers several advantages:

1. It minimizes I/O amplification by separating keys and values, allowing for more efficient storage and retrieval.
   
2. The design leverages the SSD’s performance characteristics, utilizing sequential writes and parallel random reads to enhance throughput and reduce latency.
   
3. WiscKey maintains excellent insert and lookup performance while improving SSD lifespan by reducing the number of erase cycles required.

WiscKey minimizes CPU usage compared to LevelDB by eliminating the log file, which reduces the CPU cost associated with encoding and writing key-value pairs. While LevelDB has higher CPU usage due to its single writer protocol and background compaction using one thread, WiscKey’s architecture allows for garbage collection to be decoupled and performed later, minimizing its impact on foreground performance. Consequently, WiscKey generally exhibits lower CPU usage during workloads, except when utilizing multiple background threads for prefetching.

The four critical ideas in the design of WiscKey are:

1. Separation of keys from values, with only keys stored in the LSM-tree and values in a separate log file.
   
2. Utilization of the parallel random-read characteristic of SSD devices to handle unsorted values during range queries.
   
3. Implementation of unique crash-consistency techniques to manage the value log efficiently.
   
4. Optimization of performance by removing the LSM-tree log without sacrificing consistency, reducing system-call overhead from small writes.

Summary of WiscKey: Separating Keys from Values in SSD-Conscious Storage

• Authors: Lanyue Lu, Thanumalayan Sankaranarayana Pillai, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau
  
• Conference: 14th USENIX Conference on File and Storage Technologies (FAST ‘16)
  
• Key Concept: WiscKey is a key-value store that separates keys from values to minimize I/O amplification, optimizing performance for SSDs.
  
• Performance: WiscKey outperforms LevelDB and RocksDB in various workloads, achieving up to 111× faster loading and improved random lookup speeds.
  
• Design Goals: Focus on low write/read amplification, SSD optimization, and support for modern features like range queries.