@lyse@lyse.isobeef.org what are you building now? The things you are mentioning I couldn’t even start wrapping my head around them! 😅 They sure sound expensive, tough.

Thanks @david@collantes.us, good to know, but we need to agree on what character we use, otherwise the hashes will not be the same:)

@sorenpeter@darch.dk a TAB is simply \t. Just add it to that echo line, and that’s it.

@aelaraji@aelaraji.com You could just remove the {getuser()} part because you added ~.

Ok, i know how to command working (not sure), but seems it only grab from cache. Maybe make fetch from twtxt.net if hash not found?

@prologic@twtxt.net Regarding the new way of generating twt-hashes, to me it makes more sense to use tabs as separator instead of spaces, since the you can just copy/past a line directly from a twtxt-file that already go a tab between timestamp and message. But tabs might be hard to “type” when you are in a terminal, since it will activate autocomplete…🤔

Another thing, it seems that you sugget we only use the domain in the hash-creation and not the full path to the twtxt.txt

$ echo -e "https://example.com 2024-09-29T13:30:00Z Hello World!" | sha256sum - | awk '{ print $1 }' | base64 | head -c 12

should i delete gemini support from twet? iirc in twtxt v2 it starts prohibited. And all of my fields are https

It has twts cache which used if timeline is set to jew. Maybe i.should fork twet to make wishes like newlines (i see two squares), showing conversations, showing twts if not found in cache and parsing medata to configure url, nick and followers (currenly it duplicated in config and twtxt file)

twet display twts in raw format with some formatting (sadly no newlines). And for reply messages i just seen (#hash). But which text hidden on hash? currenly im open twtxt.net/twt/hash to see this

@doesnm@doesnm.p.psf.lt I am not sure I am understanding what you mean. Can you explain?

Lol, im just join for several minutes. Wait, Merkle Trees in twtxt?

@prologic@twtxt.net YES James, it should be up to the client to deal with changes like edits and deletions. And putting this load on the clients, location-addressing with make this a lot easier since what is says it: Look in this file at this timestamp, did anything change or went missing? (And then threading will not break;)

@bender@twtxt.net It’s the experience of an ordinary person in a strange place where memories are disappearing with the help of the Memory Police. The setting feels contemporary (to the book’s 1994 publication date) rather than futuristic, except for some unexplained stuff about memories.

@prologic@twtxt.net

See https://dev.twtxt.net

Yes, that is exactly what I meant. I like that collection and “twtxt v2” feels like a departure.

Maybe there’s an advantage to grouping it into one spec, but IMO that shouldn’t be done at the same time as introducing new untested ideas.

See https://yarn.social (especially this section: https://yarn.social/#self-host) – It really doesn’t get much simpler than this 🤣

Again, I like this existing simplicity. (I would even argue you don’t need the metadata.)

That page says “For the best experience your client should also support some of the Twtxt Extensions…” but it is clear you don’t need to. I would like it to stay that way, and publishing a big long spec and calling it “twtxt v2” feels like a departure from that. (I think the content of the document is valuable; I’m just carping about how it’s being presented.)

More thoughts about changes to twtxt (as if we haven’t had enough thoughts):

1. There are lots of great ideas here! Is there a benefit to putting them all into one document? Seems to me this could more easily be a bunch of separate efforts that can progress at their own pace:
   

1a. Better and longer hashes.

1b. New possibly-controversial ideas like edit: and delete: and location-based references as an alternative to hashes.

1c. Best practices, e.g. Content-Type: text/plain; charset=utf-8

1d. Stuff already described at dev.twtxt.net that doesn’t need any changes.

1. We won’t know what will and won’t work until we try them. So I’m inclined to think of this as a bunch of draft ideas. Maybe later when we’ve seen it play out it could make sense to define a group of recommended twtxt extensions and give them a name.

2. Another reason for 1 (above) is: I like the current situation where all you need to get started is these two short and simple documents:
   https://twtxt.readthedocs.io/en/latest/user/twtxtfile.html
   https://twtxt.readthedocs.io/en/latest/user/discoverability.html
   and everything else is an extension for anyone interested. (Deprecating non-UTC times seems reasonable to me, though.) Having a big long “twtxt v2” document seems less inviting to people looking for something simple. (@prologic@twtxt.net you mentioned an anonymous comment “you’ve ruined twtxt” and while I don’t completely agree with that commenter’s sentiment, I would feel like twtxt had lost something if it moved away from having a super-simple core.)

3. All that being said, these are just my opinions, and I’m not doing the work of writing software or drafting proposals. Maybe I will at some point, but until then, if you’re actually implementing things, you’re in charge of what you decide to make, and I’m grateful for the work.

@prologic@twtxt.net Done. Also, I went ahead and made two changes: changed hexadecimal to base64 for hashes (wasn’t sure if anyone objected), and changed “MUST follow the chain” to “SHOULD follow the chain.

@prologic@twtxt.net Thanks for pointing out it lasts four hours. That’s a big window! I wonder when most people will be on. I might aim for halfway through unless I hear otherwise. (12:00Z is a bit early for me.)

@bender@twtxt.net I am also in camp no edit signals. deletes only breaks the head of a thread. all the replies are unaffected.

@lyse@lyse.isobeef.org that sounds truly idyllic! 🤭

@bender@twtxt.net yeah, that’s what I said, and linked. Want more?

Yes. I have only twtxt and scp hook in twet and it enough

Wait, webfinger? Mandate this ruin philosophy “twtxt is just text file”

I dont think that is ruined twtxt. Twtxt v2 is just standartize twtxt and yarn extensions. What is bad?

@bender@twtxt.net I believe it is Unix-Unix Copy Protocol. Not Unix Copy-Copy Protocol.

@aelaraji@aelaraji.com ooooh! It’s that kind mission! /me stands, salutes, turns around, and exits the room. LOL.

@aelaraji@aelaraji.com why, having a party with lots of libations? LOL.

@prologic@twtxt.net that “little database that could” is simply amazing, isn’t it? I run Conduwuit (nevermind, this one is RocksDB), and GoToSocial using it as a backend, no issues. And, of course, sqlite is the database of choice for a lot of things under iOS.

@prologic@twtxt.net, are you running Gitea with an SQL backend, or using sqlite? Any reason have haven’t moved to Forgejo?

@prologic@twtxt.net a wise plan! Who knows, ideas change, and often plans do not hash, right? Mature, mature! :-)

@xuu@txt.sour.is was that 2% picked out randomly? I like it! LOL.

@prologic@twtxt.net I like the, allegedly, original:

“It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.”

Not as simple as the interpretation you used, yet often context is king (or queen).

@prologic@twtxt.net and one could say that “for every simple problem, there is a solution that’s confusing, convoluted, and right.” :-P

@prologic@twtxt.net so, where are they? I want to take a peek at HomeTunnel (even though I don’t a use case for it at the moment). Show us repos! :-P

@david@collantes.us having offsets were nice because it gives you context of where the user is in relation to you.

@prologic@twtxt.net thanks. I hate it. Might as well use UUID

@lyse@lyse.isobeef.org thank you! Raining is starting to fall very steadily. All good so far. Wife’s home, a nice meal simmers. Ah! :-D

@lyse@lyse.isobeef.org on this:

3.2 Timestamps: I feel no need to mandate UTC. Timezones are fine with me. But I could also live with this new restriction. I fail to see, though, how this change would make things any easier compared to the original format.

Exactly! If anything it will make things more complicated, no?

No, json is overhead. I love twtxt for simplicity where blog is just text file and not several json files where fields are repeated…

@sorenpeter@darch.dk this will show broken, because you are hellbent on editing twtxts, aren’t you? :-D

(#2024-09-24T12:53:35Z) What does this screenshot show? The resolution it too low for reading the text…

(#2024-09-24T12:45:54Z) @prologic@twtxt.net I’m not really buying this one about readability. It’s easy to recognize that this is a URL and a date, so you skim over it like you would we mentions and markdown links and images. If you are not suppose to read the raw file, then we might a well jam everything into JSON like mastodon

(#2024-09-24T12:44:35Z) There is a increase in space/memory for sure. But calculating the hashes also takes up CPU. I’m not good with that kind of math, but it’s a tradeoff either way.

(#2024-09-24T12:39:32Z) @prologic@twtxt.net It might be simple for you to run echo -e "\t\t" | sha256sum | base64, but for people who are not comfortable in a terminal and got their dev env set up, then that is magic, compared to the simplicity of just copy/pasting what you see in a textfile into another textfile – Basically what @movq@www.uninformativ.de also said. I’m also on team extreme minimalism, otherwise we could just use mastodon etc. Replacing line-breaks with a tab would also make it easier to handwrite your twtxt. You don’t have to hardwrite it, but at least you should have the option to. Just as i do with all my HTML and CSS.

(#2024-09-24T12:34:31Z) WebMentions does would work if we agreed to implement it correctly. I never figured out how yarnd’s WebMentions work, so I decide to make my own, which I’m the only one using…

I had a look at WebSub, witch looks way more complex than WebMentions, and seem to need a lot more overhead. We don’t need near realtime. We just need a way to notify someone that someone they don’t know about mentioned or replied to their post.

@lyse@lyse.isobeef.org aha! Just like Bash would do. I figure -- is way too broad to start an autocomplete. Got to feed it a bit more! :-D

@lyse@lyse.isobeef.org that -P is a life saver when running rsync over spotty connections. In my very illiterate opinion, it should always be a default.

@aelaraji@aelaraji.com I think all replies are missing the fact that your auto-completion isn’t working. LOL. Or did I misunderstood?

Aggred. But reading twtxt in raw form sounds… I can’t do this

Some more arguments for a local-based treading model over a content-based one:

1. The format: (#<DATE URL>) or (@<DATE URL>) both makes sense: # as prefix is for a hashtag like we allredy got with the (#twthash) and @ as prefix denotes that this is mention of a specific post in a feed, and not just the feed in general. Using either can make implementation easier, since most clients already got this kind of filtering.

2. Having something like (#<DATE URL>) will also make mentions via webmetions for twtxt easier to implement, since there is no need for looking up the #twthash. This will also make it possible to make 3th part twt-mentions services.

3. Supporting twt/webmentions will also increase discoverability as a way to know about both replies and feed mentions from feeds that you don’t follow.

Finally pubnix is alive! That’s im missing? Im only reading twtxt.net timeline because twtxt-v2.sh works slowly for displaying timeline…

@movq@www.uninformativ.de Yes, the tools are surprisingly fast. Still, magrep takes about 20 seconds to search through my archive of 140K emails, so to speed things up I would probably combine it with an indexer like mu, mairix or notmuch.

@falsifian@www.falsifian.org I believe the preserve means to include the original subject hash in the start of the twt such as (#somehash)

@bender@twtxt.net Ha! Maybe I should get on the Markdown train. You’re taking away my excuses.

@bender@twtxt.net

Sorry, you’re right, I should have used numbers!

I’m don’t understand what “preserve the original hash” could mean other than “make sure there’s still a twt in the feed with that hash”. Maybe the text could be clarified somehow.

I’m also not sure what you mean by markdown already being part of it. Of course people can already use Markdown, just like presumably nothing stopped people from using (twt subjects) before they were formally described. But it’s not universal; e.g. as a jenny user I just see the plain text.

@falsifian@www.falsifian.org The GDPR does not apply to the processing of data for a purely personal or household activity that is not connected to a professional or commercial activity.

@prologic@twtxt.net Do you feel the same about published vs. privately stored data?

For me there’s a distinction. I feel very strongly that I should be able to retain whatever private information I like. On the other hand, I do have some sympathy for requests not to publish or propagate (though I personally feel it’s still morally acceptable to ignore such requests).

@lyse@lyse.isobeef.org I’d suggest making the whole content-type thing a SHOULD, to accommodate people just using some hosting service they don’t have much control over. (The same situation could make detecting followers hard, but IMO “please email me if you follow me” is still legit twtxt, even if inconvenient.)

@prologic@twtxt.net Thanks for writing that up!

I hope it can remain a living document (or sequence of draft revisions) for a good long time while we figure out how this stuff works in practice.

I am not sure how I feel about all this being done at once, vs. letting conventions arise.

For example, even today I could reply to twt abc1234 with “(#abc1234) Edit: …” and I think all you humans would understand it as an edit to (#abc1234). Maybe eventually it would become a common enough convention that clients would start to support it explicitly.

Similarly we could just start using 11-digit hashes. We should iron out whether it’s sha256 or whatever but there’s no need get all the other stuff right at the same time.

I have similar thoughts about how some users could try out location-based replies in a backward-compatible way (append the replyto: stuff after the legacy (#hash) style).

However I recognize that I’m not the one implementing this stuff, and it’s less work to just have everything determined up front.

Misc comments (I haven’t read the whole thing):

• Did you mean to make hashes hexadecimal? You lose 11 bits that way compared to base32. I’d suggest gaining 11 bits with base64 instead.

• “Clients MUST preserve the original hash” — do you mean they MUST preserve the original twt?

• Thanks for phrasing the bit about deletions so neutrally.

• I don’t like the MUST in “Clients MUST follow the chain of reply-to references…”. If someone writes a client as a 40-line shell script that requires the user to piece together the threading themselves, IMO we shouldn’t declare the client non-conforming just because they didn’t get to all the bells and whistles.

• Similarly I don’t like the MUST for user agents. For one thing, you might want to fetch a feed without revealing your identty. Also, it raises the bar for a minimal implementation (I’m again thinking again of the 40-line shell script).

• For “who follows” lists: why must the long, random tokens be only valid for a limited time? Do you have a scenario in mind where they could leak?

• Why can’t feeds be served over HTTP/1.0? Again, thinking about simple software. I recently tried implementing HTTP/1.1 and it wasn’t too bad, but 1.0 would have been slightly simpler.

• Why get into the nitty-gritty about caching headers? This seems like generic advice for HTTP servers and clients.

• I’m a little sad about other protocols being not recommended.

• I don’t know how I feel about including markdown. I don’t mind too much that yarn users emit twts full of markdown, but I’m more of a plain text kind of person. Also it adds to the length. I wonder if putting a separate document would make more sense; that would also help with the length.

@movq@www.uninformativ.de I cases of these kind of “abuse” of social trust. Then I think people should just delete their replies, unfollow the troll and leave them to shouting in the void. This is a inter-social issue, not a technical issue. Anything can be spoofed. We are not building a banking app, we are just having conversation and if trust are broken then communication breaks down. These edge-cases are all very hypothetical and not something I think we need to solve with technology.

@prologic@twtxt.net “Even though there are some 😉 that have different views on this 🤣” — coyly raises the hand… LOL.

@prologic@twtxt.net I have no specifics, only hopes. (I have seen some articles explaining the GDPR doesn’t apply to a “purely personal or household activity” but I don’t really know what that means.)

I don’t know if it’s worth giving much thought to the issue unless either you expect to get big enough for the GDPR to matter a lot (I imagine making money is a prerequisite) or someone specifically brings it up. Unless you enjoy thinking through this sort of thing, of course.

And they have arrived (well, they did around 3 hours ago, LOL). Buttery smooth, my 16 Pro (one with dark cover). It took a bit over an hour to transfer all my data.

Image

@lyse@lyse.isobeef.org yeah, tell us, @prologic@twtxt.net, what isn’t true? 🤔 You can’t just go around, “that’s not true, and that’s not true; and that, and that!” without spelling out exactly what isn’t, and why? For the love of god, why?! 😂

@falsifian@www.falsifian.org comments on the feeds as in nick, url, follow, that kind of thing? If that, then not interested at all. I envision an archive that would allow searching, and potentially browsing threads on a nice, neat interface. You will have to think, though, on other things. Like, what to do with images? Yarn allows users to upload images, but also embed it in twtxts from other sources (hotlinking, actually).

@david@collantes.us Thanks, that’s good feedback to have. I wonder to what extent this already exists in registry servers and yarn pods. I haven’t really tried digging into the past in either one.

How interested would you be in changes in metadata and other comments in the feeds? I’m thinking of just permanently saving every version of each twtxt file that gets pulled, not just the twts. It wouldn’t be hard to do (though presenting the information in a sensible way is another matter). Compression should make storage a non-issue unless someone does something weird with their feed like shuffle the comments around every time I fetch it.

@falsifian@www.falsifian.org “I was actually thinking about making an Internet Archive style twtxt archiver, letting you explore past twts” — that’s an awesome idea for a project. Something I would certainly use!

@movq@www.uninformativ.de I don’t think it has to be like that. Just make sure the new version of the twt is always appended to your current feed, and have some convention for indicating it’s an edit and which twt it supersedes. Keep the original twt as-is (or delete it if you don’t want new followers to see it); doesn’t matter if it’s archived because you aren’t changing that copy.

@prologic@twtxt.net Do you have a link to some past discussion?

Would the GDPR would apply to a one-person client like jenny? I seriously hope not. If someone asks me to delete an email they sent me, I don’t think I have to honour that request, no matter how European they are.

I am really bothered by the idea that someone could force me to delete my private, personal record of my interactions with them. Would I have to delete my journal entries about them too if they asked?

Maybe a public-facing client like yarnd needs to consider this, but that also bothers me. I was actually thinking about making an Internet Archive style twtxt archiver, letting you explore past twts, including long-dead feeds, see edit histories, deleted twts, etc.

@prologic@twtxt.net what time in UTC?

@prologic@twtxt.net cool, I will be there! Are you going to post the regular banner notice? It will serve as a reminder, at least for me.

@david@collantes.us Well, I wouldn’t recommend using my code for your main jenny use anyway. If you want to try it out, set XDG_CONFIG_HOME and XDG_CACHE_HOME to some sandbox directories and only run my code there. If @movq@www.uninformativ.de is interested in any of this getting upstreamed, I’d be happy to try rebasing the changes, but otherwise it’s a proof of concept and fun exercise.

@falsifian@www.falsifian.org hiya from the Sunshine State (also known as a never-ending hell, LOL)!

@falsifian@www.falsifian.org this one hits hard, as jenny was just updated today. :‘-(

@david@collantes.us Hello!

@prologic@twtxt.net Hi. i have noticed sometimes when i hit the back button i lose all the surrounding layout and just have a list of twts.

I wrote some code to try out non-hash reply subjects formatted as (replyto ), while keeping the ability to use the existing hash style.

I don’t think we need to decide all at once. If clients add support for a new method then people can use it if they like. The downside of course is that this costs developer time, so I decided to invest a few hours of my own time into a proof of concept.

With apologies to @movq@www.uninformativ.de for corrupting jenny’s beautiful code. I don’t write this expecting you to incorporate the patch, because it does complicate things and might not be a direction you want to go in. But if you like any part of this approach feel free to use bits of it; I release the patch under jenny’s current LICENCE.

Supporting both kinds of reply in jenny was complicated because each email can only have one Message-Id, and because it’s possible the target twt will not be seen until after the twt referencing it. The following patch uses an sqlite database to keep track of known (url, timestamp) pairs, as well as a separate table of (url, timestamp) pairs that haven’t been seen yet but are wanted. When one of those “wanted” twts is finally seen, the mail file gets rewritten to include the appropriate In-Reply-To header.

Patch based on jenny commit 73a5ea81.

https://www.falsifian.org/a/oDtr/patch0.txt

Not implemented:

• Composing twts using the (replyto …) format.
  
• Probably other important things I’m forgetting.

@lyse@lyse.isobeef.org indeed! There is no “central authority” acting as witness, and notary. The more I think of it… LOL.

@movq@www.uninformativ.de I recognise him, but yes, he has aged quite a bit. I mean, I look at myself in the mirror and can’t, often, recognise myself. Ageing is a bitch! 😅

I mean, really, it couldn’t get any better. I love it!

Image

@movq@www.uninformativ.de perfect in every way. Configurable too! Thank you!

@movq@www.uninformativ.de yes, that’s perfect! <3

@eldersnake@we.loveprivacy.club I wanted to ask you, are you running Headscale and WireGuard on the same VPS? I want to test Headscale, but currently run a small container with WireGuard, and I wonder if I need to stop (and eventually get rid of) the container to get Headscale going. Did you use the provided .deb to install Headscale, or some other method?

@prologic@twtxt.net I read it. I understand it. Hopefully a solution can be agreed upon that solves the editing issue, whilst maintaining the cryptographic hash.

@prologic@twtxt.net I know the role of the current hash is to allow referencing (replies and, thus, threads), and it also represents a “unique” way to verify a twtxt hasn’t been tampered with. Is that second so important, if we are trying to allow edits? I know if feels good to be able to verify, but in reality, how often one does it?

@movq@www.uninformativ.de could it be possible to have compressed_subject(msg_singlelined) be configurable, so only a certain number of characters get displayed, ending on ellipses? Right now the entire twtxt is crammed into the Subject:. This request aims to make twtxts display on mutt/neomutt, etc. more like emails do.

@prologic@twtxt.net I don’t trust Google with anything, sorry, pass. Oh, and you need to sign in on your Google Account (or whatever they call it these days).

@prologic@twtxt.net how about hashing a combination of nick/timestamp, or url/timestamp only, and not the twtxt content? On edit those will not change, so no breaking of threads. I know, I know, just adding noise here. :-P

@eldersnake@we.loveprivacy.club there has to be less reliance on a single point of failure. It is not so much about creating jobs in the US (which come with it, anyway), but about the ability to produce what’s needed at home too. What’s the trade off? Is it going to be a little bit more expensive to manufacture, perhaps?

@quark@ferengi.one It does not. That is why I’m advocating for not using hashes for treads, but a simpler link-back scheme.

@prologic@twtxt.net where was that idea?

@prologic@twtxt.net the basic idea was to stem the hash.. so you have a hash abcdef0123456789... any sub string of that hash after the first 6 will match. so abcdef, abcdef012, abcdef0123456 all match the same. on the case of a collision i think we decided on matching the newest since we archive off older threads anyway. the third rule was about growing the minimum hash size after some threshold of collisions were detected.

@prologic@twtxt.net Wikipedia claims sha1 is vulnerable to a “chosen-prefix attack”, which I gather means I can write any two twts I like, and then cause them to have the exact same sha1 hash by appending something. I guess a twt ending in random junk might look suspcious, but perhaps the junk could be worked into an image URL like

Image

git only uses sha1 because they’re stuck with it: migrating is very hard. There was an effort to move git to sha256 but I don’t know its status. I think there is progress being made with Game Of Trees, a git clone that uses the same on-disk format.

I can’t imagine any benefit to using sha1, except that maybe some very old software might support sha1 but not sha256.

@prologic@twtxt.net :-D Thanks! Things can come in cycles, right? This is simply another one. Another cycle, more personal than the other “alter egos”.

@aelaraji@aelaraji.com hey, hey! You are my very first reply! 👋🏻 Cheers!

Incredibly upset—more than you could imagine—because I already made the first mistake, and corrected it (but twtxt.net got it on it’s cache, ugh!) :‘-( . Can’t wait for editing to become a reality!

Alright. My first mentions—which were picked not so randomly, LOL—are @prologic@twtxt.net, @lyse@lyse.isobeef.org, and @movq@www.uninformativ.de. I am also posting my first image too, which you see below. That’s my neighbourhood, in a “winter” day. Hopefully @prologic@twtxt.net will add my domain to his allowed list, so that the image (and any other further) renders.

Image

Alright, announce_me set to true. Now, who do I pick to be my first mention? Decisions, decisions. Next twtxt will have my first mention(s). :-)

I have configured my twtxt.txt as simple as possible. I have setup a publish_command on jenny. Hopefully all works fine, and I am good to go. Next will be setting the announce_me to true. Here we go!

@sorenpeter@darch.dk hmm, how does your client handles “a little editing”? I am sure threads would break just as well. 😉

@prologic@twtxt.net, there is a parser bug on parent. Specifically on this portion:

"*If twtxt/Yarn was to grow bigger, then this would become a concern again. *But even Mastodon allows editing*, so how
+much of a problem can it really be? 😅*"