@movq@www.uninformativ.de Is there a good way to get jenny to do a one-off fetch of a feed, for when you want to fill in missing parts of a thread? I just added @slashdot@feeds.twtxt.net to my private follow file just because @prologic@twtxt.net keeps responding to the feed :-P and I want to know what he’s commenting on even though I don’t want to see every new slashdot twt.

I guess I can configure neomutt to hide the feeds I don’t care about.

@falsifian@www.falsifian.org Hah! Remind me to talk to you about how yarnd peers with each pod in its own network to do exactly that. Maybe we could open up the protocol and you could potentially pee with other pods?

@falsifian@www.falsifian.org You mean fetching the feed temporarily and then discarding all its twts again? 🤔 I don’t think there’s an easy way to do that, other than filtering in your mail client, yeah. 🤔

@movq@www.uninformativ.de I don’t know if I’d want to discard the twts. I think what I’m looking for is a command “jenny -g https://host.org/twtxt.txt” to fetch just that one feed, even if it’s not in my follow list. I could wrap that in a shell script so that when I see a twt in reply to a feed I don’t follow, I can just tap a key and the feed will get added to my maildir. I guess the script would look for a mention at the start of a selected twt and call jenny -g on the feed.

@falsifian@www.falsifian.org Ah, I see. 🤔 Maybe I’ll add that. To be honest, I have the same “problem” regarding the slashdot feed. 😅 It’s mostly stuff that I’m not interested in – but from time to time someone replies and then I want to see what it’s about.

@movq@www.uninformativ.de, that would be a nice addition. :-) I would also love the ability to hide/not show the hash when reading twtxts (after all, that’s on the header on each “email”). Could that be added as a user configurable toggle?

@falsifian@www.falsifian.org @movq@www.uninformativ.de You actually only really want the missing root Twt. You could just fetch this from any Yarn pod. There are scripts I built way back when yo do this 😅

@prologic@twtxt.net Yes, fetching the twt by hash from some service could be a good alternative, in case the twt I have does not @-mention the source. (Besides yarnd, maybe this should be part of the registry API? I don’t see fetch-by-hash in the registry API docs.)

@falsifian@www.falsifian.org Yeah this is a good idea. Opening up the little tiny API that yarnd has for “peering” between pods for this reason. It’s quite simple really and its actually open publicly, so you can just use the scripts I wrote.

One thing to bare in mind is that Twtxt (the original spec) is largely dead, this included the registry. The registry in practise was never really widely used, and suffers from “centralization” – Which registry do you use? Its for this reason we built a search engine/crawler to help with searching and discovery. Anyway I digress… LMK if you want to go down this path, happy to document it beyond the scripts I wrote.

@prologic@twtxt.net @falsifian@www.falsifian.org This just popped up in my head: How about adding a “fetch context” feature? Point jenny to some mail file that contains a twt (or pipe it to stdin) and it will try to auto-discover and fetch all related things. Like, if it sees something like @<falsifian https://www.falsifian.org/twtxt.txt>, then it will look in https://www.falsifian.org/twtxt.txt for a twt with hash tkjafka. Maybe even do this recursively until there are no new references anymore. This process could include explicitly querying some user-configurable Yarn pods as well. 🤔

It won’t always work. There’s no guarantee that tkjafka will be present in the given URL.

Hmm. 🤔

(The jenny code is getting a bit long and convoluted. I feel the need to refactor this quite a bit. That’s why I’m not implementing any of this right away.)

@falsifian@www.falsifian.org I’ve pushed a draft into the git repo.

You can now do a “oneshot fetch” for a URL:

jenny oneshot-fetch --url https://feeds.twtxt.net/hacker-news-newest/twtxt.txt --nick hacker-news-newest

This fetches the entire feed, which might be too much. So there’s also this, which only fetches a single twt:

jenny oneshot-fetch --url https://feeds.twtxt.net/hacker-news-newest/twtxt.txt --nick hacker-news-newest --only-twt-hash r6rbinq

Let me know what you think. 🤔

@falsifian@www.falsifian.org @bender@twtxt.net I pushed an alternative implementation to the fetch-context branch. This integrates the whole thing into mutt/jenny.

You will want to configure a new mutt hotkey, similar to the “reply” hotkey:

macro index,pager <esc>C "\
<enter-command> set my_pipe_decode=\$pipe_decode nopipe_decode<Enter>\
<pipe-message> jenny -c<Enter>\
<enter-command> set pipe_decode=\$my_pipe_decode; unset my_pipe_decode<Enter>" \
"Try to fetch context of current twt, like a missing root twt"

This pipes the mail to jenny -c. jenny will try to find the thread hash and the URL and then fetch it. (If there’s no URL or if the specific twt cannot be found in that particular feed, it could query a Yarn pod. That is not yet implemented, though.)

The whole thing looks like this:

https://movq.de/v/0d0e76a180/jenny.mp4

In other words, when there’s a missing root twt, you press a hotkey to fetch it, done.

I think I like this version better. 🤔

(This needs a lot of testing. 😆)

@prologic@twtxt.net How does yarn.social’s API fix the problem of centralization? I still need to know whose API to use.

Say I see a twt beginning (#hash) and I want to look up the start of the thread. Is the idea that if that twt is hosted by a a yarn.social pod, it is likely to know the thread start, so I should query that particular pod for the hash? But what if no yarn.social pods are involved?

The community seems small enough that a registry server should be able to keep up, and I can have a couple of others as backups. Or I could crawl the list of feeds followed by whoever emitted the twt that prompted my query.

I have successfully used registry servers a little bit, e.g. to find a feed that mentioned a tag I was interested in. Was even thinking of making my own, if I get bored of my too many other projects :-)

@falsifian@www.falsifian.org So yes, you would ask a pod about the missing Twt by hash, or whatever. Pods do this already, even though there aren’t that many now, so it maybe a bit less effective today. However it’s more of a small/tiny “distributed” protocol, you ask any pod.

On registries however, I think a registry is the wrong approach. I see far greater value in feed crawlers and search engines like the (half baked one) I built over at https://search.twtxt.net/

@prologic@twtxt.net What’s the difference between search.twtxt.net and the /api/plain/tweets endpoint of a registry? In my mind, a registry is a twtxt search engine. Or are registries not supposed to do their own crawling to discover new feeds?

@falsifian@www.falsifian.org to my knowledge registries were never designed to crawl the Twtxt space. If they did, they would be considered a search engine 🤣

@prologic@twtxt.net I guess I thought they were search engines. Anyway, the registry API looks like a decent one for searching for tweets. Could/should yarn.social pods implement the same API?

@falsifian@www.falsifian.org I think I’m missing something in my description. When I say “search engine” I also mean “with a crawler” that is able to self-discover feeds. A registry (as designed today, or as the spec described) required users to add their feeds to one or more registries, putting the burden on the user(s). I for example do not bother adding my feed to a registry (which one would I add it to anyway?)

@prologic@twtxt.net I believe you when you say registries as designed today do not crawl. But when I first read the spec, it conjured in my mind a search engine. Now I don’t know how things work out in practice, but just based on reading, I don’t see why it can’t be an API for a crawling search engine. (In fact I don’t see anything in the spec indicating registry servers shouldn’t crawl.)

(I also noticed that https://twtxt.readthedocs.io/en/latest/user/registry.html recommends “The registries should sync each others user list by using the users endpoint”. If I understood that right, registering with one should be enough to appear on others, even if they don’t crawl.)

Does yarnd provide an API for finding twts? Is it similar?

@falsifian@www.falsifian.org You are totally right. The specs are at least “open enough” for us to consider that as an implementation detail. We, and by we I mean @movq@www.uninformativ.de @lyse@lyse.isobeef.org @bender@twtxt.net @xuu and others should discuss this in more detail I believe and try to see if we can agree on what we’re trying to solve.

Does yarnd provide an API for finding twts? Is it similar?

No, it doesn’t. But yarns (the search engine/crawler wrote) seems more fitting here. It’s been discussed before, the possibility of building a “Twtxt Register v1” compatible API for yarns. I think a search engine + crawler + registry (especially ones that can form a bit of a “distributed network) are far more useful I think in order to support the actual decentralised Twtxt / Yarn ecosystem (which is how I prefer to describe it).