@movq@www.uninformativ.de Is there a good way to get jenny to do a one-off fetch of a feed, for when you want to fill in missing parts of a thread? I just added @slashdot@feeds.twtxt.net to my private follow file just because @prologic@twtxt.net keeps responding to the feed :-P and I want to know what he’s commenting on even though I don’t want to see every new slashdot twt.
I guess I can configure neomutt to hide the feeds I don’t care about.
@falsifian@www.falsifian.org Hah! Remind me to talk to you about how yarnd peers with each pod in its own network to do exactly that. Maybe we could open up the protocol and you could potentially pee with other pods?
@falsifian@www.falsifian.org You mean fetching the feed temporarily and then discarding all its twts again? 🤔 I don’t think there’s an easy way to do that, other than filtering in your mail client, yeah. 🤔
@falsifian@www.falsifian.org Ah, I see. 🤔 Maybe I’ll add that. To be honest, I have the same “problem” regarding the slashdot feed. 😅 It’s mostly stuff that I’m not interested in – but from time to time someone replies and then I want to see what it’s about.
@movq@www.uninformativ.de, that would be a nice addition. :-) I would also love the ability to hide/not show the hash when reading twtxts (after all, that’s on the header on each “email”). Could that be added as a user configurable toggle?
@falsifian@www.falsifian.org @movq@www.uninformativ.de You actually only really want the missing root Twt. You could just fetch this from any Yarn pod. There are scripts I built way back when yo do this 😅
@falsifian@www.falsifian.org Yeah this is a good idea. Opening up the little tiny API that yarnd has for “peering” between pods for this reason. It’s quite simple really and its actually open publicly, so you can just use the scripts I wrote.
One thing to bare in mind is that Twtxt (the original spec) is largely dead, this included the registry. The registry in practise was never really widely used, and suffers from “centralization” – Which registry do you use? Its for this reason we built a search engine/crawler to help with searching and discovery. Anyway I digress… LMK if you want to go down this path, happy to document it beyond the scripts I wrote.
@prologic@twtxt.net @falsifian@www.falsifian.org This just popped up in my head: How about adding a “fetch context” feature? Point jenny to some mail file that contains a twt (or pipe it to stdin) and it will try to auto-discover and fetch all related things. Like, if it sees something like @<falsifian https://www.falsifian.org/twtxt.txt>, then it will look in https://www.falsifian.org/twtxt.txt for a twt with hash tkjafka. Maybe even do this recursively until there are no new references anymore. This process could include explicitly querying some user-configurable Yarn pods as well. 🤔
It won’t always work. There’s no guarantee that tkjafka will be present in the given URL.
Hmm. 🤔
(The jenny code is getting a bit long and convoluted. I feel the need to refactor this quite a bit. That’s why I’m not implementing any of this right away.)
@falsifian@www.falsifian.org I’ve pushed a draft into the git repo.
You can now do a “oneshot fetch” for a URL:
jenny oneshot-fetch --url https://feeds.twtxt.net/hacker-news-newest/twtxt.txt --nick hacker-news-newest
This fetches the entire feed, which might be too much. So there’s also this, which only fetches a single twt:
jenny oneshot-fetch --url https://feeds.twtxt.net/hacker-news-newest/twtxt.txt --nick hacker-news-newest --only-twt-hash r6rbinq
Let me know what you think. 🤔
@falsifian@www.falsifian.org @bender@twtxt.net I pushed an alternative implementation to the fetch-context branch. This integrates the whole thing into mutt/jenny.
You will want to configure a new mutt hotkey, similar to the “reply” hotkey:
macro index,pager <esc>C "\
<enter-command> set my_pipe_decode=\$pipe_decode nopipe_decode<Enter>\
<pipe-message> jenny -c<Enter>\
<enter-command> set pipe_decode=\$my_pipe_decode; unset my_pipe_decode<Enter>" \
"Try to fetch context of current twt, like a missing root twt"
This pipes the mail to jenny -c. jenny will try to find the thread hash and the URL and then fetch it. (If there’s no URL or if the specific twt cannot be found in that particular feed, it could query a Yarn pod. That is not yet implemented, though.)
The whole thing looks like this:
https://movq.de/v/0d0e76a180/jenny.mp4
In other words, when there’s a missing root twt, you press a hotkey to fetch it, done.
I think I like this version better. 🤔
(This needs a lot of testing. 😆)
@falsifian@www.falsifian.org So yes, you would ask a pod about the missing Twt by hash, or whatever. Pods do this already, even though there aren’t that many now, so it maybe a bit less effective today. However it’s more of a small/tiny “distributed” protocol, you ask any pod.
On registries however, I think a registry is the wrong approach. I see far greater value in feed crawlers and search engines like the (half baked one) I built over at https://search.twtxt.net/
@falsifian@www.falsifian.org to my knowledge registries were never designed to crawl the Twtxt space. If they did, they would be considered a search engine 🤣
@prologic@twtxt.net I guess I thought they were search engines. Anyway, the registry API looks like a decent one for searching for tweets. Could/should yarn.social pods implement the same API?
@falsifian@www.falsifian.org I think I’m missing something in my description. When I say “search engine” I also mean “with a crawler” that is able to self-discover feeds. A registry (as designed today, or as the spec described) required users to add their feeds to one or more registries, putting the burden on the user(s). I for example do not bother adding my feed to a registry (which one would I add it to anyway?)
@falsifian@www.falsifian.org You are totally right. The specs are at least “open enough” for us to consider that as an implementation detail. We, and by we I mean @movq@www.uninformativ.de @lyse@lyse.isobeef.org @bender@twtxt.net @xuu and others should discuss this in more detail I believe and try to see if we can agree on what we’re trying to solve.
Does yarnd provide an API for finding twts? Is it similar?
No, it doesn’t. But yarns (the search engine/crawler wrote) seems more fitting here. It’s been discussed before, the possibility of building a “Twtxt Register v1” compatible API for yarns. I think a search engine + crawler + registry (especially ones that can form a bit of a “distributed network) are far more useful I think in order to support the actual decentralised Twtxt / Yarn ecosystem (which is how I prefer to describe it).