Man my math is rusty as hell 😅 Trying to work out how to calculate some kind of moving average for a feeds update frequency to use as a way to do some kind of exponential back-off for feeds 😂

@prologic@twtxt.net For polling feeds on other pods or?

@will@twtxt.net In general basically. But yes, a way to intelligently reduce traffic between pods and between pods and external feeds. The refresh field recently added to the spec is a “manual” / “human” driven optional field. But I’d love it if yarnd could just work out whether it should refresh a feed or not based on historical data of how often it was posted to.

I’ve thrown away one implementation so far 😂

But so far I end up with values like this:

DEBU[0481] Moving Average for http://10.0.0.109:9000/twtxt.txt = 2998.23
DEBU[0481] Moving Average for https://twtxt.net/user/prologic/twtxt.txt = 3427.83
DEBU[0481] Moving Average for http://10.0.0.109:8000/user/stats/twtxt.txt = 0.00
DEBU[0481] Moving Average for http://10.0.0.109:8000/user/help/twtxt.txt = 0.00
DEBU[0481] Moving Average for http://10.0.0.109:8000/user/twtxt/twtxt.txt = 0.00
DEBU[0481] Moving Average for http://10.0.0.109:8000/user/admin/twtxt.txt = 1.00

Which I’m not sure are actually that useful hmmm 🤔

Maybe I need to look at only a subset of a feed, like the last 6 posts? Hmmm 🤔

I have no idea what I’m doing 😅

Okay this is better, using a “window” is producing better results:

DEBU[0241] Moving Average for https://twtxt.net/user/prologic/twtxt.txt = 181.25
DEBU[0241] Moving Average for http://10.0.0.109:8000/user/admin/twtxt.txt = 1.00
DEBU[0241] Moving Average for http://10.0.0.109:8000/user/twtxt/twtxt.txt = 0.00
DEBU[0241] Moving Average for http://10.0.0.109:8000/user/help/twtxt.txt = 0.00
DEBU[0241] Moving Average for http://10.0.0.109:9000/twtxt.txt = 4.17

Okay cool, now my moving average for my own feed is going up as expecgted:

DEBU[0005] Moving Average for https://twtxt.net/user/prologic/twtxt.txt = 255.29
DEBU[0060] Moving Average for https://twtxt.net/user/prologic/twtxt.txt = 442.47
DEBU[0120] Moving Average for https://twtxt.net/user/prologic/twtxt.txt = 562.78
DEBU[0180] Moving Average for https://twtxt.net/user/prologic/twtxt.txt = 652.96
...

Now let’s see if it goes down as I post most frequently 😅

Seems to be working so far…

Sorry this is gonna be a bit of a spammer day 😂

@prologic@twtxt.net thought experiment: what happens if every feed is on a different pod?

@prologic@twtxt.net oh, you explained about the Moving average a few tweets (?) before…
I missed that msg! Nice, it brings me memories of a very old project to parse RSS and a few Leaderboards where I’ve used some statistical functions/weird math.

Good vibes with what you are doing!