Man my math is rusty as hell π Trying to work out how to calculate some kind of moving average for a feeds update frequency to use as a way to do some kind of exponential back-off for feeds π
@prologic@twtxt.net For polling feeds on other pods or?
@will@twtxt.net In general basically. But yes, a way to intelligently reduce traffic between pods and between pods and external feeds. The refresh
field recently added to the spec is a βmanualβ / βhumanβ driven optional field. But Iβd love it if yarnd
could just work out whether it should refresh a feed or not based on historical data of how often it was posted to.
Iβve thrown away one implementation so far π
But so far I end up with values like this:
DEBU[0481] Moving Average for http://10.0.0.109:9000/twtxt.txt = 2998.23
DEBU[0481] Moving Average for https://twtxt.net/user/prologic/twtxt.txt = 3427.83
DEBU[0481] Moving Average for http://10.0.0.109:8000/user/stats/twtxt.txt = 0.00
DEBU[0481] Moving Average for http://10.0.0.109:8000/user/help/twtxt.txt = 0.00
DEBU[0481] Moving Average for http://10.0.0.109:8000/user/twtxt/twtxt.txt = 0.00
DEBU[0481] Moving Average for http://10.0.0.109:8000/user/admin/twtxt.txt = 1.00
Which Iβm not sure are actually that useful hmmm π€
Maybe I need to look at only a subset of a feed, like the last 6 posts? Hmmm π€
I have no idea what Iβm doing π
Okay this is better, using a βwindowβ is producing better results:
DEBU[0241] Moving Average for https://twtxt.net/user/prologic/twtxt.txt = 181.25
DEBU[0241] Moving Average for http://10.0.0.109:8000/user/admin/twtxt.txt = 1.00
DEBU[0241] Moving Average for http://10.0.0.109:8000/user/twtxt/twtxt.txt = 0.00
DEBU[0241] Moving Average for http://10.0.0.109:8000/user/help/twtxt.txt = 0.00
DEBU[0241] Moving Average for http://10.0.0.109:9000/twtxt.txt = 4.17
Okay cool, now my moving average for my own feed is going up as expecgted:
DEBU[0005] Moving Average for https://twtxt.net/user/prologic/twtxt.txt = 255.29
DEBU[0060] Moving Average for https://twtxt.net/user/prologic/twtxt.txt = 442.47
DEBU[0120] Moving Average for https://twtxt.net/user/prologic/twtxt.txt = 562.78
DEBU[0180] Moving Average for https://twtxt.net/user/prologic/twtxt.txt = 652.96
...
Now letβs see if it goes down as I post most frequently π
Seems to be working so farβ¦
Sorry this is gonna be a bit of a spammer day π
@prologic@twtxt.net thought experiment: what happens if every feed is on a different pod?
@prologic@twtxt.net oh, you explained about the Moving average a few tweets (?) beforeβ¦
I missed that msg! Nice, it brings me memories of a very old project to parse RSS and a few Leaderboards where Iβve used some statistical functions/weird math.
Good vibes with what you are doing!