I just blocked the following ASN(s) from being able to hit twtxt.net or mills.io:

16509 - AMAZON-02
32934 - FACEBOOK

Why? Because the Claude Bot web crawler from facebookexternalhit and Metaโ€™s facebookexternalhit web crawler are both behaving badly for pages that have no cache headers. Not sure if this is malicious, an oversight, a bug or me just being stupid and not ensuring every web resource or page had appropriate Cache headers? ๐Ÿค” In any case, until I hear back from at least facebookexternalhit (whom Iโ€™ve reached out to), these ASN(s) will remain entirely blocked.

That is the entirety of Amazon Web Services and Facebook.

โค‹ Read More