Almost every website and services are getting scraped at alarming rate, are Lemmy servers facing this issue?
Please share mitigations you’ve seen applied to this.
Almost every website and services are getting scraped at alarming rate, are Lemmy servers facing this issue?
Please share mitigations you’ve seen applied to this.
I don’t host a Lemmy instance, but I post links in my comments. I sometimes generate and share unique-ish URLs to share updates with specific versions of my hobby projects. I’ve seen them queried a few times in my Apache logs by useragents claiming to be from OpenAI, Anthropic, etc. Also search engine crawler bots.
Here’s the IP whose useragent claimed to be an Anthropic bot, seems like others have encountered the same behaviour: https://abuseipdb.com/check/216.73.216.135