Anubis, which block AI scrapers from scraping websites to death, has been downloaded almost 200,000 times.
  • interdimensionalmeme@lemmy.ml
    9·
    23 days ago

    If you allow my searchxng search scraper then an AI scraper is indistinguishable.

    If you mean, “google and duckduckgo are whitelisted” then lemmy will only be searchable there, those specific whitelisted hosts. And google search index is also an AI scraper bot.