The idea that “AI scraping” is any more expensive than search engine indexing is flatly nonsense, only credible to people who have never run any network service at scale.

Folks need to learn about Common Crawl. https://commoncrawl.org/