• kcweller@feddit.nl
    link
    fedilink
    arrow-up
    72
    ·
    20 hours ago

    Robots.txt expects that the client is respecting the rules, for instance, marking that they are a scraper.

    AI scrapers don’t respect this trust, and thus robots.txt is meaningless.