Major AI firms back new standard for publishers to block scraping
OpenAI, Google, Microsoft, and Anthropic announced support for a new opt-out standard that allows publishers to control AI data scraping across multiple companies.
Proposed policy token mechanism
Kevin Kelley proposed this shared signal as part of the Internet Engineering Task Force. The system will live within a website robots.txt file to signal preferences. This approach lets publishers apply their AI training rules across multiple crawlers at once.
Current limitations for sites
Publishers currently must set scraping preferences separately for each individual AI company. The new control aims to solve this issue by standardizing the opt-out process. Sites face a continuing influx of crawlers feeding content into models without consent.
Existing industry alternatives
Some organizations have tried Really Simple Licensing to charge visiting crawlers for access. Other tools like Cloudflare now block AI crawlers by default on their platforms. These methods exist alongside the new proposed standard for broader adoption.
Reported by one outlet
Only one outlet has published this. Nothing here has been checked against a second report, so read it as that outlet's account and follow the link below for the original.
Reported by
1 independent outlet. Headline as published. Links open the original report.