Major AI Labs Adopt Unified Standard for Web Scraping Opt-Outs
OpenAI, Google, Microsoft, and Anthropic are backing a new shared signal allowing publishers to opt out of AI training across multiple platforms at once.
A group of leading artificial intelligence companies—including OpenAI, Google, Microsoft, and Anthropic—have announced plans to adopt a unified standard designed to help web publishers control how their content is scraped. This development introduces a shared "policy token" signal that lets websites apply their training preferences to multiple AI providers simultaneously, replacing the need to configure restrictions for each individual company.
The proposed mechanism was created by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF). The token operates directly inside a website's robots.txt file, a traditional location used for communicating with web crawlers regarding search indexing, site archiving, and now, artificial intelligence training data collection.
Streamlining Publisher Controls
Until now, publishers dealing with an increase in automated crawlers feeding content into machine learning models have had to manage their preferences separately for every AI developer. Various tools and methods have emerged to address unauthorized data harvesting, such as Cloudflare blocking AI scrapers by default and the Really Simple Licensing standard allowing websites to charge visiting crawlers.
The newly backed policy token aims to simplify this landscape. OpenAI, Google, Microsoft, and Anthropic have stated they will keep their crawlers away from any site that utilizes the shared signal. Microsoft has clarified that it will provide additional granular options, enabling websites to selectively block either AI training or AI grounding—a technique that links models to verified sources to minimize hallucinations. Microsoft also emphasized that choosing to opt out of AI features will not impact a site's visibility in standard Bing search results.
What it means for developers
Developers working on AI-driven applications must navigate an evolving landscape of data governance, copyright measures, and technical standards. While these new opt-in and opt-out mechanisms change how training data is sourced, developers building applications can try top AI models cheaply through one API at https://apixoai.online.
Rollout timelines for the shared token vary across the industry. Anthropic has already implemented the signal, whereas Google and Microsoft intend to roll out support over the course of next year. Meanwhile, OpenAI stated it will incorporate the standard in the coming months.
Source: The biggest names in AI are backing a new way to opt out of scraping — The Verge AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
