Major AI Providers Back Unified Web Crawling Opt-Out Standard
Google, Microsoft, Anthropic, and OpenAI are collaborating on an updated robots.txt standard to let publishers block generative AI training scrapers with a single entry.
A group of leading artificial intelligence companies has unveiled plans to support an updated standard for website crawling. Google LLC, Microsoft Corp., Anthropic PBC, and OpenAI Group PBC are backing a proposal designed to make it simpler for site owners to stop automated scrapers from gathering online content to train generative AI models.
Traditionally, web administrators rely on a root text file known as "robots.txt" to control how automated bots access their domains. Governed by the Robots Exclusion Protocol, this mechanism allows publishers to restrict crawlers from indexing specific pages, block entire sites, or enforce visit frequency rules. While originally created to manage search engine indexing, robots.txt has more recently become the main tool for handling automated scrapers deployed by AI labs.
Simplifying Data Scraping Controls
Under current web standards, restricting AI crawlers requires significant manual configuration. If a webmaster wants to block training-related scraping across different AI systems, they must research and write separate entries for each individual provider. Furthermore, if a single AI company deploys multiple web crawlers to collect training corpora, every unique bot requires its own distinct rule within the robots.txt file.
The new standard supported by Google, Microsoft, OpenAI, and Anthropic aims to eliminate this operational complexity. Under the proposed update, website owners will be able to disallow all scrapers that gather training data for generative AI through a single directive. The companies noted that this restriction would not apply to crawlers associated with models performing purely non-generative tasks.
To standardize these rules across the broader web ecosystem, the coalition has submitted a draft proposal to the Internet Engineering Task Force (IETF), the nonprofit organization responsible for maintaining the Robots Exclusion Protocol.
Separate Settings for AI Grounding and Retrieval
Beyond training data exclusions, companies are also introducing granular controls over live data retrieval. Microsoft announced a new crawling directive called AI Grounding, intended for scenarios where a model visits a webpage in real time to answer a user's prompt rather than gathering information for pre-training.
By enabling this setting, website operators can prevent Microsoft services from fetching their pages to fulfill live queries. Microsoft indicated that this safeguard will also cover third-party applications relying on Microsoft technology to process user prompts. OpenAI plans to roll out similar controls for real-time browsing and grounding.
Rollout timelines vary across the participating organizations. Both Microsoft and OpenAI intend to roll out implementation for the updated standard over the coming months. Google plans to incorporate the new protocol sometime next year. Meta Platforms Inc. has expressed support for the concept behind the opt-out mechanism, though it has not yet committed to implementing the standard.
What it means for developers
For developers building web applications, managing content platforms, or maintaining digital properties, this update promises to significantly reduce maintenance overhead. Instead of constantly monitoring new bot user-agent strings and updating robots.txt files whenever an AI lab launches a new scraper, engineers will soon be able to set a single unified rule to manage generative AI training access.
At the same time, engineering teams designing retrieval-augmented generation (RAG) systems or integrating real-time web search into AI applications need to be aware of the emerging distinction between training crawlers and grounding bots. As site owners implement separate directives for AI Grounding, tools that rely on live web retrieval may find certain high-value domains unreachable for prompt responses.
For teams experimenting with different foundation models to see how they handle content parsing, reasoning, or grounding, services like Apixo let developers try top AI models cheaply through one API at https://apixoai.online without managing multiple platform agreements. As the IETF evaluates the draft specification and providers like OpenAI, Microsoft, and Google implement it across their crawling fleets, developers should track how these rules affect both public web content accessibility and automated data collection pipelines.
Source: Google, Anthropic and others to support new AI crawling opt-out standard — SiliconANGLE AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
