Meta Removes Controversial Clause on AI Scraping Following Publisher Backlash
Meta has eliminated terms that critics warned could let it scrape news sites for AI training and ignore robots.txt instructions.
Meta has revised its commercial documentation, deleting clauses that critics argued could permit the tech company to scrape web content—including articles from news organizations—to train artificial intelligence systems. The decision follows substantial industry scrutiny and public pushback regarding whether Meta was claiming sweeping rights to use domain content across its broader product ecosystem.
At the center of the dispute was a specific phrase housed within Meta's commercial terms page. The section previously asserted that the company retained the right to utilize "any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain the integrity of, the Meta Company Products, services, and features provided under these Commercial Terms and any other applicable supplemental terms."
Industry criticism and robots.txt concerns
Observers quickly raised alarms after examining the text. Legal and media analysts, including Cornell digital and information law professor James Grimmelmann as well as Nieman Lab, viewed the language as an aggressive push by Meta. Under their interpretation, the provisions appeared designed to grant Meta permission to scrape publisher materials without financial compensation to feed machine learning models. Crucially, critics warned that the phrasing seemed crafted to override website operators' explicit blocking directives, such as instructions configured inside a domain’s standard robots.txt file.
Meta's defense and revised position
Meta leadership contested these interpretations. Andy Stone, Meta’s vice president of communications, alongside company spokesperson Daniel Roberts, maintained that Meta’s crawler policies had never altered. The representatives noted, “There is no change to how Meta uses web crawlers; any suggestion to the contrary is false.”
Despite that initial rebuttal, Meta elected to excise the disputed sentence from the commercial terms altogether. In an updated statement provided to The Verge, Roberts sought to clarify the company's intent behind the earlier wording. According to Roberts, the consolidated crawling terms were strictly intended for advertising and business offerings rather than a broader effort to siphon editorial content.
"We’ve heard you and want to clarify: the crawling terms we consolidated only applied to our ads and business products, and we’ve clarified that in the terms themselves," Roberts stated. He reiterated that the modifications did not alter Meta's web crawler operations, emphasizing that publishers retain control over their content through standard web protocols. "As always, content owners can use robots.txt to signal their preferences. This is not about scraping news sites, and these updates don’t give us new rights that supersede those preferences," he added.
What it means for developers
For developers and software engineers operating in the AI space, the dispute underscores the precarious balance between automated web data collection and site owner permissions. Meta's affirmation that robots.txt directives remain respected provides reassurance for platform builders and web publishers who rely on standardized crawl permissions to manage how automated agents interact with their sites.
As scrutiny over AI training sets intensifies, tracking the terms of service from major tech entities is increasingly critical. For developers integrating machine intelligence into their applications, understanding how different model providers source data and enforce crawling rules remains a foundational requirement. Teams looking to experiment across multiple AI architectures can try top AI models cheaply through one API at https://apixoai.online, simplifying the process of building with industry-leading frontier models while maintaining focus on application development and compliance.
The swift removal of the contentious clause illustrates how rapidly model developers must react when commercial guidelines appear to infringe upon domain governance. For both developers building scraping pipelines and publishers protecting proprietary content, standard web signals like robots.txt continue to be the primary benchmark in defining how training data can be gathered.
Source: Meta removes language that may have allowed it to scrape news sites — The Verge AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
