A so-called “policy token” will prevent scraping by the largest AI model developers. This standard, which is part of the existing robots.txt file, has the support of OpenAI, Google, Microsoft, and Anthropic.
Until now, publishers had to adopt a separate anti-scraping standard for each AI company. Kevin Kelley, Technical Lead for Standards at Anthropic, proposed the standard. While it is not an official IETF standard, it is expected to gain the support of the leading Western AI companies by next year. To adopt this standard, publishers add a separate group to their robots.txt file, specifically with the user-agent __usage_ai-training and a Disallow rule. Exceptions for specific directories remain possible to control scraping in a nuanced way.
Until now, sites set their preferences on a per-crawler basis. For example, those who wanted to block OpenAI would block GPTBot. Google used a different token, and for Anthropic, ClaudeBot was the bot to block. With the shared token, that is no longer necessary. The four companies say their AI crawlers will skip a site as soon as they encounter the token.
Meta, the largest player not included in the list of four adopters, supports the initiative but has not said whether it will also honor the token. “We believe that a common signal that conveys such preferences to multiple providers at once is an important step,” said Head of Communications Andy Stone.
Training or grounding
There will still be differences between the implementations of the AI companies. For example, Microsoft offers publishers more choice. They can specify separately whether they want to opt out of AI training or AI “grounding.” With grounding, an AI model refers to a verified source to limit hallucinations; this can help generate traffic from AI responses without the source having been used for training. Regardless of the publisher’s choice, sites will remain visible in Bing. Google previously offered similar options that do not affect search results, even though traffic to online sources has plummeted in recent years for multiple reasons, with AI being a partial culprit.
Incidentally, not everyone is adopting the token at the same pace. Anthropic already supports the standard, but Google and Microsoft will follow suit next year. OpenAI hasn’t specified a date and suggests it will adopt the standard sometime in the coming months.
Still voluntary
In any case, publishers can still prevent scraping. Robots.txt has always been a voluntary mechanism. Google had previously sought to make the decades-old protocol an internet standard. However, crawlers can ignore the instructions. This was demonstrated once again when Cloudflare accused Perplexity of circumventing crawl limits. Pressure on websites is also growing. Wikimedia saw its bandwidth increase by 50 percent due to AI crawlers, without people actually viewing the information from this source.
Other parties have taken strong measures to limit their traffic. Cloudflare previously introduced a feature to block AI scrapers and now blocks them via an opt-in. With Bot Preference Sync, the company also converts preferences for search, AI input, and AI training into robots.txt rules. Using the Really Simple Licensing (RSL) standard, publishers can require crawlers to pay for access, as an alternative to simply blocking or allowing scraping.
See also: Cloudflare introduces bot detection that analyzes entire browser sessions