Google, Microsoft, OpenAI and Anthropic are backing a shared robots.txt signal that would let website operators state one preference about whether their content may be used for generative AI training.
OpenAI said on October 30, 2026, that it will honor a common signal in robots.txt for foundational generative AI training and that its existing crawler controls will continue to apply. Google published its own plan the same day, saying the shared control is based on work from the IETF AI Preferences working group and will let sites express training preferences across participating AI providers in a single step.
Microsoft’s Bing team also announced support for the shared training preference and added separate controls for AI grounding, the use of web content to inform generated responses. On the IETF AI Preferences mailing list, Kevin Kelley, whose draft lists Anthropic as his affiliation, wrote that Anthropic had already started implementing support for the draft, while saying support across its services would take time.
How the signal works
The proposed signal is a policy token written as __usage_ai-training, and it is designed to reduce today’s provider-by-provider setup, where a publisher may need separate rules for OpenAI’s GPTBot, Google-Extended, Anthropic’s ClaudeBot or Microsoft’s own controls.
The draft uses ordinary robots.txt groups, but the User-Agent line names a policy token rather than a crawler. A site-wide opt-out would look like this:
User-Agent: __usage_ai-training
Disallow: /Under the draft, that group states a preference against AI training for every path covered by Disallow: /. The same syntax can allow exceptions, such as a media kit or press folder, by adding an Allow rule for a narrower path.
The draft defines AI Training as using an asset to modify the learned parameters of a generative AI model. It also says the proposal adds no new robots.txt directives or grammar, so parsers that do not implement the signal should keep treating crawler rules as they already do.
Support is uneven for now
The four providers are not on the same timetable. Microsoft has the clearest dates, while Google and OpenAI describe planned support without a specific launch day, and Anthropic says implementation has begun.
| Provider | Stated timing | Key detail |
|---|---|---|
| 2027, with crawler documentation pointing to early 2027 | If the shared opt-out appears and Google-Extended is not explicitly specified, Google plans to treat it as if Google-Extended were disallowed. | |
| Microsoft | February 1, 2027, as it crawls | Microsoft is adding AI Training and AI Grounding preferences and retiring NoArchive and NoCache as AI training and grounding controls after March 1, 2027. |
| OpenAI | In the coming months | OpenAI says the shared signal will not replace its existing robots.txt crawler controls. |
| Anthropic | Implementation started, with no full-service date stated in the reviewed IETF message | Kelley said the draft complements Anthropic’s existing crawler controls rather than replacing them. |
The token does not block crawling
The easiest limit to miss is technical: the draft says the policy token is not a directive for crawling behavior. A site that wants to stop a bot from fetching pages still needs a crawler-specific robots.txt rule. A training preference and a crawl rule can live in the same file, but they answer different questions.
Another detail matters: User-Agent: * does not count as the shared training preference. The draft says wildcard groups must not be interpreted as matching the policy token. A robots.txt file with no User-Agent: __usage_ai-training group expresses no preference through this mechanism.
Training, search and grounding are being split apart
For site operators, the useful way to read the announcement is as a separation of three decisions: whether content may be used for model training, whether search crawlers may fetch and index pages, and whether content may be used to ground generated answers.
Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal, but its documentation also says Google-Extended applies to training future Gemini models and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. OpenAI’s crawler documentation similarly separates OAI-SearchBot for ChatGPT search features from GPTBot for training.
Microsoft is making the split more explicit. Its new training preference can be set in robots.txt or at the page level, while its grounding preference is page-level only. Microsoft says using Disallow or DisallowExtended for grounding will reduce visibility in AI-generated experiences, while the choices do not change eligibility for traditional Bing search results, subject to normal crawl and indexing requirements.
Tech Help Canada has also covered how AI is changing search, including the shift toward conversational answers and summaries.
Microsoft has the clearest deadline
Starting February 1, 2027, Microsoft says it will honor the new preferences as it crawls. Sites using NoArchive or NoCache to manage AI training or grounding preferences are being told to move to the new settings by March 1, 2027.
After that date, Microsoft says NoArchive and NoCache will no longer be read as preferences for training or grounding. A temporary Bing Webmaster Tools grounding preference opens December 15, 2026, for sites using those tags as of that date, and the temporary preference lasts until recrawl or September 30, 2027, whichever comes first.
What remains unresolved
The new signal is still a draft, not a finished internet standard. The IETF Datatracker page says the document is an individual Internet-Draft, is not endorsed by the IETF and has no formal standing in the IETF standards process. The draft is dated October 30, 2026, and expires May 3, 2027 unless it is updated or replaced.
The current materials also do not make the signal universal across every AI developer. They do not describe a built-in auditing system, a payment model, or a retroactive process for content that may already have been collected. The signal is a machine-readable preference that participating providers have said they will honor.
I can suggest related Tech Help Canada articles based on the topic you’re reading now.
Tech Help Canada Staff researches, writes, and reviews practical content for business owners and professionals. Our coverage spans business, marketing, SEO, technology, and the tools and systems people use to grow and operate online. We focus on clear, useful information backed by research, hands-on experience, and editorial review. Learn more about our team and editorial standards. Need help with something? Contact Us
