OpenAI, Google, Microsoft and Anthropic Back Shared AI Training Signal

OpenAI, Google, Microsoft, and Anthropic have backed a shared mechanism that will let website owners communicate their preferences about AI training through a single rule in their site's robots.txt file, reducing the need to maintain separate instructions for each participating provider.

Announced on October 30, the proposal centers on a policy token called __usage_ai-training and is intended to make it easier for publishers to indicate whether their content may be used to train generative AI models.

However, the mechanism is not yet a finalized Internet standard, and the companies have announced different implementation timelines.

For publishers, the development addresses a growing operational problem: controlling how publicly accessible articles, images, and other web content may be used by AI companies without managing a different training preference for every provider.

The shared signal offers a common approach, but it does not automatically block website crawling, govern every use of content, or guarantee that participating companies will interpret every situation identically.

A Single Rule Could Replace Multiple AI Training Instructions

For years, website operators have been able to use robots.txt to communicate instructions to automated crawlers.

The difficulty is that AI providers have used different crawler names and individual controls, requiring publishers to maintain separate rules when they want to distinguish AI training from other activities.

OpenAI, Google, Microsoft, and Anthropic are now backing a common policy token intended to reduce that administrative burden.

The proposal was documented in an Internet-Draft titled Machine-Readable Training Preference Signal, submitted by Kevin Kelley of Anthropic to the Internet Engineering Task Force (IETF) on October 30, 2026.

The draft describes how a website can use an ordinary robots.txt group to express a preference about AI training for an entire site or specific paths.

The key change is that a publisher would no longer need to express the same training preference separately under every participating AI company's crawler name.

Instead, the shared token provides a common way for compatible systems to interpret that preference.

For website owners managing large content libraries, this could simplify technical administration and reduce the risk of overlooking an individual provider when updating crawler rules.

The mechanism remains voluntary.

Its practical effect depends on each provider implementing and honoring the signal, as well as on the exact scope of the rules published by the website operator.

What Is the __usage_ai-training Signal?

The proposed signal uses a policy token in the User-Agent line of a robots.txt group.

Rather than identifying one company's crawler, the token identifies the intended use of the content: AI training.

In the draft's terminology, AI training means using content to modify the learned parameters of a generative AI model.

That definition distinguishes training from other activities, such as using web content to inform a response at the time a user asks a question.

The distinction matters because a publisher may want to prevent its articles from being used to train a model while still allowing those articles to be discovered in search engines or referenced in AI-generated answers.

The shared signal is designed to communicate the training preference.

It does not, by itself, determine every other way a provider may access or use a page.

How the preference works

The proposed mechanism follows the existing structure of robots.txt.

A website can express a preference against AI training for all paths or use more specific path rules.

For example, a publisher could indicate that content across its site should not be used for AI training while making a particular directory, such as a press-kit section, available for that purpose.

The draft uses the following illustrative configuration:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Under the proposal, the rules express a preference against training on the site's content except for the specified press-kit path.

The mechanism uses existing path-matching behavior rather than introducing a new set of robots.txt directives.

Website owners should not assume that adding the token immediately changes how every AI company handles their content.

A rule only has the intended effect with providers that support and honor the mechanism, and each company's implementation schedule matters.

Why the Change Matters to Publishers and Independent Websites

The shared signal arrives as publishers seek clearer ways to manage the relationship between open-web publishing and AI development.

News organizations, independent blogs, documentation providers, educational websites, and commercial publishers invest in producing content that may later be collected by automated systems.

Some want their work to remain accessible to readers and search engines but do not want it used to train generative AI models without permission or a separate arrangement.

Until now, expressing those preferences has often required technical decisions at the crawler level.

A common signal could make that process easier for publishers that lack dedicated engineering teams.

Instead of maintaining separate training rules for multiple AI providers, a site operator could publish one shared preference and retain company-specific controls where needed.

That does not settle broader disputes about copyright, licensing, compensation, or the legal basis for using publicly accessible material.

It provides a more consistent technical method for expressing a website owner's preference.

The difference is important: a machine-readable preference is not the same as a licensing agreement or a legal ruling.

A publisher's choice to disallow AI training through robots.txt does not, on its own, establish whether earlier uses of its content were lawful or whether a company owes payment.

Those questions depend on facts, agreements, and applicable law beyond the scope of the proposed signal.

OpenAI, Google, Microsoft, and Anthropic Have Different Rollout Plans

Although the four companies have backed the shared approach, they have not announced identical implementation schedules.

That distinction is central to understanding what website owners can expect next.

OpenAI plans implementation in the coming months

OpenAI announced on October 30 that it will honor the common signal in robots.txt for preferences concerning the use of website content in foundational generative AI training.

The company said it aims to implement the signal in the coming months and will publish additional details when implementation progresses.

OpenAI also confirmed that the new mechanism will not replace its existing crawler controls, which will continue to apply.

For publishers, this means existing OpenAI-specific controls remain relevant during the transition.

A website should not assume that the shared token is already being applied by OpenAI simply because the company has announced support.

Google targets support in 2027

Google announced plans to support the shared AI training control in 2027, building on its existing Google-Extended control.

Google says that when a site uses the shared signal to opt out of AI training and does not explicitly specify Google-Extended, it plans to interpret the shared opt-out as though the publisher had disallowed Google-Extended.

Existing Google-specific controls will continue to work alongside the shared signal.

This is an important detail for publishers concerned about losing visibility in search.

Google-Extended is a separate control that allows publishers to manage certain uses of their content by Google's generative AI systems.

Google has stated that publishers can control eligible AI uses through Google-Extended without that control itself affecting a page's inclusion in ordinary Google Search results.

Publishers should nevertheless distinguish traditional search visibility from participation in AI-generated experiences.

They are related aspects of web discovery, but they are not identical.

Microsoft plans to begin honoring the preferences in February 2027

Microsoft announced support for the shared AI training preference as part of a broader update to publisher controls.

Starting February 1, 2027, Microsoft says it will honor the new preferences as its crawler visits websites.

The company is also introducing separate page-level controls for AI training and AI grounding, allowing publishers to make different choices about these uses.

Microsoft's announcement provides more detailed transition instructions for websites already using its older NoArchive and NoCache tags to manage AI-related preferences.

Those publishers should review Microsoft's migration guidance rather than assume that their existing settings will automatically carry over to the new system.

Anthropic has begun implementation

The IETF draft was submitted by Kevin Kelley, an Anthropic standards lead.

The draft's publication establishes the technical proposal, while reporting on the announcement indicates that Anthropic has begun implementing support across its services.

The timing and completeness of support across individual Anthropic services remain important considerations for publishers.

Website operators should check the company's current documentation rather than assume that every service already recognizes the signal.

The broader lesson is that public support for a shared mechanism and complete operational deployment are different milestones.

AI Training Is Not the Same as AI Grounding

One of the most important distinctions in the announcement is the difference between training a model and using web content to inform a response.

These activities may involve the same article, but they serve different technical purposes.

AI training involves using content to modify the learned parameters of a generative AI model.

The content contributes to the model's learned capabilities rather than simply being consulted as a source for a particular answer.

AI grounding involves using external information to help inform an AI-generated response.

A system may retrieve a webpage, use information from it while preparing an answer, and provide a link to the original source without that retrieval itself constituting model training under the proposed definition.

This distinction matters to a publisher deciding how its content should be used.

A news organization might want to prevent its reporting from being used to train generative models while allowing search systems and AI assistants to discover articles and reference them in responses.

Another publisher might prefer to restrict both training and use in AI-generated answers.

The shared token addresses training preferences.

Microsoft's separate AI grounding controls are designed to address the second type of use within Microsoft's own services.

The shared training token should not be mistaken for a universal control covering all AI-generated responses.

Microsoft's additional grounding controls

Microsoft's new page-level grounding preference provides three options:

• Allow: Content may be used to help inform generated responses.

  This is the default when no applicable preference is published.

• Disallow: Content must not be used for grounding under the stated Microsoft preference.

• DisallowExtended: Grounding is limited to specified Microsoft-operated experiences and excludes certain grounding services Microsoft provides to third parties.

The controls apply to Microsoft's stated use of content in experiences including Bing, Copilot, and Microsoft 365, subject to the scope of the selected preference.

Microsoft says these choices do not change eligibility for traditional Bing search results, although restricting grounding can reduce a publisher's visibility within Microsoft's AI-generated experiences.

This creates a practical decision for publishers: whether they want to restrict model training, the use of content to inform generated responses, or both.

It also highlights why a single opt-out switch for every kind of AI activity would not necessarily meet every publisher's needs.

The Proposal Is Not Yet an Official Internet Standard

Despite support from several major AI companies, the shared signal should not be described as a finalized IETF standard.

The document published on October 30 is an Internet-Draft titled Machine-Readable Training Preference Signal.

The IETF Datatracker identifies it as an individual active draft and explicitly states that it has not been endorsed by the IETF and has no formal standing in the organization's standards process.

The current draft lists May 3, 2027, as its expiration date.

Internet-Drafts are working documents that can be revised, replaced, or allowed to expire.

Their publication does not guarantee adoption as a formal standard.

The distinction matters because implementation details may evolve as the working group continues its discussions.

The proposal is based on work from the IETF's AI Preferences working group, which has been developing ways for publishers to communicate how their content may be used.

The four companies' support represents a significant step toward a common approach, but the wider standardization process remains ongoing.

Publishers should therefore avoid treating every detail in the current draft as permanently settled.

A shared signal does not force every AI company to comply

The mechanism is voluntary, and the draft does not itself impose a universal legal obligation on every AI provider.

Its effectiveness depends on providers choosing to recognize and honor the preference.

A company that does not implement the signal may continue to interpret its own crawler instructions according to its existing behavior.

The proposal also distinguishes content-use preferences from crawling restrictions.

The shared token communicates how content should be used for AI training; it is not, by itself, a directive that blocks a crawler from visiting the website.

A publisher that wants to restrict crawling must continue to use appropriate crawler-specific rules or other technical measures.

This is why the announcement is best understood as an effort to improve consistency, not as a universal technical or legal enforcement system.

What Website Owners Should Do Now

Website operators do not need to wait until every provider has completed implementation before reviewing their existing policies.

However, changes should be made carefully, particularly on sites that depend on organic search traffic or have existing contractual arrangements.

1. Review the current robots.txt file

Start by checking which crawler rules are already published.

A website may have separate instructions for search crawlers, AI training crawlers, and other automated services.

Record which rules are intended to control access and which are intended to communicate preferences about content use.

Do not remove existing controls simply because a shared mechanism has been announced.

2. Decide what uses of content the business permits

The decision should be made before changing technical configuration.

Ask whether the site should allow its content to be used for generative AI training, whether it wants to permit content to inform AI-generated answers, and whether different rules should apply to particular directories or pages.

These are separate decisions, and the appropriate configuration depends on the organization's publishing objectives, rights, and agreements.

3. Keep training and search visibility separate in the policy

Blocking AI training does not necessarily require blocking ordinary search crawling.

A publisher may want its pages to remain discoverable through Google Search or Bing while communicating a preference against training.

Review crawler-specific rules carefully so that an attempt to restrict model training does not unintentionally block search engines from accessing content needed for indexing.

The distinction between training, grounding, and ordinary search crawling should be reflected in the site's technical policy.

4. Monitor implementation announcements

The companies have announced different schedules, and the relevant technical details may change.

OpenAI says it aims to implement the signal in the coming months.

Google targets 2027, while Microsoft has announced February 1, 2027, as the date it will begin honoring the new preferences as it crawls websites.

Anthropic has begun implementation, but support across its services may take time.

Website owners should confirm each provider's current documentation before assuming that a published preference is already being applied.

5. Review older Microsoft controls before the transition deadline

Microsoft has said that its older NoArchive and NoCache preferences will no longer be read as controls for AI training or grounding after March 1, 2027.

For affected sites, Microsoft has published a migration process that includes new training and grounding preferences and, for eligible existing configurations, temporary transition settings through Bing Webmaster Tools.

Publishers using those older tags should review the official instructions and complete the relevant steps before the deadline.

The transition requirements differ depending on whether the publisher is managing training, grounding, or both.

6. Test the configuration and document the decision

After updating the site's policies, verify that the robots.txt file is accessible and that the rules match the intended paths.

Document why the organization selected its preferences, who approved the changes, and when the configuration was last reviewed.

For larger publishers, legal, editorial, SEO, and engineering teams should coordinate these decisions.

A change intended to protect content rights can have operational consequences if it accidentally blocks a search crawler or affects an AI service the publisher wants to remain discoverable through.

A technical review helps reduce that risk.

What the Shared Signal Means for Desking and Other Digital Publishers

For technology publications such as Desking, the announcement raises an important editorial and business question: how should a publisher balance content protection with visibility across search engines and AI-powered services?

The answer depends on the publication's objectives.

A website may want its reporting to appear in traditional search results, attract referral traffic, and be cited as a source in AI-generated answers while restricting the use of its articles for model training.

Those goals are not inherently incompatible, but they require publishers to understand the separate controls offered by each platform.

Publishers should avoid confusing training controls with visibility controls

A restriction on AI training does not automatically mean that a website will disappear from Google Search or Bing.

Google's existing Google-Extended control and Microsoft's newly announced preferences illustrate that AI-related uses can be managed separately from ordinary search visibility, although individual platform features and eligibility requirements differ.

OpenAI also distinguishes between its search crawler, OAI-SearchBot, and its training crawler, GPTBot.

These controls serve different purposes, and the new shared signal is not intended to replace the company's existing crawler controls.

Publishers should therefore make separate decisions about search discovery, AI training, and AI grounding instead of treating all automated access as one category.

A shared preference is not a substitute for content licensing

The new mechanism also does not establish a payment system for publishers.

Microsoft explicitly states that its new preferences do not guarantee traffic, inclusion in AI experiences, or payment.

The signal is a technical means of communicating content-use preferences, not a commercial agreement between a publisher and an AI company.

Publishers seeking licensing revenue will still need to evaluate licensing arrangements and other relevant commercial or legal options independently.

Technical implementation is becoming part of editorial strategy

As AI-powered search and answer systems become part of how audiences discover information, publishers increasingly need to consider how their content is accessed and used.

That does not mean every publication should block AI training.

Nor does it mean that allowing training guarantees additional traffic or citations.

The more useful approach is to define the publication's objectives, understand the available controls, and make an informed decision about the trade-offs.

For an independent technology publisher, a technically correct policy can help communicate its preferences consistently.

It cannot guarantee how every AI company will respond or how much referral traffic the publication will receive.

Sources Consulted

OpenAI — Making It Easier for Websites to Express AI Training Preferences

OpenAI's official announcement of its support for the shared signal, its planned implementation, and the continuation of existing crawler controls.

Google Search Central — Supporting a Shared AI Training Control for Websites

Google's official explanation of the shared signal, its relationship with Google-Extended, and its implementation target.

Microsoft Bing — Your Content, Your Choice: Clearer Preferences for Training and Grounding

Microsoft's official announcement of shared training preferences, separate AI grounding controls, the February 2027 implementation date, and the transition requirements for older NoArchive and NoCache settings.

Internet Engineering Task Force — Machine-Readable Training Preference Signal

The technical proposal defining the __usage_ai-training policy token, its intended use, and its status as an Internet-Draft.

OpenAI — Overview of OpenAI Crawlers

OpenAI's documentation distinguishing search-related crawling from crawling that may be used for training.

The Verge — The Biggest Names in AI Are Backing a New Way to Opt Out of Scraping

Independent reporting providing additional context about the companies' announcements and the broader debate over AI training preferences.