OpenAI announced on October 30, 2026 that it will honor a shared signal in robots.txt, the file websites use to express preferences about how their content is used, so that site operators can communicate a single AI training preference to multiple participating providers at once.
A Voluntary Signal Shared Across Providers
In its announcement, OpenAI described the effort as voluntary and said it addresses an existing burden: until now, website operators had to identify individual crawlers and maintain separate rules for each and every provider. A shared signal, the company said, lets publishers and website operators communicate the same preference to multiple participating providers at once. OpenAI said it aims to implement the signal in the coming months and will share another update with implementation details when it does.
The company said the new signal will not replace the controls it already provides through robots.txt, which it said will continue to apply and be respected. OpenAI framed the move as drawing on the ongoing work of the Internet Engineering Task Force’s AI Preferences working group, said it is joining other working group participants in making the announcement, and encouraged other AI providers to honor the signal. The company described the step as moving toward an open web where people can publish and discover information and website owners have clearer, simpler ways to communicate their choices about how their content is used, and said it will continue to adapt its practices as those standards conversations evolve.
How the Signal Works
The signal is defined in an individual Internet-Draft titled Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00), authored by Kevin Kelley of Anthropic, dated October 30, 2026 and expiring May 3, 2027. The IETF’s datatracker notes that an individual Internet-Draft is not endorsed by the IETF and has no formal standing in the IETF standards process.
Under the draft, a service operator binds usage preferences, such as “no AI training,” to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line. The draft introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of the robots.txt standard published as RFC 9309 are unchanged. As a result, the draft says, existing parsers deployed across the internet will continue to interpret traffic-management signals as they did previously and will ignore policy-token groups as they would any group naming a user agent they do not recognize.
The draft defines the cross-vendor policy token __usage_ai-training. Its example group pairs the line “User-Agent: __usage_ai-training” with “Disallow: /” and “Allow: /press-kit/”, which expresses a preference against AI training for all paths except those under /press-kit/. Wildcard groups, meaning “User-Agent: *”, must not be interpreted as matching the policy token, and if no group in a file contains the token, then no preference has been expressed.
The draft also states that the policy token is not a directive for crawling behavior. Content-usage preferences and crawler traffic-management signals are independent and orthogonal, it says, and private agreements may override preferences expressed through the mechanism. Its stated goals include path-specific bindings, minimal distance from RFC 9309, no large-scale duplication of rules, unchanged behavior for parsers that do not implement the mechanism, and consistent interpretation across consumers.
The Vocabulary Behind the Signal
The token’s meaning tracks the working group’s vocabulary draft, A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-08), written by Paul Keller of Open Future and Martin Thomson of Mozilla and last updated in September 2026 as a working group document intended as a Proposed Standard. That vocabulary defines AI Training as using an asset to modify the learned parameters of a generative AI model, excluding the training of models used for exclusively non-generative tasks, and it defines a generative AI model as one used, or made available for use, to generate synthetic content in one or more modalities.
The vocabulary also defines a separate AI Use category, covering assets used as input to a generative AI model rather than provided directly by a user, and a Search category for applications whose primary purpose is selecting assets and directing users to their locations. Each category receives a preference value of allowed, disallowed, or unknown, and where statements conflict, the most restrictive preference applies. The vocabulary draft states that it does not ensure preferences are followed and leaves the choice of whether to follow them to the recipient, and a note to readers cautions that its contents do not yet reflect working group consensus. One section, whose inclusion is still tracked as an open issue, states that complying with training preferences includes conveying the preferences associated with training assets along with a model when it is distributed.
The training-signal draft’s acknowledgments name working group chairs Mark Nottingham of Cloudflare and Suresh Krishnan of Cisco, vocabulary editors Thomson and Keller, and responsible area director Mike Bishop of Akamai. They also list individual participants, with affiliations noted for identification only because IETF participants contribute as individuals, from organizations including OpenAI, Google, Microsoft, Meta, Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation, and Creative Commons.
OpenAI’s Existing Crawler Controls
OpenAI’s crawler documentation, which the company said will remain in effect alongside the new signal, describes GPTBot as the agent that crawls content that may be used in training its generative AI foundation models; disallowing GPTBot indicates a site’s content should not be used for that training. OAI-SearchBot surfaces websites in ChatGPT’s search features, and sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. OAI-AdsBot validates the safety of web pages submitted as ads on ChatGPT, and the data it collects is not used to train generative AI foundation models. ChatGPT-User handles page visits triggered by user actions in ChatGPT rather than automatic crawling, and the documentation notes robots.txt rules may not apply to those visits.
Each setting is independent of the others, according to the documentation, so a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot for training. Search adjustments can take about 24 hours after a site’s robots.txt update. OpenAI said it will share implementation details for the shared signal when it implements it in the coming months.