OpenAI says it will start reading a shared robots.txt signal that tells AI providers whether a site's content may be used for training. The announcement, published October 30, 2026, is a commitment rather than a launch: OpenAI "aim[s] to implement the signal in the coming months" and will publish another update with details when it does. Existing crawler rules keep working in the meantime.
This post separates what OpenAI actually committed to from what the web has been guessing, explains the IETF standards effort that the announcement leans on, and gives site owners a checklist for today.
TL;DR: what is confirmed and what is not
| Question | Answer |
|---|---|
| What did OpenAI announce? | It will honor a common robots.txt signal for AI training preferences, shared with other participating providers |
| Is it live? | No. OpenAI aims to implement it "in the coming months" and will publish another update |
| Does it replace GPTBot rules? | No. OpenAI says existing crawler controls "will continue to apply and be respected" |
| What syntax will it read? | Not stated in OpenAI's post. The IETF draft Content-Usage: train-ai=n is the likely basis, unconfirmed |
| Is it enforceable? | No. It is voluntary and works only for providers that choose to honor it |
| Who else is in? | OpenAI says it joins other IETF AI Preferences working group participants; the post does not list them |
Update (October 30, 2026): Google, Microsoft and Anthropic join, with different timelines
Later on October 30, The Verge reported, citing Search Engine Roundtable, that OpenAI, Google, Microsoft and Anthropic all say they will adopt the shared "policy token" signal, answering the open question above about who else is in. The proposal comes from Kevin Kelley, a standards technical lead at Anthropic, through the IETF, and it lives in a site's robots.txt so a publisher can state its AI training preference once instead of per company.
Timelines differ, according to The Verge: Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says "coming months." The four say their AI crawlers will stay away from a site that carries the token. Meta's communications head Andy Stone signaled support for a common signal but, per The Verge, did not say whether Meta will honor it.
Microsoft also said it will offer finer controls that separate opting out of AI training from opting out of AI grounding (pointing a model at verified sources), and that sites keep appearing in Bing results either way. Google already offers controls over AI search features and training without hurting search visibility. The still-unconfirmed parts are the exact syntax each company will read and whether the IETF draft is final; our checklist below still applies. Related: Google Zero and the publisher crawler-blocking debate.
What OpenAI actually said
The OpenAI post, filed under Policies and Procedures, makes four points.
First, the problem: "Previously, website operators would identify individual crawlers and maintain separate rules for each and every provider." A shared signal "reduces that work for operators."
Second, the commitment: OpenAI "will honor a common signal in robots.txt" for the use of content "for foundational generative AI training." It calls the effort voluntary.
Third, the limits: the new signal "will not replace the controls we already provide through robots.txt." Existing crawler controls stay.
Fourth, the standards link: the move "draws on the ongoing work of the Internet Engineering Task Force's (IETF) AI Preferences working group," and OpenAI is "joining other participants in the working group in today's announcement of this shared signal." It also "encourage[s] other AI providers to honor the signal."
What the post does not contain is a code sample, a date, a list of co-announcing companies, or a statement about content already collected. Those gaps matter and are covered below.
How robots.txt works today, and why a shared signal helps
Robots.txt, standardized as RFC 9309, is an access preference protocol. Google's Gary Illyes put it this way in a Search Central video covered by Search Engine Journal: robots.txt "doesn't dictate what a system does with the data after it has already crawled it." To keep one AI crawler out while allowing others, he said, you have to target that crawler's user-agent token explicitly.
That is the pain OpenAI describes. A publisher who wants to stay in search but out of model training must know and maintain a growing list of tokens: GPTBot for OpenAI, Google-Extended for Gemini training and grounding, plus a long tail from other labs. Some tokens cover training only, some also cover live retrieval, and a few providers have separate agents for user-triggered fetches. We looked at the operational side of this in our guide to protecting a small site from AI bot floods.
There is also a conceptual gap. A Disallow rule blocks fetching, so it blocks everything the crawler might do with the page, including search indexing in some cases. What many publishers want is the opposite split: allow crawling, decline training. Classic robots.txt has no vocabulary for that.
The IETF draft behind the announcement
The "common signal" OpenAI refers to links to the IETF's work, and the public drafts show what that work looks like. The attachment draft, draft-ietf-aipref-attach-05 (dated August 19, 2026, authored by Google's Gary Illyes and Mozilla's Martin Thomson), defines two ways to attach usage preferences to content:
- A
Content-UsageHTTP response header. - A
Content-Usagerule inside robots.txt.
The draft's own example shows the shape:
User-Agent: *
Allow: /
Disallow: /never/
Content-Usage: train-ai=n
Content-Usage: /ai-ok/ train-ai=y
User-Agent: ExampleBot
Allow: /
Content-Usage: train-ai=y
Under this design, crawling and usage are separate stages. Allow and Disallow decide what may be fetched; Content-Usage states a preference for what may be done with what was fetched. The draft says preferences "apply only to those resources that can be crawled," and that no preferences are implied for disallowed paths. In the example, everything outside /never/ can be crawled but is marked train-ai=n, except /ai-ok/, which is marked train-ai=y. A bot named ExampleBot gets its own group that opts in.
Two details matter for implementers. Conflicting Content-Usage rules on identical paths are resolved by the vocabulary draft's process rather than the "most permissive wins" logic Allow and Disallow use. And preferences are not retroactive: the draft states that a crawler uses the robots.txt that is current at fetch time, so a changed file only affects content fetched after the crawler re-reads it. Crawlers may cache robots.txt for up to 24 hours.
What is claimed versus what is verified
Verified from OpenAI's own page: the commitment, the "coming months" timeline, the statement that existing controls stay, the IETF link, and the call for other providers to join.
Verified from the IETF documents: the Content-Usage rule exists as a working-group draft, and train-ai is a label in a companion vocabulary draft. Search Engine Journal reports that the attachment draft is still a Working Group document that has not gone to the IESG for review, and that the vocabulary draft, dated September 14, carries a note that it reflects no consensus within the working group. The attachment draft expires on February 20, 2027 unless replaced.
Not verified:
- That OpenAI will read exactly
Content-Usage: train-ai=n. The post says "a common signal" and links to working-group material, but the page text does not print a directive. Treat the IETF line as the leading candidate, not a spec OpenAI has promised. - Which other companies are announcing alongside OpenAI. The post says "other participants" without naming them.
- Whether the signal will cover content crawled before it is honored. Nothing in the post says it does, and the IETF draft explicitly says changes are not retroactive.
- Whether it governs retrieval and agent browsing. OpenAI scopes it to "foundational generative AI training."
- What happens to pages that contain no signal. Today the default for most sites is silence, and OpenAI has not said how it will treat silence.
If a later OpenAI update prints the syntax, this section is the first thing to revise.
Why this matters for the training-data fight
Publisher disputes with AI labs are mostly about consent and compensation. USA Today's $250 million suit against OpenAI and the broader AI lawsuit tracker show how much of that fight is about whether training happened without permission. A machine-readable, standards-based way to say "no" gives courts, regulators and publishers a clearer record of what was declared and when. It does not settle whether training on unlicensed content is lawful; the Wikimedia brief in NYT v. OpenAI is a good read on why training and retrieval are being argued separately.
It also lands in a week when other platforms are reshaping access. Meta's new commercial terms raised questions about crawling publisher sites, which we covered in claimed versus verified, and Reddit is moving to a primarily logged-in model. Walls are one answer to scraping; a preference signal is a softer, interoperable one. The broader trend, where sites start closing doors to automated visitors, is the subject of our opinion piece on the web locking the door.
For a lab, honoring a shared signal is cheap, visible and defensible. For a publisher, the gain depends entirely on how many labs actually follow through, and on whether the largest crawlers treat silence as permission.
What site owners should do now
Nothing in the announcement requires a change today, but there are low-risk steps.
- Audit your current robots.txt. List which AI user-agent tokens you already block or allow. OpenAI says those rules stay valid.
- Decide your policy per section. Many publishers want public marketing pages open and paywalled or licensed archives marked no-training. Write that down before touching syntax.
- Do not add unverified directives and assume they work. A line that a crawler does not parse is ignored; RFC 9309 lets crawlers decide how to handle records outside the protocol. Google's own parser supports only User-Agent, Allow, Disallow and Sitemap, per the Search Engine Journal summary of its documentation.
- Watch for OpenAI's follow-up. The implementation update is where the exact line, the crawler behavior and any dates will appear.
- Keep enforcement separate. If you need content kept from a crawler, use rate limits, authentication or a CDN bot rule. A preference signal is a request.
- Mind encoding. For non-English paths, Google's Cherry Prommawin recommended percent-encoded rules for the sake of other or older crawlers.
If you also run AI agents on your own infrastructure and worry about what they fetch or leak, AgentBeam, the agent security platform from the explainx.ai team that stops AI agents before they take dangerous actions, is built for that review.
What this means if you build with AI
Builders on the crawler side have a mirror-image task. If you run a scraper, an agent that browses, or a dataset pipeline, preference signals are becoming a standard input. Parsing Content-Usage alongside Allow and Disallow will likely be expected for polite crawlers once the drafts settle, and the draft's two-stage logic (fetch permission first, usage preference second) is simple enough to implement now. Our reporting on agents that read government and agency sites at scale shows why this is not hypothetical: automated visitors are already a measurable share of traffic, and site owners are paying attention.
For teams shipping products that use third-party web content, keep a record of which signals your pipeline honored and when. If the standard lands, "we checked" becomes a compliance question, not a courtesy.
Open questions to watch
- Syntax and date. Will OpenAI publish the exact line and a go-live date, and will it match the IETF draft?
- Defaults. Will absence of a signal be treated as permission to train?
- Other labs. OpenAI "encourage[s] other AI providers to honor the signal." Watch for Google, Anthropic, Meta and Microsoft documentation updates.
- Standards status. The attachment draft has not been sent to the IESG, and the vocabulary draft says it lacks working-group consensus. A standard could still change shape.
- Verification. There is no public way yet to audit whether a crawler honored a signal. Crawl logs on your own server are the only evidence you control.
Bottom line
OpenAI's post is a modest but real step: a top lab says in writing that it will read a cross-provider robots.txt signal for training, keep its existing controls, and keep participating in the IETF process. What it is not is a switch you can flip today, a legal settlement of the training-data debate, or a guarantee that other labs follow. Treat it as a direction of travel, keep your current per-crawler rules, and wait for the implementation update before changing production files.
Related reading
- A Hacker News launch brought a flood of AI bot traffic: protect a small site
- Meta's new business terms and AI training on news sites: claimed vs verified
- Reddit goes primarily logged-in: what Huffman said about AI scraping
- USA Today sues OpenAI for $250 million over AI training data
- Wikimedia brief in NYT v. OpenAI: training is fair use, regurgitation is not
- AI lawsuit tracker: 248 cases against 76 companies
- Agent teams are the new org chart; the web is closing doors
Official sources: OpenAI, "Making it easier for websites to express AI training preferences" (October 30, 2026); IETF, draft-ietf-aipref-attach-05; Search Engine Journal, "Google Explains The Limit Of Robots.txt For AI Content Use" (October 22, 2026).
Details are accurate as of October 30, 2026. OpenAI has not yet published implementation specifics, and the IETF drafts are works in progress.