OpenAI, Google, and Anthropic Back a Universal AI Scraping Opt-Out

OpenAI, Google, Microsoft, and Anthropic say they will adopt a unified standard giving publishers a single switch to block AI training crawlers across all four platforms at once, according to an October 30, 2026 report from The Verge. For anyone who has been watching the AI visibility and content control space closely, this is a significant shift. Right now, opting out of AI scraping means hunting down each company's individual crawler rules and filing preferences one by one. This standard changes that equation.

The proposal comes through the Internet Engineering Task Force (IETF) and was put forward by Kevin Kelley, a technical lead for standards at Anthropic. It centers on a shared "policy token" signal that lives inside a site's robots.txt file, the same file that has directed search crawlers for decades. One signal, multiple crawlers. That's the core promise.

How the AI Scraping Opt-Out Standard Actually Works

The mechanics are straightforward, and that's intentional. According to The Verge, publishers will be able to place a shared policy token inside their robots.txt file. When any of the participating AI crawlers encounter that token, they are expected to honor the publisher's stated training preferences and stop scraping the site for AI model training.

Previously, every AI company ran its own crawler under its own rules, which meant a publisher had to signal their preferences to OpenAI, Google, Microsoft, and Anthropic separately. That friction has been a real problem. Most site owners don't have the time or technical depth to audit their robots.txt for four different AI crawlers, let alone keep up as new ones emerge.

Microsoft is going a step further by offering granular controls. Publishers will be able to opt out of AI training specifically, or opt out of AI grounding, which is the process of pointing an AI model to a verified source to reduce hallucinations. Microsoft also confirmed that opting out will not affect a site's visibility in Bing search results. Google has previously implemented similar controls allowing sites to separate their AI training preferences from their standard search visibility.

As for the rollout: Anthropic has already adopted the signal. Google and Microsoft plan to support it in 2027. OpenAI says it will implement it in the "coming months." Meta's communications head Andy Stone signaled support for the idea in a public statement but stopped short of confirming the company will honor it.

What This AI Scraping Standard Means for SEO and Content Strategy

Here's the tension every content-driven site is living with right now: you want to rank in search, you want to get cited in AI Overviews and generative answers, but you also don't necessarily want your content baked into a model that someone else monetizes. Until now, those goals were hard to separate. This standard starts to untangle them.

The distinction Microsoft is making between AI training and AI grounding is the piece I'd watch most closely. Grounding is how AI systems pull verified, real-time content into their answers, which is actually a visibility opportunity for publishers. Training is a different thing: it's your content absorbed into a model, used indefinitely, with no ongoing credit or traffic signal. Being able to block training while staying eligible for grounding is a meaningful option for publishers who want AI visibility without giving their content away permanently.

For SEO agencies managing content for multiple clients, this creates a new audit item. You now need a clear stance on robots.txt AI directives for every site in your portfolio, not just a blanket decision. A news publisher and a SaaS company have very different risk profiles here. One earns from content directly; the other earns from conversion. Their robots.txt strategy should reflect that.

And yes, this happens more than most agencies want to admit: robots.txt files go untouched for years. This announcement is a good forcing function to open that file back up.

The Broader Fight Against Unauthorized AI Scraping

The IETF proposal doesn't exist in a vacuum. The Verge noted that other attempts to address unauthorized scraping are already in play. The Really Simple Licensing (RSL) standard gives publishers the ability to charge crawlers that visit their sites. Cloudflare has moved to block AI crawlers by default. Publishers and media organizations have pursued legal routes as well.

The shared policy token is the most cooperative approach yet: the AI companies themselves are agreeing to honor it. That's a different posture than what we've seen before, where publishers largely had to go on defense. Whether the honor system holds as smaller or less reputable AI companies emerge is a fair question, and one the IETF process will need to address.

I've watched the robots.txt spec do a lot of heavy lifting over the years, from blocking archival crawlers to managing crawl budget on large sites. It's a resilient mechanism because it's simple and widely understood. Attaching AI training preferences to it is pragmatic. The hard part has never been the technical signal; it's been getting the big players to agree to respect it. Getting four of the largest AI companies on record is the actual news here.

What Publishers and SEOs Should Do About the AI Scraping Policy Token Now

This is early-stage, but there are concrete steps worth taking before the standard goes live across all four platforms.

1. Audit your current robots.txt file. Pull it up and check what AI crawler directives, if any, you already have in place. Most sites have nothing beyond the standard Googlebot rules. Note what's there and what's missing.
2. Decide your stance before the token goes live. Do you want to block AI training but stay eligible for grounding and AI-powered search features? That distinction matters, especially for sites that earn from organic traffic or want citation visibility in tools like ChatGPT or Perplexity. There's no universal right answer.
3. Separate training from search indexing in your strategy. Microsoft says sites will still appear in Bing's search results regardless of their preferences, and Google's existing controls already let sites opt out of AI training without impacting search visibility. That removes the fear that blocking AI scraping will tank your rankings. You can block training and still rank.
4. Watch the IETF process for the finalized token syntax. Anthropic has already implemented the signal, which means the technical spec exists in some form. Follow IETF updates so you implement the correct directive when the others roll out.
5. Build this into your client reporting and technical SEO audits. If you manage sites for clients, robots.txt AI directives are now a standard line item. Add a review step to your onboarding and quarterly audit process.

If you want visibility into whether your content is surfacing in generative search answers, tools like AI visibility tracking can help you monitor that alongside your standard rank data. Knowing where you appear, and where you don't, is the starting point for any robots.txt decision.

Where the AI Scraping Opt-Out Movement Goes From Here

The four biggest AI companies agreeing to honor a shared opt-out signal is genuinely meaningful, even if the details are still rolling out. But the harder problem is the long tail: every AI startup, scraper, and model trainer that isn't OpenAI, Google, Microsoft, or Anthropic. A voluntary IETF standard only works if everyone participates, and participation from smaller players isn't guaranteed.

Still, setting a clear expectation at the top of the market matters. It creates a baseline that regulators, courts, and smaller companies can reference. If the big four are on record saying they'll honor this signal, it becomes harder for a smaller player to argue that ignoring it is acceptable practice.

For publishers and content teams, the immediate move is simple: understand what the policy token does, decide what's right for your site, and get your robots.txt ready. The window between now and full rollout is your chance to make a deliberate choice instead of defaulting to whatever your file said three years ago.