Technical AEO

robots.txt for AI crawlers: which bots to allow and block

Last updated

7 min read

robots.txt is a plain text file at the root of a site that tells crawlers which URLs they may fetch. A crawler is the program a company uses to fetch web pages, and in 2026 the AI crawlers named in that file do three different jobs.

Short answer

For AI visibility, robots.txt should allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot) and make a deliberate choice about training crawlers and tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot). Blocking GPTBot or ClaudeBot doesn't remove you from ChatGPT or Claude search, but blocking OAI-SearchBot or Claude-SearchBot does. User-initiated fetchers may ignore robots.txt.

Key facts

  • OpenAI says OAI-SearchBot and GPTBot are independent settings (OpenAI).
  • Google-Extended and Applebot-Extended are control tokens, not crawlers. Apple says "Applebot-Extended does not crawl webpages" (Apple).
  • OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules" (OpenAI, Perplexity).
  • Under RFC 9309, a crawler that matches a named group ignores the * group (RFC Editor).

What are the three kinds of AI crawlers?

AI crawlers fall into three groups: training crawlers, search crawlers and user-initiated fetchers. Each group does a different job, so blocking each one costs you something different.

  • Training crawlers collect pages that may be used to train AI models. GPTBot (OpenAI) and ClaudeBot (Anthropic) are training crawlers. CCBot builds Common Crawl's open archive, which Vercel describes as "often used as a training dataset for Large Language Models".
  • Search crawlers index pages so an AI search product can cite them. OAI-SearchBot, Claude-SearchBot and PerplexityBot do this for ChatGPT, Claude and Perplexity. Googlebot, Bingbot and Applebot feed classic search and AI answers at the same time.
  • User-initiated fetchers load a page when a person asks an AI tool to: ChatGPT-User, Claude-User and Perplexity-User. Google calls its equivalents user-triggered fetchers.

Google-Extended and Applebot-Extended are product tokens, not crawlers: names that control how content Google or Apple already crawled may be used.

Which AI crawlers matter in 2026?

As of November 2026, these are the tokens to know, per each vendor's documentation.

AI-related crawlers and tokens in robots.txt, November 2026
TokenCompanyWhat it's forBlocking it meansDocs
GooglebotGoogleGoogle Search, including AI Overviews and AI ModeGoogle can't crawl your pages for Search or its AI featuresGoogle
Google-ExtendedGoogleToken for Gemini training and grounding; not a crawlerContent isn't used to train Gemini or ground Gemini Apps answers; Search is unaffectedGoogle
BingbotMicrosoftBing's index, which powers CopilotBing can't crawl your pages for Bing search or CopilotBing
OAI-SearchBotOpenAIChatGPT searchPages aren't shown in ChatGPT search answers, except as navigational linksOpenAI
GPTBotOpenAIContent that may be used to train OpenAI's modelsSignals your content shouldn't be used for training; search is unaffectedOpenAI
ChatGPT-UserOpenAIVisits pages when a ChatGPT user asksMay do nothing: OpenAI says robots.txt rules may not applyOpenAI
ClaudeBotAnthropicContent that could contribute to model trainingSignals that future pages should be excluded from Anthropic's training dataAnthropic
Claude-SearchBotAnthropicIndexing for Claude's search resultsPages aren't indexed for Claude's search, which may reduce visibilityAnthropic
Claude-UserAnthropicVisits pages when a Claude user asksClaude can't retrieve your pages for user questionsAnthropic
PerplexityBotPerplexityPerplexity search results; not used for model trainingPages can't be surfaced and linked in Perplexity searchPerplexity
Perplexity-UserPerplexityVisits pages when a Perplexity user asksUsually nothing: Perplexity says it generally ignores robots.txtPerplexity
ApplebotAppleSpotlight, Siri and Safari search; data may also train Apple's models and inform its AI answersApple can't crawl your pages for its search features or AI answersApple
Applebot-ExtendedAppleToken for training Apple's foundation models; doesn't crawlContent isn't used to train Apple's models; search is unaffectedApple
CCBotCommon CrawlOpen web archive that anyone can downloadCommon Crawl stops crawling your site, so new pages stay out of future crawlsCommon Crawl

Which AI bots may ignore robots.txt?

User-initiated fetchers may ignore robots.txt, because a person asked for the page. OpenAI says that for ChatGPT-User "robots.txt rules may not apply", Perplexity says Perplexity-User "generally ignores robots.txt rules", and Google says its user-triggered fetchers "generally ignore robots.txt rules".

Anthropic is the exception among these vendors. It documents Claude-User as controllable through robots.txt and says its bots respect "industry standard directives in robots.txt".

Agent browsers are a separate case. Anthropic's Claude-Agent and Google's Google-Agent name themselves in the user-agent string and come from published IP addresses, so restrict them with firewall rules.

robots.txt is a request, not a lock

RFC 9309 says robots.txt rules "are not a form of access authorization". Well-behaved crawlers follow them. To enforce a block, use firewall rules that match each vendor's published IP ranges.

Are Google-Extended and Applebot-Extended separate crawlers?

No. Google-Extended and Applebot-Extended are tokens that control how content is used, and neither one sends its own requests.

Google says Google-Extended "doesn't have a separate HTTP request user agent string" and calls it "a standalone product token". It controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Grounding means giving the model search results at answer time. Google says it "does not impact a site's inclusion in Google Search".

Apple says "Applebot-Extended does not crawl webpages" and is used "to determine how to use the data crawled by the Applebot user agent". Disallowing it opts content out of training Apple's foundation models. For Apple's AI answers, Apple points to the nosnippet meta tag instead.

You won't see either token in your server logs, so blocking them doesn't reduce crawl traffic.

Why does a named group override the catch-all group?

A crawler that finds a group with its own name follows only that group and ignores the catch-all User-agent: * group. RFC 9309 says crawlers must obey the * group only "if no matching group exists", and Google's spec says user agent specific groups and global groups (*) "are not combined".

User-agent: *
Disallow: /checkout/
Disallow: /account/

User-agent: OAI-SearchBot
Allow: /

In this file, OAI-SearchBot may crawl /checkout/, because it reads only its own group. Repeat any Disallow lines you still need inside each named group.

Apple adds a twist: "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions." A Googlebot group with Disallow: / also blocks Applebot unless Applebot has its own group.

What should your robots.txt say?

Your robots.txt should match a decision about search and training. Here are three versions, with their trade-offs.

Allow every crawler

This file lets every crawler in, including all AI crawlers. This site uses the same approach, because being cited is the goal.

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The trade-off: any company that crawls your public pages may use them for training.

Allow search, block training

This file keeps AI search crawlers working and asks training crawlers to stay out.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The trade-off: Google-Extended also controls grounding in Gemini Apps, so blocking it can keep your pages out of grounded Gemini answers. Remove that line if Gemini visibility matters to you. For Microsoft, the training control Bing announced in 2023 is a meta tag: content tagged NOARCHIVE won't be used to train its models, but NOARCHIVE also keeps that content out of Bing Chat answers.

A shared no-training line is coming

An IETF draft defines a policy token, __usage_ai-training, that states one training preference to every provider that supports it. As of November 2026, Anthropic says it has begun recognizing the signal, Google plans to support it in 2027, and OpenAI says it "will honor a shared robots.txt signal". Named groups still take priority, so add it alongside them.

Block all AI crawlers

This file asks every AI crawler and fetcher in the table to stay out, while leaving classic search crawlers alone.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

The trade-offs: your pages stop appearing in ChatGPT search answers (except as navigational links), in Perplexity results and in Claude's search, and user-initiated fetchers may still visit. It doesn't remove you from Google AI Overviews or Copilot, which rely on Googlebot and Bingbot. For Google, use the Search generative AI control or nosnippet instead. For a brand that wants to be recommended, it's the costliest option.

I write and test these files for small stores and course businesses. I'm a quant developer, and I deploy the change myself instead of handing you a list.

Find out how AI engines see your brand

Start with an AEO audit: a fixed-price report on how ChatGPT, Google’s AI features, Gemini and Perplexity see your brand today, with a prioritized fix list. I can implement the fixes too.

How do you test robots.txt for AI crawlers?

Test robots.txt by checking the live file first, then checking how each crawler is actually treated.

  1. Fetch the live file. Load https://www.example.com/robots.txt and confirm it returns 200. Under RFC 9309 a 5xx error means complete disallow. On Cloudflare, Bot Preference Sync can prepend a managed block, so read the live file, not your repo copy.
  2. Check Google's view. Search Console's robots.txt report shows the files Google found for your top 20 hosts, when it last crawled them, and any warnings or errors.
  3. Check Bing's view. Bing's robots.txt tester checks a URL the way Bingbot and BingAdsBot would.
  4. Request a page as the bot. Send the crawler's documented user-agent string with curl, as below. A 403 or a challenge page points to a firewall rule, not robots.txt.
  5. Read your server logs. Search for each token, check the status codes it received, and confirm the IP addresses against the vendor's published list, such as OpenAI's searchbot.json or Anthropic's bots.json.
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://www.example.com/

This only tests user-agent rules. Google warns the string "can be spoofed", so firewalls that also check IP ranges may treat your test differently from the real crawler.

Sources

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI says GPTBot and OAI-SearchBot are independent settings. GPTBot covers content that may be used for training, while OAI-SearchBot controls whether your pages can appear in ChatGPT search answers.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google says Google-Extended doesn't affect inclusion in Google Search, and AI Overviews are managed through Googlebot, snippet controls and Search Console. Blocking Google-Extended does stop Google from using your content to ground answers in Gemini Apps.

Do AI crawlers obey robots.txt?

The automated crawlers documented by OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl all offer robots.txt controls. User-initiated fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.

How long does a robots.txt change take to reach AI crawlers?

OpenAI says about 24 hours for ChatGPT search, Perplexity says up to 24 hours, and Google generally caches robots.txt for up to 24 hours. RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable.

Is robots.txt enough to keep AI crawlers out?

No. RFC 9309 says robots.txt rules are not a form of access authorization, so a crawler that ignores them can still fetch your pages. To enforce a block, use firewall rules that match each vendor's published IP ranges.

Written by

TutorTechWiz AEO

An independent AEO and SEO practice run by a quant developer who audits how AI engines see a brand and then builds the fixes. About the consultant.

Find out how AI engines see your brand

Start with an AEO audit: a fixed-price report on how ChatGPT, Google’s AI features, Gemini and Perplexity see your brand today, with a prioritized fix list. I can implement the fixes too.