Technical AEO
robots.txt for AI crawlers: which bots to allow and block
Last updated
7 min read
robots.txt is a plain text file at the root of a site that tells crawlers which URLs they may fetch. A crawler is the program a company uses to fetch web pages, and in 2026 the AI crawlers named in that file do three different jobs.
Short answer
For AI visibility, robots.txt should allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot) and make a deliberate choice about training crawlers and tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot). Blocking GPTBot or ClaudeBot doesn't remove you from ChatGPT or Claude search, but blocking OAI-SearchBot or Claude-SearchBot does. User-initiated fetchers may ignore robots.txt.
Key facts
- OpenAI says OAI-SearchBot and GPTBot are independent settings (OpenAI).
- Google-Extended and Applebot-Extended are control tokens, not crawlers. Apple says "Applebot-Extended does not crawl webpages" (Apple).
- OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules" (OpenAI, Perplexity).
- Under RFC 9309, a crawler that matches a named group ignores the
*group (RFC Editor).
What are the three kinds of AI crawlers?
AI crawlers fall into three groups: training crawlers, search crawlers and user-initiated fetchers. Each group does a different job, so blocking each one costs you something different.
- Training crawlers collect pages that may be used to train AI models. GPTBot (OpenAI) and ClaudeBot (Anthropic) are training crawlers. CCBot builds Common Crawl's open archive, which Vercel describes as "often used as a training dataset for Large Language Models".
- Search crawlers index pages so an AI search product can cite them. OAI-SearchBot, Claude-SearchBot and PerplexityBot do this for ChatGPT, Claude and Perplexity. Googlebot, Bingbot and Applebot feed classic search and AI answers at the same time.
- User-initiated fetchers load a page when a person asks an AI tool to: ChatGPT-User, Claude-User and Perplexity-User. Google calls its equivalents user-triggered fetchers.
Google-Extended and Applebot-Extended are product tokens, not crawlers: names that control how content Google or Apple already crawled may be used.
Which AI crawlers matter in 2026?
As of November 2026, these are the tokens to know, per each vendor's documentation.
| Token | Company | What it's for | Blocking it means | Docs |
|---|---|---|---|---|
Googlebot | Google Search, including AI Overviews and AI Mode | Google can't crawl your pages for Search or its AI features | ||
Google-Extended | Token for Gemini training and grounding; not a crawler | Content isn't used to train Gemini or ground Gemini Apps answers; Search is unaffected | ||
Bingbot | Microsoft | Bing's index, which powers Copilot | Bing can't crawl your pages for Bing search or Copilot | Bing |
OAI-SearchBot | OpenAI | ChatGPT search | Pages aren't shown in ChatGPT search answers, except as navigational links | OpenAI |
GPTBot | OpenAI | Content that may be used to train OpenAI's models | Signals your content shouldn't be used for training; search is unaffected | OpenAI |
ChatGPT-User | OpenAI | Visits pages when a ChatGPT user asks | May do nothing: OpenAI says robots.txt rules may not apply | OpenAI |
ClaudeBot | Anthropic | Content that could contribute to model training | Signals that future pages should be excluded from Anthropic's training data | Anthropic |
Claude-SearchBot | Anthropic | Indexing for Claude's search results | Pages aren't indexed for Claude's search, which may reduce visibility | Anthropic |
Claude-User | Anthropic | Visits pages when a Claude user asks | Claude can't retrieve your pages for user questions | Anthropic |
PerplexityBot | Perplexity | Perplexity search results; not used for model training | Pages can't be surfaced and linked in Perplexity search | Perplexity |
Perplexity-User | Perplexity | Visits pages when a Perplexity user asks | Usually nothing: Perplexity says it generally ignores robots.txt | Perplexity |
Applebot | Apple | Spotlight, Siri and Safari search; data may also train Apple's models and inform its AI answers | Apple can't crawl your pages for its search features or AI answers | Apple |
Applebot-Extended | Apple | Token for training Apple's foundation models; doesn't crawl | Content isn't used to train Apple's models; search is unaffected | Apple |
CCBot | Common Crawl | Open web archive that anyone can download | Common Crawl stops crawling your site, so new pages stay out of future crawls | Common Crawl |
Which AI bots may ignore robots.txt?
User-initiated fetchers may ignore robots.txt, because a person asked for the page. OpenAI says that for ChatGPT-User "robots.txt rules may not apply", Perplexity says Perplexity-User "generally ignores robots.txt rules", and Google says its user-triggered fetchers "generally ignore robots.txt rules".
Anthropic is the exception among these vendors. It documents Claude-User as controllable through robots.txt and says its bots respect "industry standard directives in robots.txt".
Agent browsers are a separate case. Anthropic's Claude-Agent and Google's Google-Agent name themselves in the user-agent string and come from published IP addresses, so restrict them with firewall rules.
robots.txt is a request, not a lock
RFC 9309 says robots.txt rules "are not a form of access authorization". Well-behaved crawlers follow them. To enforce a block, use firewall rules that match each vendor's published IP ranges.
Are Google-Extended and Applebot-Extended separate crawlers?
No. Google-Extended and Applebot-Extended are tokens that control how content is used, and neither one sends its own requests.
Google says Google-Extended "doesn't have a separate HTTP request user agent string" and calls it "a standalone product token". It controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Grounding means giving the model search results at answer time. Google says it "does not impact a site's inclusion in Google Search".
Apple says "Applebot-Extended does not crawl webpages" and is used "to determine how to use the data crawled by the Applebot user agent". Disallowing it opts content out of training Apple's foundation models. For Apple's AI answers, Apple points to the nosnippet meta tag instead.
You won't see either token in your server logs, so blocking them doesn't reduce crawl traffic.
Why does a named group override the catch-all group?
A crawler that finds a group with its own name follows only that group and ignores the catch-all User-agent: * group. RFC 9309 says crawlers must obey the * group only "if no matching group exists", and Google's spec says user agent specific groups and global groups (*) "are not combined".
User-agent: *
Disallow: /checkout/
Disallow: /account/
User-agent: OAI-SearchBot
Allow: /
In this file, OAI-SearchBot may crawl /checkout/, because it reads only its own group. Repeat any Disallow lines you still need inside each named group.
Apple adds a twist: "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions." A Googlebot group with Disallow: / also blocks Applebot unless Applebot has its own group.
What should your robots.txt say?
Your robots.txt should match a decision about search and training. Here are three versions, with their trade-offs.
Allow every crawler
This file lets every crawler in, including all AI crawlers. This site uses the same approach, because being cited is the goal.
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
The trade-off: any company that crawls your public pages may use them for training.
Allow search, block training
This file keeps AI search crawlers working and asks training crawlers to stay out.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
The trade-off: Google-Extended also controls grounding in Gemini Apps, so blocking it can keep your pages out of grounded Gemini answers. Remove that line if Gemini visibility matters to you. For Microsoft, the training control Bing announced in 2023 is a meta tag: content tagged NOARCHIVE won't be used to train its models, but NOARCHIVE also keeps that content out of Bing Chat answers.
A shared no-training line is coming
An IETF draft defines a policy token, __usage_ai-training, that states one training preference to every provider that supports it. As of November 2026, Anthropic says it has begun recognizing the signal, Google plans to support it in 2027, and OpenAI says it "will honor a shared robots.txt signal". Named groups still take priority, so add it alongside them.
Block all AI crawlers
This file asks every AI crawler and fetcher in the table to stay out, while leaving classic search crawlers alone.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
The trade-offs: your pages stop appearing in ChatGPT search answers (except as navigational links), in Perplexity results and in Claude's search, and user-initiated fetchers may still visit. It doesn't remove you from Google AI Overviews or Copilot, which rely on Googlebot and Bingbot. For Google, use the Search generative AI control or nosnippet instead. For a brand that wants to be recommended, it's the costliest option.
I write and test these files for small stores and course businesses. I'm a quant developer, and I deploy the change myself instead of handing you a list.
Find out how AI engines see your brand
Start with an AEO audit: a fixed-price report on how ChatGPT, Google’s AI features, Gemini and Perplexity see your brand today, with a prioritized fix list. I can implement the fixes too.
How do you test robots.txt for AI crawlers?
Test robots.txt by checking the live file first, then checking how each crawler is actually treated.
- Fetch the live file. Load
https://www.example.com/robots.txtand confirm it returns 200. Under RFC 9309 a 5xx error means complete disallow. On Cloudflare, Bot Preference Sync can prepend a managed block, so read the live file, not your repo copy. - Check Google's view. Search Console's robots.txt report shows the files Google found for your top 20 hosts, when it last crawled them, and any warnings or errors.
- Check Bing's view. Bing's robots.txt tester checks a URL the way Bingbot and BingAdsBot would.
- Request a page as the bot. Send the crawler's documented user-agent string with curl, as below. A 403 or a challenge page points to a firewall rule, not robots.txt.
- Read your server logs. Search for each token, check the status codes it received, and confirm the IP addresses against the vendor's published list, such as OpenAI's searchbot.json or Anthropic's bots.json.
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://www.example.com/
This only tests user-agent rules. Google warns the string "can be spoofed", so firewalls that also check IP ranges may treat your test differently from the real crawler.
Sources
- Overview of OpenAI Crawlers, OpenAI, accessed November 2026
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic, accessed November 2026
- How do Anthropic-hosted browsers identify themselves when they browse the web on a user's behalf?, Anthropic, accessed November 2026
- Perplexity Crawlers, Perplexity, accessed November 2026
- Google's common crawlers, Google, updated October 2026
- Google user-triggered fetchers, Google, accessed November 2026
- How Google interprets the robots.txt specification, Google, updated October 2026
- Supporting a shared AI training control for websites, Google Search Central Blog, published October 2026
- AI features and your website, Google Search Central, updated December 2025
- Search generative AI control, Google Search Console Help, accessed November 2026
- robots.txt report, Google Search Console Help, accessed November 2026
- About Applebot, Apple, published September 2026
- CCBot, Common Crawl, accessed November 2026
- Announcing user-agent change for Bing crawler bingbot, Microsoft Bing, published April 2022
- Announcing new options for webmasters to control usage of their content in Bing Chat, Microsoft Bing, published September 2023
- Bing Webmaster Tools makes it easy to edit and verify your robots.txt, Microsoft Bing, published September 2020
- Optimizing Your Content for Inclusion in AI Search Answers, Microsoft Advertising, published October 2025
- RFC 9309: Robots Exclusion Protocol, RFC Editor, published September 2022
- Machine-Readable Training Preference Signal, IETF Datatracker, published November 2026
- Making it easier for websites to express AI training preferences, OpenAI, published October 2026
- Say it once: introducing Bot Preference Sync, Cloudflare, published August 2026
- The rise of the AI crawler, Vercel, published December 2024
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI says GPTBot and OAI-SearchBot are independent settings. GPTBot covers content that may be used for training, while OAI-SearchBot controls whether your pages can appear in ChatGPT search answers.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google says Google-Extended doesn't affect inclusion in Google Search, and AI Overviews are managed through Googlebot, snippet controls and Search Console. Blocking Google-Extended does stop Google from using your content to ground answers in Gemini Apps.
Do AI crawlers obey robots.txt?
The automated crawlers documented by OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl all offer robots.txt controls. User-initiated fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
How long does a robots.txt change take to reach AI crawlers?
OpenAI says about 24 hours for ChatGPT search, Perplexity says up to 24 hours, and Google generally caches robots.txt for up to 24 hours. RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable.
Is robots.txt enough to keep AI crawlers out?
No. RFC 9309 says robots.txt rules are not a form of access authorization, so a crawler that ignores them can still fetch your pages. To enforce a block, use firewall rules that match each vendor's published IP ranges.
Find out how AI engines see your brand
Start with an AEO audit: a fixed-price report on how ChatGPT, Google’s AI features, Gemini and Perplexity see your brand today, with a prioritized fix list. I can implement the fixes too.
Keep reading
- Why your website is invisible to AI search, and 7 fixes
Technical AEO ·
- How to get your brand mentioned in ChatGPT answers
AI visibility ·