How to allow or block AI crawlers in robots.txt
AI companies run several crawlers, and each does a different job. Some collect pages to train models, some build the index behind AI search answers, and some fetch a page when a user asks about it. robots.txt lets you allow or block each one by name. Whether to allow them is a business decision, and there is no single right answer. This guide covers how to make your robots.txt say what you actually intend, and how to check that nothing else overrides it.
Know which crawler does what
| Name in robots.txt | Used for |
|---|---|
GPTBot | OpenAI model training |
OAI-SearchBot | ChatGPT search results and citations |
ChatGPT-User | Pages fetched when a ChatGPT user asks |
ClaudeBot | Anthropic model training |
Claude-SearchBot | Claude's web search index |
Claude-User | Pages fetched when a Claude user asks |
Google-Extended | Training Google's AI models, including Gemini and the models behind AI Overviews and AI Mode, and grounding answers in the Gemini app. It does not affect whether you appear in Google Search, AI Overviews or AI Mode. |
Applebot-Extended | Apple model training. Applebot itself still indexes for Siri and Spotlight. |
PerplexityBot | Perplexity's search index |
Perplexity-User | Pages fetched when a Perplexity user asks |
Meta-ExternalAgent | Meta AI training and products |
Amazonbot | Amazon products and services; may be used to train Amazon AI models |
Amzn-SearchBot | Search in Amazon products such as Alexa; not used for training |
Amzn-User | Pages fetched when an Alexa user asks |
Bytespider | ByteDance, TikTok's parent company. ByteDance does not publish clear documentation of what it is used for. |
CCBot | Common Crawl, a public dataset widely used for training |
Common setups
Allow everything
You do not need any AI-specific rules. A crawler with no group of its own follows your User-agent: * group.
Appear in AI search, stay out of training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /Several User-agent lines in a row share the rules that follow. The search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are not named, so they follow your User-agent: * group.
Block all AI crawlers
Add every name in the table above to the same group. That asks each of them to stay out, but fetchers that act on a user's request may still visit (Perplexity says Perplexity-User generally ignores robots.txt), so block at your firewall or CDN if you need to be sure.
Blocking Google-Extended stops Google using your content to train its AI models and to ground answers in the Gemini app, but it does not remove you from AI Overviews or AI Mode. To leave those, use the Search generative AI control in Google Search Console (under Settings); it does not change your regular search results.
The rule people miss
A crawler that has a group naming it follows that group and ignores User-agent: * entirely (if several groups name it, their rules are combined). So if you give GPTBot its own group, it no longer sees paths you meant to keep every crawler out of. Repeat those Disallow lines in each named group.
Check your CDN is not overriding you
If your site is on Cloudflare, three settings can override your robots.txt:
- AI Crawl Control blocks named crawlers at the edge, so they are refused before robots.txt matters.
- AI bot policies, in the zone's security settings, block crawlers by what they do (Search, Agent or Training), on every page or only on pages that show ads.
- Cloudflare's managed robots.txt adds its own
Disallowrules for AI crawlers to the top of the file it serves.
Cloudflare's defaults have changed more than once. Since July 2025, new domains are asked at signup whether to allow AI crawlers, and from 15 September 2026 new domains block Training and Agent bots on pages with ads while allowing Search. Check all three in the dashboard and make sure they match what your robots.txt says. Other bot-management services can block these crawlers too; your firewall or bot logs show what was refused.
The limits of robots.txt
robots.txt is a request, not a lock. The major AI companies document that their crawlers follow it, but agents that fetch a page on a user's behalf may not, and anyone can fake a user agent. To enforce a block, refuse the requests at your firewall or CDN, matching the IP ranges the operator publishes where there are any (OpenAI, Anthropic, Google, Apple, Perplexity, Amazon and Common Crawl publish them) rather than the user agent alone. Google-Extended and Applebot-Extended cannot be blocked this way: they are robots.txt names only, and the pages are fetched by Googlebot and Applebot.
Check it worked
Run WEBCHKR again. The AI Crawler Access check reads the robots.txt your site actually serves, including any lines Cloudflare's managed robots.txt adds, and gives each of 17 AI crawlers a summary status (allowed, partly restricted or blocked) and whether it has its own rules or follows the * group. The status reflects whether the whole site is disallowed, so read complex Allow and Disallow patterns yourself. It reports rather than grades, because there is no right answer. It cannot see blocks applied at the edge, such as Cloudflare's AI Crawl Control, so check those in the dashboard.