Blocking AI crawlers with robots.txt

Most AI companies now separate training, search and user-triggered fetching into different tokens, so you can opt out of one without the others.

Block training, keep AI search

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /

Several User-agent lines in a row share one group (RFC 9309). OAI-SearchBot, Claude-SearchBot and PerplexityBot stay allowed, so you can still appear in ChatGPT, Claude and Perplexity search.

What each token means

Token	Operator	Purpose
`GPTBot`	OpenAI	training
`OAI-SearchBot`	OpenAI	search (ChatGPT search)
`ChatGPT-User`	OpenAI	user-triggered fetch (robots.txt may not apply)
`ClaudeBot`	Anthropic	training
`Claude-SearchBot`	Anthropic	search
`Claude-User`	Anthropic	user-triggered fetch
`Google-Extended`	Google	control token: Gemini training/grounding (not a crawler)
`Googlebot`	Google	search indexing
`Applebot-Extended`	Apple	control token: Apple AI training (not a crawler)
`Applebot`	Apple	search (Siri/Spotlight)
`PerplexityBot`	Perplexity	search
`Perplexity-User`	Perplexity	user-triggered fetch (generally ignores robots.txt)
`Meta-ExternalAgent`	Meta	training
`CCBot`	Common Crawl	open web corpus (widely used for training)
`Bytespider`	ByteDance	crawler (AI)
`Amazonbot`	Amazon	crawler (Alexa/AI)
`Bingbot`	Microsoft	search indexing (also feeds Copilot)
`DuckAssistBot`	DuckDuckGo	AI answers
`MistralAI-User`	Mistral	user-triggered fetch
`cohere-ai`	Cohere	AI

Limits

robots.txt is a request, not access control. Enforcement needs a CDN/WAF rule.
User-triggered agents (ChatGPT-User, Perplexity-User) say robots.txt may not apply to them.
Google-Extended and Applebot-Extended are control tokens, not crawlers; Googlebot and Applebot do the fetching.
Rules apply per host: repeat them on every subdomain.

Check which AI crawlers a site blocks.

Sources: OpenAI crawlers, Anthropic crawlers, Google crawlers, Applebot, Perplexity crawlers, RFC 9309.