How to Block (or Allow) AI Crawlers: GPTBot, ClaudeBot & More

To control AI crawlers, add rules to your robots.txt by user-agent — but the catch almost nobody mentions is that each AI company runs several bots, and blocking the wrong one can quietly remove you from AI search while doing nothing to stop training. After eleven years in SEO, this is the mistake I see most in 2026: a site blanket-blocks “GPTBot” to keep its content out of AI, then wonders why it stopped being cited in ChatGPT. Here’s how to do it deliberately.

The one thing to understand first: training vs retrieval bots

Each major AI provider runs distinct crawlers for distinct jobs, and they’re controlled separately in robots.txt:

• Training crawlers gather content to train models. Blocking these keeps your content out of the training data.
• Retrieval / search crawlers fetch pages live to answer a user’s question and cite sources. Blocking these removes you from AI search results and citations.

Per OpenAI’s and Anthropic’s own docs, blocking the training bot does not block the search bot — and vice versa. So “should I block AI crawlers?” isn’t one question. It’s two: do you want to be in the training set, and do you want to be cited in AI answers? Most site owners want out of training but in the answers — and that requires blocking precisely, not blanket-blocking.

The crawlers that actually matter (2026)

The user-agents you’ll see in your logs, by vendor:

• OpenAI — GPTBot (training), OAI-SearchBot (ChatGPT search index), ChatGPT-User (fetches on a user’s request).
• Anthropic — ClaudeBot (training), Claude-SearchBot (search), Claude-User (user-requested fetch).
• Perplexity — PerplexityBot (search index), Perplexity-User (on-demand fetch).
• Google — Google-Extended (controls use in Gemini/Vertex training; blocking it does not affect Google Search ranking).
• Common Crawl — CCBot. This one’s high-leverage: Common Crawl is a public dataset many LLMs train on, so blocking CCBot limits training inclusion across many models at once.

How to block AI crawlers in robots.txt

Put this at your site root (/robots.txt). To block a bot from everything:

User-agent: GPTBot
Disallow: /

To block the training bots while keeping the search/retrieval bots (the “out of training, still cited” setup most people actually want):

# Block training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

# Allow retrieval / search crawlers (leave them unlisted, or explicitly allow)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Changes can take about a day to register (OpenAI notes ~24 hours). And remember robots.txt is public — anyone can read which bots you block.

The big caveat: robots.txt is a request, not a wall

Compliant crawlers (OpenAI, Anthropic, Google, Perplexity’s main bot) honor robots.txt. But it’s an honor system: crawlers like Bytespider, and reportedly some stealth fetchers, have been documented ignoring it. If you need to actually stop a non-compliant bot, robots.txt won’t do it — you need a block at the server or WAF level (for example, a Cloudflare firewall rule by user-agent or verified IP range). OpenAI and others publish official IP ranges so you can verify a real bot versus an impersonator.

Should you block them at all?

My honest take: for most people who make things online, you want to be cited by AI, not hidden from it — AI referral traffic is small but converts well, and being the source an answer engine quotes is the whole game of GEO. So the common right answer is: allow the search/retrieval bots, and only block training crawlers if being in the training set genuinely bothers you. If your goal is the opposite — you’re protecting proprietary or paywalled content — then block broadly and enforce it at the edge, not just in robots.txt.

Not sure which bots can currently reach your site? My free AI Crawler Checker tests the major AI user-agents against your URL so you can see what’s actually getting through — and if you’re worried you’re accidentally invisible to AI, that’s exactly the problem I dig into in why isn’t my site showing up in ChatGPT. For the wider technical picture, see technical SEO in the AI era.

If this was useful, I send occasional, honest notes on SEO and AI search — the free GEO Starter Checklist comes with the first one. Subscribe below.

Frequently asked questions

Does blocking GPTBot remove me from ChatGPT search?

No. GPTBot is OpenAI's training crawler. ChatGPT's live search uses OAI-SearchBot, and user-requested fetches use ChatGPT-User — both separate. Blocking GPTBot keeps you out of training data but does not, by itself, remove you from ChatGPT's search results or citations.

How do I block AI crawlers but stay in AI search results?

Block the training user-agents (GPTBot, ClaudeBot, Google-Extended, CCBot) with Disallow: / while allowing the retrieval/search agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot). That keeps your content out of model training while preserving live AI-search visibility.

Will robots.txt actually stop AI crawlers?

Only the compliant ones. OpenAI, Anthropic, Google and Perplexity honor robots.txt, but bots like Bytespider have been documented ignoring it. To reliably stop a non-compliant crawler you need a server or WAF-level block by user-agent or verified IP, not just robots.txt.

Should I block AI crawlers on my site?

For most site owners, no — being cited by AI answer engines drives small but high-converting traffic. The common balanced setup is to allow retrieval/search bots and only block training crawlers. Block broadly only if you are protecting proprietary or paywalled content, and enforce it at the edge.