Every AI Crawler User-Agent in 2026 (Complete List & Reference)

Here’s the complete, current list of the AI crawler user-agents you’ll see hitting your site in 2026 — what each one does, who runs it, and whether it actually respects robots.txt. Bookmark it: the key thing most lists get wrong is that each AI company runs several bots for different jobs (training, search, on-demand fetching), and they’re controlled separately. Blocking the wrong one can quietly remove you from AI search while doing nothing about training.

*Perplexity's user-fetch has been documented ignoring robots.txt in some cases. Robots.txt is an honor system — non-compliant bots need a server or WAF-level block.

How to read this list

The crawlers fall into three functional types, and the distinction matters more than the brand names:

• Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider) gather content to train models. Block these to keep your content out of training data.
• Search/index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) build the indexes that power AI answers and citations. Block these and you disappear from AI search.
• User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User) fetch a page live because a person asked the assistant about it. Blocking these stops the AI from reading a page on a user’s behalf.

So “should I block AI crawlers?” is really three questions. Most site owners want out of training but in the answers — which means blocking the training set and allowing the search and user bots. The full how-to is in how to block (or allow) AI crawlers.

The important nuances

A few things the one-line lists miss:

• Blocking GPTBot does not remove you from ChatGPT search. That’s OAI-SearchBot. Same pattern for Anthropic (ClaudeBot ≠ Claude-SearchBot) and Google (Google-Extended ≠ Googlebot). Each token is independent.
• Google-Extended controls Gemini/Vertex AI training only. Blocking it does not affect your Google Search ranking — a common and costly misunderstanding.
• CCBot (Common Crawl) is high-leverage. Common Crawl is a public dataset many models train on, so blocking CCBot limits training inclusion across many LLMs at once.
• robots.txt is an honor system. Most operators (OpenAI, Anthropic, Google, Perplexity’s main bot) comply. Bytespider has been documented ignoring it, and even Perplexity’s on-demand fetcher has been caught bypassing it. For non-compliant bots, the only real defense is a block at the server or WAF level.
• Verify before trusting. Anyone can spoof a user-agent string. Major operators publish official IP ranges (e.g. OpenAI’s gptbot.json) so you can confirm a request is genuinely from the bot it claims to be.

How to use these in robots.txt

Match a crawler by its user-agent token. To block one bot from everything:

User-agent: GPTBot
Disallow: /

To block the training bots while keeping the search/retrieval ones (the setup most people want):

# Block training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

# Allow search + user fetchers (leave unlisted, or allow explicitly)
User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /

Want to know which of these can currently reach your site? My free AI Crawler Checker tests the major AI user-agents against your URL. And if the concern is the opposite — that you’re invisible to AI — see why isn’t my site showing up in ChatGPT.

Why this matters now

Being reachable by the right AI crawlers is the entry ticket to being cited in AI answers — the whole discipline of Generative Engine Optimization (GEO). As search shifts toward answers over links (the bigger story in the future of SEO), this list is turning into basic hygiene, like robots.txt and sitemaps before it. Keep it handy, audit your robots.txt and CDN against it, and decide deliberately who gets in.

If you want honest notes on AI search as it evolves, I send them occasionally — subscribe below, and the free GEO Starter Checklist comes with the first.

Frequently asked questions

What are the main AI crawler user-agents in 2026?

The major ones are GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot and Claude-User (Anthropic); PerplexityBot and Perplexity-User (Perplexity); Google-Extended (Gemini training); Applebot-Extended, Amazonbot, Meta-ExternalAgent, MistralAI-User, DuckAssistBot, CCBot (Common Crawl) and Bytespider (ByteDance).

Do AI crawlers respect robots.txt?

Most do — OpenAI, Anthropic, Google, Apple, Amazon and Perplexity's main crawler state they honor robots.txt. Notable exceptions include Bytespider, which has been documented ignoring it, and even Perplexity's on-demand fetcher in some cases. For non-compliant bots you need a server or WAF-level block, not just robots.txt.

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT's live search uses OAI-SearchBot and on-demand fetches use ChatGPT-User — both separate. Blocking GPTBot keeps you out of training data but does not remove you from ChatGPT search results or citations.

How do I verify an AI crawler is genuine?

User-agent strings can be spoofed, so check the request's IP against the operator's published IP ranges — for example OpenAI publishes its crawler IP lists (gptbot.json and related files). If the IP is not in the official range, treat the request as an impersonator.

What is the difference between a training crawler and a search crawler?

A training crawler (like GPTBot or CCBot) collects content to train AI models. A search/index crawler (like OAI-SearchBot or PerplexityBot) builds the index that powers live AI answers and citations. They are controlled separately, so you can block training while staying visible in AI search.