Free tool
AI crawler checker
Paste your robots.txt and see exactly which AI crawlers you're allowing or blocking — and whether you're accidentally shutting out the bots that power AI answers and citations. Everything is parsed in your browser; nothing is sent.
| Crawler | Operator | Role | Status |
|---|---|---|---|
| GPTBotCrawls the web to train OpenAI models. | OpenAI | Model training | — |
| ChatGPT-UserFetches a page when a ChatGPT user or action follows a link. | OpenAI | Live AI answersaffects citations | — |
| OAI-SearchBotIndexes pages for ChatGPT search results and citations. | OpenAI | Live AI answersaffects citations | — |
| ClaudeBotCrawls the web for Anthropic (training and product indexing). | Anthropic | Model trainingaffects citations | — |
| PerplexityBotIndexes pages so Perplexity can cite them in answers. | Perplexity | Live AI answersaffects citations | — |
| Perplexity-UserFetches a page live when a user follows a Perplexity citation. | Perplexity | Live AI answersaffects citations | — |
| Google-ExtendedOpt-out token for Gemini training & grounding (does not affect Search ranking). | Model trainingaffects citations | — | |
| GooglebotThe main Google crawler — feeds Search and AI Overviews. | Search indexaffects citations | — | |
| BingbotBing’s crawler — feeds Bing and Copilot answers. | Microsoft | Search indexaffects citations | — |
| CCBotOpen web archive used as training data by many AI models. | Common Crawl | Model training | — |
| Applebot-ExtendedOpt-out token for training Apple’s generative models. | Apple | Model training | — |
| ApplebotApple’s crawler for Siri and Spotlight suggestions. | Apple | Search indexaffects citations | — |
| meta-externalagentCrawls the web to train and ground Meta AI. | Meta | Model training | — |
| AmazonbotCrawls for Alexa answers and Amazon’s AI services. | Amazon | Live AI answersaffects citations | — |
| BytespiderByteDance/TikTok crawler, widely used for AI training. | ByteDance | Model training | — |
How it works
- The tool reads your robots.txt the way a crawler does: it finds the rule group that applies to each bot (its own
User-agentblock, or the wildcard*) and checks whether the site root is disallowed. - Blocked means that bot is told not to crawl your site (
Disallow: /). Allowed means it can. - Bots marked affects citations are the ones that fetch or index pages for live AI answers — blocking those is what quietly removes you from ChatGPT, Perplexity and Google AI answers.
- One nuance worth knowing:
Google-Extendedonly controls Gemini training and grounding — it does not change your normal Google Search ranking.
Frequently asked questions
Which AI crawlers should I allow?
If you want visibility in AI answers, allow the crawlers that fetch or index pages for live answers: OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User (Perplexity), plus Googlebot and Bingbot (which feed Google AI Overviews and Copilot). Blocking these removes you from those AI answers. Training-only bots like GPTBot and CCBot are a separate choice — blocking them does not affect citations.
Does blocking GPTBot hurt my AI visibility?
Not directly. GPTBot is used for training OpenAI models, not for answering live queries, so blocking it does not remove you from ChatGPT search results or citations. The bot that matters for ChatGPT citations is OAI-SearchBot (and ChatGPT-User for live fetches).
What does Google-Extended actually control?
Google-Extended is an opt-out token for using your content in Gemini training and grounding. It does not affect how Googlebot crawls you or how you rank in normal Google Search — those are governed by Googlebot. You can block Google-Extended and still rank fine in Search.
Is this crawler checker accurate and private?
It applies standard robots.txt group-matching (an exact User-agent block wins over the wildcard) and checks whether the root path is disallowed — the same logic crawlers use for the common allow-all / block-all cases. It runs entirely in your browser, so your robots.txt is never uploaded. Very unusual per-path rules are simplified to the site-root decision.