AI crawler checker
See which AI bots can read your site: ChatGPT, Claude, Perplexity, Gemini, Copilot and more. We check your robots.txt and your firewall. Free, no sign-up.
What this checker tests
- robots.txt rules for 20 AI bots. For each bot we apply your rules the way crawlers do, and show if it can read your homepage.
- Your firewall. We load your homepage as a normal browser and as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, to spot bot blocking that robots.txt doesn't show.
- llms.txt and your sitemap. Whether you publish an llms.txt file and list your sitemap in robots.txt.
AI search bots vs AI training bots
Not all AI bots do the same job. Search and answer bots fetch pages so tools like ChatGPT, Perplexity and Copilot can cite them. Block these and AI answers can't quote you. Training bots collect pages to train future models. Blocking them is a business choice and doesn't remove you from today's AI answers.
AI crawlers we check
Search and answer bots
| Bot | Company | Used for |
|---|---|---|
| Googlebot | Google Search and AI Overviews | |
| Bingbot | Microsoft | Bing and Copilot answers |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | ChatGPT opening a page a person asked about |
| PerplexityBot | Perplexity | Perplexity answers |
| Perplexity-User | Perplexity | Perplexity opening a page a person asked about |
| Claude-SearchBot | Anthropic | Claude search results |
| Claude-User | Anthropic | Claude opening a page a person asked about |
| Applebot | Apple | Siri and Spotlight answers |
| DuckAssistBot | DuckDuckGo | DuckDuckGo AI answers |
| MistralAI-User | Mistral | Le Chat opening a page a person asked about |
Bots that open a page because a person asked about it, such as ChatGPT-User and Perplexity-User, may not follow robots.txt the way regular crawlers do.
Training bots
| Bot | Company | Used for |
|---|---|---|
| GPTBot | OpenAI | Training OpenAI models |
| ClaudeBot | Anthropic | Training Claude models |
| Google-Extended | Gemini model training and grounding | |
| Applebot-Extended | Apple | Apple AI model training |
| Meta-ExternalAgent | Meta | Meta AI model training |
| CCBot | Common Crawl | Open web dataset many AI models train on |
| Bytespider | ByteDance | ByteDance AI model training |
| Amazonbot | Amazon | Alexa and Amazon AI |
| cohere-ai | Cohere | Cohere AI models |
How to allow AI search bots in robots.txt
Add rules like these near the top of your robots.txt file. Each one lets a bot read your whole site.
User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: /
To keep a training bot out while staying in AI search, block it by name, for example User-agent: GPTBot followed by Disallow: /.
Frequently asked questions
What is an AI crawler?
An AI crawler is a bot that reads web pages for an AI company. Some, like OAI-SearchBot and PerplexityBot, fetch pages so AI answers can cite them. Others, like GPTBot and ClaudeBot, collect pages to train AI models.
Does blocking GPTBot remove my site from ChatGPT?
Not on its own. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. You can block GPTBot and still allow OAI-SearchBot, so ChatGPT search can find and cite your pages.
My robots.txt allows AI bots. Why are they still blocked?
Many firewalls and CDNs block AI bots before they reach robots.txt, for example a "block AI bots" setting. The firewall test in this checker loads your homepage as AI bots to spot that.
What is llms.txt?
llms.txt is a short text file at the root of your site that lists your most useful pages for AI tools. It is optional, and Google has said it does not use it, but some AI tools read it.
Should I block AI training bots?
It is a business choice. Blocking training bots keeps your content out of future model training. It does not stop AI search bots, which are the ones that decide whether you get cited today.
Is this AI crawler checker free?
Yes. It is free, needs no sign-up and checks any public website.