Can AI crawlers read your site?
Free tools · AI crawler checker
Type a domain. We read its robots.txt and show, for nine AI crawlers, whether each may fetch your pages. Then we write a robots.txt that keeps every rule you have and lets the answer crawlers in.
The nine AI crawlers we check
Run a check to see whether each one may fetch your pages.
| Crawler | Run by | Job | Access |
|---|---|---|---|
| GPTBotCollects pages that may be used to train OpenAI models | OpenAI | Model training | Not checked |
| OAI-SearchBotSurfaces sites in ChatGPT search answers | OpenAI | Answers questions | Not checked |
| ChatGPT-UserFetches a page when a ChatGPT user asks for it | OpenAI | Answers questions | Not checked |
| PerplexityBotSurfaces and links sites in Perplexity search results | Perplexity | Answers questions | Not checked |
| ClaudeBotCollects pages that may be used to train Anthropic models | Anthropic | Model training | Not checked |
| Claude-SearchBotCrawls pages to improve Claude search results | Anthropic | Answers questions | Not checked |
| Google-ExtendedAI training control; it does not affect Google Search or AI Overviews, which use Googlebot | Model training | Not checked | |
| Applebot-ExtendedControls whether Applebot-crawled pages may train Apple AI models | Apple | Model training | Not checked |
| BingbotCrawls for Bing search, which Microsoft Copilot searches to answer | Microsoft | Answers questions | Not checked |
Crawler access is line one of the proof.
The free proof runs this check, then crawls your pages and ranks every issue by impact: broken pages first, then titles, then what crawlers cannot read.
Reading the result
The full walk-through, with the exact groups to add, is in the guide How do I allow GPTBot, OAI-SearchBot and PerplexityBot in robots.txt?
Which crawlers matter most for being named in AI answers?
The answer crawlers: OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot for Perplexity, Claude-SearchBot for Claude. They fetch pages so an assistant can answer a question and point to a source. A page they cannot fetch cannot be quoted.
Should I block the training crawlers?
That is your choice, and it is separate from being found. GPTBot, ClaudeBot, Google-Extended and Applebot-Extended control whether your pages may be used to train models. Blocking them does not remove anything already collected and does not stop the answer crawlers.
Does blocking Google-Extended remove me from Google AI Overviews?
No. Google documents Google-Extended as a training control that does not affect Google Search. AI Overviews draw on pages Googlebot crawls, so they follow your Googlebot rules.
My robots.txt allows everything. Why can an AI crawler still not read my site?
robots.txt is only one gate. Some hosting and CDN providers block AI crawlers at the firewall before the file is read, and a page that answers with an error or a challenge cannot be read either. Check your bot settings, then your server logs.