How do I allow GPTBot, OAI-SearchBot and PerplexityBot in robots.txt?
All guidesUpdated 6 steps
TL;DR
Add a separate group for each crawler to your robots.txt: a User-agent line with the bot's exact name, then Allow: / on the next line. A group that names a bot overrides your User-agent: * rules for that bot. Treat search and training as two decisions: OAI-SearchBot and PerplexityBot fetch pages for answers, while GPTBot collects pages for model training.
AI companies run more than one crawler, and they do different jobs. Some fetch pages so an assistant can answer a question and link to a source. Others collect pages that may be used to train future models. You can allow one kind and block the other.
robots.txt is a request, not a lock. Reputable crawlers follow it, but it does nothing against one that ignores it, and it does not remove anything already collected.
Step by step
- Step 1
Read the robots.txt you already have
Open yoursite.com/robots.txt and note every User-agent group. Look for a User-agent: * group with Disallow: /, and for any AI bot named directly. Do not delete rules you did not write until you know why they are there.
- Step 2
Know which bot does which job
Answer and search bots: OAI-SearchBot and ChatGPT-User from OpenAI, PerplexityBot from Perplexity, and Claude-SearchBot from Anthropic. Training controls: GPTBot from OpenAI, ClaudeBot from Anthropic, Google-Extended from Google and Applebot-Extended from Apple. Google-Extended is Google's AI training control; blocking it does not remove you from Google Search or AI Overviews, which use Googlebot.
- Step 3
Add one explicit group per answer bot
At the end of the file, add a group for each answer bot: a User-agent line with its exact name, then Allow: / on the next line, with a blank line between groups. A crawler that follows the robots.txt standard obeys the group that names it and ignores the User-agent: * group, so this works even when * is blocked.
- Step 4
Decide on training bots separately
If you accept your public pages being used for model training, add Allow: / groups for GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. If not, give them Disallow: / instead. Blocking the training bots does not block the answer bots, because each bot reads only its own group.
- Step 5
Add your sitemap line
If the file has no Sitemap line, add one with the full URL of your sitemap, such as https://yoursite.com/sitemap.xml. Search engines that support the Sitemap line use it to discover your pages.
- Step 6
Check the firewall, then the logs
Some CDNs and hosting firewalls block AI crawlers before they ever read robots.txt, so check those settings too. Over the following days, look for the bot names in your server logs. RankingTop's free scan reads your live robots.txt and reports which AI crawlers it lets in.
Questions people ask next
Does Allow: / for one bot override Disallow: / under User-agent: *?
Yes, for the bot you name. A crawler that follows the robots.txt standard uses the group that matches its name and falls back to the * group only when no group names it. So an OAI-SearchBot group with Allow: / lets that bot in even when * is blocked.
Will blocking GPTBot keep my site out of ChatGPT search?
No. OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User as separate controls, so blocking GPTBot affects training use, not search results. It also does nothing about what earlier models already learned.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls whether Google may use your pages for AI training. AI Overviews are part of Google Search, which crawls with Googlebot, and limiting them means using Search controls such as nosnippet or noindex, which also affect your regular listings.
Do all AI crawlers obey robots.txt?
Not necessarily. The companies named here publish their crawler names and say those crawlers follow robots.txt, but robots.txt is voluntary and cannot stop a bot that ignores it. Fetchers that act on a user's direct request may follow different rules, so read each operator's current documentation.