A robots.txt Guide for AI Search and Training Crawlers

Which bot does what?

Four companies, each running multiple, independently-controlled bots. As of October 2026:

The pattern across every company except Google: search/citation crawling, training crawling, and live user-triggered fetching are three separate jobs with three separate bot names, each independently controllable in robots.txt. Google is the exception — it uses one crawler (Googlebot) for both classic Search and its AI features, because, per its own guidance, "there are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary" beyond ordinary indexing eligibility.

What should an open-visibility robots.txt look like?

A sample file that allows every citation- and search-relevant bot, while leaving the training-only and ad-safety bots as a separate, deliberate choice:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: *
Allow: /

This sample allows everything, including the training bots (GPTBot, ClaudeBot) — a deliberate choice, not a default. Whether to allow training crawlers is a separate decision from whether to allow search/citation crawlers, covered in full in Blocking GPTBot Isn't Blocking ChatGPT. This post's job is the search/citation half: if you want to be found and cited, the search-specific bots above (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) need to be allowed, full stop.

What does blocking each one cost you?

The single most common mistake, per OpenAI's and Anthropic's own framing of their separate bots, is treating the training bot (GPTBot, ClaudeBot) as a stand-in for the whole company and blocking it expecting to disappear from search answers. It won't — OAI-SearchBot and Claude-SearchBot are the ones actually powering citations, and they're unaffected by a GPTBot or ClaudeBot disallow.

How do you verify a bot request is real, not spoofed?

Anyone can set a User-agent header to claim to be OAI-SearchBot or ClaudeBot. Verifying the real thing takes one extra step:

1. Check the published IP range, not just the user-agent string. OpenAI publishes machine-readable IP lists per bot (e.g. openai.com/searchbot.json, openai.com/gptbot.json, openai.com/chatgpt-user.json). Anthropic publishes one combined list at claude.com/crawling/bots.json. Perplexity publishes its own at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json.
2. Cross-reference your server logs' source IP against the current published range before trusting that a request claiming to be, say, GPTBot actually is GPTBot.
3. Don't rely on IP blocking as your access-control method instead of robots.txt. Anthropic's own guidance is explicit: blocking by IP address "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file" — the IP list exists for verification, not as an alternative enforcement mechanism.
4. Re-pull the IP list periodically. These ranges change as companies scale infrastructure; a verification check run against a stale list will produce false negatives.

The AI Bot Access Checker runs this kind of check for you — point it at a domain and it reports which named bots can currently fetch it, without needing to pull and cross-reference the IP lists by hand.

How often do the docs actually change?

Often enough that this post itself carries a dated caveat. OpenAI's bot documentation lists version numbers in its user-agent strings (e.g. OAI-SearchBot/1.4, GPTBot/1.4) that have changed before and will again. Anthropic runs three separate, independently-named bots today (ClaudeBot, Claude-User, Claude-SearchBot) — a split worth re-checking on any future visit to their docs, since a company adding or renaming a bot changes what a robots.txt rule actually matches. Perplexity's and OpenAI's own docs are the first-party source for all of this, and they're the ones to re-open — not a blog post, including this one — on the day you're actually editing a robots.txt file.

Checklist

• OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot all explicitly allowed, if you want to be found and cited.
• GPTBot and ClaudeBot set deliberately, not by default — a conscious training-data decision, covered fully in the companion crawler-blocking post.
• No IP-based blocking used as a substitute for robots.txt rules.
• Bot identity verified against the current published IP ranges before trusting a log entry, if verification matters for your use case.
• Robots.txt re-checked against each company's current docs at least whenever you're about to publish or audit it — not from memory.

Tools to use

• The AI Bot Access Checker — a free, one-click check of whether named AI bots can currently fetch your site.
• Your own server or CDN access logs, filtered by user-agent and cross-checked against the published IP lists above.
• A plain text editor for robots.txt itself — no special tooling required, just the correct bot names and the discipline to keep it current.

FAQ

Does blocking GPTBot stop ChatGPT from citing my site? No. GPTBot only controls training-data crawling. OAI-SearchBot is the separate bot that powers ChatGPT's search citations, and it's unaffected by a GPTBot rule.

Does Anthropic really run three separate bots? Yes, as of October 2026: ClaudeBot (training), Claude-User (user-triggered fetches), and Claude-SearchBot (search/answer indexing) — each independently controllable in robots.txt.

Can I just block an IP range instead of editing robots.txt? Anthropic specifically advises against this — IP blocking "may not work correctly or persistently" and can interfere with a bot's ability to read your robots.txt file at all. Use robots.txt as the primary control.

Will a user-triggered bot like ChatGPT-User respect my robots.txt disallow? Not reliably. OpenAI's own documentation says robots.txt rules "may not apply" to ChatGPT-User, since the fetch is triggered by a person asking about a specific page, not an automated crawl. Perplexity says the same about Perplexity-User.

What to do next

1. Check your current robots.txt against the bot table above and confirm the search/citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are explicitly allowed if you want to be found.
2. Decide the training-bot question (GPTBot, ClaudeBot) deliberately — see Blocking GPTBot Isn't Blocking ChatGPT for the full tradeoff.
3. Run the AI Bot Access Checker to confirm what you've configured actually matches what's being served — and if a bot you've allowed still can't reach your content, see is your site invisible to AI crawlers for the rendering and CDN-level checks robots.txt alone won't catch.

ContextOwl sells one-time AI visibility reports. The AI Bot Access Checker is a free check of whether named AI bots can currently reach your site.