Blocking GPTBot Isn't Blocking ChatGPT: Should You Block AI Crawlers?

Does blocking GPTBot stop ChatGPT from citing you?

No. GPTBot and the bot that actually powers ChatGPT's live search and citations are two different programs with two different user-agent names, and OpenAI's own crawler documentation treats them as independent settings. Disallowing GPTBot in robots.txt tells OpenAI not to use your pages to train future models. It has no effect on OAI-SearchBot, which is what fetches and indexes pages so ChatGPT can name and cite them in search-grounded answers, or on ChatGPT-User, which fetches a specific page when someone asks ChatGPT about it directly.

This matters because a lot of "block the AI bots" advice treats GPTBot as a stand-in for ChatGPT generally. It isn't. If your goal is to keep your content out of model training while still being discoverable in ChatGPT's answers, blocking only GPTBot is exactly the right move — not a mistake. If your goal is to disappear from ChatGPT answers entirely, blocking GPTBot alone will not do that, and you may be disappointed when your pages still turn up cited.

What does each bot actually do?

Three vendors, each with the same basic split: a crawler that feeds training, a crawler that feeds search/retrieval, and a fetcher triggered by an individual user's request.

OpenAI (per developers.openai.com/api/docs/bots):

• GPTBot — crawls content that may be used to train OpenAI's generative AI foundation models. Respects robots.txt.
• OAI-SearchBot — crawls and indexes pages so ChatGPT's search feature can surface and cite them in answers. Not used for model training. Respects robots.txt.
• ChatGPT-User — fetches a page when a person asks ChatGPT to look at or act on a specific URL (including Custom GPTs and GPT Actions). OpenAI's documentation now states that because this is a user-initiated action, "robots.txt rules may not apply" to it.
• OAI-AdsBot — checks landing pages submitted as ChatGPT ads for safety; not used to train models. A narrow, less relevant case for most sites.

Anthropic (per Anthropic's crawler support article, article ID 8896518):

• ClaudeBot — collects web content that may contribute to training Claude models. Respects robots.txt, including the non-standard Crawl-delay directive.
• Claude-SearchBot — crawls to improve the relevance and accuracy of Claude's search results. Respects robots.txt.
• Claude-User — fetches a page when someone asks Claude a question that requires visiting it. Respects robots.txt.

Anthropic documents these as three separate user-agents, each disallowed or allowed on its own line. Nothing in the documentation ties them together — disallowing ClaudeBot by name has no bearing on Claude-User or Claude-SearchBot, which each need their own entry if you want to act on them too.

Perplexity (per Perplexity's developer crawler docs):

• PerplexityBot — the automated crawler that builds the index Perplexity's answers draw on. Not used to train foundation models. Respects robots.txt.
• Perplexity-User — fetches a page live when a specific user query requires it. Generally ignores robots.txt, on the same "a person asked for this" logic OpenAI uses for ChatGPT-User.

The pattern across all three vendors: the training bot and the search bot are named differently and controlled separately, and the user-triggered fetcher increasingly sits outside robots.txt altogether because it's standing in for a human click, not an autonomous crawl.

What happens if you block GPTBot only?

Your pages stop being candidates for GPTBot's training crawls going forward (it says nothing about data already collected before you blocked it). OAI-SearchBot keeps crawling and your pages can keep appearing, named and cited, in ChatGPT's search-grounded answers. ChatGPT-User keeps fetching individual pages when a user asks about them, and per OpenAI's current documentation may do so even if you'd disallowed it specifically. This is the single most common "AI crawler" mistake: a site blocks GPTBot expecting to disappear from ChatGPT, gets confused when our AI Citation Snapshot-style checks or a full report still show citations, and assumes the block "isn't working." The block is working exactly as documented — it just isn't the block that controls what you're trying to control.

When does blocking make sense?

Blocking the training bots (GPTBot, ClaudeBot) is a defensible, common choice when you don't want your content used as raw training material — a publisher protecting original reporting, a company with contractual or licensing reasons to withhold its text, or simply a site owner who objects on principle. That's a real tradeoff with a real benefit, and it's the bot OpenAI and Anthropic both describe as training-specific.

Blocking the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) makes sense in a much narrower set of cases: content that's already paywalled or gated from the general public, where you don't want a free summary showing up as a substitute for a subscription; or content you genuinely don't want AI engines naming at all, such as internal tools or staging environments. For almost everything else — a marketing site, a SaaS product, a brand's own pages — blocking a search bot removes a free channel to be named and cited, with no offsetting benefit.

What does blocking search bots cost you?

If an AI search engine can't crawl your pages at all, it can't cite them, and it can't ground a recommendation in your own stated facts — pricing, features, service area. The engine still answers the question; it just answers it using whatever it can see, which may be a competitor's page, a review site, or an outdated secondary source about you. This is the same gap our AI search audit checklist flags as a first-pass check, because it's cheap to get wrong by accident: a legacy robots.txt rule, a security team's blanket "block all bots" policy, or a CDN-level WAF rule written before these user-agents existed can all quietly disallow a search bot nobody meant to block.

How should a team review this setting?

Robots.txt for AI bots isn't a file you write once. Review it on the same cadence as any other technical SEO setting — after a CMS migration, a new WAF or CDN rollout, or roughly quarterly — and check it against what each vendor documents today, not what a blog post said a year ago; vendors have changed this behavior before (OpenAI's own ChatGPT-User robots.txt stance is a recent example) and will again.

For most commercial sites outside publishing and paywalled content, the recommendation collapses to one line: allow everything except the training bots you have a specific reason to block, and even that's optional.

FAQ

Does blocking GPTBot remove me from ChatGPT's answers? No. GPTBot only controls training. ChatGPT's search-grounded answers and citations come from OAI-SearchBot and, for single-page lookups, ChatGPT-User — both are separate settings that blocking GPTBot does not touch.

If I block ChatGPT-User, will it stop visiting my site? Maybe not. OpenAI's current documentation says robots.txt rules "may not apply" to ChatGPT-User because the fetch is initiated by a person using ChatGPT, not an autonomous crawl. Treat it more like a browser request than a bot you can fully gate with robots.txt alone.

Does Anthropic's ClaudeBot block also block Claude-User and Claude-SearchBot? No. Anthropic documents these as three separate user-agents, each controlled by its own robots.txt entry. Disallowing one by name has no effect on the other two.

Is there one "block all AI bots" robots.txt rule I can copy? Not a safe one. A blanket block hits training, search, and user-triggered bots together, which for most non-publisher sites trades a real cost (disappearing from AI answers) for a benefit (opting out of training) you may not even want in that bundle. Decide per bot type, not per vendor.

How do I check what my current robots.txt actually allows? Read the file directly against each vendor's documented user-agent strings, since the names are specific (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) and easy to miss in a legacy rule written for generic "bot" patterns.

What to do next

1. Open your live robots.txt and list every AI-related user-agent line already in it, including any inherited from a CDN or WAF default.
2. Match each line against the bot's actual job — training, search, or user-triggered — using the breakdown above, not assumptions from older advice.
3. Decide per bot type using the scenario table, document the decision, and put a recheck of the same file on your calendar for the next CMS or infrastructure change. While you're in there, llms.txt is worth a look too — a different, lower-stakes file with much thinner evidence behind it.

This is ContextOwl's own read of current vendor documentation, not legal advice; check the AI Bot Access Checker against your live robots.txt before changing anything.