How Claude Searches the Web and Describes Brands

What are Claude's three bots?

Anthropic's own crawler documentation names three, each with a distinct job:

• ClaudeBot — crawls web content to help train future Claude models. Anthropic's documentation states that blocking it means "future materials should be excluded from our AI model training datasets." It has nothing to do with live search or citations.
• Claude-SearchBot — crawls and indexes content specifically to improve the quality and accuracy of Claude's search-based answers. This is the bot most directly tied to whether Claude can cite your pages.
• Claude-User — fetches a single page live, triggered by an actual person asking Claude to look at or use a specific URL in that moment. Blocking it stops Claude from retrieving your content when a user's query points at it directly.

All three independently honor robots.txt. Blocking ClaudeBot doesn't block Claude-SearchBot or Claude-User, and vice versa — each needs its own rule.

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

Each block is independent — remove or adjust the lines for whichever bot(s) you actually want to allow.

Which one decides search citations?

Claude-SearchBot is the one that matters for "will Claude cite my page." It's described by Anthropic as existing specifically to index content so Claude's search-based answers are more relevant and accurate — functionally the closest equivalent to OAI-SearchBot for ChatGPT or Claude-SearchBot's counterparts at other engines (see how ChatGPT chooses sources for that comparison).

ClaudeBot, by contrast, only affects whether your content might be used in training a future model — a separate question from whether today's Claude can search for and cite your page right now. Blocking ClaudeBot while leaving Claude-SearchBot open means: your content won't train future models, but Claude can still find and cite it in a live search-enabled answer. The common mistake is assuming one "Claude bot" setting covers all of this — it doesn't, and the same split exists for how Gemini and Copilot handle brand questions (see how Claude, Gemini and Copilot answer brand questions); see a robots.txt guide for AI search and training crawlers for the full pattern across engines, and whether blocking GPTBot actually blocks ChatGPT for the equivalent mistake on OpenAI's side.

What does Claude's web search actually do?

Anthropic's own API documentation describes a three-step process: Claude decides, per prompt, whether the request depends on current, changing, or specific information that's outside stable training knowledge (recent events, current prices, specifics about an organization or product that may have changed). If so, it runs one or more searches — this can repeat multiple times within a single request — then composes a final answer with inline citations pointing to the exact sources used, including the specific cited text from each page.

Requests that draw on stable knowledge (established facts, general concepts, analysis of what's already in the conversation) are answered directly, with no search and nothing to cite. This is the same two-path pattern documented for other engines: only the search path involves choosing sources at all.

What Anthropic's documentation doesn't publish is a ranking formula for which search results get used once a search runs — the documented process covers the decision to search, the mechanics of running it, and the citation format, not a scoring method for selecting among results. That's a consistent gap across most engines' public documentation, not specific to Claude.

What does blocking each one do?

How to test brand presence in Claude?

Ask a real, neutral buyer question — no brand name in the prompt — in a fresh conversation with web search enabled, and separate three outcomes: named, cited (and to whose site), and recommended.

Here's a real run, done live for this post, using Claude's actual web search tool — the same mechanism documented above, called directly through the API rather than through the consumer chat interface:

Evidence: Claude Sonnet 5, web search tool enabled, prompt: "What's the best password manager for a small team?", 10 October 2026, API session (not signed in to any consumer account). Result: 1Password named first and described as "the benchmark for team password management," specifically positioned for organizations of 5-50 people, with pricing cited ($8.99 per user/month Business, or a $24.95/month Teams Starter plan for up to 10 people). Bitwarden and Dashlane were also named and recommended for different priorities (budget and open-source; compliance features). All pricing and feature claims were cited to third-party sources (TechRepublic, TechRadar and similar comparison sites) — none were cited to the vendors' own pricing pages directly.

This matches the pattern seen across other engines on this blog: brands get named and recommended with specific, current-sounding facts attached, but the citation trail often runs through a third-party comparison site rather than the brand's own page. If a brand's own pricing or feature claims differ from what a comparison site states, that's the gap an engine is actually repeating — and it's worth noting these answers vary run to run, so this is one sample, not a settled verdict.

What is unknown?

• No published ranking formula for how Claude's search tool selects among results once a search runs — Anthropic documents the workflow, not the scoring.
• No large-sample, dated study (found for this post) comparing Claude's citation overlap with Google's top 10 the way Ahrefs measured it for ChatGPT, Gemini and Copilot. [VERIFY] if a Claude-specific version of that study exists; none was opened for this post.
• Whether the consumer Claude.ai web search behaves identically to the API's web search tool used for the live evidence above — both are documented by Anthropic as the same underlying capability, but this post didn't test the consumer interface directly, since it wasn't available this session.
• Bot behavior can change. Anthropic's own crawler documentation is actively maintained and has been updated before; re-verify bot names and robots.txt handling on the day you publish anything that depends on them.

FAQ

Does blocking ClaudeBot stop Claude from citing my site? No. ClaudeBot only controls training-data crawling. Claude-SearchBot is the one that indexes content for Claude's search and citation feature — they're independent settings in robots.txt.

Is blocking Claude's IP addresses a reliable way to opt out? No. Anthropic states directly that IP blocking "may not work correctly or persistently guarantee an opt-out," partly because it can prevent the crawler from reading your robots.txt rules at all. Use robots.txt rules, not IP blocks.

Does Claude always search the web before answering? No. Claude decides per prompt whether the request needs current or specific information. Stable-knowledge questions (general facts, math, analysis of existing conversation content) get answered directly, with no search and nothing to cite.

Does ContextOwl check Claude by default? No. ContextOwl's base check is ChatGPT; Claude is available as one of three add-on engines, alongside Gemini and Perplexity.

How do I check what Claude says about my brand? Ask a real buyer question, without your brand name, in a fresh Claude conversation with web search on, and log whether you were named, cited (and to whose site), and recommended, separately. Repeat on a different day before treating one answer as settled.

What to do next

1. Check your robots.txt for ClaudeBot, Claude-SearchBot and Claude-User separately — a rule for one doesn't cover the others. The AI Bot Access Checker reads this for you.
2. If you've blocked any Claude bot by IP address instead of robots.txt, switch to a robots.txt rule — Anthropic's own documentation says IP blocking isn't reliable.
3. Run three to five real buyer questions through Claude with web search on, logged out of any account, and log named/cited/recommended for each.

ContextOwl sells one-time AI visibility reports. The AI Bot Access Checker is a free way to see which AI crawlers, including Claude's three bots, your robots.txt currently allows.