- Claude, Gemini and Copilot each publish some detail about sourcing and leave the rest unconfirmed — this post says exactly where that line falls for each.
- Claude has three documented bots with distinct jobs: ClaudeBot (training), Claude-SearchBot (search quality), and Claude-User (user-triggered fetch).
- Gemini's "grounding with Google Search" cites sources with structured metadata, but Google hasn't published whether it shares the same crawl or index as AI Overviews and AI Mode.
- Copilot runs on Microsoft's Bing index — there's no separate "Copilot crawler"; Bingbot's crawl is what feeds it.
- ContextOwl's own reports cover ChatGPT as the base engine plus Gemini, Perplexity and Claude as add-ons. Copilot is covered here for comparison only — it isn't a ContextOwl report engine.
What does each engine actually publish about how it finds sources?
Unfamiliar term below? The AI search glossary covers "grounding," "crawler" and the rest in one place — or see AI search vs traditional search for the general retrieve-then-generate pipeline these three engines each build on differently. Claude publishes the most structured detail of the three on crawler-level behavior. Anthropic's support documentation names three separate bots and states plainly that all of them "respect 'do not crawl' signals by honoring industry standard directives in robots.txt," plus the Crawl-delay extension for pacing.
Gemini's public documentation is strongest on the citation mechanics, weaker on the retrieval side. Google's developer docs for "grounding with Google Search" state that when grounding is enabled, "the model handles the entire workflow of searching, processing, and citing information automatically," and that responses carry structured groundingChunks (the actual source URLs and titles) and groundingSupports (which link specific stretches of the answer's text back to those sources) — "the key to building inline citations," in Google's own words. What that page does not say, as of this writing, is whether this grounding process draws on the same crawl and index that power AI Overviews and AI Mode in regular Search.
Copilot's documentation is the thinnest of the three on mechanics, but Microsoft does publish real citation data through a different channel: Bing Webmaster Tools' AI Performance report, live as a public preview since 10 February 2026, which tracks "how often your content is referenced by AI systems" across "Microsoft Copilot, AI-generated summaries in Bing, and select partner integrations" — though Microsoft itself flags this data as "a sample of overall citation activity," not a complete count.
Which crawlers serve search versus training, for each engine?
| Engine | Crawler / mechanism | What it does | Respects robots.txt? |
|---|---|---|---|
| Claude | ClaudeBot | Collects content "that could potentially contribute to" model training | Yes |
| Claude | Claude-SearchBot | "Navigates the web to improve search result quality" — indexing for Claude's search feature | Yes |
| Claude | Claude-User | Fetches a page live when a person asks Claude about a specific URL | Yes |
| Gemini | Googlebot (regular Search crawl) + a separate "grounding with Google Search" retrieval step | Standard Search crawl feeds Google's index; grounding retrieves from Search at answer time | Yes (Googlebot); grounding's own crawl behavior isn't separately documented |
| Gemini | Google-Extended (a robots.txt token, not a crawler — no user agent ever appears in server logs) | Opts a site's already-crawled content out of training and grounding for Gemini Apps and Vertex AI; doesn't affect Search ranking or AI Overviews per Google's own framing | N/A — it's read from robots.txt, not a bot that visits |
| Copilot | Bingbot | Standard Bing crawl; there is no separate, publicly documented "Copilot bot" — Copilot answers draw on Bingbot's index | Yes |
Unknowns and undocumented are marked explicitly below rather than guessed into this table.
What differs in how each engine actually names and describes brands?
This is the part none of the three document, and it's worth saying so plainly rather than inferring a pattern from a handful of anecdotal chats. What is observable, without claiming to know the mechanism: Gemini's grounding output is structured enough (named source chunks mapped to exact text spans) that, when grounding is on, you can usually tell precisely which sentence came from which URL — a more auditable citation format on paper than a conversational answer with no visible source list. Copilot's citation behavior is now measurable in aggregate through Bing's own dashboard (total citations, which pages, how that changes over time), even though Microsoft explicitly warns that frequency "does not indicate ranking, authority, or the role of any page within an individual answer." Claude's documentation doesn't describe citation-selection logic at all — only crawler purpose and robots.txt behavior.
None of this is enough to rank the three by accuracy or fairness to brands. It only tells you which engine gives you more to inspect after the fact.
What is genuinely unknown, that shouldn't be guessed at?
- Whether Gemini's search grounding uses the same retrieval index and ranking signals as AI Overviews and AI Mode, or a separate pipeline — Google's grounding documentation doesn't say.
- The exact reranking or selection logic any of the three use once candidate pages are retrieved — none publish this.
- How large a role, if any, each engine's own foundation-model training data (as opposed to live retrieval) plays in naming a brand when no citation is shown.
- Whether Copilot's underlying model weighs sources differently from a standard Bing search ranking once Bingbot's crawl feeds it — Microsoft's documentation describes what's tracked (citations), not how a given citation was chosen.
Treat anything beyond what's quoted above, for any of the three, as inference — including inference in this post.
Which of these should a team actually prioritise?
Here is the honest framing, stated directly rather than implied: ContextOwl's own reports run ChatGPT as the base engine, with Gemini, Perplexity and Claude available as add-ons. Copilot is covered in this post for comparison only, because the pain point — "which platforms to prioritise" — doesn't make sense to answer without mentioning it, but ContextOwl does not report on Copilot or Google AI Overviews as product engines.
With that disclosed, the practical case for the three ContextOwl does cover: Gemini sits inside Google's own ecosystem and increasingly inside AI Overviews and the Gemini app directly, so it matters wherever your buyers already use Google. Claude has grown as a long-form research and work tool, which tends to surface more detailed brand comparisons than a quick chat answer. Perplexity, covered separately in how Perplexity chooses its sources, is worth checking because it shows its source list by default, making it the easiest of the group to audit directly. We lay out a fuller prioritisation framework, independent of which tool you use to check it, in which AI engines should you prioritise.
FAQ
Does Claude have one bot or several? Three: ClaudeBot (training), Claude-SearchBot (search indexing), and Claude-User (user-triggered page fetches). All three respect robots.txt.
Does blocking Google-Extended stop my site from appearing in AI Overviews? No. Google-Extended only controls training and grounding use for Gemini Apps and Vertex AI; it isn't a crawler and, per Google's own framing, doesn't affect Search ranking or inclusion in AI Overviews, which run through Googlebot's regular crawl.
Is there a dedicated Copilot crawler to allow or block? No separate, publicly documented one. Copilot's answers draw on Bingbot's existing crawl and index — the same bot that feeds standard Bing search.
Does ContextOwl track Copilot or Google AI Overviews? No. ContextOwl's reports cover ChatGPT as the base engine, with Gemini, Perplexity and Claude as add-ons. Copilot and Google AI Overviews are not covered.
Which engine is easiest to audit yourself without any tool? Gemini's grounding output and Perplexity's default source list are both more directly inspectable than a Claude or Copilot answer with no visible citations — when either shows a citation, you can usually trace exactly which sentence it supports.
What to do next
- Check which of the documented bots — ClaudeBot, Claude-SearchBot, Claude-User, Bingbot — can currently reach your site using the AI Bot Access Checker.
- If you use Google's own AI products, separately check Bing Webmaster Tools' AI Performance report for Copilot citation volume — it's free and already live for any verified site.
- Decide prioritisation based on where your actual buyers already spend time, not on which engine is easiest to measure — then read which AI engines should you prioritise for the fuller framework.
ContextOwl sells one-time AI visibility reports covering ChatGPT, Gemini, Perplexity and Claude. The AI Citation Snapshot is a free, single-question version of the same check.
Sources
- Anthropic, "Does Anthropic crawl data from the web, and how can site owners block it or control it?": https://support.claude.com/en/articles/8896518
- Google AI for Developers, "Grounding with Google Search" (last updated 9 October 2026): https://ai.google.dev/gemini-api/docs/generate-content/google-search
- Microsoft, "Introducing AI Performance in Bing Webmaster Tools Public Preview" (10 February 2026): https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview
- Google Search Central, "AI features and your website" (last updated 10 December 2025): https://developers.google.com/search/docs/appearance/ai-features
- Search Engine Land, "Google introduces Google-Extended to let you block Bard, Vertex AI via robots.txt" (September 2023), on what Google-Extended is and isn't: https://searchengineland.com/google-extended-crawler-432636
- Google, "Google's common crawlers" (current), on Google-Extended's present scope — training and grounding for Gemini Apps and Vertex AI, no effect on Search ranking: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
More in AI Search Basics
A results page ranks pages and lets you choose. An AI answer retrieves, selects and writes a few sentences that name a few brands for you.
There's no single citation algorithm. Retrieval, relevance, freshness and source type each play a part — and some of them are yours to influence.
AI knows a brand from memory built during training, or from pages fetched live. You can influence the second one quickly and the first one only slowly.