Same Prompt, Different Brand Lists: How Many Runs Do You Need?

How consistent are repeated runs?

Not very. The most comprehensive public test of this, run by SparkToro (Rand Fishkin) with Gumshoe.ai (Patrick O'Donnell), had 600 volunteers run 12 brand-recommendation prompts through ChatGPT, Claude and Google's AI (AI Overviews and AI Mode) a combined 2,961 times across November and December 2025, with each prompt run 60 to 100 times per platform. The finding: asking an identical prompt twice carries under a 1-in-100 chance of getting the exact same brand list back, and under a 1-in-1,000 chance of getting that same list in the same order.

Two disclosures worth stating plainly, because they affect how to read this. The study's authors run SparkToro and Gumshoe.ai, tools that sell audience and AI-tracking data — a real interest in making repeated-run variance sound important. And ContextOwl sells AI visibility reports built from prompts sent once per engine, so we have an adjacent interest in this question too. Neither interest makes the underlying numbers wrong, but both are reasons to read the method, not just the headline.

Does order or appearance hold up better?

Appearance held up meaningfully better than order. The study found that certain brand names recurred in 60% to 90% of responses for a given category and intent, even while the exact list and the order within it kept shifting run to run. Claude was slightly more likely than ChatGPT or Google's AI to repeat the same list twice in a hundred runs, but even less likely to repeat that list in the same order.

This is the practical reason "visibility rate" (how often you're named across many runs) is the more defensible unit than "rank" (where you landed on one run). Rank looks precise — "we're #2 in ChatGPT" — but on this evidence it's closer to noise dressed as a measurement. Frequency is noisier to produce (you need many runs to get a stable read) but far more honest about what it's actually telling you. Our full piece on why rank specifically doesn't hold up goes deeper on that distinction.

There's a cross-platform layer on top of the within-platform variance: the study found brand mentions disagreed across ChatGPT, Claude and Google's AI 62% of the time for the same prompt. So two sources of noise stack — run-to-run on one engine, and engine-to-engine for the same question — which is why a single screenshot from a single engine proves very little on its own. See why one ChatGPT screenshot isn't an audit for the practical version of that caution.

How many prompts and runs are enough?

There's no universal number, but the shape of the tradeoff is clear from the study's own design: 12 prompts, 60 to 100 runs each, roughly 3,000 total runs, took 600 volunteers to assemble — well beyond what any individual team can replicate by hand. More runs narrow the gap between "what we observed" and "what's actually typical," the same way a bigger poll sample narrows a margin of error. Fewer runs are cheaper and faster but carry a wider, unstated margin of error that most single-run checks never acknowledge.

A practical compromise for a team without research-study resources: a handful of prompts (five to ten, covering your real buyer questions) run multiple times each (three to five, on at least two engines) gives a materially better read than one prompt run once — not because it matches the statistical power of a 3,000-run study, but because it stops treating a single lucky or unlucky run as a finding. Whatever count you choose, state it: "named in 3 of 5 runs" is honest; "named" from one run presented as a settled fact is not.

How can a reader test repeatability in 20 minutes?

1. Pick one real buyer-style question in your category — not your brand name, the question a buyer would actually type.
2. Open five fresh chats on one engine (no prior context carried over), and ask the identical prompt in each.
3. Log, per run: which brands appeared, in what order, and whether yours was named, cited or recommended (see the difference between those three).
4. Compare the five lists. Count how many times your brand appeared (your rough visibility rate for this prompt) versus where it landed each time (your position — likely the more erratic of the two).
5. Repeat on a second engine if you have time. Expect the lists to diverge further, consistent with the 62% cross-platform disagreement rate above.

This is a directional read, not a study — five runs on one prompt won't match the precision of thousands of runs across a dozen prompts. Treat it as a fast way to see the variance for yourself, not as a number to report to a client as settled.

What does this mean for any "AI rank" number?

Treat it with real skepticism. If a tool reports "you rank #2 in ChatGPT for [category]" from a check that ran once, or even a handful of times, the evidence above says that specific rank is close to a coin flip on the next run. A more defensible version of the same claim states the run count and frames it as frequency: "named in 4 of 5 runs, appearing first in 2 of those." We go further into what a single bare score hides — the same caution applies to a bare rank number, for the same underlying reason: it hides exactly how many runs it's built from.

ContextOwl's own reports send each prompt once per engine with web search on — we call that a point-in-time snapshot for this reason, a strong read rather than a measurement to the decimal, and we're disclosing that limitation here rather than only in fine print.

FAQ

How many runs did the SparkToro and Gumshoe.ai study use? 600 volunteers ran 12 prompts across ChatGPT, Claude and Google's AI (AI Overviews and AI Mode), 60 to 100 times each per platform, for roughly 2,961 total runs over November–December 2025.

Is any engine more consistent than the others? Claude was slightly more likely to repeat the exact same brand list across runs, but even less likely to repeat that list in the same order — so "more consistent" depends on whether you mean the list or the order.

Does this mean AI visibility can't be measured at all? No — it means it needs to be measured as a rate across multiple runs, not read off one check. Frequency (how often you appear) is a stable, measurable thing; a single run's rank is not.

Should I run my own repeatability test before trusting any report, including ContextOwl's? It's a reasonable check on any vendor, including us. A quick five-run test on your own top buyer question will show you directly how much a single run can move.

What to do next

1. Run the 20-minute repeatability test above on one real buyer question before trusting any single "AI rank" check — yours or a vendor's.
2. If you're building a recurring report, state your run count every time, not just the result.
3. Read how to measure AI visibility for the full four-metric framework this variance data supports.

The AI Citation Snapshot runs one question once — useful as a first look, not as proof of a stable rank, for exactly the reasons above.