'Our AI Visibility Score Is 72': Why One Number Misleads

What can a single score hide?

Almost everything that would let you judge whether the number means anything. "Our AI visibility score is 72" tells you nothing about which prompts were asked, which engines answered them, how many times each prompt ran, or when the check happened. A 72 built from one prompt run once on one engine and a 72 built from fifty prompts run five times each across four engines are not the same kind of claim, even though they produce an identical-looking number.

This matters more than it sounds like it should, because the underlying evidence on AI-answer variance is severe: a large study of repeated identical prompts found under a 1-in-100 chance of the same brand list appearing twice on the same engine (see how many runs you actually need). A score built from too few runs isn't measuring your visibility precisely — it's reporting one noisy sample and presenting it with false confidence.

What must any score disclose?

At minimum, four things, stated next to the number rather than buried in a methodology page nobody reads:

• The prompts. How many, and ideally what they were or what categories they covered.
• The engines. Which AI systems were asked — ChatGPT alone is a very different claim than ChatGPT, Gemini, Perplexity and Claude combined.
• The run count. How many times each prompt ran per engine. One run per prompt and five runs per prompt produce meaningfully different confidence, even from an identical score.
• The date. AI answers change as models and retrieval systems change. A score from six months ago is a historical data point, not a current status.

Here's what that disclosure looks like in practice — a score sitting directly next to the inputs it's built from, not standing alone:

Compare that to the opaque version of the same claim: a card or badge reading "AI Visibility Score: 72" with no competitor median, no per-engine breakdown, no prompt count, and no date — just the number, presented as if it settles the question on its own. Both can come from a real check. Only one of them gives you enough to judge whether to trust it.

What should a score measure?

Something built from the same four defined metrics this blog uses everywhere — visibility, share of voice, sentiment, position — not a new proprietary formula invented to make one vendor's number look more sophisticated than a competitor's. A score that's just visibility rate relabeled is at least transparent about what it is. A score blending undisclosed weightings of multiple signals into one figure is harder to audit and easier to game or misrepresent, intentionally or not.

A score relative to something is more useful than a score in isolation. "72 out of 100" invites the question "72 compared to what?" A score stated against a competitor median, or against your own prior measurement, gives a reader an actual reference point instead of an arbitrary-feeling absolute number.

When is a score still useful?

As a fast, relative read — not a replacement for the evidence underneath it. A single number is genuinely useful for a quick comparison ("ahead of or behind the competitor median"), for tracking your own trend over time (assuming the method stays fixed between checks), or as a headline figure in a report that then shows its work below. The problem was never the existence of a score; it's a score presented as if it were a precise, context-free measurement rather than a snapshot built from a specific, disclosed method.

We disclose our own position here plainly: ContextOwl publishes a score and letter grade as part of our reports, so we have a commercial reason to want scores to seem credible and useful. That's exactly why we hold ours to the disclosure standard argued for above — prompts, engines, run count and date stated next to the number, every time — rather than asking you to take the figure on faith.

How should buyers test a vendor's score?

Ask the vendor directly for the four disclosures above, before trusting the number: how many prompts, which engines, how many runs per prompt, and when the check ran. If a vendor can't or won't answer those four questions about their own score, treat the number as marketing, not measurement — regardless of how precise it looks. A second useful test: ask whether the same prompt set, run again today, would produce a materially different score. If the honest answer is "yes, somewhat," that's not a flaw to hide — it's the kind of disclosure this whole argument is asking for.

FAQ

Is a single AI visibility score always meaningless? Not always — it's meaningless without disclosed method. The same number can be a reasonable quick read or close to meaningless noise, depending entirely on what's underneath it.

What's the minimum disclosure a score needs to be trustworthy? Prompt count, engines used, runs per prompt, and the date the check ran, stated next to the number — not buried in a separate methodology page.

Does ContextOwl's own score meet this standard? We publish prompts, engines, named brands and per-prompt evidence alongside our score and grade, and we disclose that it's a point-in-time snapshot from one run per engine per prompt — see the sample report for the full disclosure.

Why shouldn't I just compare scores across different vendors? Because "AI visibility score" isn't standardized — two vendors' scores can be built from completely different prompt sets, engines and run counts and still look like comparable numbers. Compare methods before comparing scores.

What to do next

1. Before trusting any AI visibility score — including ours — ask for the four disclosures: prompts, engines, run count, date.
2. If you're publishing your own score internally or to a client, attach those four disclosures to the number every time you report it.
3. Treat the score as a headline, not the whole report — read the evidence underneath it before acting on the number alone.

See a fully disclosed score, with the prompts and per-engine evidence behind it, in the sample report.