How to Build an AI Visibility Benchmark Against Competitors

What does "benchmarking AI visibility" actually mean?

It means asking the same question about your brand and your named competitors, the same way, enough times to see a pattern — then writing down exactly how you did it so a later comparison means something. The output isn't a single score. It's a short, repeatable method plus a log of what happened when you ran it.

This matters because an AI visibility number without its method attached isn't comparable to anything, including your own number from last quarter. The rest of this guide is the method, step by step, with a worksheet at the end you can copy directly.

Step 1: Which competitors belong in the set?

Three to five real names, chosen the same way a buyer would narrow a shortlist — not a list of every company technically in your category. Two filters that work:

• Ask who you actually lose deals to, or who comes up unprompted when a prospect says "we're also looking at...". That's your real competitive set, independent of who ranks well on Google.
• Avoid aspirational names. Including a company ten times your size because you'd like to be compared to them tells you nothing useful — their visibility numbers reflect a different scale of brand awareness, not a comparable fight for the same buyer's attention.

Keep the set stable once you've picked it. Swapping competitors in and out between runs breaks the comparison the same way changing the prompts does — see Step 5.

Step 2: How do you keep the comparison fair?

Same prompt wording, same engine, same day, for every brand in the set — no exceptions. The biggest way benchmarks go wrong isn't the metric, it's letting the comparison quietly stop being apples-to-apples. Concretely:

• Write each prompt once, in neutral, buyer-style language, with no brand name in it. "What's a good project management tool for a 10-person team?" not "Is [Your Brand] good for a 10-person team?"
• Run the identical prompt text for every brand's slice of the analysis — you're not writing a different prompt per competitor, you're running one prompt and checking whether each brand in the set gets named, cited or recommended in that single answer.
• Run every prompt on the same engine, in a fresh session each time, so earlier context in a chat doesn't bias later answers toward brands already discussed.
• Log the exact date. Answers are time-sensitive; a comparison spanning several days of runs should still note the date range, not just "October 2026."

Step 3: How many prompts, and how many runs?

Start with 8–10 prompts that represent real buyer questions (a mix of "best X," "X vs Y," and problem-based phrasing), and run each one at least 3–5 times before drawing a conclusion. This isn't an arbitrary minimum — it's a direct response to how much a single run can mislead you.

SparkToro and Gumshoe.ai's study — 600 volunteers, 12 prompts, nearly 3,000 runs across ChatGPT, Claude and Google's AI, run November–December 2025 — found under a 1-in-100 chance of getting the same brand list twice on a repeated prompt, and under a 1-in-1,000 chance of the same list in the same order. But frequency held up better than exact ordering: for the headphones prompts specifically, the same handful of brands appeared in roughly 55% to 77% of the 994 responses, with the single most-recommended brand showing up in 90–100% of them, even as the exact list and order kept changing. That's the pattern a benchmark is trying to catch — not a single list, but how often each brand shows up across enough tries to see the real rate, not noise from one lucky or unlucky run.

More runs is better, but 3–5 per prompt is a workable floor for a first pass, scaled up later if a particular comparison matters enough to invest in it.

Step 4: How do you compute visibility and share of voice from the log?

Once the runs are done, two numbers per brand, from the same run log:

• Visibility rate = (number of runs where the brand was named) ÷ (total runs), as a percentage.
• Share of voice = (that brand's total mentions) ÷ (total mentions across every brand named, across every run), as a percentage.

Compute both for your brand and for every named competitor, from the exact same set of runs. Share of voice in AI answers: how to calculate it walks through the counting rules in more depth if a run names a brand more than once in the same answer.

Step 5: How do you summarise it for a boss?

One chart, one sentence, nothing else on the first page. The chart: visibility rate, your brand next to each named competitor, same prompt set, same date. The sentence: the single biggest gap, stated plainly — "We're named in 30% of these runs; our top two competitors are named in 65% and 80%." A one-page AI visibility report for a CMO or client covers the full layout, including where the fix list and next step go on the same page.

Resist the urge to hand over the raw run log as the main deliverable — it's the evidence behind the summary, not the summary itself.

Step 6: How do you compare a later run to the first one?

Only by holding the method identical: same prompt wording, same competitor set, same engine. If any of those changed, you're not tracking movement — you're comparing two different measurements and calling the difference progress or decline when it might just be the method shifting underneath you.

Log the method alongside every run, not just the numbers: prompt list, engines, date, run count, competitor set. When you re-run the benchmark, diff that method block before touching the results — if it's identical, the comparison is fair; if anything changed, note it before presenting the new numbers next to the old ones.

A worked example, run live for this post

AI web search (Claude, web search enabled), prompt: "best email marketing tool for small business," 10 October 2026 — one run. The answer named six brands: Constant Contact, MailerLite, ActiveCampaign, Omnisend, EmailOctopus and Klaviyo, with Constant Contact's own blog cited twice among the top sources.

Evidence: Constant Contact and MailerLite were positioned as the default starting points ("best starting points" for most small businesses under 5,000 contacts); ActiveCampaign was named specifically for automation, Omnisend for e-commerce, EmailOctopus for budget-conscious buyers. Two of the top-cited sources were Constant Contact's own blog — a self-citation, not an independent third party. Outputs vary between runs; this is one AI web search on one day, not a benchmark. Factual corrections: team@contextowl.com.

This is exactly what Step 2 of a real benchmark would log for one brand on one run — it isn't a benchmark by itself, since it's a single run of one prompt on one day. A real benchmark repeats this same prompt, this same way, 3–5 times, across the brand you're tracking and its named rivals, before drawing any conclusion from it.

The benchmark worksheet

Copy this directly. One row per run.

Summary roll-up (fill in after all runs for one prompt set):

Method block to log with every benchmark:

• Prompts used (exact wording):
• Engine(s):
• Runs per prompt:
• Competitor set:
• Date range:

Common mistakes

• Changing the prompt per brand. Asking "is [Your Brand] good for X" for yourself and "what's the best tool for X" for competitors isn't the same test.
• One run, treated as settled. A single run is a screenshot of one moment, not a measurement — see how many runs you actually need for the full case.
• Swapping the competitor set between benchmarks. If Competitor B is dropped and Competitor C added next quarter, that's a new benchmark, not a continuation of the old one.
• Reporting the raw log instead of the summary. A boss or client needs the chart and the sentence, not fifteen rows of prompts and dates.

Tools to use

• A spreadsheet (the worksheet above, copied in) for logging runs by hand across any engine you have access to.
• Competitor AI Comparison for a fast, free, single-prompt check between your brand and one named rival, as a starting point before a fuller multi-prompt benchmark.
• How to measure AI visibility for the shared definitions of all four metrics this worksheet uses.

FAQ

How many competitors should be in a benchmark? Three to five real ones your buyers actually compare you to. More than that dilutes attention across runs you'd rather spend on repeats of the same comparison.

Can I benchmark against competitors on more than one engine? Yes, and it's worth doing — just keep each engine's results separate rather than averaging across engines, since they source and answer differently.

What's a "good" result from a benchmark? There's no universal good number — see what is a good AI visibility rate for why the honest comparison is against your own competitor set, not a borrowed target.

Do I need special software to run a benchmark? No. A spreadsheet and the worksheet above are enough for a manual benchmark; a paid report automates the run-and-log step across more prompts and engines at once.

What to do next

1. Pick three to five real named competitors and write 8–10 neutral, buyer-style prompts.
2. Run each prompt 3–5 times per brand, on the same engine, logging named/cited/recommended/position in the worksheet above.
3. Compute visibility rate and share of voice per brand, summarise in one chart and one sentence, and save the method block for next quarter's comparison.

ContextOwl sells one-time AI visibility reports. Competitor AI Comparison is a free way to run one prompt against a named rival right now.