How to Test SEO and GEO Changes: A Simple Experiment Framework

Why do most GEO claims fail as evidence?

A testable GEO or SEO experiment needs four things: a hypothesis stated before you look at results, a design that holds everything but one variable constant, enough runs to say something beyond "it happened once," and a write-up that discloses exactly what was and wasn't controlled. Most posts claiming to have "tested" a tactic are missing at least one of the four.

Because the thing being measured — an AI engine's answer — is itself noisy by design. SparkToro and Gumshoe.ai ran 2,961 prompts across ChatGPT, Claude and Google's AI using 600 volunteers over November and December 2025: the same brand list came back less than 1% of the time across repeated identical prompts, and the same list in the same order came back less than 0.1% of the time. Against that backdrop, a single before/after screenshot — "I added schema, then it got cited" — can't distinguish a real effect from the normal run-to-run churn the study above documents. See how many runs you actually need for the full run-count picture this method leans on. The missing ingredient in almost every viral "I tested X" post is a comparison: what happened to a similar page that didn't get the change, over the same period, with the same number of runs — the same gap that makes several popular GEO tactics look overhyped once you check what was actually tested.

How do you write a testable hypothesis?

State, before you run anything, what you expect and why — in a form that could turn out false. "Adding FAQ schema increases how often this page's brand is named in a 20-run prompt set, compared to a matched page without it, over the same two-week window" is testable. "Schema helps AI search" is not — it doesn't say what would count as it not helping. Write the hypothesis down and date it before looking at a single result; changing the claim after seeing the data isn't a finding, it's a story fitted to noise.

How do you match pages and hold variables constant?

Two designs cover most cases:

• Matched-page pairs (your own site). Pick two pages as similar as possible in topic, length, age and existing visibility — ideally the same template — and change exactly one thing on one of them. Run the same prompt set against both, on the same days, through the same engines. The comparison page is your control; without one, you can't tell your change from ordinary drift.
• Observational design (third-party or industry-wide signals). When you can't control the variable directly — testing whether backlinks matter, say — compare groups of existing pages that already differ on that variable, and report it as correlational, not causal. This is the Ahrefs 75,000-brand study's own design: real, useful, but explicitly not proof that changing one factor changes the outcome, as the study's own authors say directly ("correlation isn't causation").

Either way, log what you didn't control for. A matched-page test run during a week your brand also got a different, unrelated press mention isn't clean, and saying so in the write-up is part of the method, not an afterthought.

How many runs and prompts are enough?

More than one, always. Beyond that, there's no single magic number — but the variance data above sets a floor: if the same prompt returns a different brand list more than 99% of the time on its own, a two-run "before and after" comparison can't separate a real effect from ordinary noise. A reasonable minimum for a directional (not definitive) result is five runs per prompt per condition, across more than one day; ten or more per condition is stronger. Call anything below ten per condition "directional" in the write-up, not "confirmed" — the difference in wording is the honesty the method requires.

What must every write-up disclose?

A method box, every time, with:

• The exact prompts used, verbatim.
• The engines and dates of every run.
• The number of runs per prompt per condition, and whether that number is enough to call the result more than directional.
• How "named," "cited" and "recommended" were counted — these are three different things, and a write-up that blends them is less useful than one that keeps them separate.
• What wasn't controlled for — anything that changed on either page or in the wider context during the test window.
• The conclusion's actual scope — "this held for this page, this prompt set, this engine, this week," not "this works."

A downloadable experiment design template

Copy this structure for any GEO or SEO test before you run it:

Hypothesis: [one sentence, stated before any run, falsifiable]
Design: [matched-page pair / observational — name which]
Variable changed: [exactly one thing]
Held constant: [everything else you controlled for]
Pages/sample: [URLs or dataset, with size]
Prompts: [exact text, verbatim]
Engines: [named, with dates]
Runs per condition: [number — flag "directional" if under 10]
Counting rule: [how named/cited/recommended were each counted]
Result: [what actually happened, including a messy or null result]
Not controlled for: [honest list]
Conclusion scope: [exactly what this does and doesn't generalize to]

Fill in every line before publishing. A blank "not controlled for" line is a red flag, not a clean result — it usually means nobody checked.

A worked example, filled in

Hypothesis: Adding a 40-word direct-answer sentence under the H1 increases
how often a page's brand is named in a 10-prompt, single-engine test,
compared to a matched page without one, over the same 2-week window.
Design: Matched-page pair (own site)
Variable changed: Added one direct-answer sentence to Page A; Page B unchanged
Held constant: Same template, same publish date, same internal links,
same prompt set, same engine, same two-week window
Pages/sample: 2 pages (fictional: /tools/widget-a, /tools/widget-b)
Prompts: "what is the best [category] tool for a 5-person team"
(same single prompt, run repeatedly)
Engines: ChatGPT, web search on, logged out — 8–22 October 2026
Runs per condition: 8 (directional — below the 10-run bar for a stronger result)
Counting rule: "Named" = brand appears anywhere in the answer text,
regardless of citation or recommendation status
Result: Page A named in 5 of 8 runs; Page B named in 2 of 8 runs
Not controlled for: Page A also got one new external backlink mid-window;
can't fully separate that from the sentence change
Conclusion scope: Directional signal for this one page pair, one prompt,
one engine, two weeks — not a confirmed, generalizable finding

That write-up is honest about its own limits, which is exactly what makes it usable: a reader knows precisely what it does and doesn't support.

FAQ

Can a small team run a real test without special tools? Yes. A spreadsheet, a prompt list, and a shared doc for logging each run's raw answer covers the matched-page design above. The hard part is discipline — writing the hypothesis down first and running enough repeats — not tooling.

Is five runs ever enough to publish anything? Yes, if it's labeled directional and the write-up says so plainly. A directional result with a clear method is more useful than a confident claim with none.

What's the single most common mistake in a GEO "test"? No control. A page changed, something happened, and there's no matched comparison page or prior baseline to know whether the same thing would have happened anyway.

Does this apply to classic SEO tests too, not just GEO ones? Yes — titles, meta descriptions and internal-link tests follow the same logic: one variable, a real comparison, enough repeats, and a disclosed scope. AI-answer variance just makes the discipline harder to skip, because the noise is larger and more obvious.

What to do next

1. Pick one change you're considering and write its hypothesis down, in falsifiable form, before making the change.
2. Use the template above to plan the comparison — matched pages if you can control the variable, an observational design if you can't.
3. Run at least five repeats per condition, across more than one day, and disclose the run count and what wasn't controlled for in whatever you publish.
4. Check the tactic you're testing against the full SEO, AEO and GEO checklist first — if it's already rated Strong evidence there, a test may confirm what's known rather than break new ground, which is still useful, just worth knowing going in.

ContextOwl sells one-time AI visibility reports. The AI Citation Snapshot runs one real prompt through one engine live — useful as a single data point for a test like this, not as the test itself.