Running AI visibility experiments: change one thing, measure enough

You rewrote your product pages, added a buying guide, or got into a roundup. A month later your brand is named in more AI answers. Did the work do it? The honest reply is that a single before-and-after comparison cannot tell. Three things other than your change can move the count: ordinary noise from asking again, a change inside the assistant, and anything else on the web that moved during the same weeks.

This post is a design for finding out. It covers the setup, the statistics and a worked example whose numbers come from a script. The counts are made up for illustration and are not Dopbot data.

What is a fair test of a visibility change?

A fair test has six parts. Write all six down before collecting the first post-change answer.

1. A metric. The share of answers that recommend your brand, for a named set of prompts, on one named engine. Count "recommended", not "mentioned". Writing tracking prompts covers the prompt set.
2. A baseline. Enough answers before the change to know where you start. If you can afford two baselines a few days apart, compare them with each other first. The gap between two baselines of an unchanged brand is a sample of your noise.
3. One change. If you rewrite pages, add structured data and pitch a review site together, you will not know which one mattered, or whether any did.
4. A control. Prompts or products you left untouched, asked on the same days. A before-and-after comparison leaves this part out, and it is the part that separates your change from anything that moved every series at once.
5. A sample size and a threshold. Fixed in advance, as covered below.
6. A stop date. The date you will stop sampling and read the result, set before you look.

Which design controls for what?

Table: three designs and what each can and cannot tell apart.

If you sell several product lines, a staggered rollout is a strong design for a brand with several product lines. The lines you have not yet changed are a live control, sampled on the same days from the same engines.

How long should you wait after a change?

We found no published figure for this. As of October 2026, the closest documented number comes from search, not from assistants: Google's documentation says crawling can take anywhere from a few days to a few weeks after you ask for a recrawl. As of October 2026, neither OpenAI's nor Perplexity's documentation that we reviewed gives an equivalent for how fast a page change reaches an answer, and an answer can also lean on third-party pages you do not control.

So treat the lag as unknown. Sample at fixed intervals after the change, say in each of the next four weeks, and report each interval separately. Do not stop at the first interval that looks good, and do not stop because the first looks flat.

What do model updates do to an experiment?

They can move every prompt at once, which a control will show and a before-and-after comparison will not. As of October 2026, OpenAI's text guide says even different snapshots of one model family can produce different results, and Perplexity says to pin an explicit model ID and that pinning does not stop a retirement. Record the model name and the date with every answer, and if a model changes in the middle of a test, restart the clock with a new baseline. Why AI gives different answers lists the other sources of variation.

Which statistics do you need?

Two counts, before and after, in the same prompt set on the same engine: that is a comparison of two proportions. The exact tool is Fisher's exact test. It takes the 2 by 2 table of recommended and not recommended, before and after, and gives the probability of a gap at least as large if the underlying rate had not changed. In Python, scipy.stats.fisher_exact does it.

For the controlled comparison, subtract the control's change from the treated change, add up p(1-p)/n over the four cells (before and after, treated and control), take the standard error as sqrt of that sum, and divide the difference by it (a normal approximation, wider than a single change). The worked example below does this.

The Dopbot Index has one rule for change. As of October 2026 the test is not built yet, so the index lists no changes. Per its methodology, a change will be listed only when an exact test between two updates of the same size gives p below 0.01, with the counts shown. The methodology explains the strict line this way: the index looks at every brand in every category, and picking the largest changes from all of them would list the luckiest draws.

You can see why the stricter line matters for yourself. Run many comparisons and some will cross it by chance:

Table: upper bound on the probability that at least one of m independent tests crosses the line when nothing has changed anywhere. Exact tests on small counts run lower, but the pattern holds.

Test one brand and one prompt set, as planned, and 0.05 is a reasonable line. Test every prompt separately, or look at many brands, and correct for the number of tests, for example by dividing 0.05 by the number of comparisons. A p below 0.01 line is enough for about 5 comparisons, not for 20.

What does a worked example look like?

A brand adds a buying guide aimed at one use case. The prompt set for that use case gets 40 answers before and 40 after, on one engine, spread over several prompts. A control prompt set, for a use case the guide does not touch, also gets 40 answers before and 40 after. Two possible outcomes, with made-up counts:

Table: two made-up outcomes of the same design, 40 answers per cell, one engine.

Outcome A. The treated series rose 22.5 points, which looks good and has p of 0.069, short of both 0.05 and the index's 0.01. The control rose 20.0 points too. Something moved both series on that engine. This design cannot say what. Treated minus control is 2.5 points, with a 95% interval running from 26.8 points below zero to 31.8 points above, and p of 0.87. The data cannot show that the guide did anything, and with an interval this wide it cannot rule out a large effect either.

Outcome B. The treated series rose 35.0 points with p of 0.0033, which clears the index's line. The control barely moved. Treated minus control is 32.5 points, with a 95% interval of 4.4 to 60.6 points and p of 0.024. That is evidence that the treated series rose more than the control, but it does not clear 0.01, and the interval says the true effect could be as small as a few points or as large as 60.

The interval for the treated change alone, 13.1 to 52.5 points in outcome B, is narrower than the controlled one. That is the price of a control: the difference of two changes has an interval about 40% wider than one change, and the numbers above use the standard normal approximation for it. For a baseline share near 30%, the standard error of the controlled difference is 14.5 points at 40 answers per cell and 7.2 at 160. Halving it costs four times the answers.

What goes wrong in practice?

• Looking early and stopping when it looks good. This inflates false alarms well above the stated level. Fix the stop date.
• Bundling changes. You learn that something worked, and not what.
• Fixing the worst performer. A category or prompt picked because it scored lowest tends to rise on the next sample with no help at all. A control picked the same way tracks it.
• Reading the luckiest draw. With many prompts and brands, a few will jump. Pre-commit to the prompts that matter.
• Changing prompts mid-test. A new wording is a new series.
• Pooling engines. Test each separately. A rise on one engine and a flat result on another is two results.
• Treating repeated answers as independent. Asking the same prompt many times measures that prompt's variation. The intervals and p-values here assume independent answers, so for a claim about a topic, spread the sample over more prompts.
• Going from association to cause. Even outcome B shows a difference between series. It does not prove why. Keep a note of everything else you published or changed in the window.

What should you write down before you start?

A short plan, fixed before the first post-change answer:

• The change, in one sentence, and the date it went live.
• The engines, prompts, mode and location, and the model names if visible.
• Answers per cell for the treated and control series, and the dates of each sampling round after the change.
• The test, the threshold (0.05 for one planned comparison, 0.01 for a few, a correction for many) and the stop date.
• What you will do on each of three outcomes: a clear rise, no detectable change, a fall.

Report "no detectable change at this sample size" when the test is flat, together with the smallest rise that would have passed the test. A true rise of that size can still be missed, so say how often. The tables in how many answers you need before trusting a change give the smallest passing rise for 10, 20, 40 and 100 answers, with figures for how often a true rise is missed, and how to measure your brand's AI visibility gives the logging layout.

Frequently asked questions

Can I run an experiment with only 10 answers?

Only for an enormous effect. With 3 of 10 before, the later count has to reach 9 of 10 before a two-sided exact test gives p below 0.05. Below that, a flat result tells you nothing.

Should I test both engines together?

No. Test each engine on its own, and treat a rise on one and a flat result on the other as two separate findings. In the October 2026 edition of the Dopbot Index, which sampled OpenAI's API with web search (gpt-6-luna) and Perplexity's Agent API with the Sonar model (perplexity/sonar), the two engines' brand lists overlapped by 47% per category on average (Jaccard, 160 categories, excluding the alternative-to-a-brand question), so one engine's result says little about the other's.