How many AI answers you need before trusting a change

Suppose your brand was recommended in 3 of 10 AI answers last month and in 5 of 10 this month. Two more recommendations feel like movement. The useful question is how often two samples of that size would differ by that much if nothing had changed, and for 10 answers the honest reply is: most of the time.

This post gives the numbers for 10, 20, 40 and 100 answers per period, so you can decide how much sampling a claim needs before you make it. Every figure below was computed with a script, and the counts are made up for illustration. They are not Dopbot data.

When can you trust a change in recommendations?

You can trust a change when it is larger than what repeated sampling of an unchanged system would usually produce. A real change can still go undetected when there are few answers, so a change you cannot detect is not proof that nothing changed. You cannot see the sampling noise directly, so you use two tools.

• An interval for each share. The 95% Wilson interval gives the range of underlying rates consistent with a count. Unlike the plain plus-or-minus formula, it stays between 0% and 100% and behaves sensibly with small counts, and it is the interval the methodology uses. The recipe is in how to measure your brand's AI visibility.
• An exact test for the gap. Fisher's exact test takes the two counts and asks: if the underlying rate were identical in both periods, how likely is a gap at least this large? That probability is the p-value. A small p means the gap is unlikely under "nothing changed". It does not say how large the effect is, and it does not say why it happened.

Both assume each answer is an independent draw. They are not quite that, a point returned to near the end.

Is a rise from 3 to 5 recommendations real?

No, and more answers do not rescue it. The table keeps the change at two recommendations and grows the sample.

Table: made-up counts, one engine, same prompts in both periods. A p near 0.7 means that if nothing had changed, a gap at least this large would turn up about 7 times in 10, given the same total of recommendations across the two periods.

The p-value creeps up as the sample grows, because the same two extra recommendations are a smaller share of a bigger denominator. A bigger sample helps only when the share gap itself holds up, which is the next table.

What if the share really moved from 30% to 50%?

Keep the shares fixed and change only the number of answers.

Table: made-up counts with a 20-point rise in share at four sample sizes, one engine.

Only the 100-answer row clears p below 0.01, the line the index says it will require before it lists a change (described below). That is the result that clears the line, not the sample you need: at 100 answers a true move from 30% to 50% is flagged only about 58% of the time at p below 0.01. The power table below gives the sample for 80%.

At 40 answers per period, the same 20-point rise gives p of 0.110. That is not one engine's view of an index category, though. From November 2, 2026, a monthly update of one of the 40 US categories that are still updated has 40 answers across both engines, 20 per engine (the October 2026 edition has about 10 answers per category), so one engine's view of one category sits on the 20-answer row, at p of 0.333.

Read the intervals too. At 100 answers the two intervals do not touch. At 40 they overlap by about 10 points. Overlap alone does not rule out a real change, because the exact test is sharper than eyeballing two intervals, but the test agrees here.

How big a jump do you need at each sample size?

Turn the question around. Start from a baseline of 30% and find the smallest later count whose exact test passes.

Table: from a baseline of 30%, the smallest later count that passes each threshold, one engine.

With 10 answers a month, a brand at 3 of 10 has to reach 9 of 10 before the test calls it a change at p below 0.05. With 40 answers it has to go from 12 to 22. This is why small-sample dashboards that flag every wiggle are producing noise, and why a "no change" from a small sample means "no change we could detect".

How many answers catch a change that is really there?

The tables above ask what a result must look like. Power asks the planning question: if the share truly moves by a given amount, how large must each period's sample be for the test to flag it 4 times in 5 (80% power)?

Table: the first sample size per period at which exact-test power reaches 80%, computed exactly for two independent samples of equal size. The search stopped at 400 answers.

At 40 answers per period, a true move from 30% to 50% is flagged 36.8% of the time at p below 0.05 and 16.0% of the time at p below 0.01. The rest of the time you would see a real change and call it noise. Meanwhile, if nothing changes and the true rate stays at 30%, 40 answers per period flags a false change in 2.8% of repeats at p below 0.05 and 0.5% at p below 0.01. The exact test is conservative, so it errs toward silence.

Smaller changes cost far more: from a 30% baseline, a 10-point rise (375 answers per period at p below 0.05) needs about 3.7 times the answers of a 20-point rise (102). For most brands the realistic plan is to look for large moves, or to accept a long time window.

What does the Dopbot Index do with changes?

It does not show any yet. The methodology says the index has no change column, no list of biggest movers and no trend chart. It also says that updates of the new size have 40 answers per category, above the minimum of 30 answers that a change needs, and that changes will come back only when an exact test between two updates of the same size gives p below 0.01, with the counts shown. That test is not built yet.

If the index uses a two-sided Fisher exact test, a rise from 12 of 40 answers to anything below 25 of 40 would not clear p below 0.01, so it would not be listed as a change. That is deliberate. The methodology gives the reason: picking the largest changes out of every brand in every category would still list the luckiest draws.

Which cautions apply to every number here?

• Answers are not independent shoppers. From the November 2, 2026 update, each of the 40 US categories updated monthly asks the same 5 prompts 4 times on each engine (the October 2026 edition asked each prompt once per engine). Repeating one prompt measures how much that prompt's answers vary, not how shoppers who word the question differently would be served. For the same prompts in both periods, the intervals above describe that repeat-to-repeat variation; if you want to say something about a topic rather than these prompts, they are a floor on the uncertainty. To say something about a topic rather than a prompt, add prompts before you add repeats.
• One engine at a time. The engines differ in what they return, as OpenAI vs Perplexity shows. A rise on one and a fall on the other should not be netted into a single number.
• Same questions, same sample size. If either changed between periods, the comparison is not like for like. The methodology makes the same point about the move from the October 2026 edition to the monthly update.
• Many comparisons. Test 20 brands or prompts at p below 0.05 and, if the tests were independent, the chance of at least one false alarm is at most 64.2% (the ceiling if each test's false-alarm rate were exactly 5%; with Fisher's exact test at 40 answers per period it is about 44%).
• Set the sample size first. Adding answers until a result crosses the line raises the false-alarm rate well above its stated level.
• The engine can change under you. A shift that appears in every prompt on one engine on the same date is more likely to come from the engine or its search than from your work, so check other brands before crediting a change.

How should you report a change?

Give the counts, the denominator, the engine, the dates and the prompt set, then the test: "Perplexity, 5 prompts, 20 answers per period, made-up counts: 6 of 20 in period 1, 14 of 20 in period 2, p = 0.026 (Fisher exact)." Add the smallest change your sample could have detected. When the result is not significant, say "no detectable change at this sample size", not "no change".

Frequently asked questions

Should I use p below 0.05 or p below 0.01?

For one comparison you planned in advance, 0.05 is the common convention. When you compare many brands, prompts or engines, or intend to publish the claim, a stricter line such as the 0.01 the index plans to require keeps false alarms down, at the price of missing more real changes.

Which tool computes Fisher's exact test?

Most statistics packages do. In Python it is the fisher_exact function in scipy.stats, which takes a 2 by 2 table of counts: answers that recommend the brand and answers that do not, in the first period, then in the second. Free online calculators do the same. Your spreadsheet only needs the counts, as laid out in how to measure your brand's AI visibility.

Can I keep sampling until the result looks significant?

Not if you want the p-value to mean what it says. Stopping when the test first crosses the line inflates false alarms. Fix the number of answers in advance, or use a method built for repeated looks, which is beyond this post.