Why AI gives different answers to the same shopping question
Six sources of variation behind different brands for the same question, what October 2026 Dopbot data shows about short answer lists, and how to measure.
Ask an assistant "what is the best mattress to buy right now?" twice and you may get two different lists. That is not a bug report. It is how these systems work, and it decides how you should measure your brand. If a single answer is one draw from a spread of possible answers, then no single answer, yours or a customer's screenshot, is the answer.
This post separates the causes that the providers document from the ones people assume, shows what the October 2026 Dopbot Index can and cannot say about variation, and ends with a procedure for telling noise from a real change.
Where does the variation come from?
Six sources of variation follow; five are stated in the providers' documentation and the live-search row is an inference from it. Each row says what the documentation states and what you can hold constant when you measure.
| Source | What the documentation says | What you can hold constant |
|---|---|---|
| Generation | OpenAI's text guide calls model output non-deterministic | Nothing. Repeat the question instead |
| Live search | OpenAI says web search gives models up-to-date information from the internet; that results change as the web changes is our inference, not a documented measurement | Ask everything on the same day |
| Model versions | OpenAI says even different snapshots of one model family can produce different results; Perplexity says to pin an explicit model ID and that retirements are announced in its changelog | Record the model and date; use one tool version |
| Personalization | ChatGPT's temporary chats can be personalized or unpersonalized; Perplexity's memory remembers details between conversations | Turn memory and personalization off where the product allows it |
| Location | OpenAI's web search accepts an approximate user location; Perplexity's Agent API web search takes a user location (as of October 2026) | Fix the location and write it down |
| Product | Perplexity says its API and its app share a search system, with different configuration, so outputs may differ | Measure one product consistently |
Table: sources of variation, as of October 2026. Five rows come from the providers' own pages; the live-search row is an inference.
The six fall into three kinds. Generation is non-deterministic, so repeated runs can differ even when nothing else changes. Search results can move as the web moves. Model versions, accounts, location and product are conditions: they differ between two askers, or between two dates, rather than between two runs.
That split matters for measurement. You can average away the first by repeating. You can only remove the others by controlling the conditions or by recording them.
What does the October 2026 data show?
Start with what it does not show. The October 2026 edition asked each question once on each engine, so it has no repeated answers to one question and cannot say how much the same prompt varies from run to run. From November 2026, 40 US categories are re-sampled monthly with each question asked 4 times per engine, which will allow a direct measure; this post uses only the October data and reports no repeat-variation number.
What the October data can show is the shape of the answers that all this variation acts on.
Shopping answers are short lists
Excluding the alternative-to-a-brand question (1,271 answers), an answer recommended 2.3 brands on average, with a median of 2. OpenAI's API with web search (gpt-6-luna) averaged 2.2 brands over 640 answers. Perplexity's Agent API with the Sonar model (perplexity/sonar) averaged 2.5 over 631 answers.
A list of two brands is fragile in a simple arithmetic way. Take two answers to the same question that agree on all but one brand. The brand overlap, measured the way the index measures overlap (shared brands divided by brands in either), looks like this:
| Brands in each answer | Brands that differ | Shared brands | Overlap |
|---|---|---|---|
| 2 | 1 of 2 | 1 | 33.3% |
| 3 | 1 of 3 | 2 | 50.0% |
| 5 | 1 of 5 | 4 | 66.7% |
Table: arithmetic, not data. Two answers that differ by one swapped brand, at three list lengths.
At two brands, a single swap leaves two answers sharing a third of their union. So on a two-brand list, one swapped brand is enough to bring the overlap down to a third. A low overlap between two runs does not by itself show a wholesale change in the answer.
Two engines, same categories, about half the brands
The two engines, asked the same questions about the same categories, shared 47.0% of their recommended brands per category on average (excluding the alternative question, 160 categories; Jaccard, each category weighted equally). A typical category, about 8 answers excluding the alternative question, had 3.9 brands recommended by both engines and 5.3 by only one. That comparison sets two different systems with different search tools against each other, so it is not a measure of run-to-run noise. It is a reminder that "the same question" has no single answer across products. OpenAI vs Perplexity defines it fully.
Budget and premium answers share few brands
Within one category, the budget answers and the premium answers shared few brands. Per category, the brands in the budget answers and the premium answers (both engines together) overlapped by 8.2% on average (brands in both lists divided by brands in either, averaged over categories with equal weight; 155 of 160 categories, usually 2 answers each). That is a different question, not a repeated one, and the figure is observed overlap, not a measured effect of the wording. Budget vs premium goes further.
A category's answers rarely repeat each other
Across a category's roughly 8 answers (7.9 on average, excluding the alternative question), the index found 9.2 distinct recommended brands, and 48.4% of a category's brands were named in only one answer. These come from four different questions, asked of two engines, so they mix repeat noise with question wording and engine differences. They do show that at this sample size nearly half of a category's brands appear in a single answer, so a count of one says very little.
How do you tell noise from a real change?
Treat variation as three layers, and test them in this order.
- Noise. Ask one question many times, in identical, fresh sessions on one day. How often does your brand appear? That rate, with its interval, is your baseline.
- Conditions. Change one condition at a time (location, signed in or out, app or API) and repeat the same count. A difference that passes the test in the sample-size post is evidence of a condition effect.
- Drift. Repeat the same noise test a month later. A difference that passes the same test, with the conditions unchanged, is evidence of drift: the model, the web or your own pages may have moved.
Here is what the first layer looks like with made-up numbers. One shopping question is asked 8 times in fresh sessions. Brand A is recommended in 4 of the 8.
| Brand A in the 8 answers | Count | Share | 95% interval |
|---|---|---|---|
| Recommended | 4 of 8 | 50.0% | 21.5% to 78.5% |
Table: a made-up noise test. The interval says the underlying rate for this question could plausibly be anywhere from about a fifth to about four fifths.
That is the honest picture of a small repeat test. It also shows why the next step, comparing it with another condition, needs far more than 8 answers. How many answers you need before trusting a change gives the thresholds.
What should you do about it when measuring?
- Repeat each question. The index's monthly update of 40 US categories asks every prompt 4 times per engine. A manual version of 20 answers per assistant is described in how to measure your brand's AI visibility.
- Keep each engine separate. Do not average your brand's share across engines.
- Log the conditions: date, location, product, mode, model name when you can see it.
- Collect in a short window, so that search results and model versions are as constant as you can make them.
- Use counts with intervals, not single percentages, and say how many answers each rests on.
- Do not change the wording between rounds. A new wording is a new series.
- Expect a model change to move many prompts at once. If every question moves the same way on one engine on the same date, suspect a change in that engine (its model or its search) before your own work, and check the provider's changelog.
What does it mean for a brand?
Do not react to a single answer, favorable or not. A screenshot from one customer is one draw from a spread. Judge visibility by rates over repeated, fixed questions, and judge a change only when the counts clear a stated test, as in the sample-size post. The variation is also a reason to care about breadth: a brand appearing in many kinds of questions and in both engines is less exposed to one lucky or unlucky run than a brand that appears in one place.
Frequently asked questions
Is a different answer a wrong answer?
Not necessarily. Two different lists can both be reasonable. Variation changes which brands are named, and the facts about each brand can still be right or wrong independently. Check the facts about your brand in each answer, separately from whether it was listed.
How many repeats before I believe my brand is part of the usual answer?
There is no fixed number, because the answer is a rate with a margin. Eight repeats leave a wide margin, as the example above shows. Twenty per assistant gives a usable picture, and detecting a change takes more, as covered in the sample-size post.
How we write: drafted with AI tools from Dopbot Index data and public sources; every number and outside fact is traced to its source in a claims record and checked before publishing. More about how we write; corrections to hello@dopbot.com.