API answers vs app answers: what sampling can and cannot tell you

Any AI visibility number answers a narrower question than it first seems to. "Recommended in 40% of answers" means 40% of the answers that one tool collected, in one way, from one surface. This post sets out what is documented about the gap, what the Dopbot Index does and does not see, and a spot check you can run on any tracker, ours included.

In this post, "OpenAI" means OpenAI's API with web search (gpt-6-luna), and "Perplexity" means Perplexity's Agent API with the Sonar model (perplexity/sonar) and its web search tool turned on. These are the two engines behind the October 2026 edition of the index. Perplexity's documentation says Sonar chat completions support ended on September 27, 2026; the index did not use that endpoint. Product features described below are as of October 2026 and change often.

Do API answers match what people see in the apps?

No. Perplexity says its API and web interface can differ. OpenAI's help pages describe what the ChatGPT app does around a search (search triggers, location, memory, product results), and none of them says an API call does the same.

Perplexity's FAQ states that its API uses the same search system as its web interface, with differences in configuration, so outputs may differ. It adds that the underlying model might differ between the API and the web interface for a given query. Another passage says the two share search infrastructure while their tools, configuration and models differ.

OpenAI's silence is not a finding that its API answers differ; it only means there is no documented guarantee that they match. The table lists the points that matter for shopping questions, next to what the index does.

Table: documented differences between how the October 2026 index samples answers and how ChatGPT and Perplexity are described in OpenAI's help pages and Perplexity's help center and documentation. The index column is from the methodology.

Two consequences follow. First, a tracker built on an API measures a configuration that someone chose: a model, a tool setting, a location. The apps choose differently, and choose differently for different people. Second, neither side is "the truth". The app answer a shopper sees depends on who they are and where they are; the API answer depends on the settings of whoever asks.

What does the Dopbot Index sample, exactly?

The October 2026 edition attempted 1,600 answers and collected 1,587: 800 from OpenAI and 787 from Perplexity, with 13 Perplexity answers failing. Each category got five prompts (best overall, budget, premium, a specific use, and an alternative to a named brand), each asked once per engine, which is about 10 answers per category. From the US run of November 2, 2026, 40 US categories are re-sampled monthly at 40 answers each; the other categories stay frozen as the October edition.

What that setup holds constant: the prompt wording, web search on, the country of the market, no account and no memory, one model per engine. What it leaves out:

• Personalization. Real users have memory, history and device location. Our requests have none, so the index shows a generic shopper in a country, not anyone in particular.
• The app's product interface. Product options, images, links and ads are not recorded. Only the text answer and its source list are.
• Which pages shaped the answer. OpenAI's source list is the pages its answer links. Perplexity's is the pages its search retrieved, which the API does not tie to the answer text. OpenAI's own documentation notes that sources, the full list of URLs consulted, is often longer than the list of citations. So the OpenAI lists in the index leave out pages the search consulted but the answer does not link. A domain in a list is presence, not influence.
• Google. Google's AI features are not sampled at all.

What does the index show about how far apart two assistants are?

We did not measure the gap between an API and its own app. We did measure how far apart two engines are when asked the same questions. Per category, brands recommended by both engines divided by brands recommended by either averaged 47.0% over 160 categories, excluding the alternative-to-a-brand question. That compares two engines, not two surfaces of one engine, and it shows that the two engines' brand lists often differ. It does not show whether a changed surface would change the shortlist; we did not measure that. OpenAI vs Perplexity defines the measure and breaks it down by market.

What can sampling show, and what can it not?

Sampling can show:

• How often a brand is recommended for a fixed set of questions under fixed settings, with a count and an interval.
• Whether one brand is recommended clearly more often than another, once there are enough answers to see past noise. At 10 answers, one answer moves a share by 10 points; at 40 answers, by 2.5 points. How many answers you need works through the intervals.
• Which source domains appear in each engine's lists, engine by engine.
• Change over time, but only when the setup, the prompts and the sample size stay the same.

Sampling cannot show:

• What a particular person sees, with their memory, location and settings.
• How many shoppers ask a question, or how they phrase it. We sampled answers to prompts we wrote. We did not observe shoppers or their searches.
• What a product card, an ad or a carousel shows.
• Whether a recommendation led to a click or a sale.
• Why an answer recommended a brand.

How do you check any API tracker against the apps?

You cannot reproduce a shopper, but you can find out whether a tracker's shortlist is close to what an app returns with personalization turned down. This takes about an hour for a handful of prompts.

1. Pick 5 to 8 prompts the tracker already runs, spread across question types: a best-overall, a budget, a premium and a use-case question.
2. Remove personal context in the app. In ChatGPT, start a temporary chat and choose not to use existing memories or custom instructions; temporary chats do not create new memories. In Perplexity, memory and past searches are always off in incognito mode. If the app offers a location setting, match the tracker's country.
3. Ask each prompt at least 4 times, each in a new conversation, and record the brands each answer presents as a pick, exactly as the tracker's rules count them.
4. Compare brand counts. For each brand that matters to you, compute the share and its interval in the app sample and in the tracker's sample.
5. Compare shortlists too. For each prompt, note how many of the app's brands also appear in the tracker's answers for that prompt, and the reverse.
6. Record what you could not control: app version, plan, time of day, whether you were signed in.

What the comparison can say depends on the counts. This worked example uses made-up numbers: the tracker shows your brand in 16 of 40 answers (26.3% to 55.4%).

Table: illustrative counts only. Non-overlapping intervals are a rough, conservative test.

Ten app answers will rarely settle anything: even a complete miss, 0 of 10, still overlaps. Against this tracker figure, 11 app answers with no recommendation is the smallest sample that separates (the upper end falls to 25.9%), and a single recommendation needs 18 answers. If the intervals overlap, this rough check finds no clear gap, which is not proof that the two agree: two intervals can overlap even when an exact test on the two counts would show a difference, as how many answers you need explains. Use the tracker for the brand and prompts you checked with that caveat. If they do not overlap, report both and keep the manual check as a calibration. Do not average them.

What should a report say about the surface?

Name it. "Recommended in 16 of 40 answers from OpenAI's API with web search, US location, no personalization" is a statement you can defend. "Your brand appears in 40% of ChatGPT answers" is not, because the app was not sampled. Give the count and the denominator, say which engines and which prompts, and list what is not covered in one line. The methodology page sets out how the index works, and the October 2026 report gives each category's most recommended brand with its count and denominator.

Frequently asked questions

Does turning off personalization make the app match the API?

No. It removes memory and history as one reason the two can differ. The model, the tools, the search settings and the product interface can still differ, and Perplexity's own FAQ says outputs may differ even though the search system is shared.

Can I run my own API sample instead of buying a tracker?

Yes. OpenAI's documentation describes a web search tool in its Responses API with an approximate user location you set, and a way to request the full list of sources. The work is in writing neutral prompts, asking each one enough times, and recording brands consistently. How to measure your AI visibility sets out a manual version of the same method.