Competitor benchmarking in AI answers: a method for small samples

Benchmarking against competitors in AI answers sounds like benchmarking in search: list the rivals, count appearances, compare. The difference is crowding. An AI shopping answer names only a couple of brands (2.3 on average in the October 2026 edition, excluding the alternative question), so most brands are missing from most answers, and two brands that look unequal in a handful of answers may not be different. The method below is built for that: it compares brands on the same answers, keeps engines and question types apart, and says how lopsided a gap has to be before it means anything.

In this post, "OpenAI" means OpenAI's API with web search (gpt-6-luna) and "Perplexity" means Perplexity's Agent API with the Sonar model (perplexity/sonar). Both were the engines of the October 2026 edition of the Dopbot Index. Figures labeled "excluding the alternative question" leave out the October question that asks for alternatives to a named brand, because it names a brand in the prompt.

Who are your competitors in an AI answer?

Not necessarily the brands your sales team names. In the October 2026 edition, excluding the alternative question (1,271 answers), an answer recommended 2.3 brands on average, with a median of 2. Across 160 categories, the answers for a category recommended 9.2 distinct brands on average (median 8), from about 8 answers each. The competitor set an assistant draws from is the brands that show up in answers to your category's questions, and it can be longer than your list.

So build the set from answers, not from memory:

1. Collect the answers to your category prompts, per engine.
2. List every brand an answer recommends, including yours.
3. Keep brands recommended in at least 2 answers, plus any rival you track for other reasons, then trim to the 3 to 5 you will act on.
4. Fix that list before you compare counts, and review it at each update. Choosing rivals after seeing which ones beat you builds the gap into the benchmark.

Brand names need one more pass. A competitor that comes back as "Brand X" in one answer and "Brand X Co." in another is one brand, and a split count understates it.

How should you set up the comparison?

1. Fix the prompts and keep them. Write each question once, grouped by type: best overall, budget, premium, a specific use, and a buying-advice question that names no brand. Changing a prompt starts a new series.
2. Name no competitor in the prompts you measure. A prompt that names a brand steers the answer around that brand. As an illustration, in the October edition the alternative question, which asks for options other than a named brand, averaged 5.0 recommended brands per answer against 2.3 for the other questions, and the named brand itself was recommended in 21 of the 296 answers where we could find it. That question differs in more than the named brand, so this is not a measured effect.
3. Run each engine separately and report each on its own. Per category, brands recommended by both engines divided by brands recommended by either averaged 47.0%, excluding the alternative question. On average a category had 3.9 brands recommended by both engines and 5.3 by only one. A rival that is absent in one engine may be present in the other.
4. Record for each answer which of your tracked brands it recommends, counting a brand once per answer and counting only picks, not warnings or comparisons.
5. Keep the denominator beside every count, for example "12 of 40 answers, OpenAI, US".

Why count answers with both brands, only yours, only theirs and neither?

Your brand and a rival are measured on the same answers, so they are not two independent samples. A brand-by-brand comparison, such as 20 of 40 against 12 of 40, throws that structure away. Answers naming both brands say nothing about which of the two is favored. Only answers naming exactly one of you do, so those are the ones to compare.

Lay the counts out as four cells:

The test is then simple: among the answers that name exactly one of you, did the split lean to one side more than a fair coin would? An exact binomial (sign) test answers that, and its p-value is computed from the two off-diagonal counts alone.

The point is easiest to see with the same totals and different overlap. All three rows below are made-up counts for 40 answers, with your brand named in 20 and the rival in 12:

Table: illustrative counts at 40 answers. The last column is Fisher's exact test on 20 of 40 against 12 of 40, which cannot see the overlap.

Identical totals give p-values from 0.152 to 0.008, and only the last row clears the bar. When the rival is only ever named alongside your brand, the 8 answers where only you appear are one-sided evidence. When the two rarely appear together, the same totals split into 16 against 8, which is not. Report the four cells, not just the two totals.

Two caveats. The sign test treats answers as independent, but answers to the same prompt may resemble each other, so its p-values can be optimistic and a result near the line should not be trusted. And the bar matters: as of October 2026, the index's rule for listing a change, which is not built yet, is an exact test between two updates of the same size, with p below 0.01 and the counts shown. The same bar is sensible for a gap you intend to act on.

How lopsided does the split need to be?

The table gives, for each number of answers that name exactly one of you, the smallest split that reaches p below 0.01 on a two-sided exact test.

Table: exact two-sided sign test, computed by script.

At the October edition's 10 answers per category, a result can reach this bar only if at least 8 answers name exactly one of you and every one of them goes the same way. Without the alternative question a category has about 8 answers, so every one of them would have to name exactly one of you and go the same way. That is why a single category from that edition can show a rival "ahead" without telling you anything. From November, the updated US categories have 40 answers, 20 per engine, so only a clearly one-sided gap can reach the bar. Adding differently worded prompts of the same type for one engine gives more independent answers than repeating one prompt.

Which competitors does each engine name?

Besides the head-to-head test, build a map of the competitor set by engine: brands recommended by both, by OpenAI only and by Perplexity only. An example from one category, not a finding about the market: for mattresses, excluding the alternative question, 8 answers recommended 12 distinct brands. OpenAI's 4 answers recommended 9 of them and Perplexity's 4 answers recommended 9, and 6 were recommended by both.

Table: brands recommended in the mattresses category, excluding the alternative question, 4 answers per engine.

Read such a table as a to-do list. A rival that only one engine names is a prompt to ask the other engine more often, not proof of a difference. With 4 answers per engine, a brand missing from one engine's four answers may simply have been missed.

Does a question type change who your competitors are?

It can, and that is worth knowing, but the sample per type is small. From November, in the updated US categories, each of the 5 prompts is asked 4 times on each engine, so a question type has 8 answers and 4 per engine. Break results out by type to find where to look and where to gather more answers, and do not run the sign test per type on one category. In the October 2026 edition, per category, the brands in the budget answers and the premium answers shared 8.2% of the brands in either list, on average (155 categories with two answers of each type, both engines together), so a rival that matters for one type can be absent from the other. The budget and premium post gives the detail.

How do you find source gaps?

A source gap is a page or domain that appears where a rival is named and not where you are. To find one:

1. Take, per engine, the answers that name a rival and not you.
2. List the domains in those answers' source lists.
3. For each domain, count the answers that list it, and compare with the answers that name you.
4. Mark each domain by type: editorial review, retailer, forum or video, brand site.
5. For each candidate, check whether your product is on that page, and whether it could be.

Read the two engines' lists differently. In the October edition, OpenAI's source lists, the pages its answers link, held 2.2 URLs per answer on average across 800 answers, on 2.0 distinct domains. Perplexity's lists, the pages its search retrieved, held 27.3 URLs on 15.2 distinct domains across 787 answers, and not all are used by the answer. So an OpenAI gap list is short and specific, and a Perplexity list is long and includes pages the answer may not use. A domain in a list is presence, not influence, and a gap shows where to look, not why a rival was named. The reviews post and the retail listings post cover what to do about the common types.

What can a benchmark not tell you?

• Which brand shoppers prefer. The counts describe answers to prompts we wrote, not shoppers or their searches.
• The apps. Answers collected through APIs can differ from the consumer apps; the API versus app post explains how.
• Change over time, unless the prompts, the engines and the sample size stay the same between runs.
• Cause. A rival named more often may have better reviews, more retail listings or a better-matching product, and a benchmark cannot say which.

Frequently asked questions

How many competitors should I track?

Start with the 3 to 5 brands named most often in your category's answers, plus any rival you know is a real alternative to you. As of October 2026, the Dopbot tracker allows up to 5 competitors per brand.

Can I just ask the assistant to compare my brand with a rival?

That is a different measurement. A prompt that names both brands will name both, so it tells you how the assistant describes each, not how often it recommends you unprompted. For visibility, use prompts that name neither.