AI crawlers and robots.txt: what each bot does and what to allow

"Block the AI bots" is one instruction that covers at least three different jobs. A search bot decides whether your pages can be surfaced in an assistant's answers. A training bot collects content that may be used to build models. A user-initiated fetcher loads a page because a person asked a question about it. The operators document these as separate bots with separate rules, and a store that blocks all of them with one line loses things it may want to keep.

This post lists each bot's documented role as of October 2026, from the operators' own pages. It then covers the CDN layer, the trade-offs, and a way to check your own setup. Where a bot's behavior is not documented, we say so.

What does each AI bot do?

Three details change how you read the table.

The same page can serve two roles. OpenAI says that if a site allows both OAI-SearchBot and GPTBot it may use the results of one crawl for both uses. Apple says Applebot's crawled data may be used for search, to help train its foundation models, and to give AI models context for answers. The control tokens (Applebot-Extended, Google-Extended) exist to separate the uses, not the crawling.

Fallbacks exist. Apple says that if your robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. A rule written for one bot can end up governing another.

Some fetchers do not follow robots.txt by design. OpenAI and Perplexity both say their user-initiated fetchers may or generally do ignore robots.txt, since a person asked for the page. If you want to stop those, the control is at the network layer, not in robots.txt. That is also a decision to take with open eyes: those fetches are the assistant reading your page for a shopper who asked about it.

What does robots.txt actually enforce?

Nothing. The Robots Exclusion Protocol standard says its rules are not a form of access authorization. Shopify's help center calls robots.txt rules directional and advisory, and says not all crawlers are guaranteed to follow them. Cloudflare's documentation says compliance is voluntary and that some operators may disregard it.

It follows that robots.txt states a preference to well-behaved bots, and a firewall or CDN rule enforces one. Using both is normal, and each needs checking. Anthropic adds that blocking its bots by IP address may not work and can stop them reading your robots.txt, so for its bots the documented opt-out is the robots.txt rule.

Timing matters too. Both OpenAI and Perplexity say changes to your rules can take up to 24 hours to take effect for search.

Which bots should a store allow?

There is no single right answer, and "allow everything" is as much a decision as "block everything". The view below is our judgment, built on the documented roles. It assumes a store that wants to be found in AI shopping answers and has not decided whether it minds training use.

1. Allow the search bots. Googlebot, OAI-SearchBot, PerplexityBot, Claude-SearchBot and Applebot are the ones the operators describe as surfacing or indexing content for search. OpenAI and Perplexity both recommend allowing their search bots and their published IP ranges. Storebot-Google is separate: it governs Google Shopping surfaces, including the Shopping tab, so a store that wants to appear there should not block it.
2. Decide on the user-initiated fetchers deliberately. Allowing ChatGPT-User, Perplexity-User and Claude-User means an assistant can read your live page when a shopper asks about it. Anthropic notes that disabling Claude-User may reduce visibility for user-directed search.
3. Treat training as a separate question. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot relate to training. The operators' own pages say that blocking them does not remove you from search: OpenAI says each setting is independent, Google says Google-Extended does not affect Search, and Apple says disallowing Applebot-Extended still allows inclusion in search.
4. Know what is not known. OpenAI lists model responses generated before new search results as one input to shopping results. We found no operator that says whether allowing training use changes how a brand is described later. We did not test it, and our data cannot.

If you decide against training use, the lines are short. For one bot: User-agent: GPTBot on one line and Disallow: / on the next, repeated for each token you want to block. Anthropic asks that you do this for every subdomain you want to opt out.

What can the CDN layer add, and what can it break?

Cloudflare's documentation, as of the dates on its pages, describes three overlapping controls:

• Per-crawler actions. AI Crawl Control lists the AI crawlers that request your content and lets you allow or block each one. Blocking creates or updates a WAF custom rule. On the free plan, detection uses user agent strings, so it catches self-identifying crawlers.
• A managed robots.txt. When bot preference sync is turned on and at least one of your Search, Agent or Training policies blocks, Cloudflare can prepend its own directives to your robots.txt. Its example lists bots such as GPTBot, ClaudeBot, Google-Extended, CCBot and Applebot-Extended.
• Policy presets. Search, Agent and Training presets can each be set to block on all pages, block on pages with ads, or allow. Cloudflare's page, last updated July 1, 2026, said its defaults for new domains would change on September 15, 2026, so that Training and Agent bots are blocked on pages that display ads, with Search allowed. We did not confirm in an account whether that change took effect, so check your own dashboard. The page also says mixed-purpose crawlers that combine search and training are blocked by all training-block settings.

The last point is where a well-meant switch can hurt. Before turning on a training block at the CDN, open the crawler list in your account and check which bots sit in which category, because a mixed-purpose crawler may be both. Perplexity's documentation gives a Cloudflare rule that combines user agent and IP address conditions with an Allow action for its bots, and says to use its published IP lists.

On Shopify, the network layer is handled by Shopify. Shopify says you do not need to act on it and does not recommend a proxy in front of the store. Editing robots.txt.liquid is an unsupported customization that Shopify Support cannot help with, so keep a copy of the default. Shopify also says that blocking AI crawlers at either layer affects only open-web discovery, and not the product data Shopify Catalog sends to channels you have switched on (the data side is in product feeds for AI shopping). The Shopify crawlability audit walks through this store by store.

What is at stake in our data?

Our October 2026 samples show how often a brand's own pages show up in what the engines linked or retrieved. In OpenAI's API with web search (gpt-6-luna), the pages linked in an answer included the site of a brand the answer recommended in 46.6% of 800 answers. In Perplexity's Agent API with the Sonar model (perplexity/sonar), the retrieved source lists held one in 62.0% of 787 answers. Both figures cover all prompts, including the October alternative-to-a-brand question. The rule is strict: a domain whose first label equals a recommended brand's name, with spaces and punctuation removed (samsung.com for Samsung), so it misses country sites, parent companies and other store names. These are presence counts and not proof of influence.

What they suggest is narrow: brand sites are a regular part of what these two engines link or retrieve. The operators describe search bots as what lets a site be surfaced in their search products, but the pages we read do not say whether the API tools we sample draw on those bots, so we cannot say how many of these appearances depend on crawl access. Our data does not include Google, Anthropic or Apple. The October 2026 edition has the full engine-by-engine figures.

How do you check your own setup?

1. Read your live robots.txt. Open yourdomain.com/robots.txt in a browser. Look for rules under User-agent: * that apply to everyone and for named groups. Remember Apple's Googlebot fallback.
2. Check what the CDN adds. If a CDN or app sits in front of the store, request the file again and compare. Cloudflare says managed content can be prepended, so the file a bot sees may differ from the one in your theme.
3. Test the firewall with a simulated user agent. Request a product page with each bot's user agent string. Google warns that the HTTP user agent string can be spoofed, so this tests your rules, not the real bot. A pass here proves nothing about indexing or visibility.
4. Look for the real bots in your logs. Match request IPs against the operators' published lists. OpenAI and Perplexity publish JSON lists for each bot, Apple documents reverse DNS under applebot.apple.com, and Google documents how to verify its crawlers. Anthropic says a source IP on its list indicates the crawler comes from Anthropic.
5. Use the vendor's own test where one exists. For Google, the URL Inspection tool in Search Console shows the HTML Googlebot received.
6. Wait out the lag, then recheck. OpenAI and Perplexity say up to 24 hours for search to reflect a change.

A successful response to a bot only says the server answered it. Google says indexing and serving is not guaranteed even for a page that meets every requirement.

Frequently asked questions

Do I need an llms.txt file or other AI text file?

For Google's AI features, Google says you do not need new machine-readable files, AI text files or special markup, and that the usual Search requirements apply. We did not find a requirement for one in the OpenAI, Perplexity, Anthropic or Apple crawler pages we read.

Can I block a bot at the firewall instead of in robots.txt?

You can, and a firewall rule is what enforces a block, but the operators differ on whether it is a safe substitute. Anthropic says blocking its bots by IP address may not guarantee an opt-out and can stop its crawler reading robots.txt, so use the robots.txt rule for Anthropic's bots. Keep any firewall allow rules for the search bots you want, and recheck both layers after a change.

Does a robots.txt block stop Shopify from sending my products to AI channels?

Shopify says product data it syndicates through Shopify Catalog to agentic channels, such as ChatGPT or Microsoft Copilot in its own examples, reaches them independently of robots.txt. Whether a training block changes how models describe a brand later is not documented, and we have not measured it.