What your Shopify robots.txt tells the AI crawlers

Shopify lets every AI crawler in by default

That is the answer to the question most people are actually asking, and it is worth having before you touch anything. No Shopify default blocks GPTBot, ClaudeBot, Google-Extended, PerplexityBot or any other AI agent. If an AI crawler is being turned away from your store, something on your side put it there.

What has changed is the file it is being let in by. Two different Shopify defaults are live on storefronts right now, and which one you have was not your decision.

The one most people remember opens with a long list of things not to crawl: /cart, /checkout, /account, /orders, /search, and every sorted or filtered collection URL. Sensible, dull, unchanged for years.

The newer one opens with a paragraph addressed to software. Verbatim from the top of the file, on a live store:

# Shopify storefront. Public product, collection, page, blog, policy, cart, and localized HTML is crawlable.
# Agent instructions: https://yourstore.com/agents.md
# UCP discovery: https://yourstore.com/.well-known/ucp
# UCP/MCP endpoint: https://yourstore.com/api/ucp/mcp
# Agents should use UCP/MCP for catalog, cart, and checkout. Payment requires buyer approval.
#
# Checkouts are for humans. Do NOT complete checkout, payment, or order placement automatically

Those URLs are not documentation. On a store with the new default, /agents.md serves a real document telling an AI shopping assistant to prefer Shopify's Shop skill over scraping you, and /api/ucp/mcp is a live endpoint exposing search_catalog, create_cart and create_checkout over the Universal Commerce Protocol. Your store has an agent-commerce interface, and it was switched on for you.

Six public storefronts checked by hand on 8 September 2026: four were serving the new file, two the old one. The rollout is partway through, so your neighbour's store and yours can disagree. Find out which you have in one line:

curl -s https://yourstore.com/robots.txt | head -3

A comment about agents means the new default. A bare User-agent: * means the old one. Neither blocks anything AI-related, which is why the rest of this post is mostly about not breaking that.

The bot that trains on you is not the bot that recommends you

Almost every "block AI scrapers" post treats the crawlers as one thing. They are not, and the distinction is the whole decision.

OpenAI runs four agents and states plainly that "each setting is independent of the others". You can allow the one that decides whether ChatGPT can see you and refuse the one that trains on you.

Anthropic splits its three the same way, in a help-centre article last revised in April 2026.

Read those two tables together and the practical move is obvious. If your objection is to being training data, you block GPTBot and ClaudeBot, and you keep every scrap of your visibility in both assistants. Almost nobody does this, because almost nobody knows the tokens are separate.

Google is the exception, and it is the one that catches people out.

Google-Extended has no user agent of its own. Google's crawler documentation says it plainly: crawling is done with existing Google user agent strings, and the robots.txt token "is used in a control capacity". What it controls is two things at once — training future Gemini models, and grounding, which is the step that hands content from Google's index to the model at the moment someone asks a question.

So the clean split does not exist here. Blocking Google-Extended to opt out of Gemini training also opts you out of being quoted in Gemini answers. There is no token that separates them.

The compensation is that it stops there. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal", which puts the AI features inside Search itself on the Googlebot side of the line, untouched either way.

One piece of housekeeping while you are in the file. anthropic-ai is a retired token that a great many stores still carry, copied from a 2023 blocklist. It costs nothing to leave and buys nothing either. The three tokens above are the current set.

The trap: naming a bot switches off every rule you did not repeat

Here is the advice you will find on every "let AI crawl your Shopify store" page:

User-agent: GPTBot
Allow: /

Append that to a Shopify storefront and you have just done something you did not intend.

The robots.txt standard, RFC 9309, is specific about how a crawler picks its rules. It matches its own token against the user-agent lines, obeys that group, and — the sentence that matters — "if no matching group exists, crawlers MUST obey the group with a user-agent line with the * value". Only then. A named group does not extend the * group or override it line by line. It replaces it wholesale.

So the moment GPTBot appears anywhere in your file, GPTBot stops reading the * group. Every Disallow Shopify put there for you is gone for that crawler: your checkout, your cart, your account pages, your customer orders, and the sorted-and-filtered collection URLs that are the single largest crawl trap on a Shopify store. You added two lines meaning "you are welcome" and the crawler heard "and none of the other rules apply to you".

Order does not save you, because there is no order. Position in the file is irrelevant to group selection.

This is not a standards-lawyer's point. It is in your file already, in Shopify's own words. Both defaults carry a named group for Google's ad crawler, and above it, this comment:

# Google adsbot ignores robots.txt unless specifically named, some rules must be repeated.

Then Shopify repeats about a dozen rules inside that group, because it has to. Shopify's engineers hit the same rule and paid the same tax. Shopify's own help centre does it too, repeating its full rule set inside a named PetalBot group rather than trusting * to carry over.

What to do instead

Do nothing. For most stores this is the correct answer and nobody gives it. The default already allows every AI crawler, with sensible exclusions that took Shopify years to tune. Verify it, then leave it alone.

If you are blocking, naming is fine. A group that only says Disallow: / has nothing to inherit, so the replacement problem never arises:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

That blocks training in both and leaves OAI-SearchBot, Claude-SearchBot and Claude-User in the * group, still allowed, still able to name you.

If you are naming a bot in order to allow it, repeat the rules. Copy the whole Disallow list from your * group into the named group. It is ugly and it is what the standard requires.

Changing it on Shopify, without freezing it

Shopify does not let you edit /robots.txt. It generates the file, and it updates the defaults over time — which is exactly how the agentic file above arrived on four of six stores without anyone doing anything.

What you can do is add a theme template that renders it. Shopify's documentation is clear that this template ships in no theme by default; you create it yourself:

1. Online Store → Themes, find the theme, … → Edit code.
2. Right-click the Templates folder, New File, name it robots.txt.liquid.

It cannot be a JSON template. It has to be robots.txt.liquid.

The content is where stores go wrong. Shopify mirrors its defaults through a Liquid object, and the documented template loops it:

{% for group in robots.default_groups %}
  {{- group.user_agent -}}
  {% for rule in group.rules %}
    {{- rule -}}
  {% endfor %}
  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

You are allowed to replace all of that with hand-written plain text. Shopify strongly recommends you do not, and gives the reason: "the default rules are updated regularly to ensure that SEO best practices are always applied."

Read that against the first section of this post. A merchant who pasted a hardcoded robots.txt into that template at any point in the last few years is still serving whatever the defaults looked like on the day they did it. They did not receive the agentic file. They will not receive the next one. The two-line convenience froze the file permanently, and nothing in the admin will ever tell them.

Keep the loop. Append only the group you actually need, after it.

Two more things about this template that are easy to miss, both consequences of it being a theme file:

• It is per theme. Duplicate your theme, edit the copy, publish it, and your robots rules are whatever the copy had. Theme changes are the most common way a considered robots.txt quietly disappears.
• It follows theme deploys. If a developer publishes from a branch that predates your edit, the edit is gone with it, and the symptom is a crawler behaviour change weeks later that nobody connects to a theme push.

Checking that you got what you asked for

Four checks, none of which need a tool.

Read the live file, not the template. curl -s https://yourstore.com/robots.txt. The template can be right and the published theme still be wrong.

Read it for named groups, not for the word Disallow. Search it for every user-agent line. A group you forgot is the failure that looks like success — the file has no scary rules in it and a crawler is still ignoring half of what you meant.

Give OpenAI a day. Its documentation says it can take around 24 hours from a robots.txt change for their systems to adjust. If you check an hour later and nothing has moved, nothing has gone wrong.

Re-check after every theme publish. This is the one that catches real stores, and it is a two-second job that belongs in whatever checklist you already run after a deploy.

You can also read any competitor's file, always. The standard makes /robots.txt implicitly allowed, which is the one URL nobody can refuse you.

What this is and is not worth

Crawler access is the entry requirement, not the achievement. Getting every token right buys you the ability to be considered. It does not make an assistant recommend you, and anyone claiming a robots.txt edit will get you named in ChatGPT is selling you something.

What it does do is remove a category of silent, self-inflicted failure that costs nothing to fix and is invisible until you look. A store can spend a year on product copy and schema while a single inherited line keeps the crawler out, and no dashboard anywhere will mention it.

The rest of the work — structured data, answer-shaped copy, third-party corroboration — is in how to get your store recommended by ChatGPT. Crawler access is step one of that post, and this is what step one actually involves.

If you would rather have it checked for you, with your live file read line by line and the fixes written down in order, that is section five of the Store Teardown.