Shopify robots.txt and AI Crawlers: What to Allow or Block

Published October 23, 2026

Last checked: October 23, 2026.

Should your Shopify store block AI crawlers? Many merchants want to keep their content out of AI training while still showing up when someone asks ChatGPT or Perplexity for a product recommendation. Those are different bots with different jobs, and robots.txt lets you treat them differently. This guide explains what each major AI crawler does (from its operator's own documentation), how Shopify's robots.txt works, and how to edit it safely.

One fact to know before you touch anything: robots.txt doesn't control what Shopify Catalog sends to AI channels. Shopify says blocking AI crawlers at the robots.txt or network layer "affects only open-web discoverability" and "doesn't stop your product data from being sent by Shopify Catalog to the agentic storefronts that you've activated" (Shopify Help: Agentic storefronts products).

Two ways AI platforms see your store

Shopify describes two separate paths (Shopify Help: Agentic storefronts products):

Shopify Catalog. "Shopify Catalog is the primary method for agentic storefronts to receive your product data." Eligible products are listed with their title, description, options, images, price and availability, and Catalog keeps prices and inventory updated. ChatGPT and Microsoft Copilot get your products this way, and Google AI Mode and Gemini get them through the Google & YouTube sales channel.
The open web. "AI crawlers might also access your store directly through the open web." This is where robots.txt applies. It matters for things like your pages being cited as a source in ChatGPT search or Perplexity answers, and for stores that aren't eligible for Catalog, which AI channels "might still access and display ... through other means," such as "traditional web crawling and indexing."

So the question isn't "do I want to be in AI shopping?" (that's Catalog, managed in the admin). It's "which bots may read my website?"

What each AI crawler does

The table below uses only each operator's own documentation.

User agent	Operator	What it's for	Follows robots.txt?
`OAI-SearchBot`	OpenAI	Surfacing sites in ChatGPT search	Yes
`GPTBot`	OpenAI	Training OpenAI's foundation models	Yes
`ChatGPT-User`	OpenAI	Fetches a page when a ChatGPT user asks	"may not apply"
`PerplexityBot`	Perplexity	Surfacing and linking sites in Perplexity search	Yes
`Perplexity-User`	Perplexity	Fetches a page to answer a user's question	"generally ignores" it
`Google-Extended`	Google	A control token for Gemini training and grounding	Yes (it's a robots.txt token)
`Googlebot`	Google	Google Search, including AI features	Yes

The details behind each row:

OpenAI (Overview of OpenAI crawlers):

OAI-SearchBot "is for search." OpenAI says sites that opt out "will not be shown in ChatGPT search answers," though they can still appear as navigational links. Changes can take about 24 hours to apply.
GPTBot is used "to make our generative AI foundation models more useful and safe." Disallowing it signals that your content shouldn't be used to train them. Blocking GPTBot is a separate choice from blocking OAI-SearchBot.
ChatGPT-User handles user-initiated actions and "is not used for crawling the web in an automatic fashion." Because a person triggered the visit, OpenAI says "robots.txt rules may not apply." To opt out of search, OpenAI points to OAI-SearchBot, not ChatGPT-User.

Perplexity (Perplexity crawlers):

PerplexityBot "is designed to surface and link websites in search results on Perplexity" and is "not used to crawl content for AI foundation models." Perplexity recommends allowing it if you want to appear in its results.
Perplexity-User supports user actions; "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."
Perplexity says changes can take up to 24 hours to show up.

Google (Google's common crawlers):

Google-Extended is not a separate crawler. It's a token in robots.txt that controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
For AI Overviews and AI Mode, Google's guidance is that "robots.txt directives for Googlebot" are the control, and a page "must be indexed and eligible to be shown in Google Search with a snippet" to be a supporting link (Google: AI features and your website). Blocking Googlebot to avoid AI Overviews would also remove you from regular Search, so it's rarely what a store wants.

A sensible default for most stores

There's no single right answer, but here's how the trade-off looks based on the documentation above:

Allow the search bots (OAI-SearchBot, PerplexityBot, Googlebot). Blocking them removes your pages from those answers, and gives you nothing back.
Training bots are your call (GPTBot, Google-Extended). Blocking them doesn't remove you from ChatGPT search or Google Search, according to OpenAI and Google. Note that Google-Extended also covers grounding in Gemini Apps, so read Google's page before deciding.
Don't count on robots.txt for user-triggered fetchers (ChatGPT-User, Perplexity-User). Their operators say robots.txt may not apply.

Shopify's own default already allows crawlers to request public content like products, collections, pages, blogs and policies (Allow: /) while blocking admin, cart, checkout, account and filtered or sorted collection URLs (Shopify Help: Editing robots.txt.liquid). If you've never customized it, AI search bots can already read your product pages.

Step 1: Check what your store serves now

Open https://your-store.com/robots.txt in a browser. Look for User-agent: lines that name an AI bot followed by Disallow: /, or a User-agent: * group with rules that block /products. If you see no AI bots named and no unusual rules, you're on Shopify's default.

You can also run the free AI readiness checker. Among other checks, it reads your robots.txt and reports whether assistant bots and training bots are allowed to read product pages.

If your theme has an old robots.txt.liquid that copies the default groups, Shopify notes it may contain older rules such as Disallow: /search and Disallow: /policies/. Deleting the template restores the current default.

Step 2: Edit robots.txt.liquid (only if you need to)

Shopify says the default "works for most stores." It also warns that "Incorrect use of the feature can result in loss of all traffic" and that this is an unsupported customization Shopify Support can't help with (Shopify Help). Before you start, save a copy of your current /robots.txt output, as Shopify's developer docs recommend (Shopify.dev: Customize robots.txt).

To create the template:

In your Shopify admin, go to Online Store.
Next to your published theme, click the ... menu, then Edit code.
Click Add a new template and choose robots.
Click Create template, make your changes and save.

Shopify recommends keeping its Liquid loop over robots.default_groups so it can keep the default rules up to date, and adding your own rules outside that loop. Shopify's developer docs show custom groups written as plain lines, for example a crawler block:

`` User-agent: discobot Disallow: / ``

(Shopify.dev: Customize robots.txt). Following that pattern, a store that wants to opt out of OpenAI and Gemini training while staying in ChatGPT search would add, below the default loop:

``` User-agent: GPTBot Disallow: /

User-agent: Google-Extended Disallow: / ```

Don't add a group for OAI-SearchBot or PerplexityBot unless you mean to opt out of those search results. Rules apply per user agent, so blocking GPTBot doesn't affect OAI-SearchBot.

A few practical notes from Shopify's Help Center:

Remove earlier workarounds first, such as a third-party service like Cloudflare in front of your store. Separately, Shopify says bot management is handled for stores on Shopify, and it doesn't recommend a proxy in front of Shopify (agentic storefronts products).
Changes take effect immediately, but crawlers may take time to react. OpenAI and Perplexity both mention about 24 hours.
Uploading a theme from the admin Themes section won't import the file, so recheck /robots.txt after a theme switch.
To undo everything, open Edit code, right-click robots.txt.liquid and choose Delete permanently.

Step 3: Make sure Catalog is the part you actually manage

Since robots.txt doesn't decide what AI shopping channels receive, your product data does most of the work. Shopify notes it can't control other ways AI channels list your products, but Catalog is the feed it calls "authoritative." To get the most from it:

Confirm your store and products are eligible: see our Shopify Catalog requirements checklist.
If you want out of ChatGPT or Copilot, use the Catalog setting, not robots.txt. Shopify says you "can choose to remove Shopify Catalog access to your products for ChatGPT or Microsoft Copilot," though products might still appear through other methods.
Fill the fields assistants match on: categories, category metafields and barcodes. Our guides on product categories in bulk and GTINs walk through them.

Surfaced helps with that last part: it scores each product's AI readiness, applies safe fixes like categories and alt text with undo, and attributes AI-referred orders. But the robots.txt step above needs nothing more than the theme editor.

Common questions

Does blocking GPTBot remove me from ChatGPT shopping? Not according to the sources above. GPTBot is OpenAI's training crawler; ChatGPT search uses OAI-SearchBot, and Shopify products reach ChatGPT through Shopify Catalog, which robots.txt doesn't affect.

Do I need an llms.txt as well? Shopify already serves /llms.txt and /agents.md by default, and says these files don't replace Catalog. See Shopify llms.txt: do you need one?.

My products aren't in ChatGPT and robots.txt looks fine. What next? Work through why Shopify products don't show in ChatGPT, which covers eligibility, exclusions and ranking.