AI Crawler Checker
Is your site blocking ChatGPT, Claude, Perplexity or Gemini? Check what any site's robots.txt tells 13 AI bots, split into the ones that power AI answers and the ones that only collect training data.
Answer bots vs. training bots
AI companies run different bots for different jobs. Search and user bots fetch pages when someone asks a question, so blocking them keeps your pages out of AI answers and links. Training bots gather data for future models; blocking them doesn't affect today's AI search results. A robots.txt written before 2024 often blocks both by accident, for example through a broad User-agent: * rule.
How to read the result
- Check the answer bots first. For a store or a business site you usually want all of them allowed.
- Decide on training bots on purpose, and write named rules so the choice is explicit.
- Paste a product or article URL to test a specific path: a rule like
Disallow: /productsonly shows up there. - Run the check again after editing robots.txt.
More free tools: AI shopping readiness checker for Shopify stores, AI Readiness Index of top brands, and the script tag checker.
Frequently asked questions
What does this checker do?
It reads the site's public robots.txt and applies the standard matching rules (RFC 9309) for each AI bot: a group that names the bot wins over the * group, and the longest matching rule decides. It reads one public file and changes nothing.
Which bots matter for showing up in AI answers?
Search and user bots such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User fetch pages to answer questions. If they are blocked, assistants can't read or link your pages. Training bots such as GPTBot, ClaudeBot, Google-Extended and CCBot only collect data for future models.
Should I block training crawlers?
That's a business choice. Blocking them doesn't remove you from AI search results, which use the search and user bots. Many stores allow the search bots and decide on training bots separately.
Is robots.txt the only thing that can block AI bots?
No. Firewalls and bot protection (for example a CDN's AI bot blocking setting) can block them even when robots.txt allows them, and some vendors say user-initiated fetchers may not follow robots.txt. This tool checks robots.txt only.
How do I edit robots.txt on Shopify?
In Online Store > Themes > Edit code, add the robots.txt.liquid template and add your rules there. Shopify's default rules stay in place unless you remove them.