AI Crawler Checker
Paste your robots.txt below and this AI crawler checker tells you which AI crawlers, like GPTBot, ClaudeBot, and Google-Extended, your site is blocking. Blocking them makes you invisible in AI answers. Free, runs in your browser, nothing stored.
Find yours at yourdomain.com/robots.txt, copy it all, paste here.
Paste your robots.txt and hit Check.
Why checking your AI crawlers matters
AI engines read the web with their own crawlers, and they obey your robots.txt. If yours blocks GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, those engines cannot read you, so you cannot be cited or recommended. This is the most common and most fixable reason a brand is missing from AI answers, and it is invisible until someone checks. It is the floor of AI SEO, also called AEO.
How to use this checker
- Open
yourdomain.com/robots.txtin your browser. - Copy everything and paste it into the box above.
- Click Check to see which AI crawlers are allowed or blocked.
Found a blocked crawler? Fix it.
If anything is blocked by mistake, rebuild a clean file with our robots.txt generator. And remember, crawlability is necessary but not sufficient: see the five reasons ChatGPT ignores brands for what comes after.
AI crawler checker: FAQ
- What does the AI crawler checker do?
- It reads your robots.txt and tells you which AI crawlers, such as GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, your site is allowing or blocking. Blocking them, often by accident, means you cannot appear in those engines' answers. The check runs entirely in your browser, and nothing is stored.
- How do I find my robots.txt?
- Go to your domain followed by /robots.txt, for example yourdomain.com/robots.txt. Copy everything you see and paste it into the box above. If nothing loads, you may not have a robots.txt yet, which usually means crawlers are allowed by default.
- Which AI crawlers should I allow?
- If you want AI search visibility, allow the major ones: GPTBot and OAI-SearchBot (OpenAI), ChatGPT-User, ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot and Perplexity-User, Google-Extended (Gemini), and Applebot-Extended (Apple). Blocking any of them removes you from that engine's answers.
- What happens if I block AI crawlers?
- You become invisible to that engine. If GPTBot is blocked, ChatGPT cannot read your site, so it will not cite or recommend you, no matter how good your content is. This is one of the most common and most fixable reasons a brand is missing from AI answers.
- My crawlers are allowed but I'm still not in AI answers. Why?
- Being crawlable is necessary but not sufficient. You also need content written to be quoted and third-party corroboration the engines trust. Crawlability is the floor. See the five reasons a brand gets skipped for the full picture.
Crawlable? Good. Now, are you recommended?
See where you stand across every AI engine, free.