robots.txt Generator for AI Crawlers
This robots.txt generator builds a file that controls which crawlers, including AI crawlers like GPTBot, ClaudeBot, and Google-Extended, can access your site. Choose which to allow, add your sitemap, and copy the file to your domain root. Free, no signup.
# Your robots.txt will appear here. Pick your crawlers and hit Generate.
Save as robots.txt and upload to your root (yourdomain.com/robots.txt).
What is robots.txt and why does it matter for AI?
robots.txt is a plain text file at the root of your domain that tells crawlers what they may access. AI engines send their own crawlers, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more, and they check robots.txt before reading you. If you block them, even by accident, you cannot appear in their answers. This is the floor of AI SEO, also called AEO.
How to create a robots.txt for AI crawlers
- Keep the AI crawlers you want to allow checked above.
- Add your sitemap URL.
- Click Generate and copy or download the file.
- Upload it to your root so it lives at
yourdomain.com/robots.txt.
Should you allow or block AI crawlers?
If you want AI search visibility, allow them. Blocking GPTBot or Google-Extended is one of the most common reasons a brand is invisible in AI answers, and it is usually an accident. Already have a robots.txt? Run it through our AI Crawler Checker to see if you are blocking anything by mistake, and read the reasons ChatGPT ignores brands.
robots.txt generator: FAQ
- What is a robots.txt file?
- robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your site they may access. Search engines and AI crawlers check it before reading your pages, so it is the first gate between your content and the engines.
- How do I let AI crawlers access my site?
- Add rules that allow the AI crawler user agents, such as GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. This generator does it for you: keep the crawlers you want checked, add your sitemap, and copy the file to your root. If a crawler is not blocked, it is allowed by default, but stating it explicitly is clearer.
- Should I block AI crawlers?
- If you want your brand to show up in AI answers, no. Blocking GPTBot, ClaudeBot, or Google-Extended makes you invisible to those engines. Some publishers block them to protect content, but for most businesses that want AI search visibility, blocking them is self-sabotage.
- What are the AI crawler user agents?
- The main ones in 2026 are GPTBot and OAI-SearchBot (OpenAI), ChatGPT-User (ChatGPT browsing), ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot and Perplexity-User (Perplexity), Google-Extended (Gemini), and Applebot-Extended (Apple). New ones appear regularly, so it is worth reviewing your robots.txt periodically.
- Where do I put the robots.txt file?
- At the root of your domain, so it lives at yourdomain.com/robots.txt. That is the only place crawlers look for it.
- Does robots.txt affect AI search visibility?
- Directly. If your robots.txt blocks the AI crawlers, you cannot appear in those engines' answers, no matter how good your content is. It is the floor of AI SEO: get it right first, or nothing else you do will show up.
Crawlable is step one. Are you recommended?
See where you stand across every AI engine, free.