Free robots.txt AI crawler checker
See which AI assistants' crawlers your robots.txt lets in, whether your server actually lets them through, and whether you publish an llms.txt. It takes a few seconds and only reads public files.
What this checks
- robots.txt rules for 9 AI services: ChatGPT (GPTBot, OAI-SearchBot, ChatGPT-User), Claude (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity, Google AI (Google-Extended), Copilot (Bingbot), Meta AI, Apple Intelligence, Mistral and Common Crawl. Results are grouped: allowed everywhere, allowed except some paths, or blocked.
- Whether your server lets them in: we request your page once each as GPTBot, ClaudeBot and PerplexityBot (our user agent says it's SniffPup) and report a 401, 403, 429 or 503 answer. A robots.txt that says yes doesn't help if your firewall says no.
- llms.txt: whether
/llms.txtexists, has a# titleline and how many links it lists. - Sitemap: whether your robots.txt lists a
Sitemap:.
What it can't tell you
- robots.txt is a request, not a lock. It can't tell you which crawlers really follow it, or what an AI assistant has already learned about your site.
- Crawlers can come from IP ranges your firewall treats differently from ours, so the "server lets them in" result is a strong hint, not a guarantee.
- It reads the robots.txt of the exact host you enter.
wwwand other subdomains have their own. - It checks live web URLs only, not native mobile apps or pages behind a login.
- No issues found means none in this one check at that moment. It is never proof of security.
Check it yourself
From a terminal, using your own domain:
curl -s https://your-app.com/robots.txt
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.1" https://your-app.com/
curl -s https://your-app.com/llms.txt | head -5A Disallow: / under User-agent: GPTBot (or under * with no GPTBot group) blocks it. A 403 from the second command means your server turns that user agent away.
How to fix it
- Decide which assistants you want reading your site. To allow everyone, a robots.txt with
User-agent: *andAllow: /is enough. - To block one, add its own group, for example
User-agent: GPTBotthenDisallow: /. - If your host or firewall has an "AI bots" or bot-fight setting, check it matches your robots.txt.
- Optionally add an
/llms.txt: a short Markdown summary of your site with links to its key pages.
This checker answers one question. The free scan checks everything at once, and how our scanner works explains what it does and never does.
Questions
Will allowing AI crawlers help people find my app?
It lets assistants like ChatGPT, Claude and Perplexity read your pages, which they need before they can cite or recommend them. It doesn't guarantee they will.
Is blocking AI crawlers bad?
No. It's your choice. This checker just shows what your robots.txt and your server actually say, so the result matches what you meant.
What is llms.txt?
An optional Markdown file at /llms.txt that summarizes your site for AI assistants and links to your most useful pages. It's a community proposal, not a standard, and not every assistant reads it.
Why does it say my server blocked a crawler that robots.txt allows?
We requested your page with that crawler's user agent and got a 401, 403, 429 or 503. That usually means a firewall or bot-protection rule. If that's not intended, allow the crawler there too.