Robots.txt Tester & AI Crawler Checker — who can crawl your site?

What robots.txt does — and doesn't do

robots.txt is a plain text file at the root of a site (https://example.com/robots.txt) that tells crawlers which URLs they may fetch. Every well-behaved crawler reads it before anything else: Googlebot and Bingbot, and now a growing list of AI crawlers — OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, PerplexityBot and more. One wrong line can remove a site from Google, or from ChatGPT's answers, overnight. It's a request, not a lock: it controls crawling, not indexing or access, and badly behaved bots ignore it.

What this tester checks

Line-by-line validation

The file is parsed the way RFC 9309 — the Robots Exclusion Protocol standard Google follows — describes, and every line gets a verdict. Typos such as Dissalow (crawlers silently skip the line), rules that appear before any User-agent (they belong to no group), full URLs where a path is expected, relative sitemap addresses, a $ in the middle of a pattern, and directives Google doesn't support — Crawl-delay, Host and Noindex, which Google stopped honouring in 2019 — are all flagged with an explanation. Files over 500 KiB are flagged too, because Google stops reading there.

A URL tester that shows the deciding rule

Enter any path, pick a crawler, and the tester answers Allowed or Blocked and highlights the line that decided it. The rules are the standard ones:

• A crawler obeys the group that names it; only if there is none does it fall back to User-agent: *. Several groups naming the same crawler are combined.
• Among that group's rules, the longest matching path wins, and Allow wins a tie.
• * matches any sequence of characters, and a $ at the end anchors the end of the URL — so Disallow: /*.pdf$ blocks PDFs only.
• The query string is part of the URL being matched, so Disallow: /*? blocks every URL with parameters.

Twenty search and AI crawlers at a glance

The crawler matrix tests your home page and the page you entered against twenty crawlers in three groups: search engines (Googlebot, Bingbot, Applebot, DuckDuckBot), AI search and assistants (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot) and AI training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai). Click any cell to load that combination into the tester.

Blocking AI crawlers: search vs training

The distinction that matters is what a crawler is for. Training crawlers collect pages to train language models; blocking them doesn't change your search rankings at all. AI search crawlers are how ChatGPT search, Claude and Perplexity find pages to quote and link to; block those and your content can't be cited — or send you visitors — from AI answers. A common middle ground is to block training and allow search:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Note that Google-Extended and Applebot-Extended aren't crawlers: they're switches that Google and Apple read to decide whether pages fetched by Googlebot and Applebot may be used for AI. And a crawler with its own group ignores everything under User-agent: * — the tester warns when that leaves a bot with fewer restrictions than you intended.

When robots.txt says yes but the bot still can't get in

robots.txt isn't the only gatekeeper. Firewalls and CDNs can block crawlers by user agent before robots.txt is ever consulted. Cloudflare has blocked AI crawlers by default on new sites since July 2025, and its newer per-purpose crawler controls can block more than intended. That's why the tester also requests your page live with each crawler's user agent and compares the answer with an ordinary browser request: a 403 or a challenge page that only bots receive is a firewall rule, not robots.txt. (Real crawlers come from their own networks; firewalls that verify bots by IP address may block our test request even when the genuine crawler gets through, so treat a block here as a lead to check in your CDN's bot settings.)

Status codes matter too

How the server answers for /robots.txt changes everything. A 404 (or any 4xx except 429) means "no rules": crawlers may fetch everything. A 5xx or 429 is treated by Google as "disallow everything" — crawling of the whole site pauses until robots.txt is reachable again. A 200 with an HTML page (common on single-page apps that answer every path with the app) means crawlers find no rules at all. The summary at the top of the result calls out each case.

Sitemaps and llms.txt

Sitemap: lines in robots.txt are how crawlers other than Google find your sitemap; the tester lists them and checks that each one answers. If none are declared, it looks in the usual places and tells you what to add. It also checks for llms.txt, a proposed Markdown summary of a site for AI tools. It doesn't grant or deny access — robots.txt does — and Google Search ignores it, so treat it as optional.

Edit and test before you deploy

Press Edit & test changes to change the file in the page: every verdict, the crawler matrix and the validation update as you type. Nothing is saved to your site; copy the result when it does what you want.

Related tools

Crawl access is one half of visibility; the page itself is the other. The SEO & AI Visibility Audit checks noindex tags, canonicals and what AI crawlers can actually read on the page, and the Redirect Checker shows the status code and redirect chain each crawler gets.

Get alerted when crawler access changes

The most expensive robots.txt mistake is the one nobody sees: a staging file with Disallow: / shipped to production, or a CDN setting that starts turning crawlers away. Monity re-reads robots.txt with every check and alerts you in email, Slack or Teams when AI crawlers such as GPTBot, ClaudeBot or PerplexityBot lose (or regain) access, when robots.txt starts blocking the page from Googlebot, or when the page turns noindex. The free plan includes three monitors, forever.