Your robots.txt says yes, your server says no: the AI crawler block nobody checks

WebAudit 360 TeamWebAudit 360 TeamAugust 11, 20267 min read
Your robots.txt says yes, your server says no: the AI crawler block nobody checks

You allowed the AI crawlers in robots.txt. You checked the file, the rules look right, and every GEO tool you've run says you're fine. So why does ChatGPT still answer questions about your industry without ever mentioning you?

Because robots.txt is a request, not a lock. It tells well-behaved bots what they're permitted to fetch. It has no power to make your server actually serve them. Between that text file and the crawler sits a CDN, maybe a WAF, maybe a security plugin, and any one of them can return a 403 to GPTBot while your robots.txt cheerfully says "allowed."

This is the failure mode that file-based checks cannot see. Every rule passes. The site is still invisible.

What this check actually does

Instead of reading a file about permission, we fetch your page three times while pretending to be the crawlers themselves, using their real user-agent strings:

  • GPTBot (ChatGPT)
  • ClaudeBot (Claude)
  • PerplexityBot (Perplexity)

Then we look at what came back. Each request gets 8 seconds before we call it a timeout.

A response counts as blocked in three situations, and the third one surprises people:

  1. An HTTP error. Anything 400 or above. A 403 is the classic signature of a WAF rule matching on user-agent.
  2. A bot-challenge page. The status says 200, so a naive check thinks everything worked, but the body is an interstitial. We look for the tells: "Just a moment", "Attention Required", cf-chl, cf-browser-verification, "enable JavaScript and cookies", "access denied", and captcha markup. A crawler that receives this page reads a challenge screen, not your content.
  3. An almost-empty body. Under 200 characters. Your page already passed our readability check as a normal browser, so if the crawler gets a stub, it got a different, emptier response than a human would.

That second category is the one worth sitting with. A 200 status code with a Cloudflare challenge inside it looks like success to almost every monitoring tool you own.

Why a block here costs more than a robots.txt block

A robots.txt block is at least honest. The crawler asks, gets told no, and moves on. You can find it in thirty seconds by opening the file.

A server-side block is silent in both directions. The crawler doesn't get a polite refusal, it gets a wall. And on your side, nothing logs it as a problem, because from your server's point of view it did exactly what it was configured to do: it stopped a bot.

The result is that you can pass a GEO audit, have a perfect robots.txt, publish genuinely good content, and still be absent from AI answers for months without a single signal telling you why.

How this differs from the robots.txt check

These two checks sound similar and test completely different layers. It's worth being precise about the split, because fixing one does nothing for the other.

AI crawler permissions AI crawler reachability
What it reads Your robots.txt rules The actual HTTP response
Layer A text file stating intent CDN, WAF, origin server
Typical failure Disallow: / under a bot name 403 or challenge page to a bot user-agent
Where you fix it The robots.txt file Cloudflare, your WAF, a security plugin

You need both to pass. A site can allow every AI bot in robots.txt and block all three at the edge. The reverse happens too: a permissive CDN in front of a robots.txt that still carries a leftover Disallow: / from staging.

Finding the block yourself

You don't need a tool for a first look. Curl the page as a crawler and read what comes back:

curl -sS -o /dev/null -w '%{http_code}\n' \
  -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot' \
  https://yourdomain.com/

A 200 is a start, not an all-clear. Drop the -o /dev/null and read the body. If the first screenful mentions checking your browser or enabling JavaScript, you found your block even though the status code says otherwise.

Run it against your real pages, not just the homepage. WAF rules get applied per-path, and it's common to see a permissive homepage in front of blocked article routes.

The usual culprits

In practice, server-side AI blocks come from a short list.

Cloudflare's bot controls. The "Block AI Scrapers and Crawlers" toggle is one click and it catches GPTBot, ClaudeBot, and PerplexityBot together. Plenty of sites have it on without anyone remembering turning it on. Super Bot Fight Mode produces the same outcome by a different route.

Security plugins. WordPress security suites ship user-agent blocklists that get updated with new bot names over time. You install the plugin for spam protection and inherit an AI-crawler block you never chose.

Rate limiters that see a bot and assume the worst. Some rules match anything with "bot" in the user-agent string. GPTBot, ClaudeBot, and PerplexityBot all contain it.

Origin-level rules. An .htaccess or nginx if ($http_user_agent ...) block, often copied from a Stack Overflow answer about blocking scrapers years ago.

The fix in all four cases is the same shape: allowlist the AI crawler user-agents explicitly, ahead of whatever general bot rule is catching them. Do it at the layer that's blocking, since adding an Allow line to robots.txt will not change what your WAF does.

Honest limits of this check

We test one URL, from one network location, at one moment. That covers the common case well and misses a few real ones.

Geographic WAF rules can serve a challenge to traffic from one region and pass another. Rate-based rules may let a single probe through and block a crawler doing a real multi-page pass. And some challenge pages are novel enough that our marker list won't recognize them, so an unusual interstitial can read as reachable.

A pass here means the page answered three specific crawlers correctly when we asked. It's strong evidence, not a guarantee for every bot on every route forever. If AI engines still can't see you after this check passes, look at whether the page is readable at all before assuming the crawlers are the problem.

FAQ

Does a 403 to GPTBot hurt my Google rankings?

No. Googlebot is a separate crawler with a separate user-agent, and blocking OpenAI's bot has no effect on classic search indexing. That's precisely why this problem survives so long: your Search Console stays green while ChatGPT and Perplexity see nothing.

I want to block AI training but still appear in AI answers. Is that possible?

Partly. Some crawlers are training-focused, some fetch pages to ground live answers, and a few do both. You can allow the answer-oriented bots while blocking others, but you have to do it deliberately at both layers, and you should read each bot's own documentation rather than trusting a plugin's grouping.

Our CDN blocks bots for security reasons. Is allowlisting these three risky?

Low risk. These are named, documented crawlers from companies with public IP ranges and abuse contacts. The bots you actually want to stop are the ones lying about their user-agent, and a user-agent allowlist for three honest crawlers doesn't help those at all.

Why did the check say a bot was blocked when I can load the page fine?

You're loading it as a browser. The block matches on user-agent, so your visit never triggers the rule. That asymmetry is the whole reason this check exists.

Key Takeaways

  • robots.txt grants permission but cannot force your server to respond, so a site can allow GPTBot in robots.txt and still return a 403 to it from a CDN or WAF.
  • Testing AI crawler access means fetching the page with each crawler's real user-agent and inspecting the response, not reading a rules file.
  • A 200 status code does not prove reachability: a Cloudflare-style challenge page ("Just a moment", "enable JavaScript and cookies") is served with a 200 and contains no content for the crawler to read.
  • Server-side AI crawler blocks are silent, since Google Search Console stays healthy while ChatGPT, Claude, and Perplexity receive nothing.
  • Fix a server-side block at the layer causing it (Cloudflare bot controls, WAF rules, security plugins, origin rules); adding an Allow line to robots.txt will not override a WAF.

Want to know what GPTBot, ClaudeBot, and PerplexityBot actually receive from your server? Run a free audit: we fetch your page as each one and report exactly what came back. More in the GEO explainers.

← Back to the blog