Page could not be read: what's blocking crawlers (and AI engines) from your site

You ran an audit and got a single result: "Page could not be read." No scores, no pass/fail list, just one gray card. It feels like the tool broke. It didn't. We reached your site, something answered, and what came back was not a readable HTML page.

That distinction matters more than any individual check we could have run. Titles, structured data, Core Web Vitals: none of it exists for a machine that never gets past the front door. And the machines that matter are lined up behind us at the same door. Googlebot. GPTBot, ClaudeBot, and PerplexityBot deciding whether an AI answer can cite you. The bot that renders your link preview in Slack. If an honest auditor can't read your page, assume some of them can't either.

There are three common causes. Each has a different fix, and you can identify yours in about five minutes.

What "could not be read" actually means

Our audit fetches your URL once, the way any polite crawler would: a single GET request that identifies itself honestly as WebAudit360Bot/1.0. For the audit to run, three things must be true about the response.

1. The status code is a success (2xx). A 403, 404, 500, or a redirect to a login page ends the audit there.
2. The content type is HTML. A PDF, a JSON payload, or an image can't be graded as a web page.
3. The HTML contains something to read. A title, headings, text, links. Any of them.

Fail any one of those and running the per-element checks would produce garbage: thirty "missing X" failures on a page that never really loaded. So we stop and tell you the real problem instead.

Cause 1: a bot wall answered instead of your page

The most common case. A firewall, CDN security layer, or bot-protection product (Cloudflare's bot modes, Akamai, PerimeterX, various WordPress security plugins) intercepted the request and served a challenge page or a flat 403 Forbidden. Your actual page never entered the conversation.

The frustrating detail: challenge pages are sometimes served with HTTP 200 and a "verifying you are human" interstitial, so even the status code looks healthy while the content is a dead end for every non-human reader.

Blocking bots is a legitimate choice. The problem is that most sites never made it. A security plugin shipped with aggressive defaults, or someone enabled "block all bots" during a scraping incident and forgot, and now everything without a browser fingerprint gets walled: SEO tools, uptime monitors, link-preview bots, and the AI crawlers that decide whether ChatGPT or Perplexity can ever cite you.

The fix: open your WAF or bot-protection settings and change blanket rules into targeted ones. Allow verified search crawlers (most products have a one-click "verified bots" allowance). Challenge suspicious behavior, not every unfamiliar user agent. And if you do want to keep AI crawlers out, do it deliberately in robots.txt, where it's a documented choice, rather than as a side effect of a firewall rule. We wrote about that trade-off in AI crawlers and robots.txt.

Cause 2: your HTML is an empty JavaScript shell

This one is the sneakiest, because the site looks perfect in your browser. We audited a company homepage this week that returned HTTP 200 with a valid text/html content type, and the entire body was a <div id="root"></div> and some script tags. No title. Zero words of text. Zero links. The real page is assembled by JavaScript in the visitor's browser, and a crawler doing a plain fetch receives the empty scaffolding.

That's the signature of a client-side-rendered single-page app: React, Vue, or Angular deployed without server-side rendering or prerendering.

Google can execute JavaScript, but rendering happens in a second, deferred pass, and it's the exception among machine readers (Google's own docs describe the queue). Most AI crawlers fetch raw HTML and do not run your JavaScript at all. To them, your site is a blank page with excellent branding.

The fix, in order of preference:

1. Server-side rendering or static generation. Every major framework supports it now (Next.js, Nuxt, SvelteKit, Astro). Marketing pages, docs, and anything you want found or cited should arrive as real HTML.
2. Prerendering for the pages that matter, if a full SSR migration is off the table. Even prerendering only public routes moves you from unreadable to fully auditable.
3. At absolute minimum, put a real <title>, meta description, and an <h1> with a paragraph of actual content into the served shell. It's triage, not a cure, but machines get something to read.

The test: hit view-source: on your URL (the raw served HTML, unlike the DevTools Elements panel, which shows the page after JavaScript ran). If you can't find your headline text in there, neither can a crawler.

Cause 3: the URL isn't an HTML page at all

Less common, but we see it: the audited URL serves a PDF, a JSON API response, or an image, or it 302-redirects every visitor to a login screen. Nothing is wrong with your site; the wrong thing was audited. Point the audit at the page a first-time human visitor would land on. If your homepage genuinely redirects everyone to a login, that's worth knowing too: it means your public face is a login form, and search and AI engines have nothing else to index.

Confirm which one you have

Five minutes, no tools you don't already have:

1. view-source:yourdomain.com in the browser. Real content in the raw HTML? Then rendering isn't your problem, look at blocking.
2. Fetch as a bot. curl -sI -A "Mozilla/5.0 (compatible; WebAudit360Bot/1.0)" https://yourdomain.com/ and read the status line. A 403 or 503 here, while your browser loads the site fine, is a bot wall in action.
3. Search Console's URL Inspection ("view crawled page") shows exactly what Googlebot received, including the rendered HTML.
4. Check robots.txt for user agents you didn't mean to block. Then remember that robots.txt only restrains polite crawlers; a WAF rule blocks everyone.

Why this outranks every other fix

An answer engine can't quote a page it can't read. A zero-word response has zero citable sentences, no matter how good the content looks in a browser. The same wall stops the crawlers that feed classic search, the scrapers behind link previews, and every auditing tool you'll ever run. Whatever else is on your roadmap, readability comes first, because until it's fixed, nobody (human tools included) can even measure the rest.

Common mistakes

• "It works on my machine." Your browser runs JavaScript, carries your cookies, and has a human fingerprint. Crawlers have none of those. Test with curl or view-source:, never the rendered page.
• Allowlisting only Googlebot. Your WAF's "good bots" list may predate AI crawlers entirely. If being cited by AI answers matters to you, check who else is on it.
• Fixing robots.txt while the firewall still 403s. The two layers are independent; the stricter one wins.
• Prerendering only the homepage. The deep pages people actually search for stay invisible.

FAQ

Google indexes my site fine. Why did the audit fail?

Googlebot gets special treatment: bot walls verify and admit it, and Google renders JavaScript in a deferred second pass. Most other machine readers, including AI crawlers, get neither courtesy. Passing for Google alone means everything newer than Google can't see you.

We block bots on purpose. Is that wrong?

No. Some sites have good reasons to keep automated readers out. Just make it a decision instead of an accident: state it in robots.txt, know that it removes you from AI answers and previews, and expect audits from any tool to stay unscored.

I fixed it. What now?

Refresh the audit. If the page is readable, you'll get the full GEO, SEO, and technical report instead of this card, and a baseline to improve from.

Key takeaways

• "Page could not be read" means the response wasn't a readable HTML page: a blocked request, an empty JavaScript shell, or a non-HTML resource.
• The same wall that stops an auditor stops AI answer engines like ChatGPT, Claude, and Perplexity, which fetch raw HTML and don't run JavaScript.
• Check the raw response with view-source: or curl with a bot user agent; the browser view proves nothing about what machines see.
• Bot protection should challenge suspicious behavior and admit verified crawlers, not blanket-block every non-browser.
• Client-side-rendered apps need SSR, static generation, or prerendering before search and AI visibility can even be measured.
• Fix readability first: every other audit finding depends on it.