There's a category of SEO problem that no amount of great content can overcome: the page is telling search engines not to list it at all. It's called a noindex directive, it's a single line of code, and when it lands on the wrong page it makes that page effectively invisible: no ranking, no traffic, no matter how good it is.
The maddening part is how easily it happens by accident. So this one's worth checking with your own eyes.
What "indexable" means
A page is indexable when search engines are allowed to add it to their index, the giant catalog they search through to answer queries. If a page isn't indexed, it can't rank, full stop.
The main thing that blocks indexing is a noindex directive, and it can live in two places:
A meta robots tag in the page's <head>:
<meta name="robots" content="noindex" />
An HTTP header sent with the page (easy to miss because you can't see it in the rendered page):
X-Robots-Tag: noindex
Our check looks at both, including the header, which a lot of tools quietly ignore. A page can look perfectly fine in the browser while a response header is telling Google to drop it.
How pages get noindexed by accident
Almost nobody noindexes their money pages on purpose. It happens through:
- A launch leftover. The site was built behind a sitewide
noindexso it wouldn't get indexed early, and someone forgot to remove it at launch. Entire sites have sat invisible for months this way. - A staging config that shipped. The staging environment noindexes everything (correctly), and that setting rode along to production.
- An SEO plugin toggle. A per-page or per-template "hide from search engines" checkbox gets ticked, sometimes by a different team member.
- A CMS default that noindexes certain page types (tag archives, author pages), fine until it catches a page you actually want indexed.
How to check and fix it
- View the page source and search for
noindex. If it's in a<meta name="robots">tag and the page should rank, remove it. - Check the response headers (your browser's network tab, under the document request) for
X-Robots-Tag: noindex. This is the sneaky one: fix it at the server, CDN, or framework level. - Confirm in Search Console. Google's URL Inspection tool tells you whether a page is indexed and why not. It's the source of truth.
- Don't confuse this with robots.txt.
robots.txtcontrols crawling (whether bots fetch the page); noindex controls indexing (whether it's listed). They're different levers, as the next section explains.
noindex vs robots.txt vs canonical
These three get tangled constantly:
- noindex says "you may read this page, but don't list it." Use it for thin or private pages you don't want in search.
- robots.txt
Disallowsays "don't even fetch this." Confusingly, a disallowed page can sometimes still appear in results without a description, and if you block crawling, Google can't even see a noindex tag to obey it. Don't block a page in robots.txt if your goal is to noindex it. - canonical says "this is a duplicate of another page" (see canonical URLs).
For AI specifically, indexability is separate again from whether you allow AI crawlers. You can be fully indexable in Google and still block GPTBot, or vice versa. Check both.
When noindex is the right call
Plenty of pages should be noindexed: thank-you pages, internal search results, thin tag archives, duplicate filter views, staging. The goal isn't "index everything." It's "make sure the pages you want found aren't accidentally hidden." Be deliberate either way.
Common mistakes
- A sitewide noindex left over from launch or staging. Check this first if traffic mysteriously never arrives.
X-Robots-Tag: noindexin the headers that no one sees because they only check the HTML.- Blocking a page in robots.txt and adding noindex. The block stops Google reading the noindex, so it can backfire.
- Noindexing pages you meant to rank via an overzealous plugin or CMS default.
FAQ
What's the difference between noindex and robots.txt?
robots.txt controls whether bots crawl a page; noindex controls whether it gets listed. To keep a page out of search, let it be crawled but add noindex; don't block it in robots.txt.
Why is my page not showing in Google even though it loads fine?
The most common cause is a noindex directive in the meta tag or the X-Robots-Tag header. Check both, then confirm in Search Console.
Can a page be blocked by an HTTP header I can't see?
Yes. X-Robots-Tag: noindex travels in the response headers, invisible in the rendered page. Check your browser's network tab.
Should I noindex anything on purpose?
Yes: thin, private, or duplicate pages (thank-you pages, internal search, staging). Just be deliberate about which.
Key Takeaways
- A
noindexdirective keeps a page out of the search index entirely, so it cannot rank or get traffic no matter how good the content is. - noindex can live in two places: a
<meta name="robots" content="noindex">tag in the<head>or anX-Robots-Tag: noindexHTTP response header that is invisible in the rendered page and missed by many tools. - Most accidental noindexing comes from a launch or staging leftover, a shipped staging config, an SEO plugin toggle, or a CMS default, so a sitewide noindex is the first thing to check when traffic never arrives.
- noindex controls whether a page is listed while robots.txt controls whether it is crawled; blocking a page in robots.txt stops Google from reading its noindex tag, so the two should not be combined to hide a page.
Want to know if any important pages are accidentally set to noindex, in the tag or the headers? Run a free audit. More in the SEO explainers.
