There's a setting on your site that decides whether AI answer engines can read you at all, and most owners have never looked at it. It's robots.txt, a plain text file at your domain root, and a single line in it can quietly remove you from the places people now go to ask questions.
Worth a look, then.
What these crawlers are
When you ask Perplexity "what's the best X for Y," it doesn't think from memory. It fetches live pages, reads them, and writes an answer with citations. The thing doing the fetching is a bot with a name, and robots.txt is where you tell named bots what they may and may not touch.
The ones worth knowing right now:
- GPTBot: OpenAI's crawler for training and, increasingly, for grounding answers.
- OAI-SearchBot: OpenAI's crawler specific to ChatGPT search.
- ClaudeBot: Anthropic's crawler.
- PerplexityBot: Perplexity's crawler.
- Google-Extended: controls whether Google uses your content for its AI features (separate from normal Google indexing).
- CCBot, Bytespider, Amazonbot: Common Crawl and others that feed a long tail of models.
Block these and you're invisible in those engines, the same way blocking Googlebot would make you vanish from Google.
The trade-off, stated plainly
Here's the honest tension. Some businesses deliberately block AI crawlers because they don't want their content used to train models they get nothing back from. That's a legitimate stance, especially for original journalism or paid research.
But for most sites (a SaaS product, a local service, a store, a blog that wants readers) blocking these bots is self-defeating. You're not protecting a moat. You're opting out of discovery. If your goal is to be the source an AI answer cites, you have to let the AI read you. You can't be quoted from a page nobody's allowed to fetch.
So the question isn't "are AI crawlers good or bad." It's "do I want to appear in AI answers?" If yes, allow them. If you genuinely don't, block them on purpose, not by accident, which is how most blocks happen.
How to check and fix it
Open https://yourdomain.com/robots.txt in a browser. You're looking for Disallow rules attached to the bots above. A block looks like this:
User-agent: GPTBot
Disallow: /
To welcome them instead, either say nothing about them (the default is "allowed") or be explicit:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
A subtle gotcha: crawlers obey the most specific group that matches their name. If you have a permissive User-agent: * block but a separate User-agent: GPTBot group, GPTBot follows its own group and ignores the wildcard. So if you add named groups, make sure each one still carries any Disallow rules you actually want enforced. Otherwise naming a bot can accidentally grant it access to admin or checkout paths you meant to keep private.
After editing, re-fetch the file and read it back. It's easy to leave a stray Disallow: / from a staging config that's been quietly killing you for months.
Don't confuse this with blocking by accident
The most common pattern we see isn't a deliberate block. It's a leftover. A site launches behind Disallow: /, goes live, and nobody removes the line. Or a security plugin adds aggressive bot rules. Or a CDN's "block AI scrapers" toggle gets flipped on without anyone realizing it also blocks the engines that send traffic.
If you're not sure, assume nothing and check the live file today.
How this differs from regular SEO robots rules
Allowing AI crawlers is related to, but separate from, your normal robots.txt and indexability setup. Googlebot reads your pages for the classic search index; GPTBot and friends read them for answers. You can be perfectly indexable in Google and still be blocked from every AI engine, or vice versa. Both matter now, so check both.
FAQ
Does blocking AI crawlers protect my content from being copied?
Not really. It stops cooperative, named bots that respect robots.txt, but it doesn't stop bad actors who ignore it. Mostly it just removes you from the engines that would have cited and linked you.
If I allow AI bots, will my content be used for training?
Possibly, depending on the bot. Some crawlers are for answering, some for training, and a few do both. If training specifically is your concern, you can allow answer-focused bots (like OAI-SearchBot and PerplexityBot) while blocking training-focused ones. Read each bot's docs.
What's the safest default for a normal business site?
Allow the major AI crawlers. The discovery upside almost always outweighs the downside for sites that want to be found.
Will this slow my server down?
These crawlers are polite and infrequent compared to your real traffic. If volume ever becomes an issue, you can rate-limit rather than block outright.
Key Takeaways
robots.txtat your domain root controls whether named AI crawlers like GPTBot, ClaudeBot, OAI-SearchBot, and PerplexityBot can read your pages; aDisallow: /line under a bot's name removes you from that engine entirely.- Blocking an AI crawler does not protect content from bad actors who ignore
robots.txt; it mostly removes you from the engines that would have cited and linked you. - For most business sites, allowing the major AI crawlers is the safest default because the discovery upside outweighs the downside.
- Crawlers obey the most specific
User-agentgroup that matches their name, so a named group can override a wildcardDisallowand accidentally expose private paths. Google-Extendedcontrols Google's AI use of your content separately from normal Googlebot indexing, so a page can be indexed in Google search yet blocked from AI answers.
Not sure what your robots.txt is currently telling these bots? Run a free audit: we report exactly which AI crawlers your site allows and blocks, bot by bot. More in the GEO explainers.
