Robots.txt and AI Crawlers: Is 'Disallow All' Killing Your AI Traffic?

A single leftover line in robots.txt can quietly remove your site from AI search. If Disallow: / is sitting under User-agent: *, you are telling ChatGPT's crawler, Perplexity's crawler, Google's crawler, and the rest to skip your pages, which means they have nothing to cite when someone asks about your category. The fix usually takes two minutes. The hard part is knowing which crawlers matter and which rules actually do what people think they do.

This guide covers what robots.txt does and does not control for AI, the crawlers worth knowing by name, and two copy-paste configurations: one for maximum AI visibility, and one for blocking model training while staying in AI search.

Key takeaways

Yes, more than most site owners realize. The major AI companies run named crawlers and, for the most part, respect robots.txt. When OpenAI's OAI-SearchBot or Perplexity's PerplexityBot is disallowed, it does not fetch your pages, so your content cannot become a cited source in those answers. Blocking is the difference between being in the running and not existing as far as the engine is concerned.

Two clarifications matter, because they are where people go wrong.

First, robots.txt is about crawling, not indexing or training in the broad sense. It asks a crawler not to fetch listed paths. It cannot remove what a model already absorbed during earlier training, and it cannot stop third-party datasets, like Common Crawl, that may have copied your site before you added any rules. If your goal is to keep specific pages out of results entirely, robots.txt is the wrong tool on its own (more on that below).

Second, robots.txt is a request, not a wall. Reputable crawlers honor it. Some scrapers ignore it. For the bots that feed mainstream AI answers, though, compliance is the norm, so the file is the right first lever to check.

In the late 1990s, some publishers blocked early search engine crawlers because they worried about losing control of their content. It backfired. The sites that let Google in captured the traffic; the ones that walled themselves off faded from the web's main discovery layer.

AI search is the same decision arriving again. Buyers increasingly start in ChatGPT, Perplexity, Gemini, and Google's AI Overviews instead of a list of blue links. If your robots.txt keeps those crawlers out, you are repeating the 1990s mistake with a new set of bots. The upside of getting it right is that AI answers now drive measurable referral traffic and shape first impressions, so being readable is the price of entry.

Different bots do different jobs. Some crawl to train models, some index pages so an assistant can cite them, and some fetch a single page live when a user asks about it. Blocking the wrong one has consequences you may not intend. Here are the ones that matter most in 2026.

User-agent names and behaviors change. Treat this as a starting map, not a permanent spec, and check each company's published crawler docs before writing strict rules.

The single most expensive mistake here involves Google. People assume there is a dedicated "AI Overviews bot" they can block to stay out of AI answers while keeping normal search. There is not. AI Overviews and AI Mode are features of Google Search, and they use the same Googlebot. Disallow Googlebot and you vanish from both at once.

Google-Extended is a separate control, and it is narrower than its name suggests. It only governs whether your content trains Gemini models and grounds Vertex AI responses. It does not affect crawling for Search, ranking, or AI Overviews. So blocking Google-Extended is a training opt-out, not an AI-answers opt-out. If you want to stay visible in Google's AI answers, leave Googlebot allowed and decide on Google-Extended separately.

If you want AI engines to read, cite, and send traffic to your site, allow crawling by default and block only the paths that should never be public. This is the right setup for most businesses.

That is the whole pattern. No standalone Disallow: /, a clear sitemap pointer, and a short blocklist for areas that add nothing to an AI answer. Anything not disallowed is fair game, which is what you want.

Some publishers are fine appearing in AI answers but do not want their writing used to train models. You can split the difference, because training crawlers and search crawlers are usually different bots. Allow the search and live-fetch crawlers, and disallow the training ones.

This keeps you eligible for citations in ChatGPT search, Perplexity, and Google's AI answers while telling the main training crawlers to stay out. Two honest caveats: it only affects bots that comply, and it does nothing about content already collected. It is a forward-looking signal, not a delete button.

A subtle rule makes this work. When a crawler finds more than one matching group, it obeys the most specific one. GPTBot follows its own GPTBot block and ignores the User-agent: * group entirely. That is why naming bots explicitly is the only reliable way to give different crawlers different rules.

Getting crawlers in the door is necessary but not sufficient. Once your pages are readable, a few other layers decide whether AI engines actually cite you.

Use noindex (a meta robots tag or X-Robots-Tag header), not robots.txt, when you want a page kept out of results entirely. Remember the order of operations: the page must be crawlable for the crawler to read the noindex.

Make your content easy for a model to lift and trust. That is the work of answer engine optimization: clear answers near the top of the page, clean structure, and consistent facts. Structured data removes ambiguity about what a page is, and the debate over whether llms.txt files matter for AEO is worth reading before you add one. To turn crawlability into actual mentions, see how to earn AI citations.

Finally, measure it. The only way to know whether AI engines now read and cite you is to check the answers themselves, on a schedule, across the engines your buyers use. Our guide on tracking your brand in AI search walks through the method, and Elmo is an open-source tool built to automate that loop across ChatGPT, Perplexity, Gemini, Claude, and Google's AI Overviews. Fixing robots.txt gets you in the room. Tracking tells you whether it worked.