robots.txt used to be a file nobody argued about. Now legal wants every AI bot blocked, marketing wants the company inside ChatGPT answers, and the person who edits the file has a plugin with a single toggle that says “block AI crawlers”. That toggle is how a site disappears from ChatGPT search while its content stays in the training data anyway. The four crawlers that matter each control a different thing. The companies that run them document exactly what.
The rule: allow the search agents, decide about training separately, and do not expect llms.txt to do either.
Who crawls, and what each one controls
| User agent | Run by | What it is for | What blocking it does |
|---|---|---|---|
| GPTBot | OpenAI | Training foundation models | Content is not used for training. No effect on ChatGPT search results |
| OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT search | Site is not shown in ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetching a page when a user asks about it | Not used to decide search appearance. User-initiated, so robots.txt may not apply |
| Google-Extended | Gemini training and grounding | No effect on inclusion or ranking in Google Search | |
| Googlebot | Search, including AI Overviews and AI Mode | Out of Search entirely | |
| PerplexityBot | Perplexity | Surfacing sites in Perplexity results | Not cited in Perplexity answers |
| Perplexity-User | Perplexity | Fetching a page a user asked about | Generally ignores robots.txt |
Every line in that table comes from the OpenAI, Google and Perplexity crawler documentation. None of it is inference.
Decision 1: training is separate from being found
Blocking GPTBot and Google-Extended keeps your pages out of the next model’s training set. It does not remove you from ChatGPT search or from Google, because those run on different agents. So the decision is a licensing decision. Legal can make it without touching visibility.
The cost is real but unmeasurable: a model that never saw your site in training knows less about you when it answers without searching. For a publisher whose content is the product, blocking training can be the right call. For a company that sells something and wants to be recommended, I have not seen a case where it was.
Decision 2: never block the search agents
OAI-SearchBot and PerplexityBot are the ones that decide whether you appear inside answers. Blocking them is the equivalent of blocking Googlebot. It is what the “block all AI bots” toggle does. Several WordPress security and SEO plugins ship that toggle. I find it switched on in sites whose owners are paying for AI visibility work at the same time.
The check takes one minute: open /robots.txt and look for OAI-SearchBot and PerplexityBot under a Disallow. If the file blocks a wildcard group that these agents fall into, the same applies.
The other cost is on your side. A large site with thin sections spends crawl on pages that will never be cited. The AI search agents have less patience than Googlebot. The pruning logic is the same as for crawl budget: fewer pages, each worth fetching.
Decision 3: Google’s AI features are not controlled by Google-Extended
This is the one most often gotten wrong. Google-Extended controls Gemini training and grounding in Google’s other products. It has no effect on AI Overviews or AI Mode in Search. Google’s own documentation on AI features says the controls for those are the Search controls: nosnippet, data-nosnippet, max-snippet and noindex.
Since 31 August 2026 there is a separate switch for that: an AI controls setting in Search Console that removes a site from AI Overviews, AI Mode and the AI features in Discover. Google’s help page says it is not used as a ranking or inclusion signal elsewhere in Search, which is what the snippet controls could never offer. It also removes your links from inside the answers, and for most companies those links are the visibility they were paying for. The opt-out exists now. The reason to use it still almost never does.
What llms.txt does not do
llms.txt is a proposal from September 2024: a markdown file at the site root that summarises the site for language models. It is a reasonable idea and a well-written spec.
It is also, as of today, unsupported. None of the three crawler documentation pages above mention reading it. The confusion comes from the labs publishing llms.txt files for their own developer docs, which the proposal’s page lists as adoption. Publishing a file and reading it from other sites are different things. Only the first is documented.
I have added it to sites when a client asked. It costs ten minutes and does no harm. What it does not do is change what an engine says about you. A plan that has llms.txt as its first line is a plan that has not looked at where answers come from. That is the same trap as optimising for the word “AI” in what AI startups rank for: visible effort, no mechanism.
What does change the answer
The agents fetch pages. What they take from them is the sentence that answers the question, an entity that reads the same everywhere and the third-party pages that mention you. The method for checking all three is in how to measure AI search visibility.
So the order is: fix robots.txt so the search agents can get in, make the training decision on its own terms and then spend the time on what the agents read.
The block you cannot see in robots.txt
On 31 October 2026 I tested fifteen crawler user agents against this site. Fourteen returned 200. ClaudeBot, the crawler that builds Anthropic’s search index, returned 403 from nginx on every URL, including robots.txt. The file said allow all. The block sat at the hosting level, in a rule no setting in the control panel showed, and it took a support ticket to remove. If one engine never finds you and the others do, test the user agent before you touch the file.
Where this sits
The robots.txt review is the first hour of AI search optimization work, because nothing after it matters if the agents are blocked. If you are not sure what your file says today, tell me what your robots.txt says and I will tell you which of the three decisions it has already made for you.