Does blocking AI training bots hurt AI visibility? Search eligibility holds, but 'costs nothing' is overstated

The advice to block AI training bots and allow the answering bots, at no cost to AI visibility, is Overstated. OpenAI documents its training crawler and its search crawler as separate robots.txt switches, so the retrieval half holds OpenAI · 2024A. Google-Extended also controls grounding in Gemini Apps, so one of the three tokens usually named is not training-only Google · 2024A. The one outcome study finds the median GPTBot blocker at 0.003 ChatGPT citations per Google organic appearance, against 0.417 for sites allowing it cloro · 2026C. What leaving the training data costs has never been measured.

Blocking GPTBot or ClaudeBot leaves ChatGPT and Claude search eligibility intact

OpenAI documents GPTBot and OAI-SearchBot as independent robots.txt controls. GPTBot crawls content used to train foundation models, while OAI-SearchBot decides whether a site appears in ChatGPT search answers OpenAI · 2024A. The publisher FAQ states the same split: a site must allow OAI-SearchBot to appear in ChatGPT search, and disallowing GPTBot opts content out of training OpenAI · 2026A. Where both bots are allowed, OpenAI may crawl a page once and share the result between the two uses.

Anthropic runs the same design with three agents Anthropic · 2026A. ClaudeBot collects training data, Claude-SearchBot indexes for search and Claude-User fetches pages on demand, each blockable on its own. Apple's Applebot-Extended governs training for Apple's foundation models, while Applebot keeps feeding Apple search Apple · 2025A. For these three vendors, the retrieval half of the claim is first-party documented.

Blocking Google-Extended affects Gemini, not Google Search or AI Overviews

Google-Extended is a robots.txt control token with no separate crawler, and Google gives it two jobs Google · 2024A. It governs whether crawled content may train future Gemini models. It also governs grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. Grounding is retrieval at answer time, the half the claim treats as untouched.

Google states that the token does not affect inclusion or ranking in Google Search Google · 2024A. AI Overviews and AI Mode need nothing beyond standard indexing and snippet eligibility Google · 2025A. A Google-Extended block is therefore free in Search and AI Overviews, and not free in the Gemini app.

Removing a site from AI Overviews takes a different lever, a Search Console control that Google says leaves AI training to Google-Extended Google · 2026A. One quarterly crawler report still labels Google-Extended's role "Training opt-out", omitting grounding TechnologyChecker.io · 2026B. Of the three tokens in the usual snippet, that is the one the claim mislabels.

Sites that block GPTBot are cited far less by ChatGPT, for reasons the data cannot separate

The only study matching robots.txt rules to citation outcomes by token cuts against "costs nothing". Across 1,058 prominent domains, the median GPTBot blocker earned 0.003 ChatGPT citations per Google organic appearance, against 0.417 for allowers cloro · 2026C. It is one vendor, a two-day citation window and a correlational design: Tier C, treat as provisional.

The usual rebuttal is that blocklists also block OAI-SearchBot. The study's own block rates limit that reading, with 13.9% of domains blocking GPTBot at the root against 3.4% for OAI-SearchBot cloro · 2026C. At least three in four GPTBot blockers therefore left OAI-SearchBot open in robots.txt.

Site type explains part of the gap. GPTBot blockers were also cited about half as often by Perplexity, 0.566 against 1.143, an engine GPTBot does not feed cloro · 2026C. A halving on an unrelated engine points to who blocks, and the ChatGPT gap is more than a hundred times larger.

Edge rules are the other blind spot, because a robots.txt crawl cannot see them. Cloudflare's Training category judges multi-purpose crawlers by all their behaviours, so a training block also blocks them Lee · Cloudflare · 2026A. Site owners have reported verified Googlebot and Bingbot receiving HTTP 403 after setting that block, in a report that is anecdotal Montti · Search Engine Journal · 2026B.

Training-data visibility tracks AI search visibility in the first disclosed-method study

The claim assumes training data carries no visibility value, and the first disclosed-method evidence points the other way. Across 3,115 entity-question pairs on logged-out ChatGPT, search visibility rose with parametric visibility in every mention-rate bin Druck et al. · Graphite · 2026B. Parametric visibility is what a model says about an entity from its weights, without searching. The design is correlational, covers entity-comparison prompts only and does not test a block.

A network-traffic analysis shows one route for that link. In 21 of 27 ChatGPT conversations, the model's first search query already named brands the user never typed Mohanadasan · Snippet Digital; Keyword Insights · 2026C. Brands named that way reached the answer 68.9% of the time, against 2.1% for brands only fetched. It is a single-account Tier C study, with a reproducible mechanism and unreproduced magnitudes.

The retrieval path still carries most of the weight on discovery prompts. Across 34,960 unbranded observations, brands appeared in 2.8% of GPT answers when neither brand nor own domain reached retrieval Tannenbaum · Aiso Boost Ltd · 2026B. With the own domain cited, the figure rose to 49.0%. That study is observational and vendor-authored.

The opposite advice now circulates: allow GPTBot, ClaudeBot, Google-Extended and CCBot so models learn the brand. The paper behind it calls those tactics exploratory and its crawler advice inference, because training sets are undisclosed Druck et al. · Graphite · 2026B. No study shows that allowing training crawlers changes what a model says about a brand.

No study has yet measured what a training-only block costs a brand

The only causal estimate cannot isolate training. News publishers that blocked LLM crawlers lost about 7% of weekly visits within six weeks, an effect confined to the top 50 publishers Zhao · Rutgers Business School; The Wharton School, University of Pennsylvania · 2026A. Those blocks bundled training and retrieval agents before May 2024, and the paper is not peer reviewed.

The counterweight is weaker and points the other way. Across more than 6,000 publisher sites, Raptive found no statistically significant traffic change from blocking AI bots Griffin · Raptive · 2025C. It discloses no cohort sizes or test, and it never measured citations. Neither study covers brand or retail sites.

Timing makes the gap slow to close. Recent models reached users 2.4 to 4.6 months after their knowledge cutoff Druck et al. · Graphite · 2026B. The parametric study treats that gap as the minimum time for new content to reach a model's weights. A before-and-after test of a training block therefore has to span at least one model release.

Three designs would settle the question. One would match training-only blocks to citations, with edge rules observed and site type controlled. Another would track brand mentions with search switched off across a model release that followed a GPTBot block. A third would rerun the publisher analysis on post-2024 data, with GPTBot and OAI-SearchBot separated.