Which AI crawlers to block, and which blocks quietly cost you money
Blocking GPTBot opts you out of OpenAI model training and has no effect on whether ChatGPT can cite you. Blocking OAI-SearchBot removes you from ChatGPT's citable index entirely. Most robots.txt files we audit block the wrong one, or both, without intending to.
Three categories, three consequences
Training crawlers — GPTBot, ClaudeBot, CCBot, meta-externalagent. They collect content into training corpora. Blocking them affects future model knowledge: slow, diffuse, hard to measure. This is a rights decision more than a visibility one.
Answer indexers — OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot. They build the indexes that answer engines cite from. Blocking them removes you from those answers. This is a visibility decision with immediate, measurable cost.
Agent fetchers — ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher. They fetch a specific URL because a user asked about it. Blocking them breaks the experience for a user who is actively trying to read your page.
The rules people get wrong
"We blocked GPTBot so we're out of ChatGPT." No. Different agent. Your content can still be cited live via OAI-SearchBot.
"We want out of AI training but to stay in AI Overviews." Disallow Google-Extended. It's a control token, not a crawler, and it separates Gemini training from Search indexing. You cannot, however, appear in AI Overviews while blocking Googlebot — they come from the same index.
"Robots.txt handles it." Not always. Perplexity-User and the Internet Archive both state they are not governed by robots.txt. Enforce those at the CDN.
"The user agent said it was Googlebot." Strings are trivially spoofed, and Bytespider is among the most impersonated agents on the web. Verify by reverse DNS or the operator's published IP ranges before acting on a block or an allow.
Audit before you copy
A sane default
Allow answer indexers and agent fetchers. Decide on training crawlers as a policy question with legal, not as an SEO tactic. Then verify with your logs a week later, because a spoofed agent will keep arriving regardless of what the file says.
Our AI crawler directory lists every agent with its token, verification method and the specific cost of blocking it, and the robots.txt generator writes the file with the categories kept separate.
See this for your brand
Nostimates shows you how your brand shows up across twelve AI engines and search — in a dashboard, or as data in your own tools. Tell us what you want to measure.