Definition

Training crawler

A training crawler gathers web content into corpora used to train foundation models. GPTBot, ClaudeBot, CCBot and meta-externalagent are examples.

Blocking training crawlers affects future model knowledge, not live answers. The effect is slow, diffuse and hard to measure.

The opt-out decision is more often about rights and policy than about visibility, and should be made on that basis rather than as an SEO tactic.

Related terms

AI crawler robots.txt Retrieval-augmented generation (RAG)

Nostimates measures training crawler across twelve AI answer surfaces. Request access →

All terms

AEO (Answer Engine Optimisation)GEO (Generative Engine Optimisation)AI Overviews AI Mode Citation share n-sampling Query fan-out Grounding Share of voice (AI)Extraction success rate Surface Answer engine Citation Brand mention Mention rate Confidence interval Statistical significance Citation volatility Non-determinism Reproducibility Identity-free collection Personalisation Locale Residential proxy Bot detection Collection cost Credit Evidence retention Audit trail Parser version Canary prompt Data freshness Cadence Prompt set Head term Long-tail query Competitive set llms.txt robots.txt AI crawler Retrieval-augmented generation (RAG)Hallucination Answer box Featured snippet Zero-click search Entity Entity salience Knowledge graph Structured data JSON-LD schema.org FAQ schema Answer-first content Chunk Embedding Semantic search Prompt injection Market coverage Sentiment analysis Position tracking White-label data