Definition

robots.txt

robots.txt is a root-level file that grants or denies crawler access per user-agent token. In the AI era it carries a distinction most sites get wrong: training crawlers and answer-engine indexers are separate agents.

Blocking GPTBot opts you out of OpenAI training but leaves ChatGPT citations intact; blocking OAI-SearchBot removes you from ChatGPT's citable index entirely.

Some agents — notably user-initiated fetchers — state that they are not governed by robots.txt, so enforcement there requires CDN rules.

Related terms

AI crawler llms.txt Training crawler

Nostimates measures robots.txt across twelve AI answer surfaces. Request access →

All terms

AEO (Answer Engine Optimisation)GEO (Generative Engine Optimisation)AI Overviews AI Mode Citation share n-sampling Query fan-out Grounding Share of voice (AI)Extraction success rate Surface Answer engine Citation Brand mention Mention rate Confidence interval Statistical significance Citation volatility Non-determinism Reproducibility Identity-free collection Personalisation Locale Residential proxy Bot detection Collection cost Credit Evidence retention Audit trail Parser version Canary prompt Data freshness Cadence Prompt set Head term Long-tail query Competitive set llms.txt AI crawler Training crawler Retrieval-augmented generation (RAG)Hallucination Answer box Featured snippet Zero-click search Entity Entity salience Knowledge graph Structured data JSON-LD schema.org FAQ schema Answer-first content Chunk Embedding Semantic search Prompt injection Market coverage Sentiment analysis Position tracking White-label data