Definition

AI crawler

An AI crawler is an automated agent that fetches web pages on behalf of an AI system, whether for model training, answer-engine indexing, or an on-demand user request.

The three categories behave differently and should be governed differently. Treating them as one bucket is how sites accidentally remove themselves from AI answers.

User-agent strings are trivially spoofed, so verification should use reverse DNS or the operator's published IP ranges rather than the string alone.

Nostimates measures ai crawler across twelve AI answer surfaces. Request access →

All terms

AEO (Answer Engine Optimisation)GEO (Generative Engine Optimisation)AI Overviews AI Mode Citation share n-sampling Query fan-out Grounding Share of voice (AI)Extraction success rate Surface Answer engine Citation Brand mention Mention rate Confidence interval Statistical significance Citation volatility Non-determinism Reproducibility Identity-free collection Personalisation Locale Residential proxy Bot detection Collection cost Credit Evidence retention Audit trail Parser version Canary prompt Data freshness Cadence Prompt set Head term Long-tail query Competitive set llms.txt robots.txt Training crawler Retrieval-augmented generation (RAG)Hallucination Answer box Featured snippet Zero-click search Entity Entity salience Knowledge graph Structured data JSON-LD schema.org FAQ schema Answer-first content Chunk Embedding Semantic search Prompt injection Market coverage Sentiment analysis Position tracking White-label data