The AI crawler directory

Fetches a page on demand because a user or a ChatGPT action asked for that specific URL.

honours robots.txt

Fetches a page because a Claude user's request required it.

honours robots.txt

Fetches sites on behalf of Vertex AI customers building grounded agents.

honours robots.txt

Fetches a specific page because a user's query required it.

ignores robots.txt

Fetches specific URLs on behalf of Meta AI user requests.

honours robots.txt

Fetches pages on behalf of Le Chat users.

honours robots.txt

Builds the search index ChatGPT uses to cite live sources in answers.

honours robots.txt

Crawls the web to improve the quality of Claude's search-grounded answers.

honours robots.txt

Indexes pages so Perplexity can cite them.

honours robots.txt

Indexes pages for You.com's AI search answers.

honours robots.txt

Supports DuckDuckGo's AI-assisted answers.

honours robots.txt

We collect AI answers from the engines themselves. We do not crawl customer or third-party websites.

honours robots.txt

Crawls public pages to build training corpora for OpenAI foundation models.

honours robots.txt

Crawls public content for Anthropic model training.

honours robots.txt

A robots.txt control token governing whether your content trains Gemini and grounds Vertex AI, without affecting Search.

honours robots.txt

Robots.txt token controlling whether Apple uses your content to train Apple Intelligence models.

honours robots.txt

Collects public web data to train Meta AI models.

honours robots.txt

Collects training data for ByteDance models including Doubao.

honours robots.txt

Crawls web content for TikTok search and recommendation systems.

honours robots.txt

Crawls public content to improve Amazon products and services, and may be used to train Amazon's AI models. (Alexa's on-demand answering uses a separate agent, Amzn-User.)

honours robots.txt

Crawls content for Cohere model training and retrieval products.

honours robots.txt

Dedicated training-corpus crawler.

honours robots.txt

Builds the Common Crawl corpus, which underpins a large share of every major model's training data.

honours robots.txt

Collects web data resold as training and market-intelligence datasets.

honours robots.txt

Builds a structured knowledge graph sold to AI and data customers.

honours robots.txt

Third-party bot lists attribute this user agent to DeepSeek, but DeepSeek publishes no official crawler documentation, declared user agent or IP ranges — treat it as unverified.

honours robots.txt

Collects openly licensed content for open research datasets.

honours robots.txt

Builds a decentralised index sold for AI use.

honours robots.txt

Preserves pages in the Wayback Machine.

ignores robots.txt

Builds Semrush's backlink and keyword databases.

honours robots.txt

Builds the Ahrefs backlink index.

honours robots.txt

Google's primary crawler. Feeds classic Search, AI Overviews and AI Mode from the same index.

honours robots.txt

Microsoft's search crawler, feeding Bing and by extension Copilot answers.

honours robots.txt

Legacy Microsoft crawler, largely superseded by Bingbot but still seen in logs.

honours robots.txt

Powers Siri, Spotlight and Safari suggestions.

honours robots.txt

Baidu's primary crawler, feeding both classic results and ERNIE AI answers.

honours robots.txt

Crawls for Petal Search and Huawei assistant features.

honours robots.txt

Crawls to build the index for Shenma (SM.CN), the mobile search engine from UCWeb and Alibaba.

honours robots.txt

Crawls for Sogou search and Tencent-adjacent AI features.

honours robots.txt

Naver's crawler, feeding Korea's default search and its AI answer surfaces.

honours robots.txt

Crawls for Yandex Search and its Alice assistant answers.

honours robots.txt

Renders Open Graph link previews when a URL is shared.

ignores robots.txt