← AI crawler directory
AI training
ClaudeBot
AnthropicHonours robots.txt
Crawls public content for Anthropic model training. Training-only. Blocking has no effect on Claude's live browsing of your pages.
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com
How to verify it
Anthropic publishes crawler IPs at claude.com/crawling/bots.json.
What blocking costs
Training-only. Blocking has no effect on Claude's live browsing of your pages.
robots.txt rule
# Block ClaudeBot
User-agent: ClaudeBot
Disallow: /
# Allow everything else
User-agent: *
Allow: /Robots.txt is enough
Anthropic documents ClaudeBot as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.
Generate a full robots.txt
Pick which AI crawlers to allow or block across all 42 agents and copy the file out.
Open the generator →Other Anthropic agents