← AI crawler directory
AI training
CCBot
Common CrawlHonours robots.txt
Builds the Common Crawl corpus, which underpins a large share of every major model's training data. Blocking CCBot is the broadest single training opt-out available, because so many labs derive from Common Crawl.
User-agent string
CCBot/2.0 (https://commoncrawl.org/faq/)
How to verify it
Reverse DNS to commoncrawl.org infrastructure on AWS.
What blocking costs
Blocking CCBot is the broadest single training opt-out available, because so many labs derive from Common Crawl.
robots.txt rule
# Block CCBot
User-agent: CCBot
Disallow: /
# Allow everything else
User-agent: *
Allow: /Robots.txt is enough
Common Crawl documents CCBot as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.
Generate a full robots.txt
Pick which AI crawlers to allow or block across all 42 agents and copy the file out.
Open the generator →