← AI crawler directory
AI training
cohere-training-data-crawler
CohereHonours robots.txt
Dedicated training-corpus crawler. Training-only, no live answer effect.
User-agent string
cohere-training-data-crawler/1.0
How to verify it
Cohere published ranges.
What blocking costs
Training-only, no live answer effect.
robots.txt rule
# Block cohere-training-data-crawler
User-agent: cohere-training-data-crawler
Disallow: /
# Allow everything else
User-agent: *
Allow: /Robots.txt is enough
Cohere documents cohere-training-data-crawler as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.
Generate a full robots.txt
Pick which AI crawlers to allow or block across all 42 agents and copy the file out.
Open the generator →Other Cohere agents