← AI crawler directory
AI training
AI2Bot
Allen Institute for AIHonours robots.txt
Collects openly licensed content for open research datasets. Non-commercial research. Often left allowed even on strict opt-out lists.
User-agent string
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)
How to verify it
AllenAI published ranges.
What blocking costs
Non-commercial research. Often left allowed even on strict opt-out lists.
robots.txt rule
# Block AI2Bot
User-agent: AI2Bot
Disallow: /
# Allow everything else
User-agent: *
Allow: /Robots.txt is enough
Allen Institute for AI documents AI2Bot as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.
Generate a full robots.txt
Pick which AI crawlers to allow or block across all 42 agents and copy the file out.
Open the generator →