← AI crawler directory
SEO tool
AhrefsBot
AhrefsHonours robots.txt
Builds the Ahrefs backlink index. Same trade-off as SemrushBot: blocking cuts both ways.
User-agent string
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
How to verify it
Reverse DNS to ahrefs.com.
What blocking costs
Same trade-off as SemrushBot: blocking cuts both ways.
robots.txt rule
# Block AhrefsBot
User-agent: AhrefsBot
Disallow: /
# Allow everything else
User-agent: *
Allow: /Robots.txt is enough
Ahrefs documents AhrefsBot as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.
Generate a full robots.txt
Pick which AI crawlers to allow or block across all 42 agents and copy the file out.
Open the generator →