← AI crawler directory
SEO tool
SemrushBot
SemrushHonours robots.txt
Builds Semrush's backlink and keyword databases. Blocking hides your backlink profile from competitor research — and from your own.
User-agent string
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
How to verify it
Reverse DNS to semrush.com.
What blocking costs
Blocking hides your backlink profile from competitor research — and from your own.
robots.txt rule
# Block SemrushBot
User-agent: SemrushBot
Disallow: /
# Allow everything else
User-agent: *
Allow: /Robots.txt is enough
Semrush documents SemrushBot as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.
Generate a full robots.txt
Pick which AI crawlers to allow or block across all 42 agents and copy the file out.
Open the generator →