← AI crawler directory
AI answer
YouBot
You.comHonours robots.txt
Indexes pages for You.com's AI search answers. Small audience. Rarely worth blocking or optimising for.
User-agent string
Mozilla/5.0 (compatible; YouBot (+http://www.you.com))
How to verify it
You.com published ranges.
What blocking costs
Small audience. Rarely worth blocking or optimising for.
robots.txt rule
# Block YouBot
User-agent: YouBot
Disallow: /
# Allow everything else
User-agent: *
Allow: /Robots.txt is enough
You.com documents YouBot as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.
Generate a full robots.txt
Pick which AI crawlers to allow or block across all 42 agents and copy the file out.
Open the generator →