Nostimates
← AI crawler directory

Search

Googlebot

GoogleHonours robots.txt

Google's primary crawler. Feeds classic Search, AI Overviews and AI Mode from the same index. Blocking Googlebot removes you from Search and from both Google AI surfaces. There is no way to appear in AI Overviews while blocking Googlebot.

User-agent string

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

How to verify it

Reverse DNS to googlebot.com then forward-confirm the IP.

What blocking costs

Blocking Googlebot removes you from Search and from both Google AI surfaces. There is no way to appear in AI Overviews while blocking Googlebot.

robots.txt rule

# Block Googlebot
User-agent: Googlebot
Disallow: /

# Allow everything else
User-agent: *
Allow: /

Robots.txt is enough

Google documents Googlebot as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.

Generate a full robots.txt

Pick which AI crawlers to allow or block across all 42 agents and copy the file out.

Open the generator →