← AI crawler directory
AI agent
Google-CloudVertexBot
GoogleHonours robots.txt
Fetches sites on behalf of Vertex AI customers building grounded agents. Affects third-party Vertex AI applications only, not Google Search or AI Overviews.
User-agent string
Mozilla/5.0 (compatible; Google-CloudVertexBot/1.0; +https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
How to verify it
Google's published crawler IP ranges.
What blocking costs
Affects third-party Vertex AI applications only, not Google Search or AI Overviews.
robots.txt rule
# Block Google-CloudVertexBot
User-agent: Google-CloudVertexBot
Disallow: /
# Allow everything else
User-agent: *
Allow: /Robots.txt is enough
Google documents Google-CloudVertexBot as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.
Generate a full robots.txt
Pick which AI crawlers to allow or block across all 42 agents and copy the file out.
Open the generator →Other Google agents