Nostimates
← All free tools

AI crawler robots.txt generator

This generator writes a robots.txt with an explicit rule per AI crawler. The important distinction it enforces: blocking a training crawler like GPTBot keeps you in ChatGPT answers, while blocking an answer indexer like OAI-SearchBot removes you from them entirely.

Block these crawlers

Your robots.txt

# Generated with nostimates.com/tools/robots-txt-generator

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Runs entirely in your browser

Nothing you type here is sent to us or stored anywhere. Copy the output and go.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects training data. ChatGPT's live citations come from OAI-SearchBot, which is a separate agent with a separate robots.txt token.
Can I block AI training but stay in Google AI Overviews?
Partly. Disallowing Google-Extended opts you out of Gemini training while keeping Googlebot indexing intact. AI Overviews are built from the Googlebot index, so you cannot appear there while blocking Googlebot.