← AI crawler directory
AI training
GPTBot
OpenAIHonours robots.txt
Crawls public pages to build training corpora for OpenAI foundation models. Blocking it removes your content from future training runs. It does not affect whether ChatGPT can cite you live — that is OAI-SearchBot.
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
How to verify it
Reverse DNS to openai.com plus OpenAI's published IP ranges at openai.com/gptbot.json.
What blocking costs
Blocking it removes your content from future training runs. It does not affect whether ChatGPT can cite you live — that is OAI-SearchBot.
robots.txt rule
# Block GPTBot
User-agent: GPTBot
Disallow: /
# Allow everything else
User-agent: *
Allow: /Robots.txt is enough
OpenAI documents GPTBot as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.
Generate a full robots.txt
Pick which AI crawlers to allow or block across all 42 agents and copy the file out.
Open the generator →Other OpenAI agents