Builds a structured knowledge graph sold to AI and data customers. It is operated by Diffbot.

Does Diffbot respect robots.txt?

Yes. Diffbot states that Diffbot honours robots.txt directives, so a Disallow rule for the Diffbot token is sufficient.

AI training

Diffbot

DiffbotHonours robots.txt

Builds a structured knowledge graph sold to AI and data customers. Feeds third-party knowledge graphs that some assistants consult.

Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)

How to verify it

Diffbot published IP ranges.

What blocking costs

Feeds third-party knowledge graphs that some assistants consult.

# Block Diffbot
User-agent: Diffbot
Disallow: /

# Allow everything else
User-agent: *
Allow: /

Robots.txt is enough

Diffbot documents Diffbot as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.

Generate a full robots.txt

Pick which AI crawlers to allow or block across all 42 agents and copy the file out.