← AI crawler directory
AI training
Bytespider
ByteDanceHonours robots.txt
Collects training data for ByteDance models including Doubao. Heavy crawler with a reputation for aggressive rates. Rate-limit before you block outright if you care about Doubao visibility.
User-agent string
Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com)
How to verify it
Difficult — Bytespider is among the most spoofed agents. Verify by ASN, not string.
What blocking costs
Heavy crawler with a reputation for aggressive rates. Rate-limit before you block outright if you care about Doubao visibility.
robots.txt rule
# Block Bytespider
User-agent: Bytespider
Disallow: /
# Allow everything else
User-agent: *
Allow: /Robots.txt is enough
ByteDance documents Bytespider as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.
Generate a full robots.txt
Pick which AI crawlers to allow or block across all 42 agents and copy the file out.
Open the generator →Other ByteDance agents