← AI crawler directory
AI training
TikTokSpider
ByteDanceHonours robots.txt
Crawls web content for TikTok search and recommendation systems. Relevant if TikTok search is a discovery channel for your category.
User-agent string
Mozilla/5.0 (compatible; TikTokSpider; +https://www.tiktok.com)
How to verify it
ByteDance ASN checks.
What blocking costs
Relevant if TikTok search is a discovery channel for your category.
robots.txt rule
# Block TikTokSpider
User-agent: TikTokSpider
Disallow: /
# Allow everything else
User-agent: *
Allow: /Robots.txt is enough
ByteDance documents TikTokSpider as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.
Generate a full robots.txt
Pick which AI crawlers to allow or block across all 42 agents and copy the file out.
Open the generator →Other ByteDance agents