Question
Should I block Bytespider?
Bytespider is ByteDance's crawler, associated with data collection for its AI and TikTok ecosystem. Many sites block it because it can crawl aggressively and offers little Western search-visibility upside. Block it if it strains your origin or you do not want ByteDance training use; keep it allowed if you target ByteDance/TikTok-driven discovery.
Reasons teams block it
- Crawl load: Bytespider can hit sites frequently and add origin cost.
- Low visibility return for sites that do not target ByteDance products.
- Training opt-out: some publishers prefer not to feed ByteDance models.
Reasons to keep it
- You want reach in ByteDance/TikTok-adjacent surfaces.
- It behaves within your rate limits and adds no real cost.
Block it in robots.txt
User-agent: Bytespider
Disallow: /If it ignores robots.txt
Some operators report user-agents that do not fully honor robots.txt. If logs show continued hits after a Disallow, enforce at the edge (firewall or WAF rule) by user-agent and verified IP rather than relying on robots.txt alone.
Related pages: what is CCBot, verify AI crawler authenticity, AI crawler user agents.