Bytespider: ByteDance's crawler
Bytespider belongs to ByteDance. It crawls the web for its operator. The operator documents it as a search crawler; its use for AI training is widely reported but not documented. Blocking it keeps your pages out of whatever the operator builds from the crawl.
| Operator | ByteDance |
|---|---|
| Used for | Toutiao Search |
| Type | Crawler |
| robots.txt name | Bytespider |
| User agent | Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36 |
| Published IP ranges | None published |
| Blocked by top sites | 15.7% of 740 (study) |
| Official documentation | zhanzhang.toutiao.com/docs/intro/26899 |
ByteDance's only official page (in Chinese) describes Bytespider as the crawler for Toutiao Search. Its use for AI training is widely reported but not documented by ByteDance. ByteDance says genuine requests resolve by reverse DNS to hostnames under bytedance.com.
Allow or block Bytespider in robots.txt
To block it on the whole site:
User-agent: Bytespider
Disallow: /To allow it explicitly (useful when a rule for all bots would block it):
User-agent: Bytespider
Allow: /robots.txt is a request, not a wall. A CDN or firewall rule can turn a bot away even when robots.txt allows it, and nothing in the file will show that.
The free check requests your home page as a browser and as AI bots, reads your robots.txt and shows what gets blocked. No sign-up.
Check my siteIs it really Bytespider?
Anyone can send a request with this bot's name. ByteDance doesn't publish IP ranges for it, so the name alone can't prove a request is real.
See Bytespider on your own site
Bytespider doesn't run JavaScript, so analytics scripts never record it. AI Visibility in PageLens reads it from your server or CDN and shows which pages it requested, when, and which requests ended in a 404.