CCBot: Common Crawl's open dataset crawler
CCBot belongs to Common Crawl. It builds a public archive of the web that anyone can download, and that many AI labs use as training data. Blocking it keeps your pages out of future versions of that dataset, and so out of models trained on it.
| Operator | Common Crawl |
|---|---|
| Used for | Common Crawl dataset |
| Type | Open dataset crawler |
| robots.txt name | CCBot |
| User agent | CCBot/2.0 (https://commoncrawl.org/faq/) |
| Published IP ranges | https://index.commoncrawl.org/ccbot.json |
| Blocked by top sites | 16.8% of 740 (study) |
| Official documentation | commoncrawl.org/ccbot |
Common Crawl is a non-profit that publishes an open crawl of the web. Many AI labs train on that dataset, which is why sites that opt out of AI training often block CCBot. Common Crawl warns that other crawlers falsely identify as CCBot.
Allow or block CCBot in robots.txt
To block it on the whole site:
User-agent: CCBot
Disallow: /To allow it explicitly (useful when a rule for all bots would block it):
User-agent: CCBot
Allow: /Common Crawl says CCBot checks robots.txt before fetching and obeys Crawl-delay.
robots.txt is a request, not a wall. A CDN or firewall rule can turn a bot away even when robots.txt allows it, and nothing in the file will show that.
The free check requests your home page as a browser and as AI bots, reads your robots.txt and shows what gets blocked. No sign-up.
Check my siteIs it really CCBot?
Anyone can send a request with this bot's name. Common Crawl publishes the IP ranges its requests come from, so a request that carries the name from another address is an impostor. You can check a request's IP address against that list.
See CCBot on your own site
CCBot doesn't run JavaScript, so analytics scripts never record it. AI Visibility in PageLens reads it from your server or CDN and shows which pages it requested, when, and which requests ended in a 404.