ClaudeBot: Anthropic's training crawler
ClaudeBot belongs to Anthropic. It downloads pages to build training data for AI models. Blocking it keeps your pages out of future training. It does not remove you from AI search results or from answers.
| Operator | Anthropic |
|---|---|
| Used for | Claude |
| Type | Training crawler |
| robots.txt name | ClaudeBot |
| User agent | Not published by the operator |
| Published IP ranges | https://claude.com/crawling/bots.json |
| Blocked by top sites | 13.6% of 740 (study) |
| Official documentation | support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from- |
Anthropic documents the robots.txt name but not a full user-agent string. It warns that blocking its IP addresses may not work as an opt-out, because the bot then can't read your robots.txt.
Allow or block ClaudeBot in robots.txt
To block it on the whole site:
User-agent: ClaudeBot
Disallow: /To allow it explicitly (useful when a rule for all bots would block it):
User-agent: ClaudeBot
Allow: /Anthropic says its bots honour robots.txt, including the non-standard Crawl-delay. The opt-out must be set on every subdomain.
robots.txt is a request, not a wall. A CDN or firewall rule can turn a bot away even when robots.txt allows it, and nothing in the file will show that.
The free check requests your home page as a browser and as AI bots, reads your robots.txt and shows what gets blocked. No sign-up.
Check my siteIs it really ClaudeBot?
Anyone can send a request with this bot's name. Anthropic publishes the IP ranges its requests come from, so a request that carries the name from another address is an impostor. PageLens checks every request from this bot against that list.
See ClaudeBot on your own site
ClaudeBot doesn't run JavaScript, so analytics scripts never record it. AI Visibility in PageLens reads it from your server or CDN and shows which pages it requested, when, and which requests ended in a 404.