AI bots directory

The crawlers and agents behind ChatGPT, Claude, Perplexity, Gemini and other AI tools: what each one does, how to allow or block it in robots.txt, and how many of the world's top websites block it. Facts come from each operator's own documentation; the blocking numbers are from our study of 740 top websites (October 2026).

Bot	Operator	Type	Blocked by top sites
`GPTBot`	OpenAI	Training crawler	14.7%
`OAI-SearchBot`	OpenAI	AI search crawler	7.8%
`ChatGPT-User`	OpenAI	Fetch for a person	9.7%
`ClaudeBot`	Anthropic	Training crawler	13.6%
`Claude-SearchBot`	Anthropic	AI search crawler	—
`Claude-User`	Anthropic	Fetch for a person	8%
`PerplexityBot`	Perplexity	AI search crawler	10.9%
`Perplexity-User`	Perplexity	Fetch for a person	8.4%
`Google-Extended`	Google	robots.txt token	13.2%
`Applebot-Extended`	Apple	robots.txt token	9.3%
`meta-externalagent`	Meta	Training crawler	12.6%
`CCBot`	Common Crawl	Open dataset crawler	16.8%
`Bytespider`	ByteDance	Crawler	15.7%
`Amazonbot`	Amazon	Training crawler	—
`DuckAssistBot`	DuckDuckGo	Fetch for a person	—
`MistralAI-User`	Mistral	Fetch for a person	—

Three kinds of AI bots

Training crawlers (GPTBot, ClaudeBot) collect pages for model training. AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index assistants search. Fetches for a person (ChatGPT-User, Claude-User, Perplexity-User) open one page because someone asked about it. They are controlled separately in robots.txt, so you can stay out of training and still appear in AI answers.

Can this bot reach your site?

The free check requests your home page as a browser and as AI bots, reads your robots.txt and shows what gets blocked. No sign-up.

Check my site

See what they do on your site

None of these bots run JavaScript, so analytics scripts never see them. AI Visibility in PageLens reads them from your server: which pages AI crawlers read, which ones assistants open to answer people, and the visitors they send.