AI Crawler Checker
Check which AI crawlers and assistants (ChatGPT, Claude, Perplexity, Gemini and more) are allowed to read your website, based on its robots.txt. Free, no sign-up.
Why it matters
AI assistants are a new traffic source: people ask ChatGPT or Perplexity and click through to the sites they cite. Blocking their user agents (ChatGPT-User, Claude-User, Perplexity-User) can cut you out of those answers; blocking training crawlers (GPTBot, ClaudeBot, CCBot) keeps your content out of future models. This tool shows what your robots.txt currently says for each of them.
How many visitors do AI assistants actually send you?
robots.txt only tells you who may visit. PageLens shows who actually does: visits referred by ChatGPT, Perplexity, Claude and Gemini, and every AI crawler hitting your pages — without cookies.
Create a free PageLens account: 7-day trial, no card, no cookie banner needed.
Start free trial