Free AI crawler checker: can ChatGPT, Claude and Perplexity read your site?
See what stops AI assistants from reading and citing your website: robots.txt rules, CDN blocks, pages that are empty without JavaScript, and a missing sitemap or llms.txt. Takes 10 seconds, no sign-up.
What the check looks at
- Can AI reach you? Your robots.txt rules for each assistant, and whether your server or CDN actually serves pages to requests carrying their names (the part most tools miss).
- Can AI read you? How much text your home page has without JavaScript, since AI crawlers don't run it; title, description and structured data.
- Can AI find you? A sitemap, and an llms.txt summary for AI tools.
What other checks miss
robots.txt only says what you allow. Your CDN or firewall decides what actually gets through, and a "block AI bots" setting there overrides the file without leaving a trace in it. So besides reading robots.txt, this check requests your home page five more times: as GPTBot, ChatGPT-User, ClaudeBot and PerplexityBot, and once as a made-up crawler. If the AI-named requests are turned away and the made-up one isn't, the block is aimed at AI.
Which AI bots it checks
| Bot | Operator | robots.txt rule | Live request |
|---|---|---|---|
GPTBot | OpenAI | ✓ | ✓ |
OAI-SearchBot | OpenAI | ✓ | — |
ChatGPT-User | OpenAI | ✓ | ✓ |
ClaudeBot | Anthropic | ✓ | ✓ |
Claude-SearchBot | Anthropic | ✓ | — |
Claude-User | Anthropic | ✓ | — |
PerplexityBot | Perplexity | ✓ | ✓ |
Perplexity-User | Perplexity | ✓ | — |
Google-Extended | ✓ | — | |
Applebot-Extended | Apple | ✓ | — |
meta-externalagent | Meta | ✓ | — |
Amazonbot | Amazon | ✓ | — |
CCBot | Common Crawl | ✓ | — |
Bytespider | ByteDance | ✓ | — |
DuckAssistBot | DuckDuckGo | ✓ | — |
MistralAI-User | Mistral | ✓ | — |
What it can't tell you
This shows whether AI can use your site. PageLens AI Visibility shows what it actually does: which pages ChatGPT, Claude and Perplexity read, which ones they open while answering people, and the visitors and revenue they send.
Questions
Does blocking GPTBot keep me out of ChatGPT?
No. GPTBot collects pages for training. ChatGPT search results come from OAI-SearchBot, and answers that open your page live come from ChatGPT-User. You can block training and still allow both of those.
Why does the checker request my site as GPTBot?
Many sites block AI bots at the CDN or firewall (for example Cloudflare's "Block AI bots" setting), which robots.txt doesn't show. The checker sends one request per bot name, clearly labelled as coming from this tool, and compares the answer with a normal browser request.
What is llms.txt?
A short Markdown file at /llms.txt that summarizes your site for AI tools: what you do, for whom, prices and links to key pages. It is optional and cheap to add.
Does ChatGPT run JavaScript when it reads a page?
AI crawlers generally read the HTML your server sends and don't run scripts. If your text only appears after JavaScript runs, as in many single-page apps, they see an almost empty page. The check counts the words that are readable without JavaScript.
What is GPTBot?
GPTBot is OpenAI's crawler for training data. It is separate from OAI-SearchBot, which builds the index for ChatGPT search, and from ChatGPT-User, which opens a page when a person asks ChatGPT about it. The AI bots directory on this site describes each one.
How do I allow AI crawlers on my site?
Make sure robots.txt has no Disallow rule for the agents you want (for example OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot), then check your CDN or firewall: bot-blocking settings there apply even when robots.txt allows a bot.
My robots.txt allows AI bots. Why does the check say they may be blocked?
Because the server answered differently. A request carrying the bot's name got an error or a challenge page while a browser got the page. That usually means a CDN or firewall rule. Some CDNs only stop requests that claim a bot's name without coming from that company's IP addresses, so confirm in your CDN's bot settings.
How is this different from other AI crawlability checkers?
Most checkers read robots.txt and compare the page with and without JavaScript. This one also sends requests named after GPTBot, ChatGPT-User, ClaudeBot and PerplexityBot, plus a control request from a made-up crawler, so it can tell a block aimed at AI from one aimed at every bot.
Is it free, and do you keep my results?
It is free with no sign-up, limited to a few checks per minute. Results are not saved to any account; the grade shown on a badge is kept for a day so the badge loads quickly.
Does a good score mean ChatGPT recommends my site?
No. It means nothing on your side stops AI assistants from reading and citing you. Whether they do depends on your content and on what others write about you. PageLens shows what they actually do: which pages they read, use in answers and send visitors to.