Can ChatGPT access my website? How to check in two minutes, and what to fix

If ChatGPT can't read your pages, it can't quote them, recommend them or link to them. Most sites never check. Three things decide whether an AI assistant gets in, and you can test all three for free in a couple of minutes.

1. robots.txt: are AI bots allowed?

Open yoursite.com/robots.txt. Look for groups that name AI user agents, or a User-agent: * group with Disallow: /, which blocks everyone. The ones that matter for answers today:

• OAI-SearchBot builds the index ChatGPT search uses. Blocked here, you are unlikely to be cited in ChatGPT answers.
• ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person asks the assistant about it.
• GPTBot and ClaudeBot collect training data. Blocking them is a choice about training, not about appearing in answers.

Reading robots.txt by hand is easy to get wrong, because a specific group overrides * and the most specific path wins. The free AI crawler checker reads it the way crawlers do and shows each AI bot as allowed or blocked, plus whether you have an llms.txt.

2. Your CDN or firewall: are they blocked before robots.txt matters?

robots.txt is a request. A firewall is a wall. Many sites block AI bots at the CDN without knowing it:

• Cloudflare has a one-click setting that blocks AI bots, and since mid-2025 it is offered by default to new sites. Check Security → Bots and the AI crawler settings in your dashboard.
• Bot protection on other CDNs and WAFs, and some WordPress security plugins, challenge unknown agents with a JavaScript or CAPTCHA page that AI fetchers can't pass.

The quick test: ask ChatGPT (with search on) or Claude to "open https://yoursite.com/pricing and summarise it". If it says it can't access the page, or summarises something generic, the request was probably stopped before it reached your content. Try a deep page, not only the home page.

3. Is the content in the HTML?

AI crawlers download the HTML and mostly don't run JavaScript. If your page is a single-page app that renders everything in the browser, the bot may see an empty <div id="root">.

Check it with one command and look for your actual text in the output:

curl -s -A "ChatGPT-User" https://yoursite.com/pricing | grep -i "your headline"

If nothing comes back, render that page on the server or pre-render it at build time (Next.js, Nuxt, Astro and SvelteKit all can).

4. Then check that they actually come

Allowed doesn't mean visited. To see real reads you need your server or CDN logs, because AI bots never show up in JavaScript analytics. Five ways to see AI crawlers covers logs, Cloudflare, middleware and WordPress. On a new site, expect the first reads within days, mostly from training crawlers; here is what the first week looked like on ours.

Checklist

PageLens Analytics does the last step automatically: it reads AI crawler and assistant requests on your server, checks them against the IP ranges OpenAI, Anthropic, Perplexity and Google publish, and shows which pages they read. The checker above is free and needs no account.