We checked 740 of the world's top websites for AI bots: who blocks ChatGPT, and how (2026 data)

Most advice about AI search starts with "check your robots.txt". We wanted to know what the biggest sites actually do, so we ran our AI visibility check on the top 1,500 domains of the Tranco list (list 4672X), a research ranking of the most visited domains, on October 12, 2026.

Many of those domains are APIs and CDNs with no home page (gstatic.com, akamai.net and so on). 740 served a home page to a normal browser request, and every number below is out of those 740 websites.

The short version

• 15% block GPTBot in robots.txt, and about as many block ClaudeBot (14%) and Google-Extended (13%).
• 13% block at least one agent that answers people: OAI-SearchBot, ChatGPT-User, Perplexity's bots or Claude-User.
• Of the 109 sites that block GPTBot, 69 (63%) also block ChatGPT's search or browsing agent. Only 40 block training and keep ChatGPT search open.
• 9% turn away requests that carry an AI bot's name at the server, even though their robots.txt allows those bots.
• About 14% publish an llms.txt; 45% have structured data on the home page; 38% have no sitemap we could find.

1. robots.txt: training bots are blocked more than answer bots

Share of the 740 websites whose robots.txt keeps each agent off the home page (in practice, off the site):

Most of these are deliberate: 92 of the 109 GPTBot blocks name GPTBot explicitly, the rest come from a rule for all bots.

Each agent has a page in our AI bots directory with its user agent, robots.txt rules and published IP ranges.

2. Blocking GPTBot usually blocks ChatGPT search too

GPTBot only collects training data. Whether a site appears in ChatGPT search depends on OAI-SearchBot, and ChatGPT-User is the agent that opens a page while ChatGPT answers someone. OpenAI documents them as separate agents. One nuance: OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person starts those requests, so a robots.txt block of it is a stated preference more than a guarantee.

Of the 109 websites that block GPTBot:

• 69 also block OAI-SearchBot or ChatGPT-User. Among them are cnn.com, bbc.com, reddit.com, imdb.com, espn.com, nature.com and Amazon's stores. They opt out of ChatGPT as a whole.
• 40 block only training and leave ChatGPT search and browsing open, including forbes.com, bloomberg.com, airbnb.com, figma.com, ted.com and nintendo.com.

Both are legitimate choices. The second one is the one most businesses want: stay out of training, stay in the answers. If you copied a "block AI bots" robots.txt from a blog post, check which group you're in.

3. robots.txt isn't the whole story

For each site we also requested the home page with each bot's name in the user agent (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot), and compared the answer with a normal browser request. On 130 sites, at least one bot-named request was turned away (a 403, a challenge page or no answer) although robots.txt allows that bot.

We then re-tested those 130 sites with a fourth request: a generic, made-up crawler name. That splits them into groups:

• 69 (9.3% of all sites) let the browser and the generic crawler through, but turn away requests named after AI bots. 37 of them turn away ChatGPT-User, the agent that opens pages to answer people.
• 37 (5%) turn away any bot-like request, AI or not.
• 19 did not repeat on the second test, and 5 blocked the browser too.

One caution: our requests come from our servers, not from OpenAI's or Anthropic's. Some CDNs block requests that claim to be GPTBot but don't come from the IP ranges OpenAI publishes, and let the real GPTBot in. From the outside, "blocks AI bots" and "blocks impostors" look the same, which is why we don't name sites in this group. If you're one of them, your CDN's bot settings (in Cloudflare: Security → Bots) will tell you which it is.

The useful lesson for a site owner: robots.txt can say "allowed" while the server says no, and nothing in robots.txt will show it.

4. Can AI read the page once it gets it?

AI crawlers don't run JavaScript. On 22% of home pages, fewer than 120 words were readable in the HTML before scripts run. This number is an upper bound: some sites send bots a short interstitial page instead of the real one. But for a single-page app, an empty HTML shell is what every AI crawler sees.

llms.txt is still a minority practice even among the largest sites: about one in seven. Several more sites answered /llms.txt with something that isn't one (an IP address, a bare "OK", an error page), which a naive check would count.

What to do on your own site

1. Decide training and answers separately. Blocking GPTBot and ClaudeBot keeps you out of training; to stay in ChatGPT and Claude answers, allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User. Our robots.txt guide has a copy-paste version.
2. Check the server, not only the file. CDN bot settings and firewalls override robots.txt.
3. Make sure the text is in the HTML. Server rendering or prerendering for the pages you want quoted.
4. Then measure what AI actually does. A check shows what AI can do. Your server logs, or AI Visibility in PageLens, show which pages AI crawlers read, which ones assistants open to answer people, and whether any of it brings visitors.

You can run the same check on your site for free: pagelens.io/tools/ai-crawler-checker.

Method and limits

• Domains: top 1,500 of Tranco list 4672X. A domain counts as a website if a normal browser request to its home page got a page (2xx or 3xx, no challenge). 740 did.
• robots.txt rules were evaluated for the home page path with the most specific matching user-agent group, as crawlers do.
• Bot requests used each bot's name and said they came from the PageLens checker. Run from Central Asia on October 12, 2026; results can differ by location and change at any time.
• We report shares, not judgements: blocking AI bots is a valid choice.

Questions about the data or the method? Write to hello@pagelens.io.