How to see which AI crawlers read your website: 5 ways, from free to automatic

AI crawlers read websites all day, and almost no site owner can see it. The reason is simple: analytics tools count visitors with JavaScript that runs in a browser, and AI crawlers download the HTML without running any JavaScript. To see them you have to look where the request arrives: your server or your CDN.

Here are five ways to do that, starting with the ones that cost nothing.

First: which bots to look for

There are three kinds, and they mean different things:

The third kind is the one to watch. It is the closest thing to "my page was used in an answer". The AI bots directory has the user agent and robots.txt rules for each.

1. Server access logs (free, if you run the server)

Nginx and Apache already write every request to a log. Count requests per AI bot:

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn

See which pages one of them asked for (the path is the 7th field in the default log format):

grep "ChatGPT-User" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

And the requests that ended in a 404, which are pages a bot expected to find and didn't (the status is the 9th field):

grep -E "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | awk '$9 == 404 {print $7}' | sort | uniq -c | sort -rn | head

Good for: a one-off look on a VPS. Limits: logs rotate after days or weeks, a CDN cache in front of the server hides requests from it, and you get a number, not a trend.

2. Your CDN's dashboard (free, if your site runs through one)

A CDN sees every request before your server does, including the ones it answers from cache. Cloudflare has an AI crawler section in its dashboard that lists AI crawlers requesting your site and lets you allow or block each. Other CDNs and hosts (Vercel, Netlify, Fastly) show requests in their logs, where you can filter by user agent.

Good for: a quick answer to "which AI companies crawl me, and how much". Limits: usually no view per page over time, nothing about what happened next (did anyone visit, did they buy), and retention depends on your plan.

3. A few lines of middleware (free, for Next.js and similar)

If your framework lets you run code on every request, log the bots there. In Next.js:

// middleware.ts (proxy.ts in Next.js 16)
import { NextResponse, type NextRequest } from 'next/server'

const AI = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User/i

export function middleware(req: NextRequest) {
  const bot = req.headers.get('user-agent')?.match(AI)?.[0]
  if (bot) console.log(JSON.stringify({ bot, path: req.nextUrl.pathname }))
  return NextResponse.next()
}

export const config = { matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'] }

The lines land in your host's function logs. The same idea works in Express, a Cloudflare Worker or any server framework.

Good for: developers who want the raw events. Limits: you still have to store, count and chart them, and middleware runs before the page, so it doesn't know the response status.

4. WordPress

WordPress runs PHP on every uncached request, so a short snippet in a small plugin or your child theme can log AI bots:

add_action('template_redirect', function () {
    $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
    if (preg_match('/GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User/i', $ua, $m)) {
        error_log('AI bot ' . $m[0] . ' ' . ($_SERVER['REQUEST_URI'] ?? '/'));
    }
});

The catch: page caches (WP Rocket, LiteSpeed, a CDN) serve cached pages without running PHP, so cached pages go uncounted. Exclude bot user agents from the cache, or count at the CDN instead.

5. Analytics that reads bots from your server (automatic)

The four ways above give you raw requests. Turning them into something you can act on means storing them, grouping them by page and by company, and joining them with what people did afterwards. That is what AI Visibility in PageLens does: a small snippet on your server (Next.js middleware, a Cloudflare Worker, Express or the WordPress plugin) reports bot requests, and the dashboard shows:

• which pages AI crawlers read, and which ones assistants opened to answer people;
• pages bots asked for that don't exist, and pages they never found;
• visits that came from ChatGPT, Claude and Perplexity, and the revenue from them;
• which requests were real: each bot is checked against the IP ranges its company publishes.

Sites on hosted builders such as Wix, Squarespace or Framer can't run server code. If the domain goes through Cloudflare, the Worker covers them; otherwise the CDN or host dashboard is the only place to look.

Check that the bots are real

Anyone can send a request that says GPTBot. OpenAI, Anthropic and Perplexity publish the IP ranges their bots use, so a request with a bot's name from another address is an impostor. In our study of 740 top websites, 9% of sites turned away requests carrying AI bot names even though their robots.txt allowed those bots. Before you read much into your numbers, also make sure the bots can reach you at all: the free AI crawler checker tests that in ten seconds.

What to do with the numbers

• Read a lot, never opened for an answer: AI knows the page exists and doesn't find it useful for questions. Put the question in the heading and the answer in the first two sentences.
• Opened for answers, no visits: you are being used as a source without being clicked. Check how the page describes you and whether it gives a reason to click through.
• Important page never read: link to it from your home page and sitemap, and check robots.txt and your CDN.
• Bots hitting 404s: they expect a page there, often /pricing, /docs or /llms.txt. Create it or redirect.

The field numbers are for the default "combined" log format of Nginx and Apache. If your format differs, adjust them.