What AI crawlers did on a brand-new website in its first five days (real data)
pagelens.io went live on 7 October 2026. Our server forwards every request from a known bot to PageLens, and humans are counted by the normal script, so we can see both sides of the same site. Here is everything AI companies' bots did in the first five days (7–11 October, times in UTC), and what a new site can take from it.
The short version
- Bots outnumbered people about 3.5 to 1. 750 bot requests against 207 human page views, after leaving out
curlrequests (many of them were our own checks). - AI crawlers read 152 pages. GoogleOther came first, on day 2. GPTBot came on day 5 and read 67 different pages in a single day, essentially the whole sitemap.
- Assistants opened our pages live 11 times while answering someone: ChatGPT-User 9 times, Claude-User twice.
- AI assistants sent us zero visitors. Not one click from ChatGPT, Claude, Perplexity or Gemini yet.
Who came, and when
| Bot | Company | What it does | Requests | Different pages | First seen |
|---|---|---|---|---|---|
GoogleOther | General crawling outside Search (research, product teams) | 69 | 56 | 8 Oct | |
GPTBot | OpenAI | Collects pages for model training | 67 | 67 | 11 Oct |
Bytespider | ByteDance | Collects pages for ByteDance's models | 10 | 10 | 9 Oct |
ChatGPT-User | OpenAI | Opens a page because a ChatGPT user asked something | 9 | 7 | 9 Oct |
ClaudeBot | Anthropic | Collects pages for model training | 3 | 3 | 8 Oct |
Claude-User | Anthropic | Opens a page because a Claude user asked something | 2 | 1 | 8 Oct |
meta-externalagent | Meta | Collects pages for Meta's AI | 2 | 1 | 10 Oct |
PerplexityBot | Perplexity | Builds Perplexity's search index | 1 | 1 | 10 Oct |
The very first AI request of any kind was Claude-User opening the home page on 8 October, one day after launch. Someone had asked Claude about PageLens, or pasted the link, before any training crawler had read the site.
The most interesting minute: ChatGPT checks us out
On 10 October between 05:02 and 05:03 UTC, ChatGPT-User opened six pages in a row: the home page, the install guide index, the API docs, the terms, the privacy policy and the blog. That pattern looks like a person asking ChatGPT to evaluate the product — "is this legit, how does it handle data, does it have an API?" — and the assistant reading the pages that answer those questions.
We can't see the question, and we can't prove what the person did next. But it is a useful reminder: your terms, privacy policy and docs are now read by AI on behalf of buyers, not only by lawyers. If they are vague, the answer the person gets will be vague too.
GPTBot read the whole site in one day
GPTBot didn't trickle in. It showed up on day 5 and fetched 67 different URLs on 11 October, one request per page, which matches the size of our sitemap. A clean sitemap and fast pages make that single pass count; a crawler that hits errors or slow pages on its one visit may not come back soon.
GoogleOther behaved differently: 69 requests spread over four days, mostly install guides and docs, a few pages more than once.
Reads are not visits
152 crawler reads and 11 live fetches turned into zero visitors. That's normal for a five-day-old site, and it's the main thing to understand about AI traffic: being read is the first step, being used in an answer is the second, and a click only comes when the answer links to you and the person wants more than the answer gave them. You won't see this funnel in Google Analytics at all, because GA4 drops bots and most AI clicks arrive without a referrer.
What a new site can do with this
- Don't block the bots you want answers from. Blocking
OAI-SearchBot,ChatGPT-UserorPerplexityBotkeeps you out of those answers. Check yours with the free AI Crawler Checker; our guide to which AI crawlers to block explains the trade-offs. - Make the pages AI reads for buyers answer buyer questions. Pricing, what data you collect, where it's stored, how to cancel. In plain text, not behind a script.
- Keep a sitemap and tell search engines about changes. We ping IndexNow on every deploy, which Bing (and through it ChatGPT search) picks up.
- Publish an
llms.txtwith a short, factual description of what you do and what it costs. Ours is at pagelens.io/llms.txt. - Measure the whole funnel, not just clicks. Reads and live fetches show interest weeks before visits do.
How we measured it
Bots that don't run JavaScript are invisible to a normal analytics script, so pagelens.io sends a copy of each bot request from the server to PageLens (bot user agents only; humans are counted by the script). Bots are classified by user agent and kept out of visitor numbers. User agents can be faked; we don't verify them against published IP ranges yet, so treat individual rows as "claims to be". The totals for the big crawlers are consistent with what their operators document.
This is the same report PageLens shows any site under AI Visibility. You can explore it with sample data in the live demo.