OAI-SearchBot: OpenAI's ai search crawler
OAI-SearchBot belongs to OpenAI. It builds the index an AI assistant searches when it answers questions. Block it and your pages can't be found or cited in that assistant's search results.
| Operator | OpenAI |
|---|---|
| Used for | ChatGPT search |
| Type | AI search crawler |
| robots.txt name | OAI-SearchBot |
| User agent | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot |
| Published IP ranges | https://openai.com/searchbot.json |
| Blocked by top sites | 7.8% of 740 (study) |
| Official documentation | developers.openai.com/api/docs/bots |
The version number in the user agent changes over time.
Allow or block OAI-SearchBot in robots.txt
To block it on the whole site:
User-agent: OAI-SearchBot
Disallow: /To allow it explicitly (useful when a rule for all bots would block it):
User-agent: OAI-SearchBot
Allow: /OpenAI recommends allowing it. Sites that opt out are not shown in ChatGPT search answers, though they can still appear as navigational links. Changes take about 24 hours.
robots.txt is a request, not a wall. A CDN or firewall rule can turn a bot away even when robots.txt allows it, and nothing in the file will show that.
The free check requests your home page as a browser and as AI bots, reads your robots.txt and shows what gets blocked. No sign-up.
Check my siteIs it really OAI-SearchBot?
Anyone can send a request with this bot's name. OpenAI publishes the IP ranges its requests come from, so a request that carries the name from another address is an impostor. PageLens checks every request from this bot against that list.
See OAI-SearchBot on your own site
OAI-SearchBot doesn't run JavaScript, so analytics scripts never record it. AI Visibility in PageLens reads it from your server or CDN and shows which pages it requested, when, and which requests ended in a 404.