QuestionUpdated 2026-09-21

Can you trust a GPTBot user agent in your logs?

Not on its own. A user agent is text any program can send. On our site 2,290 requests calling themselves OpenAI's crawler came from outside OpenAI's published ranges, against 1,304 that matched. The check that separates them takes a few lines and should run when the request arrives.

What the string proves

Nothing. GPTBot, ClaudeBot, PerplexityBot are text in a request header, and any script can put them there. Scrapers do it because many sites wave AI crawlers through while blocking unknown bots.

How often the claim fails

Our own log, 25.07.2026 to 20.09.2026, every request carrying an AI crawler's name, checked against the provider's published IP ranges or reverse DNS at the moment it arrived:

| Provider | Passed | Failed | Share passed | | --- | --- | --- | --- | | Google | 2,342 | 77 | 96% | | Apple | 911 | 155 | 85% | | Perplexity | 518 | 715 | 42% | | OpenAI | 1,304 | 2,290 | 36% | | Amazon | โ€” | 2,906 | 0% |

A further 4,597 requests named providers our check had nothing published to compare with; those are recorded as claims, not as visits. Full method in the study.

What "failed" does and does not mean

A failed check is not proof of impersonation. A provider can use an address before publishing it, and a reverse lookup can fail for reasons that have nothing to do with the requester. What it does mean is narrower: that request has not been shown to be what it claims, and should not be counted as though it had.

Why it matters to a brand

If you decide which pages to open to AI crawlers, or report AI interest to your team, on the strength of user agent counts, you are deciding on numbers where roughly half the checkable requests do not check out. The fix is not to block more; it is to record the result of the check next to the request, so the number you report is the verified one.

How to run the check yourself: /learn/how-to-verify-ai-crawlers-in-your-logs.

Sources

Related

โ† All pages