How to verify AI crawlers in your server logs

Take the requesting IP, check it against the provider's published address ranges, or resolve it to a hostname and confirm that hostname resolves back to the same IP. Store the result with the request. Report only the verified count as the provider's traffic.

Step 1 — Keep the raw request

For every request whose user agent names an AI provider, keep the remote IP, the user agent, the path and the timestamp. Most access logs already have all four. Do not keep only the user agent: it is the one field that proves nothing.

Step 2 — Use the provider's own rule

Providers document one of two ways to confirm their crawler:

Published IP ranges. The provider publishes a list of the address blocks

its crawler uses. Check whether the requesting IP falls inside one.

Reverse DNS. Resolve the IP to a hostname, check the hostname ends in the

provider's documented domain, then resolve that hostname forward and confirm it returns the same IP. The forward step matters: without it, anyone who controls the reverse record of their own address can pass.

Use whichever the provider documents, fetched from the provider's own site, and refresh the lists regularly — ranges change.

Step 3 — Record one of three outcomes

| Outcome | Meaning | | --- | --- | | verified | the request matched the provider's documented rule | | failed | the provider documents a rule and the request did not match it | | claim only | you have no documented rule for that provider to compare with |

Three outcomes, not two. Folding "claim only" into either of the others is the most common way this goes wrong: it either inflates the verified count or accuses a provider of something your check cannot establish.

Step 4 — Check at the moment of the request

Run the check when the request arrives, not months later. Ranges move; an address that matched in July may not match in December, and re-checking old requests against new lists rewrites your history.

What to expect

On our own site, of 11,557 requests we could check this way, 45% matched (study). Your ratio will differ. The point of the procedure is not the ratio; it is that the number you report as "OpenAI visited us" becomes a number someone can check.

What this does not tell you

Whether a verified crawler used your page. That is a later step on the evidence ladder, described at /evidence, and it needs different data: the answers, not the logs.