Why name-based rules fail
A user agent is text. Scrapers use AI crawler names because many sites wave them through. On our site, 2,290 requests carrying OpenAI's crawler name came from outside OpenAI's published ranges, against 1,304 that matched (study).
The rule
- Match the claim. The request's user agent names a provider you want to
allow.
- Verify the origin. The IP is inside the provider's published ranges, or
its reverse DNS resolves to the provider's documented domain and back to the same IP.
- Allow only if both hold. If the name matches and the origin does not,
handle it as any unknown client.
- Refresh the lists. Fetch published ranges from the provider on a
schedule; ranges change.
Most CDNs and firewalls can express this as "verified bot" rules or as an IP allow-list per user agent. Where they cannot, do the check in your own application and log the outcome with the request.
Record the outcome
Keep verified, failed and claim-only as three separate outcomes (/learn/how-to-verify-ai-crawlers-in-your-logs). Our own crawler identifies itself as described at /bot. What a verified crawl does and does not prove is at /evidence.