GlossaryUpdated 2026-09-21

Verified crawler

A verified crawler request is one whose origin was checked, at the moment it arrived, against the provider's documented rule — published IP ranges or forward-confirmed reverse DNS — and matched. It is the only kind of crawler visit that can be attributed to a provider.

How it differs from a user agent

A user agent is text the requester chooses. A verified request is one whose network origin agrees with what the provider documents. Of 11,557 requests we could check this way, 45% matched (study).

Three outcomes, not two

Verified, failed, and claim only — the last for providers our check has nothing published to compare with. Folding "claim only" into either of the others inflates one number or accuses a provider of something unestablished.

Where this was measured

Related

← All terms