What a crawler request is
A crawler request means a program asked your server for a URL. It says nothing about what happened next: whether the page was stored, whether it was ever retrieved for a question, whether any answer used it.
Analytics products that report "AI traffic" count these requests. Some go further and draw a chart titled visibility. The chart is real; the title is not.
What we measured on our own pages
Between 25.07.2026 and 20.09.2026 we recorded every verified crawler request to our pages, and every time the same provider cited the same page in an answer:
| Provider | Pages crawled | Pages later cited | | --- | --- | --- | | Google | 800 | 0 | | OpenAI | 719 | 4 | | Apple | 666 | 0 | | Perplexity | 414 | 34 |
2,743 pages fetched by verified crawlers; 42 came back in an answer from the same provider. The median gap between the first fetch and the first citation was 11 days. The full table and its method are in the study.
Two reasons the crawl number is worse than it looks
- Many "AI crawler" requests are not verifiable. A user agent string is a
claim. Of 11,557 requests we could check against a provider's published ranges, 45% matched (study).
- Even verified requests are not use. A provider can fetch 800 of your
pages and cite none — which is exactly what Google did on ours over two months.
What to count instead
Count outcomes you can check in an answer: whether your product was named, and whether the link pointed at your page. Those are the two things a buyer sees. Everything before them is plumbing — worth knowing, not worth reporting as visibility.
What none of this proves is causation. A crawl followed by a citation is a sequence, not a mechanism; the rules we use to keep those apart are at /evidence.
Sources
- How many crawled pages a model actually cites
- How many AI crawler visits can be verified
- Evidence methodology, version 1.0