- Period
- 2026-07-25 — 2026-09-21
- Method
- A page counts as crawled when a request for that exact path passes verification against the provider's published ranges or reverse DNS, and as cited when the same provider returns that path in an answer. Paths are compared without query or fragment; the provider must match on both sides.
- Method version
- 1.0
- Published
- 2026-09-21
What was measured
Our own site, 2026-07-25 to 2026-09-21. 45 product records were published in that period, alongside the hub pages, the machine representations and the editorial pages — every one of them a URL a crawler can request.
Two events are recorded for each URL. A verified crawl: a request for that exact path whose origin matched the provider's published ranges or reverse DNS at the moment it arrived. A citation: the same path appearing in an answer from the same provider. The lag is the number of days between the first of each.
Results by provider
| Provider | Pages crawled | Pages cited | Both | Median lag | | --- | --- | --- | --- | --- | | google | 800 | 0 | 0 | — | | openai | 719 | 4 | 4 | 30 days | | apple | 666 | 0 | 0 | — | | perplexity | 414 | 35 | 34 | 11 days | | yandex | 86 | 0 | 0 | — | | microsoft | 58 | 0 | 0 | — | | xai | 0 | 3 | 0 | — | | anthropic | 0 | 1 | 0 | — |
Across all providers the median lag between the first verified crawl of a page and its first citation by the same provider is 11 days.
What stands out
Citations come from 4 of the 8 providers that crawl us. The rest fetch pages and have not returned one in an answer we recorded. That asymmetry is the finding: a crawl is a request, not a decision, and counting crawls as visibility counts something no buyer ever sees.
Two providers show zero crawls, and that is our method, not their behaviour
Anthropic and xAI appear with no crawled pages and with citations. Our verifier had no published IP ranges or reverse-DNS rule from either to check their requests against, so every request from them was recorded as a claim rather than as verified — and this table counts verified crawls. Their rows are a property of what can be checked, not evidence that they never fetched anything. The same applies to Meta, which crawls us and has not been recorded citing us.
What this does not say
A citation from a provider does not prove the crawl caused it. Models retrieve at answer time as well, and a page can be reached in a way our log never sees. The pairing here is the weakest claim the data supports — same provider, same path, first event on each side — and it is stated as a lag, not as a cause.
It also says nothing about answers we did not observe. We measure the questions we ask on the schedule we ask them; a page cited in a conversation we never ran is not in this table, and its absence is not evidence of anything.
Why the gap matters commercially
Dashboards that report "AI traffic" count crawler requests. This table is the same data with one more column, and the column changes the meaning: of 2 743 pages fetched by verified crawlers, 43 came back in an answer. Anyone buying AI visibility on the strength of crawl counts is buying the first number and being shown it as if it were the second.