GuideUpdated 2026-09-21

What can and cannot be measured about AI answers

You can measure what appears in an answer: whether your product was named, which URL was linked, and whether a stated fact matches your record — per model and per date. You can estimate change with baselines and controls. You cannot measure why a model answered as it did, or turn a mention into revenue.

Measurable directly

  • Named or not — the product appears in the answer text.
  • Linked or not, and where — your product page, another page of your site,

someone else's. In our records, when a seller's URL was present it pointed at the product page in 643 answers and elsewhere on the site in 1,445 (study).

  • Fact match — a stated value equals the value in a versioned, sourced

record.

  • Verified crawls — requests that passed the provider's documented check.

Each of these must carry the provider, the exact model name and the date.

Measurable only with design

  • Change after an intervention — needs a frozen question set, a baseline,

the noise between two unchanged runs and a control group. The checklist is at /checklists/before-and-after-test-on-ai-answers.

  • Which models favour which sources — needs the same questions across

models on the same days.

Not measurable from the outside

  • Why a model answered as it did. The answer shows what it said, not what

it read or weighed.

  • Answers to questions you did not ask. Absence from your record is not

absence from the world.

  • Revenue from a mention. A mention is not a visit, and a visit is not a

sale.

The ladder that keeps each claim on its own step is at /evidence.

Sources

Related

← All pages