ChecklistUpdated 2026-09-21
Running a before-and-after test on AI answers
Use it before changing a product page with the aim of changing AI answers. Without these steps a better answer afterwards is a sequence, not evidence: models drift on their own, and several changes at once cannot be told apart.
Before the change
- Write the questions down and freeze them. Changing the questions between measurements compares nothing.
- Choose the models and name each one in full.
- Measure twice, two weeks apart, without changing anything. The difference between the two runs is your noise.
- Pick comparable products you will not change, and measure them the same way. They are the control.
The change
- Change one thing: identifiers, one fact, the canonical URL — not several at once.
- Record the date and the exact change.
After
- Measure the same questions, same models, on the same schedule, for both the changed and the unchanged products.
- Compare the change against the noise and against the control, not against a single earlier answer.
- Report what was measured and what was not: an improvement in the changed products that also appears in the unchanged ones is not your change.
Why each item is on the list
Related
← All checklists