The screenshot problem: why one AI answer proves nothing
A single AI answer screenshot documents one non-deterministic event. In our testing, only 18% of prompts returned an identical citation set across 25 repeats, so roughly four times out of five a client repeating the query will see something different from the screenshot in their report.
What happens in the meeting
You show a screenshot: the client's brand cited in an AI Overview. Someone in the room types the same query into their phone. The brand isn't there.
You now have to explain non-determinism, in a meeting, defensively, having led with evidence that just failed in front of everyone. The methodology conversation you avoided by using a screenshot arrives anyway, in the worst possible conditions.
What the data says about repeats
In our volatility study — 400 prompts, 25 samples, six engines — only 18% of prompts returned an identical citation set every time. Median pairwise overlap between two random samples of the same prompt was 62% on AI Overviews and 47% on ChatGPT.
That means for a typical ChatGPT prompt, two people asking the same question see about half the same sources. A screenshot captured one of those halves.
The honest replacement
Replace the screenshot with three things:
- A rate: "cited in 34% of 25 samples".
- An interval: "±17 points at this sample size".
- The evidence, on request: all 25 answers, timestamped.
This is less impressive on a slide and enormously more durable in a challenge. The client who tests it finds a distribution that matches what you told them, rather than a contradiction.
Say the quiet part in the report
When screenshots are still fine
As illustration, never as evidence. Showing what an answer looks like is useful context. Showing one and implying it's the steady state is where the credibility goes.
See this for your brand
Nostimates shows you how your brand shows up across twelve AI engines and search — in a dashboard, or as data in your own tools. Tell us what you want to measure.