AEOsim blog
A gap report is only as good as its recall
Table of contents
An absence is not automatically a gap
Most AI visibility reports include a gap list: prompts where a competitor appears and you do not, or attributes the engine never mentions for your product. Gap lists drive content plans, so they matter. They are also the least reliable part of most reports.
The reason is simple. A gap is an absence, and absence is ambiguous. "The engine didn't mention our free returns" can mean the engine genuinely does not associate your brand with free returns. It can also mean the engine said it in a later turn the tool never ran, or in a run the tool did not sample, or in a phrasing the extraction step failed to catch.
In statistics, this is recall: of everything that was actually there, how much did the measurement catch? A gap report with low recall manufactures gaps. Teams then write content to fix problems that did not exist, while the real drop-off goes unnoticed.
Three ways a gap can be false
1. Sampling miss
AI answers vary run to run. Researchers found brand sets in repeated same-day runs overlapped by a Jaccard similarity of only 0.33 to 0.48 (Schulte et al., 2026). A tool that runs a prompt once can easily catch the draw where your brand was absent and report it as a gap. The same study found at least seven runs per prompt were needed before the error on a per-brand detection rate fell below 0.10.
2. Turn miss
A tool that measures only the first response cannot see a brand that enters at turn three. The reverse also holds: a brand present at turn one may be gone by the decision. Absence at one turn says little about absence across the conversation (Visibility Has Two Axes).
3. Extraction miss
Someone, or some model, has to read each answer and decide whether a brand or attribute was mentioned. Abbreviations, misspellings, Hinglish, product-line names instead of brand names, and indirect references ("the Indian brand with the lifetime warranty") all slip past naive matching. Every one of those becomes a false gap.
There is a fourth, subtler case. In multi-turn chats an engine sometimes drops a brand without any reason in the conversation, simply losing track of its own shortlist. AEOsim calls the share of genuine, justified exclusions γ (Visibility Has a Third Axis). A drop caused by engine incoherence is real in the log but says nothing about your brand, so it should not drive a content fix.
How to estimate recall before claiming a gap
You do not need perfect recall. You need to know roughly what it is, so a gap can be stated with the right confidence. Four checks cover most of it:
- Sampling recall. Run the prompt or conversation enough times to see a stable distribution. If your brand appears in 3 of 20 runs, that is a 15% presence rate, not a gap. Report the rate and its interval, and call it a gap only when the upper bound is low.
- Turn recall. Check presence across every turn of the conversation, not only the first. A gap claim should say which turn it applies to.
- Extraction recall. Hand-label a random sample of answers, say 50, and compare against the automated extraction. The share of real mentions the extractor caught is its recall. If it is 85%, the extractor misses about 15% of real mentions, and every miss shows up as a false absence. Re-check whenever the extraction model changes.
- Incoherence check. Compare drop rates on turns where the buyer added a constraint against turns where they did not. Drops on neutral turns estimate how often the engine loses its own shortlist, which is not a brand gap.
Then apply a simple rule: a gap is only a gap if it survives all four. If the brand is absent in most runs, at the turn that matters, with extraction checked, and mainly on turns where a real constraint was applied, it has a genuine gap worth a content fix.
AEOsim's alignment framework makes the same point from another angle: one observation can establish a direct factual contradiction, but it cannot establish that a problem recurs (AI Answer Alignment).
What a trustworthy gap report contains
| Field | Why it is there |
|---|---|
| Presence rate with n and interval | Separates a true gap from a sampling miss |
| Turn where the gap appears | Separates a discovery gap from a decision-turn gap |
| Constraint active at that turn | Tells you which objection or criterion to answer |
| Extraction recall for the run | Tells you how many absences may be misses |
| Engine and surface | Gaps often appear on one engine only |
| Competitor that filled the slot | Shows whose evidence the engine preferred |
| The page the engine pulled | Points to the page that needs the fix |
Before acting on a gap list, check it has these fields. If all you have is a list of missing mentions, treat it as a set of hypotheses to test, not a content brief.
Related reading: Chat Is the New Funnel · How many chats is enough? · Why AI visibility tools disagree
See a gap report built this way for your category. Book an Analytika demo.
Sources
- Don't Measure Once: Measuring Visibility in AI Search, Schulte et al., 2026
- Measuring AI Search Visibility Beyond the First Response, AEOsim
- Visibility Has a Third Axis, AEOsim
- AI Answer Alignment Beyond Factual Accuracy, AEOsim and Kojable