AEOsim blog
Syntheticity is a spectrum, not a flaw
Table of contents
"Is your data real?" is the wrong question
Every AI visibility tool simulates buyers. None of them can see what real people type into ChatGPT, because no engine publishes that log. So the first objection a sceptical buyer raises is fair: if the prompts are made up, what is the number worth?
The honest answer is that "real versus synthetic" is not a useful split. Every prompt sits somewhere on a spectrum, defined by how far its generating process is from the process that produces actual buyer behaviour. And every position on that spectrum has a bias that points in a known direction.
The useful question is not "is this data real?" It is: where does each prompt sit, which way does its bias point, and does that bias move the metric being reported? That question has an answer. The binary one does not.
Five rungs, five biases
Take one buyer intent, "which carry-on bag fits a budget airline's cabin limit", and generate the prompt five ways (Bridge The Real2Sim Gap):
| Rung | How the prompt was made | Its main bias |
|---|---|---|
| 1 | Transcribed word for word from a customer call or chat log | Represents buyers who already found you; small samples |
| 2 | The same question with names and jargon normalised | Loses the messy phrasing engines react to |
| 3 | A buyer given the task and asked to type what they would send | Lab setting; people behave a little more carefully |
| 4 | A template built from constraints seen across many calls, filled with plausible values | Covers the space well; phrasing is tidier than real life |
| 5 | A model given the category name and asked to write prompts | Echoes whatever the model already believes about the category |
Two things follow from the ladder.
Rung does not equal value. A rung-1 prompt can be a settled question that separates nobody. A rung-5 prompt can be the probe that finds a real gap. The rung tells you the failure mode, not the usefulness. Information value depends on whether the prompt separates your brand from competitors, which is a separate test.
Even the "real" rung is biased. Production logs describe the buyers who already reached you and already phrase things the way your system answers well. A keyword export reshaped into a question is plausible in intent and synthetic in phrasing, because nobody talks to a chat assistant the way they type into a search box.
Why it compounds across turns, and why single-turn tools have it too
In a multi-turn simulation, every buyer turn is generated, not just the first. If the opening prompt sits at rung 4, the follow-ups usually sit there too, or further down. A scripted constraint turn is tidier than a real buyer's, and a model-driven user simulator tends to ask the questions a model finds natural. Each turn adds a little more distance from real behaviour.
The sharpest version of this is a closed loop. If one model writes the buyer's turns and the same class of model answers them, both draw their priors from the same public web. The simulated buyer asks what the engine expects to be asked, and the result can look cleaner than reality (Synthetic Data, Part 1).
None of this is unique to multi-turn tools. A single-turn tracker that templates SEO keywords into questions is also at rung 4 or 5. It simply has one turn's worth of syntheticity instead of five, and it never reaches the decision turn where the bias would show up. Measuring less of the conversation does not make the measurement more real.
The gain from simulation is worth the cost when it is controlled. Simulation can do three things observation cannot:
- Adversarial probing. Put your top three sales objections in a buyer's words and see what the engine says, before a real buyer asks.
- Forced elicitation. Ask the engine to commit to one pick. Open questions often produce a shortlist with no decision in it.
- Multi-turn control. Fix which constraint enters at which turn, and vary one thing at a time.
How to disclose and control syntheticity
- Record the rung per prompt. Store how each prompt was generated as a field, next to its source and intent. Report the mix, not a "real" label.
- Ground in business artifacts. Sales call notes, support ticket categories, return reasons and CRM objection fields move prompts up the ladder in intent, even when phrasing is templated.
- Break the closed loop. Use a different model family, or human-written constraint turns, for the simulated buyer than for the engine under test.
- Keep a small real panel. A handful of verbatim buyer questions, run alongside the designed set, acts as a calibration check on absolute levels.
- Validate outside the simulation. Check whether simulated findings line up with things you can observe: AI referral traffic, questions sales hears, and shifts in how inbound leads phrase their needs.
An AI visibility tool that can tell you the mix, the bias direction, and how it validates has taken syntheticity seriously. One that says the data is simply "real" has not thought about it.
Related reading: Chat Is the New Funnel · Building a prompt set that represents real buyers · How many chats is enough?
See the prompt mix behind your numbers. Book an Analytika demo.
Sources
- Synthetic Data in LLM Visibility Tools, Part 1, AEOsim and Kojable
- Bridge The Real2Sim Gap, AEOsim and Kojable