Synthetic Data in LLM Visibility Tools, Part 1: How we Build the Prompt Set

Originally published on LinkedIn. Article 1/n of a series on Synthetic Data in LLM Visibility Tools, by Piush Vaish (Kojable) and Arnav Narang (AEOsim).

Table of Contents

Before we can evaluate an AI system, we need to decide what we're going to ask it.

That sounds obvious, but the prompt set can make or break the study. If the questions are too narrow, repetitive, or unlike the way people actually ask for help, the results won't tell us much. Great analysis later cannot compensate for weak inputs at the start.

This is how we build a prompt corpus that is intentional, traceable, and useful for testing a system in the real world.

People don't ask the same thing twice

Real users rarely phrase a question in exactly the same way. They leave out context. They change their mind halfway through. They use shorthand, make typos, add constraints, switch currencies, or ask the same question differently depending on whether they are in a hurry.

A useful prompt set needs to reflect that. We design for two types of variation:

1. Variation that should not change the answer. "What sales tax applies here?" and "Can you tell me the applicable sales tax?" should lead to the same result.
2. Variation that should change the answer. A different jurisdiction, date, currency, or a small word such as "not" can completely change what a correct response looks like.

Both matter. If we only test paraphrasing, we miss important edge cases. If we only test factual changes, we miss whether the system handles normal human language.

Start with the users we want to represent

We first define what population the prompt set is meant to represent.

For a finance assistant, that means English-language finance and compliance questions from the user roles, jurisdictions, products, and channels we support. We also decide which company sizes, task types, languages, tool setups, and time periods should be represented.

This prevents a common failure mode: creating a dataset that looks varied on paper but does not resemble real usage.

We also separate two goals:

• Prompts designed to mirror typical user behaviour
• Prompts designed to deliberately stress-test the system

Both are valuable, but they should not be confused.

Use a mix of real and synthetic prompts

We do not generate the entire corpus from scratch with AI. Instead we combine three sources:

• 60% de-identified real questions from consented production, search, or support data, weighted to resemble actual traffic
• 20% task-elicited questions from moderated sessions with finance, treasury, payments, and compliance users
• 20% synthetic stress tests designed to cover rare, risky, or adversarial cases that may not appear often in real traffic

Synthetic prompts are useful because they give us control. We can test a specific contrast, introduce a tricky condition, or cover a rare scenario reliably. But synthetic data should not be treated as a substitute for real language. People do not always talk the way a prompt generator expects them to.

For each prompt we store more than the text. We also retain its source, topic, task, intent, user role, geography, language, privacy class, required tools, conversation state, and prompt family. We preserve both the original and normalised text, along with a content hash and immutable ID, so the dataset remains auditable over time.

Syntheticity is a spectrum, not a bucket

The 60/20/20 split is how we manage the corpus and it is worth being precise about what it does and does not mean. It is a budget, not a purity test. A prompt is not simply real or synthetic. It sits somewhere on a spectrum defined by how far its generating process is from the process that produces actual user behaviour.

A search query lifted from a keyword export and reshaped into a question is highly synthetic in one direction: the intent is plausible, but the phrasing is a search artifact. People do not talk to chat interfaces the way they type into search boxes. A prompt written by a solutions engineer who has sat through two hundred discovery calls is less synthetic in phrasing and more synthetic in coverage, because one person's recall of two hundred calls is a biased sample of a much larger population.

Even the 60% has a direction of bias. Production logs describe the users who already found you and already phrase things in ways your system tends to answer.

Once you accept the spectrum, the useful question changes. Not "is this data real" but "where does this prompt sit, which way does its bias point, and does that bias move the metric I am reporting?" That is answerable. The binary version is not. So we record the generating process per prompt as a first-class field, alongside the source label.

What grounds the set when there is no log at all

The three sources above assume you have consented production data. Plenty of evaluation work does not.

Anyone measuring brand visibility inside assistants is in exactly that position. There is no Search Console for ChatGPT. No query log, no impression counts, no keyword volumes. Nobody outside the labs has the real distribution of what buyers type. The common workaround is to pull a client's top search keywords, template them into questions, and call the result grounded. The keywords are real. The questions are not, and the mapping between the two is an assumption rather than a measurement.

Better grounding usually sits inside the business rather than in a keyword tool: CRM objection fields, support ticket categories, sales call transcripts, churn reason codes, the questions a solutions engineer answers in the second half of every demo. These describe the decision a user is making rather than the phrasing they once used on Google. They are harder to collect and worth the effort.

The failure mode to avoid is a closed loop. If one model generates your prompts and the same class of model is the thing under test, the generator's priors and the system's priors come from the same public web. You end up asking a model to imagine the questions, and the imagined questions are shaped by whatever was already dominant in its training data.

Build prompt families, not isolated questions

Rather than creating one-off prompts, we work in prompt families.

A family starts with one canonical task, then creates controlled variations around it. The underlying task might be: "Can this company reclaim sales tax on this expense?" From there we might create:

• A short and direct version
• A detailed version with extra context
• Casual and professional wording
• A version with a typo or missing information
• A follow-up question in a multi-turn conversation
• A similar-looking prompt that actually means something different
• A version where a date, country, amount, or currency changes
• A version with conflicting constraints
• A request that requires access to private information or an external system
• An unsafe or unsupported action

This gives us useful comparisons. We can test whether the model treats true paraphrases consistently, while still catching cases where a tiny factual change should produce a different answer.

We also have domain experts review a representative sample. At least two independent reviewers check that prompts are plausible, that the intended meaning is clear, and that supposed equivalents actually read as equivalent in context. We record agreement rates and how disagreements are resolved.

Randomise deliberately

We do not manually decide every combination of topic, role, template, geography, industry, and integration. We randomise those factors or use a documented experimental design, then filter out combinations that are impossible or nonsensical. There is little value in pairing a sales tax question with a product feature that only exists in Europe.

Every generation factor becomes a field in the final dataset. It should not disappear inside a script where nobody can inspect it later.

We also hold out entire prompt families, templates, and topics before evaluation begins. That means the system is tested on genuinely unseen material, not slight variations of examples already used during development.

Not every prompt is worth the same

A balanced corpus still has to answer one more question before you can compute anything on top of it: how much does each prompt count?

Real query sets carry a volume distribution. A generated set carries whatever distribution the generator happened to produce, which tends to be flat, or shaped by how many variations the model felt like emitting. Frequency is the obvious weight and it is not sufficient on its own. A question asked in the last mile before a decision is worth more than one asked during idle browsing, whatever their relative frequency.

So importance is better treated as a vector than a scalar. Frequency is one component. Proximity to the outcome you care about is another. Risk is a third, and in regulated settings it can outrank both, since a rare question with a compliance consequence deserves more weight than a common one with none.

The weighting scheme belongs in the manifest with everything else. If a score moves, you want to know whether the system changed or the weights did.

Check the dataset before using it

Before a prompt batch is considered ready, we verify that:

• It meets the target volume and balance requirements
• Prompts are unique and fall within sensible length ranges
• Impossible combinations have been removed
• Critical roles and jurisdictions are covered
• Held-out families are genuinely separate from development data
• The weighting scheme is recorded and reproducible
• The corpus can be reproduced from a documented random seed

Reproducibility is especially important. If we cannot regenerate the dataset, or a matching dataset from the recorded inputs and seed, then we do not have a repeatable study. We have a one-off snapshot.

Keep the full audit trail

Every generated batch has an immutable manifest that records:

• The generator code commit and dependency-lock hash
• The templates and transformation rules used
• The randomisation seed
• Target counts and balance constraints
• Any model used to generate synthetic prompts, including its version
• The generation date
• Which families were held out and which were used for development

The raw generated records remain the source of truth. Charts, summaries, and coverage tables are derived from that raw data rather than being manually assembled afterwards.

Why this matters

A strong prompt corpus gives us a realistic mix of everyday questions, deliberately collected edge cases, and controlled stress tests. More importantly, it helps us distinguish between changes that should not affect an answer and changes that absolutely should. That is what makes an evaluation meaningful.

None of this makes generated data unusable. It makes unquantified generated data unusable. If you are buying or building in this space, the questions worth asking are: what grounded the prompt set, and can you inspect it? How is prompt importance weighted, and does the weighting encode anything about the business or only about the language? A vendor who can answer those has done the work. A vendor who says the data is real has either not thought about it or is hoping you have not.

This process does not tell us whether the system's answers are good. That is the next stage.

Next in the series: the two failure modes that survive a clean build. Branded prompts leaking into unbranded sets and inflating the score, and what happens to all of this when a user does not stop at one question.

A practical example of the approach: AI citations fintech Reddit Gemini study.

Piush Vaish (Kojable) and Arnav Narang (AEOsim). This piece was originally published on LinkedIn. Measure AI search visibility with prompt sets built for real buying behaviour on Analytika.