The envelope on the desk is labeled in marker: "Concept test, 1,000 responses." Inside is a spreadsheet with a thousand rows, each with an age, an income band, a region and an answer to the question that matters, whether the respondent would buy the new product at the proposed price. The work was finished overnight. Nobody was contacted to answer this survey. The rows were generated by a language model that had been given the product description and the notes from forty customer interviews conducted last quarter. The envelope in this essay is invented, but studies built this way are becoming common, and the question it raises is not going away: what, exactly, does the envelope contain?
The appeal is easy to see. In an article published this week in MIT Sloan Management Review, the marketing researchers Neeraj Arora, Ishita Chakraborty and Yohei Nishimura describe how large language models are compressing research that used to take months and cost tens of thousands of dollars into days, through synthetic "digital twins" of consumers for rapid concept testing, AI-moderated interviews and faster analysis of unstructured data.1 A team that can test ten concepts in the time it once took to test one will learn faster, and it would be foolish to wave that away.
My argument is that the envelope holds two different kinds of material that look identical on a spreadsheet. One is simulated responses: answers a model generated because they were plausible given what it had been told. The other is observed human responses: answers that particular people actually gave. Simulated responses can be very useful, and in some studies they track real answers remarkably well. But they become evidence about a population of customers only through validation against that population, for the specific use at hand. Generating more of them does not add observations of real customers, and it does not remove an error the model shares across every row. This is not a claim that synthetic research is worthless, nor that a thousand human respondents are automatically representative. It is a claim about what each kind of row can be trusted to say.
Concept test
1,000 responses
1,000 generated · 40 people heard
A thousand draws from one model, sharing whatever it got wrong.
Forty people, actually heard. This is what the rest stands on.
Generating is not measuring
Start with what a survey of people does. It selects respondents from a population, asks them a question, and records what they say. Each answer is an observation of one person, made on one day, with all the noise that implies: people misread questions, answer carelessly, or say what they think the asker wants to hear. But each answer is also, in an important sense, independent. One person's misreading does not cause the next person's. That independence is what lets the noise average out as the sample grows, and it is the basis of the margin of error printed under every poll.
A synthetic panel works differently. Each row is a sample from the same model, conditioned on the same instructions and the same grounding material. The model may vary its answers from row to row, and it can be prompted to vary them in realistic ways. But whatever the model believes about these customers, it believes in every row. If it overestimates how much they care about price, that overestimate is present a thousand times. Asking it more often refines our picture of what the model thinks. It does not, by itself, tell us anything new about what customers think.
This is why the label on the envelope matters. "1,000 responses" invites a reader to imagine a thousand people. What the envelope actually holds is a thousand draws from one instrument, plus forty people who were heard.
The strongest case
The best synthetic work is far more than a model guessing from a demographic profile, and it deserves to be described at its best. The most careful example I know is a study by Joon Sung Park and colleagues at Stanford and elsewhere, first posted in November 2024. They recruited 1,052 Americans in a sample stratified by age, region, education, ethnicity, gender, income, political ideology and other characteristics. Each person sat for a two-hour interview about their life, and the researchers built an AI agent for each one from the transcript.2
They then asked both the people and their agents a battery of questions from the General Social Survey, and asked the people the same questions again two weeks later. People matched their own earlier answers 81.25 percent of the time; the agents matched the people's answers 68.85 percent of the time. Dividing one by the other gives the headline figure: the agents were 85 percent as accurate as the people were at predicting themselves. Agents given only demographic descriptions reached 71 percent on the same measure, and the interview-based agents showed smaller accuracy gaps across political, racial and gender groups. In five well-known social science experiments, the agents reproduced the same four results that the human participants reproduced.2
Arora, Chakraborty and Nishimura have made a related case for commercial research. In a 2025 paper in the Journal of Marketing, they argued for a hybrid of people and language models across the research process. In qualitative work, they found the hybrid produced richer data than human-only work and matched people at summarizing themes. In quantitative work, the model correctly picked the direction and valence of answers, and its synthetic data improved when it was given examples and relevant documents to draw on. They demonstrated the approach by working with a Fortune 500 food company to replicate a 2019 study with GPT-4.3
These are serious results. They show that a model grounded in real human material can reproduce real human answers well, under the conditions tested.
What, exactly, was validated?
"Under the conditions tested" is doing a great deal of work in that sentence, and it is worth slowing down. The Park study validated agents built from two-hour interviews with specific people, predicting those same people's answers to established survey items and well-studied experiments. That is an impressive target. It is not the same target as predicting how many customers who were never interviewed would buy a product that did not exist when the model was trained. The Arora study validated, among other things, that the model got the direction of answers right in a replication for one company. Getting the direction right is useful for choosing which concept to test further. It is not the same as getting the level right, which is what a revenue forecast needs.
A synthetic method can be validated against at least five different targets: the answers of particular people, answers to particular questions, the proportions in a population, the ranking of options, or actual behavior such as purchases. Success on one does not certify the others. A model that ranks three package designs in the same order as customers might still be badly wrong about what share of them would pay the price on the box.
Marko Sarstedt and colleagues reached a similar conclusion in a 2024 review of studies that compared synthetic "silicon samples" with human ones. They found that results vary considerably across domains, and argued that the most promising uses are upstream in the research process, in qualitative pretesting and pilot studies, where the aim is to improve the design of the study that follows.4 That is a sensible division of labor. It also shows why the question "is synthetic research accurate?" has no general answer. The useful question is narrower: accurate for which target, in which domain, checked against whom?
Precision around the wrong answer
The most seductive feature of a synthetic panel is how stable its numbers become. Run it once with a hundred respondents and the share who say they would buy might come out at 58 percent; run it again and it might be 66. Run it with three thousand and the answer settles down, run after run, near the same figure. That stability looks like the narrowing margin of error of a large human survey. It is something else.
Figure 2 is a model, with invented numbers, built to show the difference. Suppose the model's honest view, given its instructions, is that 62 percent of customers would buy, and suppose a careful survey of real customers finds 48 percent. Small synthetic runs scatter widely around 62. Large ones cluster tightly around it. At no size do they move toward 48, because the gap is not noise that averages out. It is a property of the instrument, shared by every row.
| Synthetic panel size | Center of repeated runs | Approximate spread of runs | Human benchmark |
|---|---|---|---|
| 30 rows | 62% | about 47% to 76% | 48% |
| 300 rows | 62% | about 57% to 67% | 48% |
| 3,000 rows | 62% | about 60% to 64% | 48% |
The practical consequence is that the usual arithmetic of sample size does not transfer. With a human sample, more respondents shrink the uncertainty about the population, under assumptions about how they were selected. With a synthetic sample, more rows shrink the uncertainty about the model. Whether the model is close to the population is a separate question, and only observed human answers can settle it. A synthetic result reported with a confidence interval computed as if its rows were people is reporting the wrong kind of precision.
A workflow that uses both
None of this argues for throwing the envelope away. It argues for organizing the work so that each kind of row does what it is good at.
Explore and pretest synthetically. Use simulated respondents to find confusing questions, generate hypotheses, compare many concepts cheaply, and design a better human study. This is where Sarstedt and colleagues see the most promise, and where an error costs least.4
Reserve real people for validation. Hold back some questions, concepts or customer groups, ask real people, and compare. Validation does not have to repeat the whole study. It has to test the specific target the decision depends on: the level, if the decision is a forecast; the ranking, if it is a choice between options.
Check the groups that matter. A model that is accurate on average can be wrong for a particular segment. Park's team found that interview-grounded agents narrowed gaps across groups compared with demographic prompts; that is a result worth checking for, not assuming.2
Go back to people when things change. A model grounded in last quarter's interviews knows last quarter's customers. A new market, a new price or a shift in circumstances is a reason to collect new human evidence before trusting old grounding.
Label every row. Keep simulated and observed rows separate, each tagged with its origin, its date and its intended use, so that a chart built later cannot quietly mix them.
Objections, considered
Human surveys are flawed too. They are. Samples are skewed by who agrees to answer, questions are misread, and people's stated intentions often differ from what they later do. A thousand human respondents are not automatically representative, and a careless human survey can be worse than a careful synthetic one. The comparison is not between a perfect human study and an imperfect model. It is between two instruments, each with its own errors, and the argument here is only that the model's errors must be checked against people rather than assumed away.
For some questions, synthetic methods already outperform weak human research. Plausibly, especially for wording, early concept screening and the kinds of qualitative depth the Arora paper describes.3 If a method has been shown to work for a stated task, using it for that task is good practice. The standard proposed here is not "human only." It is demonstrated usefulness for the stated task, with transparent provenance and honest limits.
Validation defeats the purpose of saving time and money. It does not have to. A validation sample can be small relative to the synthetic exploration it checks, and it can be reused across studies until the population or the question changes. The cost of skipping it is paid later, in a launch or a forecast built on a number that was precise, stable and wrong.
The number of answers is not the number of people you have heard from.
Before you open the next envelope
- What human evidence grounds the model? Which people, how many, when, and how they were chosen.
- Which held-out questions does it predict well? Against which real respondents, and for which target: individual answers, proportions, rankings or behavior.
- Which groups were checked? Whether accuracy holds for the segments the decision actually turns on.
- What is it allowed to decide? Whether the result is for exploring and pretesting, or for a forecast, a price or a launch.
Back to the envelope. Nothing inside needs to be thrown out; the generated rows may be the fastest way to find out which concept deserves a real test. But the label is wrong, and a pen can fix it. Strike through "1,000 responses." Underneath, write what the envelope actually holds: a thousand generated answers, grounded in forty people who were heard, made on a known date, for pretesting. Then put the forty interviews where everyone can see them, because they are the only part of the envelope that came from a customer.