The average can hide the damage.

A pilot report lies folded on a desk. The cover says AI BUSINESS ADVISER, PILOT RESULTS, and under that, in the largest type on the page, one line: average change in weekly profit, approximately zero. Someone has pressed a rubber stamp beside it that reads NO EFFECT ON AVERAGE. Unfold the cover and there are two sheets inside. The first is for half of the users, and their profit rose 10 percent. The second is for the other half, and their profit fell 10 percent. The report, its users and its numbers are invented, chosen so the arithmetic is easy: two groups of equal size, moving by equal amounts in opposite directions, average zero. A committee that reads only the cover concludes that the tool does nothing. Nobody who used it had that experience.

The cover reports the overall average effect. The inside reports the effects across relevant groups. My argument is that an average treatment effect can hide materially different outcomes for different users, and that a responsible rollout needs evidence about those differences wherever they could change the decision. The same argument has a second half that matters as much as the first. Any data set can be sliced until some slice tells a pleasing story, and the discipline is to name the groups that matter before the results arrive, not to go looking for them afterward.

This is not a claim that averages are useless, or that some kind of user always benefits from AI while another always loses. The evidence below comes from one experiment, with one tool, in one country, and its pattern belongs to that setting. The claim is narrower: an average answers one question, and a rollout usually has to answer two.

What an average answers

The strongest way to learn what a tool does is a randomized experiment. Some people, chosen by lot, get the tool; the rest get something else. Because the lottery decides who is in which group, the two groups are alike on average in everything except the tool, and the difference between their average outcomes can be credited to it. That difference is the average treatment effect, and it is a real achievement. It is the closest thing to a fair answer that the messy world allows.

But it is an answer to a specific question, and the question has three parts. It says what happened on average, in this population, under this intervention, compared with this alternative. Change any of the three and the answer can change. What it does not say, and was never designed to say, is what happened to each person in the study.

The reason is arithmetic. An average is a weighted sum. An average change of zero can mean that nobody changed, or that some people gained and others lost in amounts that happened to cancel. The cover of the folded report is accurate in both cases. Only the inside sheets tell them apart, and the two cases call for opposite decisions: one says the tool is harmless and pointless, the other says it is consequential and needs managing.

An adviser in every pocket

Nicholas Otis, Rowan Clarke, Solène Delecourt, David Holtz and Rembrand Koning ran exactly this kind of test with small businesses in Kenya. They recruited 640 entrepreneurs for a five-month randomized trial.² Half, chosen at random, got a business mentor built on GPT-4 that they could message through WhatsApp. The other half received a standard business guide.¹² Recruitment began in May 2023, and the researchers tracked each business's revenue and profit through repeated surveys.³ Very few people dropped out: 634 of the 640 completed at least one survey after the treatment began.³

One detail of the design matters for everything that follows. What was randomized was access. Being offered an adviser is not the same as using it, and the experiment measures the effect of the offer. Most people did use it. In the working paper, 88.2 percent of the businesses that had been performing poorly before the trial asked the adviser at least one question, against 84.2 percent of those that had been performing well.³ But an organization that reads a result like this as the effect of using AI, rather than the effect of being given it, has quietly changed the question.

The researchers split the entrepreneurs at the median of their performance before the trial began, into high and low performers.² Here is what the version of their paper revised in July 2024 reports. Overall, they could not reject the hypothesis that the adviser had no average effect. The effect of the adviser for high performers was 0.27 standard deviations larger than for low performers. Looked at separately, high performers did just over 15 percent better with the adviser, and low performers did about 8 percent worse.¹

"Could not reject" deserves a careful reading. It does not mean the researchers proved the average effect was zero. It means their data were consistent with zero, and also with modest effects in either direction; the study could not tell which. A board summary that turns this into "AI had no effect" has converted an absence of evidence into evidence of absence, and has also thrown away the inside sheets.

The numbers moved somewhat as the work matured. In an account the same researchers published in MIT Sloan Management Review in April 2026, the gain for high performers is 15 percent and the decline for businesses that were already struggling is nearly 10 percent.⁴ Different versions of a study can report different estimates as the analysis is refined, and it would be wrong to average them or to mix one version's gain with another's loss. What every version I have read agrees on is the direction: the same offer helped one group and hurt the other. The size of the gap is less settled than its sign.

Same questions, different choices

The obvious explanation would be that the struggling businesses used the adviser badly: asked fewer questions, vaguer questions, or got worse answers. The researchers looked and did not find that. The rise in inequality, they write, did not come from differences in the questions entrepreneurs asked or the advice they received, but from how they selected from that advice and put it into practice.¹ Low performers were especially likely to act on generic suggestions, cutting prices and spending on advertising, while the high performers appear to have been better at screening what the adviser told them.³ In the 2026 account, the authors put it more bluntly: weaker performers followed generic or misleading advice because they lacked the judgment to filter it.⁴

That mechanism is worth holding in mind, because it is general. An adviser that produces a fluent list of plausible options leaves the hardest part of the job, choosing among them, to the person who asked. Whoever is better placed to choose gets more out of it. For someone without the context to reject a bad suggestion, a confident and generic answer can be worse than no answer at all.

Two cautions keep this from turning into a slogan. "Low performer" here describes a business's results in a few months before the trial, not the worth or the intelligence of the person running it. A business can struggle because of a poor location, thin margins or a bad season, and the label says nothing about anyone's skill with AI. And the study shows that the gap appeared, not which remedy would close it. Better prompts, narrower tasks, a human mentor alongside the machine: each is a hypothesis, and each would need its own test.

The average depends on the mix

There is a second way an average misleads, and it bites at exactly the moment a pilot becomes a rollout. Because an average is weighted by how many people are in each group, it changes whenever the mix of people changes, even if the tool and its effect on each group stay exactly the same.

Go back to the invented report. Its pilot happened to be half group 1 and half group 2, so the average was zero. Suppose the first users were volunteers, and volunteers skew toward the people who expect to benefit: the pilot might have been four in five group 1, with an average gain of 6 percent. Roll that tool out to everyone, and the mix becomes even; the average falls to zero. Roll it out to a branch network where most users are in group 2, and the average turns negative. Nothing about the tool changed. Only the population did. The sheet below lets you change the mix yourself. Drag across it, or tab to it and use the arrow keys.

The same arithmetic runs the other way, and that is the case that should worry a board. A clearly positive average can sit on top of a group that is being harmed. In the model, a user base that is four in five group 1 shows a healthy average gain while one user in five loses 10 percent. If that fifth are the customers with the least room for error, the average is reporting good news about a rollout that is doing damage.

What a rollout owes the second sheet

None of this means the average should be ignored. It means the evaluation should carry the inside sheets with it. For a tool that will reach many people, I would ask for a short table before the rollout decision, with one row for each group that was named before the analysis began. Each row would give the group's outcome before the tool arrived, how many people were in the treatment and comparison groups, the estimated effect with its uncertainty, how many people dropped out of each arm, and the count of consequential errors: the advice acted on that cost someone money, a customer or a license.

The groups should be chosen because they could change the decision. Baseline performance, as in the Kenyan study, is one obvious candidate. Others are the kind of work, the stakes of a mistake, and how much the user can check the tool's answer. A group that would receive the same decision whatever its result does not need a row.

Small groups need humility. A row with forty people in it will have a wide interval, and the honest response is to say the result is unclear and collect more data, not to rank the groups. And the table is not finished when the tool ships. The population that uses a tool after launch is rarely the population that tested it, the work changes, and the model itself is updated. The groups that were monitored in the pilot should be monitored in production, with the same rows.

Where a group is harmed, the response is a choice to be tested, not a conclusion to be drawn. A rollout might give one group more support, narrow the tasks the tool handles for them, or put a person in to review advice before it is acted on. Each of those has costs, and none of them is known to work until it has been tried against a comparison.

Objections, considered

Slice the data enough and you will find anything. This is the strongest objection, and it is correct. The classic demonstration comes from ISIS-2, a trial of more than 17,000 heart-attack patients in which aspirin clearly reduced deaths. When the investigators split the results by star sign, aspirin appeared to do nothing for patients born under Libra or Gemini, a finding produced by the play of chance and nothing else.⁵ The defense is not to stop looking at groups but to look in a disciplined way: name the groups in advance, report every one of them rather than the interesting ones, show the uncertainty, and replicate before treating a pattern as general. The Kenyan team pre-registered their test for differences by baseline performance, which is why it deserves more weight than a split found afterward.³

Sometimes the average is the right number. Also true. When a decision will apply to everyone and there is no practical way to treat groups differently, the average is what the decision turns on, and a positive one can justify going ahead. The inside sheets still matter in that case, because they say who will bear the cost of a decision taken on the average, and whether that cost is acceptable.

The answer to a harmed group is support, not exclusion. Often it is. Withholding a useful tool from the people who are struggling can widen the very gap the experiment found. But support is itself an intervention. A mentor, a training course or a narrower version of the tool might close the gap or might not, and it needs the same evaluation as the tool it is meant to repair.

Kenyan microbusinesses are not my company. Correct, and the study does not claim to be. It tested one GPT-4 interface, in 2023, with small businesses in one country. Enterprise staff, other markets and later models may show smaller gaps, larger ones, or gaps between different groups. One experiment is not a law. What transfers is not the numbers but the method, and the warning that a flat average can be the sum of real movements in opposite directions.

An average result is not a uniform experience.

Before the next rollout

• Who improved? Which group, by how much, with what uncertainty, and how many people were in it.
• Who got worse? And what did the decline cost them: money, customers, or a mistake that cannot be undone.
• Were the groups named before the results? If not, treat the pattern as a hypothesis for the next test.
• What would change the rollout? The result, for each group, that would mean more support, a narrower scope, review, or stopping.
• Who keeps watching after launch? The same rows, in production, as the users and the work change.

Back to the folded report on the desk. The cover is not wrong; the average really was about zero. The mistake would be to read the cover and file the report. Unfold it, staple the inside sheets to the front, and send all three pages to the committee. The decision they make may be the same. But they will know whom it is being made for.