A pilot needs a way to be wrong.

An index card lies face up on a conference table. Across the top it says PILOT SCORECARD, and beneath that, in a sponsor's neat hand, three lines with ticks beside them: demo completed, thirty-eight people attended, the assistant handled all twenty test claims. Someone has copied a comment from the session at the bottom, "Can we have it now?", and pressed a rubber stamp beside it that reads SUCCESS. The insurer, the pilot and the card are invented, and so is every number that follows. The steering committee looks at the card for about a minute and approves the next tranche of funding, which was already in the year's budget. Nobody in the room could say what the card would have had to show for them to decide anything else.

The pilot did produce something. It showed demonstrated feasibility: the technology can do the task, in front of people, on the cases it was given. What it did not produce is decision-changing evidence, a result that could have sent the money somewhere else. My argument is that a pilot has decision value only when some result it could plausibly return would change the next investment decision, and that the sponsor should write down which result that is before the pilot begins. A demonstration designed only to succeed answers whether the thing works. It leaves untested whether the thing is worth buying.

This is not an argument that every prototype must be run as an experiment. Early on, feasibility is often the real uncertainty: nobody knows whether a model can read a scanned claim form at all, and a week of building settles it. The requirement is narrower. Whatever is being run, say which uncertainty it resolves, and do not let the answer to one question stand in for the answer to another.

Learning as the unit of progress

The best-known case for treating early work as an experiment is Eric Ries's The Lean Startup, published in 2011.¹ Ries starts from the observation that a startup works under extreme uncertainty, and he proposes a measure of progress suited to it. In his words, "the unit of progress for Lean Startups is validated learning": learning that has been checked against experiments, not assumed.²

The method that goes with it is a loop. Build something, measure what happens, learn from the measurement, and then decide whether to pivot or persevere.² The smallest thing worth building is what Ries calls a minimum viable product, whose job is to begin the learning as quickly as possible.² For a pilot, the important step is the last one. Measuring is only half the loop. It closes when a measurement settles a decision, and a team that cannot say which decision is pending has built and measured without learning anything it can act on.

The book has limits, and they matter here. It is a framework drawn from case histories and the author's own practice, not a controlled test of its advice. It was written for startups, where the product itself is the open question, and it does not show that every team should put a minimum viable product into a live workflow where a mistake costs a customer money or does harm. What I take from it is the question it keeps asking, what have we learned that changes what we do next, and not a set of instructions for an insurer's claims floor.

What the trials found

Whether thinking this way improves decisions is an empirical question, and a group of researchers centered on Bocconi University in Milan has tested it with randomized trials. In the first, published in Management Science in 2020, Arnaldo Camuffo, Alessandro Cordova, Alfonso Gambardella and Chiara Spina followed 116 Italian startups for about a year.³ The treated founders, chosen at random, were taught to build a framework for predicting how their idea would perform and to run rigorous tests of their hypotheses. The control group followed its intuitions about how to judge its idea. The founders taught to behave like scientists performed better, and they were more likely to pivot to a different idea.³

One trial of 116 firms can mislead, so the team ran the test again at a larger scale. In a 2024 paper in the Strategic Management Journal, Camuffo, Gambardella, Spina and three colleagues report four randomized control trials involving 759 firms.⁴ The larger sample sharpened the picture. Firms trained in the scientific approach terminated more of their ideas, and their pivots followed a pattern rather than simply becoming more frequent: they tended to make a few strategic changes, where other firms more often made none or kept changing.⁴ The authors interpret this as more efficient search. Founders who think like scientists hold their own theory with a measure of doubt, because they see that the world may not turn out as they predicted, and so they find out sooner which ideas are not viable.⁴

Two cautions. The trials tested a way of making decisions, taught to entrepreneurs at an early stage. They did not test enterprise AI pilots, and nothing in them shows that the method guarantees revenue. And the finding I lean on most is the least glamorous one. A measurable effect of the scientific approach was more ideas abandoned. In a startup, that is a founder's year saved. In a large company, it is a tranche not spent.

Four questions a demonstration leaves open

Feasibility deserves its due. Whether the technology can do the task at all is a fair first question, and sometimes the hardest. But once it is answered, four more remain, and a successful demonstration does not answer any of them on its own.

Does it improve an outcome that matters, counted in finished work rather than impressed spectators? For whom: every claims handler, or only the experienced ones who already know how to check its work, or only on simple claims? At what total cost, counting the licences but also the review time, the support tickets and the hours spent correcting it? And under what conditions: the twenty test claims chosen for the session, or a Monday in January when the queue is full of storm damage?

The demonstration on the index card was built to succeed, and there was nothing dishonest in that. A demonstration is supposed to show a thing at its best. The trouble starts when the card is read as if it had answered the other four questions, because nothing about a demonstration can come out badly enough to change the decision. If no result could have sent the money elsewhere, the result carried no information about where the money should go.

Write the decision before the test

The remedy is to write the decision down before the pilot starts, in terms a result could fail. For the invented claims assistant, the sponsor might write: expand if claims closed correctly rise by at least 10 percent against the comparison group, with consequential errors up by no more than one per 1,000 claims and no unacceptable rise in support effort or in handlers abandoning the tool. Stop if the gain is under 3 percent, or if consequential errors rise by more than 2.5 per 1,000. Anything in between means revise and test again.

Those thresholds are invented, and they should be. Each is a business judgment: how much improvement pays for the tool, how many additional errors the company and its customers can absorb, and who answers for them. There is no universal benchmark to borrow. What matters is that the line is drawn while nobody knows which side of it the pilot will land on, because a line drawn after the results arrive has a way of running just beneath them.

Writing the rule first also forces the three outcomes to be three different actions. Expand, revise and stop should each carry a budget consequence the committee has agreed to in advance. The ledger below applies one prewritten rule to three invented pilots whose demonstrations all went well. Drag across the sheet, or tab to it and use the arrow keys, to place a result of your own.

A comparison you can believe

A decision rule is only as good as the comparison underneath it. Ten percent more claims closed than what? The strongest answer is random assignment: some teams, chosen by lot, use the tool, and the rest carry on as before. That is what gave the Bocconi trials their force. Where a lottery is impractical, weaker designs still beat a plain before and after. In a staged rollout, teams start on different dates and the late starters serve as the comparison for the early ones. A matched comparison sets the pilot teams against teams with similar work. An interrupted time series looks for a break in a long run of weekly figures at the moment the tool arrived. Each has known weaknesses, and the pilot report should state them.

The traps they guard against are familiar. Volunteers who ask to try a new tool are often the people who would have improved anyway. Novelty lifts effort for a few weeks and then fades. The workload shifts underneath the test, and a quiet month makes any tool look good. A simple before and after absorbs all three and reports them as the tool's effect.

Even careful researchers find the comparison can move beneath them. In February 2026 the research group METR explained why it was redesigning its study of AI tools and experienced software developers. Its early 2025 study had found that tasks took 19 percent longer when the developers could use AI. In the follow-up, many developers declined to take part because they did not want to work without AI, 30 to 50 percent said they were choosing not to submit some tasks for the same reason, and developers running several AI agents at once made time on a task hard to measure.⁵ METR concluded that its new data gave an unreliable signal of the current effect.⁵ The lesson is not a number to freeze, from 2025 or from any other year. It is that who takes part, and which work they bring, can change during a test, so the comparison has to be inspected before the result is believed.

Keep the original card on the table

When the results arrive, the most useful object in the room is the scorecard written before the test. It keeps out two quiet distortions. One is moving the line: deciding afterward that a 6 percent gain was always the target. The other is relabeling. Pilots often turn up something nobody planned, and sometimes it is valuable. Suppose the handlers barely used the assistant's drafting but came to rely on its summaries of long medical reports. That is a new hypothesis, and it deserves its own card, its own comparison and its own rule. It is not a reason to stamp the original pilot SUCCESS.

Nor is every stopped pilot a triumph. It is tempting to call any failed experiment valuable learning, and often it is. A pilot that rejects an expensive idea at modest cost can be worth more than one that makes money, because it keeps the next tranche away from an idea that would not have paid. But the claim has to be earned. The pilot must have reduced an uncertainty that mattered, at a cost in proportion to what was at stake. An experiment too small to detect anything, or one that tested a question nobody needed answered, has not taught anything by failing.

Objections, considered

Small pilots miss real effects. They do. A pilot with four teams and six weeks can easily fail to detect an improvement that is really there, and a rule that says stop below 3 percent can kill a good tool on noise. The answer is not to drop the rule but to write it with that weakness in view. A pilot may reduce uncertainty without settling it, and the rule can include a fourth outcome, extend the test, for a result too imprecise to support any of the other three. What the rule should never allow is that every result, however small, means expand.

Some things should not be tested on live work. A tool that helps decide benefit eligibility, or flags patients for follow-up, should not be randomized onto real cases just to get a clean comparison. Shadow operation does much of the work instead: the tool runs alongside the people, its recommendations are recorded but not acted on, and an independent reviewer compares the two. That answers fewer questions than a live trial, and the report should say which ones remain open.

Gates slow learning. If every prototype needs a written hypothesis, a comparison design and a committee, teams will stop building prototypes, and the company will learn less. The rule should be in proportion to what the next decision costs. A two-week feasibility test needs one sentence. A rollout to two thousand handlers needs the whole card.

The best result may be one nobody predicted. True, and a rigid rule can blind a sponsor to it. But noticing a surprise is compatible with keeping score. Register the new use as a new hypothesis, give it its own test, and leave the original result where it fell.

If no result can change the decision, you are staging a demonstration.

Before the next pilot starts

• What uncertainty does it resolve? Feasibility, outcome, who benefits, total cost or conditions, named before the work begins.
• Which result would stop the next expenditure? The stop line, written beside the expand and revise lines, with the budget consequence of each.
• What is the comparison? Random assignment, a staged rollout, a matched group or a long time series, and the weakness of whichever is chosen.
• Who keeps the original card? The person who will put the prewritten rule beside the results, and send any new idea to a test of its own.

Back to the index card on the conference table. Turn it over and the reverse is blank: no hypothesis, no comparison, no line a result could fall below. The stamp on the front is accurate as far as it goes. The demonstration did succeed, and the honest word for what it showed is feasible. Fill in the reverse before the next tranche is spent, and the next stamp will say something a committee can act on: expand, revise or stop.