In 1896 Sakichi Toyoda finished a power loom built of steel and wood, the first of its kind in Japan. Among its parts was a small device that a visitor to the mill might never notice. A loom weaves by carrying a crosswise thread, the weft, back and forth through the lengthwise ones. If the weft broke on an ordinary loom, the machine kept going and wove cloth with a flaw in it until someone saw the problem. Toyoda's loom carried a weft-breakage automatic stopping device.12 When the thread broke, the loom stopped.
That small device is the seed of the Toyota Production System, the most studied way of making things in the world, and it answers a question that companies are now asking about AI agents. An agent is software that acts: it reads the invoices, updates the records, sends the email. When it acts wrongly, who notices? Most organizations reach for one of two answers. One is a person watching every agent: a human approves each action, or reviews each result, before it counts. The other is an agent that knows when to stop: it is built to recognize when something is abnormal, halt, and call a person to that one spot.
My argument is that the first answer, which feels like the responsible one, is the old ordinary loom with a weaver standing beside it, and that the second is what Toyota has been building for more than a century. The watcher does not scale, and it puts people in the wrong place. The stop cord does, but only after someone has done the work that tells the machine what abnormal looks like. This is not a claim that agents need no oversight; some actions deserve a person's signature every time. The claim is about the default. Oversight that depends on a person seeing every action fails quietly as volume grows. Oversight that depends on the machine stopping at the abnormality gets better every time it fires.
Two defaults, both failing
Surveys published this fall show companies swinging between the two. Each was run by a vendor or drawn from self-selected readers, and each measures different things in a different sample, so none is a census and they should not be added together. Read one at a time, they point the same way.
A survey by the software company Nylas asked 1,030 American professionals who build, buy or influence AI systems about their agents. Fifty-four percent said an agent had made a mistake in the past 90 days, most often by starting a workflow the team then had to reverse.4 A survey of 417 governance, risk and audit leaders by the software vendor Optro found that 34 percent of organizations had acted on an agent decision or output that turned out to be wrong.5 In a survey of 300 technology and risk leaders for Redwood Software, 77 percent said they had suffered some negative consequence from an agent nobody was governing, most often an unexpected cloud bill.6
The reaction is visible too. VentureBeat surveys a different self-selected group of its readers and panel members each month, so its comparisons are between two groups, not changes inside one. Among respondents whose organizations deploy autonomous agents, the share that let agents push some changes to production on automated checks alone, or were building toward it, fell from 75 percent in July to 56 percent in August. The share expecting to keep a person reviewing agents' production changes for the foreseeable future rose from 20 percent to 42 percent.7 The retreat from trust is a retreat toward the watcher.
And the watcher has a cost of its own. In the Optro survey, 46 percent said employees were now spending more time reviewing, validating or correcting AI output, not less.5 The machine was supposed to free the person, and the person has become its supervisor.
What the loom knew
Jeffrey Liker's The Toyota Way, published in 2004, distilled the company's practice into 14 management principles, grouped under philosophy, process, people and problem solving.3 Three of them read as if they had been written for this moment. The first is Principle 5: "Build a culture of stopping to fix problems, to get quality right the first time." This is where Liker places jidoka, the idea that grew from Toyoda's loom, and andon, the signal that calls for help.
Toyota's own description of jidoka is more exact than the slogan. The company translates the word loosely as automation with a human touch. The point, it says, is to stop immediately when something abnormal is detected, so that no defective product is made, and to raise productivity enough "to eliminate the need for people to be simply watching over machines."1 Either the equipment detects the abnormality and stops itself, or an operator stops the line by pulling a cord. When the line stops, the andon board lights up to show where. People need only respond to the abnormality, Toyota writes, so no one has to stand watch over the equipment.
Notice what that does to the people. They are not removed; they are moved, from beside the machine to the spot where it failed, while the evidence is still there. Liker opens his chapter on Principle 5 with a line from Fujio Cho, then Toyota's president, recalling that Taiichi Ohno used to say a problem found by stopping the line should be fixed no later than the next morning.3 The stop starts a repair, and the repair is meant to make the same stop unnecessary next time.
A person saw every step, and approved the wrong one along with the rest.
Nobody watched. The run stopped at the abnormality and called its owner.
An agent with a stop cord
Translated to software, jidoka asks four unexotic things of an agent. It carries written expectations of normal: how many records a task should touch, which actions it has taken before, what a reasonable amount looks like. It stops cleanly when one is broken, before the abnormal action takes effect. The stop reaches a named person who owns the work, not a channel where no one has to answer. And every stop ends in a countermeasure, a fix to the task, the data or the expectation, so the same stop does not recur tomorrow.
Figure 1 shows the difference on one task. In the gold version a person approves the fourth step, which posts 4,812 updates where the task called for about forty, because the three steps before it were fine and the screen looked like every other screen that day. In the blue version nobody approves the routine steps, and the agent stops with nothing posted. Its owner gets one call about one abnormality.
The gold version has more human attention in it than the blue one. The trouble is where it goes: mostly to steps that were never in doubt. An earlier dispatch here, "Approval needs a budget of attention.", made the case that approval systems should be judged by where they send attention, not by how many approvals they log. Jidoka is the oldest working answer to where it should go: to the abnormality, and only there.
You cannot build the stop until you know normal
The blue version has a catch. The agent could stop only because someone had told it that the task produces about forty updates, and somebody had to know that. Toyota is blunt about the order of the work. Its description of jidoka says that to build machines that stop on abnormality, you must first be able to do the work smoothly and correctly by hand, and that rather than starting with a machine, you must first try doing it thoroughly by hand.1 Only then do you know what an abnormality looks like well enough to build its detection into the machine.
That is Liker's Principle 8: "Use only reliable, thoroughly tested technology that serves your people and processes." In summary, its advice is that technology should support people rather than replace them, and that a process is often best worked out by hand before technology is added to it.3 It is also the principle Liker himself softened most. In the book's second edition, it became "Adopt and adapt technology that supports your people and processes," a change that reads as an admission that companies cannot always wait for a technology to be proven before learning to use it.3 What survived the rewrite is the order. You still learn the process before you hand it over.
Most agent projects skip this step, and the surveys suggest how often. In the Optro sample, 96 percent of leaders agreed that workflows should change to make use of AI, and 9 percent had actually redesigned their workflows to accommodate agents safely.5 An agent placed on an unexamined workflow cannot stop at an abnormality, because no one ever wrote down what normal is, so the organization falls back on the only detector it has: a person looking at the output. The watcher is not a choice. It is what you get when you skip the hand work.
This is narrower than the advice to fix a process before automating it, which "Redraw the floor before you pave the path." argued is right only when the tool leaves the work's purpose intact. Even when an agent changes the work completely, someone has to do the new work by hand long enough to learn its normal shape. A test follows. If the team cannot write down, in a sentence or two, what a wrong output from this agent would look like, it is not ready to give the agent a stop cord, and it will end up paying for a watcher instead.
The run log is the shop floor
The third principle is the easiest to agree with and the hardest to practice. Principle 12 reads: "Go and see for yourself to thoroughly understand the situation (genchi genbutsu)." In Liker's account, it means understanding a problem from the place where it happens, not from what other people or a computer screen report, and he applies it to senior managers as much as to engineers.3 At Toyota the place is the gemba, the floor where the work is done.
For an agent, the floor is the trace of a run: what it was asked, what it read, what it did, and what changed. Here the surveys are sobering. In the Nylas sample, 53 percent said they could fully audit what their agents did the previous week. Where agents had an identity of their own, 66 percent reported full audit capability. Where agents acted through an employee's account, the figure was 51 percent.4 In the Redwood sample, 42 percent had a central registry of the agents they had deployed, and 51 percent monitored agent activity in real time.6
A leader who cannot open the trace of last Tuesday's runs cannot go and see, and is approving autonomy on the strength of a report, the habit genchi genbutsu was meant to break. Every stop is an invitation to go and see, and it is useful only if there is something to look at on arrival: the inputs, the step that tripped the stop, the expectation it broke.
More people at the problem, fewer at the machine
Figure 2 puts the two defaults on one sheet. The numbers are made up; the shapes follow from the mechanisms. Under the watcher, review time grows in step with the workload, because every action costs the same few seconds, until it passes a working day and either the organization adds watchers or watching turns into clicking. Under the stop cord, human time depends on how often the agent stops, and each stop, handled properly, makes the next one less likely.
The flat line depends entirely on the countermeasures. An agent that stops and is merely restarted, again and again, for the same reason, has turned the stop cord into a nuisance alarm, and people learn to ignore those. Toyota is explicit about where improvement comes from. However well machines, robots or software perform, its description says, they cannot evolve on their own; only people can carry out kaizen, the steady improvement of the work.1 The stop gives people a reason and a place to improve the work; the improving is still theirs.
That division of labor is more demanding than either default. Letting the agent run asks nothing of people. Approving everything asks for attention but not judgment. The stop cord asks for judgment at the moment and place it is needed, and then for a better process. It needs fewer people watching and more who understand the work.
Objections, considered
Agents are not looms, and a wrong answer is not a broken thread. A broken thread is unambiguous. A plausible but wrong paragraph, or a correct-looking record with the wrong customer on it, is not, and no simple rule will catch it. That limits the argument. But most consequential agent actions are not paragraphs. They are counts, amounts, records touched, first-time actions, actions outside the usual hours or the usual systems, and those can be checked against expectations. Where only a person reading the output can check it, a person should read it, and the task is not yet ready for an agent alone.
Some decisions need a person's signature every time. Agreed. Jidoka does not abolish approval; it makes approval rare enough to mean something. The few actions that are irreversible, large or regulated should always wait for a person, and those approvals improve when they are not buried among thousands of routine ones.
The surveys are vendor marketing. They are, or they are self-selected readers, and I have tried to describe each one as what it is and not to add them together. None establishes an industry rate. Each, in its own sample, shows part of the pattern, and a reader who discounts every number still has the mechanism, which depends on none of them.
The Toyota Way is a book by an admirer. Liker has spent his career studying Toyota, and the book reads as a case for its methods rather than an audit of them. I lean on it for its organization of Toyota's ideas, and on Toyota's own account for what jidoka means. The idea at the center, a machine that stops at an abnormality and a person who fixes the cause, is older than the book and does not depend on Toyota's record.
A person who has to watch every action the machine takes has not been freed by it, but hired by it.
Before the next agent goes live
- Write down what wrong looks like. For each agent, list the conditions under which it should stop: counts, amounts, first-time actions, low confidence. If the list is empty, the agent is not ready.
- Name the person the stop calls. Every stop should reach one owner who knows the work, not a shared channel.
- Make every stop end in a countermeasure. Count the stops by cause each week. A cause that keeps recurring is a process that has not been fixed.
- Go and read a run. Before approving more autonomy for an agent, open the trace of one of its runs from last week and see whether you can tell what it did and why.
Toyoda's weft-breakage stop was not a clever way to do without the weaver. It was a way to stop wasting the weaver's attention on cloth that was fine, and to bring it to the one spot where the thread had broken. The agents arriving in offices now are far more capable than any loom. What they mostly lack is that small, unglamorous device: a written sense of what normal looks like, and the willingness to stop when the work stops looking like it.