Simple enough to be obviously right.

Picture a printed page from a code review, the kind someone carries into a meeting with a pen. On it is a short block that sends an update to another service. The call sits inside a try, and under it is the handler for anything that goes wrong: catch (Exception e), an opening brace, a comment that says TODO, and a closing brace. Nothing else. The block is invented for this essay, but anyone who has read much code has met one like it. It compiles. It passes every check that runs when things go well. And it has already decided what the system will do on its worst day, without anyone deciding.

That block shows two different things a design can have. The first is what the system can do: here, keep running whatever happens to the update, because no failure can get past the handler. The second is what anyone can show it does: the behavior a reader who did not write the code could state, path by path. On the path where the update succeeds, the two agree. On the path where it fails, the block can do something, carry on, but nobody can say what that leaves behind: an update that never arrived, a record half written, a caller told that all is well.

My argument is that a system is only as reliable as the parts of it someone can see working, and that complexity grows the parts nobody can see. Every branch, option and handler is a path that has to be right, and the best evidence we have says that catastrophic failures gather in the paths nobody designed. This is not a claim that simple is always better, or that simplicity prevents failure. Some problems are complicated all the way down, and the simple design is usually the harder one to find. The claim is narrower: simplicity is what makes a failure visible before it happens, and visibility is the part of reliability anyone can check.

Two ways to build a design

In his 1980 Turing Award lecture, published in Communications of the ACM in 1981, C. A. R. Hoare told the story of a meeting.¹ In October 1965, at St. Pierre de Chartreuse in France, the international working group on the ALGOL programming language set aside the draft it had been refining and took up a different design, more ambitious and, to Hoare, far less attractive. He warned the group against its obscurity, its complexity and its overambition, and the warnings went unheeded.

From that experience he drew the sentence the lecture is remembered for. There are two ways of constructing a software design, he wrote. One is to make it so simple that its correctness is plain to anyone who looks. The other is to make it "so complicated that there are no obvious deficiencies." The first way, he added, is far more difficult.

The line is often repeated as a joke about complexity, but it is more exact than a joke. Both designs can arrive with no visible defects. The difference is where the absence comes from. In the simple design, you look and see that nothing is wrong. In the complicated one, you look and fail to see anything wrong, which is a statement about the looking, not about the design.

Hoare knew the second kind from the inside. A week before the Chartreuse meeting he had sat through an inquest on his own team's software for the Elliott 503 Mark II computer, a project of more than thirty person-years that had missed two delivery dates, produced a compiler too slow to use, and was abandoned. When he lists the causes, the first one he names is his own: overambition. The same lecture also records the opposite instinct. His team's earlier compiler checked every array subscript against its bounds while the program ran, and when the team later asked customers whether they wanted an option to switch the checks off, they unanimously said no. They preferred a program that would show them its mistakes to one that might run a little faster without them.

What the failures look like

Hoare's sentence is a claim about design. Three decades later, a team at the University of Toronto, led by Ding Yuan and Michael Stumm, went looking for where real failures come from.² They took 198 failures, sampled at random from user reports, in five distributed data systems that are widely used in production and were built to tolerate the failure of their own parts: Cassandra, HBase, the Hadoop Distributed File System, Hadoop MapReduce and Redis. Forty-eight of the failures were catastrophic, meaning that most or all users saw an outage or lost data.

The finding was lopsided. In 92 percent of the catastrophic failures, the software had already noticed something wrong and said so: an error that was not fatal in itself had been explicitly signaled, as an exception thrown or an error returned. The failure came from what the code did next. The machine knew. The part that was supposed to decide what to do about it did the wrong thing.

Many of those decisions were not subtle. In 35 percent of the catastrophic failures, the fault in the handler fell into one of three trivial patterns. The handler was empty, or did nothing but write a line to the log. The handler caught an overly general exception and shut down the whole cluster. Or the handler held a comment such as TODO or FIXME. In one MapReduce case the authors describe, the developers had left a TODO and a logging statement in a handler for an error that would rarely occur. When it did, it "took down a production cluster of 4,000 nodes."

Two things about these numbers matter for design. The first is that these were not neglected systems. They were mature, heavily used, and built specifically so that a failing disk or machine would not take them down. The capability was there, in the architecture. What failed was the small piece of code that was supposed to use it.

The second is that the trivial cases did not take clever tools to find. The authors wrote what they call a simple static checker, Aspirator, which reads the code and flags these patterns without running anything. They estimate that more than 30 percent of the catastrophic failures they studied would have been prevented had it been used and its findings fixed. Reading was enough. The gap was on the page.

Why complexity hides the gap

Why would mature systems carry handlers like these? Not because their authors were careless. Error handlers run rarely. The main path runs millions of times a day, and daily use watches it, exercises it and complains when it breaks. A handler for a timeout from a neighbor during a configuration change might run once a year. Its emptiness stays invisible precisely because it is rare: nobody sees it misbehave until the day it matters, and on that day it is usually running alongside other trouble.

Now add capability. Every option, mode and fallback a system gains adds paths, and paths multiply rather than add. A step with three ways to fail, inside a feature with two modes, behind a flag with two settings, has twelve combinations that each have to be right. The paths that run every day get shown to work by use. The rest are promises. Hoare's complicated design is not one where somebody hid the defects. It is one where the number of paths has outrun anyone's ability to look at each one.

Seen this way, the 92 percent is less surprising than it first looks. In a system built to tolerate failure, the error handling is where the tolerance lives. It is the most capable code in the system and the least observed, so it is where what the system can do and what anyone can show it does drift furthest apart.

Two versions of one step

Go back to the printout in Figure 1. Version A can do more. Whatever goes wrong inside the call, the program carries on. Version B can do less: if the update fails, the failure leaves the step and goes to whoever called it, and the caller has to decide. But Version B can be shown to do what it does. A reviewer can read it, follow the failure up one level, and see whether the caller handles it. If the caller does not, the gap is right there, in the place a reader would look. Version A has the same gap, covered by a pair of braces that make it look closed.

Neither version is right for every program, and sometimes the best place to handle a failure is exactly where Version A catches it. The point is that A's extra capability came with a path nobody designed, while B's smaller capability came with a path anyone can state. When the HBase developers fixed one of the data losses in the study, caused by a handler that did nothing but log, the fix was a few lines in the handler saying what to do instead.² That is all a designed path is: a decision, written down where a reviewer can read it.

A working test for simple

Simplicity is hard to define, and line counts are a poor stand-in for it. A five-line block can be complicated if its behavior on failure depends on things nobody wrote down; a long function can be simple if every branch says what it does. A better test is the one Hoare's sentence implies. Can someone who did not write the code say what it does on each path? If so, the design is simple in the sense that matters, whatever its size. If not, it is complicated, however short. The test turns simplicity from a matter of taste into a question anyone can ask in a review.

It also explains why Hoare called the simple way the harder one. Making each path statable means deciding what each path should do, and many of those decisions are unwelcome: drop the update, try again, stop the whole job, or wake someone up. An empty handler puts the decision off. A TODO records that it was put off. Neither one makes it, and so neither one can be shown to be right.

Objections, considered

Real systems are complex, and simplicity is a luxury. Some complexity is essential. A distributed database has to cope with machines that die halfway through a write, and no amount of taste removes that. The argument is about the complexity added on top: the option nobody questioned twice, the catch-all that avoids a crash today. Hoare does not say the simple way is always open. He says it is far more difficult, which is a reason to budget for it, not a reason to skip it.

This is just "test your error handling" again. The study's own title is about testing, and earlier essays here were about evidence after the fact: "The backup has to come back." on restoring a service, and "The patch is not the finish line." on proving a fix worked. This one is about what makes evidence possible in the first place. A design with fewer paths has less to prove, and the trivial patterns were found by a checker that read the code without running it. Whether anyone can see what a handler does is a property of the design before it is a property of any test.

The study is old, and narrow. It is from 2014, and it covers five open-source data systems, sampled from failures that users reported, which are not all failures. Its numbers describe those systems, not all software, and I use them as evidence of a pattern rather than as a rate. Hoare's claim is older and does not depend on them. What the study adds is that the pattern shows up in mature code, written by people who thought hard about failure for a living.

Swallowing an error keeps the system up, and a crash is worse. Sometimes carrying on is the right response. But the study counts the opposite reflex, catching too broad an exception and shutting down the whole cluster, among the same trivial patterns: 8 percent of the catastrophic failures. The answer is neither silence nor alarm. It is a handled path that someone chose and wrote down, which is simpler than either, because its behavior can be stated.

Simplicity is in the eye of the beholder. It is, if simplicity means elegance. The working test is meant to take the beholder out of it: a second reader either can or cannot say what each path does, and that is a fact about the code, not about anyone's taste.

Every capability a system gains is another path someone has to be able to explain, and the paths nobody can explain are where it fails.

Before the next design review

• Search for empty handlers and handlers that only log. Each one is a decision. Write down what it should be, then make the code say it.
• Search the error handling for TODO and FIXME. Date each one and give it an owner, or fix it.
• Find every catch-all handler. Ask whether it should exist, and what it does to the rest of the system when it fires.
• Ask how many paths a change adds before asking what it can do. For every flag, mode and fallback, name someone who can say what it does.

Back to the printout. The TODO in Version A is not a missing line of code. It is a sentence someone meant to write: what the system should do when the update fails. Until it is written, the system can survive the failure, and nobody can say what it survived into. Write the sentence, and the block becomes a little less capable and a great deal more obviously right.