The margin was the time.

Two temperatures from one room in Frankfurt, less than half an hour apart: 21.8°C and 51.9°C. Just after 11 p.m. on Wednesday, August 26, the cooling failed in the main room of Proton's datacenter in Frankfurt. At about 11:15 the air began to warm from its normal 21.8°C, and in under thirty minutes it reached 51.9°C, with some probes reading 60°C. Servers and network gear started to die one at a time. Proton's chief technology officer, Bart Butler, published the incident report last Friday.¹

The cause was mundane. The datacenter's operator replaced the air filters on both of the redundant compressors that power the cooling, in the middle of the night, without telling Proton first, and then did not tell Proton when the cooling failed. So the room had a second compressor, and it did not help. What a second compressor is really for is time: the stretch between a fault and the harm it causes, in which someone notices, decides and acts. Proton's report says the thing that made this night unusual is how little of that stretch there was.¹

My argument is that redundancy should be judged by the time it buys, not by the count of spare parts, and that the time is shrinking. As racks draw more power, a room that loses its cooling runs out of margin in minutes, and the deliberate human steps in a recovery, being told, deciding, failing over by hand, do not get faster to match. This is not a claim that Proton or its operator was careless beyond what the report says, that redundancy is useless, or that every data center is one filter change from disaster. Most nights the second compressor does its job. The claim is about what is left when it does not.

Two of everything, failing together

Proton's report makes the point itself: cooling systems are usually redundant, losing all of it is rare, and so there is normally plenty of time before temperatures become critical.¹ That night, the two compressors were serviced in the same operation, so the redundancy and the thing it protected against arrived together. About an hour later, at around midnight, the heat took out both the primary and the backup network switch on one critical rack. That rack, the report says, held several primary database copies, and that is when users began to notice.¹

Two pairs of redundant parts, and each pair failed as one, because each pair shared something: a maintenance schedule in one case, a room in the other. Richard Cook, the physician and safety researcher, described the pattern in his short 1998 treatise on how complex systems fail. Such systems are heavily defended, he wrote, so a catastrophe needs several failures at once, usually small ones that look harmless on their own.⁶ A filter change is routine. Working at night is routine. A missed phone call is ordinary. Together they removed the cooling and the warning at the same time. No. 25, Two suppliers can share one failure, made the case about shared failure between vendors, and the same logic holds at the scale of a compressor. What is new here is what the room did with the time it had.

The clock got shorter

Here is Proton's sentence on why the night went so fast: "what used to take 3-4 hours to go critical went critical in 20 minutes."¹ The report blames a large increase in server power density in recent years, from higher-power processors and the GPUs used for AI. That is the company's estimate for its own hardware, not a measurement, but the mechanism behind it is simple. With the cooling off, nearly every watt a server draws becomes heat that stays in the room. Double the power in the same space and, roughly, you halve the time it takes the air to reach any given temperature. The thermal mass that used to give an operator hours now gives minutes.

The industry has said so for a decade. A 2014 Schneider Electric white paper on temperature rise during cooling outages warned that higher rack density shortens the time operators have before the air entering the equipment reaches critical levels.⁵ ASHRAE's data center committee, in a 2016 white paper, said that after a chiller or air-handling failure a rise of 30°C in the cold aisle in as little as five minutes is not uncommon, and gave the interval its name: ride-through time, the time before equipment starts shutting itself down to protect itself.⁴

And density is still rising. Uptime Institute's annual survey of data center operators, released on July 28, found power demands per rack climbing, driven by AI and by other enterprise work, and a growing number of operators with racks at 30 kilowatts or more.² The average of the most typical rack density reported passed 11 kilowatts for the first time, up from 9 the year before, though without the newest high-density sites it was 7.8, up from 7.5.³ The day after the survey came out, Uptime's research director, Daniel Bizo, put the consequence plainly: higher density means "less margin for error as ride-through times shorten." In rooms under 10 kilowatts a rack, operators had several, sometimes tens of, minutes before the equipment's intake air got too hot; racks cooled by liquid through cold plates may not survive more than a few seconds without circulation.⁷

These numbers measure different things. Proton's are room air and its own threshold for "critical"; ASHRAE's are the cold aisle; Bizo's are the air entering a server against its tolerance. They should not be stacked on one axis, and I have not done so. But they all point the same way, and Proton's room, warming by about 30°C in under half an hour, was slower than ASHRAE's fast case. It still ran out the clock.

The steps that need minutes

What is the time for? In Proton's case, for a handful of hard decisions made by on-call engineers under pressure. Should they bring service back first, or save the hardware? Should they fail over to the copies in the same Frankfurt building, which is faster, or to Zurich, which is safer if the heat cannot be stopped? Should they move everything, or only what had already failed?¹

Each of those choices runs through a step that is slow on purpose. Almost all of Proton's systems fail over automatically, but the primary databases do not move without a person watching. The reason is a failure called split brain: if a copy takes over while the original is only briefly unreachable, the two can each accept changes the other never sees, and the records disagree in ways that are hard to reconcile. The standard procedure is to fail over to a copy in the same building, for speed, but that night the copy in the same building might be the next thing to die. Proton's automation handles a clean loss of a whole site well. It does not handle, in the report's words, random servers dying one by one. And because Proton restricts access to the out-of-band controllers that let engineers manage machines when the network is down, it had to wake more staff to help.¹

Every one of these is a sensible design. Supervised failover protects the data; same-building copies protect performance; locked-down management access protects security. Each one also spends minutes from the same budget, and the operator's silence spent the first of them before anyone at Proton knew the clock had started. The report says that silence dramatically cut the time Proton had to respond.¹

Cook has a line for what the engineers did next: "All practitioner actions are gambles."⁶ Proton chose to save the hardware, accepting a longer outage, because, it says, a server shortage tied to the AI boom meant lost equipment could not be replaced quickly.¹ That is Proton's account of the market, and I have not found independent figures on it. The choice was not about money already spent; the hardware was not yet lost. It was a bet on what would be hardest to get back. Cooling returned at 00:45. Most services were back for most users by 01:30, and the less critical ones, such as push notifications and payment processing, by about 02:00.¹

Figure 2 puts the night on one clock, and one thing stands out. On Proton's own estimate, an older room would have given it 3 to 4 hours before anything went critical. Cooling came back an hour and three quarters after it failed, and most services half an hour before the shortest old budget would have run out. In that room, the same lapse would most likely have been a scare. In this one, it was an outage.

The heat stayed in the hardware

The night had a second surprise. Once the cooling was back, Proton failed its primary databases over to a surviving copy in Frankfurt where there was one, and to Zurich where there was not, on the assumption that Frankfurt would now be easy to bring back. It was not. Many network cards had reached 105°C, against a normal 45°C, and gone into a protection mode that keeps them disabled until the machine is fully powered off and restarted.¹ The heat kept drawing on the time budget after the cooling problem itself was fixed.

No email was lost, though delivery was delayed in both directions. Some servers did not survive, and Proton does not yet know whether the heat has shortened the life of the rest. The engineers worked through the night and into the next day to restore full redundancy, with some primary databases left in Zurich and others in Frankfurt.¹

Proton also names its own part. A failure like this one, it says, was a known limitation of its database setup, liable to make recovery slower than normal; the work to fix it was already under way and planned for the end of the year, and new datacenter space due within weeks will reduce its dependence on a single site.¹ That is Cook's fifth point in miniature: complex systems run as broken systems, with known flaws and scheduled fixes, kept working by people.⁶ The incident simply arrived before the fix did.

Objections, considered

This was a maintenance failure by the operator, not a density story. Partly true, and Proton puts the operator's unannounced work and missing alarm first. But the same lapse in a room that took 3 to 4 hours to go critical would have left time to notice, call and fix it before any equipment was at risk. Both are true. This dispatch is about the second, because the operator's mistake can be prevented by contract, and the density will not go back down.

The bottleneck is manual failover, so automate it. Proton keeps it manual on purpose, to avoid split brain, which is a real risk to the integrity of the data. Automating it trades one failure for another. The better questions are whether the manual step fits inside the ride-through the room now has, and whether the copies live somewhere the same fault cannot reach.

One company's estimate is not data. Fair. The 3 to 4 hours and the 20 minutes are Proton's testimony about its own equipment, and I have treated them that way. The trend does not rest on them: the Schneider and ASHRAE papers, Uptime's density figures and Bizo's ride-through ranges are independent, and they agree on the direction.

A spare part is worth the time it buys, and the time is what the people need.

Before you rely on someone else's room

Most health and dental organizations never see a compressor. They rent space, buy hosting or run on software someone else operates. The questions still apply, one contract away.

• Ask for the time, not the count. At today's density, how long does the room or the service have after a total loss of cooling before equipment starts to shut down?
• Ask who tells whom. Does the contract require notice before maintenance on redundant systems, and immediate forwarding of alarms to the tenant?
• Time the human steps. Which recovery steps are manual by design, how long do they take at two in the morning, and do they fit inside that ride-through?
• Ask where the copy lives. If the standard failover target is in the same building, what happens when the building is the problem?

Back to the slip from Frankfurt. The gold lines say the room had two compressors, two switches, redundancy two of two. The blue lines under them tell the rest: 23:00, no alarm, 21.8°C, 51.9°C, twenty minutes where there used to be hours. The margin was never the second compressor. It was the time between the fault and the harm, and how much of it was left when someone found out.