The backup has to come back.

Two pieces of paper are clipped together on a desk. The first is a printed backup report for an order system, and every line on it ends in the same word: SUCCESS. The nightly database copy ran, the transaction logs were shipped, the file share and the configuration were saved, and the newest copy is stamped 08:00. The second sheet is a recovery exercise record for the same system. It has lines for the scenario, the recovery point, the start time and the problems found, and near the bottom it has a line labeled "first useful transaction." That line is empty. Both papers are invented, and nothing here suggests that backup reports lie. The first sheet may be entirely accurate. It simply answers a different question from the one the second sheet leaves blank.

A completed backup job is evidence that a copying process ran. It says that some data was read from one place and written to another, and that the software saw no error while doing it. That is a stored backup, and it is worth having. Whether the business could work again after losing the original is a separate claim. It is a claim about demonstrated service recovery: a useful service, restored within a time the business can tolerate, missing no more recent work than the business can tolerate.

My argument is that the second claim becomes credible only through recovery exercises that resemble the disruption you fear, and not through a larger count of stored copies. More copies can help, and some copies should sit where an attacker or an accident cannot reach them. But a copy that has never been restored into a working service is a possibility, not a result. This is not a claim that backups are unreliable, or that every organization must rehearse every disaster every quarter. It is a claim about what kind of evidence answers the question a board actually asks.

A copy is not a service

Start with what a successful backup job can and cannot know. It knows that the data it was told to copy was copied. It usually does not know whether it was told to copy everything the service needs. It cannot know whether the copy will still be readable when the original is gone, whether the people who need it will be able to reach it, or whether the machinery to turn it back into a running system will exist on the bad day. Those are the questions that decide recovery, and a green row in a console is silent on all of them.

Put them as a short checklist. Is the copy accessible, stored somewhere that the same failure, or the same attacker, cannot destroy? Is it usable, complete and consistent rather than a half-written file or a database caught mid-transaction? Is it recent enough, so that the work lost since it was taken is work the business can redo or live without? And is it restorable with what will actually be at hand during the disruption: the credentials to open it, the encryption keys to decrypt it, the servers or cloud capacity to load it onto, and people who know the steps? A backup can pass the first three and fail the fourth, and the fourth is where most of the time goes.

Notice that none of this is a criticism of backup software. The software is doing its job when it reports success. The trouble comes when the report is read as an answer to a larger question that nobody asked the software.

Two tolerances, not one

The United States government has a careful vocabulary for the larger question. NIST Special Publication 800-34, the National Institute of Standards and Technology's contingency planning guide for federal information systems, was written by Marianne Swanson and four colleagues and issued in revised form in 2010.¹ It lays out a seven-step process that runs from a policy, through a business impact analysis and a choice of recovery strategies, to a written plan, and then to two steps that are easy to skip: testing, training and exercises, and keeping the plan current.¹ The guide was prepared for federal agencies, which must follow it; anyone else may use it voluntarily. I lean on it here as a disciplined structure, not as a rule that binds every reader.

The business impact analysis asks two questions that sound alike and are not. The first is how long the service can be unavailable before the damage becomes unacceptable. NIST calls the outer limit the maximum tolerable downtime, and the target for restoring a particular system the recovery time objective, which it says should normally be shorter.¹ The second question is how much recent work the business can afford to lose. NIST frames this as the recovery point objective: the point in time, before the disruption, to which data can be recovered from the most recent backup copy.¹ The guide adds a detail that matters more than it looks. Unlike the recovery time, the recovery point is not counted as part of the tolerable downtime.¹ They are two different measures.

Figure 2 puts them on separate rulers, using invented times. The last recoverable copy was taken at 08:00. The disruption struck at 09:00. The service became usable again at 12:00. Under that scenario, the restored data is one hour old, and the service was down for three hours. Those numbers do not add up to a four-hour anything. One hour of lost orders is a loss of work that customers or staff must reconstruct; three hours of outage is a loss of time in which nobody could place an order at all. A business might tolerate the first and not the second, or the reverse. A single resilience score that blends them hides exactly the trade-off that a recovery plan has to make. Drag across the sheet, or tab to it and use the arrow keys, to move the recovery point and the moment service returns.

Four safeguards, one copy

The clearest public account of the gap between copies and recovery is a postmortem GitLab published on February 10, 2017, about an outage of its hosted service ten days earlier.² I use it narrowly. It is a documented, bounded case from years ago, and it says nothing about how reliable GitLab is today.

On the evening of January 31, replication between GitLab.com's primary database and its secondary had broken. While trying to rebuild the secondary, an engineer ran a command to clear its data directory, and it ran on the primary instead. The engineer stopped it within a second or two, by which point around 300 gigabytes had been removed.²

GitLab had four mechanisms that might have saved the data. A daily database dump was supposed to be uploaded to cloud storage, but it was running an older version of the dump tool than the database required, so it had been failing, and the storage was empty. The failure notices were sent by email and were rejected by the receiving mail system, so nobody saw them. Disk snapshots offered by the cloud provider were enabled for other servers but not for the database servers. Replication to the secondary was designed for failover, not for recovering from a deletion, and it was the thing that had broken. What remained was a disk snapshot taken about six hours earlier, made to refresh a staging environment rather than for disaster recovery.²

That copy saved the company, slowly. Restoring GitLab.com took more than 18 hours, and much of that went to copying the data across slow cloud disks.² Database changes made after the snapshot were lost: by GitLab's count, roughly 5,000 projects, 5,000 comments and 700 new user accounts, along with issues and snippets. The Git repositories and wikis were not removed, because they were stored separately.² That distinction matters. The loss was real and bounded, and it fell on one part of the service, not all of it.

The postmortem does not make this a story about one person's mistake, and neither should we. Its root-cause analysis turns to the system around the engineer. The backup procedure had no owner, and, in the postmortem's words, "nobody was responsible for testing this procedure."² The fixes it lists follow: an owner for data durability, automated testing of database backup recovery, more frequent snapshots, and monitoring of the backups themselves.² Notice the shape of the failure. GitLab did not lack backups. It had several kinds. What it lacked was a recent, routine demonstration that any one of them would come back.

An exercise that ends in a transaction

What does a demonstration look like? NIST's guide distinguishes a tabletop exercise, which is discussion only and deploys no equipment, from a functional exercise, in which people perform their duties in a simulated operational environment.¹ For systems whose availability is rated moderate, it recommends functional exercises that include an element of recovery from backup media; for high-impact systems, a full-scale exercise that fails over to the alternate location.¹ Testing, it says, should be done "as close to an operating environment as possible."¹

For a business service, I would add one requirement: the exercise should end with something the business recognizes as work. A server that boots is not the finish line. Neither is a database that accepts connections. The useful steps run roughly like this:

Recover into an isolated environment, so that the exercise cannot disturb production and cannot quietly borrow from it. Production is often the hidden dependency: the sign-in service, the license server or the shared secret store that the recovered system reaches for and finds, because the original is still running.

Complete a representative transaction, chosen in advance by the business owner. For the order service in Figure 1, that means placing an order, taking a payment and producing the confirmation a customer would see. The time at which that first works is the time the service was recovered.

Reconcile the data. Compare what was restored with what should be there: order counts, payment totals, the last records before the recovery point. A restore that completes but silently drops a table is worse than one that fails loudly.

Write down the dependencies and the timings, including the ones that were improvised. If someone had to phone a supplier for a key, that call is part of the recovery path and belongs in the plan.

Compare the result with the tolerance. Did the restored data fall within the recovery point the business agreed to? Did usable service return within the recovery time? The answer may be yes for one and no for the other, and both answers deserve to be reported as they are.

In Figure 1, the exercise failed at the second step. The copy was fine, which is why the gold stamp is honest. The service could not take a payment, because the key for the payment provider was held by the supplier, and the administrators could not sign in, because their accounts lived in the production identity service the exercise had assumed away. Both problems are cheap to fix once found. Found on the day of a real loss, they would have been the longest part of the outage.

Choose the disaster you rehearse

A recovery exercise tests one path through one scenario, so the choice of scenario decides what the result can prove. The classic scenario is a broken server or a corrupted database, and it is a fair place to start. It is not the only shape a bad day takes.

Consider a compromised administrator account. The attacker who holds it may be able to delete the backups along with the originals, which is why a copy that the same credentials cannot alter is worth more than a third copy that they can. Consider an unavailable identity provider: the recovered system may be intact and nobody may be able to log in to it. Consider a missing encryption key, which turns a perfect copy into unreadable bytes. Consider a supplier that is itself having the bad day, so that the managed service, the support line or the cloud region you planned to recover into is not there. Each scenario changes which copy is usable and which steps are possible.

No organization needs to exercise all of these on the same schedule, and pretending otherwise would bury people in drills. NIST itself scales its expectations, from a tabletop exercise for low-impact systems to full failover for high-impact ones, each at a frequency the organization defines.¹ The sensible approach is to rank the scenarios by likelihood and harm for each important service, rehearse the top ones in proportion, and say plainly which ones have not been tested.

Objections, considered

Full recovery tests are disruptive and expensive. They can be. Failing over a production system carries its own risk, and a complete rehearsal of a large estate can take weeks of effort. But the choice is not between a full production failover and nothing. Restoring one service into an isolated environment tests much of the claim at modest cost. A tabletop walk-through can find missing keys and phone numbers for the price of a meeting. The honest course is to run graduated exercises and keep their limits visible: this one proved the database restores; that one did not test the payment path.

Our platform provider handles recovery. Managed databases and cloud services often do take care of many recovery steps, and do them well. The customer still owns the parts the provider cannot see: which of its services depend on which others, which keys and identities the application needs, and what counts as a correct business result. A provider can restore a database to a point in time. It cannot tell you whether the restored orders reconcile with the payments.

A passed exercise is no guarantee. True. An exercise is bounded evidence about one scenario on one day. Systems change after it, and the next disaster may not resemble the rehearsal. But the alternative is not certainty; it is an untested assumption. A recent result with its gaps written down is the best evidence an organization can have, and it decays, which is an argument for repeating it rather than for skipping it.

Backups do other jobs. They do. Organizations keep copies for archives, for legal holds and for audits, and those purposes may call for long retention and slow retrieval. Nothing here argues against them. The point is narrower: a copy kept for a records request is not, by being kept, evidence that the order service can be running by noon.

A backup protects a possibility; a recovery exercise tests the promise.

Questions for the next report

• Show me the last recovery result for this service. Not the backup console: the date of the exercise, the scenario, and what it achieved.
• Which business transaction proved recovery? Name the order, payment or record that worked, and the time it first did.
• How old was the recovered data? Compare it with the recovery point the business agreed to, on its own ruler.
• How long until the service was usable? Compare that with the recovery time, separately.
• What failed in the last exercise? And who owns fixing it before the next one.
• Which scenarios have not been tested? A compromised administrator, a missing key, an unavailable supplier: say which have been rehearsed and which are still assumptions.

Back to the two clipped sheets. The backup report is still accurate, and every line on it still says SUCCESS. The exercise record now has a second page. The key went into escrow, the administrators got accounts that do not depend on production, and the order service was rebuilt again from the 08:00 copy. The line beside "first useful transaction" is no longer blank. In pencil, it reads: 12:00, order placed and paid, confirmation sent.