Gary Bordey / POV

· Systems and data · For CIOs, CFOs and architects

When everything has to change at once.

A system can keep serving the business while its routes to the next version disappear. Preserving those routes is part of the work.

Picture two sheets of paper on the same desk. The first is the weekly service report for an order platform: every service up, availability at 99.95 percent, no failed batch jobs, no open incidents. The second is a compatibility worksheet for the same platform. It lists the application, the database underneath it, and the steps that would carry both to the versions the business will need. Two years ago the worksheet showed a bridge, an intermediate version of the application that could run against the old database and the new one, with the vendor's support behind it. This year the same row carries a pencil note: support ended. Nothing on the service report changed. The software has not rotted. The terms around it have.

The two sheets answer different questions. The service report measures continuity of service: is the business being served today? The worksheet measures freedom to change: can the system still be moved, one rehearsed step at a time, to where it has to go? Both are real obligations, and a good week on the first sheet says nothing about the second. My argument is that a working system can lose its affordable, supportable ways to change long before it stops delivering its service. When that happens, waiting does not merely postpone the job. It can change the job's shape, so that work which once could be done in separate, testable stages needs a new bridge, a period of running two systems side by side, or one larger transition in which many parts move together. Keeping credible intermediate states available is part of keeping the business running.

The title names a risk, not a law. Some upgrades can safely skip versions. Some systems deserve to be contained or retired rather than modernized. A large replacement program is not, by itself, evidence that anyone neglected anything, and an old system is not automatically a bad one. The end of a support agreement does not make software stop working on that date. It changes what an organization has agreed it is willing to run.

Service report

Order platform · week 32

Services up12 of 12

Availability99.95%

Failed batch jobs0

Open incidents0

Question answered: is it running?

All services up

This sheet has been green for years.

Compatibility worksheet

Order platform · route A0 + D0 to A2 + D2

Each step of the route, checked two years ago and now
StepMoveTwo years agoNow
1A0+D0 → B+D0tested, supportedsupported support ended
2B+D0 → B+D2tested, supportedsupported support ended
3B+D2 → A2+D2testedtested

Question answered: can it move, one step at a time?

No supported bridge

The software still runs. What lapsed is the agreement to run it.

Figure 1. Two invented documents for a hypothetical platform; no real product, vendor or company is shown. A0 and A2 are application versions, D0 and D2 database versions, and B an intermediate application that works with both databases. Gold marks continuity of service, blue marks a verified step toward change, graphite marks a step no longer acceptable on the original terms.

Where the first part left off

Last week's essay, The savings that became a project, was about organizations: how repeated decisions to postpone preventive work can wear away the capacity to do it at all. This one turns to the technical side. Both test a suspicion from the first essay on this site: that many disastrous IT overruns are the deferred lifecycle arriving all at once.

A larger study sharpens the picture of the disasters without proving the suspicion. In 2022, Bent Flyvbjerg and five colleagues analyzed 5,392 IT projects and found that cost overruns follow a power law: many projects overrun modestly, and a fat tail of projects overrun by extreme amounts. As a possible cause, they proposed interdependence among the technical components of a system, and showed in a computer simulation that a problem in one component can set off chain reactions through others.7 That demonstrates a plausible mechanism, not why any real project overran, and it says nothing about deferral in particular. But it points at coupled work, and the question here is how coupling builds up while a system works fine.

A debt that depends on the next change

The most useful engineering account of that build-up is Managing Technical Debt, by Philippe Kruchten, Robert Nord and Ipek Ozkaya, published by Addison-Wesley in 2019.1 In an excerpt from its first chapter, Ozkaya and Nord define technical debt as design or implementation choices that are expedient in the short term but set up a technical context that can make a future change more costly or impossible. They stress that the most expensive debt often has less to do with the quality of the code than with design decisions made over time at the level of the whole system, and that some of it comes from events outside the designers' control. How much of it matters, they write, "depends on how you want your system to evolve."2

That last point is the one an executive can use. Debt in this sense is not a property of old code. It is a relation between the system and a change someone will need to make. A tangled module nobody needs to touch costs little; a clean system whose database cannot move without everything else moving too can cost a great deal. So ask of any constraint: which future change does it make harder?

The metaphor has limits, and the authors acknowledge some of them: this debt is not quite like a mortgage or a car loan.2 There is no stated rate, and a system that is retired never has to be repaid. The gold and blue distinction in this essay is my own framing, not the book's. What the book supplies is the insistence that the cost of a technical decision lives in the changes it affects.

What an audit can show

Britain's tax authority offers the best documented public example of a large estate trying to regain its freedom to change. HM Revenue and Customs runs one of the largest and most complex IT estates in the country, including legacy tax systems more than 30 years old. In its February 2025 report on the cost of the tax system, the National Audit Office recalled that HMRC had recognized the state of its IT as a significant risk, in part because operational maintenance and systems upgrades had been postponed to secure cost savings.3

HMRC's main response was a program called Securing our Technical Future, set up in 2018 to stabilize an estate, some of it aged and out of support, by fixing high-priority vulnerabilities, leaving three ageing data centers, and preparing services to move to cloud and government hosting.4 The auditors reported that the program, now closed, cost more than initially expected and underdelivered, because of unforeseen technical complexity and because some of its 2023-24 funding was moved to higher priorities. By March 2024, 372 of 545 services had been migrated and 49 remediated.3 Those are different achievements. Moving a service to a new platform is not the same as fixing what made it fragile.

The report also found that three of HMRC's eight most costly digital programs in 2023-24 involved remediating legacy systems, and that by March 2023 their whole-life costs had grown by between 60 and 390 percent, with their lives extended by 21 to 36 months as their scope changed.3 That range covers the three programs together, not this one alone. Running HMRC's digital systems cost £785 million in 2023-24, 18 percent more in real terms than in 2019-20, and among HMRC's reasons was the rising cost of supporting old systems as the people who understand them become scarce.3

HMRC's own account deserves equal weight. In an assessment approved on June 3, 2024, its chief executive recorded that a 2023 review had refocused the program on migrating the critical national infrastructure services behind HMRC's most important operations, that a government review in February 2024 had given it a green rating for delivery confidence, and that it remained value for money and deliverable.4 The auditors, for their part, said the scope was reduced because of wider financial constraints, which also reduced its benefits.3 These accounts do not contradict each other. A program can succeed against a revised scope and fall short of its original ambition.

HMRC is not unusual. The audit office's 2022 review of the environment department found that years of low investment had created a serious risk of critical service failure or cyber-attack, and that money for some of that work, and staff at the Environment Agency, had been diverted to preparations for leaving the European Union.6 Those were legitimate priorities. What the audits cannot show is the dependency chain inside any of these estates, or what an earlier migration would have cost. For that mechanism we need a model, and it has to be labeled as one.

A route with a bridge in it

Figure 2 is a deliberately constructed example, not a reconstruction of HMRC or anyone else. A business must move from application A0 on database D0 to application A2 on database D2. A2 cannot run on D0, and A0 cannot run on D2. An intermediate application, B, can run on both. With B available, the job splits into three moves, each of which can be rehearsed and, if necessary, reversed on its own: install B against the old database, move the database, then replace B with A2. Each move still needs its own testing. Compatibility on paper proves neither performance nor a safe way back.

Now add one rule that many organizations really do have: nothing runs in production without a current support arrangement. In the earlier scenario, B is supported and accepted for the workload, and the staged route is open. In the later scenario, B's support has expired and equivalent support cannot be bought on the original terms. B will still execute. It is simply no longer an acceptable step under the organization's own policy, so the two middle states drop out of the route. What is left is the starting state, the destination, and a gap between them.

Support assumption for the bridge, B

Held fixed: the policy that every production intermediate must be supported. Every configuration is invented.

A four-state migration route with and without a supported bridge: an illustrative model Four states on graph paper: existing service A0 on D0, bridge installed B on D0, database moved B on D2, and target service A2 on D2. When B is supported, three inked arrows join the states into a staged route. When B's support has expired, the two middle states and their arrows turn to dashed pencil, and only a coordinated change or another remedy connects the start to the target.

With B supported, the route has three separate moves, each rehearsed and reversible on its own.

The same model without interaction: each state under both support assumptions
StateAppDatabaseEarlier scenarioLater scenario
Existing serviceA0D0Starting pointStill running; its own risks need review
Bridge installedBD0Supported stepSupport condition not met
Database movedBD2Supported stepSupport condition not met
Target serviceA2D2DestinationDestination
Figure 2. Illustrative compatibility and support model. These are invented configurations, not measured company results. Loss of an approved intermediate state narrows the staged path; alternative remediation remains possible. Gold marks the states that serve the business, blue marks supported moves, graphite marks states and moves that fail the support condition.

Notice what changed. No code was edited, and A0 runs exactly as well as before, which is why the service report stayed green. Only an assumption about support moved, and the figure lets you restore it: renew B's agreement and the staged route comes back. Time does not make software incompatible. It changes the terms on which an organization will accept each step, and the steps are what make a large change manageable.

The gap is not a dead end. The business can pay for a replacement bridge, negotiate a bounded exception to its own policy, contain the old system and stop changing it, retire the service, or move application and database together in one coordinated transition. Sometimes the coordinated move is the best one. But each is different work from the original three steps. A replacement bridge has to be found and tested; an exception has to be justified and watched; a coordinated transition has fewer points at which to stop and go back, and more parts that must work the first time.

So the economic question about the delay is not whether waiting was free. It is whether the benefit of waiting exceeded the extra preparation, support, coordination and exposure that waiting introduced. Those are different kinds of cost: planned work, the temporary cost of running two systems at once, the risk of interrupting the business, and plain uncertainty. They should not be summed into one confident number without evidence, but they should be named before the decision to wait, not discovered after.

Paying for the next upgrade now

The constructive version of this story comes from Shopify. In December 2021, Yi Qing Sim, an engineer on its database team, described how the company moved its MySQL fleet from Percona Server 5.7.21 to 5.7.32, retiring its last database machine on the old version in early September 2021.5 That is a minor upgrade, not the larger move to MySQL 8.0, and the team chose it partly for that reason. Among its motives were replication improvements, the practice of staying current to receive security and stability fixes, and the expectation of moving to 8.0 eventually. Building the tooling for the small upgrade, she wrote, would help prepare for the large one.

The tooling is the point. The team extended Mason, an internal service that replaces failed database machines automatically, so that it could also build machines at a new version and roll them out shard by shard. The new version ran for a few months on canary shards, some of them load tested regularly, before the main rollout, which moved in staggered batches. And they tested the way back before they needed it. Downgrading from 5.7.32 to 5.7.21 turned out to fail, because the server would not start properly; the team patched its own build of the database and then passed its rollback tests. The stated aim was to make upgrades "a checklist of tasks going forward, as opposed to a full-fledged project."5

This is a company describing its own work, and the article reports no independently measured return. The lesson that travels is not its architecture. It is the decision to spend effort on the capacity to perform the next change, and to prove the way back while the stakes are small.

A review for the route, not only the service

Most organizations already review the service. What I propose alongside it is a short review of the route, a one-page record of the current state, the target state and every acceptable intermediate state between them. It is my suggestion, not an industry standard, and it should be scaled to the system: a page for a critical platform, a few lines for a minor one. For each move between states it asks for five things.

  1. Evidence. A compatibility result, a test under a representative workload, or a documented support commitment, matched to the risk of the move.
  2. Business meaning. What a migrated record means, how it is reconciled, and which system may authorize action while both are running. A matching row count is not enough.
  3. Recovery. Which changes can be reversed, which need data reconciliation, and the last point at which the old service can safely resume. Restoring a backup does not undo new writes or actions already taken outside the system.
  4. Expiry and owner. Which assumption can lapse, such as a support agreement or a person's expertise, who watches it, and what decision its lapse triggers.
  5. Retirement. Evidence that the old component and the jobs that depend on it can actually be switched off.

The review measures demonstrated ability to move, not a score. A "changeability index" with two decimal places would be false precision. The honest unit is the transition that has been rehearsed, with a date on the assumptions that make it acceptable.

Objections, considered

Upgrades cause outages too. They do. Preserving the freedom to change means preparation and a tested way back, not installing every release the week it ships. Shopify's months on canary shards are the opposite of haste.

A single replacement may be cheaper than years of small upgrades. It may. The argument is for comparing the full alternatives, including the cost of running old and new together, while there is still a choice. It protects the choice; it does not prescribe small steps.

Expired support does not make software stop working. Correct, and the model says so. Running, tested compatibility, vendor support and the risk an organization has agreed to accept are four different things. The model changes only the third, and shows how the fourth turns it into a constraint.

A national tax authority is nothing like my business. The scale and constraints differ, and none of HMRC's costs should be imported into anyone else's plan. What transfers is the question: which intermediate states does our next change depend on, and when do they lapse?

Shopify has resources we do not. True. Borrow the principle of making the next change cheaper, not the tooling or the staffing. For a small organization that may mean one rehearsed upgrade a year and a rollback someone has actually performed.

The audits do not prove the mechanism. They do not. The audits document deferral, exposure and a hard remediation; the research proposes interdependence through simulation; the bridge model is constructed. Together they establish a credible risk worth managing, not an estimate of how often deferral ends this way.

Keeping a system running includes keeping a credible way to change it.

Before the next uptime report

  • Which intermediate state makes our next migration possible? Name it, with its versions, and say what route remains without it.
  • When does its support or acceptance expire? A date and an owner, not a line in a vendor's lifecycle page that nobody reads.
  • What evidence shows we can recover? A way back that has been performed, with the last safe point to resume the old service written down.
  • What must be retired to finish the job? The old component and every job that still depends on it, so the migration ends rather than becoming permanent coexistence.

Back to the desk. The service report is still green, and it should be; continuity is the business's first claim on its technology. It just should not stand alone. Beside it belongs the worksheet, dated, with each step of the route marked tested or not, supported until a stated date, and owned by someone who will notice when that date approaches. A board that asks for both sheets every quarter will rarely be told that everything has to change at once.

Sources

  1. Philippe Kruchten, Robert Nord and Ipek Ozkaya, Managing Technical Debt: Reducing Friction in Software Development, Addison-Wesley Professional, SEI Series in Software Engineering, 2019. Book record; the essay draws on the authors' published Chapter 1 excerpt (source 2), not on the full book.
  2. Ipek Ozkaya and Robert Nord, Managing the Consequences of Technical Debt: 5 Stories from the Field, Software Engineering Institute, Carnegie Mellon University, May 13, 2019. Excerpt from Chapter 1: definition, design strategy versus code quality, external events, and the limits of the loan analogy.
  3. National Audit Office, The administrative cost of the tax system, HC 675, Session 2024-25, February 10, 2025. HMRC's IT estate and postponed upgrades at paragraphs 2.11 to 2.12 (pp. 29-30); Securing our Technical Future, the 372 and 49 of 545 services, and the 60 to 390 percent range at paragraph 2.12 (p. 30); running costs at paragraph 2.13 (p. 31); reduced scope at paragraph 2.18 (p. 33).
  4. HM Revenue and Customs, Securing our Technical Future Programme: Accounting Officer Assessment, approved June 3, 2024, published July 16, 2024. Background, objectives, 2023 reprioritization, the February 2024 Gate 5 review and the conclusion.
  5. Yi Qing Sim, Upgrading MySQL at Shopify, Shopify Engineering, December 2, 2021. Versions, motivations, Mason tooling, canary shards, rollback testing and the aim of repeatable upgrades.
  6. National Audit Office, Department for Environment, Food & Rural Affairs: Modernising ageing digital services, HC 948, Session 2022-23, December 6, 2022. Summary paragraphs 11 and 14.
  7. Bent Flyvbjerg, Alexander Budzier, Jong Seok Lee, Mark Keil, Daniel Lunn and Dirk W. Bester, The Empirical Reality of IT Project Cost Overruns: Discovering A Power-Law Distribution, Journal of Management Information Systems 39, no. 3 (2022); author manuscript deposited at arXiv, September 30, 2022. Sample, distribution finding and simulated interdependence mechanism as stated in the abstract.

Read next

Every essay, by date