“In this world, nothing can be said to be certain – except debt and taxes.”
That’s how that quote goes, right?1
If you’ve spent any time in the tech industry, it might as well be. The phrase “technical debt” tends to get thrown around a lot, and while it is a very real, important concept, I’ve often observed its incorrect usage in the wild. As such, I thought it’d be a worthy topic to cover this week.
I like to define technical debt very simply, as the difference between an existing solution and an ideal solution.2
Since it’s wintertime, let’s use the example case of clearing a driveway of freshly fallen snow. I have two possible solutions – using a snowblower, or using a shovel.
Let’s say that I pick the shovel.
There are two primary ways we might calculate technical debt:
- The difference in effectiveness between the existing and ideal solutions.
- Using my shovel, I’ll complete the job much more slowly than if I’d chosen the snowblower. Each time I clean my driveway, the aggregate difference will continue to grow larger.
- The cost to switch from the existing solution to the ideal solution.
- If I wanted to switch to the snowblower, I’ll need to consider the cost of driving out to the snowblower store3 and the amount of money I’d spend on buying it.
This concept is called “debt” because you’re “paying” for your past choices – either in pieces by enduring a less effective solution, or in one lump sum by switching to a better solution.
It’s also important to note that technical debt is contextual.4 You can’t just say “there’s technical debt” without referencing a specific example of a way in which the current solution is sub-optimal, because the entire concept of technical debt requires comparison between an existing and an ideal solution.
If the context changes, it’s likely the measure of technical debt will change, too. Perhaps I chose the shovel because I didn’t have any storage space for a snowblower. If I later move into a larger house with garage space, then I’ll no longer need to worry about storage, so the switch cost of buying a snowblower goes down.
You’ve (Probably) Experienced Tech Debt
Snowblowers and shovels are a relatively trivial example of technical debt, not to mention completely unnecessary out here on the west coast. How about another, more accessible example? There’s one particularly common experience which nicely illustrates technical debt.
Have you ever moved (or helped a friend move) to a new place? During the three years I lived in Boston, I moved six times – three different apartments and three different offices. Each time, I followed a simple series of steps to achieve success:
- Pack my shit into a car.
- Drive the car to my new place.
- Unload my shit.
- Repeat as necessary.
While technically correct, the steps outlined above omit a surprising amount of context. Let’s dig into the variables a bit. Presuming that you already know where you’ve decided to move, you still have to consider the following:
- WHAT are you moving? what are you keeping? what are you selling and how?5 what are you throwing away? how do you deal with all the trash? how do you keep fragile belongings safe? how will you disassemble your large furniture?
- HOW will you move? do you hire movers? do you do it yourself? do you need help? do you get a U-Haul, rent a car, or borrow a friend’s ride? oh, and what about boxes? how much stuff do you really have? how many trips will you need to make?
- WHEN will you move? when will you start? how long will it take? what’s the weather like? how will traffic conditions be? how long do you have the moving van for? when can your friends help? how do you coordinate with your work schedule?
Just reading that long list of questions is anxiety-inducing enough, isn’t it?
They’re important considerations, though. After all, every decision you make may have far-reaching effects on your moving experience. A couple poor choices and before you know it, you’re getting ticketed for double parking your unwieldy U-Haul on a one-way street during rush hour, and you’re inconceivably sweaty from single-handedly lugging just a fraction of your over-stuffed, structure-less boxes up four flights of narrow stairs on the hottest day of the year.
I’m convinced this is why most people hate moving. The general sequence of events remains the same, but the little, unique factors in each move make all the difference between easy-peezy and life on hard mode. You really have to design a custom moving solution suited to the task at hand – or risk a quickly growing pile of technical debt.
When tech debt starts building, it’s usually for one of four reasons:
- Inadequate Planning – You don’t properly consider the context for your solution.
- e.g., you thought you had a lot less shit than you actually do.
- Limited Resources – You don’t have the time, money, or energy to solve for everything.
- e.g., you can’t afford a U-Haul, so a friend’s sedan will have to do.
- Competing Goals – Solving for one of your goals means underachieving on another.
- e.g., you can go fast and break some shit, or slowly and carefully.
- Evolving Context – Things change unexpectedly, midstream.
- e.g., a bad traffic accident backs up the highway for an hour.
In each of the above scenarios, it leaves open the possibility that one aspect of the move isn’t optimized, which leaves room for technical debt to appear and grow.
You probably recalculated your technical debt multiple times during your last move. It happens intuitively whenever you’re contemplating a change in plans. Should I call a friend for help? Do I need a bigger car? Should I just hire someone to do this for me?
A rational decision-maker chooses to “pay off” technical debt when the cost to switch to a better solution is cheaper than continuing to experience the present inefficiency.
When Technical Debt Costs $15 BILLION
Often times, technical debt builds up over the course of years and even decades, rather than a single weekend.
One of my favorite examples of a massive, real-life technical debt scenario is the “Big Dig“. If you haven’t heard of this before, then I’ll bet you $100 that you didn’t live in Boston in the 90s.
The “Big Dig” – a.k.a., the Central Artery/Tunnel Project (CA/T) – was a massive, 25-year city infrastructure project (1982-2007), undertaken to relieve the traffic congestion that had been plaguing Boston since the Great Depression.6 Despite the city’s best efforts throughout the 20th century, population growth and the mass consumer adoption of automobiles continued to push the city’s thoroughfares to their limits. By one estimate in the 80s, traffic jams would last 16 hours by 2010.7 Boston simply could not handle the traffic.
So how did this happen, and why wasn’t every other major American city experiencing the same issues? After all, Boston hadn’t even been in the top ten for largest American cities since the 1950s.
You see, Boston was first settled way back in 1630, a time in which one iconic American invention was conspicuously absent – the automobile.8 As such, the streets just weren’t designed for vehicle traffic. The Massachusetts Bay Transportation Authority (MBTA) notes, “People traveled within the city on foot, and rarely went beyond its borders; for most could not afford horses and wagons.”
Over the next couple hundred years, pedestrians and the rise of public transportation9 continued to be the primary drivers of traffic, using streets that were already proving to be problematic. Historian Annie Haven Thwing10 once observed, “Many of the old streets were so narrow that it was difficult for two vehicles to pass each other, and so crooked that after a fire the town invariably ordered them straightened.”11
By the 1900s, Boston’s population had grown to around 500,000. Just eight years later, Henry Ford released his Model T – the “Thin Lizzy” – which for the first time made the automobile accessible to the general consumer.
Things went downhill from there.
Flash forward to the 1950s. To combat the growing congestion on its centuries-old roads, Boston took on an unprecedented project to build a 6-lane, elevated highway through the heart of the city. Formally named the Central Artery,12 it was constructed over the course of eight years and was designed to carry about 75,000 vehicles a day.
Almost as soon as it opened, the Central Artery was hated for a number of reasons:
- Its construction displaced about 20,000 Boston residents.
- It split the city in two, separating the historic North End and Waterfront neighborhoods from downtown Boston.
- Commuters had expected to take the highway into the city, park their cars underneath the structure, and walk to work. Unfortunately, the areas beneath the highway weren’t developed and attracted all sorts of unsavory individuals and activities.
The worst part of it all, was that the Central Artery actually failed to achieve its primary goal of reducing traffic. By 1990, it was struggling to support nearly triple its designed capacity (200k vehicles / day), and traffic jams lasting up to ten hours were not inconceivable.
At this point, the technical debt of the city infrastructure was absolutely massive. If Boston hoped to maintain its status as one of America’s most thriving cities, it would have to pay its debt.
It would take more than 25 years to do so. Planning for the “Big Dig” started in 1982, and construction finally began in 1991 after federal funding was secured. The project was supposed to take 10 years, with a budget of just under $2 Billion. If you read this section header, however, you know how this ends.
The project was riddled with delays and problems, and by the end of it, costs had skyrocketed (to $15B) and construction time was almost doubled (to 17 years). A large number of factors contributed to this super-high switch cost, including but not limited to:
- 70% of Boston was built on land-filled Massachusetts Bay, which meant that it wasn’t safe to build serious infrastructure upon. Any heavy supports needed to be drilled into the bedrock below the water.
- Planners didn’t want to recreate the Central Artery’s issues. This meant no displacing residents and no above-ground highways. They would have to go under ground.13
- Residents of nearby neighborhoods lobbied to prohibit construction between the hours of 11pm and 7am.
- Boston could not shut down – all of its residents and professionals would continue life as usual throughout the construction period.
- At one point, they had to dig to within 4 vertical meters of the currently-operating subway Red Line, which was built back in 1914 and whose structural integrity was in doubt.
Finally, in 2007, after decades of construction, the Big Dig was announced as complete. All told, the megaproject involved four major parts:
- Rerouting the Central Artery underground.
- Building the Ted Williams Tunnel to connect the highway to the airport.
- Building the Zakim Bridge over the Charles River to connect Boston and Charlestown.14
- Developing the Rose Kennedy Greenway, an urban park replacing the Central Artery.
So was it worth it?
Various reports calculated a 62% reduction in vehicle hours of travel, with savings for travelers estimated at $166 MM per year.15 From a strictly economic perspective, many cited the Big Dig as one of the worst examples of a large-scale, federally-funded city infrastructure project. Indeed, it was a costly bit of technical debt to tackle, and many still rightfully wonder to what extent it’s actually been paid.16
All that said, I moved to Boston in 2007 – the year they completed the Big Dig – and I have to say that in researching this article, I didn’t realize how many awesome things happened because of it: the beautiful Zakim bridge, the easy ride to the airport, the picturesque entryway to the North End, the burgeoning Innovation District, and much more. Yes, many goals weren’t met and many lessons should be learned from the Big Dig, but I believe a city is as much about its identity and what it means to its residents, and so I’m glad it happened. (Let’s just hope it lasts a little longer than the Central Artery.)
TLDR; the Big Dig is a prime example of massive technical debt in a system. The pre-existing traffic solution was broken, and the city elected to take on the huge switch cost of building a better solution.
But Where’s the Technology??
Now, astute readers among you may be wondering (or grumbling) that for all my talk of technical debt, I haven’t written very much about “technology”. I know.
This was a conscious decision. We didn’t need to talk about technology because all of the concepts covered here are directly transferrable to the land of zeroes and ones. Indeed, any technical system is built to accomplish tasks and solve problems, so it’s not unlike your battle plan for moving to a new apartment.
I’ll also bet that more of you have moved, shoveled your driveway, or experienced traffic than have migrated to a new database engine, refactored complex data models, or deprecated hardware/software. Remember, this blog is all about relatability! Get it? Got it? Good.
So what’s the big takeaway, then?
Technical debt is costly and often impossible to avoid. We can minimize it by considering its four primary drivers: inadequate planning, limited resources, competing goals, and evolving contexts.
Remember this, and you’ll be able to speak loudly and proudly whenever you say…
- At least, Ponce de León might have hoped so.
- Technical debt is also sometimes called “design debt”. I find this term more technically (if not empirically) correct, since most technical solutions can trace any inadequacies back to their initial designs.
- I guess snowblower stores exist?
- No, not “consensual”. It’s definitely not consensual…
- Gotta love the Craigslist fire-sale.
- No shit. There’s an article from the Daily Boston Globe published December 21, 1930, which references a city proposal to fix the traffic.
- Ahhh beautiful Boston – where you might as well stay at work because going home and back will take you over 32 hours.
- The bicycle wasn’t even invented yet. HECK, America wasn’t even invented yet!
- The MBTA website has a really fabulous write-up on the history of its system of ferries, omnibuses, horse-drawn cars, cable cars, and underground trains.
- Thwing’s amazingly specific catalog, The Crooked & Narrow Streets of the Town of Boston, 1630 – 1822, is worth a read if this type of thing is your bag.
- This is a good example of switch cost lowering as a result of an event (e.g., the fire). When the buildings lining a street have been decimated by a fire, it’s an opportune time to redo the streets because previously complicating factors (e.g., the buildings and their inhabitants) are no longer an issue.
- Nicknames for the Central Artery included “The Distressway”, “the largest parking lot in the world”, and “the other Green Monster“. I’ll give you one guess as to whether Bostonians thought the project was a success.
- Hence the name, “Big Dig”.
- It remains the widest cable-stayed bridge in the world.
- Damn right, I’m citing Wikipedia.
- The Boston Globe actually estimated it will eventually cost closer to $22 Billion, and the interest won’t be paid off until 2038.