What is technical debt?

In order to move faster, save money, or achieve some other short-term objective, decision-makers will delay the harder, slower, more expensive fix until later. At that point, the fix will inevitably be even harder, slower, and more expensive than it was before, and they’ll also have suffered with lower performance in the meantime. Technical debt is what results from changes that make future fixes and maintenance more difficult or even impossible.

These changes can be to code, design, technical documentation, development environments, third-party tools, development practices, and more. The choices can range from deferring bug fixes and skimping on automated testing to delaying necessary major architecture changes.

“Technical Debt is an invisible, insidious, pernicious killer of our software,” writes Chief Scientist and Senior Agility Instructor Dan Rawsthorne.

But is it really? Saying technical debt is always bad is like saying borrowing money is always a bad idea. In this post we’ll cover the downsides to technical debt, what causes technical debt, how to prevent tech debt, tips for paying it off, and myths around tech debt. Let’s get started.

The downsides of technical debt

In a nutshell, tech debt is bad for a few reasons:

• It's self-perpetuating because no one wants to work on it
• No one wants to work on it in part because there's no glory in working on it
• It's bad for morale because it's tedious and causes bickering
• It delays new releases
• It's time-consuming to fix

In Exploring Scrum: the Fundamentals the authors describe technical debt as the difference between running on a flat surface in good shoes on a beautiful day and running in water up to your waist. It takes a lot more time and energy to cover the same distance. “Technical Debt is probably the number one impediment to Teams being agile,” the authors write.

Tech debt is buggy, slow, and difficult-to-maintain code. Lack of adequate documentation misleads developers or forces them to dig through months of pull requests to discover what broke and when. The build process is slow and inefficient. Outages happen. Panic sets in. Fixes take hours or days, pulling people off scheduled work. Every fix creates more problems. Sometimes you have to throw away code, which never feels great. It also tends to accumulate in a company’s most popular, profitable applications.

It’s often a self-perpetuating problem. Buying new things is fun. Paying back debt isn’t.

No one wants to be the one to work on tech debt because the work is often boring and no one outside Engineering appreciates or respects it. “Nobody likes working with a significant handicap and being unproductive day after day,” IT Management Consultant Erik Dietrich writes about tech debt. Developers aren’t exactly chomping at the bit to spend the better part of their day doing something simple like adding a checkbox to a form. “They know that they’re going to have to manufacture endless explanations for why seemingly simple things take them a long time. When new developers are hired or consultants brought in, they know that they’re going to have to face confused looks, followed by those newbies trying to hide mild contempt.”

It’s not great for morale. Tech debt leads to discord and bickering among teammates. New developers blame long timers, maintenance programmers blame developers, the greenfield project team blames the the legacy project team.

“When you are working around problems, when it becomes harder and harder to do your work for fear of breaking the existing code, when every solution you try hits another bug that stops you, life becomes frustrating,” writes Dr. Mark J. Balbes, Senior Director at WWT. “It's easy to blame others on the team for the problems you are facing.”

For management, delaying new releases and hitting pause on support for older products isn’t an easy sell. “For a manager, a code base high in technical debt means that feature delivery slows to a crawl, which creates a lot of frustration and awkward moments in conversation about business capability,” IT Management Consultant Erik Dietrich writes.

And the fixes are time-consuming, usually taking longer than a Hack Day, or week. All this incentivizes teams to continue kicking the can down the road until something really breaks.

Badly managed technical debt, according to Ozkaya, “will bankrupt the software-development industry. Practitioners today have no choice but to treat technical-debt management as one of the core software engineering practices.”

Examples of technical debt in the wild

There's no shortage of technical debt examples out there to learn from.

Microsoft’s crashed their servers in 2013, leading to a seven-hour outage. They’d launched a new SaaS version of Visual Studio without cleaning up their technical debt, leaving their servers unable to handle millions of users’ simultaneous requests.

Evaldas, Technical Team Lead at Euromonitor International, wrote that his current codebase is an example of technical debt. The databases contain a decade-plus-old “huge relational monolith.”

Symptoms:

• Lots of production objects are dead (not used, or wouldn't compile anymore)
• Objects that were supposed to go to production but never did are there
• Databases calling objects from other databases in a manydirectional way
• Mixed up naming conventions
• Lots of dynamic SQL
• Business layer and sometimes even presentation layer has been put to database, primarily stored procedures which means changes often have to be done in many places and tracked
• Lots of quick fixes and hard-coded conditionals

“So yep, most of these changes mean that sometimes we're making changes on old objects that will not make an impact at all,” Evaldas wrote. “There's a huge risk of breaking things apart, risk that someone will forget to make a change in one place due to hard-coding.”

Ipek Ozkaya Senior Member of Technical Staff, Carnegie Mellon Software Engineering Institute wrote about a successful maritime equipment company that amassed 3 million lines of code over 16 years. In that time new companies entered the marketplace and staff turned over and they launched more and more products that would end up under warranty or maintenance contracts. Technical debt made small changes or additions to the products difficult and often resulted in broken code. When the company launched new products they had to do lots of manual and labor-intensive regression testing with existing products over several days for every release. And new hires had no idea why the code was breaking because the documentation for design and program choices was sparse and outdated.

Common causes of technical debt accumulation

Technical debt has two main causes: Shortcuts and changing realities.

Shortcuts

“In my experience, focusing on fast always creates Technical Debt,” the authors of Exploring Scrum: the Fundamentals write. Sometimes you need to build fast more than you need to build right. Many teams build applications that can’t scale or aren’t resilient for perfectly rational business reasons. Business realities can mean you need to get something out the door today and worry about tomorrow later. Other times a Project Leader who’s overestimated Velocity will push for suboptimal code today and worry about the ramifications later instead of going back to stakeholders and admitting error.

One tip here: If "fast" is the overriding objective, consider making the initial "fast" version completely discardable. Be real about the fact that you're creating tech debt and embrace it by setting the expectation that this is an experiment that will have to be tossed.

Changing realities

Sometimes events outside your organization, and outside your control, cause technical debt. Even with perfect code and testing the marketplace and best practices are constantly changing. Remember when LAMP, JBoss, VMware were the hotness? New entrants and conventions cause your stack or architecture to become outdated. In that case migrating to newer technologies is often difficult. Debt accumulates as management postpones it, and postpones it…

Conversely, another good way to accumulate debt is by doing a big migration too soon. There's always a new, sexy technology that would be "fun" to try, but making these choices quickly is a huge, huge creator of debt.

Tips for preventing technical debt

The most effective way to prevent technical debt is to follow best practices when writing code. Again, this isn’t always advisable when you’ve got to get something out the door. In that case, your best bet is to add appropriate “Cleanup stories” (more on this later) to the backlog.

But if you’re writing debt-saddled code because of poor planning, it’s essential your Product Manager, Engineering Manager, or whoever is in charge of estimates buys the team enough time to code correctly so the debt doesn’t pile up needlessly.

The authors of Exploring Scrum: the Fundamentals have a lot of tips for preventing unnecessary tech debt. They center around writing code correctly the first time. What’s correctly written code? It has a few attributes.

1. It follows the “once and only once” rule. That is, don’t write the code to do one thing multiple times.
2. It’s modular, and its modules are cohesive and relate loosely to each other.
3. Its modules are named according to what they’re supposed to do.
4. It’s readable and your coding practices are common and well-known.
5. The variables, methods, and classes are named based on what they're supposed to do.
6. It comes with documentation on the intent of your design decisions, how you’ve grouped your entities into modules, etc.
7. Its documentation has been updated as you’ve refactored and added to your code.

How do you know your code is readable? How do you know your naming practices are descriptive? How do you know your documentation makes sense?

Teamwork makes the dream work. Have another developer read the code and tell you what it does. Write comments that break the code into virtual modules based on what each section does.

Side note: Engineers often love to leave "TODO" comments for themselves or "FIX" in the code. It's better than absolutely nothing, but not by much, because it's often not clear what the context for the choice was. It's good to agree on a naming convention for the project so you can insert comments with searchable labels. For example, "TODO-team-settings," "FIX-teams-setting." In your comment expand on the context of the todo/fix that's needed and what led you to not address it in the moment. At the end of the project, you can just search the codebase for these keywords and come up with a quick list of all the outstanding todo/fixes.

Have someone who didn’t write the code read the comments and try to parse it. Put many eyes on the code. Pair program. Have lots of conversations. Reviews reviews reviews. Review architecture, test, design, code, etc.

You’ll also want extensive automated test coverage, which tells future developers what they need to know about the code before they go in making changes. You need two types: Black box and unit tests. Black box tests show you what the code does in the wild. Unit tests protect the internal environment.

Refactor early and often. One good refactoring rule of thumb: Always leave the code better than you found it.

Optional: Test Driven Development (TDD), eXtreme Programming (XP)

Tips for paying off technical debt

“The most effective way to handle technical debt is by ensuring it is continuously taken into consideration and part of technological decisions being made on an ongoing basis,” R&D Group Manager Adi Belan wrote.

The Exploring Scrum: the Fundamentals authors insist that the cardinal rule of tech debt is “First, do no harm.” Before you tackle existing tech debt, make sure you’re following the steps above to stop creating more debt.

After that, it’s a four-step process:

1. Refactor - Focus your efforts on the code near the code you need to change.
2. Rewrite - Last resort. Sometimes necessary.
3. Document - Document so future devs can tell what changes you made and why.
4. Test - Add black box tests.

It being a book about Scrum, the authors recommend what they call “Cleanup stories.” “Basically, a Cleanup Story is a Story that apologizes to the code base about something bad that happened, and promises to fix it.” When you find something broken, write down what’s broken and what needs to be done to fix it.

Whatever you call it, it’s a best practice to note tech debt wherever you find or create it. This makes it far easier to continuously assess the risks and costs of change versus maintaining legacy code. That can mean refactoring and bug fixes on the small end or migrating to newer systems and technologies on the large end.

It may be useful to plan a one-time or regular hackathon or something similar where the team focuses on maintenance. That’s what Balbes’ team did when faced with crippling tech debt. First, his team had one pair decouple the code into three major sections. “From there, we used a technique of divide and conquer,” Balbes wrote. “Some sections we wrapped with automated tests and refactored. Others we simply rewrote. We continued to fix bugs as we found them. And of course, before any of this, we added the software to our continuous integration server and automated build.”

Remember that dealing with tech debt is typically a thankless job that can kill enthusiasm if not handled well. Belan notes that many companies will task a junior developer with the job of managing legacy applications. This is a mistake for two reasons. First, n00bs don’t understand the system well enough to predict how even minor changes can affect other mission-critical services. They often don’t even know which services are mission-critical. The second reason it’s a mistake to assign tech debt-riddled systems to junior devs is that it kills morale. Tasking Senior Engineers with managing legacy code and rewarding and celebrating them for doing it well shows the task is a vote of confidence and seniority, not a punishment.

The book "Working Effectively with Legacy Code" by Michael Feathers is an excellent reference for techniques on how to deal with code that was written without tests.

Note: Paying off tech debt is a great use-case for timeboxing in software development.

Myths around technical debt

Myth: Technical debt is always bad

“Tech debt gets used to describe anything an Engineer doesn’t like about their codebase,” Jon Thornton, Engineering Manager at Squarespace, said recently on the Changelog Podcast.

As mentioned earlier, borrowing money isn’t always bad. The truth is that most businesses, and many other worthwhile endeavors, couldn’t have gotten off the ground without someone borrowing some money. Rather, borrowing too much and carrying the debt too long is bad. Tech debt is the same.

Instead of money, technical debt borrows time and effort. Sometimes tech debt pays off. Ozkaya describes good technical debt as a strategic investment strategy. Maybe you need to write good-enough-for-now code hit a deadline, maintain your SLAs, or manage business continuity.

Thornton needed to build an MVP that could send a few hundred emails unreliably to fast-track user feedback collection. They ended up trashing that MVP, but not because it was a mistake to build. Quite the opposite. The MVP did its job, and when it was done working, the Squarespace Engineering team pitched it and got to work on the real product with the feedback in hand.

Sometimes it makes more sense to pay the time and effort piper later. “If the software product is successful, this strategy can provide you with greater returns than if you had remained debt free,” Ozkaya writes.

Myth: Technical debt = bad code

Ozkaya again: Technical debt has “less to do with intrinsic code quality than with design strategy implemented over time.” As mentioned earlier, sometimes technical debt has nothing to do with choices you make as an organization and everything to do with where the market goes. Other times you choose to write code that’s great for what you need it to do today, but will need tests, documentation, etc. in the future. That’s not necessarily bad code. It’s just code that will need to be addressed again later.

Going forward

Like many things in life, tech debt isn’t all bad. But, left unchecked, it can come back to bite you in the butt. It can sap morale, hinder progress, and even cause catastrophic outages if left to compound for too long. The key is to only create tech debt where it makes sense, create realistic estimates that leave room for coding best practices, and continually address the debt you have as you go along.