Paying Back Technical Debt
Kyle Brown, IBM Fellow, CTO Cloud Architecture
Let’s face it, 2020 was a weird year. One of the smaller oddities was that for the first time since 2012, the U.S. Personal savings rate grew (and at a phenomenal rate) rather than holding roughly steady [1]. While most of this is pandemic-related, it may partially be a sign that consumers have begun to realize that you can’t just keep borrowing without repaying debts you have incurred. What I wish would occur is that businesses would realize that the same principle applies to technical debts just as much as it does to financial debts. This analogy may be a bit jarring to some, but it’s actually a well established idea that was first introduced by Ward Cunningham over 15 years ago[2], and further developed by Joshua Kerievsky in 2005[3].
Briefly, a Technical Debt is incurred by a development team when they put an important software development activity off in order to complete other activities. The intention is usually that they will “go back” and finish off that activity, but intentions often do not translate into activity. The types of activity put off may be as simple as writing documentation, but it may also be more involved activities like changing a piece of code to make it more understandable or maintainable or updating design documentation that has gone out of date due to changes in the code.
I’ve recently been involved in several customer situations where I felt kinship to a consumer debt counselor talking with a couple that is facing an outsized mortgage, a rising set of outlandish credit card balances, and a new baby on the way. In each case, we were close to being overwhelmed with the technical debt, and we had to find some way to reduce that debt while still continuing to develop new features and move forward. What I’ve come up with is a set of practices that have close parallels to the steps that credit counselors advise their customers to adopt. Let’s take a look at each of these and see how they can apply in technical debt situations that so many projects face.
Step 1: Take stock of your situation. Find out what you owe.
This step is the most critical. Once a team has decided that they must pay back their technical debt (which is not an easy decision — and one that must be taken together with the business) they have to figure out what debts are actually owed. The best way I have found to do this is to perform a top-to-bottom design and code review.
Begin by looking at your design documentation; is it up to date? Does it capture the most important points of your design? You may then want to prioritize a review of the code by examining which areas of the code have given you the most trouble — what parts are the hardest to modify? What parts have the highest error rates? What parts are most important to your business? Finding the answers to these questions will help you rank the activities you need to take on in order to figure out how to improve the situation you are in.
Code and documentation aren’t the only parts of your system that can incur technical debt. Another key element to consider is operational debt — for instance, are you running on back-levels of platform software like a database or application server? Is your operations team manually performing tasks that should be automated, wasting time and money? And do you have monitoring in place to make it possible to discover problems before they bring your site down or do you waste time in post-mortem analysis?
It’s often best to bring in an outside expert to help you take stock of your project status. Bringing in an outsider will allow you to receive an assessment based purely on the technical merits of your solution, independent of office politics or personal attachment to sections of the code.
The final assessment needs to include a description of what needs to change, ranked by priority, together with recommendations on how to change the code and cost estimates for each change listed. Once you have these facts in hand, you can begin negotiating with the business owners on which debts you pay back and in what order.
Step 2: Stop incurring new debt
While the previous step is the most important in the grand scheme of things this one usually leads to the most rancorous discussions with the business. The hardest part of doing this is learning to change your organization’s culture so that you don’t accumulate more debt than you can reasonably service. Referring to our financial debt analogy, this is a very hard thing too — changing your spending habits and only spending on what you need, rather than buying what you want on credit is a hard lesson to learn.
In order to fix the problems you have found you have to take the time to implement the fixes, and that means putting aside new development while you correct the problems. There’s no perfect way to do this, and whatever approach you take will need to be taken after negotiating with the business on how to balance technical debt reduction and new development. Some strategies we’ve seen that work are:
Include debt reduction activities in with your user stories. If you have developed a set of sized and ranked activities from the previous step, then you can work with the business to make sure that these are include a few of these in each development cycle. The tricky part is balancing the debt reduction activities with the new development that touches on the same area of code. For instance if you’re developing an e-commerce site and you’ve found that most of your problems are in checkout, you may want to put off new development in that area only while you repay technical debt (for instance, refactoring code or updating documentation) in that section. In this case, adding new promotions or changing your product pages would be fair game for new development while these changes proceed.
Employ a beta strategy. If your infrastructure is built to sustain two web sites (one your main site and the other a “beta“site) then you can continue new development on the beta site while refactoring the main code stream. This has the advantage of not slowing down any new development, while at the cost of more difficult integration later on when changes to the beta site must be re-integrated with the main site.
Step 3: Choose your debt reduction strategy
Again in this case, we can draw a parallel to credit card debt reduction strategies and consider two different ways of prioritizing your debt repayments. The first possible strategy is the “highest interest rate first” approach. In this strategy in the credit world, you repay the credit cards with the highest rate first since they cost you the most in interest payments. By analogy in the technical sphere, this means that you can consider taking on the tasks that have the biggest impact first. That allows you to get them out of the way, which would usually then clear the path for other changes, and may also have the biggest payback in terms of improved performance, maintainability, etc.
The other possible strategy you can take is the “lowest balance first” strategy. In credit cards terms, this means paying off the credit cards that have the lowest balance first — this leads to an immediate sense of accomplishment that something has been done quickly. An analogous strategy for technical debt would be to tackle the smallest fixes first — this is especially useful if you must convince the business or management of the utility of paying back technical debt, or if your company has a very results-oriented culture where showing immediate progress is important to sustain funding for larger efforts.
Step 4: Stick with the plan!
The key here is to make the debt reduction part of your ongoing activities for the long run. This is not a one-shot deal; many companies became disillusioned with terms like “Refactoring”[4] when they came into vogue a few years ago because they expected short-term results from a process that is best looked at as a long-term investment. You will always incur new debt; the trick is to make sure that you can repay it in a reasonable amount of time rather than let it accumulate.
Step 5: Track and reassess your progress
Finally, you need to be able to report on the progress you’ve made in your debt repayment activities. It’s particularly important to capture metrics that help justify the time spent in these activities to management and the business. For instance, many times you need to refactor code to improve performance; having the right statistics on hand to show improvements in the user experience are important. Likewise tracking improvements in the rate in which you can add new features to a simplified code base is another important metric that demonstrates value to the business.
Following these steps won’t solve all of your problems with technical debt, but they will at least allow you to systematically determine what needs to be done, what value that brings to the development process, and how well the changes address the problem. If you keep at it then this should make your development artifacts easier to maintain and should make your development process less stressful as a result.
Acknowledgements
The author would like to thank Rachel Reinitz for her very insightful and helpful comments on this article.
[1] https://fred.stlouisfed.org/series/PSAVERT
[2] Ward Cunningham, The Wycash Portfolio Management System, ACM SIGPLAN OOPS Messenger, April 1993
[3] Joshua Kerievsky, “Refactoring to Patterns”, Addison-Wesley, 2005
[4] Martin Fowler, “Refactoring, Improving the Design of Existing Code”, Addison-Wesley, 1999