The past, present and future of technical debt

Abstract

Listen

Translate article

While technical debt has emerged as a formal concept relatively recently [2] we have had technical debt from earliest days of software development, it has simply evolved in nature. So what can we learn from past types of technical debt to allow us to prepare for its future forms?When we look back over recent software history, we can see five identifiable evolutions of software systems [5], each one roughly aligning with a decade.Before and through 1980s, software systems were largely monolithic and tended to run single computers, with software being developed as monolithic programs. As we moved into 1990s, distributed systems became mainstream and standard style for an enterprise system became three-tier client server. The Internet became a mainstream technology in late 1990s, and organisations developed Internet-connected systems, which were always on rather than just online and could support difficult and unpredictable quality properties. In current era, we are building Internet-native systems, where the Internet is system. These systems are built from a combination of open source components, remote Internet connected services and custom and their services often form part of Internet via publicly accessible APIs.Following current trends, it seems that next phase of evolution will be to Intelligent-Connected systems, as artificial intelligence (machine learning in particular) becomes mainstream [1], users expect context specific assistance, and fast, reliable networks allow us to connect things (devices) to our systems as well as traditional computers [3].Software engineering practice evolves in response to new challenges and each era of computing has introduced new techniques and technology but each has also introduced its own types of technical debt too.In monolithic era, focus was structuring a single program, with spaghetti code, poor naming and unrestricted use of goto emerging as examples of earliest types of technical debt. As we moved into distributed era, we ended up tangling presentation and business logic code in our client/server user interfaces, while in Internet-connected era we distorted our systems to meet performance and scalability concerns at all costs and often ended up with poorly-designed automated tests being an inflexible technical debt of their own. More recently Internet-native systems often introduce a mishmash of microservices with poorly understood choreography and diverse internal implementations, references to external APIs that became unsupported or difficult to use and public APIs with many versions, all of which have to be maintained forever due to callers who would not migrate to new versions.So what types of technical debt do we expect in future?In intelligent-connected era, amongst other things, applications will have machine learning features and we'll need large datasets to train machine learning models and provide context-specific user experiences and we'll have lots of non-computing devices connected to our systems, providing data. So we'll probably get machine learning debt [4], ML models that we can't explain, models that we can't improve because people rely their quirks (even if wrong). We'll also have large inflexible data sets which our systems and models rely on, and we'll have unknown and unpredictable collections of things connecting to our services, which we can't change because other people own them.While this sounds like a daunting set of challenges, intelligent-connected era is only just beginning, so we have not yet incurred significant amounts of technical debt. By looking to past as our guide to future we can be forewarned and start find solutions to our future technical debt before we have become overwhelmed by it!