Assessing the quality of large, software-intensive systems: A case study

Abstract

This paper presents a case study in carrying out an audit of a large, softwareintensive system. We discuss our experience in structuring the team for obtaining maximum effectiveness under a short deadline. We also discuss the goals of an audit, the methods of gathering and assimilating information, and specific lines of inquiry to be followed. We present observations on our approach in light of our experience and feedback from the customer. In the past decade, as engineers have attempted to build software-intensive systems of a scale not dreamed of heretofore, there have been extraordinary successes and failures. Those projects that have failed have often been spectacular and highly visible [3], particularly those commissioned with public money. Such failures do not happen all at once; like Brooks’ admonition that schedules slip one day at a time [2], failures happen incrementally. The symptoms of a failing project range from the subtle (a customer’s vague feelings of uneasiness) to the ridiculous (the vendor slips the schedule for the eighth time and promises that another $30 million will fix everything). A project that has passed the “failure in progress” stage and gone on to full-fledged meltdown can be spotted by one sure symptom: the funding authority curtails payment and severely slows development. When that happens, the obvious question is asked by every involved party: “What now?” The answer is often an audit. This paper summarizes the experience of an audit undertaken by the Software Engineering Institute (SEI) in the summer of 1994 to examine a large, highly visible development effort exhibiting the meltdown symptom suggested above. The customer was a government agency in the process of procuring a large software-intensive system from a major contractor. The audit team included the authors of this paper, as well as members from other organizations. Members of the team had extensive backgrounds and expertise in software engineering, in large systems development,