Evaluating testing effectiveness during software evolution: a time‐series cross‐section approach

Abstract

SUMMARYAs software evolution organizations (SEOs) formalize their activities into standard repeatable processes, capability maturity models suggest that they gather and analyze data to quantitatively manage their activities. This study examines the use of a time‐series cross‐section (TSCS) modeling approach to inform the quantitative management of the testing activities within a mature SEO. This paper describes the TSCS modeling approach and provides an exemplar case study that details its use. Data for the case study come from readily available, commonly gathered production and testing defect reports in the quantitative measurement efforts of ait CMMI‐DEV Level 3 assessed software development and evolution organization. Covering six independent projects for 43 months, the case study defines quality in terms of reported defects in the production code base. The predictors are the six projects and defect data identified in the unit, system, and regression testing activities logs. The case study also details how results from the TSCS analysis triggered causal analysis to improve the testing activities. The study provides evidence that TSCS models can be used as a quantitative problem identification tool for the mature (SMmm Level 3 or higher) SEO's testing activities. Copyright © 2011 John Wiley & Sons, Ltd.