Accuracy and efficiency comparisons of single- and multi-cycled software classification models

Abstract

Software classification models have been regarded as an essential support tool in performing measurement and analysis processes. Most of the established models are single-cycled in the model usage stage, and thus require the measurement data of all the model’s variables to be simultaneously collected and utilized for classifying an unseen case within only a single decision cycle. Conversely, the multi-cycled model allows the measurement data of all the model’s variables to be gradually collected and utilized for such a classification within more than one decision cycle, and thus intuitively seems to have better classification efficiency but poorer classification accuracy. Software project managers often have difficulties in choosing an appropriate classification model that is better suited to their specific environments and needs. However, this important topic is not adequately explored in software measurement and analysis literature. By using an industrial software measurement dataset of NASA KC2, this paper explores the quantitative performance comparisons of the classification accuracy and efficiency of the discriminant analysis (DA)- and logistic regression (LR)-based single-cycled models and the decision tree (DT)-based (C4.5 and ECHAID algorithms) multi-cycled models. The experimental results suggest that the re-appraisal cost of the Type I MR, the software failure cost of Type II MR and the data collection cost of software measurements should be considered simultaneously when choosing an appropriate classification model.