Software Operational Profile vs. Test Profile: Towards a Better Software Testing Strategy

Abstract

Software Operational Profile (SOP) is a software specification based on how users use the software. This specification corresponds to a quantitative representation of software that identifies the most used software parts. As software reliability depends on the context in which users operate the software, the SOP is used in software reliability engineering. However, there are evidences of a misalignment between the software tested parts and SOP. Therefore, this paper investigates a possible misalignment between SOP and the tested software parts to obtain, based on experimental data, more evidence of this misalignment. We performed an exploratory study composed of four activities to verify: a) whether there are significant variations in how users operate the software; b) whether there is a misalignment between SOP and the tested software parts; c) if failures occur in untested SOP parts in case of misalignment; d) in case of misalignment between SOP and untested software parts, whether a test strategy based on the amplification of the existent test set with additional test data generated automatically, can contribute to reduce the misalignment. We collected data form four software while users were operating them. We analyzed this collected data in an attempt to reach the goals of this work. To evaluate the originality of this research, we performed a Literature Systematic Review (SLR) and presented its conclusions. The obtained results evidence that there are significant variations in how users operate the software and also that there is a misalignment between SOP and the tested software parts when we evaluated the four software mentioned above. There are also indications of the occurrence of failures in the untested SOP parts. Although the test strategy mentioned above has reduced the possible misalignment, the test strategy is not enough to avoid it, thus denoting the need of specifics test strategies using SOP as a test criterion. These results indicate that SOP becomes relevant not only to software reliability engineering but also to contribute to testing activities, regardless of the adopted strategy.