Chapter Three - Automated Test Oracles: State of the Art, Taxonomies, and Trends

Abstract

Abstract Test oracle methods have changed significantly over time, which has resulted in clear shifts in the research literature. Over the years, the testing techniques, strategies, and criteria utilized by researchers went through technical developments due to the improvement of technologies and programming languages. Software testing designers, known as testers, currently have several resources to increase their confidence in the software under test correctness. All of these software testing resources are supposed to include a mechanism to decide whether a particular execution is considered a failure or not. In software testing environments, this decision is the responsibility of the test oracle. Despite the evolution and adaptation of testing techniques over more than 30 years, test oracles remain a particular and relevant issue. In this chapter, using literary evidence from a pool of about 300 studies directly related to test oracles, we present a classification of test oracles based on a taxonomy that considers their source of information and notations. Based on this classification, we perform a quantitative analysis to highlight the shifts in (evolution of) research on test oracles. Exploring geographical and quantitative information, we analyzed the maturity of this field using coauthorship networks among studies published between 1978 and 2013. Further, we determine the most prolific authors and their countries, main conferences and journals, supporting tools, and academic efforts and use a comparative analysis between academia and industry. Finally, from these analyses, we draw an analytic reflection about contemporary test oracle approaches and a criticism about oracle trends.