Abstract
Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice.
Highlights
MUCH work on software testing seeks to automate as much of the test process as practical and desirable, to make testing faster, cheaper, and more reliable
Automated test input generation been the subject of many significant advances in both Search-Based Testing [3], [5], [83], [126], [128] and Dynamic Symbolic Execution [75], [108], [161]; yet none of these advances address the issue of checking generated inputs with respect to expected behaviours—that is, providing an automated solution to the test oracle problem
This paper has provided a comprehensive survey of test oracles, covering specified, derived and implicit oracles and techniques that cater for the absence of test oracles
Summary
MUCH work on software testing seeks to automate as much of the test process as practical and desirable, to make testing faster, cheaper, and more reliable. In 2001, Baresi and Young [17] presented a partial survey that covered four topics prevalent at the time the paper was published: assertions, specifications, state-based conformance testing, and log file analysis. While these topics remain important, they capture only a part of the overall landscape of research in test. We identify key milestones and track the volume of past publications Based on this data, we plot growth trends for four broad categories of solution to the test oracle problem, which we survey in Sections 4, 5, 6, and 7.
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
