Abstract
Background: The existing literature in software engineering reports adverse effects of confirmation bias on software testing. Confirmation bias among software testers leads to confirmatory behaviour, which is designing or executing relatively more specification consistent test cases (confirmatory behaviour) than specification inconsistent test cases (disconfirmatory behaviour). Objective: We aim to explore the antecedents to confirmatory and disconfirmatory behaviour of software testers. Furthermore, we aim to understand why and how those antecedents lead to (dis)confirmatory behaviour. Method: We follow grounded theory method for the analyses of the data collected through semi-structured interviews with twelve software testers. Results: We identified twenty antecedents to (dis)confirmatory behaviour, and classified them in nine categories. Experience and Time are the two major categories. Experience is a disconfirmatory category, which also determines which behaviour (confirmatory or disconfirmatory) occurs first among software testers, as an effect of other antecedents. Time Pressure is a confirmatory antecedent of the Time category. It also contributes to the confirmatory effects of antecedents of other categories. Conclusion: The disconfirmatory antecedents, especially that belong to the testing process, e.g., test suite reviews by project team members, may help circumvent the deleterious effects of confirmation bias in software testing. If a team's resources permit, the designing and execution of a test suite could be divided among the test team members, as different perspectives of testers may help to detect more errors. The results of our study are based on a single context where dedicated testing teams focus on higher levels of testing. The study's scope does not account for the testing performed by developers. Future work includes exploring other contexts to extend our results.
Highlights
C ONFIRMATION bias is the cognitive tendency to look for evidence that confirms, rather than refutes, one’s prior beliefs [1]
We provide a list of disconfirmatory antecedents that can be used by practitioners to alleviate confirmation bias
We applied grounded theory to explore the antecedents to confirmatory and disconfirmatory behaviours and to understand how they occur among software testers
Summary
C ONFIRMATION bias is the cognitive tendency to look for evidence that confirms, rather than refutes, one’s prior beliefs [1]. Confirmation bias occurs when developers or testers exercise a program with the data that is consistent with its specified behaviour instead of inconsistent data [2]. Confirmation bias leads to confirmatory behaviour by software testers during testing [3]. If requirements specification state that ...the phone number field accepts seven digits from 0 to 9; a consistent test case would validate the behaviour of the field by providing in, e.g., 0123456 as an input test data. An inconsistent test case would validate the field’s behaviour with inconsistent data, e.g., a - a letter instead of a digit. The higher the level of confirmation bias, the more adverse effects it has on software testing [4], [5], [6], [7]. Calikli and Bener observed a positive correlation between software defect density and confirmation bias
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
