Abstract
A key goal of empirical research in software engineering is to assess practical significance, which answers the question whether the observed effects of some compared treatments show a relevant difference in practice in realistic scenarios. Even though plenty of standard techniques exist to assess statistical significance, connecting it to practical significance is not straightforward or routinely done; indeed, only a few empirical studies in software engineering assess practical significance in a principled and systematic way. In this paper, we argue that Bayesian data analysis provides suitable tools to assess practical significance rigorously. We demonstrate our claims in a case study comparing different test techniques. The case study's data was previously analyzed (Afzal <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> , 2015) using standard techniques focusing on statistical significance. Here, we build a multilevel model of the same data, which we fit and validate using Bayesian techniques. Our method is to apply cumulative prospect theory on top of the statistical model to quantitatively connect our statistical analysis output to a practically meaningful context. This is then the basis both for assessing and arguing for practical significance. Our study demonstrates that Bayesian analysis provides a technically rigorous yet practical framework for empirical software engineering. A substantial side effect is that any uncertainty in the underlying data will be propagated through the statistical model, and its effects on practical significance are made clear. Thus, in combination with cumulative prospect theory, Bayesian analysis supports seamlessly assessing practical significance in an empirical software engineering context, thus potentially clarifying and extending the relevance of research for practitioners.
Highlights
A MAIN goal of research in empirical software engineering (ESE) is assessing practical significance: what is the impact of the research findings in realistic scenarios? To this end, statistical analysis has been used extensively in ESE for decades
We demonstrate how to build a rigorous model of practical significance, and the advantages of reporting significance results in a way that is grounded in concrete decision-making scenarios—in contrast to the traditional approach that presents general statistics in a more abstract form
In order to present the background of our research, we introduce the essential terminology and techniques of Bayesian statistical analysis and cumulative prospect theory (CPT)
Summary
A MAIN goal of research in empirical software engineering (ESE) is assessing practical significance: what is the impact of the research findings in realistic scenarios? To this end, statistical analysis has been used extensively in ESE for decades. A common example are effect size measures (such as Cohen’s d, or the size of coefficients in a regression model): if the effect size of a technique A is markedly bigger than the one of another technique B, this is taken as an indication that A performs better than B in practice. This common approach overlooks the issue that assessing practical significance on statistical measures such as effect sizes makes it hard to ensure that the statistics accurately reflect expert knowledge. In Bayesian statistics, this is expressed as estimating the probability P (θ | D) of the parameters given the data, which obeys Bayes’ theorem:
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
