Establishing driving simulator validity: drawbacks of null-hypothesis significance testing when compared to equivalence tests and Bayes factors

Abstract

The validity of driving simulators is a topic that has been addressed ever since their first use. In practice, a comparison of field data and simulator data is often used for validation purposes. A systematic literature search was conducted which identified large heterogeneity in both study designs and statistical methods used in research on simulator validity. Most of the included studies concluded simulator validity (thus equivalence of field data and simulator data) from non-significant results in Null-Hypothesis Significance Testing (NHST), which is a well-known yet somewhat controversial practice. In this work, the consequences of both study design choice and the choice of the statistical method are outlined and alternatives such as using Bayes factor hypothesis testing and equivalence tests are discussed and applied to a real data example. The present paper is intended to shed light on the influence of the statistical method on conclusions drawn from experimental data. For research on driving simulator validity, we highly encourage the use of equivalence tests or Bayes factor hypothesis tests, in particular when evidence of the data suggests that an effect might be absent.