A plague on both your houses: The debate about how to deal with ‘inconclusive’ conclusions when calculating error rates

Abstract

To the Editor, There is an ongoing debate about how to deal with ‘inconclusive’ conclusions when calculating error rates. Recent contributions to this debate include, in chronological order, Dror and Scurich (2020), Weller and Morris (2020), Biedermann and Kotsoglou (2021), Arkes and Koehler (2021), Dror (2022) and Arkes and Koehler (2022). All the proposed solutions are inappropriate because they do not address the real problem. The real problem is not what to do with ‘inconclusive’ conclusions but the fact that forensic practitioners report conclusions as ‘identification’, ‘inconclusive’ and ‘exclusion’ at all. The real solution is for practitioners to abandon this practice and, instead, adopt the logically correct framework for interpretation of forensic evidence, the likelihood-ratio framework. Rather than reporting each of their conclusions as one of three categories, practitioners should report them as continuously-valued likelihood-ratio values.1 When conducting tests, if the test pair is a same-source pair, then a good output by the forensic-evaluation system would be a large positive log-likelihood-ratio value, a poorer output would be a small positive log-likelihood-ratio value, a worse output would be a small negative log-likelihood-ratio value, and a much worse output would be a large negative log-likelihood-ratio value. Mutatis mutandis, for a different-source pair, for which a good output would be a large negative log-likelihood-ratio value.2 When calculating error rates in a categorical framework, a penalty value of 1 is assigned to each response that is the incorrect category and a penalty value of 0 is assigned to each response that is the correct category, then the penalty values are averaged. In contrast, an appropriate validation metric for a system that outputs likelihood ratios assigns a small penalty value to each log-likelihood-ratio value that is large in the correct direction, a less small penalty value to each log-likelihood-ratio value that is small in the correct direction, a large penalty value to each log-likelihood-ratio value that is small in the incorrect direction, and an even larger penalty value to each log-likelihood-ratio value that is large in the incorrect direction, then the penalty values are averaged. The penalty values are assigned using a continuously-valued function of the continuously-valued log-likelihood-ratio outputs of the forensic-evaluation system. A well-established validation metric that has these properties is the log-likelihood-ratio cost (Cllr; Brümmer and du Preez, 2006). It has been in use in forensic science since at least 2007 (González-Rodríguez et al., 2007), and its use is recommended in the recently-published Consensus on validation of forensic voice comparison (Morrison et al., 2021).3 An appendix in the latter publication provides an introduction to Cllr and its interpretation.