Accuracy assessment methods for likelihood ratio-based evidence evaluation

Abstract

There is an increasing interest in scientific assessment methods for measuring the accuracy of the evidence evaluation process in any discipline in forensic science. The reasons for such interest are multiple, but the American Daubert rules have played an important role in motivating the forensic laboratories to find a proper way of expressing theaccuracyof their techniques [1]. Regarding theevidenceevaluationand interpretation process, and usually aimed at reporting figures of merit to a factfinder, the assessment frameworks proposed in the literature have been many, but some of them have become particularly popular. In this contribution,we reviewsome frequentlyusedmethods for the assessment of the evidence evaluation process, assuming that the weight of such evidence is expressed in the form of likelihood ratios. Threemethods used in the literature are described and compared: rates ofmisleading evidence (frequently found as false positive and negative rates); hypothesisdependent likelihood-ratio histograms and its cumulative version, Tippett plots; andmethodsbasedonstrictlyproper scoring rules,withemphasis in the information-theoretical Empirical Cross-Entropy (ECE). Moreover, we present a novel assessment method to be used jointly with Tippett plots, and based on theoretical bounds on the probability of observing misleading evidence [2]. This method, which has been coined ‘Limit Tippett plots’, is especially adequate to detect inaccuracies in the models used to represent the data, demonstrating its usefulness for any likelihood ratio-based evidence evaluation technique, and showing to incorporate rich and useful information to conventional Tippett plots. Results are contributed in two ways. First, some illustrating examples are generated using synthetic data, where typical errors in the evidence evaluation process are simulated. Second, some results are presented using forensic automatic speaker-recognition systems. On the one hand, both sets of results demonstrate the adequacy of the presented method, Limit Tippet plots, to detect inaccuracies in the models used for likelihood ratio computation. On the other hand, the novel proposed technique is compared to other assessment frameworks, and some discussion is presented about whether themethods are complementary in daily forensic work andwith regards to reporting to a factfinder.