The Measurement of Performance in Probabilistic Diagnosis

Abstract

Owing to the inherent uncertainty of diagnostic tasks, diagnostic advice should be offered in a probabilistic, rather than deterministic form. Since the late fifties a lot of effort has been invested in constructing probabilistic diagnosis rules. Much less has been done to devise rational tools for evaluating them ; conventional error rates reflect but one aspect of performance in a rather crude way. The aim of this paper and its successors is to offer a body of evaluation tools. After defining a general framework and stating its limitations we apply some graphical techniques to the acute abdominal pain data that will serve as illustration in the next papers as well: dot diagrams, the triangular diagram for the three-disease case, and three tabular representations based on categorization of the probabilities, viz. the usual (forced) classification matrix, from which various classification rates are read off; the classification matrix with doubt, in which uncertain and confident diagnoses are distinguished; and the exclusion matrix, which spots diseases that are judged improbable. Together these matrices give a good first impression of the behaviour of a probabilistic diagnosis system. The outlined techniques of later papers are needed for a more complete analysis.