Tying up loose ends: a method for constructing and evaluating decision aids that meet blunt and sharp-end goals

Abstract

We present a methodological framework for constructing and evaluating decision aids – fast and frugal trees (FFTs) – ideally suited to the front line of an organisation. Their performance can be analysed in signal detection theory, allowing for transparent selection of FFTs given managerial-level trade-offs among type I and II errors. We extend FFTs from binary classification to selection from multiple actions (FFT multiple) as well as performance analysis to organisational goal states beyond type I and II error reduction. Concepts and framework are introduced and a tutorial-style example application (threat assessment at military checkpoints) is provided. Throughout, we discuss ways to deal with missing or incomplete data and show that the performance of decision aids may be overestimated if the effectiveness of actions is not heeded. The methodology can be used to construct and evaluate decision aids in any area characterised by dichotomised cues and a one-to-many mapping between categorisation outcomes and actions.