Traditional Signal Detection Theory Research Articles

Three-Way ROCs for Forensic Decision Making

Firearm examiners use a comparison microscope to judge whether bullets or cartridge cases were fired by the same gun. Examiners can reach one of three possible conclusions: Identification (a match), Elimination (not a match), or Inconclusive. Numerous error rate studies report that firearm examiners commit few errors when they conduct these examinations. However, the studies also report many inconclusive judgments (> 50%), and how to score these responses is controversial. There have recently been three Signal Detection Theory (SDT) primers in this domain. Unfortunately, these analyses rely on hypothetical data and fail to address the inconclusive response issue adequately. This article reports an SDT analysis using data from a large error rate study of practicing firearm examiners. First, we demonstrate the problem of relying on the traditional two-way SDT model, which either drops or combines inconclusive responses; in addition to lacking ecological validity, this approach leads to implausible results. Second, we introduce readers to the three-way SDT model. We demonstrate this approach in the forensic firearms domain. While the three-way approach is statistically complicated, it is well suited to evaluate performance for any forensic domain in which three possible decision categories exist.

Comparing Traditional and Fuzzy Signal Detection Analysis for Classification Data

This paper provides a comparative analysis of the results of applying traditional signal detection theory (tSDT) and fuzzy signal detection theory (fSDT) methods to classification data. Fuzzy signal detection theory generalizes tSDT by allowing for non-binary assignment of events and responses in order to capture the uncertainty inherent in detection judgments. Because traditional signal detection has been used extensively in classification research, the use of fSDT may ultimately provide new insights in this domain. Results were analyzed from a classification learning experiment in which category discriminability, base-rates and payoffs varied across conditions. Several quantitative and qualitative differences were observed across theoretical perspectives. The difference between derived hit and false alarm rates was smaller in the fSDT analysis, even though the data were the same. Differences between accounts of the base-rate and payoff conditions, as well as between category discriminability levels, were actually reversed in some cases, depending on whether tSDT or fSDT were used. The implications of these differences for interpreting classification data are considered.

A signal improvement to Signal Detection Analysis: Fuzzy SDT on the ROCs.

Fuzzy Signal Detection Theory (FSDT) combines traditional Signal Detection Theory (SDT) with Fuzzy Set Theory to generalize signal detection analysis beyond the traditional categorical decision-making model. This advance upon SDT promises to improve measurement of performance in domains in which stimuli do not fall into discrete, mutually exclusive categories; a situation which characterizes many detection problems in real-world operational contexts. FSDT allows for events to simultaneously be in more than one state category (e.g., both signal and nonsignal). The present study derived FSDT Receiver Operating Characteristic (ROC) functions to test whether application of FSDT meets the Gaussian and equal variance assumptions of traditional SDT and, therefore, whether the standard representation of the SDT decision space can be extended to the broader case of FSDT. Results supported the contention that FSDT does meet these traditional SDT assumptions, and further, that it yields higher sensitivity scores than traditional SDT when the category membership of events is ambiguous. ROC analyses indicate that use of traditional SDT formulas with fuzzy hit and false alarm rates is thus justified. The implications of this advance to both theoretical and practical domains are adumbrated.

Fuzzy Signal Detection Theory: A Monte Carlo Investigation

Fuzzy Signal Detection Theory (FSDT) has the potential of enhancing performance measures in detection tasks in which precision of Signal Detection Theory (SDT) analysis is limited by discrete mutually exclusive categorization of the state of the world and/or responses available to the observer. While there have been empirical efforts to demonstrate the benefits and tenability of FSDT, the question still remains whether traditional SDT computational procedures for measures of sensitivity and bias can be used with FSDT procedures. Through the use of Monte Carlo simulation and ROC analysis, the current study examined whether data analyzed by FSDT met the assumptions of traditional SDT on which sensitivity and bias measures are predicated. The results indicated that FSDT does in fact meet the normality and equal variance assumptions of SDT. However, the results also indicated that further theoretical elaboration of ‘fuzzy criterion setting’ is necessary.

From basic to applied research: theory and application of the a–b signal detection theory model

The goal of this research was three-fold: (1) address the basic versus applied research debate within the domain-specific signal detection theoretical framework, which scientists and practitioners have used extensively along the full spectrum of this continuum; (2) provide empirical support to validate the adequacy of the model; and (3) show the potential generalisability of the model to different domain-specific areas within applied settings. The results from the basic empirical study suggest that the a–b signal detection theory (SDT) model provides a more accurate theoretical framework for examining the underlying processes involved in signal detection. The findings from the domain-specific studies showed the potential applicability of the a–b SDT model for approaching applied problems. From a theoretical and applied point of view, this research suggests that the traditional SDT contention that the detection and response processes are independent from each other does not hold true for either basic signal detection tasks or domain-specific areas.

Theoretical Implications and Practical Applications of the a-b Signal Detection Theory Model for Human-Automation Interaction

The purpose of this research was threefold: 1) Present the a-b Signal Detection Theory (SDT) model as an alternative theoretical framework to overcome the limitations of traditional SDT, 2) Provide basic empirical support to validate the adequacy of the model, and 3) Show the potential generalizability of the model to different domain-specific areas within applied settings. The results from the basic empirical study suggest that the a-b SDT model provides a more accurate theoretical framework for examining the underlying processes involved in signal detection and decision making. Furthermore, the findings from the domain-specific study show the potential applicability and generalizability of the a-b SDT model for examining human-automation interaction. The end product of this work is particularly important for researchers and practitioners who are interested in applying a basic fundamental framework for examining how sensory, perceptual, and cognitive factors may affect humans' decision-making accuracy and response bias while interacting with automated systems in a wide range of applied settings.

Functions for traditional and multilevel approaches to signal detection theory

In the present article, functions written in the freeware R are presented that calculate several measures from traditional signal detection theory for each individual in a sample, along with summary statistics for the sample. Bias-corrected and accelerated bootstrap confidence intervals are also produced. Arguments are made for using an alternative approach--multilevel generalized linear models--and a function is presented for it. These functions are part of the R package sdtalt, which is available on the Comprehensive R Archive Network. Recent data from memory recognition studies are used to illustrate these functions.

Evaluating signal detection models of perceptual decision confidence

Event Abstract Back to Event Evaluating signal detection models of perceptual decision confidence Introduction In making perceptual decisions, humans and animals are able to report the level of confidence associated with the decision (1). In this work we investigate the mechanism underlying this confidence reporting procedure. In particular, is all the information used for perceptual decision making available to confidence reporting mechanisms? Some models of perception suggest that there are multiple channels of information, and subjective reports such as confidence ratings can only tap into one of the channels (e.g. cortical, but not subcortical channels). Is this popular view correct? We capitalize on an original psychophysical finding (2) that subjective reports of perceptual confidence and perceptual performance (d’) can dissociate, and apply formal model comparison techniques to identify the mechanism underlying confidence reporting. Methods and Results Signal detection theory (SDT) can be extended to characterize sensitivity and response bias in the type 2 task, i.e. the task of discriminating between one’s own correct and incorrect perceptual judgments using confidence ratings (3). We considered several such SDT models, including: 1: A simple SDT model where type 2 decisions are made by setting criteria on a transformation of the primary decision axis; 2: A decision noise model where type 2 sensitivity is affected by variation in type 2 criterion setting; 3: A late noise model where the noisy perceptual signal becomes even noisier when making confidence judgments; 4: A two-channel model where only one channel contributes to confidence judgments. Psychophysical data from the metacontrast masking paradigm dissociates perceptual performance from confidence in a manner that cannot plausibly be accounted for by differences in type 2 criterion setting (2). As such, this data presents a stringent test for models of type 2 performance. We compared models by evaluating the likelihood of each model, given the metacontrast masking data, using the Akaike information criterion. All models could account for perceptual performance, but only the late noise model provided a close fit to the observed performance-confidence dissociation. Discussion Our results suggest that traditional SDT models are not adequate to model type 2 performance, because they do not provide a process that allows for the kind of performance-confidence dissociation observed in the metacontrast masking paradigm. However, this extra process need not be an extra information processing channel. Our best-fitting model was a hierarchical, single-channel model where noisy perceptual signals accrued further noise when used for rating confidence. This suggests that confidence decisions may be made by mechanisms downstream from perceptual decision mechanisms.

The Complexity of Signal Detection in Air Traffic Control Alert Situations

Air traffic controllers continually monitor the traffic situation in their sectors and take action when they detect potentially hazardous situations. Automation systems simultaneously and independently monitor the situation and provide alerts when the situation meets defined criteria. The decisions made by the controllers and the automation systems may agree or disagree. Signal Detection Theory (SDT) provides a theoretical framework for understanding how controllers and automation systems make these decisions. However, traditional SDT provides an incomplete explanation of decision-making in the real-world ATC situations. In this paper, we examine instances where controllers take actions independently of the alert and where controllers take actions in response to an alert, but delay their actions until more information is available. Results from this study are applicable to other domains where operators are tasked to monitor situations while simultaneously monitoring the output of an alerting system.

Application of Fuzzy Signal Detection Theory to the Discrimination of Morphed Tank Images

The effect of response set size on performance on a detection task was evaluated using both fuzzy and traditional signal detection theory. Fuzzy categories of stimuli were created using morphing software to blend profile images of American (M1A1) and Iraqi (T55) tanks to different degrees. These combinations were used to create static images varying from 100% T55 to 0% T55 (100% MIA1). Participants were asked to indicate the degree to which each image did not resemble an American tank. Consistent with previous research, results indicated that the FSDT model conforms to the normality assumption of traditional SDT. In addition, forcing observers to make binary decisions impaired performance relative to multi-category response sets in the FSDT analysis but not the traditional analysis. However, there were more model convergence failures in the FSDT analysis relative to the traditional analysis, mostly associated with conditions in which there were 100 response categories.

A More Parsimonious Approach to Estimating Signal Detection Theory Measures

Signal Detection Theory (SDT) has proven to be an effective tool for analyzing perceptual and decision-making performance in many different areas. The main contribution of SDT is its capability to separate the discrimination process from the decision process by distinguishing between measures of sensitivity and measures of response bias. However, the usefulness of the traditional SDT measures of sensitivity, d', and response bias, c, is questionable in applied settings where observers do not make decisions based on an underlying psychophysical continuum. The purpose of our research was to derive alternative measures of sensitivity and response bias, overcoming the limitations of the traditional SDT model. Results from a Monte Carlo simulation showed support for our more parsimonious model.

Comparison of Fuzzy Signal Detection and Traditional Signal Detection Theory: Analysis of Duration Discrimination of Brief Light Flashes

Both Traditional and Fuzzy Signal Detection Theory (SDT) methods of analyses were compared using a duration discrimination task of brief light flashes. Both difficulty and response bias (by instruction) were manipulated. The two methods clearly provided different estimates of SDT parameters. The Fuzzy SDT analyses inflated performance while the traditional SDT analyses underestimated performance. In some cases performance on the ‘easier’ condition was worse than on the more ‘difficult’ condition. This may be due to differences in the range of durations discriminated, which was narrower in the ‘easier’ condition. Range of stimuli used can impact performance, although it or maybe an artifact of the procedure for the crisp analysis. Fuzzy SDT met the assumptions of traditional SDT, the assumption of normally distributed noise and signal plus noise distribution holds. In addition, the equal variance assumption holds for the more difficult discrimination for all participants.

Comparison of Fuzzy Signal Detection and Traditional Signal Detection Theory: Approaches to Performance Measurement

A recent advance upon Signal Detection Theory (SDT) promises to enhance measurement of performance in complex real world domains where stimuli do not fall into discrete, mutually exclusive categories. This development, Fuzzy Signal Detection Theory (FSDT) combines traditional SDT with Fuzzy Set Theory (FST) to extend signal detection analysis beyond the traditional crisp, categorical decision making model. FSDT allows for events to simultaneously be in more than one state category (e.g. signal and non-signal), so that stimulus and response dimensions can be continuous rather than categorical. This study compared the differences in methods of analyses from FSDT and traditional SDT using the same data set. Data suggests that FSDT analysis and traditional SDT provide different vistas into signal detection performance. FSDT provided a better description of the effects of stimulus uncertainty on observers' response bias and sensitivity. This is because the FSDT model explicitly captures this uncertainty and can provide insight into system performance in domains in which stimulus categories vary along a continuum.

Application of Fuzzy Signal Detection Theory to Vigilance: The Effect of Criterion Shifts

A recent advance on Signal Detection Theory (SDT) promises to enhance measurement of performance in complex real world domains. This development, Fuzzy Signal Detection Theory (FSDT), combines traditional SDT with Fuzzy Set Theory to extend signal detection analysis beyond the traditional crisp, categorical model. FSDT permits events to simultaneously be in more than one state category (e.g. signal and non-signal), so that the stimulus and response dimensions can be continuous rather than categorical. Consequently, FSDT can be employed in settings where the degree to which an event is a signal for detection may vary. This study is an initial test of application of FSDT to vigilance, a domain in which SDT has been widely applied. Results indicate that manipulations of stimulus probability impacts response bias in a fuzzy vigilance task, but that these effects differ somewhat from tasks employing traditional signal detection.

Equivalence of Julesz Ensembles and FRAME Models

In the past thirty years, research on textures has been pursued along two different lines. The first line of research, pioneered by Julesz (1962, IRE Transactions of Information Theory, IT-8:84–92), seeks essential ingredients in terms of features and statistics in human texture perception. This leads us to a mathematical definition of textures in terms of Julesz ensembles (Zhu et al., IEEE Trans. on PAMI, Vol. 22, No. 6, 2000). A Julesz ensemble is a set of images that share the same value of some basic feature statistics. Images in the Julesz ensemble are defined on a large image lattice (a mathematical idealization being Z2) so that exact constraint on feature statistics makes sense. The second line of research studies Markov random field (MRF) models that characterize texture patterns on finite (or small) image lattice in a statistical way. This leads us to a general class of MRF models called FRAME (Filter, Random field, And Maximum Entropy) (Zhu et al., Neural Computation, 9:1627–1660). In this article, we bridge the two lines of research by the fundamental principle of equivalence of ensembles in statistical mechanics (Gibbs, 1902, Elementary Principles of Statistical Mechanics. Yale University Press). We show that 1). As the size of the image lattice goes to infinity, a FRAME model concentrates its probability mass uniformly on a corresponding Julesz ensemble. Therefore, the Julesz ensemble characterizes the global statistical property of the FRAME models 2). For a large image randomly sampled from a Julesz ensemble, any local patch of the image given its environment follows the conditional distribution specified by a corresponding FRAME model. Therefore, the FRAME model describes the local statistical property of the Julesz ensemble, and is an inevitable texture model on finite (or small) lattice if texture perception is decided by feature statistics. The key to derive these results is the large deviation estimate of the volume of (or the number of images in) the Julesz ensemble, which we call the entropy function. Studying the equivalence of ensembles provides deep insights into questions such as the origin of MRF models, typical images of statistical models, and error rates in various texture related vision tasks (Yuille and Coughlan, IEEE Trans. on PAMI, Vol. 2, No. 2, 2000). The second thrust of this paper is to study texture distance based on the texture models of both small and large lattice systems. We attempt to explain the asymmetry phenomenon observed in texture “pop-out” experiments by the asymmetry of Kullback-Leibler divergence. Our results generalize the traditional signal detection theory (Green and Swets, 1988, Signal Detection Theory and Psychophysics, Peninsula Publishing) for distance measures from iid cases to random fields. Our theories are verified by two groups of computer simulation experiments.

Analysis of the performance of a model-based optimal auditory signal processor.

Traditionally, psychophysical data have been predicted either by constructing models of the peripheral auditory system or by applying signal detection theory (SDT). Frequently, the theoretical detection performance predicted by SDT is greater than that observed experimentally and a nonphysiologically based "internal noise" source is often added to the system to compensate for the discrepancy. A more appropriate explanation may be that traditional SDT approaches either incorporate little or no physiology or make simplifying assumptions regarding the density functions describing the physiological data. In the work presented here, an integrated approach, which combines SDT and a physiologically based model of the human auditory system, is proposed as an alternate method of quantifying detection performance. To validate this approach, the predicted detection performance for a simultaneous masking task is compared to predictions obtained from traditional methods and to experimental data. Additionally, the sensitivity of the integrated method is thoroughly investigated. The results suggest that by combining SDT with a physiologically based auditory model, thereby capitalizing on the strengths of each individual method, the previously observed discrepancies can be partially explained as the result of physical processes inherent in the auditory system rather than unspecified "internal noise" and more accurate predictions of psychophysical behavior can be obtained.

Signal detection by human observers: a cutoff reinforcement learning model of categorization decisions under uncertainty.

Previous experimental examinations of binary categorization decisions have documented robust behavioral regularities that cannot be predicted by signal detection theory (D.M. Green & J.A. Swets, 1966/1988). The present article reviews the known regularities and demonstrates that they can be accounted for by a minimal modification of signal detection theory: the replacement of the "ideal observer" cutoff placement rule with a cutoff reinforcement learning rule. This modification is derived from a cognitive game theoretic analysis (A.E. Roth & I. Erev, 1995). The modified model reproduces all 19 experimental regularities that have been considered. In all cases,it outperforms the original explanations. Some of these previous explanations are based on important concepts such as conservatism, probability matching, and "the gambler's fallacy" that receive new meanings given the current results. Implications for decision-making research and for applications of traditional signal detection theory are discussed.

Testing models of decision making using confidence ratings in classification.

Classification implies decision making (or response selection) of some kind. Studying the decision process using a traditional signal detection theory analysis is difficult for two reasons: (a) The model makes a strong assumption about the encoding process (normal noise), and (b) the two most popular decision models, optimal and distance-from-criterion models, can mimic each other's predictions about performance level. In this article, the authors show that by analyzing certain distributional properties of confidence ratings, a researcher can determine whether the decision process is optimal, without knowing the form of the encoding distributions. Empirical results are reported for three types of experiments: recognition memory, perceptual discrimination, and perceptual categorization. In each case, the data strongly favored the distance-from-criterion model over the optimal model.