Use Of Signal Detection Theory Research Articles

How do people build up visual memory representations from sensory evidence? Revisiting two classic models of choice

In many decision tasks, we have a set of alternative choices and are faced with the problem of how to use our latent beliefs and preferences about each alternative to make a single choice. Cognitive and decision models typically presume that beliefs and preferences are distilled to a scalar latent strength for each alternative, but it is also critical to model how people use these latent strengths to choose a single alternative. Most models follow one of two traditions to establish this link. Modern psychophysics and memory researchers make use of signal detection theory, assuming that latent strengths are perturbed by noise, and the highest resulting signal is selected. By contrast, many modern decision theoretic modeling and machine learning approaches use the softmax function (which is based on Luce’s choice axiom; Luce, 1959) to give some weight to non-maximal-strength alternatives. Despite the prominence of these two theories of choice, current approaches rarely address the connection between them, and the choice of one or the other appears more motivated by the tradition in the relevant literature than by theoretical or empirical reasons to prefer one theory to the other. The goal of the current work is to revisit this topic by elucidating which of these two models provides a better characterization of latent processes in m-alternative decision tasks, with a particular focus on memory tasks. In a set of visual memory experiments, we show that, within the same experimental design, the softmax parameter β varies across m-alternatives, whereas the parameter d′ of the signal-detection model is stable. Together, our findings indicate that replacing softmax with signal-detection link models would yield more generalizable predictions across changes in task structure. More ambitiously, the invariance of signal detection model parameters across different tasks suggests that the parametric assumptions of these models may be more than just a mathematical convenience, but reflect something real about human decision-making.

Interindividual variations in associative visual learning: Exploration, description, and partition of response characteristics.

Relying on existing literature to identify suitable techniques for characterizing individual differences presents practical and methodological challenges. These challenges include the frequent absence of detailed descriptions of raw data, which hinders the assessment of analysis appropriateness, as well as the exclusion of data points deemed outliers, or the reliance on comparing only extreme groups by categorizing continuous variables into upper and lower quartiles. Despite the availability of algorithmic modeling in standard statistical software, investigations into individual differences predominantly focus on factor analysis and parametric tests. To address these limitations, this application-oriented study proposes a comprehensive approach that leverages behavioral responses through the use of signal detection theory and clustering techniques. Unlike conventional methods, signal detection theory considers both sensitivity and bias, offering insights into the intricate interplay between perceptual ability and decision-making processes. On the other hand, clustering techniques enable the identification and classification of distinct patterns within the dataset, allowing for the detection of singular behaviors that form the foundation of individual differences. In a broader framework, these combined approaches prove particularly advantageous when analyzing large and heterogeneous datasets provided by data archive platforms. By applying these techniques more widely, our understanding of the cognitive and behavioral processes underlying learning can be expedited and enhanced.

Preclinical Evaluation of Attention and Impulsivity Relevant to Determining ADHD Mechanisms and Treatments.

People with Attention-Deficit Hyperactivity Disorder (ADHD) exhibit inattention, hyperactivity, and/or impulsivity. Symptoms of ADHD emerge in childhood and can continue throughout adulthood. Clinical assessments to diagnose ADHD can include administration of continuous performance tests (CPTs). CPTs provide an objective measure of inattention, requiring individuals to respond to targets (attention), and inhibit response to non-targets (impulsivity). When investigating the mechanisms of, and novel treatments for, ADHD it is important to measure such behavioral domains (attention and impulsivity). Some well-established preclinical tasks purport to assess attention in rodents but, unlike CPTs, do not require non-target inhibition, limiting their ADHD-relevance.Recently developed tasks recreate CPTs for rodents. The 5-Choice CPT (5C-CPT) contains non-target stimuli, enabling use of signal detection theory to evaluate performance, consistent with CPTs. The 5C-CPT has been adapted for use in humans, enabling direct cross-species comparisons of performance. A newer task, the rodent CPT (rCPT), is a touchscreen-based analog of CPTs, utilizing symbols instead of a simple stimulus array. Currently, the rCPT may be more akin to a go/no-go task, equally presenting targets/non-targets, although numerous variants exist - a strength. The 5C-CPT and rCPT emulate human CPTs and provide the most up-to-date information on ADHD-relevant studies for understanding attention/impulsivity.

Eyewitness Identification in Its Social Context

Much of the literature on eyewitness identification neglects the social context in which identifications are made. As the number of cognitive psychologists conducting eyewitness research increased so did the use of signal detection theory and ROC analyses. With the resulting need for larger sample size, researchers moved toward conducting studies on internet platforms that allow for crowd-sourcing research participants. These methods make it next to impossible to ask research questions that explore the ways in which social interactions influence the identifications made by witnesses. Yet, the possibility of social context effects on witness memory are prevalent in applied contexts and research supports their existence. In addition, some eyewitness identifications may not be governed by memory at all. We argue that a consideration of social context effects is required to fully explore the reliability of witness identifications and propose a number of avenues for future research.

Alleviating the concerns with the SDT approach to reasoning: reply to Singmann and Kellen (2014).

Alleviating the concerns with the SDT approach to reasoning: reply to Singmann and Kellen (2014).

Evaluating Probabilistic Forecasts with Bayesian Signal Detection Models

We propose the use of signal detection theory (SDT) to evaluate the performance of both probabilistic forecasting systems and individual forecasters. The main advantage of SDT is that it provides a principled way to distinguish the response from system diagnosticity, which is defined as the ability to distinguish events that occur from those that do not. There are two challenges in applying SDT to probabilistic forecasts. First, the SDT model must handle judged probabilities rather than the conventional binary decisions. Second, the model must be able to operate in the presence of sparse data generated within the context of human forecasting systems. Our approach is to specify a model of how individual forecasts are generated from underlying representations and use Bayesian inference to estimate the underlying latent parameters. Given our estimate of the underlying representations, features of the classic SDT model, such as the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC), follow immediately. We show how our approach allows ROC curves and AUCs to be applied to individuals within a group of forecasters, estimated as a function of time, and extended to measure differences in forecastability across different domains. Among the advantages of this method is that it depends only on the ordinal properties of the probabilistic forecasts. We conclude with a brief discussion of how this approach might facilitate decision making.

Performance assessment of a 2D array of plastic scintillation detectors for IMRT quality assurance

The purposes of this work are to assess the performance of a 2D plastic scintillation detectors array prototype for quality assurance in intensity-modulated radiation therapy (IMRT) and to determine its sensitivity and specificity to positioning errors of one multileaf collimator (MLC) leaf and one MLC leaf bank by applying the principles of signal detection theory. Ten treatment plans (step-and-shoot delivery) and one volumetric modulated arc therapy plan were measured and compared to calculations from two treatment-planning systems (TPSs) and to radiochromic films. The averages gamma passing rates per beam found for the step-and-shoot plans were 95.8% for the criteria (3%, 2 mm), 97.8% for the criteria (4%, 2 mm), and 98.1% for the criteria (3%, 3 mm) when measurements were compared to TPS calculations. The receiver operating characteristic curves for the one leaf errors and one leaf bank errors were determined from simulations (theoretical upper limits) and measurements. This work concludes that arrays of plastic scintillation detectors could be used for IMRT quality assurance in clinics. The use of signal detection theory could improve the quality of dosimetric verifications in radiation therapy by providing optimal discrimination criteria for the detection of different classes of errors.

Signal detection in animal psychoacoustics: Analysis and simulation of sensory and decision-related influences

Signal detection theory (SDT) provides a framework for interpreting psychophysical experiments, separating the putative internal sensory representation and the decision process. SDT was used to analyse ferret behavioural responses in a (yes–no) tone-in-noise detection task. Instead of measuring the receiver-operating characteristic (ROC), we tested SDT by comparing responses collected using two common psychophysical data collection methods. These (Constant Stimuli, Limits) differ in the set of signal levels presented within and across behavioural sessions. The results support the use of SDT as a method of analysis: SDT sensory component was unchanged between the two methods, even though decisions depended on the stimuli presented within a behavioural session. Decision criterion varied trial-by-trial: a ‘yes’ response was more likely after a correct rejection trial than a hit trial. Simulation using an SDT model with several decision components reproduced the experimental observations accurately, leaving only ∼10% of the variance unaccounted for. The model also showed that trial-by-trial dependencies were unlikely to influence measured psychometric functions or thresholds. An additional model component suggested that inattention did not contribute substantially. Further analysis showed that ferrets were changing their decision criteria, almost optimally, to maximise the reward obtained in a session. The data suggest trial-by-trial reward-driven optimization of the decision process. Understanding the factors determining behavioural responses is important for correlating neural activity and behaviour. SDT provides a good account of animal psychoacoustics, and can be validated using standard psychophysical methods and computer simulations, without recourse to ROC measurements.

Observer detection and discrimination performance as a function of clutter: a signal detection approach

In this paper we investigate the use of signal detection theory (SDT) in predicting target detection and discrimination in disorganized clutter. Two normal observers performed monocular visual search experiments at 25 cm, in the dark. They detected Gabor gratings on an achromatic background cluttered with 2000 or 500 random dots. The targets were displayed at pseudorandom locations from 0–20° and 20–47°, by method of constant stimuli. A contrast-based detection and orientation-based discrimination task was completed in a yes/no or 2-alternative-forced-choice (2AFC) task. The hit rate, false alarm rate, detectability, criterion and bias were analysed. The psychometric function indicated low detection and discrimination thresholds in low clutter that increased in high clutter. Increased clutter showed high hit rates and a false alarm rate that increased with low detectability and liberal criterion. In the detection task, low clutter showed high hit rates and low false alarm rates in the central field. Therefore, SDT proves useful to predict observer performance in visual scenes with disorganized clutter.

Application of signal detection theory to verbal memory testing to distinguish patients with psychogenic nonepileptic seizures from patients with epileptic seizures

Distinguishing psychogenic nonepileptic seizures (PNES) from epileptic seizures (ES) is a difficult task that is often aided by neuropsychological evaluation. In the present study, signal detection theory (SDT) was used to examine differences between these groups in neuropsychological performance on the Wechsler Memory Scale, Third Edition, Word List Test (WMS-III WLT). The raw WMS-III WLT scores on this task failed to discriminate the two groups; however, with the use of SDT, patients with PNES were found to have a negative response bias and increased memory sensitivity as compared with patients with ES. When patients with left (LTLE) and right (RTLE) temporal lobe epilepsy were compared, the patients with LTLE demonstrated decreased memory sensitivity but a similar response bias as compared with the patients with RTLE. Memory impairment in patients with PNES may be related to faulty decision-making strategies, rather than true memory impairment, whereas memory performance differences between the LTLE and RTLE groups are likely related to actual differences in memory abilities.

Getting technical about awareness

It has recently been argued that post-decision wagering provides an objective measure of awareness. We critically evaluate this claim, emphasizing the distinction between performance without awareness and a reluctance to gamble in full awareness of weak sensory evidence. We address two key methodological issues. The first is the design of the pay-off matrix to reward a strategy of wagering that reflects the strength of sensory evidence. The second is the use of signal detection theory to analyze the resulting data. We argue that proper treatment of these issues is essential if post-decision wagering is to prove valuable in validating claims of perception without awareness in normal subjects and neuropsychological patients.

How high should total pain-relief score be to obviate the need for analgesic remedication in acute pain? Estimation using signal detection theory and individual-patient meta-analysis

A pain relief score 50% of the maximum is often used as a clinically meaningful outcome in meta-analyses of analgesic trials. This arbitrary value requires validation. To determine the optimum pain relief score for predicting pain relief sufficient to obviate the need for analgesic remedication in acute post-surgical pain. Individual-patient meta-analysis of randomized controlled trials and use of signal detection theory to identify the optimum cut-off point on the total pain relief score (TOTPAR). Analgesic remedication was used as the clinical outcome. Seven parallel-group, active and placebo-controlled trials of minor analgesics. The predictive value of the TOTPAR score [expressed as a percentage of the maximum score (%maxTOTPAR)] for remedication was excellent for all the trials. The pooled estimate of the area under the receiver operating curve, an index of discriminative power, was outstanding 0.96 (95% CI 0.95-0.97). The pooled estimate of the optimal %maxTOTPAR for predictive purposes was 44.11 (95% CI 44.00-44.23). The analysis suggests that the arbitrary 50% cut-off point TOTPAR score often used in meta-analyses of analgesic trials in dental pain is reasonably acceptable. This is different to the 33% cut-off point reported for analgesic trials of acute breakthrough cancer pain and some chronic pain states such as diabetic neuropathy and postherpetic neuralgia. These differences deserve careful consideration when reading reports of analgesic trials and meta-analyses. Remedication itself should be considered as the preferred outcome measure for analgesic trials.

A signal detection model applied to the stimulus: Understanding covariances in face recognition experiments in the context of face sampling distributions

We provide a description and interpretation of signal detection theory as applied to the analysis of an individual stimulus in a recognition experiment. Despite the common use of signal detection theory in this context, especially in the face recognition literature, the assumptions of the model have rarely been made explicit. In a series of simulations, we first varied the stability of d ' and C in face sampling distributions and report the pattern of correlations between the hit and false alarm rate components of the model across the simulated experiments. These kinds of correlation measures have been reported in recent face recognition papers and have been considered to be theoretically important. The simulation data we report revealed widely different correlation expectations as a function of the parameters of the face sampling distribution, making claims of theoretical importance for any particular correlation questionable. Next, we report simulations aimed at exploring the effects of face sampling distribution parameters on correlations between individual components of the signal detection model (i.e. hit and false alarm rates), and other facial measures such as typicality ratings. These data indicated that valid interpretations of such correlations need to make reference to the parameters of the relevant face sampling distribution.

Neural network models of categorical perception.

Studies of the categorical perception (CP) of sensory continua have a long and rich history in psychophysics. In 1977, Macmillan, Kaplan, and Creelman introduced the use of signal detection theory to CP studies. Anderson and colleagues simultaneously proposed the first neural model for CP, yet this line of research has been less well explored. In this paper, we assess the ability of neural-network models of CP to predict the psychophysical performance of real observers with speech sounds and artificial/novel stimuli. We show that a variety of neural mechanisms are capable of generating the characteristics of CP. Hence, CP may not be a special model of perception but an emergent property of any sufficiently powerful general learning system.

Visual signal detection in structured backgrounds. I. Effect of number of possible spatial locations and signal contrast.

Several studies have investigated the effect of signal location uncertainty on the detectability of simple visual signals in uncorrelated Gaussian noise with a deterministic background. For this case, human performance in locating a signal in a forced-choice experiment has been successfully predicted for 2-1800 alternative locations with the use of signal detection theory and the usual assumption that the observer's internal response is Gaussian distributed. Gaussian uncorrelated noise is far from realistic medical image noise, which includes not only fluctuations in intensity of quantum origin but also other anatomical objects lying in the x-ray path (structured backgrounds). Our goal is to determine whether signal detection theory with the Gaussian assumption is adequate for the case of structured backgrounds, or whether other more complex models need to be developed to predict human performance as a function of the number of possible signal locations in structured backgrounds. We present experimental data suggesting that an assumed Gaussian internal response accurately predicts the decrease in observer performance as the number of alternative locations is increased. The one exception is a lower-than-predicted performance for the detection of low-contrast signals for two alternative locations. Performance as measured by the index of detectability d' is also found to be linear with signal contrast. Together these findings extend the applicability of signal detection theory with Gaussian internal response functions to the case of complex structured backgrounds.

Analysis of cognitive function in animals, the value of SDT

The search for a better understanding of cognitive decline in man has lead to the use of increasingly complex procedures in animal research. The analysis of the data generated in such experiments has been greatly facilitated by the wider use of computer assisted techniques. These techniques can only be as good as the hypotheses they are used to test. Signal detection theory (SDT) provides a rational framework within which to work. The procedures are derived from human cognitive neuropsychology and are already used to some extent in primate but to a lesser degree in rodent research. The use of SDT offers two main advantages: first, a testable hypothesis as to the manner in which competing processes arrive at choice between various courses of action; second, the statistical procedures offer clear advantages over more traditional approaches by reducing the chances of misinterpretation. Though relatively easy to apply some care must be exercised in the protocol design and the choice of SDT indices if the full value of the approach is to be achieved. If experimental designs can be developed to include the appropriate use of SDT analysis; both the power of such protocols, and their value in the understanding of cognitive function, will provide a major step forward for animal-based research.

Incomplete neutralization and pragmatics in German

Earlier studies suggest that the neutralization of the voicing contrast in word pairs like Bund and Bunt is incomplete, but further research is required to evaluate the competing interpretations. Subjects here were given a range of different speaking tasks: reading continuous text, repeating spoken sentences (in circumstances where the concern of the experiment was carefully hidden), dictating the potential homophones to a German-speaking writer and, finally, reading from a word list. Discriminant analysis was used to combine the five spectro-temporal variables measured from sound spectrograms of these productions to categorize the tokens as voiced or voiceless in each condition. Correct categorization for discriminant analysis varied between 55% and 78% depending on the communicative task but was significant in all conditions. The data show that speakers can control the degree of neutralization depending on pragmatics and that information about the underlying contrast is distributed over much of the word. In Experiment 2, recorded productions from some conditions of Experiment 1 were played for a group of listeners. Through use of signal detection theory (and the statistic d ′), it is shown that listeners discriminated the intended word with accuracy very similar to that of discriminant analysis. They even tended to make errors on the same tokens. Apparently, the variables we measured capture information that is roughly equivalent to that employed by native listeners. The absence of complete neutralization implies that the German syllable-final devoicing rule cannot be stated in terms of the same [− voice] feature that is employed in the lexical specification of words. Instead, syllable-final devoicing is an effect that resembles implementation rules (since it is graded) and operates directly upon a syllable-like representation.

Quantification of the Müller-Lyer illusion using signal detection theory.

An experiment was done to investigate the use of signal-detection theory in quantifying the magnitude of the Müller-Lyer Illusion. The results showed a d' of 2.5 for the illusion. Some implications of this approach are discussed.

A non‐parametric method for fitting ROC curves for evaluation of industrial inspection performance

AbstractThe use of signal detection theory (SDT) in evaluation of the performance of an inspector on certain tasks requires the identification of the receiver operating characteristic (ROC) curve of the inspector. In this paper a functional form, Y = f(X), for the ROC equation is proposed, and two alternative approaches are developed for estimation of its parameters. This method does not require prior assumptions on the statistical distribution of the noise and signal‐plus‐noise. The application of the method is illustrated by the use of an example data set.

The Use of Signal Detection Theory in Research on Human-Computer Interaction

The use of signal detection theory (SDT) in evaluating attentional and decision-making performance in the context of human-computer interaction is reviewed. SDT provides a means for distinguishing between accuracy and criterion setting in decision-making environments. This is useful in evaluating the effectiveness of the decision-making performance of an intelligent machine, a human user, or a human-machine system. SDT is also useful in deciding how best to allocate subtasks and functions in human-computer monitoring systems. Applications of SDT to the evaluation of rule bases in expert systems and the design of computer assistance in human-computer monitoring are discussed.