Signal Detection Theory Model Research Articles

Everyday amnesia: Residual memory for high confidence misses and implications for decision models of recognition.

Despite studying a list of items only minutes earlier, when reencountered in a recognition memory test, undergraduate participants often say with total confidence that they have not studied some of the items before. Such high confidence miss (HCM) responses have been taken as evidence of rapid and complete forgetting and of everyday amnesia (Roediger & Tekin, 2020). We investigated (a) if memory for HCMs is completely lost or whether a residual memory effect exists and (b) whether dominant decision models predict the effect. Participants studied faces (Experiments 1a, 2, and 3) or words (Experiment 1b), then completed a single-item recognition memory task, followed by either (a) a two-alternative forced-choice recognition task, in which the studied and nonstudied alternatives on each trial were matched for their previous old/new decision and confidence rating (Experiments 1 and 2) or (b) a second single-item recognition task in which the targets and foils were HCMs and high confidence correct rejections, respectively (Experiment 3). In each experiment, participants reliably distinguished HCMs from high-confidence correct rejections. The unequal variance signal detection and dual-process signal detection models were fit to the single-item recognition data, and the parameter estimates were used to predict the memory effect for HCMs. The dual-process signal detection model predicted the residual memory effect (as did another popular model, the mixture signal detection theory model). However, the unequal variance signal detection model incorrectly predicted a negative, or no, effect, invalidating this model. The residual memory effect for HCMs demonstrates that everyday amnesia is not associated with complete memory loss and distinguishes between decision models. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Reward Sensitivity and Noise Contribute to Negative Affective Bias: A Learning Signal Detection Theory Approach in Decision-Making.

In patients with mood disorders, negative affective biases - systematically prioritising and interpreting information negatively - are common. A translational cognitive task testing this bias has shown that depressed patients have a reduced preference for a high reward under ambiguous decision-making conditions. The precise mechanisms underscoring this bias are, however, not yet understood. We therefore developed a set of measures to probe the underlying source of the behavioural bias by testing its relationship to a participant's reward sensitivity, value sensitivity and reward learning rate. One-hundred-forty-eight participants completed three online behavioural tasks: the original ambiguous-cue decision-making task probing negative affective bias, a probabilistic reward learning task probing reward sensitivity and reward learning rate, and a gambling task probing value sensitivity. We modelled the learning task through a dynamic signal detection theory model and the gambling task through an expectation-maximisation prospect theory model. Reward sensitivity from the probabilistic reward task (β = 0.131, p = 0.024) and setting noise from the probabilistic reward task (β = -0.187, p = 0.028) both predicted the affective bias score in a logistic regression. Increased negative affective bias, at least on this specific task, may therefore be driven in part by a combination of reduced sensitivity to rewards and more variable responses.

Benefits of multinomial processing tree models with discrete and continuous variables in memory research: an alternative modeling proposal to Juola et al. (2019).

Signal detection theory (SDT) and two-high threshold models (2HT) are often used to analyze accuracy data in recognition memory paradigms. However, when reaction times (RTs) and/or confidence levels (CLs) are also measured, they usually are analyzed separately or not at all as dependent variables (DVs). We propose a new approach to include these variables based on multinomial processing tree models for discrete and continuous variables (MPT-DC) with the aim to compare fits of SDT and 2HT models. Using Juola et al.'s (2019, Memory & Cognition, 47[4], 855-876) data we have found that including CLs and RTs reduces the standard errors of parameter estimates and accounts for interactions among accuracy, CLs, and RTs that classical versions of SDT and 2HT models do not. In addition, according to the simulations, there is an increase in the proportion of correct model selections when relevant DV are included. We highlight the methodological and substantive advantages of MPT-DC in the disentanglement of contributing processes in recognition memory.

Modelling decision-making biases.

Biases are a fundamental aspect of everyday life decision-making. A variety of modelling approaches have been suggested to capture decision-making biases. Statistical models are a means to describe the data, but the results are usually interpreted according to a verbal theory. This can lead to an ambiguous interpretation of the data. Mathematical cognitive models of decision-making outline the structure of the decision process with formal assumptions, providing advantages in terms of prediction, simulation, and interpretability compared to statistical models. We compare studies that used both signal detection theory and evidence accumulation models as models of decision-making biases, concluding that the latter provides a more comprehensive account of the decision-making phenomena by including response time behavior. We conclude by reviewing recent studies investigating attention and expectation biases with evidence accumulation models. Previous findings, reporting an exclusive influence of attention on the speed of evidence accumulation and prior probability on starting point, are challenged by novel results suggesting an additional effect of attention on non-decision time and prior probability on drift rate.

Three-Way ROCs for Forensic Decision Making

Firearm examiners use a comparison microscope to judge whether bullets or cartridge cases were fired by the same gun. Examiners can reach one of three possible conclusions: Identification (a match), Elimination (not a match), or Inconclusive. Numerous error rate studies report that firearm examiners commit few errors when they conduct these examinations. However, the studies also report many inconclusive judgments (> 50%), and how to score these responses is controversial. There have recently been three Signal Detection Theory (SDT) primers in this domain. Unfortunately, these analyses rely on hypothetical data and fail to address the inconclusive response issue adequately. This article reports an SDT analysis using data from a large error rate study of practicing firearm examiners. First, we demonstrate the problem of relying on the traditional two-way SDT model, which either drops or combines inconclusive responses; in addition to lacking ecological validity, this approach leads to implausible results. Second, we introduce readers to the three-way SDT model. We demonstrate this approach in the forensic firearms domain. While the three-way approach is statistically complicated, it is well suited to evaluate performance for any forensic domain in which three possible decision categories exist.

Incorporating Functional Response Time Effects into a Signal Detection Theory Model.

Signal detection theory (SDT; Tanner & Swets in Psychological Review 61:401-409, 1954) is a dominant modeling framework used for evaluating the accuracy of diagnostic systems that seek to distinguish signal from noise in psychology. Although the use of response time data in psychometric models has increased in recent years, the incorporation of response time data into SDT models remains a relatively underexplored approach to distinguishing signal from noise. Functional response time effects are hypothesized in SDT models, based on findings from other related psychometric models with response time data. In this study, an SDT model is extended to incorporate functional response time effects using smooth functions and to include all sources of variability in SDT model parameters across trials, participants, and items in the experimental data. The extended SDT model with smooth functions is formulated as a generalized linear mixed-effects model and implemented in the gamm4 R package. The extended model is illustrated using recognition memory data to understand how conversational language is remembered. Accuracy of parameter estimates and the importance of modeling variability in detecting the experimental condition effects and functional response time effects are shown in conditions similar to the empirical data set via a simulation study. In addition, the type 1 error rate of the test for a smooth function of response time is evaluated.

Predicting and Measuring Decision Rules for Social Recognition in a Neotropical Frog.

AbstractMany animals use signals to recognize familiar individuals but risk making mistakes because the signal properties of different individuals often overlap. Furthermore, outcomes of correct and incorrect decisions yield different fitness payoffs, and animals incur these payoffs at different frequencies depending on interaction rates. To understand how signal variation, payoffs, and interaction rates shape recognition decision rules, we studied male golden rocket frogs, which recognize the calls of territory neighbors and are less aggressive to neighbors than to strangers. We first quantified patterns of individual variation in call properties and predicted optimal discrimination thresholds using signal variation. We then measured thresholds for discriminating between neighbors and strangers using a habituation-discrimination field playback experiment. Territorial males discriminated between calls differing by 9%-12% in temporal properties, slightly higher than the predicted thresholds (5%-10%). Finally, we used a signal detection theory model to explore payoff and interaction rate parameters and found that the empirical threshold matched those predicted under ecologically realistic assumptions of infrequent encounters with strangers and relatively costly missed detections of strangers. We demonstrate that receivers group continuous variation in vocalizations into discrete social categories and that signal detection theory can be applied to understand evolved decision rules.

Probing doors to visual awareness: Choice set, visibility, and confidence

ABSTRACT Visibility and confidence are two subtly (sensation-based and intuition-based) different ways to assess our visual experiences. We investigated how different choice sets affect the ability of confidence and visibility judgments to retrieve conscious information from perceptual decision-making processes. Six participants made introspective judgments on sinusoidal gratings at close to chance signal-to-noise values. We used signal detection theory and mixed regression models to analyze choices and pupilometric recordings. The results suggest that allowing null responses (e.g., “I saw nothing”) asymmetrically affects the signal distribution variances associated with null and non-null responses; confidence is more effective than visibility in accessing unconscious information; and bias-free methods hinder accurate introspective judgment by forbidding free decision criteria. Our findings underscore the importance of the context afforded by choice and stimulus sets, which determines the extent to which elicited introspective judgments are based on the actual information from the past events they are intended to measure.

Many phish in the : A coexisting-choice-criteria model of security behavior.

Normative decision theory proves inadequate for modeling human responses to the social-engineering campaigns of advanced persistent threat (APT) attacks. Behavioral decision theory fares better, but still falls short of capturing social-engineering attack vectors which operate through emotions and peripheral-route persuasion. We introduce a generalized decision theory, under which any decision will be made according to one of multiple coexisting choice criteria. We denote the set of possible choice criteria by . Thus, the proposed model reduces to conventional Expected Utility theory when , while Dual-Process (thinking fast vs. thinking slow) decision making corresponds to a model with . We consider a more general case with , which necessitates careful consideration of how, for a particular choice-task instance, one criterion comes to prevail over others. We operationalize this with a probability distribution that is conditional upon traits of the decisionmaker as well as upon the context and the framing of choice options. Whereas existing signal detection theory (SDT) models of phishing detection commingle the different peripheral-route persuasion pathways, in the present descriptive generalization the different pathways are explicitly identified and represented. A number of implications follow immediately from this formulation, ranging from the conditional nature of security-breach risk to delineation of the prerequisites for valid tests of security training. Moreover, the model explains the "stepping-stone" penetration pattern of APT attacks, which has confounded modeling approaches based on normativerationality.

Sensor-based modeling of problem-solving in virtual reality manufacturing systems

Problem-solving is the process to achieve a goal when the solution path is uncertain. Recently, technological advancements have changed problems’ characteristics and their solutions in engineering fields. Strong problem-solving skills are essential to allow engineers to assess new problems and quickly implement solutions. Engineering problem-solving skills are first educated in schools and usually evaluated through written exams. However, high grades in exams do not represent sufficient problem-solving skills in real-world engineering problems. Decision making with insufficient problem-solving skills in real world may result in costly consequences. Therefore, it is imperative to evaluate and reinforce problem-solving skills of engineering students in real-world problems. With the rapid technological advancements, availability of virtual reality (VR) and eye-tracking facilitates the study of engineering problem-solving. The immersive environment created by VR enables students to better understand and solve real-world engineering problems. On the other hand, eye-tracking allows for studying fundamental cognitive processes during information processing. It is critical to integrate VR simulation with data-driven modeling of eye movements to evaluate and enhance engineering problem-solving skills. In this paper, we integrate sensing technology (i.e., eye-tracking) and virtual reality (VR) to model problem-solving in manufacturing systems. A novel data-driven model that integrates signal detection theory (SDT) with Conflict & Error (C&E) is developed to quantify engineering problem-solving skills. First, we simulate a manufacturing system in a VR game environment. Students are given an assembly problem to produce a car toy that satisfies some particular requirements in the VR manufacturing system while eye-tracking data are collected throughout the assembly process. Second, eye-tracking data are analyzed with a SDT model to quantify problem-solving skills. Third, a joint SDT-C&E model is developed to analyze eye-tracking data and benchmark with results generated from the SDT model. Experimental results show that the joint SDT-C&E model is more effective to quantify problem-solving skills of engineering students than the SDT model.

A window of subliminal perception

Under labels such as unconscious processing and subliminal perception, identification of stimuli falling below the subjective threshold (whether truly unconscious or not) has been found remarkably accurate in some experiments while completely at chance in others. Here, we first identify that an apparent window of subliminal perception arises in humans under specific stimulus conditions using different experimental paradigms and analysis methods. We then show that the standard signal detection theory (SDT) model is unable to account for this window and extend it until it is. We finally compare a range of models on empirical data. The models performing best are notable for their absence of hierarchical levels, indicating that the window could be a base property of any phenomenally conscious system. The models explain previously incompatible findings in the literature, and they allow for estimations of peaks in subthreshold perception across the spectrum of stimulus saliency, which may be used in further studies of subliminal perception.

Monotonicity of rank order probabilities in signal detection models of simultaneous detection and identification

We examine different models of recognition memory in a simultaneous detection and identification task, which features multiple simultaneously presented test stimuli. A common finding from eyewitness identification research investigating such tasks is that the more confident decision makers are about detecting the presence of a target, the higher the probability that they also correctly identify it. We demonstrate that for members of the signal detection theory (SDT) model framework, predicting such a relationship is — contrary to previous assertions — not entailed by a monotonic diagnosticity ratio. Instead, it can be shown that this prediction follows if latent memory signals’ rank order probabilities exhibit monotonicity under changes in the response criterion. For a selection of common SDT models, we prove that this monotonicity property holds in situations in which two test stimuli are presented simultaneously. Threshold models such as the two-high-threshold model (2HTM), however, do not necessarily possess this feature. Leveraging this fact, we show that in the presence of lures which resemble a target, the 2HTM is unable to make the same predictions as many reasonable SDT models with monotonic rank order probabilities. This enables us to construct a critical, distribution-free test between these models. An empirical investigation implementing this test reveals a clear failure of the 2HTM to account for the qualitative response patterns, which are consistent with the predictions of SDT models with monotonic rank order probabilities.

Testing the foundations of signal detection theory in recognition memory.

Signal detection theory (SDT) plays a central role in the characterization of human judgments in a wide range of domains, most prominently in recognition memory. But despite its success, many of its fundamental properties are often misunderstood, especially when it comes to its testability. The present work examines five main properties that are characteristic of existing SDT models of recognition memory: (a) random-scale representation, (b) latent-variable independence, (c) likelihood-ratio monotonicity, (d) ROC function asymmetry, and (e) nonthreshold representation. In each case, we establish testable consequences and test them against data collected in the appropriately designed recognition-memory experiment. We also discuss the connection between yes-no, forced-choice, and ranking judgments. This connection introduces additional behavioral constraints and yields an alternative method of reconstructing yes-no ROC functions. Overall, the reported results provide a strong empirical foundation for SDT modeling in recognition memory. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

On Joining a Signal Detection Choice Model with Response Time Models

AbstractIn a signal detection theory (SDT) approach to multiple choice exams, examinees are viewed as choosing, for each item, the alternative that is perceived as being the most plausible, with perceived plausibility depending in part on whether or not an item is known. The SDT model is a process model and provides measures of item difficulty, item discrimination, and the relative plausibility of each alternative. It is shown how to incorporate information from response times into the model, which has potential benefits for estimation and also offers a way to study underlying processes. The SDT model is joined with a lognormal response time (RT) model in a manner similar to that used in hierarchical models. In addition, a mixture extension of the RT model is joined in a novel way with the SDT choice model, using the idea that the probability of “knowing” in the SDT model might be related to the probability of working in one of two speed states in the mixture RT model. A semiparametric version of the mixture RT model is also used to assess robustness. The fused SDT/RT models are examined with science items from the 2015 Program for International Student Assessment.

Causal role for the primate superior colliculus in the computation of evidence for perceptual decisions.

Trained monkeys performed a two-choice perceptual decision-making task in which they reported the perceived orientation of a dynamic Glass pattern, before and after unilateral, reversible, inactivation of a brainstem area involved in preparing eye movements, the superior colliculus (SC). Surprisingly, we found that unilateral SC inactivation produced significant decision biases and changes in reaction times consistent with a causal role for the primate SC in perceptual decision-making. Fitting signal detection theory and sequential sampling models to the data revealed that SC inactivation produced a decrease in the relative evidence for contralateral decisions, as if adding a constant offset to a time-varying evidence signal for the ipsilateral choice. The results provide causal evidence for an embodied cognition model of perceptual decision-making and provide compelling evidence that the SC of primates, a brainstem structure, plays a causal role in how evidence is computed for decisions, a process usually attributed to the forebrain.

Individual differences in the perception of cue-outcome contingencies: A signal detection analysis

In a signal detection theory (SDT) approach to associative learning, one assumes that, when a subject is exposed to a flow of stimuli, an association is created between the internal representations of a cue and of an outcome, allowing the representation of the cue to activate the representation of the outcome. The outcome activation is a random variable drawn from a Gaussian distribution with mean m (sensitivity to the contingency) and standard deviation d (variability in outcome activation). Depending on whether the outcome activation is above or below various decision thresholds, the participant perceives either a negative, a null, or a positive contingency between the cue and the outcome. This study presents a detailed SDT analysis of the performance of four participants on whom data in a contingency assessment task were collected almost daily during several months. Parameters from the SDT model proved relatively stable over time, except if feedback was provided to the subject. In that case, for some participants but not all, the sensitivity increased. The decision criteria were also affected. Some of these changes endured despite the discontinuation of feedback. The variability in outcome activation was not affected by the feedback.

Coping with Danger and Deception: Lessons from Signal Detection Theory.

AbstractSignal detection theory (SDT) has been used to model optimal stimulus discrimination for more than four decades in evolutionary ecology. A popular standard model that maximizes payoff per encounter was recently criticized for being too simplistic, leading to erroneous predictions. We review a number of SDT models that have received less attention but have explicitly taken repeated encounters into account, focusing on prey choice, mate search, aggressive mimicry, and the aiding of kin. We show how these models can be seen as variants of a second standard model that can be analyzed in a unified framework. In contrast to the simpler model, in this second model a higher probability of an undesirable or dangerous event occurring may either decrease or increase the receiver's acceptance rates. In each instance, the latter outcome requires undesirable events to be undesirable in a relative rather than an absolute sense. Increasing the abundance of desirable signalers or the payoff from accepting them may also either raise or reduce acceptance rates. Our synthesis highlights fundamental similarities among models previously studied on a case-by-case basis and challenges some long-held beliefs. For example, some classic predictions of Batesian mimicry can be reversed when model prey are protected by low profitability rather than harmful defense.

Sdtlu: An R package for the signal detection analysis of eyewitness lineup data.

In a standard eyewitness lineup scenario, a witness observes a culprit commit a crime and is later asked to identify the culprit from a set of faces, the lineup. Signal detection theory (SDT), a powerful modeling framework for analyzing data, has recently become a common way to analyze lineup data. The goal of this paper is to introduce a new R package, sdtlu (Signal Detection Theory - LineUp), that streamlines and automates the SDT analysis of lineup data. sdtlu provides functions to process lineup data, determine the best-fitting SDT parameters, compute model-based performance measures such as area under the curve (AUC) and diagnosticity, use bootstrapping to determine uncertainty intervals around these parameters and measures, and compare parameters across two different data sets. The package incorporates closed-form solutions for both simultaneous and sequential lineups that allow for model-based analyses without Monte Carlo simulation. Show-ups are also supported. The package can estimate the base-rate of lineups that include a guilty suspect when the guilt or innocence of each suspect in the data set is unknown, as in "real-world" lineups. The package can also produce a full set of graphs, including data and model-based ROC curves and the underlying SDT model.

Woodlice change the habitat use of spiders in a different food chain.

BackgroundIn old field systems, the common woodlouse may have an indirect effect on a nursery web spider. Woodlice and nursery web spiders feed in different food chains, yet previous work demonstrated that the presence of woodlice is correlated with higher predation success by nursery web spiders upon their grasshopper prey. This finding suggested a new hypothesis which links two seemingly disparate food chains: when woodlice are present, the spider predator or the grasshopper prey changes their location in the vegetative canopy in a way that increases their spatial overlap and therefore predation rate. However, warming temperatures may complicate this phenomenon. The spider cannot tolerate thermal stress, meaning warming temperatures may cause the spider to move downwards in the vegetative canopy or otherwise alter its response to woodlice. Therefore, we would expect warming and woodlice presence to have an interactive effect on predation rate.MethodsWe conducted behavioral experiments in 2015, 2017, and 2018 to track habitat domains—the use of the vegetative canopy space by grasshoppers and spiders—in experimental cages. Then, we used three models of spider movement to try to explain the response of spiders to woodlice: expected net energy gain, signal detection theory, and individual-based modelling.ResultsHabitat domain observations revealed that spiders shift upward in the canopy when woodlice are present, but the corresponding effect on grasshopper prey survival was variable over the different years of study. Under warming conditions, spiders remained lower in the canopy regardless of the presence of woodlice, suggesting that thermal stress is more important than the effect of woodlice. Our modelling results suggest that spiders do not need to move away from woodlice to maximize net energy gain (expected net energy gain and signal detection theory models). Instead spider behavior is consistent with the null hypothesis that they move away from unsuccessful encounters with woodlice (individual-based simulation). We conclude that mapping how predator behavior changes across biotic (e.g. woodlouse presence) and abiotic conditions (e.g. temperature) may be critical to anticipate changes in ecosystem dynamics.

Estimating the proportion of guilty suspects and posterior probability of guilt in lineups using signal-detection models.

BackgroundThe majority of eyewitness lineup studies are laboratory-based. How well the conclusions of these studies, including the relationship between confidence and accuracy, generalize to real-world police lineups is an open question. Signal detection theory (SDT) has emerged as a powerful framework for analyzing lineups that allows comparison of witnesses’ memory accuracy under different types of identification procedures. Because the guilt or innocence of a real-world suspect is generally not known, however, it is further unknown precisely how the identification of a suspect should change our belief in their guilt. The probability of guilt after the suspect has been identified, the posterior probability of guilt (PPG), can only be meaningfully estimated if we know the proportion of lineups that include a guilty suspect, P(guilty). Recent work used SDT to estimate P(guilty) on a single empirical data set that shared an important property with real-world data; that is, no information about the guilt or innocence of the suspects was provided. Here we test the ability of the SDT model to recover P(guilty) on a wide range of pre-existing empirical data from more than 10,000 identification decisions. We then use simulations of the SDT model to determine the conditions under which the model succeeds and, where applicable, why it fails.ResultsFor both empirical and simulated studies, the model was able to accurately estimate P(guilty) when the lineups were fair (the guilty and innocent suspects did not stand out) and identifications of both suspects and fillers occurred with a range of confidence levels. Simulations showed that the model can accurately recover P(guilty) given data that matches the model assumptions. The model failed to accurately estimate P(guilty) under conditions that violated its assumptions; for example, when the effective size of the lineup was reduced, either because the fillers were selected to be poor matches to the suspect or because the innocent suspect was more familiar than the guilty suspect. The model also underestimated P(guilty) when a weapon was shown.ConclusionsDepending on lineup quality, estimation of P(guilty) and, relatedly, PPG, from the SDT model can range from poor to excellent. These results highlight the need to carefully consider how the similarity relations between fillers and suspects influence identifications.