Perceptual Sensitivity Research Articles

Efficient coding of natural images in the mouse visual cortex

How the activity of neurons gives rise to natural vision remains a matter of intense investigation. The mid-level visual areas along the ventral stream are selective to a common class of natural images—textures—but a circuit-level understanding of this selectivity and its link to perception remains unclear. We addressed these questions in mice, first showing that they can perceptually discriminate between textures and statistically simpler spectrally matched stimuli, and between texture types. Then, at the neural level, we found that the secondary visual area (LM) exhibited a higher degree of selectivity for textures compared to the primary visual area (V1). Furthermore, textures were represented in distinct neural activity subspaces whose relative distances were found to correlate with the statistical similarity of the images and the mice’s ability to discriminate between them. Notably, these dependencies were more pronounced in LM, where the texture-related subspaces were smaller than in V1, resulting in superior stimulus decoding capabilities. Together, our results demonstrate texture vision in mice, finding a linking framework between stimulus statistics, neural representations, and perceptual sensitivity—a distinct hallmark of efficient coding computations.

Explainable machine learning predictions of perceptual sensitivity for retinal prostheses

Objective. Retinal prostheses evoke visual precepts by electrically stimulating functioning cells in the retina. Despite high variance in perceptual thresholds across subjects, among electrodes within a subject, and over time, retinal prosthesis users must undergo ‘system fitting’, a process performed to calibrate stimulation parameters according to the subject’s perceptual thresholds. Although previous work has identified electrode-retina distance and impedance as key factors affecting thresholds, an accurate predictive model is still lacking. Approach. To address these challenges, we (1) fitted machine learning models to a large longitudinal dataset with the goal of predicting individual electrode thresholds and deactivation as a function of stimulus, electrode, and clinical parameters (‘predictors’) and (2) leveraged explainable artificial intelligence (XAI) to reveal which of these predictors were most important. Main results. Our models accounted for up to 76% of the perceptual threshold response variance and enabled predictions of whether an electrode was deactivated in a given trial with F1 and area under the ROC curve scores of up to 0.732 and 0.911, respectively. Our models identified novel predictors of perceptual sensitivity, including subject age, time since blindness onset, and electrode-fovea distance. Significance. Our results demonstrate that routinely collected clinical measures and a single session of system fitting might be sufficient to inform an XAI-based threshold prediction strategy, which has the potential to transform clinical practice in predicting visual outcomes.

Specificity of tilt illusion reduction through perceptual learning.

Human perceptual ability can be improved by perceptual learning through repeated exposure or training. Perceptual learning studies have focused on achieving accurate perception of stimuli by improving perceptual sensitivity. However, eliminating illusions can also be one of the ways of accurate perception. To determine whether the illusion can be attenuated by perceptual learning, the current study used a tilt illusion where the orientation of the grating presented in the center (central grating) was misperceived because of the orientation of the grating presented in the periphery (surrounding grating). In Experiment 1, participants were trained either in the illusion training condition, in which they trained with illusory stimuli presenting both surrounding and central gratings together, or in the control training condition, where only the central grating was presented. The results confirmed that the tilt illusion was reduced only in the illusion training condition. Experiment 2 tested the transfer effect of learning, which is not often observed in perceptual learning. During training, the orientation of the surrounding grating was fixed to see whether the elimination of the illusion also occurred in the surrounding grating with an orientation that was not used during training. A decrease in the illusion was found only in the case of a surrounding grating with trained orientations, and not in the case of surrounding gratings with untrained orientations. These results suggest that the reduction in tilt illusion through training is due to perceptual learning.

Overlapping yet dissociable contributions of superiority illusion features to Ponzo illusion strength and metacognitive performance

Humans are typically inept at evaluating their abilities and predispositions. People dismiss such a lack of metacognitive insight into their capacities while even enhancing (albeit illusorily) self-evaluation such that they should have more desirable traits than an average peer. This superiority illusion helps maintain a healthy mental state. However, the scope and range of its influence on broader human behavior, especially perceptual tasks, remain elusive. As belief shapes the way people perceive and recognize, the illusory self-superiority belief potentially regulates our perceptual and metacognitive performance. In this study, we used hierarchical Bayesian estimation and machine learning of signal detection theoretic measures to understand how the superiority illusion influences visual perception and metacognition for the Ponzo illusion. Our results demonstrated that the superiority illusion correlated with the Ponzo illusion magnitude and metacognitive performance. Next, we combined principal component analysis and cross-validated regularized regression (relaxed elastic net) to identify which superiority components contributed to the correlations. We revealed that the “extraversion” superiority dimension tapped into the Ponzo illusion magnitude and metacognitive ability. In contrast, the “honesty-humility” and “neuroticism” dimensions only predicted Ponzo illusion magnitude and metacognitive ability, respectively. These results suggest common and distinct influences of superiority features on perceptual sensitivity and metacognition. Our findings contribute to the accumulating body of evidence indicating that the leverage of superiority illusion is far-reaching, even to visual perception.

Multiple timescales of context shape the perceptual sensitivity to natural pairings of musical pitch and timbre

Musical pitch and timbre (specifically, spectral shape) covary perceptually: lower pitches are associated with darker timbres (less higher-frequency energy) and higher pitches are associated with brighter timbres (more higher-frequency energy). We examined this relationship at multiple timescales of context: trial-level (rates of performance improvement within a testing block), block-level (differences in performance across blocks), session- level (performance as a function of testing block order), and longer-term experience (musical training / background). Musical pitches were labeled as low (C4) or high (G4) from various instruments (trumpet, oboe, trombone, tuba). Testing blocks paired pitch and timbre together to either respect (Consistent) or violate (Reversed) their covariance. All timescales of context contributed to performance, producing clear patterns across experiments. Pitch labeling was poorer in Reversed blocks, but showed steeper rates of improvement than did Consistent blocks (which were often near/at ceiling levels). These patterns were moderated by which condition was tested first in the experiment and by the introduction of trial-by-trial feedback. Greater musical training was consistently correlated with higher accuracy, but adding feedback extinguished this pattern for Consistent blocks. Thus, context on a host of timescales shapes perceptual sensitivity to the natural covariance between musical pitch and timbre.

Threat Expectation Does Not Improve Perceptual Discrimination despite Causing Heightened Priority Processing in the Frontoparietal Network.

Threat cues have been widely shown to elicit increased sensory and attentional neural processing. However, whether this enhanced recruitment leads to measurable behavioral improvements in perception is still in question. Here, we adjudicate between two opposing theories: that threat cues do or do not enhance perceptual sensitivity. We created threat stimuli by pairing one direction of motion in a random dot kinematogram with an aversive sound. While in the MRI scanner, 46 subjects (both men and women) completed a cued (threat/safe/neutral) perceptual decision-making task where they indicated the perceived motion direction of each moving dot stimulus. We found strong evidence that threat cues did not increase perceptual sensitivity compared with safe and neutral cues. This lack of improvement in perceptual decision-making ability occurred despite the threat cue resulting in widespread increases in frontoparietal BOLD activity, as well as increased connectivity between the right insula and the frontoparietal network. These results call into question the intuitive claim that expectation automatically enhances our perception of threat and highlight the role of the frontoparietal network in prioritizing the processing of threat-related environmental cues.

Perceptual confirmation bias and decision bias underlie adaptation to sequential regularities.

Our perception does not depend exclusively on the immediate sensory input. It is also influenced by our internal predictions derived from prior observations and the temporal regularities of the environment, which can result in choice history biases. However, it is unclear how this flexible use of prior information to predict the future influences perceptual decisions. Prior information may bias decisions independently of the current sensory input, or it may modulate the weight of current sensory input based on its consistency with the expectation. To address this question, we used a visual decision-making task and manipulated the transitional probabilities between successive noisy grating stimuli. Using a reverse correlation analysis, we evaluated the contribution of stimulus-independent decision bias and stimulus-dependent sensitivity modulations to choice history biases. We found that both effects coexist, whereby there was increased bias to respond in line with the predicted orientation alongside modulations in perceptual sensitivity to favor perceptual information consistent with the prediction, akin to selective attention. Furthermore, at the individual differences level, we investigated the relationship between autistic-like traits and the adaptation of choice history biases to the sequential statistics of the environment. Over two studies, we found no convincing evidence of reduced adaptation to sequential regularities in individuals with high autistic-like traits. In sum, we present robust evidence for both perceptual confirmation bias and decision bias supporting adaptation to sequential regularities in the environment.

Dissociable components of attention exhibit distinct neuronal signatures in primate visual cortex.

Attention can be deployed in multiple forms and facilitates behavior by influencing perceptual sensitivity and choice bias. Attention is also associated with a myriad of changes in sensory neural activity. Yet, the relationship between the behavioral components of attention and the accompanying changes in neural activity remains largely unresolved. We examined this relationship by quantifying sensitivity and bias in monkeys performing a task that dissociated eye movement responses from the focus of covert attention. Unexpectedly, bias, not sensitivity, increased at the focus of covert attention, whereas sensitivity increased at the location of planned eye movements. Furthermore, neuronal activity within visual area V4 varied robustly with bias, but not sensitivity, at the focus of covert attention. In contrast, correlated variability between neuronal pairs was lowest at the location of planned eye movements, and varied with sensitivity, but not bias. Thus, dissociable behavioral components of attention exhibit distinct neuronal signatures within the visual cortex.

Presaccadic attention does not facilitate the detection of changes in the visual field.

Planning a rapid eye movement (saccade) changes how we perceive our visual world. Even before we move the eyes visual discrimination sensitivity improves at the impending target of eye movements, a phenomenon termed "presaccadic attention." Yet, it is unknown if such presaccadic selection merely affects perceptual sensitivity, or also affects downstream decisional processes, such as choice bias. We report a surprising lack of presaccadic perceptual benefits in a common, everyday setting-detection of changes in the visual field. Despite the lack of sensitivity benefits, choice bias for reporting changes increased reliably for the saccade target. With independent follow-up experiments, we show that presaccadic change detection is rendered more challenging because percepts at the saccade target location are biased toward, and more precise for, only the most recent of two successive stimuli. With a Bayesian model, we show how such perceptual and choice biases are crucial to explain the effects of saccade plans on change detection performance. In sum, visual change detection sensitivity does not improve presaccadically, a result that is readily explained by teasing apart distinct components of presaccadic selection. The findings may have critical implications for real-world scenarios, like driving, that require rapid gaze shifts in dynamically changing environments.

Japanese and Korean speakers' perceptual boundary and discrimination sensitivity of Japanese fricative /s/ and affricate /ʦ/

Japanese and Korean speakers' perceptual boundary and discrimination sensitivity of Japanese fricative /s/ and affricate /ʦ/

3D-PSSIM: Projective Structural Similarity for 3D Mesh Quality Assessment Robust to Topological Irregularities.

Despite acceleration in the use of 3D meshes, it is difficult to find effective mesh quality assessment algorithms that can produce predictions highly correlated with human subjective opinions. Defining mesh quality features is challenging due to the irregular topology of meshes, which are defined on vertices and triangles. To address this, we propose a novel 3D projective structural similarity index ( 3D- PSSIM) for meshes that is robust to differences in mesh topology. We address topological differences between meshes by introducing multi-view and multi-layer projections that can densely represent the mesh textures and geometrical shapes irrespective of mesh topology. It also addresses occlusion problems that occur during projection. We propose visual sensitivity weights that capture the perceptual sensitivity to the degree of mesh surface curvature. 3D- PSSIM computes perceptual quality predictions by aggregating quality-aware features that are computed in multiple projective spaces onto the mesh domain, rather than on 2D spaces. This allows 3D- PSSIM to determine which parts of a mesh surface are distorted by geometric or color impairments. Experimental results show that 3D- PSSIM can predict mesh quality with high correlation against human subjective judgments, across the presence of noise, even when there are large topological differences, outperforming existing mesh quality assessment models.

A Comparative Study of the Perceptual Sensitivity of Topological Visualizations to Feature Variations.

Color maps are a commonly used visualization technique in which data are mapped to optical properties, e.g., color or opacity. Color maps, however, do not explicitly convey structures (e.g., positions and scale of features) within data. Topology-based visualizations reveal and explicitly communicate structures underlying data. Although our understanding of what types of features are captured by topological visualizations is good, our understanding of people's perception of those features is not. This paper evaluates the sensitivity of topology-based isocontour, Reeb graph, and persistence diagram visualizations compared to a reference color map visualization for synthetically generated scalar fields on 2-manifold triangular meshes embedded in 3D. In particular, we built and ran a human-subject study that evaluated the perception of data features characterized by Gaussian signals and measured how effectively each visualization technique portrays variations of data features arising from the position and amplitude variation of a mixture of Gaussians. For positional feature variations, the results showed that only the Reeb graph visualization had high sensitivity. For amplitude feature variations, persistence diagrams and color maps demonstrated the highest sensitivity, whereas isocontours showed only weak sensitivity. These results take an important step toward understanding which topology-based tools are best for various data and task scenarios and their effectiveness in conveying topological variations as compared to conventional color mapping.

Phase of neural oscillations as a reference frame for attention-based routing in visual cortex

Selective attention allows the brain to efficiently process the image projected onto the retina, selectively focusing neural processing resources on behaviorally relevant visual information. While previous studies have documented the crucial role of the action potential rate of single neurons in relaying such information, little is known about how the activity of single neurons relative to their neighboring network contributes to the efficient representation of attended stimuli and transmission of this information to downstream areas. Here, we show in the dorsal visual pathway of monkeys (medial superior temporal area) that neurons fire spikes preferentially at a specific phase of the ongoing population beta (∼20 Hz) oscillations of the surrounding local network. This preferred spiking phase shifts towards a later phase when monkeys selectively attend towards (rather than away from) the receptive field of the neuron. This shift of the locking phase is positively correlated with the speed at which animals report a visual change. Furthermore, our computational modeling suggests that neural networks can manipulate the preferred phase of coupling by imposing differential synaptic delays on postsynaptic potentials. This distinction between the locking phase of neurons activated by the spatially attended stimulus vs. that of neurons activated by the unattended stimulus, may enable the neural system to discriminate relevant from irrelevant sensory inputs and consequently filter out distracting stimuli information by aligning the spikes which convey relevant/irrelevant information to distinct phases linked to periods of better/worse perceptual sensitivity for higher cortices. This strategy may be used to reserve the narrow windows of highest perceptual efficacy to the processing of the most behaviorally relevant information, ensuring highly efficient responses to attended sensory events.

Think twice: Re-assessing confidence improves visual metacognition

Metacognition is a fundamental feature of human behavior that has adaptive functional value. Current understanding of the factors that influence metacognition remains incomplete, and we lack protocols to improve metacognition. Here, we introduce a two-step confidence choice paradigm to test whether metacognitive performance may improve by asking subjects to reassess their initial confidence. Previous work on perceptual and mnemonic decision-making has shown that (type 1) perceptual sensitivity benefits from reassessing the primary choice, however, it is not clear whether such an effect occurs for type 2 confidence choices. To test this hypothesis, we ran two separate online experiments, in which participants completed a type 1 task followed by two consecutive confidence choices. The results of the two experiments indicated that metacognitive sensitivity improved after re-evaluation. Since post-decisional evidence accumulation following the first confidence choice is likely to be minimal, this metacognitive improvement is better accounted for by an attenuation of metacognitive noise during the process of confidence generation. Thus, here we argue that metacognitive noise may be filtered out by additional post-decisional processing, thereby improving metacognitive sensitivity. We discuss the ramifications of these findings for models of metacognition and for developing protocols to train and manipulate metacognitive processes.

Invited Session I: 30 years of normalization in the visual system: Spatiotemporal fluctuations in E/I balance & their impact on perception.

The Normalization Model has helped illuminate our understanding of neural computations in a wide variety of contexts. Here, I will describe our recent efforts to understand how intrinsic fluctuations in the ratio of excitatory to inhibitory conductances play out in space and time, and how these fluctuations impact perception. We find that spontaneous cortical activity is organized into traveling waves that traverse visual cortex several times per second. Recording in Area MT of the common marmoset, we find that these intrinsic traveling waves (iTWs) regulate both the gain of the stimulus-evoked spiking response and the monkey's perceptual sensitivity. A large-scale spiking network model recapitulates the properties of iTWs measured in vivo and explains the observed fluctuations in response gain as resulting from momentary fluctuations in E/I balance. The model predicts that iTWs are sparse, in the sense that only a small fraction of the neural population participates in any individual iTW. As a result, iTWs can occur without inducing correlated variability, which we have found, in separate experiments, can impair sensory discrimination. We thus refer to the model as the sparse-wave model of iTWs. Taken together, these findings lead to the conclusion that traveling waves strongly modulate neural and perceptual sensitivity.

Poster Session II: Spillover effects of color discrimination training on color category boundaries and color appearance.

Perceptual learning refers to the increase in perceptual sensitivity that results from several days of training on a perceptual task. Although perceptual learning has been shown to be effective in a variety of perceptual tasks, few studies have examined perceptual learning in color perception. In this study, we investigated how color discrimination training at a base color affected various aspects of color perception for entire hues. The training consisted of five days of S color discrimination (300 trials/day) at either the negative or positive L-M base color, depending on the observer groups. Before and after the training, three types of color perception tests (color difference, unique hue, and color category boundary) were conducted for colors with various hues to examine the changes in color perception due to the training. The results showed that the color discrimination thresholds in the training decreased as expected with repeated trials. Interestingly, the training also affected the performance of the three types of tests; the perceived color difference around the training color tended to increase, and some of the unique hues and the color category boundaries shifted significantly toward the training color. These results suggest that only a few days of color discrimination training can spill over to the entire color space and induce distortion of the perceptual color space.

Poster Session I: Perspective-correct rendering for active observers.

Stereoscopic, head-tracked display systems can show users realistic, world-locked virtual objects and environments (i.e., rendering perspective-correct binocular images with accurate motion parallax). However, discrepancies between the rendering pipeline and physical viewing conditions can lead to perceived instability in the rendered content resulting in reduced immersion and, potentially, visually-induced motion sickness. Precise requirements to achieve perceptually stable world-locked rendering (WLR) are unknown due to the challenge of constructing a wide field of view, distortion-free display with highly accurate head and eyetracking. We present a custom-built system capable of rendering virtual objects over real-world references without perceivable drift under such constraints. This platform is used to study acceptable errors in render camera position for WLR in augmented and virtual reality scenarios, where we find an order of magnitude difference in perceptual sensitivity. We conclude with an analytic model which examines changes to apparent depth and visual direction in response to camera displacement errors, and visual direction is highlighted as a potentially important consideration for WLR alongside depth errors from incorrect disparity.

Concurrent attention to hetero-depth surfaces in 3-D visual space is governed by theta rhythm.

When simultaneously confronted with multiple attentional targets, visual system employs a time-multiplexing approach in which each target alternates for prioritized access, a mechanism broadly known as rhythmic attentional sampling. For the past decade, rhythmic attentional sampling has received mounting support from converging behavioral and neural findings. However, so compelling are these findings that a critical test ground has been long overshadowed, namely the 3-D visual space where attention is complicated by extraction of the spatial layout of surfaces extending beyond 2-D planes. It remains unknown how attentional deployment to multiple targets is accomplished in the 3-D space. Here, we provided a time-resolved portrait of the behavioral and neural dynamics when participants concurrently attended to two surfaces defined by motion-depth conjunctions. To characterize the moment-to-moment attentional modulation effects, we measured perceptual sensitivity to the hetero-depth surface motions on a fine temporal scale and reconstructed their neural representations using a time-resolved multivariate inverted encoding model. We found that the perceptual sensitivity to the two surface motions rhythmically fluctuated over time at ~4 Hz, with one's enhancement closely tracked by the other's diminishment. Moreover, the behavioral pattern was coupled with an ongoing periodic alternation in strength between the two surface motion representations in the same frequency. Together, our findings provide the first converging evidence of an attentional "pendulum" that rhythmically traverses different stereoscopic depth planes and are indicative of a ubiquitous attentional time multiplexor based on theta rhythm in the 3-D visual space.

Catecholaminergic neuromodulation and selective attention jointly shape perceptual decision-making.

Perceptual decisions about sensory input are influenced by fluctuations in ongoing neural activity, most prominently driven by attention and neuromodulator systems. It is currently unknown if neuromodulator activity and attention differentially modulate perceptual decision-making and/or whether neuromodulatory systems in fact control attentional processes. To investigate the effects of two distinct neuromodulatory systems and spatial attention on perceptual decisions, we pharmacologically elevated cholinergic (through donepezil) and catecholaminergic (through atomoxetine) levels in humans performing a visuo-spatial attention task, while we measured electroencephalography (EEG). Both attention and catecholaminergic enhancement improved decision-making at the behavioral and algorithmic level, as reflected in increased perceptual sensitivity and the modulation of the drift rate parameter derived from drift diffusion modeling. Univariate analyses of EEG data time-locked to the attentional cue, the target stimulus, and the motor response further revealed that attention and catecholaminergic enhancement both modulated pre-stimulus cortical excitability, cue- and stimulus-evoked sensory activity, as well as parietal evidence accumulation signals. Interestingly, we observed both similar, unique, and interactive effects of attention and catecholaminergic neuromodulation on these behavioral, algorithmic, and neural markers of the decision-making process. Thereby, this study reveals an intricate relationship between attentional and catecholaminergic systems and advances our understanding about how these systems jointly shape various stages of perceptual decision-making.

Catecholaminergic neuromodulation and selective attention jointly shape perceptual decision-making

Perceptual decisions about sensory input are influenced by fluctuations in ongoing neural activity, most prominently driven by attention and neuromodulator systems. It is currently unknown if neuromodulator activity and attention differentially modulate perceptual decision-making and/or whether neuromodulatory systems in fact control attentional processes. To investigate the effects of two distinct neuromodulatory systems and spatial attention on perceptual decisions, we pharmacologically elevated cholinergic (through donepezil) and catecholaminergic (through atomoxetine) levels in humans performing a visuo-spatial attention task, while we measured electroencephalography (EEG). Both attention and catecholaminergic enhancement improved decision-making at the behavioral and algorithmic level, as reflected in increased perceptual sensitivity and the modulation of the drift rate parameter derived from drift diffusion modeling. Univariate analyses of EEG data time-locked to the attentional cue, the target stimulus, and the motor response further revealed that attention and catecholaminergic enhancement both modulated pre-stimulus cortical excitability, cue- and stimulus-evoked sensory activity, as well as parietal evidence accumulation signals. Interestingly, we observed both similar, unique, and interactive effects of attention and catecholaminergic neuromodulation on these behavioral, algorithmic, and neural markers of the decision-making process. Thereby, this study reveals an intricate relationship between attentional and catecholaminergic systems and advances our understanding about how these systems jointly shape various stages of perceptual decision-making.