Perceptual Inference Research Articles

N-methyl-d-aspartate receptor hypofunction causes recurrent and transient failures of perceptual inference.

Perception integrates external sensory signals with internal predictions that reflect prior knowledge about the world. Previous research suggests that this integration is governed by slow alternations between an external mode, driven by sensory signals, and an internal mode, shaped by prior knowledge. Using a double-blind, placebo-controlled, cross-over experiment in healthy human participants, we investigated the effects of the N-Methyl-D-aspartate receptor (NMDAR) antagonist S-ketamine on the balance between external and internal modes. We found that S-ketamine causes a shift of perception toward the external mode. A case-control study revealed that individuals with paranoid Scz, a disorder repeatedly associated with NMDAR hypofunction, spend more time in the external mode. This NMDAR-dependent increase in the external mode suggests that the symptoms of schizophrenia are caused by recurring dissociations of perception from prior knowledge about the world.

A dissociation between the use of implicit and explicit priors in perceptual inference

The brain constantly uses prior knowledge of the statistics of its environment to shape perception. These statistics are often implicit (not directly observable) and learned incrementally from observation, but they can also be explicitly communicated to the observer, especially in humans. Here, we show that priors are used differently in human perceptual inference depending on whether they are explicit or implicit in the environment. Bayesian modeling of learning and perception revealed that the weight of the sensory likelihood in perceptual decisions was highly correlated across participants between tasks with implicit and explicit priors, and slightly stronger in the implicit task. By contrast, the weight of priors was much less correlated across tasks, and it was markedly smaller for explicit priors. The model comparison also showed that different computations underpinned perceptual decisions depending on the origin of the priors. This dissociation may resolve previously conflicting results about the appropriate use of priors in human perception.

High-level visual prediction errors in early visual cortex.

Perception is shaped by both incoming sensory input and expectations derived from our prior knowledge. Numerous studies have shown stronger neural activity for surprising inputs, suggestive of predictive processing. However, it is largely unclear what predictions are made across the cortical hierarchy, and therefore what kind of surprise drives this up-regulation of activity. Here, we leveraged fMRI in human volunteers and deep neural network (DNN) models to arbitrate between 2 hypotheses: prediction errors may signal a local mismatch between input and expectation at each level of the cortical hierarchy, or prediction errors may be computed at higher levels and the resulting surprise signal is broadcast to earlier areas in the cortical hierarchy. Our results align with the latter hypothesis. Prediction errors in both low- and high-level visual cortex responded to high-level, but not low-level, visual surprise. This scaling with high-level surprise in early visual cortex strongly diverged from feedforward tuning. Combined, our results suggest that high-level predictions constrain sensory processing in earlier areas, thereby aiding perceptual inference.

The Many Roles of Precision in Action.

Active inference describes (Bayes-optimal) behaviour as being motivated by the minimisation of surprise of one's sensory observations, through the optimisation of a generative model (of the hidden causes of one's sensory data) in the brain. One of active inference's key appeals is its conceptualisation of precision as biasing neuronal communication and, thus, inference within generative models. The importance of precision in perceptual inference is evident-many studies have demonstrated the importance of ensuring precision estimates are correct for normal (healthy) sensation and perception. Here, we highlight the many roles precision plays in action, i.e., the key processes that rely on adequate estimates of precision, from decision making and action planning to the initiation and control of muscle movement itself. Thereby, we focus on the recent development of hierarchical, "mixed" models-generative models spanning multiple levels of discrete and continuous inference. These kinds of models open up new perspectives on the unified description of hierarchical computation, and its implementation, in action. Here, we highlight how these models reflect the many roles of precision in action-from planning to execution-and the associated pathologies if precision estimation goes wrong. We also discuss the potential biological implementation of the associated message passing, focusing on the role of neuromodulatory systems in mediating different kinds of precision.

Perceptual inference corrects function word errors in reading: Errors that are not noticed do not disrupt eye movements

Both everyday experience and laboratory research demonstrate that readers often fail to notice errors such as an omitted or repeated function word. This phenomenon challenges central tenets of reading and sentence processing models, according to which each word is lexically processed and incrementally integrated into a syntactic representation. One solution would propose that apparent failure to notice such errors reflects post-perceptual inference; the reader does initially perceive the error, but then unconsciously ’corrects’ the perceived string. Such a post-perceptual account predicts that when readers fail to explicitly notice an error, the error will nevertheless disrupt reading, at least fleetingly. We present a large-scale eyetracking experiment investigating whether disruption is detectable in the eye movement record when readers fail to notice an omitted or repeated two-letter function word in naturalistic sentences. Readers failed to notice both omission and repetition errors over 36% of the time. In an analysis that included all trials, both omission and repetition resulted in pronounced eye movement disruption, compared to reading of grammatical control sentences. But in an analysis including only trials on which readers failed to notice the errors, neither type of error disrupted eye movements on any measure. Indeed, there was evidence in some measures that reading was relatively fast on the trials on which errors were missed. It does not appear that when an error is not consciously noticed, it is initially perceived, and then later corrected; rather, linguistic knowledge influences what the reader perceives.

Top-Down Priors Disambiguate Target and Distractor Features in Simulated Covert Visual Search.

Several models of visual search consider visual attention as part of a perceptual inference process, in which top-down priors disambiguate bottom-up sensory information. Many of these models have focused on gaze behavior, but there are relatively fewer models of covert spatial attention, in which attention is directed to a peripheral location in visual space without a shift in gaze direction. Here, we propose a biologically plausible model of covert attention during visual search that helps to bridge the gap between Bayesian modeling and neurophysiological modeling by using (1) top-down priors over target features that are acquired through Hebbian learning, and (2) spatial resampling of modeled cortical receptive fields to enhance local spatial resolution of image representations for downstream target classification. By training a simple generative model using a Hebbian update rule, top-down priors for target features naturally emerge without the need for hand-tuned or predetermined priors. Furthermore, the implementation of covert spatial attention in our model is based on a known neurobiological mechanism, providing a plausible process through which Bayesian priors could locally enhance the spatial resolution of image representations. We validate this model during simulated visual search for handwritten digits among nondigit distractors, demonstrating that top-down priors improve accuracy for estimation of target location and classification, relative to bottom-up signals alone. Our results support previous reports in the literature that demonstrated beneficial effects of top-down priors on visual search performance, while extending this literature to incorporate known neural mechanisms of covert spatial attention.

How to reward animals based on their subjective percepts: A Bayesian approach to online estimation of perceptual biases.

Elucidating the neural basis of perceptual biases, such as those produced by visual illusions, can provide powerful insights into the neural mechanisms of perceptual inference. However, studying the subjective percepts of animals poses a fundamental challenge: unlike human participants, animals cannot be verbally instructed to report what they see, hear, or feel. Instead, they must be trained to perform a task for reward, and researchers must infer from their responses what the animal perceived. However, animals' responses are shaped by reward feedback, thus raising the major concern that the reward regimen may alter the animal's decision strategy or even intrinsic perceptual biases. We developed a method that estimates perceptual bias during task performance and then computes the reward for each trial based on the evolving estimate of the animal's perceptual bias. Our approach makes use of multiple stimulus contexts to dissociate perceptual biases from decision-related biases. Starting with an informative prior, our Bayesian method updates a posterior over the perceptual bias after each trial. The prior can be specified based on data from past sessions, thus reducing the variability of the online estimates and allowing it to converge to a stable estimate over a small number of trials. After validating our method on synthetic data, we apply it to estimate perceptual biases of monkeys in a motion direction discrimination task in which varying background optic flow induces robust perceptual biases. This method overcomes an important challenge to understanding the neural basis of subjective percepts.

Conspiracy beliefs and perceptual inference in times of political uncertainty.

Sociopolitical crises causing uncertainty have accumulated in recent years, providing fertile ground for the emergence of conspiracy ideations. Computational models constitute valuable tools for understanding the mechanisms at play in the formation and rigidification of these unshakeable beliefs. Here, the Circular Inference model was used to capture associations between changes in perceptual inference and the dynamics of conspiracy ideations in times of uncertainty. A bistable perception task and conspiracy belief assessment focused on major sociopolitical events were administered to large populations from three polarized countries. We show that when uncertainty peaks, an overweighting of sensory information is associated with conspiracy ideations. Progressively, this exploration strategy gives way to an exploitation strategy in which increased adherence to conspiracy theories is associated with the amplification of prior information. Overall, the Circular Inference model sheds new light on the possible mechanisms underlying the progressive strengthening of conspiracy theories when individuals face highly uncertain situations.

Predictive coding with spiking neurons and feedforward gist signaling.

Predictive coding (PC) is an influential theory in neuroscience, which suggests the existence of a cortical architecture that is constantly generating and updating predictive representations of sensory inputs. Owing to its hierarchical and generative nature, PC has inspired many computational models of perception in the literature. However, the biological plausibility of existing models has not been sufficiently explored due to their use of artificial neurons that approximate neural activity with firing rates in the continuous time domain and propagate signals synchronously. Therefore, we developed a spiking neural network for predictive coding (SNN-PC), in which neurons communicate using event-driven and asynchronous spikes. Adopting the hierarchical structure and Hebbian learning algorithms from previous PC neural network models, SNN-PC introduces two novel features: (1) a fast feedforward sweep from the input to higher areas, which generates a spatially reduced and abstract representation of input (i.e., a neural code for the gist of a scene) and provides a neurobiological alternative to an arbitrary choice of priors; and (2) a separation of positive and negative error-computing neurons, which counters the biological implausibility of a bi-directional error neuron with a very high baseline firing rate. After training with the MNIST handwritten digit dataset, SNN-PC developed hierarchical internal representations and was able to reconstruct samples it had not seen during training. SNN-PC suggests biologically plausible mechanisms by which the brain may perform perceptual inference and learning in an unsupervised manner. In addition, it may be used in neuromorphic applications that can utilize its energy-efficient, event-driven, local learning, and parallel information processing nature.

Human perception of impact sounds suggests auditory intuitive physics

Upon hearing objects collide, humans can estimate many of the underlying physical attributes, such as the objects’ material and mass. Although the physics of sound generation are well established, the inverse problem that listeners must solve – of inferring physical parameters from sound – remains poorly understood. In this work, we show that humans leverage an understanding of acoustical physics to constrain their perceptual inferences, allowing them to disambiguate multiple object properties from a single impact sound. We derived a linear generative model of impact sounds, combining theoretical acoustics with empirically measured statistics of object resonances. We used an analysis-by-synthesis algorithm to infer mode parameters from recorded object impulse responses. We then fit distributions to these parameters, from which object impulse responses could be sampled. Perceptual experiments demonstrated that humans could judge material and mass from sound alone, even when both of the underlying objects varied. However, performance with synthetic sounds was impaired if the simulated physical regularities were altered to be unnatural. The results suggest that listeners use internal physical models to separate the acoustic contributions of the objects that interact to create sound.

Copy raising reconsidered

There is no consensus in the literature regarding the defining characteristics of copy raising (CR), or in other words what determineswhether a CR-like expression is CR or not. As a result, existing analyses target different data sets. In this paper, I propose a different approach to these constructions, which takes a functional perspective. I propose to abandon the term copy raising, which is misleading in a number of ways. Instead, I distinguish between perceptual depiction reports and perceptual inference reports and show that the functions which they fulfill are not particular to CR-like constructions, but are in fact more general. Such an approach, I claim, resolves existing conundrums surrounding CR. The analysis is formalized in the framework of Head-driven Phrase Structure Grammar (HPSG) and is inspired by previous accounts of CR in related frameworks such as LFG and SBCG, as well as HPSG analyses. In the spirit of HPSG, the analysis employs type inheritance hierarchies to distinguish between what is shared by the two constructions and what is construction-specific in order to account for alternative realizations of a single lexeme and to ascribe constructional (or extra-lexical) meaning to linguistic elements.

Sensory processing in humans and mice fluctuates between external and internal modes.

Perception is known to cycle through periods of enhanced and reduced sensitivity to external information. Here, we asked whether such slow fluctuations arise as a noise-related epiphenomenon of limited processing capacity or, alternatively, represent a structured mechanism of perceptual inference. Using 2 large-scale datasets, we found that humans and mice alternate between externally and internally oriented modes of sensory analysis. During external mode, perception aligns more closely with the external sensory information, whereas internal mode is characterized by enhanced biases toward perceptual history. Computational modeling indicated that dynamic changes in mode are enabled by 2 interlinked factors: (i) the integration of subsequent inputs over time and (ii) slow antiphase oscillations in the impact of external sensory information versus internal predictions that are provided by perceptual history. We propose that between-mode fluctuations generate unambiguous error signals that enable optimal inference in volatile environments.

Disentangling sensory precision and prior expectation of change in autism during tactile discrimination

Predictive coding theories suggest that core symptoms in autism spectrum disorders (ASD) may stem from atypical mechanisms of perceptual inference (i.e., inferring the hidden causes of sensations). Specifically, there would be an imbalance in the precision or weight ascribed to sensory inputs relative to prior expectations. Using three tactile behavioral tasks and computational modeling, we specifically targeted the implicit dynamics of sensory adaptation and perceptual learning in ASD. Participants were neurotypical and autistic adults without intellectual disability. In Experiment I, tactile detection thresholds and adaptation effects were measured to assess sensory precision. Experiments II and III relied on two-alternative forced choice tasks designed to elicit a time-order effect, where prior knowledge biases perceptual decisions. Our results suggest a subtler explanation than a simple imbalance in the prior/sensory weights, having to do with the dynamic nature of perception, that is the adjustment of precision weights to context. Compared to neurotypicals, autistic adults showed no difference in average performance and sensory sensitivity. Both groups managed to implicitly learn and adjust a prior that biased their perception. However, depending on the context, autistic participants showed no, normal or slower adaptation, a phenomenon that computational modeling of trial-to-trial responses helped us to associate with a higher expectation for sameness in ASD, and to dissociate from another observed robust difference in terms of response bias. These results point to atypical perceptual learning rather than altered perceptual inference per se, calling for further empirical and computational studies to refine the current predictive coding theories of ASD.

Evidence for Reduced Sensory Precision and Increased Reliance on Priors in Hallucination-Prone Individuals in a General Population Sample.

There is increasing evidence that people with hallucinations overweight perceptual beliefs relative to incoming sensory evidence. Past work demonstrating prior overweighting has used simple, nonlinguistic stimuli. However, auditory hallucinations in psychosis are often complex and linguistic. There may be an interaction between the type of auditory information being processed and its perceived quality in engendering hallucinations. We administered a linguistic version of the conditioned hallucinations (CH) task to an online sample of 88 general population participants. Metrics related to hallucination-proneness, hallucination severity, stimulus thresholds, and stimulus detection rates were collected. Data were used to fit parameters of a Hierarchical Gaussian Filter (HGF) model of perceptual inference to determine how latent perceptual states influenced task behavior. Replicating past results, higher CH rates were observed both in those with recent hallucinatory experiences as well as participants with high hallucination-proneness; CH rates were positively correlated with increased prior weighting; and increased prior weighting was related to hallucination severity. Unlike past results, participants with recent hallucinatory experiences as well as those with higher hallucination-proneness had higher stimulus thresholds, lower sensitivity to stimuli presented at the highest threshold, and had lower response confidence, consistent with lower precision of sensory evidence. We replicate the finding that increased CH rates and recent hallucinations correlate with increased prior weighting using a linguistic version of the CH task. Results support a role for reduced sensory precision in the interplay between prior weighting and hallucination-proneness.

The influence of natural image statistics on upright orientation judgements

Humans have well-documented priors for many features present in nature that guide visual perception. Despite being putatively grounded in the statistical regularities of the environment, scene priors are frequently violated due to the inherent variability of visual features from one scene to the next. However, these repeated violations do not appreciably challenge visuo-cognitive function, necessitating the broad use of priors in conjunction with context-specific information. We investigated the trade-off between participants' internal expectations formed from both longer-term priors and those formed from immediate contextual information using a perceptual inference task and naturalistic stimuli. Notably, our task required participants to make perceptual inferences about naturalistic images using their own internal criteria, rather than making comparative judgements. Nonetheless, we show that observers' performance is well approximated by a model that makes inferences using a prior for low-level image statistics, aggregated over many images. We further show that the dependence on this prior is rapidly re-weighted against contextual information, even when misleading. Our results therefore provide insight into how apparent high-level interpretations of scene appearances follow from the most basic of perceptual processes, which are grounded in the statistics of natural images.

Relative fluency (unfelt vs felt) in active inference

For a growing number of researchers, it is now accepted that the brain is a predictive organ that predicts the content of the sensorium and crucially the precision of—or confidence in—its own predictions. In order to predict the precision of its predictions, the brain has to infer the reliability of its own beliefs. This means that our brains have to recognise the precision of their predictions or, at least, their accuracy. In this paper, we argue that fluency is product of this recognition process. In short, to recognise fluency is to infer that we have a precise ‘grip’ on the unfolding processes that generate our sensations. More specifically, we propose that it is changes in fluency — from unfelt to felt — that are both recognised and realised when updating predictions about precision. Unfelt fluency orients attention to unpredicted sensations, while felt fluency supervenes on—and contextualises—unfelt fluency; thereby rendering certain attentional processes, phenomenologically opaque. As such, fluency underwrites the precision we place in our predictions and therefore acts upon our perceptual inferences. Hence, the causes of conscious subjective inference have unconscious perceptual precursors.

Big number, big body: Jersey numbers alter body size perception.

Vision has been shown to be an active process that can be shaped by top-down influences. Here, we add to this area of research by showing a surprising example of how visual perception can be affected by cognition (i.e., cognitive penetration). Observers were presented, on each trial, with a picture of a computer-generated football player and asked to rate the slenderness of the player on an analog scale. The results of two experiments showed that observers perceived athletes wearing small jersey numbers as more slender than those with high numbers. This finding suggests that the cognition of numbers quantitatively alters body size perception. We conjecture that this effect is the result of previously learned associations (i.e., prior expectations) affecting perceptual inference. Such associations are likely the result of implicit learning of the statistical regularities of number and size attributes co-occurrences by the nervous system. We discuss how these results are consistent with previous research on statistical learning and how they fit into the Bayesian framework of perception. The current finding supports the notion of top-down influences of cognition on perception.

Award Lecture: 2021 Tillyer Award Lecture: The Early Visual Encoding: Computations and Consequences.

The visual system takes sensory measurements of the light incident at the eyes and uses these to make perceptual inferences about external world. The sensory measurements do not preserve all of the information available in the light signal. One approach to understanding the implications of the initial stages of visual processing is ideal observer analysis, which evaluates the information available to support psychophysical discriminations at various stages of the early visual representation. We are interested in extending this type of analysis to take into account the statistical structure of natural images. To do so, we developed an open-source computational model of the initial visual encoding, ISETBio (isetbio.org). ISETBio incorporates specification of visual displays, retinal image formation through the eye's optics, spatio-spectral sampling by the retinal cone mosaic, Poisson noise in the cone photopigment excitations, transduction of excitations to photocurrent, and fixational eye movements. In this talk, I will introduce ISETBio and illustrate a set of insights it enables about visual processing by reviewing a number of computational examples. These examples will include ways that combining ISETBio with Bayesian image-reconstruction methods helps us understand how the interaction of the visual encoding and the statistical structure of natural images shapes visual performance.

Same but different: The latency of a shared expectation signal interacts with stimulus attributes

Predictive coding theories assert that perceptual inference is a hierarchical process of belief updating, wherein the onset of unexpected sensory data causes so-called prediction error responses that calibrate erroneous inferences. Given the functionally specialised organisation of visual cortex, it is assumed that prediction error propagation interacts with the specific visual attribute violating an expectation. We sought to test this within the temporal domain by applying time-resolved decoding methods to electroencephalography (EEG) data evoked by contextual trajectory violations of either brightness, size, or orientation within a bound stimulus. We found that following ∼170 ms post stimulus onset, responses to both size violations and orientation violations were decodable from physically identical control trials in which no attributes were violated. These two violation types were then directly compared, with attribute-specific signalling being decoded from 265 ms. Temporal generalisation suggested that this dissociation was driven by latency shifts in shared expectation signalling between the two conditions. Using a novel temporal bias method, we then found that this shared signalling occurred earlier for size violations than orientation violations. To our knowledge, we are among the first to decode expectation violations in humans using EEG and have demonstrated a temporal dissociation in attribute-specific expectancy violations.

Multisensory decisions from self to world.

Classic Bayesian models of perceptual inference describe how an ideal observer would integrate 'unisensory' measurements (multisensory integration) and attribute sensory signals to their origin(s) (causal inference). However, in the brain, sensory signals are always received in the context of a multisensory bodily state-namely, in combination with other senses. Moreover, sensory signals from both interoceptive sensing of one's own body and exteroceptive sensing of the world are highly interdependent and never occur in isolation. Thus, the observer must fundamentally determine whether each sensory observation is from an external (versus internal, self-generated) source to even be considered for integration. Critically, solving this primary causal inference problem requires knowledge of multisensory and sensorimotor dependencies. Thus, multisensory processing is needed to separate sensory signals. These multisensory processes enable us to simultaneously form a sense of self and form distinct perceptual decisions about the external world. In this opinion paper, we review and discuss the similarities and distinctions between multisensory decisions underlying the sense of self and those directed at acquiring information about the world. We call attention to the fact that heterogeneous multisensory processes take place all along the neural hierarchy (even in forming 'unisensory' observations) and argue that more integration of these aspects, in theory and experiment, is required to obtain a more comprehensive understanding of multisensory brain function. This article is part of the theme issue 'Decision and control processes in multisensory perception'.