Sensory Cue Integration Research Articles

The Impact of Melanoma Imaging Biomarker Cues on Detection Sensitivity and Specificity in Melanoma versus Clinically Atypical Nevi.

Incorporation of dermoscopy and artificial intelligence (AI) is improving healthcare professionals' ability to diagnose melanoma earlier, but these algorithms often suffer from a "black box" issue, where decision-making processes are not transparent, limiting their utility for training healthcare providers. To address this, an automated approach for generating melanoma imaging biomarker cues (IBCs), which mimics the screening cues used by expert dermoscopists, was developed. This study created a one-minute learning environment where dermatologists adopted a sensory cue integration algorithm to combine a single IBC with a risk score built on many IBCs, then immediately tested their performance in differentiating melanoma from benign nevi. Ten participants evaluated 78 dermoscopic images, comprised of 39 melanomas and 39 nevi, first without IBCs and then with IBCs. Participants classified each image as melanoma or nevus in both experimental conditions, enabling direct comparative analysis through paired data. With IBCs, average sensitivity improved significantly from 73.69% to 81.57% (p = 0.0051), and the average specificity improved from 60.50% to 67.25% (p = 0.059) for the diagnosis of melanoma. The index of discriminability (d') increased significantly by 0.47 (p = 0.002). Therefore, the incorporation of IBCs can significantly improve physicians' sensitivity in melanoma diagnosis. While more research is needed to validate this approach across other healthcare providers, its use may positively impact melanoma screening practices.

Chaotic neural dynamics facilitate probabilistic computations through sampling

Cortical neurons exhibit highly variable responses over trials and time. Theoretical works posit that this variability arises potentially from chaotic network dynamics of recurrently connected neurons. Here, we demonstrate that chaotic neural dynamics, formed through synaptic learning, allow networks to perform sensory cue integration in a sampling-based implementation. We show that the emergent chaotic dynamics provide neural substrates for generating samples not only of a static variable but also of a dynamical trajectory, where generic recurrent networks acquire these abilities with a biologically plausible learning rule through trial and error. Furthermore, the networks generalize their experience in the stimulus-evoked samples to the inference without partial or all sensory information, which suggests a computational role of spontaneous activity as a representation of the priors as well as a tractable biological computation for marginal distributions. These findings suggest that chaotic neural dynamics may serve for the brain function as a Bayesian generative model.

Effects of older age on visual and self-motion sensory cue integration in navigation.

Older adults demonstrate impairments in navigation that cannot be explained by general cognitive and motor declines. Previous work has shown that older adults may combine sensory cues during navigation differently than younger adults, though this work has largely been done in dark environments where sensory integration may differ from full-cue environments. Here, we test whether aging adults optimally combine cues from two sensory systems critical for navigation: vision (landmarks) and body-based self-motion cues. Participants completed a homing (triangle completion) task using immersive virtual reality to offer the ability to navigate in a well-lit environment including visibility of the ground plane. An optimal model, based on principles of maximum-likelihood estimation, predicts that precision in homing should increase with multisensory information in a manner consistent with each individual sensory cue's perceived reliability (measured by variability). We found that well-aging adults (with normal or corrected-to-normal sensory acuity and active lifestyles) were more variable and less accurate than younger adults during navigation. Both older and younger adults relied more on their visual systems than a maximum likelihood estimation model would suggest. Overall, younger adults' visual weighting matched the model's predictions whereas older adults showed sub-optimal sensory weighting. In addition, high inter-individual differences were seen in both younger and older adults. These results suggest that older adults do not optimally weight each sensory system when combined during navigation, and that older adults may benefit from interventions that help them recalibrate the combination of visual and self-motion cues for navigation.

Experimentally disambiguating models of sensory cue integration.

Sensory cue integration is one of the primary areas in which a normative mathematical framework has been used to define the “optimal” way in which to make decisions based upon ambiguous sensory information and compare these predictions to behavior. The conclusion from such studies is that sensory cues are integrated in a statistically optimal fashion. However, numerous alternative computational frameworks exist by which sensory cues could be integrated, many of which could be described as “optimal” based on different criteria. Existing studies rarely assess the evidence relative to different candidate models, resulting in an inability to conclude that sensory cues are integrated according to the experimenter's preferred framework. The aims of the present paper are to summarize and highlight the implicit assumptions rarely acknowledged in testing models of sensory cue integration, as well as to introduce an unbiased and principled method by which to determine, for a given experimental design, the probability with which a population of observers behaving in accordance with one model of sensory integration can be distinguished from the predictions of a set of alternative models.

Deep learning-level melanoma detection by interpretable machine learning and imaging biomarker cues.

.Significance: Melanoma is a deadly cancer that physicians struggle to diagnose early because they lack the knowledge to differentiate benign from malignant lesions. Deep machine learning approaches to image analysis offer promise but lack the transparency to be widely adopted as stand-alone diagnostics.Aim: We aimed to create a transparent machine learning technology (i.e., not deep learning) to discriminate melanomas from nevi in dermoscopy images and an interface for sensory cue integration.Approach: Imaging biomarker cues (IBCs) fed ensemble machine learning classifier (Eclass) training while raw images fed deep learning classifier training. We compared the areas under the diagnostic receiver operator curves.Results: Our interpretable machine learning algorithm outperformed the leading deep-learning approach 75% of the time. The user interface displayed only the diagnostic imaging biomarkers as IBCs.Conclusions: From a translational perspective, Eclass is better than convolutional machine learning diagnosis in that physicians can embrace it faster than black box outputs. Imaging biomarkers cues may be used during sensory cue integration in clinical screening. Our method may be applied to other image-based diagnostic analyses, including pathology and radiology.

Interhemispheric control of sensory cue integration and self-motion perception

Spatial orientation necessitates the integration of visual and vestibular sensory cues, in-turn facilitating self-motion perception. However, the neural mechanisms underpinning sensory integration remain unknown. Recently we have illustrated that spatial orientation and vestibular thresholds are influenced by interhemispheric asymmetries associated with the posterior parietal cortices (PPC) that predominantly house the vestibulo-cortical network. Given that sensory integration is a prerequisite to both spatial orientation and motion perception, we hypothesized that sensory integration is similarly subject to interhemispheric influences. Accordingly, we explored the relationship between vestibulo-cortical dominance – assessed using a biomarker, the degree of vestibular-nystagmus suppression following transcranial direct current stimulation over the PPC – with visual dependence measures obtained during performance of a sensory integration task (the rod-and-disk task). We observed that the degree of visual dependence was correlated with vestibulo-cortical dominance. Specifically, individuals with greater right hemispheric vestibulo-cortical dominance had reduced visual dependence. We proceeded to assess the significance of such dominance on behavior by correlating measures of visual dependence with self-motion perception in healthy subjects. We observed that right-handed individuals experienced illusionary self-motion (vection) quicker than left-handers and that the degree of vestibular cortical dominance was correlated with the time taken to experience vection, only during conditions that induced interhemispheric conflict. To conclude, we demonstrate that interhemispheric asymmetries associated with vestibulo-cortical processing in the PPC functionally and mechanistically link sensory integration and self-motion perception, facilitating spatial orientation. Our findings highlight the importance of dynamic interhemispheric competition upon control of vestibular behavior in humans.

Concurrent intramodal learning enhances multisensory responses of symmetric crossmodal learning in robotic audio-visual tracking

Tracking an audio-visual target involves integrating spatial cues about target position from both modalities. Such sensory cue integration is a developmental process in the brain involving learning, with neuroplasticity as its underlying mechanism. We present a Hebbian learning-based adaptive neural circuit for multi-modal cue integration. The circuit temporally correlates stimulus cues within each modality via intramodal learning as well as symmetrically across modalities via crossmodal learning to independently update modality-specific neural weights on a sample-by-sample basis. It is realised as a robotic agent that must orient towards a moving audio-visual target. It continuously learns the best possible weights required for a weighted combination of auditory and visual spatial target directional cues that is directly mapped to robot wheel velocities to elicit an orientation response. Visual directional cues are noise-free and continuous but arising from a relatively narrow receptive field while auditory directional cues are noisy and intermittent but arising from a relatively wider receptive field. Comparative trials in simulation demonstrate that concurrent intramodal learning improves both the overall accuracy and precision of the orientation responses of symmetric crossmodal learning. We also demonstrate that symmetric crossmodal learning improves multisensory responses as compared to asymmetric crossmodal learning. The neural circuit also exhibits multisensory effects such as sub-additivity, additivity and super-additivity.

Experimentally disambiguating models of sensory cue combination

Sensory cue integration is one of the primary areas in which a normative mathematical framework has been used to define the “optimal” way in which to make decisions based upon ambiguous sensory information and compare these predictions to behaviour. The conclusion from such studies is that sensory cues are integrated in a statistically optimal fashion. However, numerous alternative computational frameworks exist by which sensory cues could be integrated, many of which could be described as “optimal” based on different criteria. Existing studies rarely assess the evidence relative to different candidate models, resulting in an inability to conclude that sensory cues are integrated according to the experimenter’s preferred framework. The aims of the present paper are to summarise and highlight the implicit assumptions rarely acknowledged in testing models of sensory cue integration, as well as to introduce an unbiased and principled method by which to determine, for a given experimental design, the probability with which a population of observers behaving in accordance with one model of sensory integration can be distinguished from the predictions of a set of alternative models.

Cybersickness: a Multisensory Integration Perspective.

In the past decade, there has been a rapid advance in Virtual Reality (VR) technology. Key to the user's VR experience are multimodal interactions involving all senses. The human brain must integrate real-time vision, hearing, vestibular and proprioceptive inputs to produce the compelling and captivating feeling of immersion in a VR environment. A serious problem with VR is that users may develop symptoms similar to motion sickness, a malady called cybersickness. At present the underlying cause of cybersickness is not yet fully understood. Cybersickness may be due to a discrepancy between the sensory signals which provide information about the body's orientation and motion: in many VR applications, optic flow elicits an illusory sensation of motion which tells users that they are moving in a certain direction with certain acceleration. However, since users are not actually moving, their proprioceptive and vestibular organs provide no cues of self-motion. These conflicting signals may lead to sensory discrepancies and eventually cybersickness. Here we review the current literature to develop a conceptual scheme for understanding the neural mechanisms of cybersickness. We discuss an approach to cybersickness based on sensory cue integration, focusing on the dynamic re-weighting of visual and vestibular signals for self-motion.

Crossmodal Correspondences: Standing Issues and Experimental Guidelines

Crossmodal correspondences refer to the systematic associations often found across seemingly unrelated sensory features from different sensory modalities. Such phenomena constitute a universal trait of multisensory perception even in non-human species, and seem to result, at least in part, from the adaptation of sensory systems to natural scene statistics. Despite recent developments in the study of crossmodal correspondences, there are still a number of standing questions about their definition, their origins, their plasticity, and their underlying computational mechanisms. In this paper, I will review such questions in the light of current research on sensory cue integration, where crossmodal correspondences can be conceptualized in terms of natural mappings across different sensory cues that are present in the environment and learnt by the sensory systems. Finally, I will provide some practical guidelines for the design of experiments that might shed new light on crossmodal correspondences.

Weighted cue integration in the rodent head direction system

How the brain combines information from different sensory modalities and of differing reliability is an important and still-unanswered question. Using the head direction (HD) system as a model, we explored the resolution of conflicts between landmarks and background cues. Sensory cue integration models predict averaging of the two cues, whereas attractor models predict capture of the signal by the dominant cue. We found that a visual landmark mostly captured the HD signal at low conflicts: however, there was an increasing propensity for the cells to integrate the cues thereafter. A large conflict presented to naive rats resulted in greater visual cue capture (less integration) than in experienced rats, revealing an effect of experience. We propose that weighted cue integration in HD cells arises from dynamic plasticity of the feed-forward inputs to the network, causing within-trial spatial redistribution of the visual inputs onto the ring. This suggests that an attractor network can implement decision processes about cue reliability using simple architecture and learning rules, thus providing a potential neural substrate for weighted cue integration.

Ideal-observer models of perceptual contrast enhancement

In standard ideal-observer models of sensory cue integration [1], the perceptual estimate resulting from the combination of two cues lies in the interval bounded by the estimates of each cue separately. For example, this type of model accounts well for the psychophysical result - observers give an estimate in-between the haptic-alone and the visual-alone estimates, when asked to estimate ridges height with both vision and touch [2]. Nevertheless, a class of perceptual illusion is supposedly not accounted for by this type of model, namely contrast illusion, such as the size-weight illusion [3,4]. In the size-weight illusion, when asked to estimate the weight of two objects of the same mass but not the same size, observers estimate the larger as lighter. Using standard ideal-observer models, we showed that it is possible to account for this class of illusion provided that statistical correlation between each cue estimate is taken into account. Our argument is based on statistical inference models such as linear minimum-variance unbiased estimation, maximum a posteriori estimation, and least relative surprise estimation. This psychophysical model is general as long as the perceptual estimate deals with a physical quantity that is proportional to another physical quantity also available as a cue, such as mass and volume for a given material in the size-weight illusion.

Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons

The richness of perceptual experience, as well as its usefulness for guiding behaviour, depends on the synthesis of information across multiple senses. Recent decades have witnessed a surge in our understanding of how the brain combines sensory cues. Much of this research has been guided by one of two distinct approaches: one is driven primarily by neurophysiological observations, and the other is guided by principles of mathematical psychology and psychophysics. Conflicting results and interpretations have contributed to a conceptual gap between psychophysical and physiological accounts of cue integration, but recent studies of visual-vestibular cue integration have narrowed this gap considerably.

Review: Sensory Cue Integration

Review: Sensory Cue Integration

Visual-haptic cue integration with spatial and temporal disparity during pointing movements

Many perceptual cue combination studies have shown that humans can integrate sensory information across modalities as well as within a modality in a manner that is close to optimal. While the limits of sensory cue integration have been extensively studied in the context of perceptual decision tasks, the evidence obtained in the context of motor decisions provides a less consistent picture. Here, we studied the combination of visual and haptic information in the context of human arm movement control. We implemented a pointing task in which human subjects pointed at an invisible unknown target position whose vertical position varied randomly across trials. In each trial, we presented a haptic and a visual cue that provided noisy information about the target position half-way through the reach. We measured pointing accuracy as function of haptic and visual cue onset and compared pointing performance to the predictions of a multisensory decision model. Our model accounts for pointing performance by computing the maximum a posteriori estimate, assuming minimum variance combination of uncertain sensory cues. Synchronicity of cue onset has previously been demonstrated to facilitate the integration of sensory information. We tested this in trials in which visual and haptic information was presented with temporal disparity. We found that for our sensorimotor task temporal disparity between visual and haptic cue had no effect. Sensorimotor learning appears to use all available information and to apply the same near-optimal rules for cue combination that are used by perception.