Multisensory Signals Research Articles

The Relationship Between Cognition and Sensorimotor Behavior in an F1 Driving Simulation: An Explorative Study.

Sensorimotor control simultaneously engages multiple cognitive processes, like decision making, intention, processing, and the integration of multisensory signals. The reciprocal relationship of cognition and sensorimotor learning is well documented. However, little is known if the status of cognitive skills relates to immediate sensorimotor performance of performing a novel skill. Thus, we aim to explore whether cognitive skills in general and executive functions (EFs) in particular may relate to novel sensorimotor performance and adaptive skills. Therefore, 23 male participants engaged in a novel driving simulation for 2 days. On the first day, they accustomed to the F1 simulation until meeting a preset threshold (adaption). On the second day, they aimed to drive as fast as possible (performance). In addition, we measured EFs and global cognition. We found meaningful relationships between response inhibition (Stroop Color and Word Test), the driving performance (r = 0.48, p = 0.013), and the adaptive ability (r = 0.34, p = 0.012). All other tests of executive functioning and global cognition remained non-significant. Our results illustrate an association of driving performance and adaptive abilities and the EF selective attention/inhibition in a novel F1 simulation. Given the novelty of the task, the ability to adjust sensorimotor behavior to keep the car on the track seems to be the primary necessary skill to navigate the lap and achieve fast times.

Antero-Posterior vs. Lateral Vestibular Input Processing in Human Visual Cortex.

Visuo-vestibular integration is crucial for locomotion, yet the cortical mechanisms involved remain poorly understood. We combined binaural monopolar galvanic vestibular stimulation (GVS) and functional magnetic resonance imaging (fMRI) to characterize the cortical networks activated during antero-posterior and lateral stimulations in humans. We focused on functional areas that selectively respond to egomotion-consistent optic flow patterns: the human middle temporal complex (hMT+), V6, the ventral intraparietal (VIP) area, the cingulate sulcus visual (CSv) area and the posterior insular cortex (PIC). Areas hMT+, CSv, and PIC were equivalently responsive during lateral and antero-posterior GVS while areas VIP and V6 were highly activated during antero-posterior GVS, but remained silent during lateral GVS. Using psychophysiological interaction (PPI) analyses, we confirmed that a cortical network including areas V6 and VIP is engaged during antero-posterior GVS. Our results suggest that V6 and VIP play a specific role in processing multisensory signals specific to locomotion during navigation.

A Continuous Illusion of Having a Sixth Finger.

Our body is central to our sense of self and personal identity, yet it can be manipulated in the laboratory in surprisingly easy ways. Several multisensory illusions have shown the flexibility of the mental representation of our bodies by inducing the illusion of owning an artificial body part or having a body part with altered features. Recently, new studies showed we can embody additional body parts such as a supernumerary finger. Newport etal. recently reported a novel six-finger illusion using conflicting visual and tactile signals induced with the mirror box to create the illusory perception of having a sixth finger for a brief moment. In this study, we aimed to replicate this result and to investigate whether the experience of embodiment of a sixth finger could be prolonged for an extended duration by applying continuous visual-tactile stimulation. Results showed that a continuous illusion of having a sixth finger can be clearly induced. This shows that the six-finger illusion does not reflect merely a momentary confusion due to conflicting multisensory signals but can reflect an enduring representation of a supernumerary finger.

Conscious Multisensory Integration: Introducing a Universal Contextual Field in Biological and Deep Artificial Neural Networks.

Conscious awareness plays a major role in human cognition and adaptive behavior, though its function in multisensory integration is not yet fully understood, hence, questions remain: How does the brain integrate the incoming multisensory signals with respect to different external environments? How are the roles of these multisensory signals defined to adhere to the anticipated behavioral-constraint of the environment? This work seeks to articulate a novel theory on conscious multisensory integration (CMI) that addresses the aforementioned research challenges. Specifically, the well-established contextual field (CF) in pyramidal cells and coherent infomax theory (Kay et al., 1998; Kay and Phillips, 2011) is split into two functionally distinctive integrated input fields: local contextual field (LCF) and universal contextual field (UCF). LCF defines the modulatory sensory signal coming from some other parts of the brain (in principle from anywhere in space-time) and UCF defines the outside environment and anticipated behavior (based on past learning and reasoning). Both LCF and UCF are integrated with the receptive field (RF) to develop a new class of contextually-adaptive neuron (CAN), which adapts to changing environments. The proposed theory is evaluated using human contextual audio-visual (AV) speech modeling. Simulation results provide new insights into contextual modulation and selective multisensory information amplification/suppression. The central hypothesis reviewed here suggests that the pyramidal cell, in addition to the classical excitatory and inhibitory signals, receives LCF and UCF inputs. The UCF (as a steering force or tuner) plays a decisive role in precisely selecting whether to amplify/suppress the transmission of relevant/irrelevant feedforward signals, without changing the content e.g., which information is worth paying more attention to? This, as opposed to, unconditional excitatory and inhibitory activity in existing deep neural networks (DNNs), is called conditional amplification/suppression.

The role of context in experiments and models of multisensory decision making

The availability of signals from multiple senses is often beneficial for perceptual decisions. To study such benefits, models of multisensory decision-making are typically fed with the behavioural performance as measured separately with unisensory component signals. Critically, by doing so, the approach implicitly makes the so-called context invariance assumption, which states that processing of a signal is independent of the experimental context in which it is embedded. However, context invariance is not necessarily true and is difficult to test directly. Here, we aim to assess context invariance indirectly in two testable scenarios. First, to consider the role of stimulus context, we compared unisensory performance in trials that either included a task-irrelevant signal in another modality, or not (unisensory vs. multisensory signal trials). We found that performance was faster but less sensitive in trials that contained a task-irrelevant signal. Hence, stimulus context invariance was violated. Second, to consider the role of instruction context, we compared unisensory performance when participants were asked to detect targets from either one or two modalities (unisensory vs. multisensory instructions). We found that performance was deteriorated in multi- compared to unisensory instructions, which was largely due to modality switch costs in multisensory instructions. Hence, instruction context invariance did not hold either. As performance was variant in both scenarios, context invariance cannot generally assumed to be true. We conclude that models of multisensory decision making should critically consider potential violations of the context invariance assumption as a potentially confounding factor.

Multisensory Integration Develops Prior to Crossmodal Recalibration.

It has been hypothesized that crossmodal recalibration plays a crucial role for the development of multisensory integration capabilities [1]. To test the developmental trajectory of multisensory integration and crossmodal recalibration, we used a combined ventriloquist/ventriloquist aftereffect paradigm [2] in children aged 5-9 years. The ventriloquist effect (indicating multisensory integration), that is, the shift of auditory localization toward simultaneously presented but spatially discrepant visual stimuli, was larger in children than in adults, which was attributed to a lower auditory localization precision in the children. In fact, the size of the ventriloquist effect depended on the visual stimulus reliability in both children and adults. In all groups, the ventriloquist effect was best explained by a causal inference model. In contrast to their multisensory integration capabilities, 5-year-old children did not recalibrate. The immediate ventriloquist aftereffect (indicating recalibration after a single exposure to a spatially discrepant audio-visual stimulus) emerged in 6- to 7-year-old children, whereas the cumulative ventriloquist aftereffect (reflecting recalibration to the audio-visual spatial discrepancies over the complete experiment) was not observed before the age of 8 years. First, in contrast to common beliefs, the present results provide evidence that multisensory integration precedes rather than follows crossmodal recalibration during development. Second, we report developmental evidence for a dissociation of the processes involved in multisensory integration and immediate as well as cumulative recalibration. We speculate that multisensory integration is a prerequisite for crossmodal recalibration, because the multisensory percept, rather than unimodal cues, might comprise a crucial signal for the calibration of the sensory systems.

Redundant visual signals reduce the intensity of alcohol impairment

BackgroundHumans interact with multiple stimuli across several modalities each day. The “redundant signal effect” refers to the observation that individuals respond more quickly to stimuli when information is presented as multisensory, redundant stimuli (e.g., aurally and visually), rather than as a single stimulus presented to either modality alone. Studies of alcohol effects on human performance show that alcohol induced impairment is reduced when subjects respond to redundant multisensory stimuli. However, redundant signals do not need to involve multisensory stimuli to facilitate behavior as studies have shown facilitating effects by redundant unisensory signals that are delivered to the “same sensory” (e.g., two visual or two auditory signals). MethodsThe current study examined the degree to which redundant visual signals would reduce alcohol impairment and compared the magnitude of this effect with that produced by redundant multisensory signals. On repeated test sessions, participants (n = 20) received placebo or 0.65 g/kg alcohol and performed a two-choice reaction time task that measured how quickly participants responded to four different signal conditions. The four conditions differed by the modality of the target presentation: visual, auditory, multisensory, and unisensory. ResultsAlcohol slowed performance in all conditions and reaction times were generally faster in the redundant signal conditions. Both multisensory and unisensory redundant signals reduced the impairing effects of alcohol compared with single signals. ConclusionsThese findings indicate that the ability of redundant signals to counteract alcohol impairment does not require multisensory input. Duplicate signals to the same modality can also reduce alcohol impairment.

Toward Enhanced Teleoperation Through Embodiment.

Telerobotics aims to transfer human manipulation skills and dexterity over an arbitrary distance and at an arbitrary scale to a remote workplace. A telerobotic system that is transparent enables a natural and intuitive interaction. We postulate that embodiment (with three sub-components: sense of ownership, agency, and self-location) of the robotic system leads to optimal perceptual transparency and increases task performance. However, this has not yet been investigated directly. We reason along four premises and present findings from the literature that substantiate each of them: (1) the brain can embody non-bodily objects (e.g., robotic hands), (2) embodiment can be elicited with mediated sensorimotor interaction, (3) embodiment is robust against inconsistencies between the robotic system and the operator's body, and (4) embodiment positively correlates to dexterous task performance. We use the predictive encoding theory as a framework to interpret and discuss the results reported in the literature. Numerous previous studies have shown that it is possible to induce embodiment over a wide range of virtual and real extracorporeal objects (including artificial limbs, avatars, and android robots) through mediated sensorimotor interaction. Also, embodiment can occur for non-human morphologies including for elongated arms and a tail. In accordance with the predictive encoding theory, none of the sensory modalities is critical in establishing ownership, and discrepancies in multisensory signals do not necessarily lead to loss of embodiment. However, large discrepancies in terms of multisensory synchrony or visual likeness can prohibit embodiment from occurring. The literature provides less extensive support for the link between embodiment and (dexterous) task performance. However, data gathered with prosthetic hands do indicate a positive correlation. We conclude that all four premises are supported by direct or indirect evidence in the literature, suggesting that embodiment of a remote manipulator may improve dexterous performance in telerobotics. This warrants further implementation testing of embodiment in telerobotics. We formulate a first set of guidelines to apply embodiment in telerobotics and identify some important research topics.

Stereotactic electroencephalography in humans reveals multisensory signal in early visual and auditory cortices

Unequivocally demonstrating the presence of multisensory signals at the earliest stages of cortical processing remains challenging in humans. In our study, we relied on the unique spatio-temporal resolution provided by intracranial stereotactic electroencephalographic (SEEG) recordings in patients with drug-resistant epilepsy to characterize the signal extracted from early visual (calcarine and pericalcarine) and auditory (Heschl's gyrus and planum temporale) regions during a simple audio-visual oddball task. We provide evidences that both cross-modal responses (visual responses in auditory cortex or the reverse) and multisensory processing (alteration of the unimodal responses during bimodal stimulation) can be observed in intracranial event-related potentials (iERPs) and in power modulations of oscillatory activity at different temporal scales within the first 150 msec after stimulus onset. The temporal profiles of the iERPs are compatible with the hypothesis that MSI occurs by means of direct pathways linking early visual and auditory regions. Our data indicate, moreover, that MSI mainly relies on modulations of the low-frequency bands (foremost the theta band in the auditory cortex and the alpha band in the visual cortex), suggesting the involvement of feedback pathways between the two sensory regions. Remarkably, we also observed high-gamma power modulations by sounds in the early visual cortex, thus suggesting the presence of neuronal populations involved in auditory processing in the calcarine and pericalcarine region in humans.

Divisively Normalized Integration of Multisensory Error Information Develops Motor Memories Specific to Vision and Proprioception.

Both visual and proprioceptive information contribute to the accuracy of limb movement, but the mechanism of integration of these different modality signals for movement control and learning remains controversial. We aimed to elucidate the mechanism of multisensory integration for motor adaptation by evaluating single-trial adaptation (i.e., aftereffect) induced by visual and proprioceptive perturbations while male and female human participants performed reaching movements. The force-channel method was used to precisely impose several combinations of visual and proprioceptive perturbations (i.e., error), including an instance when the directions of perturbation in both stimuli opposed each another. In the subsequent probe force-channel trial, the lateral force against the channel was quantified as the aftereffect to clarify the mechanism by which the motor adaptation system corrects movement in the event of visual and proprioceptive errors. We observed that the aftereffects had complex dependence on the visual and proprioceptive errors. Although this pattern could not be explained by previously proposed computational models based on the reliability of sensory information, we found that it could be reasonably explained by a mechanism known as divisive normalization, which was the reported mechanism underlying the integration of multisensory signals in neurons. Furthermore, we discovered evidence that the motor memory for each sensory modality developed separately in accordance with a divisive normalization mechanism and that the outputs of both memories were integrated. These results provide a novel view of the utilization and integration of different sensory modality signals in motor adaptation.SIGNIFICANCE STATEMENT The mechanism of utilization of multimodal sensory information by the motor control system to perform limb movements with accuracy is a fundamental question. However, the mechanism of integration of these different sensory modalities for movement control and learning remains highly debatable. Herein, we demonstrate that multisensory integration in the motor learning system can be reasonably explained by divisive normalization, a canonical computation, ubiquitously observed in the brain (Carandini and Heeger, 2011). Moreover, we provide evidence of a novel idea that integration does not occur at the sensory information processing level, but at the motor execution level, after the motor memory for each sensory modality is separately created.

Visual body form and orientation cues do not modulate visuo-tactile temporal integration.

Body ownership relies on spatiotemporal correlations between multisensory signals and visual cues specifying oneself such as body form and orientation. The mechanism for the integration of bodily signals remains unclear. One approach to model multisensory integration that has been influential in the multisensory literature is Bayesian causal inference. This specifies that the brain integrates spatial and temporal signals coming from different modalities when it infers a common cause for inputs. As an example, the rubber hand illusion shows that visual form and orientation cues can promote the inference of a common cause (one's body) leading to spatial integration shown by a proprioceptive drift of the perceived location of the real hand towards the rubber hand. Recent studies investigating the effect of visual cues on temporal integration, however, have led to conflicting findings. These could be due to task differences, variation in ecological validity of stimuli and/or small samples. In this pre-registered study, we investigated the influence of visual information on temporal integration using a visuo-tactile temporal order judgement task with realistic stimuli and a sufficiently large sample determined by Bayesian analysis. Participants viewed videos of a touch being applied to plausible or implausible visual stimuli for one's hand (hand oriented plausibly, hand rotated 180 degrees, or a sponge) while also being touched at varying stimulus onset asynchronies. Participants judged which stimulus came first: viewed or felt touch. Results show that visual cues do not modulate visuo-tactile temporal order judgements. This is not in line with the idea that bodily signals indicating oneself influence the integration of multisensory signals in the temporal domain. The current study emphasises the importance of rigour in our methodologies and analyses to advance the understanding of how properties of multisensory events affect the encoding of temporal information in the brain.

Relationships Between Personality Features and the Rubber Hand Illusion: An Exploratory Study.

The rubber hand illusion paradigm allows investigating human body ownership by inducing an illusion of owning a life-sized fake hand. Despite the wide consensus on the fact that integration of multisensory signals is the main interpretative framework of the rubber hand illusion, increasing amount of data show that additional factors might contribute to the emergence of the illusion and, in turn, explain the strong inter-individual differences of the illusory patterns. Here, we explored whether and how personality features contribute to the emergence of the illusion by administering to healthy participants the rubber hand illusion paradigm along with two well-known personality tests, i.e., the Personality Assessment Inventory (PAI) and the Rorschach test. Results showed that two Rorschach domains (i.e., “Perception and Thinking Problems” and “Self and Other Representation”) were positively correlated with the illusory mislocalization of the own left hand toward the fake hand. Further analyses suggested that while the tendency to perceive unconventionally is related to mislocalizing the own hand toward the fake hand, the association of the RHI index and other personality features measured by the Rorschach remain uncertain. However, our findings in general suggest that personality features might have a role in the emergence of the rubber hand illusion. This, in turn, could explain the high inter-individual variability of the illusory effects.

A Deep Coupled Network for Health State Assessment of Cutting Tools Based on Fusion of Multisensory Signals

The cutting tool is a key part of a machine system, which plays an important role in modern manufacturing systems. To avoid an unexpected tool failure, it is necessary to carry out health condition assessment of cutting tools. In this paper, a deep coupled restricted Boltzmann machine (DCRBM) is proposed for health state assessment of cutting tools based on fusion of vibration signals and acoustic emission (AE) signals. Because of the complementary of multisensory signals, it is necessary to develop a fusion strategy for the fusion of multisource signals. The proposed DCRBM is symmetric with each side consisting of several hidden layers and one coupled layer, which is constructed by two basic restricted Boltzmann machines with similarity constraints. Vibration signals and AE signals, which are connected with the two sides of DCRBM, respectively, are mapped into a feature space, where similar representations are learned. The parameters of the deep architecture are learned by optimizing the new objective function. Experimental results on fusion of vibration signals and AE signals demonstrate the promising performance of DCRBM for health state assessment of cutting tools compared with other fusion strategies.

Multisensory integration in virtual interactions with distant objects

Statistically optimal integration of multimodal signals is known to take place in direct interactions with environmental objects. In the present study we tested whether the same mechanism is responsible for perceptual biases observed in a task, in which participants enclose visual objects by manually controlled visual cursors. We manipulated the relative reliability of visual object information and measured the impact of body-related information on object perception as well as the perceptual variability. The results were qualitatively consistent with statistically optimal sensory integration. However, quantitatively, the observed bias and variability measures systematically differed from the model predictions. This outcome indicates a compensatory mechanism similar to the reliability-based weighting of multisensory signals which could underlie action’s effects in visual perception reported in diverse context conditions.

Using Multisensory Haptic Integration to Improve Monitoring in the Intensive Care Unit

ABSTRACT Introduction Alarm fatigue and medical alarm mismanagement reduces the quality of patient care and creates stressful work environments for clinicians. Here, the feasibility of a novel “pre-alarm” system that utilizes multisensory integration of auditory and haptic stimuli is examined as a possible solution. Methods Three vital signs (heart rate, blood pressure, and blood oxygenation) were represented by three musically distinct sounds that were combined into soundscapes and progressed through five pre-alarm zones (very low to very high). Three haptic conditions were tested with the auditory stimulus to determine the best combination of auditory and haptic stimulation. Qualitative data was collected through surveys and the NASA TLX index. Results Alterations in frequency and timbre were most effective at transmitting information regarding changing vital sign zones with comparatively higher accuracy and quicker reaction time (RT), p <.01. The addition of haptic stimuli to the auditory soundscape caused no significant decline in study participant accuracy or RT. However, two weeks after training, participants performed the tasks significantly faster (p <.001) and felt the alarm monitoring task was significantly less cognitively demanding (p <.01), compared to the unisensory condition. Participants also felt more confident in identifying changing vital signs with the addition of haptic stimuli. Discussion The current study demonstrates that multisensory signals do not diminish the perception of transmitted information and suggest efficient training benefits over unimodal signals. Multisensory training may be beneficial over time compared to unisensory training due to a stronger consolidation effect. The potential integration of haptic input with existing auditory alarm systems and training is supported.

Tool wear predicting based on multi-domain feature fusion by deep convolutional neural network in milling operations

Tool wear monitoring has been increasingly important in intelligent manufacturing to increase machining efficiency. Multi-domain features can effectively characterize tool wear condition, but manual feature fusion lowers monitoring efficiency and hinders the further improvement of predicting accuracy. In order to overcome these deficiencies, a new tool wear predicting method based on multi-domain feature fusion by deep convolutional neural network (DCNN) is proposed in this paper. In this method, multi-domain (including time-domain, frequency domain and time–frequency domain) features are respectively extracted from multisensory signals (e.g. three-dimensional cutting force and vibration) as health indictors of tool wear condition, then the relationship between these features and real-time tool wear is directly established based on the designed DCNN model to combine adaptive feature fusion with automatic continuous prediction. The performance of the proposed tool wear predicting method is experimentally validated by using three tool run-to-failure datasets measured from three-flute ball nose tungsten carbide cutter of high-speed CNC machine under dry milling operations. The experimental results show that the predicting accuracy of the proposed method is significantly higher than other advanced methods.

Multisensory digital colorimetry to identify and determination of active substances in drugs

The original approach was developed and used for screening analysis of more than 40 pharmaceuticals (nonsteroidal analgesics and anti-inflammatory drugs, β-adrenoblockers, secretolytics, fluoroquinolones, metabolic stimulants and others) by digital colorimetry with the use of office flatbed scanner. The proposed technique is based on the simultaneous several colorimetric reactions with different molecular sensors placed in a 96-well plate, and its subsequent optical scanning. Computer processing of the bitmap image and multisensory signal discretization allows to obtain a unique barcode suitable for identification and determination of active substances in drugs. This approach can be used for testing the quality of pharmaceuticals.

A multi-sensory code for emotional arousal.

People express emotion using their voice, face and movement, as well as through abstract forms as in art, architecture and music. The structure of these expressions often seems intuitively linked to its meaning: romantic poetry is written in flowery curlicues, while the logos of death metal bands use spiky script. Here, we show that these associations are universally understood because they are signalled using a multi-sensory code for emotional arousal. Specifically, variation in the central tendency of the frequency spectrum of a stimulus-its spectral centroid-is used by signal senders to express emotional arousal, and by signal receivers to make emotional arousal judgements. We show that this code is used across sounds, shapes, speech and human body movements, providing a strong multi-sensory signal that can be used to efficiently estimate an agent's level of emotional arousal.

Quantifying Facial Expression Intensity and Signal Use in Deaf Signers.

We live in a world of rich dynamic multisensory signals. Hearing individuals rapidly and effectively integrate multimodal signals to decode biologically relevant facial expressions of emotion. Yet, it remains unclear how facial expressions are decoded by deaf adults in the absence of an auditory sensory channel. We thus compared early and profoundly deaf signers (n= 46) with hearing nonsigners (n= 48) on a psychophysical task designed to quantify their recognition performance for the six basic facial expressions of emotion. Using neutral-to-expression image morphs and noise-to-full signal images, we quantified the intensity and signal levels required by observers to achieve expression recognition. Using Bayesian modeling, we found that deaf observers require more signal and intensity to recognize disgust, while reaching comparable performance for the remaining expressions. Our results provide a robust benchmark for the intensity and signal use in deafness and novel insights into the differential coding of facial expressions of emotion between hearing and deaf individuals.

Zona incerta GABAergic neurons integrate prey-related sensory signals and induce an appetitive drive to promote hunting.

The neural substrates for predatory hunting, an evolutionarily conserved appetitive behavior, remain largely undefined. Photoactivation of zona incerta (ZI) GABAergic neurons strongly promotes hunting of both live and artificial prey. Conversely, photoinhibition of these neurons or deletion of their GABA function severely impairs hunting. Here electrophysiological recordings reveal that ZI neurons integrate prey-related multisensory signals and discriminate prey from non-prey targets. Visual or whisker sensory deprivation reduces calcium responses induced by prey introduction and attack and impair hunting. ZI photoactivation largely corrects the hunting impairment caused by sensory deprivations. Motivational and reinforcing assays reveal that ZI photoactivation is associated with a strong appetitive drive, causing repetitive self-stimulatory behaviors. These ZI neurons project to the periaqueductal gray matter to induce hunting and motivation. Thus, we have delineated the function of ZI GABAergic neurons in hunting, which integrates prey-related sensory signals into prey detection and attack and induces a strong appetitive motivational drive.