Multisensory Effects Research Articles

Multisensory effects of mask wearing on speech intelligibility and the benefit of multilingualism.

Due to the pandemic of the Covid-19 disease, it became common to wear masks on some public spaces. By covering mouth and nose, visual-related speech cues are greatly reduced, while the auditory signal is both distorted and attenuated. The present study aimed to analyze the multisensory effects of mask wearing on speech intelligibility and the differences in these effects between participants who spoke 1, 2 and 3 languages. The study consisted of the presentation of sentences from the SPIN test to 40 participants. Participants were asked to report the perceived sentences. There were four conditions: auditory with mask; audiovisual with mask; auditory without mask; audiovisual without mask. Two sessions were conducted, one week apart, each with the same stimuli but with a different signal-to-noise ratio. Results demonstrated that the use of the mask decreased speech intelligibility, both due to a decrease in the quality of auditory stimuli and due to the loss of visual information. Signal-to-noise ratio largely affects speech intelligibility and higher ratios are needed in mask-wearing conditions to obtain any degree of intelligibility. Those who speak more than one language are less affected by mask wearing, as are younger listeners. Wearing a facial mask reduces speech intelligibility, both due to visual and auditory factors. Older people and people who only speak one language are affected the most.

Assessing the impact of multisensory attributes of Textile-based Assistive Devices (TADs) on the experiences of end-users

Researchers have defined and measured functional attributes of assistive devices, yet there are multiple dimensions of a device worn temporarily or for life that remain unexamined. The purpose of this study was to explore how the sensory attributes of assistive devices impact consumers in order to build a foundation for a holistic approach to product development. Two online surveys targeting wearers of textile-based assistive devices (TADs) (n=29) and caregivers (n=8) were used for data collection. Survey questions related to sensory experiences (visual, tactile, auditory, olfactory) and describe how device usage and purchasing was impacted. Results suggest that multisensory experiences of users and caregivers involve more complexity than functionality alone. Multisensory effects were diverse in both groups. Some expressed a desire for their device to be more visible and expressive, while others chose not to use, expressing fear of judgment based on visibility. Insight into multisensory aesthetics and how design can be used to conceal, reveal, or provide opportunities for self-expression, as well as avoid embarrassment for the wearer, will help designers understand how assistive devices can be improved and developed to meet the needs of individuals with disabilities. Findings suggest that existing measures of success should include questions about sensory attributes.

Grimes’s Music Video “Shinigami Eyes”

This article draws on recent findings in neuroscience to provide a close reading of Grimes’s music video “Shinigami Eyes.” It includes interviews with a neuroscientist who specializes in multisensory integration, a vision scientist, a color grader, and a media scholar. While multisensory integration is a burgeoning field, accessible texts and reviews are lacking. There haven’t been analyses that apply the findings from the field of multisensory integration to real media objects. Drawing on neuroscience studies, this article explores the concept of Bayesian predictions (a viewer’s experience of a present event weighted against previous experiences); the ways neuroscience explains color, size and placement, and emotion work; and processes such as the inverse effectiveness (when neurons boost both lower-res and higher-res signals, with the most perceptible often for sound), super-additive multisensory effects, the attentional blink, the ventriloquist effect, congruence and incongruity, memory, distractors, sound before image and vice versa, and other features. A conversation with the color grader for “Shinigami Eyes” helps fill things in. This article emerges from a belief that modules on neuroscience and industry studies can be incorporated as components of media study courses, and that neuroscience-informed analyses can facilitate interest and engagement with close readings.

Spoken language processing activates the primary visual cortex.

Primary visual cortex (V1) is generally thought of as a low-level sensory area that primarily processes basic visual features. Although there is evidence for multisensory effects on its activity, these are typically found for the processing of simple sounds and their properties, for example spatially or temporally-congruent simple sounds. However, in congenitally blind individuals, V1 is involved in language processing, with no evidence of major changes in anatomical connectivity that could explain this seemingly drastic functional change. This is at odds with current accounts of neural plasticity, which emphasize the role of connectivity and conserved function in determining a neural tissue's role even after atypical early experiences. To reconcile what appears to be unprecedented functional reorganization with known accounts of plasticity limitations, we tested whether V1's multisensory roles include responses to spoken language in sighted individuals. Using fMRI, we found that V1 in normally sighted individuals was indeed activated by comprehensible spoken sentences as compared to an incomprehensible reversed speech control condition, and more strongly so in the left compared to the right hemisphere. Activation in V1 for language was also significant and comparable for abstract and concrete words, suggesting it was not driven by visual imagery. Last, this activation did not stem from increased attention to the auditory onset of words, nor was it correlated with attentional arousal ratings, making general attention accounts an unlikely explanation. Together these findings suggest that V1 responds to spoken language even in sighted individuals, reflecting the binding of multisensory high-level signals, potentially to predict visual input. This capability might be the basis for the strong V1 language activation observed in people born blind, re-affirming the notion that plasticity is guided by pre-existing connectivity and abilities in the typically developed brain.

Does Task-Irrelevant Brightness Modulation Affect Auditory Contrast Processing? Exploring the Interplay Between Temporal Synchrony and Stimulus Salience

Abstract Stimulus factors such as timing, spatial location, and stimulus effectiveness affect whether and how information across the senses is integrated. Extending recent work highlighting interactions between stimulus factors, here we investigated the influence of visual information on auditory processing, complementing previous studies on the influence of auditory information on visual processing. We hypothesized that task-irrelevant and spatially non-informative visual information would enhance auditory contrast processing, when visual information was at an optimal salience level and changed synchronously with the sound. We asked human observers to indicate the location of an amplitude-modulated white-noise sound, while its loudness against a constant white-noise background varied across trials. To test for the influence of task-irrelevant visual information, we modulated screen brightness smoothly (Experiment 1) or transiently (Experiment 2) in phase or out of phase with the amplitude modulation of the target sound. In addition, to test for the interaction between temporal synchrony and stimulus salience, maximum brightness varied systematically across trials. Auditory contrast thresholds were compared across conditions. Results showed that task-irrelevant visual information did not alter auditory contrast thresholds regardless of the nature of modulation of brightness, contrary to our expectations. Nonetheless, task-irrelevant visual information modulated in phase with the target sound reduced auditory contrast thresholds if we accounted for individual differences in the optimal salience required for the largest multisensory effects. Our results are discussed in light of several stimulus factors that might be critical in modulating multisensory enhancement.

Effects of cortico-cortical paired associative stimulation based on multisensory integration to brain network connectivity in stroke patients: study protocol for a randomized doubled blind clinical trial

Introduction: Brain has a spontaneous recovery after stroke, reflecting the plasticity of the brain. Currently, TMS is used for studies of single-target brain region modulation, which lacks consideration of brain networks and functional connectivity. Cortico-cortical paired associative stimulation (ccPAS) promotes recovery of motor function. Multisensory effects in primary visual cortex(V1) directly influence behavior and perception, which facilitate motor functional recovery in stroke patients. Therefore, in this study, dual-targeted precise stimulation of V1 and primary motor cortex(M1) on the affected hemisphere of stroke patients will be used for cortical visuomotor multisensory integration to improve motor function.MethodThis study is a randomized, double-blind controlled clinical trial over a 14-week period. 69 stroke subjects will be enrolled and divided into sham stimulation group, ccPAS low frequency group, and ccPAS high frequency group. All groups will receive conventional rehabilitation. The intervention lasted for two weeks, five times a week. Assessments will be performed before the intervention, at the end of the intervention, and followed up at 6 and 14 weeks. The primary assessment indicator is the ‘Fugl-Meyer Assessment of the Upper Extremity ’, secondary outcomes were ‘The line bisection test’, ‘Modified Taylor Complex Figure’, ‘NIHSS’ and neuroimaging assessments. All adverse events will be recorded.DiscussionCurrently, ccPAS is used for the modulation of neural circuits. Based on spike-timing dependent plasticity theory, we can precisely intervene in the connections between different cortices to promote the recovery of functional connectivity on damaged brain networks after stroke. We hope to achieve the modulation of cortical visuomotor interaction by combining ccPAS with the concept of multisensory integration. We will further analyze the correlation between analyzing visual and motor circuits and explore the alteration of neuroplasticity by the interactions between different brain networks. This study will provide us with a new clinical treatment strategy to achieve precise rehabilitation for patient with motor dysfunction after stroke.Trial registrationThis trial was registered in the Chinese Clinical Trial Registry with code ChiCTR2300067422 and was approved on January 16, 2023.

Virtual reality as a technology of multimodal correction of post-stroke motor and cognitive disturbances in conditions of multitasking functioning (literature review)

The article presents an overview of innovative technologies based on the methods of sensorimotor retraining of patients using virtual reality technology as a promising in the comprehensive rehabilitation of patients who have suffered a cerebral stroke. High level of evidence studies (RCTs, meta-analyses, and systematic reviews) index in the PubMed, Cochrane Library, ClinicalTrials.gov databases are analyzed. Training with multisensory effects on visual, auditory, vestibular, and kinesthetic analyzers in multitasking conditions have beneficial effects on cognitive and motor training, retraining, neuropsychological status of the patien,t and an increase in the level of motivation to achieve success in the rehabilitation. The synergistic nature of the multimodal effects of virtual reality makes it possible to expand the possibilities and increase the effectiveness of medical rehabilitation in patients who have undergone cerebral stroke.

Innovative technologies and multimodal correction in medical rehabilitation of motor and neuropsychological disturbances due to stroke

The article presents an overview of innovative technologies based on the methods of sensorimotor retraining of the patient using various types of biofeedback (BFB) as the most promising in the medical rehabilitation (MR) of patients with cerebral stroke (CS). The works of a high level of evidence (RCTs, national and international clinical guidelines, meta-analyses, systematic reviews) of the Medline, Pubmed, PubMed Cochrane Library databases are analyzed, ClinicalTrials.gov. It is emphasized that training with multisensory effects on visual, auditory, vestibular and kinesthetic analyzers have a beneficial effect on cognitive-motor training and retraining, neuropsychological status of the patient and increase the level of motivation to achieve success in the rehabilitation process. The synergy of multimodal effects of digital technologies, BFB, virtual reality, and the brain-computer interface will expand the capabilities and improve the efficiency of MR of after stroke-patients.

Multisensory Effects on Illusory Self-Motion (Vection): the Role of Visual, Auditory, and Tactile Cues.

A critical component to many immersive experiences in virtual reality (VR) is vection, defined as the illusion of self-motion. Traditionally, vection has been described as a visual phenomenon, but more recent research suggests that vection can be influenced by a variety of senses. The goal of the present study was to investigate the role of multisensory cues on vection by manipulating the availability of visual, auditory, and tactile stimuli in a VR setting. To achieve this, 24 adults (Mage = 25.04) were presented with a rotating stimulus aimed to induce circular vection. All participants completed trials that included a single sensory cue, a combination of two cues, or all three cues presented together. The size of the field of view (FOV) was manipulated across four levels (no-visuals, small, medium, full). Participants rated vection intensity and duration verbally after each trial. Results showed that all three sensory cues induced vection when presented in isolation, with visual cues eliciting the highest intensity and longest duration. The presence of auditory and tactile cues further increased vection intensity and duration compared to conditions where these cues were not presented. These findings support the idea that vection can be induced via multiple types of sensory inputs and can be intensified when multiple sensory inputs are combined.

Audiovisual speech is more than the sum of its parts: Auditory-visual superadditivity compensates for age-related declines in audible and lipread speech intelligibility.

Multisensory input can improve perception of ambiguous unisensory information. For example, speech heard in noise can be more accurately identified when listeners see a speaker's articulating face. Importantly, these multisensory effects can be superadditive to listeners' ability to process unisensory speech, such that audiovisual speech identification is better than the sum of auditory-only and visual-only speech identification. Age-related declines in auditory and visual speech perception have been hypothesized to be concomitant with stronger cross-sensory influences on audiovisual speech identification, but little evidence exists to support this. Currently, studies do not account for the multisensory superadditive benefit of auditory-visual input in their metrics of the auditory or visual influence on audiovisual speech perception. Here we treat multisensory superadditivity as independent from unisensory auditory and visual processing. In the current investigation, older and younger adults identified auditory, visual, and audiovisual speech in noisy listening conditions. Performance across these conditions was used to compute conventional metrics of the auditory and visual influence on audiovisual speech identification and a metric of auditory-visual superadditivity. Consistent with past work, auditory and visual speech identification declined with age, audiovisual speech identification was preserved, and no age-related differences in the auditory or visual influence on audiovisual speech identification were observed. However, we found that auditory-visual superadditivity improved with age. The novel findings suggest that multisensory superadditivity is independent of unisensory processing. As auditory and visual speech identification decline with age, compensatory changes in multisensory superadditivity may preserve audiovisual speech identification in older adults. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Anxiety biases audiovisual processing of social signals

In everyday life, information from multiple senses is integrated for a holistic understanding of emotion. Despite evidence of atypical multisensory perception in populations with socio-emotional difficulties (e.g., autistic individuals), little research to date has examined how anxiety impacts on multisensory emotion perception. Here we examined whether the level of trait anxiety in a sample of 56 healthy adults affected audiovisual processing of emotion for three types of stimuli: dynamic faces and voices, body motion and dialogues of two interacting agents, and circles and tones. Participants judged emotion from four types of displays – audio-only, visual-only, audiovisual congruent (e.g., angry face and angry voice) and audiovisual incongruent (e.g., angry face and happy voice) – as happy or angry, as quickly as possible. In one task, participants based their emotional judgements on information in one modality while ignoring information in the other, and in a second task they based their judgements on their overall impressions of the stimuli. The results showed that the higher trait anxiety group prioritized the processing of angry cues when combining faces and voices that portrayed conflicting emotions. Individuals in this group were also more likely to benefit from combining congruent face and voice cues when recognizing anger. The multisensory effects of anxiety were found to be independent of the effects of autistic traits. The observed effects of trait anxiety on multisensory processing of emotion may serve to maintain anxiety by increasing sensitivity to social-threat and thus contributing to interpersonal difficulties.

Tactile Echoes: Multisensory Augmented Reality for the Hand.

Touch interactions are central to many human activities, but there are few technologies for computationally augmenting free-hand interactions with real environments. Here, we describe Tactile Echoes, a finger-wearable system for augmenting touch interactions with physical objects. This system captures and processes touch-elicited vibrations in real-time in order to enliven tactile experiences. In this article, we process these signals via a parametric signal processing network in order to generate responsive tactile and auditory feedback. Just as acoustic echoes are produced through the delayed replication and modification of sounds, so are Tactile Echoes produced through transformations of vibrotactile inputs in the skin. The echoes also reflect the contact interactions and touched objects involved. A transient tap produces discrete echoes, while a continuous slide yields sustained feedback. We also demonstrate computational and spatial tracking methods that allow these effects to be selectively assigned to different objects or actions. A large variety of distinct multisensory effects can be designed via ten processing parameters. We investigated how Tactile Echoes are perceived in several perceptual experiments using multidimensional scaling methods. This allowed us to deduce low-dimensional, semantically grounded perceptual descriptions. We present several virtual and augmented reality applications of Tactile Echoes. In a user study, we found that these effects made interactions more responsive and engaging. Our findings show how to endow a large variety of touch interactions with expressive multisensory effects.

Introducing CatchU TM : A Novel Multisensory Tool for Assessing Patients’ Risk of Falling †

To date, only a few studies have investigated the clinical translational value of multisensory integration. Our previous research has linked the magnitude of visual-somatosensory integration (measured behaviorally using simple reaction time tasks) to important cognitive (attention) and motor (balance, gait, and falls) outcomes in healthy older adults. While multisensory integration effects have been measured across a wide array of populations using various sensory combinations and different neuroscience research approaches, multisensory integration tests have not been systematically implemented in clinical settings. We recently developed a step-by-step protocol for administering and calculating multisensory integration effects to facilitate innovative and novel translational research across diverse clinical populations and age-ranges. In recognizing that patients with severe medical conditions and/or mobility limitations often experience difficulty traveling to research facilities or joining time-demanding research protocols, we deemed it necessary for patients to be able to benefit from multisensory testing. Using an established protocol and methodology, we developed a multisensory falls-screening tool called CatchU TM (an iPhone app) to quantify multisensory integration performance in clinical practice that is currently undergoing validation studies. Our goal is to facilitate the identification of patients who are at increased risk of falls and promote physician-initiated falls counseling during clinical visits (e.g., annual wellness, sick, or follow-up visits). This will thereby raise falls-awareness and foster physician efforts to alleviate disability, promote independence, and increase quality of life for our older adults. This conceptual overview highlights the potential of multisensory integration in predicting clinical outcomes from a research perspective, while also showcasing the practical application of a multisensory screening tool in routine clinical practice.

Dazzle, Dangle, and Jangle: Sensory Effects of Scandinavian Gold Bracteates

Abstract Small pendant discs known as Scandinavian gold bracteates are visually impressive indicators of status and identity during the early medieval Migration Period (c. 450–550 CE). Much of the emphasis in bracteate studies has been on typological classification and iconographic interpretation of the pictures, along with decipherment of the inscriptions, yet the sensory impression made by bracteates has been neglected. For decades, archaeologists considered it futile to speculate on the experiential; however, recent research has begun to contend with the materiality of senses and emotions in the past. In this paper, I focus on the both sensory effect of experiencing the pendants and the emotional impact of the objects on those who wore and handled them. Since bracteates are often discovered in women’s graves and show evidence of wear on their suspension loops, it is assumed that these objects were actually used in life. A spotlight on agency turns the study of bracteates from an emphasis on typology, iconography, and runology to trace instead the agency and multisensory effects of these objects on people and the effects of people’s actions on these items. This allows an examination of how bracteates were entangled with the senses and emotions of those who made, wore, and gazed upon these small but important objects. I consider what it may have felt like to make, receive, wear, lose, or bequeath a bracteate – or to deposit it into a bog or a grave.

Multisensory effects on causal perception

Multisensory effects on causal perception

What is a multisensory cortex? A laminar, connectional, and functional study of a ferret temporal cortical multisensory area.

Now that examples of multisensory neurons have been observed across the neocortex, this has led to some confusion about the features that actually designate a region as "multisensory." While the documentation of multisensory effects within many different cortical areas is clear, often little information is available about their proportions or net functional effects. To assess the compositional and functional features that contribute to the multisensory nature of a region, the present investigation used multichannel neuronal recording and tract tracing methods to examine the ferret temporal region: the lateral rostral suprasylvian sulcal area. Here, auditory-tactile multisensory neurons were predominant and constituted the majority of neurons across all cortical layers whose responses dominated the net spiking activity of the area. These results were then compared with a literature review of cortical multisensory data and were found to closely resemble multisensory features of other, higher-order sensory areas. Collectively, these observations argue that multisensory processing presents itself in hierarchical and area-specific ways, from regions that exhibit few multisensory features to those whose composition and processes are dominated by multisensory activity. It seems logical that the former exhibit some multisensory features (among many others), while the latter are legitimately designated as "multisensory."

LINKING VISUAL-SOMATOSENSORY INTEGRATION TO COGNITIVE AND MOTOR OUTCOMES IN AGING

Ability to successfully integrate simultaneous information relayed across multiple sensory systems is an integral aspect of daily functioning. Unisensory impairments have been individually linked to slower gait, functional decline, and increased risks for falls in aging. Yet, research investigating age-related changes in multisensory integration (MSI) processes still remains relatively scarce. To date, there has been converging evidence for larger behavioral multisensory effects in older compared to younger adults; however, the question of whether larger effects are actually beneficial remains largely unanswered. Findings from our studies provide support for differential multisensory processing in aging, where decreased magnitude of visual-somatosensory integration was associated with worse balance, increased falls, and slower gait. Furthermore, we established a link between visual-somatosensory integration and cognition in aging. That is, magnitude of visual-somatosensory integration was largest in older adults with normal cognitive functioning, and presence of MCI/dementia significantly decreased magnitude of visual-somatosensory integration which in turn adversely impacted balance and gait performance. While the effect of MSI has been attributed to basic degenerative changes in neuronal architecture during the aging process, this speculative interpretation has yet to be formally tested. Future studies are clearly needed to establish the structural and functional correlates of MSI in aging, specifically visual-somatosensory integration, in order to further establish the link between differential multisensory effects with other important age-related clinical outcomes. Nevertheless, these studies stress the importance of successful MSI in aging, and highlight the need for multisensory based interventions that could potentially ameliorate disability.

Do Changes in Language Context Affect Visual Memory in Bilinguals?

Language is often present when people are encoding visual memories. For bilinguals, this language context can have different forms (i.e., Language A, Language B, or both Language A and B), and can change over the course of events. The current study examined whether a change in language context during a visual event or between visual events affects a bilingual’s ability to remember visual information. English-Spanish bilinguals and control participants encoded three lists of novel shapes amid different task-irrelevant language contexts. Following each list, participants completed a free recall test in which they drew the novel shapes they remembered. Results indicated that a change in language context between events, but not during events, affected visual memory. Specifically, the switch in language context between the second and third event (such as an English context in list 2 switching to a Spanish context in list 3) produced a reliable memory advantage for the English-Spanish bilinguals (relative to the control participants). The results offer preliminary evidence that task-irrelevant language context can influence a bilingual’s ability to remember non-linguistic information, as well as further evidence for context effects and multi-sensory effects in memory.

A mulsemedia framework for delivering sensory effects to heterogeneous systems

Technological advances in computing have allowed multimedia systems to create more immersive experiences for users. Beyond the traditional senses of sight and hearing, researchers have observed that the use of smell, taste, and touch in such systems is becoming increasingly well-received, leading to a new category of multimedia systems called mulsemedia—multiple sensorial media—systems. In parallel, these systems introduce heterogeneous technologies to deliver different sensory effects such as lighting, wind, vibration, and smell, under varied conditions and restrictions. This new paradigm shift poses many challenges, mainly related to mulsemedia integration, delay, responsiveness, sensory effects intensities, wearable and other heterogeneous devices for delivering sensory effects, and remote delivery of mulsemedia components. In addition, new approaches to interacting with multimedia applications have emerged such as multi-touch interfaces, voice processing, and brain–computer interfaces, giving rise to new kinds of complex interactive systems. In this article, we underpin fundamental challenges for delivering multisensory effects to heterogeneous systems. We propose an interoperable mulsemedia framework for coping with these challenges, meeting the emerging requirements. It is achieved through the evolution of an open distributed mulsemedia system. We changed its core following architectural and design patterns to accommodate different profiles of communication, connectivity, and sensory effects metadata standard according to the need of mulsemedia applications and devices available in the user’s environment. The results include case studies where the framework has been duly applied.

Concurrent intramodal learning enhances multisensory responses of symmetric crossmodal learning in robotic audio-visual tracking

Tracking an audio-visual target involves integrating spatial cues about target position from both modalities. Such sensory cue integration is a developmental process in the brain involving learning, with neuroplasticity as its underlying mechanism. We present a Hebbian learning-based adaptive neural circuit for multi-modal cue integration. The circuit temporally correlates stimulus cues within each modality via intramodal learning as well as symmetrically across modalities via crossmodal learning to independently update modality-specific neural weights on a sample-by-sample basis. It is realised as a robotic agent that must orient towards a moving audio-visual target. It continuously learns the best possible weights required for a weighted combination of auditory and visual spatial target directional cues that is directly mapped to robot wheel velocities to elicit an orientation response. Visual directional cues are noise-free and continuous but arising from a relatively narrow receptive field while auditory directional cues are noisy and intermittent but arising from a relatively wider receptive field. Comparative trials in simulation demonstrate that concurrent intramodal learning improves both the overall accuracy and precision of the orientation responses of symmetric crossmodal learning. We also demonstrate that symmetric crossmodal learning improves multisensory responses as compared to asymmetric crossmodal learning. The neural circuit also exhibits multisensory effects such as sub-additivity, additivity and super-additivity.