Multisensory Features Research Articles

The influence of multisensory indoor environment on the perception of orange juice

Drinking (as well as eating) represents a fundamental multisensory experience of our lives. Recent studies have proposed the implementation of multisensory contexts to improve the taste perception of food and beverages. This topic becomes relevant when considering the possibility of promoting the consumption of healthy beverages (e.g., orange juice). Only a few studies have concentrated on the multisensory features of the indoor environment (e.g., acoustic and visual context) reporting effects on the perception of healthy beverages. This study investigates how multisensory indoor environments modify the taste perception while drinking orange juice. By controlling the room microclimate and combining the correlated colour temperature of lighting (i.e. warm and cool) and the background wall colour (i.e., red, green) in the Sens i-Lab, with different audio stimuli (i.e. low and high-frequency pure tone), participants were invited to taste the low-sugar orange juice and to rate the fruit juice aroma, sweetness, sourness, bitterness, freshness, and thickness. The results showed that listening to a high-frequency tone in an environment with warm lighting and a red background increases the sweetness perception of orange juice compared to the control environment. In addition, compared to the control condition, the juice aroma was rated more intense in almost all multisensory conditions and thicker with cool lighting and a green background while listening to a low-frequency tone. Our findings suggested that the development of multisensory design can contribute to interior design strategies and design innovation in food architecture, where people can improve their taste in healthy foods and beverages.

Perceived barriers and solutions identified by healthcare professionals in utilizing web-based reminiscence therapy to support dementia care during the pandemic.

This paper presents findings from a qualitative study conducted in Ontario, Canada, exploring healthcare professionals' perceptions of barriers and solutions for implementing Web-Based Reminiscence Therapy (WBRT) in an institutionalized settings for dementia care during the COVID-19 pandemic. The study identified five major barriers, including the lack of on-boarding/educational training, need for technology availability and technical support, limited attention span of persons with dementia (PWD), availability of multi-sensory features, and time constraints due to staff workload. Seven major themes emerged related to proposed solutions/suggestions: (1) involving younger generations, (2) focusing on technology training, (3) integrating with other digital platforms, (4) adding narratives/descriptions to recollect memories, (5) ensuring accessibility, (6) adding QR codes for retrieving information, and (7) combining digital/traditional reminiscence methods. These findings provide valuable insights for implementing WBRT to facilitate dementia care and for the future refinement of its application.

Speaking in front of cartoon avatars: A behavioral and psychophysiological study on how audience design impacts on public speaking anxiety in virtual environments

Public speaking anxiety is defined as a strong difficulty in speaking in front of an audience and has been shown to impair work performance and social relationships. Virtual Reality (VR) offers an efficient tool for modulating public speaking anxiety through a wide range of customizations concerning environmental settings. However, scientific research needs to understand better what features of the simulated environment are more important to increase or reduce participants’ perceived discomfort. The present study investigates the role of visual (human vs cartoon characters) and acoustic (human vs robotic voice) audience features on perceived anxiety, sense of presence, and perceived realism in an interactive VR public speaking scenario. 42 participants (mean age = 24 y.o; Females = 30) performed four public speaking sessions, characterized by different levels (high vs. low) of graphic and acoustic audience design. Both explicit (questionnaires) and implicit physiological measures (Electrodermal activity-EDA) collected during audience interaction were used to assess the participants’ experiences.The results showed that the features of the simulated audience played a crucial role in perceived anxiety during a virtual public speaking. Specifically, the more realistic level of graphic and acoustic stimuli resulted in higher levels of self-reported anxiety as compared to the lower realism level. However, the experienced realism and the sense of presence seem more affected by the graphical than acoustic features of the virtual environment. By contrast, the acoustic features impact on the interaction realism with the virtual audience. Interestingly, the robotic voice (lower acoustic realism) increased electrodermal response during the interaction with the audience, interpreted as a break in the sense of presence. A positive correlation between anxiety, sense of presence, and experienced realism was found. As well, perceived anxiety is correlated with electrodermal activity during the performance. Nevertheless, physiological activity is more affected by the first experience than the realism features, suggesting the presence of a habituation effect across the repeated sessions. Taken together, the results of our study showed that multisensory features (graphical and acoustic) of the virtual environment play a fundamental role in creating realistic public speaking experiences and might be used within gamification strategies for soft skill training (e.g., for improving public speaking anxiety).

The Effectiveness of Role-play Integrated Multisensory Learning to Enhance Students' Confidence and Speaking Skill

Lack of self-confidence and speaking abilities can hinder student learning. Students without self-confidence struggle to express their views and opinions, resulting in poor speaking skills. This study examines whether Role-play Integrated Multisensory Learning (RIML) boosts students' self-confidence and speaking skills. This quantitative experimental study uses an explanatory model and non-equivalent control group design. This study included 30 control and 30 experimental students. Grade 1 elementary school 93 Kendari State Elementary School children were purposively sampled for the study. This study collected data using observation. In this study, students' confidence and speaking skills in five were measured in four ways. This study analyzed data using normality and homogeneity tests. Additionally, SPSS with an independent sample t-test was used to assess the data. Participants' self-confidence and speaking skills improved significantly after the Role-play Integrated Multisensory Learning (RIML) intervention. The experimental group had a high average score of 5,9.0, rising to 91.3 on student confidence and 83.2 on student speaking skills. Additionally, the significance value for each variable is 0.000 <0.05. The findings in this study indicate an improvement in elementary school students' confidence and speaking abilities after receiving an intervention in the form of Role-play Integrated Multisensory Learning (RIML).RIML learning uses all senses in roleplay learning to assist students in understanding the context of learning and practicing speaking skills to gain confidence. Future studies may integrate multisensory features into other learning approaches to stimulate students with varied learning styles.

Virtually tasty: An investigation of the effect of ambient lighting and 3D-shaped taste stimuli on taste perception in virtual reality

Taste perception is influenced by sensory information not only about the food itself but also about the external environment where the food is tasted. Prior studies have shown that both visual attributes of the environment (e.g., light colour, location) and the shape associated to food (e.g., plates, cutlery) can influence people's taste perception and expectations. However, previous studies are typically based on non-edible shapes usually shown as 2D images or presented as 3D tangible objects aimed to be perceived by subjects' hand. Therefore, the effect of mouthfeel of differently shaped foods on taste perception remains unclear. Capitalising on the advantages of virtual reality (VR) to manipulate multisensory features, we explore the effects of coloured (red, blue, neutral) virtual environments on the taste (sweet, neutral) perception of differently shaped taste samples (rounded/spiky shapes according to the Kiki-Bouba paradigm). Overall, our results showed increased ratings of sweetness when participants tasted Bouba-shaped samples (rounded) relative to Kiki-shaped samples (spiky) suggesting that tactile attributes perceived inside the mouth can influence sweetness perception. Furthermore, we concluded that lighting colour in a virtual setting might dampen experiences of sweetness. However, this effect may only be present when there is a cross-modal correspondence with taste. Based on our findings, we conclude by describing considerations for designing eating experiences in VR.

A comparison of multisensory features of two auditory cortical areas: primary (A1) and higher-order dorsal zone (DZ).

From myriads of ongoing stimuli, the brain creates a fused percept of the environment. This process, which culminates in perceptual binding, is presumed to occur through the operations of multisensory neurons that occur throughout the brain. However, because different brain areas receive different inputs and have different cytoarchitechtonics, it would be expected that local multisensory features would also vary across regions. The present study investigated that hypothesis using multiple single-unit recordings from anesthetized cats in response to controlled, electronically-generated separate and combined auditory, visual, and somatosensory stimulation. These results were used to compare the multisensory features of neurons in cat primary auditory cortex (A1) with those identified in the nearby higher-order auditory region, the Dorsal Zone (DZ). Both regions exhibited the same forms of multisensory neurons, albeit in different proportions. Multisensory neurons exhibiting excitatory or inhibitory properties occurred in similar proportions in both areas. Also, multisensory neurons in both areas expressed similar levels of multisensory integration. Because responses to auditory cues alone were so similar to those that included non-auditory stimuli, it is proposed that this effect represents a mechanism by which multisensory neurons subserve the process of perceptual binding.

Domain organisation emerges in cross-modal but not within-modal semantic feature integration

ABSTRACT A large body of literature supports theories positing a distributed, perceptually grounded semantic memory system. Prominent models have assumed distributed features are integrated into networks using either shallow or deep hierarchies. Previous behavioural tests of modality effects in shallow and deep hierarchies inspired by, but not implemented in, connectionist models support deep hierarchy architectures. We behaviourally replicate and model speeded dual feature verification in a sample of general-purpose modality-specific computational models of semantic memory trained on feature production norms for 541 concepts. The cross-modal advantage in semantic processing shown behaviourally and in simulations supports hierarchically organised distributed models of semantic memory and provides novel insight into the division of labour in these models. Analyses of the emergent model structure suggest animacy distinctions arise from the self-organisation of statistical co-occurrences among multisensory features but weakly among unisensory features. These findings suggest a privileged role of the multisensory convergence area for category representation.

Whether attentional loads influence audiovisual integration depends on semantic associations.

Neuronal studies have shown that selectively attending to a common object in one sensory modality results in facilitated processing of that object's representations in the ignored sensory modality. Thus, the audiovisual (AV) integration of common objects can be observed under modality-specific selective attention. However, little is known about whether this AV integration can also occur under increased attentional load conditions. Additionally, whether semantic associations between multisensory features of common objects modulate the influence of increased attentional loads on this cross-modal integration remains unknown. In the present study, participants completed an AV integration task (ignored auditory stimuli) under various attentional load conditions: no load, low load, and high load. The semantic associations between AV stimuli were composed of animal pictures presented concurrently with semantically congruent, semantically incongruent, or semantically unrelated auditory stimuli. Our results demonstrated that attentional loads did not disrupt the integration of semantically congruent AV stimuli but suppressed the potential alertness effects induced by incongruent or unrelated auditory stimuli under the condition of modality-specific selective attention. These findings highlight the critical role of semantic association between AV stimuli in modulating the effect of attentional loads on the AV integration of modality-specific selective attention.

Multisensory responses in a belt region of the dorsal auditory cortical pathway

A basic function of the cerebral cortex is to receive and integrate information from different sensory modalities into a comprehensive percept of the environment. Neurons that demonstrate multisensory convergence occur across the necortex but are especially prevalent in higher order association areas. However, a recent study of a cat higher order auditory area, the dorsal zone (DZ) of auditory cortex, did not observe any multisensory features. Therefore, the goal of the present investigation was to address this conflict using recording and testing methodologies that are established for exposing and studying multisensory neuronal processing. Among the 482 neurons studied, we found that 76.6% were influenced by non-auditory stimuli. Of these neurons, 99% were affected by visual stimulation, but only 11% by somatosensory. Furthermore, a large proportion of the multisensory neurons showed integrated responses to multisensory stimulation, constituted a majority of the excitatory and inhibitory neurons encountered (as identified by the duration of their waveshape) and exhibited a distinct spatial distribution within DZ. These findings demonstrate that the DZ of auditory cortex robustly exhibits multisensory properties and that the proportions of multisensory neurons encountered are consistent with those identified in other higher order cortices.

Virtual Reality for the Treatment of Anxiety Disorders: A Scoping Review.

Virtual reality in head-mounted displays (HMD-VR) may be a valuable tool in occupational therapy to address anxiety. Findings from the virtual reality exposure therapy (VRET) literature may facilitate translation of HMD-VR to occupational therapy psychosocial practice. To explore how HMD-VR has been used to treat anxiety through VRET and could be translated to occupational therapy. We searched seven electronic databases for articles published between 2000 and 2020: CINAHL, Cochrane Library, Embase, ERIC, Ovid MEDLINE, PsycINFO, and Web of Science. Search terms included HMD-VR constructs, products, and therapy concepts. Study Selection and Data Collection: We used Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines to report studies implementing VRET to treat anxiety. At least two reviewers assessed each citation, and a third resolved disagreements. Articles were included if they were in English, reported experimental data, and used HMD-VR. Letters, commentaries, book chapters, technical descriptions, theoretical papers, conference proceedings (≤4 pages), and reviews were excluded. Twenty-eight studies used HMD-VR to treat posttraumatic stress disorder (n = 3), specific phobias (n = 19), and performance-based social anxiety (n = 6); protocols and levels of evidence varied (randomized controlled trials, n = 11; controlled trials without randomization, n = 6; case-control or cohort studies, n = 11). Qualitative examination indicates HMD-VR is an effective treatment tool. HMD-VR can be a valuable tool for occupational therapy to simulate environments where clients with anxiety disorders participate. Eliciting presence through multisensory features and body representation may enhance outcomes. What This Article Adds: Drawing from the VRET literature, this scoping review suggests that HMD-VR can be used by occupational therapy practitioners to simulate ecologically valid environments, evaluate client responses to fearful stimuli, and remediate anxiety though immersion in virtual tasks when participation in natural contexts is unfeasible. Having ecologically valid environments is particularly important for people with anxiety disorders because they need support to cope when they encounter triggers in everyday life environments.

Restricted Auditory Aspatialism

Some philosophers have argued that we do not hear sounds as located in the environment. Others have objected that this straightforwardly contradicts the phenomenology of auditory experience. And from this they draw metaphysical conclusions about the nature of sounds—that they are events or properties of vibrating surfaces rather than waves or sensations. I argue that there is a minimal, but recognizable, notion of audition to which this phenomenal objection does not apply. While this notion doesn’t correspond to our ordinary notion of auditory experience, it does—in conjunction with our lack of an uncontroversial individuation of the senses and recent interest in distinctively multisensory features of perceptual experiences—raise the possibility of more expansive notions of audition, including some that do plausibly count as corresponding to our everyday notion of audition, that lack the spatial phenomenology cited in the objection. Until this possibility is ruled out, the phenomenal objection and metaphysical conclusions drawn from it remain inconclusive.

A Deep Learning Approach for Brain Computer Interaction-Motor Execution EEG Signal Classification

Recently Noninvasive Electroencephalogram (EEG) systems are gaining much attention. Brain-computer Interface (BCI) systems rely on EEG analysis to identify the mental state of the user, change in cognitive state and response to the events. Motor Execution (ME) is a very important control paradigm. This paper introduces a robust and useful User-Independent Hybrid Brain-computer Interface (UIHBCI) model to classify signals from fourteen EEG channels that are used to record the reactions of the brain neurons of nine subjects. Through this study the researchers identified relevant multisensory features of multi-channel EEG that represent the specific mental processes depending on two different evaluation models (Audio/Video) and (Male/Female). The Deep Belief Network (DBN) was applied independently on the two models where, the overall achieved classification rates were better in ME classification compared to the state of art. For evaluation four models were tested in addition to the proposed model, Linear Discriminant Analysis (LDA), Support Vector Machine (SVM), Brain-computer Interface Lower-Limb Motor Recovery (BCI LLMR) and Hybrid Steady-State Visual Evoked Potential Rapid Serial Visual Presentation Brain-computer Interface (Hybrid SSVEP-RSVP BCI). Results indicated the proposed model, LDA, SVM, BCI LLMR and Hybrid SSVEP-RSVP BCI accuracies for (A/V) model are 94.44%, 66.67%, 61.11%, 83.33% and 89.67% respectively, while for (M/F) model, the overall accuracies are 94.44%, 88.89%, 83.331%, 85.44% and 89.45%. Finally, the proposed model achieved superiority over the state of art algorithms in both (A/V) and (M/F) models.

Young people's narratives of hearing voices: Systemic influences and conceptual challenges.

Despite the prevalence of voice hearing in childhood and adolescence, little qualitative research has been undertaken with young people directly to advance phenomenological and aetiological insights into their experiences and interpretations. Consequently, the researchers sought demographic, contextual, and qualitative data from 74 young people from eleven countries, aged 13-18years (28%=male; 61%=female; 21%=Transgender and Gender Non-Binary [TGNB]), who self-identified as hearing voices. A Foucauldian-informed narrative analysis yielded four analytic chapters, offering novel perspectives into individual, relational, systemic, and cultural interpretative narratives surrounding multisensory and multi-self voice hearing. Overall, young people reported heterogenous experiences of voice hearing and associated sensory experiences, and most participants reported voice hearing beginning between ages 8 and 11. Further, the emotions felt by the child, as well as reactions displayed by people around the child in relation to the voices, influenced voice-related distress and the nature of the voices in a triadic relationship. A continuum of multisensory features of voice content, nature, and relational significance is tentatively proposed to capture the breadth and depth of voice hearing for adolescents to offer a possible framework for future study and intervention design. Specifically, participants described that voice-related distress could be exacerbated by observed anxiety or internalized stigma about voice hearing, social isolation, and attribution to illness. These findings suggest that we may need to reconsider how the experience of hearing voices in childhood influences their relationships and how relationships influence the voice hearing experience. Further, young people seem to have a broad understanding of what the term "hearing voices" means, which could inform how researchers and practitioners work with this group of young people. Finally, participants described benefitting from multisensory coping strategies, such as imagery and meditation, which could offer important considerations for tailoring therapeutic interventions for adolescent voice hearers.

Transforming Space into Place: A Person-Environment Interchange Approach for Designing an Assisted Living Facility Courtyard

This study tested whether it is applicable to design a long-term care facility courtyard for older adults with a person-environment interchange approach. We created design protocols for an underutilized courtyard at an assisted living facility to promote interactions amongst residents, others, and nature. Then, we conducted semi-structured individual interviews with nine residents to explore their perceptions on the existing courtyard and the design protocols by presenting simulated prototyped landscaping in a video format. Findings suggest that residents appreciate multisensory features in their courtyard, and they are willing to play active roles in creating and sustaining their outdoor courtyard environments.

What is a multisensory cortex? A laminar, connectional, and functional study of a ferret temporal cortical multisensory area.

Now that examples of multisensory neurons have been observed across the neocortex, this has led to some confusion about the features that actually designate a region as "multisensory." While the documentation of multisensory effects within many different cortical areas is clear, often little information is available about their proportions or net functional effects. To assess the compositional and functional features that contribute to the multisensory nature of a region, the present investigation used multichannel neuronal recording and tract tracing methods to examine the ferret temporal region: the lateral rostral suprasylvian sulcal area. Here, auditory-tactile multisensory neurons were predominant and constituted the majority of neurons across all cortical layers whose responses dominated the net spiking activity of the area. These results were then compared with a literature review of cortical multisensory data and were found to closely resemble multisensory features of other, higher-order sensory areas. Collectively, these observations argue that multisensory processing presents itself in hierarchical and area-specific ways, from regions that exhibit few multisensory features to those whose composition and processes are dominated by multisensory activity. It seems logical that the former exhibit some multisensory features (among many others), while the latter are legitimately designated as "multisensory."

Crossmodal Correspondences in the Sounds of Chinese Instruments.

People tend to associate stimuli from different sensory modalities, a phenomenon known as crossmodal correspondences. We conducted two experiments to investigate how Chinese participants associated musical notes produced by four types of Chinese instruments (bowed strings, plucked strings, winds, and percussion instruments) with different colors, taste terms, and fabric textures. Specifically, the participants were asked to select a sound to match each color patch or taste term in Experiment 1 and to match the experience of touching each fabric in Experiment 2. The results demonstrated some associations between pitch and color, taste term, or the smoothness of fabrics. Moreover, certain types of Chinese instruments were preferentially chosen to match some of the colors, taste terms, and the texture of certain fabrics. These findings therefore provided insights about the perception of Chinese music and shed light on how to apply the multisensory features of sounds to enhance the composition, performance, and appreciation of music.

Bodily awareness and novel multisensory features

According to the decomposition thesis, perceptual experiences resolve without remainder into their different modality-specific components. Contrary to this view, I argue that certain cases of multisensory integration give rise to experiences representing features of a novel type. Through the coordinated use of bodily awareness—understood here as encompassing both proprioception and kinaesthesis—and the exteroceptive sensory modalities, one becomes perceptually responsive to spatial features whose instances couldn’t be represented by any of the contributing modalities functioning in isolation. I develop an argument for this conclusion focusing on two cases: 3D shape perception in haptic touch and experiencing an object’s egocentric location in crossmodally accessible, environmental space.

Auditory-induced response in the primary sensory cortex of rodents.

The details of auditory response at the subthreshold level in the rodent primary somatosensory cortex, the barrel cortex, have not been studied extensively, although several phenomenological reports have been published. Multisensory features may act as neuronal representations of links between inputs from one sensory modality to other sensory modalities. Here, we examined the basic multisensory postsynaptic responses in the rodent barrel cortex using in vivo whole-cell recordings of neurons. We observed robust responses to acoustic stimuli in most barrel cortex neurons. Acoustically evoked responses were mediated by hearing and reached approximately 60% of the postsynaptic response amplitude elicited by strong somatosensory stimuli. Compared to tactile stimuli, auditory stimuli evoked postsynaptic potentials with a longer latency and longer duration. Specifically, auditory stimuli in barrel cortex neurons appeared to trigger “up states”, episodes associated with membrane depolarization and increased synaptic activity. Taken together, our data suggest that barrel cortex neurons have multisensory properties, with distinct synaptic mechanisms underlying tactile and non-tactile responses.

Students’ Attitudes Toward High-Immersion Virtual Reality Assisted Language Learning

This pilot study delivers a preliminary report on students’ attitudes toward virtual reality assisted language learning (VRALL). Learners (N = 22) participated in a VRALL lesson and then completed a post-experience survey. Virtual reality (VR) technology allows for learning experiences that increasingly remove geographic limitations to foreign language learners. Thanks to multisensory features of VR, including 360-degree, three-dimensional visualizations, students’ experiences are highly immersive. Descriptive statistics provide preliminary evidence that VR technology is engaging and immersive for learners. Moreover, VR could play a role in computer-assisted language learning (CALL), especially when pitfalls such as cybersickness are overcome.

Cortical multisensory connectivity is present near birth in humans.

How the newborn brain adapts to its new multisensory environment has been a subject of debate. Although an early theory proposed that the brain acquires multisensory features as a result of postnatal experience, recent studies have demonstrated that the neonatal brain is already capable of processing multisensory information. For multisensory processing to be functional, it is a prerequisite that multisensory convergence among neural connections occur. However, multisensory connectivity has not been examined in human neonates nor are its location(s) or afferent sources understood. We used resting state functional MRI (fMRI) in two independent cohorts of infants to examine the functional connectivity of two cortical areas known to be multisensory in adults: the intraparietal sulcus (IPS) and the superior temporal sulcus (STS). In the neonate, the IPS was found to demonstrate significant functional connectivity with visual association and somatosensory association areas, while the STS showed significant functional connectivity with the visual association areas, primary auditory cortex, and somatosensory association areas. Our findings establish that each of these areas displays functional communication with cortical regions representing various sensory modalities. This demonstrates the presence of cortical areas with converging sensory inputs, representing that the functional architecture needed for multisensory processing is already present within the first weeks of life.