Cross-modal Effects Research Articles

Cross-modal Effect between Taste and Shape Controlled by Curvature Entropy

Recently, cross-modal effects in which perceptions interact with each other have been drawing attention. In the case of the cross-modal effect between vision and taste, the effect of the angularity of shapes on taste has been widely studied. However, there has been little research on the other features of shapes. Also, previous studies have shown that the emotional valence arisen from visual perception causes the cross-modal effect. Therefore, this study focuses on the complexity of shapes, which is said to influence emotional valence, as a visual stimulus and aims to confirm the cross-modal effect induced by its sensation. First, by using particle swarm optimization algorithm, closed curve shapes were generated based on curvature entropy, a quantitative index of the complexity of shapes. Second, cup holders, which had these closed curve shapes on their sides, were created by using a 3D printer. Third, by comparing the tastes of orange juice in these cup holders, the effect of the complexity of shapes on the perception of sweetness was confirmed. Finally, we propose index of Angularity derived from curvature, which has been suggested to be relevant in previous studies, and additionally test for cross-modal effects on this index. The results suggest that the complexity of shapes controlled by curvature entropy and the Angularity of shapes controlled by index of Angularity derived from curvature weaken the perception of sweetness. This finding can be used for reducing sugar intake in bottle packaging.

Exploring the Links between Colours and Tastes/Flavours

The colour and other visual appearance properties of food and drink constitute a key factor determining consumer acceptance and choice behaviour. Not only do consumers associate specific colours with particular tastes and flavours, but adding or changing the colour of food and drink can also dramatically affect taste/flavour perception. Surprisingly, even the colour of cups, cutlery, plates, packages, and the colour of the environment itself, have also been shown to influence multisensory flavour perception. The taste/flavour associations that we hold with colour are context-dependent, and are often based on statistical learning (though emotional mediation may also play a role). However, to date, neither the computational principles constraining these ubiquitous crossmodal effects nor the neural mechanisms underpinning the various crossmodal associations (or correspondences) that have been documented between colours and tastes/flavours have yet been established. It is currently unclear to what extent such colour-taste/flavour correspondences ought to be explained in terms of semantic congruency (i.e., statistical learning), and/or emotional mediation. Bayesian causal inference has become an increasingly important tool in helping researchers to understand (and predict) the multisensory interactions between the spatial senses of vision, audition, and touch. However, a network modelling approach may be of value moving forward. As made clear by this review, there are substantial challenges, both theoretical and practical, that will need to be overcome by those wanting to apply computational approaches both to understanding the integration of the chemical senses in the case of multisensory flavour perception, and to understanding the influence of colour thereon.

Cross-modal effects of visual characteristics of font types on scent impressions

Cross-modal effects of visual characteristics of font types on scent impressions

The cross-modal affective priming effect: Effects of the valence and arousal of primes

Although studies have investigated the influence of the emotionality of primes on the cross-modal affective priming effect, it is unclear whether this effect is due to the contribution of the arousal or the valence of primes. We explored how the valence and arousal of primes influenced the cross-modal affective priming effect. In Experiment 1 we manipulated the valence of primes (positive and negative) that were matched by arousal. In Experiments 2 and 3 we manipulated the arousal of primes under the conditions of positive and negative valence, respectively. Affective words were used as auditory primes and affective faces were used as visual targets in a priming task. The results suggest that the valence of primes modulated the cross-modal affective priming effect but that the arousal of primes did not influence the priming effect. Only when the priming stimuli were positive did the cross-modal affective priming effect occur, but negative primes did not produce a priming effect. In addition, for positive but not negative primes, the arousal of primes facilitated the processing of subsequent targets. Our findings have great significance for understanding the interaction of different modal affective information.

Environmental thermal influence over soundscape perception: a test room experimental campaign involving the psychological and physiological description of the indoor environment

Human environmental perception leads occupants’ behaviour when interacting with buildings components, affecting the final building energy performance. A solid understanding of human comfort perception includes simultaneous multisensory stimuli and cross-modal interactions among different comfort domains. This study aims to explore the cross-modal effect between thermal and acoustic domains. Each of the 40 subjects took part in a multisensory survey under two different stationary environmental temperature settings. Results show that people in thermally warm conditions are less confident in describing the provided acoustic records. To perform the same procedure but providing a decreasing air temperature ramp would lead to a better interpretation of the results of this campaign.

The Role of Joint Influence on the Cross-Modal Stroop Effect: Investigating Time Course and Asymmetry

ABSTRACT We conducted three experiments to analyze the respective roles of the joint influence account and the word production architecture account within the cross-modal Stroop effect. We utilized a varied time course of targets and distractors and investigated the possible asymmetry of visual and auditory targets to determine the underlying mechanism of interference. In Experiment 1, we used color patches as the visual target. We found color-word Stroop asymmetry that supported the word-production architecture account; there was a Stroop interference effect when participants named visual but not auditory targets. In Experiment 2, we used printed words as the color carrier instead of color patches. The size of interference in visual targets in Experiment 2 was still greater than in auditory targets and was driven by additive interference from the visual distractor dimension, which supported the joint influence account. In Experiment 3, which was a replication of Experiment 2 that removed the integrated incongruent visual distractor, we concluded that target and distractor integration played a key role in moderating the size of the interference effect, and that the lesser interference after removing the incongruent visual distractor supported the joint influence account. In almost every possible combination of conditions, we detected color-word Stroop asymmetry, and while the results were consistent with both the joint influence and word production architecture account when color patches were used as the color carrier, the joint influence account could more completely explain the findings regardless of the color carrier. Further research should be done to establish how target and distractor integration operates within and across modalities, as well as the contexts in which the word production architecture account applies.

The effect of gender stereotypes on cross-modal spatial attention

Using a spatial-cueing paradigm in which trait words were set as visual cues and gender words were set as auditory targets, we examined whether cross-modal spatial attention was influenced by gender stereotypes. Results of an experiment conducted with 24 participants indicate that they tended to focus on targets in the valid-cue condition (i.e., the cues located at the same position as targets), regardless of the modality of cues and targets, which is consistent with the cross-modal attention effect found in previous studies. Participants tended to focus on targets that were stereotype-consistent with cues only when the cues were valid, which shows that stereotype-consistent information facilitated visual–auditory cross-modal spatial attention. These results suggest that cognitive schema, such as gender stereotypes, have an effect on cross-modal spatial attention.

Evidence for an A-Modal Number Sense: Numerosity Adaptation Generalizes Across Visual, Auditory, and Tactile Stimuli.

Humans and other species share a perceptual mechanism dedicated to the representation of approximate quantities that allows to rapidly and reliably estimate the numerosity of a set of objects: an Approximate Number System (ANS). Numerosity perception shows a characteristic shared by all primary visual features: it is susceptible to adaptation. As a consequence of prolonged exposure to a large/small quantity (“adaptor”), the apparent numerosity of a subsequent (“test”) stimulus is distorted yielding a robust under- or over-estimation, respectively. Even if numerosity adaptation has been reported across several sensory modalities (vision, audition, and touch), suggesting the idea of a central and a-modal numerosity processing system, evidence for cross-modal effects are limited to vision and audition, two modalities that are known to preferentially encode sensory stimuli in an external coordinate system. Here we test whether numerosity adaptation for visual and auditory stimuli also distorts the perceived numerosity of tactile stimuli (and vice-versa) despite touch being a modality primarily coded in an internal (body-centered) reference frame. We measured numerosity discrimination of stimuli presented sequentially after adaptation to series of either few (around 2 Hz; low adaptation) or numerous (around 8 Hz; high adaptation) impulses for all possible combinations of visual, auditory, or tactile adapting and test stimuli. In all cases, adapting to few impulses yielded a significant overestimation of the test numerosity with the opposite occurring as a consequence of adaptation to numerous stimuli. The overall magnitude of adaptation was robust (around 30%) and rather similar for all sensory modality combinations. Overall, these findings support the idea of a truly generalized and a-modal mechanism for numerosity representation aimed to process numerical information independently from the sensory modality of the incoming signals.

Olfactory Stimulation Modulates Visual Perception Without Training.

Considerable research shows that olfactory stimulations affect other modalities in high-level cognitive functions such as emotion. However, little known fact is that olfaction modulates low-level perception of other sensory modalities. Although some studies showed that olfaction had influenced on the other low-level perception, all of them required specific experiences like perceptual training. To test the possibility that olfaction modulates low-level perception without training, we conducted a series of psychophysical and neuroimaging experiments. From the results of a visual task in which participants reported the speed of moving dots, we found that participants perceived the slower motions with a lemon smell and the faster motions with a vanilla smell, without any specific training. In functional magnetic resonance imaging (fMRI) studies, brain activities in the visual cortices [V1 and human middle temporal area (hMT)] changed based on the type of olfactory stimulation. Our findings provide us with the first direct evidence that olfaction modulates low-level visual perception without training, thereby indicating that olfactory-visual effect is not an acquired behavior but an innate behavior. The present results show us with a new crossmodal effect between olfaction and vision, and bring a unique opportunity to reconsider some fundamental roles of olfactory function.

Rhythmic Modulation of Visual Perception by Continuous Rhythmic Auditory Stimulation.

At any given moment our sensory systems receive multiple, often rhythmic, inputs from the environment. Processing of temporally structured events in one sensory modality can guide both behavioral and neural processing of events in other sensory modalities, but whether this occurs remains unclear. Here, we used human electroencephalography (EEG) to test the cross-modal influences of a continuous auditory frequency-modulated (FM) sound on visual perception and visual cortical activity. We report systematic fluctuations in perceptual discrimination of brief visual stimuli in line with the phase of the FM-sound. We further show that this rhythmic modulation in visual perception is related to an accompanying rhythmic modulation of neural activity recorded over visual areas. Importantly, in our task, perceptual and neural visual modulations occurred without any abrupt and salient onsets in the energy of the auditory stimulation and without any rhythmic structure in the visual stimulus. As such, the results provide a critical validation for the existence and functional role of cross-modal entrainment and demonstrates its utility for organizing the perception of multisensory stimulation in the natural environment.SIGNIFICANCE STATEMENT Our sensory environment is filled with rhythmic structures that are often multi-sensory in nature. Here, we show that the alignment of neural activity to the phase of an auditory frequency-modulated (FM) sound has cross-modal consequences for vision: yielding systematic fluctuations in perceptual discrimination of brief visual stimuli that are mediated by accompanying rhythmic modulation of neural activity recorded over visual areas. These cross-modal effects on visual neural activity and perception occurred without any abrupt and salient onsets in the energy of the auditory stimulation and without any rhythmic structure in the visual stimulus. The current work shows that continuous auditory fluctuations in the natural environment can provide a pacing signal for neural activity and perception across the senses.

The Role of Unimodal Feedback Pathways in Gender Perception During Activation of Voice and Face Areas.

Cross-modal effects provide a model framework for investigating hierarchical inter-areal processing, particularly, under conditions where unimodal cortical areas receive contextual feedback from other modalities. Here, using complementary behavioral and brain imaging techniques, we investigated the functional networks participating in face and voice processing during gender perception, a high-level feature of voice and face perception. Within the framework of a signal detection decision model, Maximum likelihood conjoint measurement (MLCM) was used to estimate the contributions of the face and voice to gender comparisons between pairs of audio-visual stimuli in which the face and voice were independently modulated. Top–down contributions were varied by instructing participants to make judgments based on the gender of either the face, the voice or both modalities (N = 12 for each task). Estimated face and voice contributions to the judgments of the stimulus pairs were not independent; both contributed to all tasks, but their respective weights varied over a 40-fold range due to top–down influences. Models that best described the modal contributions required the inclusion of two different top–down interactions: (i) an interaction that depended on gender congruence across modalities (i.e., difference between face and voice modalities for each stimulus); (ii) an interaction that depended on the within modalities’ gender magnitude. The significance of these interactions was task dependent. Specifically, gender congruence interaction was significant for the face and voice tasks while the gender magnitude interaction was significant for the face and stimulus tasks. Subsequently, we used the same stimuli and related tasks in a functional magnetic resonance imaging (fMRI) paradigm (N = 12) to explore the neural correlates of these perceptual processes, analyzed with Dynamic Causal Modeling (DCM) and Bayesian Model Selection. Results revealed changes in effective connectivity between the unimodal Fusiform Face Area (FFA) and Temporal Voice Area (TVA) in a fashion that paralleled the face and voice behavioral interactions observed in the psychophysical data. These findings explore the role in perception of multiple unimodal parallel feedback pathways.

Exploring Cross-Modal Influences on the Evaluation of Indoor-Environmental Conditions

The comfort requirements of occupants influence indoor-environmental factors and energy performance of buildings. Occupants are typically exposed to a multitude of indoor-environmental factors, including a variety of different thermal, auditory, visual, and air quality conditions. However, the bulk of past research and derivative indoor-environmental codes and standards concerning the comfort of occupants address the multiple indoor-environmental stimuli in isolation. Starting from a brief review of past research on multi-perceptual indoor-environmental assessments of occupants, the present study pursues an experimental approach to explore the potential cross-modal effects on the evaluation of indoor-environmental thermal, visual, and acoustic aspects. In this context, a laboratory space including two adjacent identical mock-up office rooms was used to conduct multi-aspect parametric studies with human participants. Different thermal, visual, and auditory conditions were maintained in these two units. In the course of the present study, 296 participants were exposed, on a short-term basis, to different combinations of thermal, visual, and auditory conditions. The experiments were intended to explore if the evaluation of one aspect of the indoor environment could be influenced by differences in the values pertaining to the other aspects. The experimental results are presented and discussed, including their limitations.

Galvanic Vestibular Stimulation Produces Cross-Modal Improvements in Visual Thresholds.

BackgroundStochastic resonance (SR) refers to a faint signal being enhanced with the addition of white noise. Previous studies have found that vestibular perceptual thresholds are lowered with noisy galvanic vestibular stimulation (i.e., “in-channel” SR). Auditory white noise has been shown to improve tactile and visual thresholds, suggesting “cross-modal” SR.ObjectiveWe investigated galvanic vestibular white noise (nGVS) (n = 9 subjects) to determine the cross-modal effects on visual and auditory thresholds.MethodsWe measured auditory and visual perceptual thresholds of human subjects across a swath of different nGVS levels in order to determine if some individual-subject determined best nGVS level elicited a reduction in thresholds as compared the no noise condition (sham).ResultsWe found improvement in visual thresholds (by an average of 18%, p = 0.014). Subjects with higher (worse) visual thresholds with no stimulation (sham) improved more than those with lower thresholds (p = 0.04). Auditory thresholds were unchanged by vestibular stimulation.ConclusionThese results are the first demonstration of cross-modal improvement with galvanic vestibular stimulation, indicating galvanic vestibular white noise can produce cross-modal improvements in some sensory channels, but not all.

Cross-Modal Effect of Presenting Visual and Force Feedback That Create the Illusion of Stair-Climbing

Image therapy, which creates illusions with a mirror and a head mount display, assists movement relearning in stroke patients. Mirror therapy presents the movement of the unaffected limb in a mirror, creating the illusion of movement of the affected limb. As the visual information of images cannot create a fully immersive experience, we propose a cross-modal strategy that supplements the image with sensual information. By interacting with the stimuli received from multiple sensory organs, the brain complements missing senses, and the patient experiences a different sense of motion. Our system generates the sense of stair-climbing in a subject walking on a level floor. The force sensation is presented by a pneumatic gel muscle (PGM). Based on motion analysis in a human lower-limb model and the characteristics of the force exerted by the PGM, we set the appropriate air pressure of the PGM. The effectiveness of the proposed system was evaluated by surface electromyography and a questionnaire. The experimental results showed that by synchronizing the force sensation with visual information, we could match the motor and perceived sensations at the muscle-activity level, enhancing the sense of stair-climbing. The experimental results showed that the visual condition significantly improved the illusion intensity during stair-climbing.

The development of visuotactile congruency effects for sequences of events

Sensitivity to the temporal coherence of visual and tactile signals increases perceptual reliability and is evident during infancy. However, it is not clear how, or whether, bidirectional visuotactile interactions change across childhood. Furthermore, no study has explored whether viewing a body modulates how children perceive visuotactile sequences of events. Here, children aged 5–7 years (n = 19), 8 and 9 years (n = 21), and 10–12 years (n = 24) and adults (n = 20) discriminated the number of target events (one or two) in a task-relevant modality (touch or vision) and ignored distractors (one or two) in the opposing modality. While participants performed the task, an image of either a hand or an object was presented. Children aged 5–7 years and 8 and 9 years showed larger crossmodal interference from visual distractors when discriminating tactile targets than the converse. Across age groups, this was strongest when two visual distractors were presented with one tactile target, implying a “fission-like” crossmodal effect (perceiving one event as two events). There was no influence of visual context (viewing a hand or non-hand image) on visuotactile interactions for any age group. Our results suggest robust interference from discontinuous visual information on tactile discrimination of sequences of events during early and middle childhood. These findings are discussed with respect to age-related changes in sensory dominance, selective attention, and multisensory processing.

Enhancing Reading Comprehension in First Graders: The Effects of Two Training Programs Provided in Listening or Written Modality

<p style="text-align:justify">It has been well documented that oral language skills are precursors of the development of written abilities, meaning that improving oral language skills, e.g. vocabulary, listening comprehension, could have positive effects reading comprehension. The main aim of the present study was to analyze the effect of a training program focused on four components of comprehension ability provided in listening or written modality on first-graders’ reading and listening comprehension. The training programs were implemented by school teachers as part of the class’s normal school activities, under the supervision of experts. Their efficacy was compared with the results obtained in a passive control group. Our results showed that both the training programs produced positive effects on listening and reading comprehension (by comparison with a passive control group), and that the gains in reading comprehension were maintained at a five-month follow-up. Findings demonstrated specific improvements in listening comprehension with the listening modality program, as well as an improvement in the modality not directly trained. A similar cross-modality effect was found from the written modality training group, therefore confirming the strict relation between oral and written abilities. Reading comprehension performance of first graders can be improved with activities provided in listening and written modality. Hence, it is possible to enhance reading comprehension even before decoding and fluency in reading are acquired.</p>

Compensatory cross-modal effects of sentence context on visual word recognition in adults

Reading involves mapping combinations of a learned visual code (letters) onto meaning. Previous studies have shown that when visual word recognition is challenged by visual degradation, one way to mitigate these negative effects is to provide "top–down" contextual support through a written congruent sentence context. Crowding is a naturally occurring visual phenomenon that impairs object recognition and also affects the recognition of written stimuli during reading. Thus, access to a supporting semantic context via a written text is vulnerable to the detrimental impact of crowding on letters and words. Here, we suggest that an auditory sentence context may provide an alternative source of semantic information that is not influenced by crowding, thus providing “top–down” support cross-modally. The goal of the current study was to investigate whether adult readers can cross-modally compensate for crowding in visual word recognition using an auditory sentence context. The results show a significant cross-modal interaction between the congruency of the auditory sentence context and visual crowding, suggesting that interactions can occur across multiple levels of processing and across different modalities to support reading processes. These findings highlight the need for reading models to specify in greater detail how top–down, cross-modal and interactive mechanisms may allow readers to compensate for deficiencies at early stages of visual processing.

Crossmodal Effects in Task Switching: Modality Compatibility with Vocal and Pedal Responses.

Modality compatibility refers to the similarity between the stimulus modality and the modality of response-related sensory consequences (e.g., vocal output produces audible effects). While previous studies found higher costs of task switching with stimulus-response modality-incompatible tasks (auditory-manual and visual-vocal), the present study was aimed to explore the generality of modality compatibility by examining a new response modality (pedal responses). Experiment 1 showed that the effect of modality compatibility generalizes to pedal responses when these replaced manual responses used in previous studies (i.e., higher switch costs when switching between auditory-pedal and visual-vocal tasks compared to switching between auditory-vocal and visual-pedal tasks). However, in single-task conditions there was no influence of modality compatibility. Experiment 2 was designed to examine whether modality compatibility depends on the frequency of task switches. To this end, one task occurred very frequently, overall decreasing the task switching frequency. Importantly, the results showed a robust task-switching benefit of modality-compatible mappings even for a highly frequent task, suggesting that the sustained representation of potentially competing response modalities affects task-switching performance independent from the actual frequency of the tasks. Together, the data suggest that modality compatibility is an emergent phenomenon arising in task-switching situations based on the necessity to maintain but at the same time separate competing modality mappings, which are characterized by ideomotor ‘‘backward’’ linkages between anticipated response effects and the stimuli that called for this response in the first place.

Choice-dependent cross-modal interaction in the medial prefrontal cortex of rats

Cross-modal interaction (CMI) could significantly influence the perceptional or decision-making process in many circumstances. However, it remains poorly understood what integrative strategies are employed by the brain to deal with different task contexts. To explore it, we examined neural activities of the medial prefrontal cortex (mPFC) of rats performing cue-guided two-alternative forced-choice tasks. In a task requiring rats to discriminate stimuli based on auditory cue, the simultaneous presentation of an uninformative visual cue substantially strengthened mPFC neurons' capability of auditory discrimination mainly through enhancing the response to the preferred cue. Doing this also increased the number of neurons revealing a cue preference. If the task was changed slightly and a visual cue, like the auditory, denoted a specific behavioral direction, mPFC neurons frequently showed a different CMI pattern with an effect of cross-modal enhancement best evoked in information-congruent multisensory trials. In a choice free task, however, the majority of neurons failed to show a cross-modal enhancement effect and cue preference. These results indicate that CMI at the neuronal level is context-dependent in a way that differs from what has been shown in previous studies.

Neural plasticity: The substratum of music-based interventions in neurorehabilitation.

The plastic nature of the human brain lends itself to experience and training-based structural changes leading to functional recovery. Music, with its multimodal activation of the brain, serves as a useful model for neurorehabilitation through neuroplastic changes in dysfunctional or impaired networks. Neurologic Music Therapy (NMT) contributes to the field of neurorehabilitation using this rationale. The purpose of this article is to present a discourse on the concept of neuroplasticity and music-based neuroplasticity through the techniques of NMT in the domain of neurological rehabilitation. The article draws on observations and findings made by researchers in the areas of neuroplasticity, music-based neuroplastic changes, NMT in neurological disorders and the implication of further research in this field. A commentary on previous research reveal that interventions based on the NMT paradigm have been successfully used to train neural networks using music-based tasks and paradigms which have been explained to have cross-modal effects on sensorimotor, language and cognitive and affective functions. Multimodal gains using music-based interventions highlight the brain plasticity inducing function of music. Individual differences do play a predictive role in neurological gains associated with such interventions. This area deserves further exploration and application-based studies.