Cross-modal Information Processing Research Articles

The role of semantically related gestures in the language comprehension of simultaneous interpreters in noise

ABSTRACT Manual co-speech gestures can facilitate language comprehension, especially in adverse listening conditions. However, we do not know whether gestures influence simultaneous interpreters’ language comprehension in adverse listening conditions, and if so, whether this influence is modulated by interpreting experience, or by active simultaneous interpreting (SI). We exposed 24 interpreters and 24 bilinguals without interpreting experience to utterances with semantically related gestures, semantically unrelated gestures, or without gestures while engaging in comprehension (interpreters and bilinguals) or in SI (interpreters only). Tasks were administered in clear and noisy speech. Accuracy and reaction time were measured, and participants’ gaze was tracked. During comprehension, semantically related gestures facilitated both groups’ processing in noise. Facilitation was not modulated by interpreting experience. However, when interpreting noisy speech, interpreters did not benefit from gestures. This suggests that the comprehension component, and specifically crossmodal information processing, in SI differs from that of other language comprehension.

Cross-modal Semantically Augmented Network for Image-text Matching

Image-text matching plays an important role in solving the problem of cross-modal information processing. Since there are nonnegligible semantic differences between heterogenous pairwise data, a crucial challenge is how to learn a unified representation. Existing methods mainly rely on the alignment between regional image features and corresponding entity words. However, the regional features in the image are often more concerned with the foreground entity information, and the attribute information of the entities and the relational information are ignored. How to effectively integrate entity-attribute alignment and relationship alignment has not been fully studied. Therefore, we propose a Cross-Modal Semantically Augmented Network for Image-Text Matching (CMSAN), which combines the relationships between entities in the image with the semantics of relational words in the text. CMSAN (1) proposes an adaptive word-type prediction model that classifies the words into four types, i.e., entity word, attribute word, relation word, and unnecessary word. It can align different image features at multiple levels. CMSAN (2) designs a sophisticated relationship alignment module and an entity-attribute alignment module that maximizes the exploitation of the semantic information, which enables the model to have more discriminative power and further improves the matching accuracy.

MeshCLIP: Efficient cross-modal information processing for 3D mesh data in zero/few-shot learning

Text, 2D, and 3D information are crucial information representations in modern science and management disciplines. However, complex and irregular 3D data produce data scarcity and expensive generation that limit their processing and application. In this paper, we present MeshCLIP, a new cross-modal information learning paradigm to directly process 3D mesh data end-to-end in a zero/few-shot manner. Specifically, we design a novel pipeline based on visual factors and graphics principles, bridging the gap between 3D mesh data and other modal data, thereby joining 2/3D visual and textual information for zero/few-shot learning. Then, we construct a self-attention adapter for 3D mesh key information learning when training only a few priors, significantly improving the model’s discriminative ability. Extensive experiments demonstrate that the proposed MeshCLIP can achieve state-of-the-art results on multiple challenging 3D mesh datasets. In the whole 3D domain, the proposed zero-shot approach significantly outperforms the existing other 3D representation methods with an accuracy 3 × better ( increased by 41.5%) on the ModelNet40 dataset. Furthermore, in few-shot learning, the proposed MeshCLIP uses only a few supervised priors (only less than 10% of the sample size) to achieve results close to those of methods trained on a full dataset.

Excitatory Crossmodal Input to a Widespread Population of Primary Sensory Cortical Neurons.

Crossmodal information processing in sensory cortices has been reported in sparsely distributed neurons under normal conditions and can undergo experience- or activity-induced plasticity. Given the potential role in brain function as indicated by previous reports, crossmodal connectivity in the sensory cortex needs to be further explored. Using perforated whole-cell recording in anesthetized adult rats, we found that almost all neurons recorded in the primary somatosensory, auditory, and visual cortices exhibited significant membrane-potential responses to crossmodal stimulation, as recorded when brain activity states were pharmacologically down-regulated in light anesthesia. These crossmodal cortical responses were excitatory and subthreshold, and further seemed to be relayed primarily by the sensory thalamus, but not the sensory cortex, of the stimulated modality. Our experiments indicate a sensory cortical presence of widespread excitatory crossmodal inputs, which might play roles in brain functions involving crossmodal information processing or plasticity.

Auditory, Visual, and Cross-Modal Mismatch Negativities in the Rat Auditory and Visual Cortices.

When the brain tries to acquire an elaborate model of the world, multisensory integration should contribute to building predictions based on the various pieces of information, and deviance detection should repeatedly update these predictions by detecting “errors” from the actual sensory inputs. Accumulating evidence such as a hierarchical organization of the deviance-detection system indicates that the deviance-detection system can be interpreted in the predictive coding framework. Herein, we targeted mismatch negativity (MMN) as a type of prediction-error signal and investigated the relationship between multisensory integration and MMN. In particular, we studied whether and how cross-modal information processing affected MMN in rodents. We designed a new surface microelectrode array and simultaneously recorded visual and auditory evoked potentials from the visual and auditory cortices of rats under anesthesia. Then, we mapped MMNs for five types of deviant stimuli: single-modal deviants in (i) the visual oddball and (ii) auditory oddball paradigms, eliciting single-modal MMN; (iii) congruent audio-visual deviants, (iv) incongruent visual deviants, and (v) incongruent auditory deviants in the audio-visual oddball paradigm, eliciting cross-modal MMN. First, we demonstrated that visual MMN exhibited deviance detection properties and that the first-generation focus of visual MMN was localized in the visual cortex, as previously reported in human studies. Second, a comparison of MMN amplitudes revealed a non-linear relationship between single-modal and cross-modal MMNs. Moreover, congruent audio-visual MMN exhibited characteristics of both visual and auditory MMNs—its latency was similar to that of auditory MMN, whereas local blockage of N-methyl-D-aspartic acid receptors in the visual cortex diminished it as well as visual MMN. These results indicate that cross-modal information processing affects MMN without involving strong top-down effects, such as those of prior knowledge and attention. The present study is the first electrophysiological evidence of cross-modal MMN in animal models, and future studies on the neural mechanisms combining multisensory integration and deviance detection are expected to provide electrophysiological evidence to confirm the links between MMN and predictive coding theory.

Visual speech cues recruit neural oscillations to optimise auditory perception: Ways forward for research on human communication.

In pandemic times, when visual speech cues are masked, it becomes particularly evident how much we rely on them to communicate. Recent research points to a key role of neural oscillations for cross-modal predictions during speech perception. This article bridges several fields of research – neural oscillations, cross-modal speech perception and brain stimulation – to propose ways forward for research on human communication. Future research can test: (1) whether “speech is special” for oscillatory processes underlying cross-modal predictions; (2) whether “visual control” of oscillatory processes in the auditory system is strongest in moments of reduced acoustic regularity; and (3) whether providing information to the brain via electric stimulation can overcome deficits associated with cross-modal information processing in certain pathological conditions.

Learning Dual Semantic Relations With Graph Attention for Image-Text Matching

Image-Text Matching is one major task in cross-modal information processing. The main challenge is to learn the unified visual and textual representations. Previous methods that perform well on this task primarily focus on not only the alignment between region features in images and the corresponding words in sentences, but also the alignment between relations of regions and relational words. However, the lack of joint learning of regional features and global features will cause the regional features to lose contact with the global context, leading to the mismatch with those non-object words which have global meanings in some sentences. In this work, in order to alleviate this issue, it is necessary to enhance the relations between regions and the relations between regional and global concepts to obtain a more accurate visual representation so as to be better correlated to the corresponding text. Thus, a novel multi-level semantic relations enhancement approach named Dual Semantic Relations Attention Network(DSRAN) is proposed which mainly consists of two modules, separate semantic relations module and the joint semantic relations module. DSRAN performs graph attention in both modules respectively for region-level relations enhancement and regional-global relations enhancement at the same time. With these two modules, different hierarchies of semantic relations are learned simultaneously, thus promoting the image-text matching process by providing more information for the final visual representation. Quantitative experimental results have been performed on MS-COCO and Flickr30K and our method outperforms previous approaches by a large margin due to the effectiveness of the dual semantic relations learning scheme. Codes are available at https://github.com/kywen1119/DSRAN.

Mechanism of Cross-modal Information Influencing Taste

Studies on the integration of cross-modal information with taste perception has been mostly limited to uni-modal level. The cross-modal sensory interaction and the neural network of information processing and its control were not fully explored and the mechanisms remain poorly understood. This mini review investigated the impact of uni-modal and multi-modal information on the taste perception, from the perspective of cognitive status, such as emotion, expectation and attention, and discussed the hypothesis that the cognitive status is the key step for visual sense to exert influence on taste. This work may help researchers better understand the mechanism of cross-modal information processing and further develop neutrally-based artificial intelligent (AI) system.

See, Hear, and Learn to Describe--Crossmodal Information Processing Opens the Way to Smarter AI

See, Hear, and Learn to Describe--Crossmodal Information Processing Opens the Way to Smarter AI

Cross-modal representations in early visual and auditory cortices revealed by multi-voxel pattern analysis.

Primary sensory cortices can respond not only to their defined sensory modality but also to cross-modal information. In addition to the observed cross-modal phenomenon, it is valuable to research further whether cross-modal information can be valuable for categorizing stimuli and what effect other factors, such as experience and imagination, may have on cross-modal processing. In this study, we researched cross-modal information processing in the early visual cortex (EVC, including the visual area 1, 2, and 3 (V1, V2, and V3)) and auditory cortex (primary (A1) and secondary (A2) auditory cortex). Images and sound clips were presented to participants separately in two experiments in which participants' imagination and expectations were restricted by an orthogonal fixation task and the data were collected by functional magnetic resonance imaging (fMRI). We successfully decoded categories of the cross-modal stimuli in the ROIs except for V1 by multi-voxel pattern analysis (MVPA). It was further shown that familiar sounds had the advantage of classification accuracies in V2 and V3 when compared with unfamiliar sounds. The results of the cross-classification analysis showed that there was no significant similarity between the activity patterns induced by different stimulus modalities. Even though the cross-modal representation is robust when considering the restriction of top-down expectations and mental imagery in our experiments, the sound experience showed effects on cross-modal representation in V2 and V3. In addition, primary sensory cortices may receive information from different modalities in different ways, so the activity patterns between two modalities were not similar enough to complete the cross-classification successfully.

Crossmodal matching

Multimodal displays, i.e., displays that distribute information across multiple sensory channels (mainly vision, hearing, and touch), have received considerable attention in recent years. To be effective, their design needs to be based on a firm understanding of how information is processed both within and across modalities. However, most studies on crossmodal information processing, to date, suffer from a methodological shortcoming: they fail to perform crossmodal matching to ensure that modality is not confounded with other stimulus properties, such as salience. One reason for this shortcoming is the fact that there is no agreed-upon crossmodal matching technique, and that existing approaches suffer from limitations. The goal of the present study is to develop and validate a more reliable crossmodal matching method that employs repeated matching. To this end, six participants were asked to use this technique and match a series of 54 modality pairings involving vision, audition, and touch. Results show that the intra-individual variability of participants’ matches was significantly less than observed in an earlier technique that involved bidirectional matching and visual feedback. The findings from this research confirm the need for improved crossmodal matching procedures and for employing them in advance of conducting experiments on multisensory information processing.

The Crossmodal Effect of Attention on Preferences: Facilitation versus Impairment

This article builds a conceptualization for the crossmodal effect of attention on preferences, predicting when and why an irrelevant auditory signal will facilitate or impair preferences for visually processed target products located in the direction of the signal. Extending perspectives on crossmodal attention, this conceptualization posits that the functional tendency to pay visual attention toward an auditory signal will translate to a facilitation effect on preferences. However, given a goal of signal avoidance, crossmodal functionality dictates a lowering of visual attention toward the signal, impairing preferences for targets in that direction. Finally, a two-stage model of involuntary and voluntary attention is invoked to reconcile opposing predictions: an aversive noise is held to produce initial facilitation because of an involuntary appraisal mechanism, before a more deliberative attention-allocation process produces impairment. Results from five experiments support these predictions, contributing to the literature on crossmodal information processing and also that on preference formation.

Contrasting Effects of Dual-task Paradigm and of Timing Interruption Paradigm in Interval Timing of the Context of Culti-modal Processing

Distribution of attention in time information processing is one of the hot areas of research science, and the dual-task paradigm is one of the most common ways to study distribution of attention. It requires an individual to perform two tasks simultaneously, the less the attention allocated to a temporal interval, the shorter it is judged(Brown, 1997). The attention sharing effect is discussed within the framework of the scalar expectancy model of timing. In such paradigm the parallel processing itself may interfere with time perception and lead to unexpected deviations. In order to avoid such interference, the timing interruption paradigm would be a better solution as it uses a blank interruption instead. The researchers consistently found a similar position and interruption effect in both paradigms (Casini Macar, 1997; Cortin, Remblai,, 2006; Remblai, Cortin,, 2003). Furthermore, the results showed both the discontinuity and interference of current information processingwere belong to interruption effect, but to varying extents (Cortin, Masse, 2000; Macar, 2002). However, though the position and interruption effect were similar in the two paradigms, they have not been explored in a same stimuli series. As we know, information exchange with the outside world is not dependent on single sensory channel, but rather the interaction of cross-modal information processing. It would be valuable to explore the position and interruption effect in the context ofcross-modal processing. It would not only help to uncover the cognitive mechanism of time processing, but also have important practical values as it is more similar with daily life. Therefore, the present study was designed to investigate the position and interruption effect in the two paradigms in the cross-modal conditions. To this end, the study consisted of two experiments. In experiment 1, 2500 ms and 4500 ms were set for the target time intervals, using the same stimulus sequence (visual presentation, with aural interruption), participants were allocated to control, break and interference condition respectively. In experiment 2,the target intervals were set to 1500 ms and 2500 ms. Results of experiment 1 showed that the interruption effect is more significant in break condition regardless of target time intervals. Furthermore, under the 2500ms, position effect were found in all three conditions, whereas under the 4500ms condition, the position effect only existed in the break condition. Experiment 2 found that there was position effect consistently, regardless of the interpolation conditions or target time intervals. Besides, the interrupt effect was more significant in the break condition. Such results imply that whether the position effect is the same in the two paradigms depends on the length ofthe target intervals to some extent. Besides, in the timing interruption paradigm, the break effect on timing is more significant than the interference effect in dual-task paradigm.

動的な変形に対する視触覚間同時性判断

Simultaneity is important in cross-modal information processing. However, it is still unclear how simultaneity is perceived between different sensory modalities. Various factors such as spatial location or attention are known to affect simultaneity judgments. In the present study, we focused on the simultaneity judgments of dynamic events, and investigated what kinds of dynamic properties affect these judgments. We presented the deformation of a virtual object in vision and haptics with various stimulus onset asynchronies. Participants judged whether the deformation occurred simultaneously. We measured the effects of duration, velocity, and the amount of deformation on the visual-haptic simultaneity judgments. The results showed that the point of subjective simultaneity changed depending on the duration of deformation. For a shorter duration (400 ms), the visual deformation needed to precede the haptic one to be judged as simultaneous, while for a longer duration (800 ms, 1 200 ms), the asymmetry was diminished, suggesting that information relevant to the duration of the event was used for the vision-haptics simultaneity judgments of dynamic events.

What speakers do and what addressees look at

This study investigates whether addressees visually attend to speakers’ gestures in interaction and whether attention is modulated by changes in social setting and display size. We compare a live face-to-face setting to two video conditions. In all conditions, the face dominates as a fixation target and only a minority of gestures draw fixations. The social and size parameters affect gaze mainly when combined and in the opposite direction from the predicted with fewer gestures fixated on video than live. Gestural holds and speakers’ gaze at their own gestures reliably attract addressees’ fixations in all conditions. The attraction force of holds is unaffected by changes in social and size parameters, suggesting a bottom-up response, whereas speaker-fixated gestures draw significantly less attention in both video conditions, suggesting a social effect for overt gaze-following and visual joint attention. The study provides and validates a video-based paradigm enabling further experimental but ecologically valid explorations of cross-modal information processing.

Synesthesia: Strong and Weak

In this review, we distinguish strong and weak forms of synesthesia. Strong synesthesia is characterized by a vivid image in one sensory modality in response to stimulation in another one. Weak synesthesia is characterized by cross-sensory correspondences expressed through language, perceptual similarity, and perceptual interactions during information processing. Despite important phenomenological dissimilarities between strong and weak synesthesia, we maintain that the two forms draw on similar underlying mechanisms. The study of strong and weak synesthetic phenomena provides an opportunity to enrich scientists' understanding of basic mechanisms involved in perceptual coding and cross-modal information processing.

Short-term Memory and Cross-modal Information Processing in Autistic Children

The short-term memory and cross-modal information processing capacities of seven autistic/schizophrenic children were tested with visual-visual and auditory-visual matching and sequencing tasks presented under three conditions: (1) simultaneous, (2) successive, and (3) delayed presentation. Although there were large individual differences in the patterning of intact functions, most of the children showed severe visual-visual short-term memory deficits and several children were unable to make cross-modal associations between auditory and visual stimuli. The results suggest that perceptual disabilities rather than emotional disturbance may underly the delayed intellectual and language development of psychotic children and may contribute to the development of bizarre behavior patterns.