Cross-modal Matching Research Articles

Dual-View Semantic Inference Network for image-text matching

Recently, image-text matching based on local region-word semantic alignment has attracted considerable research attention. The fine-grained interplay can be achieved by aggregating the similarities of the region-word pairs. However, the similarities of aligned region-word pairs are treated equally in most cross-modal matching literatures, without considering their respective importance. Moreover, the local alignment methods are prone to bring about a global semantic drift due to the ignorance of thematic considerations for the image-text pairs. In this paper, a novel Dual-View Semantic Inference (DVSI) network is proposed to leverage both local and global semantic matching in a holistic deep framework. For the local view, a region enhancement module is proposed to mine the priorities for different regions in the image, which provides differentiate abilities to discover the latent region-word relationships. For the global view, the overall semantics of image is summarized for global semantic matching to avoid global semantic drift. The two views are unified together for final image-text matching. Extensive experiments conducted on MSCOCO and Flicr30K demonstrate the effectiveness of the proposed DVSI.

Twenty-four or four-and-twenty: Language modulates cross-modal matching for multidigit numbers in children and adults

Does number–word structure have a long-lasting impact on transcoding? Contrary to English, German number words comprise decade–unit inversion (e.g., vierundzwanzig is literally translated as four-and-twenty). To investigate the mental representation of numbers, we tested the effect of visual and linguistic–morphological characteristics on the development of verbal–visual transcoding. In a longitudinal cross-linguistic design, response times (RTs) in a number-matching experiment were analyzed in Grade 2 (119 German-speaking and 179 English-speaking children) and in Grade 3 (131 German-speaking and 160 English-speaking children). To test for long-term effects, the same experiment was given to 38 German-speaking and 42 English-speaking adults. Participants needed to decide whether a spoken number matched a subsequent visual Arabic number. Systematic variation of digits in the nonmatching distractors allowed comparison of three different transcoding accounts (lexicalization, visual, and linguistic–morphological). German speakers were generally slower in rejecting inverted number distractors than English speakers. Across age groups, German speakers were more distracted by Arabic numbers that included the correct unit digit, whereas English speakers showed stronger distraction when the correct decade digit was included. These RT patterns reflect differences in number–word morphology. The individual cost of rejecting an inverted distractor (inversion effect) predicted arithmetic skills in German-speaking second-graders only. The moderate relationship between the efficiency to identify a matching number and arithmetic performance could be observed cross-linguistically in all age groups but was not significant in German-speaking adults. Thus, findings provide consistent evidence of a persistent impact of number–word structure on number processing, whereas the relationship with arithmetic performance was particularly pronounced in young children.

MDA: Multimodal Data Augmentation Framework for Boosting Performance on Sentiment/Emotion Classification Tasks

Multimodal data analysis has drawn increasing attention with the explosive growth of multimedia data. Although traditional unimodal data analysis tasks have accumulated abundant labeled datasets, there are few labeled multimodal datasets due to the difficulty and complexity of multimodal data annotation, nor is it easy to directly transfer unimodal knowledge to multimodal data. Unfortunately, there is little related data augmentation work in multimodal domain, especially for image–text data. In this article, to address the scarcity problem of labeled multimodal data, we propose a Multimodal Data Augmentation framework for boosting the performance on multimodal image–text classification task. Our framework learns a cross-modality matching network to select image–text pairs from existing unimodal datasets as the multimodal synthetic dataset, and uses this dataset to enhance the performance of classifiers. We take the multimodal sentiment analysis and multimodal emotion analysis as the experimental tasks and the experimental results show the effectiveness of our framework for boosting the performance on multimodal classification task.

The temporal dynamics of first and second language processing: ERPs to spoken words in Mandarin-English bilinguals

The dynamics of bilingual spoken word recognition remain poorly characterized, especially for individuals who speak two languages that are highly dissimilar in their phonological and morphological structure. The present study compared first language (L1) and second language (L2) spoken word processing within a group of adult Mandarin-English bilinguals (N = 34; ages 18–25). Event-related potentials (ERPs) were recorded while participants completed the same cross-modal matching task separately in their L1 Mandarin and L2 English. This task consisted of deciding whether spoken words matched pictures of items. Pictures and spoken words either matched (e.g., Mandarin: TANG2-tang2; English: BELL-bell), or differed in word-initial phonemes (e.g., Mandarin: TANG2-lang2; English: BELL-shell), word-final phonemes (e.g., Mandarin: TANG2-tao2; English: BELL-bed), or whole words (e.g., Mandarin: TANG2-xia1: English: BELL-ham). Each mismatch type was associated with a pattern of modulation of the Phonological Mapping Negativity, the N400, and the Late N400 that was distinct from those of the other mismatch types yet similar between the two languages. This was interpreted as evidence of incremental processing with similar temporal dynamics in both languages. These findings support models of spoken word recognition in bilingual individuals that adopt an interactive-activation framework for both L1 and L2 processing.

Modality adversarial neural network for visible-thermal person re-identification

Existing Visible-Thermal Person Re-identification (VT-REID) methods usually adopt two-stream networks for cross-modality images. The two streams are trained to extract features from different modality images respectively. In contrast, we design a Modality Adversarial Neural Network (MANN) to solve VT-REID problem. Our proposed MANN includes a one-stream feature extractor and a modality discriminator. The heterogeneous images are processed by the feature extractor to generate modality-invariant features. And the designed modality discriminator aims to distinguish whether the extracted features are from visible or thermal modality. Moreover, our advanced dual-constrained triplet loss is introduced for better cross-modality matching performance. The experiments on two cross-modality person re-identification datasets show that MANN can effectively learn modality-invariant features and outperform state-of-the-art methods by a large margin.

Deep Multi-Patch Matching Network for Visible Thermal Person Re-Identification

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Visible Thermal Person Re-Identification</i> (VTReID) is a cross-modality retrieval problem in computer vision. Accurate VTReID is very challenging due to large modality discrepancies. In this work, we design a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Multi-Patch Matching Network</i> (MPMN) framework to simultaneously mitigate the heterogeneity of coarse-grained and fine-grained visual semantics. In view of cross-modality matching, we verify that aligning modality distributions of the original features is likely to suffer from the selective alignment behavior, i.e., only focuses on easiest dimensions or subspaces. Inspired by adversarial learning, we propose a new <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Multi-Patch Modality Alignment</i> (MPMA) loss to jointly balance and reduce the modality discrepancies of multi-patch features by mining hard subspaces and abandoning easy subspaces. Since multi-patch features are potentially complementary to each other, the semantic correlations between different patches should be exploited during training. Motivated by knowledge distillation, we put forward a new <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cross-Patch Correlation Distillation</i> (CPCD) loss to transfer the semantic knowledges across different patches. To balance multi-patch tasks, an effective <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Patch-Aware Priority Attention</i> (PAPA) method is further introduced to dynamically prioritize hard patch tasks during training. This paper experimentally demonstrates the effectiveness of the proposed methods, achieving superior performance over the state-of-the-art methods on RegDB and SYSU-MM01 datasets.

HAL: Improved Text-Image Matching by Mitigating Visual Semantic Hubs

The hubness problem widely exists in high-dimensional embedding space and is a fundamental source of error for cross-modal matching tasks. In this work, we study the emergence of hubs in Visual Semantic Embeddings (VSE) with application to text-image matching. We analyze the pros and cons of two widely adopted optimization objectives for training VSE and propose a novel hubness-aware loss function (Hal) that addresses previous methods' defects. Unlike (Faghri et al. 2018) which simply takes the hardest sample within a mini-batch, Hal takes all samples into account, using both local and global statistics to scale up the weights of “hubs”. We experiment our method with various configurations of model architectures and datasets. The method exhibits exceptionally good robustness and brings consistent improvement on the task of text-image matching across all settings. Specifically, under the same model architectures as (Faghri et al. 2018) and (Lee et al. 2018), by switching only the learning objective, we report a maximum R@1 improvement of 7.4% on MS-COCO and 8.3% on Flickr30k.1

Cold and heavy: grasping the temperature\u2013weight illusion

The apparent heaviness of weights placed on the skin depends on their temperature. We studied the effects of such a temperature–weight illusion (TWI) on perception and action in 21 healthy volunteers. Cold (18 °C), thermal-neutral (32 °C, skin temperature) and warm (41 °C) test objects were placed onto the palm of the non-dominant hand. Their veridical mass was 350 g (light) or 700 g (heavy). Perception of heaviness was assessed with two psychophysical experiments (magnitude estimation, cross modal matching). Cold heavy objects felt about 20% heavier than thermal-neutral objects of the same mass, shape and material. In a subsequent grip-lift experiment, the test objects were grasped with a precision grip of the dominant hand and lifted off the palm of the non-dominant hand. The grip and lift forces exerted by the fingertips were recorded. The temperature of the objects had significant effects (ANOVA, p < 0.05) on the peak grip and lift forces and on the peak grip force rate (i.e., the initial force incline). The peak grip force was about 10% higher when cold heavy objects were grasped and lifted, compared to lifts of otherwise identical thermal-neutral objects. The TWI was less pronounced when light objects or warm objects were handled. In conclusion, cooling of an object increases its apparent heaviness (perception) and influences scaling of the fingertip forces during grasping and lifting (action).

Crossmodal Congruency Enhances Performance of Healthy Older Adults in Visual-Tactile Pattern Matching.

One of the pivotal challenges of aging is to maintain independence in the activities of daily life. In order to adapt to changes in the environment, it is crucial to continuously process and accurately combine simultaneous input from different sensory systems, i.e., crossmodal or multisensory integration. With aging, performance decreases in multiple domains, affecting bottom-up sensory processing as well as top-down control. However, whether this decline leads to impairments in crossmodal interactions remains an unresolved question. While some researchers propose that crossmodal interactions degrade with age, others suggest that they are conserved or even gain compensatory importance. To address this question, we compared the behavioral performance of older and young participants in a well-established crossmodal matching task, requiring the evaluation of congruency in simultaneously presented visual and tactile patterns. Older participants performed significantly worse than young controls in the crossmodal task when being stimulated at their individual unimodal visual and tactile perception thresholds. Performance increased with adjustment of stimulus intensities. This improvement was driven by better detection of congruent stimulus pairs, while the detection of incongruent pairs was not significantly enhanced. These results indicate that age-related impairments lead to poor performance in complex crossmodal scenarios and demanding cognitive tasks. Crossmodal congruency effects attenuate the difficulties of older adults in visuotactile pattern matching and might be an important factor to drive the benefits of older adults demonstrated in various crossmodal integration scenarios. Congruency effects might, therefore, be used to develop strategies for cognitive training and neurological rehabilitation.

Vision-Language Navigation Policy Learning and Adaptation.

Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments. In this paper, we study how to address three critical challenges for this task: the cross-modal grounding, the ill-posed feedback, and the generalization problems. First, we propose a novel Reinforced Cross-Modal Matching (RCM) approach that enforces cross-modal grounding both locally and globally via reinforcement learning (RL). Particularly, a matching critic is used to provide an intrinsic reward to encourage global matching between instructions and trajectories, and a reasoning navigator is employed to perform cross-modal grounding in the local visual scene. Evaluation on a VLN benchmark dataset shows that our RCM model significantly outperforms baseline methods by 10 percent on Success Rate weighted by Path Length (SPL) and achieves the state-of-the-art performance. To improve the generalizability of the learned policy, we further introduce a Self-Supervised Imitation Learning (SIL) method to explore and adapt to unseen environments by imitating its own past, good decisions. We demonstrate that SIL can approximate a better and more efficient policy, which tremendously minimizes the success rate performance gap between seen and unseen environments (from 30.7 to 11.7 percent).

RGB-IR Person Re-identification by Cross-Modality Similarity Preservation

Person re-identification (Re-ID) is an important problem in video surveillance for matching pedestrian images across non-overlapping camera views. Currently, most works focus on RGB-based Re-ID. However, RGB images are not well suited to a dark environment; consequently, infrared (IR) imaging becomes necessary for indoor scenes with low lighting and 24-h outdoor scene surveillance systems. In such scenarios, matching needs to be performed between RGB images and IR images, which exhibit different visual characteristics; this cross-modality matching problem is more challenging than RGB-based Re-ID due to the lack of visible colour information in IR images. To address this challenge, we study the RGB-IR cross-modality Re-ID (RGB-IR Re-ID) problem. Rather than applying existing cross-modality matching models that operate under the assumption of identical data distributions between training and testing sets to handle the discrepancy between RGB and IR modalities for Re-ID, we cast learning shared knowledge for cross-modality matching as the problem of cross-modality similarity preservation. We exploit same-modality similarity as the constraint to guide the learning of cross-modality similarity along with the alleviation of modality-specific information, and finally propose a Focal Modality-Aware Similarity-Preserving Loss. To further assist the feature extractor in extracting shared knowledge, we design a modality-gated node as a universal representation of both modality-specific and shared structures for constructing a structure-learnable feature extractor called Modality-Gated Extractor. For validation, we construct a new multi-modality Re-ID dataset, called SYSU-MM01, to enable wider study of this problem. Extensive experiments on this SYSU-MM01 dataset show the effectiveness of our method. Download link of dataset: https://github.com/wuancong/SYSU-MM01.

Cross-Modality Matching for Evaluating User Experience of Emerging Mobile EEG Technology

Emerging technology for brain-state monitoring offers the possibility to conduct measurements outside the laboratory. However, user-experience research is lacking. In this article, we present and test an approach for determining the development of user experience in the course of time using the so-called cross-modality matching (CMM). We conducted experiments with 24 subjects and evaluated seven mobile electroencephalography (EEG) devices. Using the CMM method, we registered the headset pressure of the EEG devices and subject's mood. We are able to identify a correlation between headset pressure and mood and to observe time trends. Subjects rated the heaviest, pin-based device as less comfortable in the course of time. The gel-based EEG cap is the most comfortable device regarding its long-time properties. The CMM approach for user-experience evaluation of new EEG technologies is direct, rapid, and easy to perform. This fact creates new opportunities for future studies in the field of user experience and human factors.

Cross-modality earth mover’s distance-driven convolutional neural network for different-modality data

Cross-modality matching refers to the problem of comparing similarity/dissimilarity of a pair of data points of different modalities, such as an image and a text. Deep neural networks have been popular to represent data points of different modalities due to their ability to extract effective features. However, existing works use simple distance metrics to compare the deep features of multiple modalities, which do not fit the nature of cross-modality matching, because it imposes the features of different modalities to be of the same dimension and do not allow cross-feature matching. To solve this problem, we propose to use convolutional neural network (CNN) models with soft-max activation layer to represent a pair of different-modality data points to two histograms (not necessarily of the same dimensions) and compare their dissimilarity by using earth mover’s distance (EMD). The EMD can match the features extracted by the two CNN models of different modalities freely. Moreover, we develop a joint learning framework to learn the CNN parameters specifically for the EMD-driven comparison, supervised by the relevance/irrelevance labels of the data pairs of different modalities. The experiments over applications such as image–text retrieval, and malware detection show its advantage over existing cross-modality matching methods.

Cross-modal correspondences in sine wave: Speech versus non-speech modes.

The present study aimed to investigate whether or not the so-called "bouba-kiki" effect is mediated by speech-specific representations. Sine-wave versions of naturally produced pseudowords were used as auditory stimuli in an implicit association task (IAT) and an explicit cross-modal matching (CMM) task to examine cross-modal shape-sound correspondences. A group of participants trained to hear the sine-wave stimuli as speech was compared to a group that heard them as non-speech sounds. Sound-shape correspondence effects were observed in both groups and tasks, indicating that speech-specific processing is not fundamental to the "bouba-kiki" phenomenon. Effects were similar across groups in the IAT, while in the CMM task the speech-mode group showed a stronger effect compared with the non-speech group. This indicates that, while both tasks reflect auditory-visual associations, only the CMM task is additionally sensitive to associations involving speech-specific representations.

Disentangled Variational Representation for Heterogeneous Face Recognition

Visible (VIS) to near infrared (NIR) face matching is a challenging problem due to the significant domain discrepancy between the domains and a lack of sufficient data for training cross-modal matching algorithms. Existing approaches attempt to tackle this problem by either synthesizing visible faces from NIR faces, extracting domain-invariant features from these modalities, or projecting heterogeneous data onto a common latent space for cross-modal matching. In this paper, we take a different approach in which we make use of the Disentangled Variational Representation (DVR) for crossmodal matching. First, we model a face representation with an intrinsic identity information and its within-person variations. By exploring the disentangled latent variable space, a variational lower bound is employed to optimize the approximate posterior for NIR and VIS representations. Second, aiming at obtaining more compact and discriminative disentangled latent space, we impose a minimization of the identity information for the same subject and a relaxed correlation alignment constraint between the NIR and VIS modality variations. An alternative optimization scheme is proposed for the disentangled variational representation part and the heterogeneous face recognition network part. The mutual promotion between these two parts effectively reduces the NIR and VIS domain discrepancy and alleviates over-fitting. Extensive experiments on three challenging NIR-VIS heterogeneous face recognition databases demonstrate that the proposed method achieves significant improvements over the state-of-the-art methods.

The Recognition of Solid Object Shape: The Importance of Inhomogeneity.

A single experiment evaluated the haptic-visual cross-modal matching of solid object shape. One set of randomly shaped artificial objects was used (sinusoidally modulated spheres, SMS) as well as two sets of naturally shaped objects (bell peppers, Capsicum annuum and sweet potatoes, Ipomoea batatas). A total of 66 adults participated in the study. The participants’ task was to haptically explore a single object on any particular trial and subsequently indicate which of 12 simultaneously visible objects possessed the same shape. The participants’ performance for the natural objects was 60.9 and 78.7 percent correct for the bell peppers and sweet potatoes, respectively. The analogous performance for the SMS objects, while better than chance, was far worse (18.6 percent correct). All of these types of stimulus objects possess a rich geometrical structure (e.g., they all possess multiple elliptic, hyperbolic, and parabolic surface regions). Nevertheless, these three types of stimulus objects are perceived differently: Individual members of sweet potatoes and bell peppers are largely identifiable to human participants, while the individual SMS objects are not. Analyses of differential geometry indicate that these natural objects (e.g., bell peppers and sweet potatoes) possess heterogeneous spatial configurations of distinctly curved surface regions, and this heterogeneity is lacking in SMS objects. The current results therefore suggest that increases in surface structure heterogeneity facilitate human object recognition.

Active Visual-Tactile Cross-Modal Matching

Tactile and visual modalities frequently occur in cognitive robotics. Their matching problem is of highly interesting in many practical scenarios since it provides different properties about objects. In this paper, we investigate the active visual-tactile cross-modal matching problem which is formulated as retrieving the relevant sample in unlabeled gallery visual dataset in response to the tactile query sample. Such a problem exhibits a nontrivial challenge that there does not exist sample-to-sample pairing relation between tactile and visual modalities. To this end, we design a shared dictionary learning model which can simultaneously learn the projection subspace and the latent shared dictionary for the visual and tactile measurements. In addition, an optimization algorithm is developed to effectively solve the shared dictionary learning problem. Based on the obtained solution, the visual-tactile cross-modal matching algorithm can be easily developed. Finally, we perform experimental validations on the PHAC-2 datasets to show the effectiveness of the proposed visual-tactile cross-modal matching framework and method.

Using Eye Tracking and Heart-Rate Activity to Examine Crossmodal Correspondences QoE in Mulsemedia

Different senses provide us with information of various levels of precision and enable us to construct a more precise representation of the world. Rich multisensory simulations are thus beneficial for comprehension, memory reinforcement, or retention of information. Crossmodal mappings refer to the systematic associations often made between different sensory modalities (e.g., high pitch is matched with angular shapes) and govern multisensory processing. A great deal of research effort has been put into exploring cross-modal correspondences in the field of cognitive science. However, the possibilities they open in the digital world have been relatively unexplored. Multiple sensorial media (mulsemedia) provides a highly immersive experience to the users and enhances their Quality of Experience (QoE) in the digital world. Thus, we consider that studying the plasticity and the effects of cross-modal correspondences in a mulsemedia setup can bring interesting insights about improving the human computer dialogue and experience. In our experiments, we exposed users to videos with certain visual dimensions (brightness, color, and shape), and we investigated whether the pairing with a cross-modal matching sound (high and low pitch) and the corresponding auto-generated vibrotactile effects (produced by a haptic vest) lead to an enhanced QoE. For this, we captured the eye gaze and the heart rate of users while experiencing mulsemedia, and we asked them to fill in a set of questions targeting their enjoyment and perception at the end of the experiment. Results showed differences in eye-gaze patterns and heart rate between the experimental and the control group, indicating changes in participants’ engagement when videos were accompanied by matching cross-modal sounds (this effect was the strongest for the video displaying angular shapes and high-pitch audio) and transitively generated cross-modal vibrotactile effects.&lt;?vsp -1pt?&gt;

The role of sex-related voice variation in children's gender-role stereotype attributions.

In the absence of clear sex differences in vocal anatomy, the expression of gender in pre-pubertal children's voices has a strong behavioural dimension. However, whether children are sensitive to this gender-related variation in the voice and use it to make inferences about their peers' masculinity and femininity remains unexplored. Using a cross-modal matching task, thirty-one 7- to 8-year-olds and forty-two adults were asked to associate prototypical voices of boys and girls, and their re-synthesized masculinized and feminized versions, to fictional stereotypically masculine, gender-neutral, and stereotypically feminine child characters. We found that listeners spontaneously associated stereotypically masculine and feminine descriptors of a child character with masculinized voices and feminized voices, respectively. Adults made overall more stereotypical associations and were less influenced by character sex than children. Our observations highlight for the first time the contribution of acoustic cues to gender stereotyping from childhood, and its potential implications for the gender schema literature. Statement of contribution What is already known on this subject? Research on stereotyping shows children's schematic processing of the visible aspects of gender expression Psychoacoustic research shows that variation in children's voices affects adults' judgments of their masculinity What does this study add? Children and adults linked voice variation to gender-stereotypical characterizations of child characters Adults made overall more stereotypical associations than children and were less influenced by character's sex Our results highlight the existence of a vocal component in children's and adults' gender schemas.

Bifocal high-definition tACS over early sensory regions modulates crossmodal matching: Combined evidence from EEG and tACS/behavioral studies

Cross-talk between distant brain regions is an integral part of complex brain dynamics. In order to study such interactions, bifocal high-definition tACS (hd-tACS) represents a promising tool to modulate oscillatory synchrony between distant brain regions in a phase- and frequency-specific manner. Here, we investigated cortico-cortical synchronization in alpha and gamma bands underlying multisensory integration. First, we employed a paradigm on crossmodal matching in an EEG study to show that stimulus-driven benefits of crossmodal congruence are paralleled by decreases in gamma band coherence between early sensory cortices. Second, we used bifocal hd-tACS to target visual and somatosensory regions in a similar crossmodal matching paradigm. In- or anti-phase stimulation between the two regions (0°/180° phase shift) was applied with either 10 or 40 Hz and was controlled by sham. As an additional control, electrical stimulation was applied ipsilateral or contralateral to the presentation of visual-tactile stimuli. As hypothesized based on our EEG data, we found contralateral gamma stimulation to decrease behavioral benefits of crossmodal congruence compared with sham. To our surprise, both in- and anti-phase stimulation showed such an effect. We conclude that 40 Hz stimulation might have synchronized gamma band activity across modalities irrespective of stimulation phase. Ipsilateral stimulation showed no interaction with stimulus congruence. Yet, overall performance was increased by ipsilateral alpha and decreased by ipsilateral gamma stimulation. We suggest that, in the absence of sensory input, tACS modulated general cortical excitation/inhibition and related hemispheric competition. Taken together, we present evidence for the involvement of gamma synchronization in crossmodal matching. Our results further suggest that conventional approaches to decrease functional coupling by anti-phase tACS might not always be effective. Our study design – incorporating both EEG and behavioral studies – also demonstrates how acute effects of tACS can be studied bypassing the unsolved problem of tACS-related artifacts in electrophysiological recordings.