Cross-modal Relationship Research Articles

Relationship-Embedded Representation Learning for Grounding Referring Expressions.

Grounding referring expressions in images aims to locate the object instance in an image described by a referring expression. It involves a joint understanding of natural language and image content, and is essential for a range of visual tasks related to human-computer interaction. As a language-to-vision matching task, the core of this problem is to not only extract all the necessary information (i.e., objects and the relationships among them) in both the image and referring expression, but also make full use of context information to align cross-modal semantic concepts in the extracted information. Unfortunately, existing work on grounding referring expressions fails to accurately extract multi-order relationships from the referring expression and associate them with the objects and their related contexts in the image. In this paper, we propose a cross-modal relationship extractor (CMRE) to adaptively highlight objects and relationships (spatial and semantic relations) related to the given expression with a cross-modal attention mechanism, and represent the extracted information as a language-guided visual relation graph. In addition, we propose a Gated Graph Convolutional Network (GGCN) to compute multimodal semantic contexts by fusing information from different modes and propagating multimodal information in the structured relation graph. Experimental results on three common benchmark datasets show that our Cross-Modal Relationship Inference Network, which consists of CMRE and GGCN, significantly surpasses all existing state-of-the-art methods.

Multiple Holdouts With Stability: Improving the Generalizability of Machine Learning Analyses of Brain–Behavior Relationships

BackgroundIn 2009, the National Institute of Mental Health launched the Research Domain Criteria, an attempt to move beyond diagnostic categories and ground psychiatry within neurobiological constructs that combine different levels of measures (e.g., brain imaging and behavior). Statistical methods that can integrate such multimodal data, however, are often vulnerable to overfitting, poor generalization, and difficulties in interpreting the results. MethodsWe propose an innovative machine learning framework combining multiple holdouts and a stability criterion with regularized multivariate techniques, such as sparse partial least squares and kernel canonical correlation analysis, for identifying hidden dimensions of cross-modality relationships. To illustrate the approach, we investigated structural brain–behavior associations in an extensively phenotyped developmental sample of 345 participants (312 healthy and 33 with clinical depression). The brain data consisted of whole-brain voxel-based gray matter volumes, and the behavioral data included item-level self-report questionnaires and IQ and demographic measures. ResultsBoth sparse partial least squares and kernel canonical correlation analysis captured two hidden dimensions of brain–behavior relationships: one related to age and drinking and the other one related to depression. The applied machine learning framework indicates that these results are stable and generalize well to new data. Indeed, the identified brain–behavior associations are in agreement with previous findings in the literature concerning age, alcohol use, and depression-related changes in brain volume. ConclusionsMultivariate techniques (such as sparse partial least squares and kernel canonical correlation analysis) embedded in our novel framework are promising tools to link behavior and/or symptoms to neurobiology and thus have great potential to contribute to a biologically grounded definition of psychiatric disorders.

Characteristic Sounds Facilitate Object Search in Real-Life Scenes.

Real-world events do not only provide temporally and spatially correlated information across the senses, but also semantic correspondences about object identity. Prior research has shown that object sounds can enhance detection, identification, and search performance of semantically consistent visual targets. However, these effects are always demonstrated in simple and stereotyped displays that lack ecological validity. In order to address identity-based cross-modal relationships in real-world scenarios, we designed a visual search task using complex, dynamic scenes. Participants searched for objects in video clips recorded from real-life scenes. Auditory cues, embedded in the background sounds, could be target-consistent, distracter-consistent, neutral, or just absent. We found that, in these naturalistic scenes, characteristic sounds improve visual search for task-relevant objects but fail to increase the salience of irrelevant distracters. Our findings generalize previous results on object-based cross-modal interactions with simple stimuli and shed light upon how audio–visual semantically congruent relationships play out in real-life contexts.

Online Fast Adaptive Low-Rank Similarity Learning for Cross-Modal Retrieval

The semantic similarity among cross-modal data objects, e.g., similarities between images and texts, are recognized as the bottleneck of cross-modal retrieval. However, existing batch-style correlation learning methods suffer from prohibitive time complexity and extra memory consumption in handling large-scale high dimensional cross-modal data. In this paper, we propose a Cross-Modal Online Low-Rank Similarity function learning (CMOLRS) method, which learns a low-rank bilinear similarity measurement for cross-modal retrieval. We model the cross-modal relations by relative similarities on the training data triplets and formulate the relative relations as convex hinge loss. By adapting the margin in hinge loss with pair-wise distances in feature space and label space, CMOLRS effectively captures the multi-level semantic correlation and adapts to the content divergence among cross-modal data. Imposed with a low-rank constraint, the similarity function is trained by online learning in the manifold of low-rank matrices. The low-rank constraint not only endows the model learning process with faster speed and better scalability, but also improves the model generality. We further propose fast-CMOLRS combining multiple triplets for each query instead of standard process using single triplet at each model update step, which further reduces the times of gradient updates and retractions. Extensive experiments are conducted on four public datasets, and comparisons with state-of-the-art methods show the effectiveness and efficiency of our approach.

Dual-supervised attention network for deep cross-modal hashing

Cross-modal hashing has received intensive attention due to its low computation and storage efficiency in cross-modal retrieval task. Most previous cross-modal hashing methods mainly focus on extracting correlated binary codes from the pairwise label, but largely ignore the semantic categories of cross-modal data. On the other hand, human perception exploits category information to connect cross-modal samples. Inspired by this fact, we propose to embed category information into hash codes. More specifically, we introduce semantic prediction loss into our framework to enhance hash codes with category supervision. In addition, there always exists a large gap between features from different modalities (e.g. text and images), leading cross-modal hashing to link irrelevant features for retrieval task. To address this issue, this paper proposes Dual-Supervised Attention Network for Deep Hashing (DSADH) to learn the cross-modal relationship via an elaborately-designed attention mechanism. Our cross-modal network applies cross-modal attention block to efficiently encode rich and relevant features to learn compact hash codes. Extensive experiments on three challenging benchmarks demonstrate that our proposed method significantly improves the retrieval results.

A model of synesthetic metaphor interpretation based on cross-modality similarity

Synesthesia focuses on the cross-modality connections between different perceptual modalities (visual, auditory, olfactory, gustatory, and haptic). It presents the aesthetics and creativity of our language and cognitive systems. A model that automatically interprets synesthetic metaphor is proposed in this paper. The interpretation model can learn the semantic knowledge of a feature, and simulate the cross-modality semantic similarity. The semantic knowledge has three parts: feature, the perceptual modality of the feature, and its sentimental orientation. They are used for supervising feature generation and expansion. By multi-step synonyms expansion, our model explores the cross-modality associations between the source and target domains which belong to different modalities. The appropriate feature is selected by a heuristic way to describe the latent meaning of the target domain in the current context. The experimental result shows that our model can fully consider the semantic knowledge and embody the cross-modality relations. It achieves an accuracy of 84%.

EmbraceNet: A robust deep learning architecture for multimodal classification

Classification using multimodal data arises in many machine learning applications. It is crucial not only to model cross-modal relationship effectively but also to ensure robustness against loss of part of data or modalities. In this paper, we propose a novel deep learning-based multimodal fusion architecture for classification tasks, which guarantees compatibility with any kind of learning models, deals with cross-modal information carefully, and prevents performance degradation due to partial absence of data. We employ two datasets for multimodal classification tasks, build models based on our architecture and other state-of-the-art models, and analyze their performance on various situations. The results show that our architecture outperforms the other multimodal fusion architectures when some parts of data are not available.

Tau PET in autosomal dominant Alzheimer's disease: relationship with cognition, dementia and other biomarkers.

Tauopathy is a hallmark pathology of Alzheimer's disease with a strong relationship with cognitive impairment. As such, understanding tau may be a key to clinical interventions. In vivo tauopathy has been measured using cerebrospinal fluid assays, but these do not provide information about where pathology is in the brain. The introduction of PET ligands that bind to paired helical filaments provides the ability to measure the amount and distribution of tau pathology. The heritability of the age of dementia onset tied to the specific mutations found in autosomal dominant Alzheimer's disease families provides an elegant model to study the spread of tau across the course of the disease as well as the cross-modal relationship between tau and other biomarkers. To better understand the pathobiology of Alzheimer's disease we measured levels of tau PET binding in individuals with dominantly inherited Alzheimer's disease using data from the Dominantly Inherited Alzheimer Network (DIAN). We examined cross-sectional measures of amyloid-β, tau, glucose metabolism, and grey matter degeneration in 15 cognitively normal mutation non-carriers, 20 asymptomatic carriers, and 15 symptomatic mutation carriers. Linear models examined the association of pathology with group, estimated years to symptom onset, as well as cross-modal relationships. For comparison, tau PET was acquired on 17 older adults with sporadic, late onset Alzheimer disease. Tau PET binding was starkly elevated in symptomatic DIAN individuals throughout the cortex. The brain areas demonstrating elevated tau PET binding overlapped with those seen in sporadic Alzheimer's disease, but with a greater cortical involvement and greater levels of binding despite similar cognitive impairment. Tau PET binding was elevated in the temporal lobe, but the most prominent loci of pathology were in the precuneus and lateral parietal regions. Symptomatic mutation carriers also demonstrated elevated tau PET binding in the basal ganglia, consistent with prior work with amyloid-β. The degree of tau tracer binding in symptomatic individuals was correlated to other biomarkers, particularly markers of neurodegeneration. In addition to the differences seen with tau, amyloid-β was increased in both asymptomatic and symptomatic groups relative to non-carriers. Glucose metabolism showed decline primarily in the symptomatic group. MRI indicated structural degeneration in both asymptomatic and symptomatic cohorts. We demonstrate that tau PET binding is elevated in symptomatic individuals with dominantly inherited Alzheimer's disease. Tau PET uptake was tied to the onset of cognitive dysfunction, and there was a higher amount, and different regional pattern of binding compared to late onset, non-familial Alzheimer's disease.

Cross-modal motion regeneration using Multimodal Deep Belief Network

In this paper, we propose a Multimodal Deep Belief Network (MDBN) for learning a generative model of 2D and 3D skeletal data. The MDBM learns the cross-modal relationship between these data in the form of a joint probability distribution over the space of multimodal inputs. It can regenerate any missing modality by sampling from the conditional distribution over the given data modality. The skeletal sequences are converted into the motion images which help us utilize the impressive power of the generative deep networks in the image processing. Furthermore, we use the variation of information (VI) as the training criterion, instead of the conventional maximum likelihood. It is proven that VI is efficient in cross-modal learning where some data modalities are missing. Our experimental results have shown that the model has an outstanding performance on the over-complete MHAD and CMU Mocap datasets in data-driven motion regeneration on a full-body 2D and 3D skeleton structures.

Cognitive Load Assessment from EEG and Peripheral Biosignals for the Design of Visually Impaired Mobility Aids

Reliable detection of cognitive load would benefit the design of intelligent assistive navigation aids for the visually impaired (VIP). Ten participants with various degrees of sight loss navigated in unfamiliar indoor and outdoor environments, while their electroencephalogram (EEG) and electrodermal activity (EDA) signals were being recorded. In this study, the cognitive load of the tasks was assessed in real time based on a modification of the well‐established event‐related (de)synchronization (ERD/ERS) index. We present an in‐depth analysis of the environments that mostly challenge people from certain categories of sight loss and we present an automatic classification of the perceived difficulty in each time instance, inferred from their biosignals. Given the limited size of our sample, our findings suggest that there are significant differences across the environments for the various categories of sight loss. Moreover, we exploit cross‐modal relations predicting the cognitive load in real time inferring on features extracted from the EDA. Such possibility paves the way for the design on less invasive, wearable assistive devices that take into consideration the well‐being of the VIP.

Multimodal characterization of gray matter alterations in neuromyelitis optica

Objective: To investigate structural and functional alterations of gray matter (GM) and examine their clinical relevance in neuromyelitis optica (NMO) using multimodal magnetic resonance imaging (MRI) techniques. Methods: A total of 35 NMO and 36 healthy controls (HC) were recruited in this study. Cortical lesions were investigated by double inversion recovery technique. Five voxel-wise MRI measurements were obtained for each participant in the GM including gray matter volume (GMV), fractional anisotropy (FA), mean diffusivity (MD), amplitude of low-frequency fluctuation (ALFF), and weighted functional connectivity strength (wFCS). Between-group differences, cross-modality relationships, and MRI-clinical correlations were examined. Results: No cortical lesions were found in NMO. Compared to HC, NMO patients exhibited significantly decreased GMV in deep GM and cortical regions involving visual function and cognition. Diffusion GM abnormalities were widespread in the patients. Decreased ALFF and wFCS were observed in the patients in sensorimotor, visual, cognition, and cerebellar sites. GM structural alterations were correlated with cognitive but not physical disability scores of the patients. Conclusion: Despite the lack of focal cortical lesions, patients with NMO exhibit both structural and functional alterations of GM in cerebrum and cerebellum that predominantly involve deep GM, visual, motor, and cognitive regions. GM alterations are associated with cognitive impairment but not physical disability.

Takete and Maluma in Action: A Cross-Modal Relationship between Gestures and Sounds.

Despite Saussure’s famous observation that sound-meaning relationships are in principle arbitrary, we now have a substantial body of evidence that sounds themselves can have meanings, patterns often referred to as “sound symbolism”. Previous studies have found that particular sounds can be associated with particular meanings, and also with particular static visual shapes. Less well studied is the association between sounds and dynamic movements. Using a free elicitation method, the current experiment shows that several sound symbolic associations between sounds and dynamic movements exist: (1) front vowels are more likely to be associated with small movements than with large movements; (2) front vowels are more likely to be associated with angular movements than with round movements; (3) obstruents are more likely to be associated with angular movements than with round movements; (4) voiced obstruents are more likely to be associated with large movements than with small movements. All of these results are compatible with the results of the previous studies of sound symbolism using static images or meanings. Overall, the current study supports the hypothesis that particular dynamic motions can be associated with particular sounds. Building on the current results, we discuss a possible practical application of these sound symbolic associations in sports instructions.

Unsupervised multi-graph cross-modal hashing for large-scale multimedia retrieval

With the advance of internet and multimedia technologies, large-scale multi-modal representation techniques such as cross-modal hashing, are increasingly demanded for multimedia retrieval. In cross-modal hashing, three essential problems should be seriously considered. The first is that effective cross-modal relationship should be learned from training data with scarce label information. The second is that appropriate weights should be assigned for different modalities to reflect their importance. The last is the scalability of training process which is usually ignored by previous methods. In this paper, we propose Multi-graph Cross-modal Hashing (MGCMH) by comprehensively considering these three points. MGCMH is unsupervised method which integrates multi-graph learning and hash function learning into a joint framework, to learn unified hash space for all modalities. In MGCMH, different modalities are assigned with proper weights for the generation of multi-graph and hash codes respectively. As a result, more precise cross-modal relationship can be preserved in the hash space. Then Nystrom approximation approach is leveraged to efficiently construct the graphs. Finally an alternating learning algorithm is proposed to jointly optimize the modality weights, hash codes and functions. Experiments conducted on two real-world multi-modal datasets demonstrate the effectiveness of our method, in comparison with several representative cross-modal hashing methods.

Recognition of faces and names: multimodal physiological correlates of memory and executive function.

We sought to characterize electrophysiological, eye-tracking and behavioral correlates of face-name recognition memory in healthy younger adults using high-density electroencephalography (EEG), infrared eye-tracking (ET), and neuropsychological measures. Twenty-one participants first studied 40 face-name (FN) pairs; 20 were presented four times (4R) and 20 were shown once (1R). Recognition memory was assessed by asking participants to make old/new judgments for 80 FN pairs, of which half were previously studied items and half were novel FN pairs (N). Simultaneous EEG and ET recording were collected during recognition trials. Comparisons of event-related potentials (ERPs) for correctly identified FN pairs were compared across the three item types revealing classic ERP old/new effects including 1) relative positivity (1R>N) bi-frontally from 300 to 500ms, reflecting enhanced familiarity, 2) relative positivity (4R>1R and 4R>N) in parietal areas from 500 to 800ms, reflecting enhanced recollection, and 3) late frontal effects (1R>N) from 1000 to 1800ms in right frontal areas, reflecting post-retrieval monitoring. ET analysis also revealed significant differences in eye movements across conditions. Exploration of cross-modality relationships suggested associations between memory and executive function measures and the three ERP effects. Executive function measures were associated with several indicators of saccadic eye movements and fixations, which were also associated with all three ERP effects. This novel characterization of face-name recognition memory performance using simultaneous EEG and ET reproduced classic ERP and ET effects, supports the construct validity of the multimodal FN paradigm, and holds promise as an integrative tool to probe brain networks supporting memory and executive functioning.

Emergence of auditory-visual relations from a visual-visual baseline with auditory-specific consequences in individuals with autism.

Empirical studies have demonstrated that class-specific contingencies may engender stimulus-reinforcer relations. In these studies, crossmodal relations emerged when crossmodal relations comprised the baseline, and intramodal relations emerged when intramodal relations were taught during baseline. This study investigated whether auditory-visual relations (crossmodal) would emerge after participants learned a visual-visual baseline (intramodal) with auditory stimuli presented as specific consequences. Four individuals with autism learned AB and CD relations with class-specific reinforcers. When A1 and C1 were presented as samples, the selections of B1 and D1, respectively, were followed by an edible (R1) and a sound (S1). Selections of B2 and D2 under the control of A2 and C2, respectively, were followed by R2 and S2. Probe trials tested for visual-visual AC, CA, AD, DA, BC, CB, BD, and DB emergent relations and auditory-visual SA, SB, SC, and SD emergent relations. All of the participants demonstrated the emergence of all auditory-visual relations, and three of four participants showed emergence of all visual-visual relations. Thus, the emergence of auditory-visual relations from specific auditory consequences suggests that these relations do not depend on crossmodal baseline training. The procedure has great potential for applied technology to generate auditory-visual discriminations and stimulus classes in the context of behavior-analytic interventions for autism.

Perception and pain thresholds for cutaneous heat and cold, and rectal distension: associations and disassociations

Hypersensitivity to somatic or visceral pain has been reported in numerous clinical conditions such as fibromyalgia or the irritable bowel syndrome, and general hypersensitivity has been proposed to be the underlying mechanism. However, cross-modal relationships especially between somatic and visceral pain have rarely been investigated even in healthy volunteers. Furthermore, psychological influences on pain have rarely been characterized across modalities. Sixty-one healthy participants underwent testing of perception and pain thresholds for cutaneous thermode heat and cold, as well as for rectal balloon distension. Psychological testing for anxiety, depression, and pain experience (including catastrophizing and coping) as well as cardiac interoception was performed. Measurement quality and the correlations between the different modalities were examined. Significant correlations existed between the perception thresholds for cold/heat (τB =-0.28, p=0.002) and cold/distension (τB = -0.21, p=0.03) and for the pain thresholds for cold/heat (r=-0.61, p<0.001) and heat/distension (r=0.33, p=0.01). No association was found between pain thresholds and anxiety, depression, psychological experience with and processing of pain, or cardiac interoception. Retest reliabilities for pain measurements were satisfying after short intertrial intervals (all intraclass correlation coefficients >0.8), but less so after longer intervals. The individuals contributing to the respective correlations differ between measurements. Moderate associations were found for pain thresholds across modalities. The strength of the associations and their stability over time warrants further investigation and might serve to increase the understanding of conditions affecting multiple pain modalities.

Questioning the utility of the concept of amodality: Towards a revised framework for understanding crossmodal relations

The idea that sensory experiences should be distinguished into modal and amodal (in terms of crossmodal redundancy, rather than amodal completion) runs through many areas of psychology; developmental psychologists, in particular, appear to hold opposing views about whether infants start by having modally distinct or amodal experiences of the properties that are commonly considered to be amodal, such as space, duration, and intensity. Many other phenomena in adults — such as ‘crossmodal correspondences’ (Spence, 2011) — are deemed interesting, in part, precisely because they are not simply picking-up on amodal properties or redundantly-coded features. However, while many researchers have, over the years, used the term ‘amodal’, we would argue that there is a great deal of uncertainty over which dimensions of experience should be considered as amodal, and according to which criteria. Here, we argue that virtually nothing is amodal. To the extent that this claim holds, it may help to shed further light on early multisensory development while at the same time calling into question various theoretical accounts about both the acquisition of crossmodal relations in early development and the acquisition of novel associations (or correspondences) in adulthood. This discussion may also help to elucidate the kinds of evidence that can legitimately be taken to support arguments concerning the existence, or otherwise, of neonatal synaesthesia (see Deroy and Spence, submitted).

Prosodic temporal alignment of co-speech gestures to speech facilitates referent resolution.

Using a referent detection paradigm, we examined whether listeners can determine the object speakers are referring to by using the temporal alignment between the motion speakers impose on objects and their labeling utterances. Stimuli were created by videotaping speakers labeling a novel creature. Without being explicitly instructed to do so, speakers moved the creature during labeling. Trajectories of these motions were used to animate photographs of the creature. Participants in subsequent perception studies heard these labeling utterances while seeing side-by-side animations of two identical creatures in which only the target creature moved as originally intended by the speaker. Using the cross-modal temporal relationship between speech and referent motion, participants identified which creature the speaker was labeling, even when the labeling utterances were low-pass filtered to remove their semantic content or replaced by tone analogues. However, when the prosodic structure was eliminated by reversing the speech signal, participants no longer detected the referent as readily. These results provide strong support for a prosodic cross-modal alignment hypothesis. Speakers produce a perceptible link between the motion they impose upon a referent and the prosodic structure of their speech, and listeners readily use this prosodic cross-modal relationship to resolve referential ambiguity in word-learning situations.

Image Retrieval Approach Based on Sparse Canonical Correlation Analysis

图像语义检索的一个关键问题就是要找到图像底层特征与语义之间的关联,由于文本是表达语义的一种有效手段,因此提出通过研究文本与图像两种模态之间关系来构建反映两者间潜在语义关联的有效模型的思路.基于该模型,可使用自然语言形式(文本语句)来表达检索意图,最终检索到相关图像.该模型基于稀疏典型性相关分析(sparse canonical correlation analysis,简称sparse CCA),按照如下步骤训练得到:首先利用隐语义分析方法构造文本语义空间,然后以视觉词袋(bag of visual words)来表达文本所对应的图像,最后通过Sparse CCA 算法找到一个语义相关空间,以实现文本语义与图像视觉单词间的映射.使用稀疏的相关性分析方法可以提高模型可解释性和保证检索结果稳定性.实验结果验证了Sparse CCA 方法的有效性,同时也证实了所提出的图像语义检索方法的可行性.;A key issue of semantic-based image retrieval is how to bridge the semantic gap between the low-level feature of image and high-level semantics, which can be expressed by means of free text effectively. The cross-modal relationship between the text and image is studied by a modeling semantic correlation between text and image. Based on the model, an approach to image retrieval is proposed so that images are retrieved according to meaning of the query text rather than query keywords. First, an algorithm for solving sparse canonical correlation analysis (CCA) is designed in this paper. Then a semantic space is learned by way of latent semantic analysis from text corpus, and images are represented by bag of visual words. After that, a semantic correlation space, by which the map between visual words of image and the high-level semantics is made explicit, can be constructed. The proposed method solves CCA in a sparse framework in order to make the result more interpretable and stable. The experimental result demonstrates that Sparse CCA outperform CCA in the context, and also substantiates the feasibility of the proposed approach to image retrieval.

Kernel Cross-Modal Factor Analysis for Information Fusion With Application to Bimodal Emotion Recognition

In this paper, we investigate kernel based methods for multimodal information analysis and fusion. We introduce a novel approach, kernel cross-modal factor analysis, which identifies the optimal transformations that are capable of representing the coupled patterns between two different subsets of features by minimizing the Frobenius norm in the transformed domain. The kernel trick is utilized for modeling the nonlinear relationship between two multidimensional variables. We examine and compare with kernel canonical correlation analysis which finds projection directions that maximize the correlation between two modalities, and kernel matrix fusion which integrates the kernel matrices of respective modalities through algebraic operations. The performance of the introduced method is evaluated on an audiovisual based bimodal emotion recognition problem. We first perform feature extraction from the audio and visual channels respectively. The presented approaches are then utilized to analyze the cross-modal relationship between audio and visual features. A hidden Markov model is subsequently applied for characterizing the statistical dependence across successive time segments, and identifying the inherent temporal structure of the features in the transformed domain. The effectiveness of the proposed solution is demonstrated through extensive experimentation.