Data Of Different Modalities Research Articles

Multi-Pathway Generative Adversarial Hashing for Unsupervised Cross-Modal Retrieval

Cross-modal hashing aims to map heterogeneous cross-modal data into a common Hamming space, which can realize fast and flexible retrieval across different modalities. Unsupervised cross-modal hashing is more flexible and applicable than supervised methods, since no intensive labeling work is involved. However, existing unsupervised methods learn the hashing functions by preserving inter- and intra-correlations while ignoring the underlying manifold structure across different modalities, which is extremely helpful in capturing the meaningful nearest neighbors of different modalities for cross-modal retrieval. Furthermore, existing works mainly focus on pairwise relation modeling while ignoring the correlations within multiple modalities. To address the above-mentioned problems, in this paper, we propose a multi-pathwaygenerativeadversarialhashing approach for unsupervised cross-modal retrieval, which makes full use of a generative adversarial network's ability for unsupervised representation learning to exploit the underlying manifold structure of cross-modal data. The main contributions can be summarized as follows: First, we propose a multi-pathwaygenerativeadversarialnetwork to model cross-modal hashing in an unsupervised fashion. In the proposed network, given the data of one modality, the generative model tries to fit the distribution over the manifold structure and selects informative data of other modalities to challenge the discriminative model. The discriminative model learns to distinguish the generated data and the true positive data sampled from the correlation graph to achieve better retrieval accuracy. These two models are trained in an adversarial way to improve each other and promote hashing function learning. Second, we propose a correlation graph-based approach to capture the underlying manifold structure across different modalities so that data of different modalities but within the same manifold can have a smaller Hamming distance to promote retrieval accuracy. Extensive experiments compared with state-of-the-art methods on three widely used datasets verify the effectiveness of our proposed approach.

CM-GANs

It is known that the inconsistent distributions and representations of different modalities, such as image and text, cause the heterogeneity gap, which makes it very challenging to correlate heterogeneous data and measure their similarities. Recently, generative adversarial networks (GANs) have been proposed and have shown their strong ability to model data distribution and learn discriminative representation. It has also been shown that adversarial learning can be fully exploited to learn discriminative common representations for bridging the heterogeneity gap. Inspired by this, we aim to effectively correlate large-scale heterogeneous data of different modalities with the power of GANs to model cross-modal joint distribution. In this article, we propose Cross-modal Generative Adversarial Networks (CM-GANs) with the following contributions. First, a cross-modal GAN architecture is proposed to model joint distribution over the data of different modalities. The inter-modality and intra-modality correlation can be explored simultaneously in generative and discriminative models. Both compete with each other to promote cross-modal correlation learning. Second, the cross-modal convolutional autoencoders with weight-sharing constraint are proposed to form the generative model. They not only exploit the cross-modal correlation for learning the common representations but also preserve reconstruction information for capturing the semantic consistency within each modality. Third, a cross-modal adversarial training mechanism is proposed, which uses two kinds of discriminative models to simultaneously conduct intra-modality and inter-modality discrimination. They can mutually boost to make the generated common representations more discriminative by the adversarial training process. In summary, our proposed CM-GAN approach can use GANs to perform cross-modal common representation learning by which the heterogeneous data can be effectively correlated. Extensive experiments are conducted to verify the performance of CM-GANs on cross-modal retrieval compared with 13 state-of-the-art methods on 4 cross-modal datasets.

Cross-media retrieval based on linear discriminant analysis

Existing cross-media retrieval approaches usually project low-level features from different modalities of data into a common subspace, in which the similarity of multi-modal data can be measured directly. However, most of the previous subspace learning methods ignore the discriminative property of multi-modal data which may lead to suboptimal cross-media retrieval performance. To address this problem, we propose a novel approach to cross-media retrieval framework based on Linear Discriminant Analysis (LDA), which integrates the correlation between textual features and visual features to learn a pair of projection matrices so that we can project the low-level heterogeneous features into a shared feature space by the transformation matrices. Thus the discriminative characteristic of textual modality is transferred to the corresponding visual features via the correlation analysis process. Experiments on three benchmark datasets show the effectiveness of our approach.

Trading-Off Accuracy and Energy of Deep Inference on Embedded Systems: A Co-Design Approach

Deep neural networks have seen tremendous success for different modalities of data including images, videos, and speech. This success has led to their deployment in mobile and embedded systems for real-time applications. However, making repeated inferences using deep networks on embedded systems poses significant challenges due to constrained resources (e.g., energy and computing power). To address these challenges, we develop a principled co-design approach. Building on prior work, we develop a formalism referred to as Coarse-to-Fine Networks (C2F Nets) that allow us to employ classifiers of varying complexity to make predictions. We propose a principled optimization algorithm to automatically configure C2F Nets for a specified trade-off between accuracy and energy consumption for inference. The key idea is to select a classifier on-the-fly whose complexity is proportional to the hardness of the input example: simple classifiers for easy inputs and complex classifiers for hard inputs. We perform comprehensive experimental evaluation using four different C2F Net architectures on multiple real-world image classification tasks. Our results show that optimized C2F Net can reduce the Energy Delay Product (EDP) by 27 to 60 percent with no loss in accuracy when compared to the baseline solution, where all predictions are made using the most complex classifier in C2F Net.

Ensembles of change-point detectors: implications for real-time BMI applications.

Brain-machine interfaces (BMIs) have been widely used to study basic and translational neuroscience questions. In real-time closed-loop neuroscience experiments, many practical issues arise, such as trial-by-trial variability, and spike sorting noise or multi-unit activity. In this paper, we propose a new framework for change-point detection based on ensembles of independent detectors in the context of BMI application for detecting acute pain signals. Motivated from ensemble learning, our proposed "ensembles of change-point detectors" (ECPDs) integrate multiple decisions from independent detectors, which may be derived based on data recorded from different trials, data recorded from different brain regions, data of different modalities, or models derived from different learning methods. By integrating multiple sources of information, the ECPDs aim to improve detection accuracy (in terms of true positive and true negative rates) and achieve an optimal trade-off of sensitivity and specificity. We validate our method using computer simulations and experimental recordings from freely behaving rats. Our results have shown superior and robust performance of ECPDS in detecting the onset of acute pain signals based on neuronal population spike activity (or combined with local field potentials) recorded from single or multiple brain regions.

Using deep neural networks along with dimensionality reduction techniques to assist the diagnosis of neurodegenerative disorders.

The analysis of neuroimaging data is frequently used to assist the diagnosis of neurodegenerative disorders such as Alzheimer's disease (AD) or Parkinson's disease (PD) and has become a routine procedure in the clinical practice. During the past decade, the pattern recognition community has proposed a number of machine learning-based systems that automatically analyse neuroimaging data in order to improve the diagnosis. However, the high dimensionality of the data is still a challenge and there is room for improvement. The development of novel classification frameworks as TensorFlow, recently released as open source by Google Inc., represents an opportunity to continue evolving these systems. In this work, we demonstrate several computer-aided diagnosis (CAD) systems based on Deep Neural Networks that improve the diagnosis for AD and PD and outperform those based on classical classifiers. In order to address the small sample size problem we evaluate two dimensionality reduction algorithms based on Principal Component Analysis and Non-Negative Matrix Factorization (NNMF), respectively. The performance of developed CAD systems is assessed using 4 datasets with neuroimaging data of different modalities.

Dementias Platform UK (DPUK) Data Portal - World-leading infrastructure facilitating innovative multi-modal research

IntroductionModern team science requires effective sharing of data and skills. The DPUK Data Portal is a collection of tools, datasets and networks that allows for epidemiologists and specialist researchers alike to access, analyse and investigate cohort and different modalities of routine data across UK and international sources.
 Objectives and ApproachThe Portal is housed on an instance of UKSeRP (UK Secure eResearch Platform), that allows customisable infrastructure to be used for multi-modal research (thus far live in genetics, imaging and clinical data) for researchers across the world using remote access technology whilst allowing governance to remain with the data provider. A central team at Swansea University is responsible for data curation and processing, and runs an access procedure for researchers to apply to use data from multiple sources to be analysed in a central analysis environment. Other modalities are similarly hosted, with input from partner sites in Cardiff and Oxford.
 ResultsDPUK facilitates data access and research on 49 cohorts, 40 UK-based and 9 international. The centralised repository model including remote access and ability to store and make available different modalities of data, from phenotypic data, to genetic and imaging data, has allowed DPUK to begin to support research of varying topics, from those studying cognitive decline and Dementia as a disease, to those maturing analytical models. By providing access to data platforms specialising in genetics, imaging and routine clinical data, as well as to specialists in disease and biology to aid with its understanding, DPUK has realised a large-scale research exercise combining major data modalities on a central platform, and allow access to such rich data across the world under an umbrella of robust governance.
 Conclusion/ImplicationsGlobally, cohorts are pooling data, expertise and desire to enrich their own aims in partnership with a federated research community to enable in-depth scrutiny of the biological origins of dementia and the development and evaluation of novel approach to disease prevention and cure.

Modality-specific Cross-modal Similarity Measurement with Recurrent Attention Network.

Nowadays, cross-modal retrieval plays an important role to flexibly find useful information across different modalities of data. Effectively measuring the similarity between different modalities of data is the key of cross-modal retrieval. Different modalities such as image and text have imbalanced and complementary relationship, and they contain unequal amount of information when describing the same semantics. For example, images often contain more details that cannot be demonstrated by textual descriptions and vice versa. Existing works based on Deep Neural Network (DNN) mostly construct one common space for different modalities, to find the latent alignments between them, which lose their exclusive modality-specific characteristics. Therefore, we propose modality-specific cross-modal similarity measurement (MCSM) approach by constructing the independent semantic space for each modality, which adopts an endto- end framework to directly generate modality-specific crossmodal similarity without explicit common representation. For each semantic space, modality-specific characteristics within one modality are fully exploited by recurrent attention network, while the data of another modality is projected into this space with attention based joint embedding, which utilizes the learned attention weights for guiding the fine-grained cross-modal correlation learning, and captures the imbalanced and complementary relationship between different modalities. Finally, the complementarity between the semantic spaces for different modalities is explored by adaptive fusion of the modality-specific cross-modal similarities to perform cross-modal retrieval. Experiments on the widely-used Wikipedia, Pascal Sentence, MS-COCO datasets as well as our constructed large-scale XMediaNet dataset verify the effectiveness of our proposed approach, outperforming 9 stateof- the-art methods.

#Healthy Selfies: Exploration of Health Topics on Instagram.

BackgroundSocial media provides a complementary source of information for public health surveillance. The dominate data source for this type of monitoring is the microblogging platform Twitter, which is convenient due to the free availability of public data. Less is known about the utility of other social media platforms, despite their popularity.ObjectiveThis work aims to characterize the health topics that are prominently discussed in the image-sharing platform Instagram, as a step toward understanding how this data might be used for public health research.MethodsThe study uses a topic modeling approach to discover topics in a dataset of 96,426 Instagram posts containing hashtags related to health. We use a polylingual topic model, initially developed for datasets in different natural languages, to model different modalities of data: hashtags, caption words, and image tags automatically extracted using a computer vision tool.ResultsWe identified 47 health-related topics in the data (kappa=.77), covering ten broad categories: acute illness, alternative medicine, chronic illness and pain, diet, exercise, health care & medicine, mental health, musculoskeletal health and dermatology, sleep, and substance use. The most prevalent topics were related to diet (8,293/96,426; 8.6% of posts) and exercise (7,328/96,426; 7.6% of posts).ConclusionsA large and diverse set of health topics are discussed in Instagram. The extracted image tags were generally too coarse and noisy to be used for identifying posts but were in some cases accurate for identifying images relevant to studying diet and substance use. Instagram shows potential as a source of public health information, though limitations in data collection and metadata availability may limit its use in comparison to platforms like Twitter.

Multimodal Poisson Gamma Belief Network

To learn a deep generative model of multimodal data, we propose a multimodal Poisson gamma belief network (mPGBN) that tightly couple the data of different modalities at multiple hidden layers. The mPGBN unsupervisedly extracts a nonnegative latent representation using an upward-downward Gibbs sampler. It imposes sparse connections between different layers, making it simple to visualize the generative process and the relationships between the latent features of different modalities. Our experimental results on bi-modal data consisting of images and tags show that the mPGBN can easily impute a missing modality and hence is useful for both image annotation and retrieval. We further demonstrate that the mPGBN achieves state-of-the-art results on unsupervisedly extracting latent features from multimodal data.

Unsupervised Generative Adversarial Cross-Modal Hashing

Cross-modal hashing aims to map heterogeneous multimedia data into a common Hamming space, which can realize fast and flexible retrieval across different modalities. Unsupervised cross-modal hashing is more flexible and applicable than supervised methods, since no intensive labeling work is involved. However, existing unsupervised methods learn hashing functions by preserving inter and intra correlations, while ignoring the underlying manifold structure across different modalities, which is extremely helpful to capture meaningful nearest neighbors of different modalities for cross-modal retrieval. To address the above problem, in this paper we propose an Unsupervised Generative Adversarial Cross-modal Hashing approach (UGACH), which makes full use of GAN's ability for unsupervised representation learning to exploit the underlying manifold structure of cross-modal data. The main contributions can be summarized as follows: (1) We propose a generative adversarial network to model cross-modal hashing in an unsupervised fashion. In the proposed UGACH, given a data of one modality, the generative model tries to fit the distribution over the manifold structure, and select informative data of another modality to challenge the discriminative model. The discriminative model learns to distinguish the generated data and the true positive data sampled from correlation graph to achieve better retrieval accuracy. These two models are trained in an adversarial way to improve each other and promote hashing function learning. (2) We propose a correlation graph based approach to capture the underlying manifold structure across different modalities, so that data of different modalities but within the same manifold can have smaller Hamming distance and promote retrieval accuracy. Extensive experiments compared with 6 state-of-the-art methods on 2 widely-used datasets verify the effectiveness of our proposed approach.

Discriminative correlation hashing for supervised cross-modal retrieval

Due to their storage and calculational efficiency, hashing techniques have been used for cross-modal retrieval on large-scale multi-modal data. Cross-modal hashing methods retrieve relevant items of one modality for the query of the other modality by mapping heterogeneous data of different modalities into a common Hamming space, where the binary codes are generated. However, the existing cross-modal hashing methods pay little attention to the discriminative property of the binary codes. In this paper, we propose a novel supervised cross-modal hashing method, named Discriminative Correlation Hashing (DCH), which integrates discriminative property into the hashing learning procedure. DCH introduces the Linear Discriminant Analysis (LDA) to preserve the discriminative property of textual modality and transfers it to the corresponding image modality by the learned unified binary code, thus making data in the common Hamming space much more discriminative. Extensive experimental results demonstrate that DCH outperforms state-of-the-art cross-modal hashing methods.

Cross-model convolutional neural network for multiple modality data representation

A novel data representation method of convolutional neural net- work (CNN) is proposed in this paper to represent data of different modalities. We learn a CNN model for the data of each modality to map the data of differ- ent modalities to a common space, and regularize the new representations in the common space by a cross-model relevance matrix. We further impose that the class label of data points can also be predicted from the CNN representa- tions in the common space. The learning problem is modeled as a minimiza- tion problem, which is solved by an augmented Lagrange method (ALM) with updating rules of Alternating direction method of multipliers (ADMM). The experiments over benchmark of sequence data of multiple modalities show its advantage.

Encoding physiological signals as images for affective state recognition using convolutional neural networks.

Affective state recognition based on multiple modalities of physiological signals has been a hot research topic. Traditional methods require designing hand-crafted features based on domain knowledge, which is time-consuming and has not achieved a satisfactory performance. On the other hand, conducting classification on raw signals directly can also cause some problems, such as the interference of noise and the curse of dimensionality. To address these problems, we propose a novel approach that encodes different modalities of data as images and use convolutional neural networks (CNN) to perform the affective state recognition task. We validate our aproach on the DECAF dataset in comparison with two state-of-the-art methods, i.e., the Support Vector Machines (SVM) and Random Forest (RF). Experimental results show that our aproach outperforms the baselines by 5% to 9%.

Joint Feature Selection and Subspace Learning for Cross-Modal Retrieval.

Cross-modal retrieval has recently drawn much attention due to the widespread existence of multimodal data. It takes one type of data as the query to retrieve relevant data objects of another type, and generally involves two basic problems: the measure of relevance and coupled feature selection. Most previous methods just focus on solving the first problem. In this paper, we aim to deal with both problems in a novel joint learning framework. To address the first problem, we learn projection matrices to map multimodal data into a common subspace, in which the similarity between different modalities of data can be measured. In the learning procedure, the l21-norm penalties are imposed on the projection matrices separately to solve the second problem, which selects relevant and discriminative features from different feature spaces simultaneously. A multimodal graph regularization term is further imposed on the projected data,which preserves the inter-modality and intra-modality similarity relationships.An iterative algorithm is presented to solve the proposed joint learning problem, along with its convergence analysis. Experimental results on cross-modal retrieval tasks demonstrate that the proposed method outperforms the state-of-the-art subspace approaches.

Canonical feature selection for joint regression and multi-class identification in Alzheimer's disease diagnosis.

Fusing information from different imaging modalities is crucial for more accurate identification of the brain state because imaging data of different modalities can provide complementary perspectives on the complex nature of brain disorders. However, most existing fusion methods often extract features independently from each modality, and then simply concatenate them into a long vector for classification, without appropriate consideration of the correlation among modalities. In this paper, we propose a novel method to transform the original features from different modalities to a common space, where the transformed features become comparable and easy to find their relation, by canonical correlation analysis. We then perform the sparse multi-task learning for discriminative feature selection by using the canonical features as regressors and penalizing a loss function with a canonical regularizer. In our experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, we use Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET) images to jointly predict clinical scores of Alzheimer's Disease Assessment Scale-Cognitive subscale (ADAS-Cog) and Mini-Mental State Examination (MMSE) and also identify multi-class disease status for Alzheimer's disease diagnosis. The experimental results showed that the proposed canonical feature selection method helped enhance the performance of both clinical score prediction and disease status identification, outperforming the state-of-the-art methods.

Nonnegative cross-media recoding of visual-auditory content for social media analysis

Cross-media semantics understanding, which focuses on multimedia data of different modalities, is a rising hot topic in social media analysis. One of the most challenging issues for cross-media sem...

Cross-Media Retrieval Method Based on Space Mapping

This paper puts forward a novel cross-media retrieval approach, which can process multimedia data of different modalities and measure cross-media similarity, such as image-audio similarity. Both image and audio data are selected for experiments and comparisons. Given the same visual and auditory features the new approach outperforms ICA, PCA and PLS methods both in precision and recall performance. Overall cross-media retrieval results between images and audios are very encouraging.

Fusing inherent and external knowledge with nonlinear learning for cross-media retrieval

Cross-media retrieval focuses on searching multimedia data of different modalities with content-based methods. However, most of those methods are designed for multimedia retrieval in single modality, such as image retrieval and audio retrieval. Though a few work has focused on cross-media retrieval, the performance is yet to be satisfactory and the potential of using cross-media retrieval for boosted retrieval performance remains largely unexplored. Hence, in this paper, we propose a novel cross-media retrieval approach for general multimedia data, such as image and audio. First, image and audio samples are mapped into an isomorphic feature subspace with kernel-based method; second, multimedia semantics is learned from inherent feature correlation by local linear regression; also a graph model is constructed to utilize external knowledge from relevance feedback; then we build a unified objective function integrating inherent and external learning results, and by solving the objective function we calculate a multimodal semantic space where cross-media retrieval among image and audio is enabled. Extensive experiments have validated the proposed methods with encouraging results.

Multimodal imaging utilising integrated MR-PET for human brain tumour assessment

The development of integrated magnetic resonance (MR)-positron emission tomography (PET) hybrid imaging opens up new horizons for imaging in neuro-oncology. In cerebral gliomas the definition of tumour extent may be difficult to ascertain using standard MR imaging (MRI) only. The differentiation of post-therapeutic scar tissue, tumour rests and tumour recurrence is challenging. The relationship to structures such as the pyramidal tract to the tumour mass influences the therapeutic neurosurgical approach. The diagnostic information may be enriched by sophisticated MR techniques such as diffusion tensor imaging (DTI), multiple-volume proton MR spectroscopic imaging (MRSI) and functional MRI (fMRI). Metabolic imaging with PET, especially using amino acid tracers such as (18)F-fluoroethyl-L-tyrosine (FET) or (11)C-L-methionine (MET) will indicate tumour extent and response to treatment. The new technologies comprising MR-PET hybrid systems have the advantage of providing comprehensive answers by a one-stop-job of 40-50 min. The combined approach provides data of different modalities using the same iso-centre, resulting in optimal spatial and temporal realignment. All images are acquired exactly under the same physiological conditions. We describe the imaging protocol in detail and provide patient examples for the different imaging modalities such as FET-PET, standard structural imaging (T1-weighted, T2-weighted, T1-weighted contrast agent enhanced), DTI, MRSI and fMRI. Hybrid MR-PET opens up new horizons in neuroimaging. Hybrid MR-PET allows brain tumour assessment in one stop. Hybrid MR-PET allows simultaneous acquisition of structural, functional and molecular images.