Multimodal Correlation Research Articles

Network health monitoring method based on multimodal spatiotemporal correlation fuzzy inference

By using the traditional active E2E (end-to-end) detection method to locate the congested link in IP network, we can only judge whether the network is congestion or not, and the congested cause cannot be effectively known. However, the passive detection method needs to acquire the syslog of all routers in the managed network, which requires a large amount of data. In this paper, a kind of multimodal spatiotemporal correlation fuzzy inference method for network health monitoring method was proposed through combining with the method of active E2E detection and passive detection. First, the set of network-congested links is obtained based on the Bayesian theorem through E2E path performance detection; then, the router syslog associated with the congested links will be acquired for a period of time, and the abnormal events in the syslog are detected based on the principle of mutual information (MI), so as to find the causes of network congestion. Based on the multimodal spatiotemporal correlation fuzzy paradigms combining with the active end-to-end detection and passive detection technology, the amount of data analysis will be greatly reduced and the operation speed will be improved. Simulation experiments and actual experiments verified the accuracy of the method this paper proposed.

Multimodal correlation deep belief networks for multi-view classification

The Restricted Boltzmann machine (RBM) has been proven to be a powerful tool in many specific applications, such as representational learning, document modeling, and many other learning tasks. However, the extensions of the RBM are rarely used in the field of multi-view learning. In this paper, we present a new RBM model based on canonical correlation analysis, named as the correlation RBM, for multi-view learning. The correlation RBM computes multiple representations by regularizing the marginal likelihood function with the consistency among representations from different views. In addition, the multimodal deep model can obtain a unified representation that fuses multiple representations together. Therefore, we stack the correlation RBM to create the correlation deep belief network (DBN), and then propose the multimodal correlation DBN for learning multi-view data representations. Contrasting with existing multi-view classification methods, such as multi-view Gaussian process with posterior consistency (MvGP) and consensus and complementarity based maximum entropy discrimination (MED-2C), the correlation RBM and the multimodal correlation DBN have achieved satisfactory results on two-class and multi-class classification datasets. Experimental results show that correlation RBM and the multimodal correlation DBN are effective learning algorithms.

Heterogeneous multimedia cooperative annotation based on multimodal correlation learning

Rich multimedia contents are dominating the current Web. In popular social media platforms such as FaceBook, Twitter, and Instagram, there are over millions of multimedia contents being created by users. In the meantime, multimedia data consists of data in multiple modalities, such as text, images, videos, audio, time series sequences, and so on. Many research efforts have been devoted to multimedia annotation to further improve the performance. However, the prevailing methods are designed for single-media annotation task. In fact, heterogeneous media content describes given labels from respective modality and is complementary to each other, and it becomes critical to explore advanced techniques for heterogeneous data analysis and multimedia annotation. Inspired by this idea, this paper presents a new multimodal correlation learning method for heterogeneous multimedia cooperative annotation, named unified space learning, which projects heterogeneous media data into one unified space. We formulate the multimedia annotation task into a semi-supervised learning framework, in which we learn different projection matrices for different media type. By doing so, different media content is aligned cooperatively, and jointly provides a more complementary profile of given semantic labels. Experimental results on data set with images, audio clips, videos and 3D models show that the proposed approach is more effective.

Multi-modal multi-concept-based deep neural network for automatic image annotation

Automatic Image Annotation (AIA) remains as a challenge in computer vision with real-world applications, due to the semantic gap between high-level semantic concepts and low-level visual appearances. Contextual tags attached to visual images and context semantics among semantic concepts can provide further semantic information to bridge this gap. In order to effectively capture these semantic correlations, we present a novel approach called Multi-modal Multi-concept-based Deep Neural Network (M2-DNN) in this study, which models the correlations of visual images, contextual tags, and multi-concept semantics. Unlike traditional AIA methods, our M2-DNN approach takes into account not only single-concept context semantics, but also multi-concept context semantics with abstract scenes. In our model, a multi-concept such as $\{``plane",``buildings"\}$ is viewed as one holistic scene concept for concept learning. Specifically, we first construct a multi-modal Deep Neural Network (DNN) as a concept classifier for visual images and contextual tags, and then employ it to annotate unlabeled images. Second, real-world databases commonly include many difficult concepts that are hard to be recognized, such as concepts with similar appearances, concepts with abstract scenes, and rare concepts. To effectively recognize them, we utilize multi-concept semantics inference and multi-modal correlation learning to refine semantic annotations. Finally, we estimate the most relevant labels for each of unlabeled images through a new strategy of label decision. The results of our comprehensive experiments on two publicly available datasets have shown that our method performs favourably compared with several other state-of-the-art methods.

Chaos sequence data classification mining research under network visualization analysis

ABSTRACTNetwork visualization analysis is a method of quantitative analysis of data in network services by using visualization techniques. However, it is difficult to classify the multimodal underlying sequence characteristics of the data by using the current method for the chaos sequence data classification mining; there is a problem of the large error of the classification of chaotic sequence data. In the article, this method is a method of network visualization analysis of chaotic sequence data classification mining based on multimodal subspace correlation transmission. This method uses the hidden Markov model to convert the chaotic sequence data into the likelihood space, the magnitude of the likelihood is identified by the symmetry Kullback–Leibler distance, extracting the underlying characteristics of multiple modal states, and carry out the transmission of the correlation between video data multimodal subspace. The accuracy and the mean square error were used to define the performance of the chaotic sequence data classification. The classification accuracy of this method is about 90%, and the root mean square error is less than 0.03, which greatly improves the classification accuracy. The packet loss rate of literature [Zheng B, Su H, Lin L. Application of unsupervised feature selection in time series data mining. Chin J Sci Instrum. 2014;35(4):834–840] is about 60%, the missing rate of literature [Mengli R. Algorithm of web hot data mining based on structured segmentation. Bull Sci Technol. 2015;4:115–117] is 50%, and the packet loss rate of this method is about 20%, which proves that this method has higher classification accuracy and application value, and is more suitable for other sequence data mining applications. The similarity between the time sequence data is given and reduces the dimension of the original data in order to obtain the coordinates module in the lower dimensional sequence data space, and then complete network visualization analysis to analyze chaos sequence data mining. The experimental simulation shows that the proposed method has a higher classification precision and greatly expands the application domain of the sequential data classification mining.

Multi-Modality Cascaded Convolutional Neural Networks for Alzheimer’s Disease Diagnosis

Accurate and early diagnosis of Alzheimer's disease (AD) plays important role for patient care and development of future treatment. Structural and functional neuroimages, such as magnetic resonance images (MRI) and positron emission tomography (PET), are providing powerful imaging modalities to help understand the anatomical and functional neural changes related to AD. In recent years, machine learning methods have been widely studied on analysis of multi-modality neuroimages for quantitative evaluation and computer-aided-diagnosis (CAD) of AD. Most existing methods extract the hand-craft imaging features after image preprocessing such as registration and segmentation, and then train a classifier to distinguish AD subjects from other groups. This paper proposes to construct cascaded convolutional neural networks (CNNs) to learn the multi-level and multimodal features of MRI and PET brain images for AD classification. First, multiple deep 3D-CNNs are constructed on different local image patches to transform the local brain image into more compact high-level features. Then, an upper high-level 2D-CNN followed by softmax layer is cascaded to ensemble the high-level features learned from the multi-modality and generate the latent multimodal correlation features of the corresponding image patches for classification task. Finally, these learned features are combined by a fully connected layer followed by softmax layer for AD classification. The proposed method can automatically learn the generic multi-level and multimodal features from multiple imaging modalities for classification, which are robust to the scale and rotation variations to some extent. No image segmentation and rigid registration are required in pre-processing the brain images. Our method is evaluated on the baseline MRI and PET images of 397 subjects including 93AD patients, 204 mild cognitive impairment (MCI, 76 pMCI +128 sMCI) and 100 normal controls (NC) from Alzheimer's Disease Neuroimaging Initiative (ADNI) database. Experimental results show that the proposed method achieves an accuracy of 93.26% for classification of AD vs. NC and 82.95% for classification pMCI vs. NC, demonstrating the promising classification performance.

Multi-modal features and correlation incorporated Naive Bayes classifier for a semantic-enriched lecture video retrieval system

ABSTRACTWith the advancement of science and technology and the development of internet sources, e-learning draws huge attention as it is capable of providing classroom lectures in the form of videos. Here, a method has been proposed for effective retrieval of the lecture videos, which employs the correlated naive Bayes (CNB) classifier. In this proposed method, Optical Character Recognition uses the tesseract classifier and GOM (Gabor Ordinal Measure) to extract the textual features and the image texture from the key frames. K-means clustering clusters the features and the classifier retrieves the relevant video. Experimentation has been done in the MATLAB and the parameters such as precision, recall, and F measure of the CNB are compared over the other methods such as K-Nearest Neighbour and naive Bayes (NB) classifiers. The CNB classifier achieves a better precision, recall, and F measure rate of 0.9366, 0.9511, and 0.9426, respectively.

Multimodal correlation of dynamic [18F]-AV-1451 perfusion PET and neuronal hypometabolism in [18F]-FDG PET

Cerebral glucose metabolism measured with [18F]-FDG PET is a well established marker of neuronal dysfunction in neurodegeneration. The tau-protein tracer [18F]-AV-1451 PET is currently under evaluation and shows promising results. Here, we assess the feasibility of early perfusion imaging with AV-1451 as a substite for FDG PET in assessing neuronal injury. Twenty patients with suspected neurodegeneration underwent FDG and early phase AV-1451 PET imaging. Ten one-minute timeframes were acquired after application of 200MBq AV-1451. FDG images were acquired on a different date according to clinical protocol. Early AV-1451 timeframes were coregistered to individual FDG-scans and spatially normalized. Voxel-wise intermodal correlations were calculated on within-subject level for every possible time window. The window with highest pooled correlation was considered optimal. Z-transformed deviation maps (ZMs) were created from both FDG and early AV-1451 images, comparing against FDG images of healthy controls. Regional patterns and extent of perfusion deficits were highly comparable to metabolic deficits. Best results were observed in a time window from 60 to 360s (r=0.86). Correlation strength ranged from r=0.96 (subcortical gray matter) to 0.83 (frontal lobe) in regional analysis. ZMs of early AV-1451 and FDG images were highly similar. Perfusion imaging with AV-1451 is a valid biomarker for assessment of neuronal dysfunction in neurodegenerative diseases. Radiation exposure and complexity of the diagnostic workup could be reduced significantly by routine acquisition of early AV-1451 images, sparing additional FDG PET.

Local and distant relationships between amyloid, tau and neurodegeneration in Alzheimer's Disease.

The relationships between β-amyloid (Aβ), tau and neurodegeneration within Alzheimer's Disease pathogenesis are not fully understood. To explore these associations in vivo, we evaluated 30 Aβ PET-positive patients (mean ± sd age 62.4 ± 8.3) with mild probable AD and 12 Aβ PET-negative healthy controls (HC) (mean ± sd age 77.3 ± 6.9) as comparison. All participants underwent 3 T MRI, 11C-PiB (Aβ) PET and 18F-AV1451 (tau) PET. Multimodal correlation analyses were run at both voxel- and region-of-interest levels. 11C-PiB retention in AD showed the most diffuse uptake pattern throughout association neocortex, whereas 18F-AV1451 and gray matter volume reduction (GMR) showed a progressive predilection for posterior cortices (p<0.05 Family-Wise Error-[FWE]-corrected). Voxel-level analysis identified negative correlations between 18F-AV1451 and gray matter peaking in medial and infero-occipital regions (p<0.01 False Discovery Rate-[FDR]-corrected). 18F-AV1451 and 11C-PiB were positively correlated in right parietal and medial/inferior occipital regions (p<0.001 uncorrected). 11C-PiB did not correlate with GMR at the voxel-level. Regionally, 18F-AV1451 was largely associated with local/adjacent GMR whereas frontal 11C-PiB correlated with GMR in posterior regions. These findings suggest that, in mild AD, tau aggregation drives local neurodegeneration, whereas the relationships between Aβ and neurodegeneration are not region specific and may be mediated by the interaction between Aβ and tau.

Structural health monitoring-based dynamic behavior evaluation of a long-span high-speed railway bridge

The dynamic performance of railway bridges under high-speed trains draws the attention of bridge engineers. The vibration issue for long-span bridges under high-speed trains is still not well understood due to lack of validations through structural health monitoring (SHM) data. This paper investigates the correlation between bridge acceleration and train speed based on structural dynamics theory and SHM system from three foci. Firstly, the calculated formula of acceleration response under a series of moving load is deduced for the situation that train length is near the length of the bridge span, the correlation between train speed and acceleration amplitude is analyzed. Secondly, the correlation scatterplots of the speed–acceleration is presented and discussed based on the transverse and vertical acceleration response data of Dashengguan Yangtze River Bridge SHM system. Thirdly, the warning indexes of the bridge performance for correlation scatterplots of speed–acceleration are established. The main conclusions are: (1) The resonance between trains and the bridge is unlikely to happen for long-span bridge, but a multimodal correlation curve between train speed and acceleration amplitude exists after the resonance speed; (2) Based on SHM data, multimodal correlation scatterplots of speed–acceleration exist and they have similar trends with the calculated formula; (3) An envelope line of polylines can be used as early warning indicators of the changes of bridge performance due to the changes of slope of envelope line and peak speed of amplitude. This work also gives several suggestions which lay a foundation for the better design, maintenance and long-term monitoring of a long-span high-speed bridge.

EP-1691: Multi-modal voxel-based correlation between DCE-CT/MRI and DWI in metastatic brain cancer

EP-1691: Multi-modal voxel-based correlation between DCE-CT/MRI and DWI in metastatic brain cancer

Semi-supervised semantic factorization hashing for fast cross-modal retrieval

Cross-modal hashing can effectively solve the large-scale cross-modal retrieval by integrating the advantages of traditional cross-modal analysis and hashing techniques. In cross-modal hashing, preserving semantic correlation is important and challenging. However, current hashing methods cannot well preserve the semantic correlation in hash codes. Supervised hashing requires labeled data which is difficult to obtain, and unsupervised hashing cannot effectively learn semantic correlation from multi-modal data. In order to effectively learn semantic correlation to improve hashing performance, we propose a novel approach: Semi-Supervised Semantic Factorization Hashing (S3FH), for large-scale cross-modal retrieval. The main purpose of S3FH is to improve semantic labels and factorize it into hash codes. It optimizes a joint framework which consists of three interactive parts, including semantic factorization, multi-graph learning and multi-modal correlation. Then, an efficient alternating algorithm is derived for optimizing S3FH. Extensive experiments on two real world multi-modal datasets demonstrate the effectiveness of S3FH.

Dual structural consistency based multi-modal correlation propagation projections for data representation

Canonical correlation analysis (CCA) is a powerful tool for analyzing multi-dimensional paired data. However, when facing semi-supervised multi-modal data (Also called multi-view Hou et al. (Pattern Recog 43(3):720–730, 2010) or multi-represented Kailing et al. (Clustering multi-represented objects with noise. In: Proceedings of the eighth Pacific-Asia conference on knowledge discovery and data mining (PAKDD). Sydney, Australia, pp 394–403) data. For convenience, we will uniformly call them multi-modal data hereafter.) which widely exist in real-world applications, CCA usually performs poorly due to ignoring useful supervised information. Meanwhile, due to the limited labeled training samples in the semi-supervised scenario, supervised extensions of CCA suffer from overfitting. Several semi-supervised extensions of CCA have been proposed recently. Nevertheless, they either just utilize the global structural information captured from the unlabeled data, or propagate label information by discovering the affinities just between the labeled and unlabeled data points in advance. In this paper, we propose a robust multi-modal semi-supervised feature extraction and fusion framework, termed as dual structural consistency based multi-modal correlation propagation projections (SCMCPP). SCMCPP guarantees the consistency between representation structure and hypotaxis structure in each modality and ensures the consistency of hypotaxis structure between two different modalities. By iteratively propagating labels and learning affinities, discriminative information of both given labels and estimated labels is utilized to improve the affinity construction and infer the remaining unknown labels. Moreover, probabilistic within-class scatter matrices in each modality and probabilistic correlation matrix between two modalities are constructed to enhance the discriminative power of features. Extensive experiments on several benchmark face databases demonstrate the effectiveness of our approach.

"Is It Rectangular?" Using I Spy as an Interactive, Game-Based Approach to Multimodal Robot Learning

Training robots about the objects in their environment requires a multimodal correlation of features extracted from visual and linguistic sources. This work abstracts the task of collecting multimodal training data for object and feature learning by encapsulating it in an interactive game, I Spy, played between human players and robots. It introduces the concept of the game, briefly describes its methodology, and finally presents an evaluation of the game's performance and its appeal to human players.

Content-oriented multimedia document understanding through cross-media correlation

This paper presents a novel method for multimedia document content analysis through modeling multimodal data correlations. We hypothesize that the correlation of different modalities from the same data source can help achieve better multimedia content understanding results than one which explores a single modality. We turn this task into two parts: multimedia data fusion and multimodal correlation propagation. During the first stage, we re-organize the training multimedia data into Modality semAntic Documents (MADs) after extracting quantized multimodal features, and then use multivariate Gaussian distributions to characterize the continuous quantity by latent topic modeling. Model parameters are asymmetrically learned to initialize multimodal correlations in the latent topic space. Accordingly, during the second stage, we construct a Multimodal Correlation Network (MCN) based on the initialized multimodal correlations, and a new mechanism of propagating inter-modality correlations and intra-modality similarities in MCN is further proposed to take the complementary from cross-modalities to facilitate multimedia content analysis. The experimental results of image-audio data retrieval on a 10-categories dataset and content-oriented web page recommendation on the USTODAY dataset show the effectiveness of our method.

P4‐375: Alzheimer's disease and mild cognitive impairment weaken connections within the default‐mode network: A multimodal study with resting‐state fMRI, diffusion MRI and FDG‐PET

A multi-modal imaging approach was applied to understand alterations in the default-mode network from normal aging to mild cognitive impairment (MCI) and Alzheimer's disease (AD). Participants included 10 probable AD, 11 amnestic MCI, and 12 age and education-matched normal control (NC) subjects. Whole-brain resting-state functional, diffusion tensor and volumetric MRI as well as FDG-PET data were acquired. Cognitive assessment on all subjects included St. Louis University test of Mental Status (SLUMS) and Wechsler Memory Scale III Logical Memory delayed recall (LM-dr) test. With the right and left isthmi of cingulate cortex (ICCs) as seed regions, functional correlation and fiber tracking connectivity distribution to all other voxels of the brain were calculated. Between-group t tests were carried. Correlation analyses were obtained between the functional connectivity of default-mode network components and (1) pons-normalized PET signal, (2) cortical thickness/volume and (3) cognitive performance. Whole-brain group analyses showed significant functional connectivity differences (NC > AD) from the right and left ICCs to medial frontal gyrus /anterior cingulate cortex and surrounding regions. There was a general trend of NC > MCI > AD on both functional and structural connectivity from the right and left ICCs to these frontal regions (Fig. 1). Significant correlations were found between PET signal in the right and left ICCs, right and left hippocampi and left ICC-right hippocampus functional connectivity (Table 1, Fig. 2). Significant correlations between PET signal and cortical thickness/volume were also found (Table 1). Significant correlations were found between the left ICC-right hippocampus functional connectivity and SLUMS score (r = 0.459, p = 0.0082) as well as LM-dr score (r = 0.375, p = 0.0347). There is a clear reduction of structural and functional connectivity within the default-mode network in AD, consistent with the reduction of metabolic activity and cortical atrophy in this network. The multi-modal correlation analyses suggest that neuronal damage in the left ICC is associated with declines in metabolic activity in this region, functional connectivity to this region and cognitive performance. Overall, the default-mode network continues to serve as a key region to understanding brain alterations due to AD. The integrated visualization of the functional and structural connections for normal, MCI and AD subjects (left to right) with the seed regions at the (a) right isthmus of cingulate cortex (ICC), (b) left ICC and (c) left lateral occipital cortex (LOC) and associated white matter regions. Thresholds: The mean correlation of > 0.2 for functional connectivity and mean connectivity distribution of > 1000 for structural connectivity. Green: functional connection only. Orange: structural connection only. Red: coexistence of both connections. GM: gray matter. WM: white matter. R = right. L = left. MeFG = medial frontal gyrus. With all subjects included, the significant correlations were shown between the left ICC-right hippocampus functional connection (the Fisher's Z-transformed values of the correlations) and the normalized PET signals in (a) left ICC and (b) right hippocampus. R = right, L = left and ICC= isthmus of cingulate cortex.

Long-term effects of motor training on resting-state networks and underlying brain structure

Acquired motor skills are coded in fronto-parietal brain networks, but how these networks evolve through motor training is unclear. On the one hand, increased functional connectivity has been shown immediately after a training session; on the other hand, training-induced structural changes are visible only after several weeks. Based on known associations between functional and structural network development during human ontogeny, we hypothesised that learning a challenging motor task leads to long-lasting changes in functional resting-state networks and the corresponding cortical and sub-cortical brain structures. Using longitudinal functional and structural MRI at multiple time points, we demonstrate increased fronto-parietal network connectivity one week after two brief motor training sessions in a dynamic balancing task, although subjects were engaged in their regular daily activities during the week. Repeated training sessions over six consecutive weeks progressively modulate these changes in accordance with individual performance improvements. Multimodal correlation analyses showed an association between structural grey matter alterations and functional connectivity changes in prefrontal and supplementary-motor areas. These coincident changes were most prominent in the first three weeks of training. In contrast, changes in fronto-parietal functional connectivity and the underlying white matter fibre structure developed gradually during the six weeks. Our results demonstrate a tight correlation between training-induced functional and structural brain plasticity on the systems level and suggest a functional relevance of intrinsic brain activity for morphological adaptation in the human brain.

Biological parametric mapping: A statistical toolbox for multimodality brain image analysis

In recent years, multiple brain MR imaging modalities have emerged; however, analysis methodologies have mainly remained modality-specific. In addition, when comparing across imaging modalities, most researchers have been forced to rely on simple region-of-interest type analyses, which do not allow the voxel-by-voxel comparisons necessary to answer more sophisticated neuroscience questions. To overcome these limitations, we developed a toolbox for multimodal image analysis called biological parametric mapping (BPM), based on a voxel-wise use of the general linear model. The BPM toolbox incorporates information obtained from other modalities as regressors in a voxel-wise analysis, thereby permitting investigation of more sophisticated hypotheses. The BPM toolbox has been developed in Matlab with a user-friendly interface for performing analyses, including voxel-wise multimodal correlation, ANCOVA, and multiple regression. It has a high degree of integration with the SPM (statistical parametric mapping) software relying on it for visualization and statistical inference. Furthermore, statistical inference for a correlation field, rather than a widely used T-field, has been implemented in the correlation analysis for more accurate results. An example with in vivo data is presented, demonstrating the potential of the BPM methodology as a tool for multimodal image analysis.