Multimodal Correlation Research Articles

Design of an Efficient Multimodal Correlation Engine for Smart IoMT Imaging System for Pre-Emptive Analysis of Breast Cancer Stages

Design of an Efficient Multimodal Correlation Engine for Smart IoMT Imaging System for Pre-Emptive Analysis of Breast Cancer Stages

Altered correlation of concurrently recorded EEG-fMRI connectomes in temporal lobe epilepsy.

Whole-brain functional connectivity networks (connectomes) have been characterized at different scales in humans using EEG and fMRI. Multimodal epileptic networks have also been investigated, but the relationship between EEG and fMRI defined networks on a whole-brain scale is unclear. A unified multimodal connectome description, mapping healthy and pathological networks would close this knowledge gap. Here, we characterize the spatial correlation between the EEG and fMRI connectomes in right and left temporal lobe epilepsy (rTLE/lTLE). From two centers, we acquired resting-state concurrent EEG-fMRI of 35 healthy controls and 34 TLE patients. EEG-fMRI data was projected into the Desikan brain atlas, and functional connectomes from both modalities were correlated. EEG and fMRI connectomes were moderately correlated. This correlation was increased in rTLE when compared to controls for EEG-delta/theta/alpha/beta. Conversely, multimodal correlation in lTLE was decreased in respect to controls for EEG-beta. While the alteration was global in rTLE, in lTLE it was locally linked to the default mode network. The increased multimodal correlation in rTLE and decreased correlation in lTLE suggests a modality-specific lateralized differential reorganization in TLE, which needs to be considered when comparing results from different modalities. Each modality provides distinct information, highlighting the benefit of multimodal assessment in epilepsy.

AM-SegNet for additive manufacturing in situ X-ray image segmentation and feature quantification

ABSTRACT Synchrotron X-ray imaging has been utilised to detect the dynamic behaviour of molten pools during the metal additive manufacturing (AM) process, where a substantial amount of imaging data is generated. Here, we develop an efficient and robust deep learning model, AM-SegNet, for segmenting and quantifying high-resolution X-ray images and prepare a large-scale database consisting of over 10,000 pixel-labelled images for model training and testing. AM-SegNet incorporates a lightweight convolution block and a customised attention mechanism, capable of performing semantic segmentation with high accuracy (∼96%) and processing speed (< 4 ms per frame). The segmentation results can be used for quantification and multi-modal correlation analysis of critical features (e.g. keyholes and pores). Additionally, the application of AM-SegNet to other advanced manufacturing processes is demonstrated. The proposed method will enable end-users in the manufacturing and imaging domains to accelerate data processing from collection to analytics, and provide insights into the processes’ governing physics.

EMCI: An Explainable Multimodal Correlation Integration Model for Unveiling Spatial Transcriptomics and Intercellular Signaling

Current integration methods for single-cell RNA sequencing (scRNA-seq) data and spatial transcriptomics (ST) data are typically designed for specific tasks, such as deconvolution of cell types or spatial distribution prediction of RNA transcripts. These methods usually only offer a partial analysis of ST data, neglecting the complex relationship between spatial expression patterns underlying cell-type specificity and intercellular cross-talk. Here, we present eMCI, an explainable multimodal correlation integration model based on deep neural network framework. eMCI leverages the fusion of scRNA-seq and ST data using different spot–cell correlations to integrate multiple synthetic analysis tasks of ST data at cellular level. First, eMCI can achieve better or comparable accuracy in cell-type classification and deconvolution according to wide evaluations and comparisons with state-of-the-art methods on both simulated and real ST datasets. Second, eMCI can identify key components across spatial domains responsible for different cell types and elucidate the spatial expression patterns underlying cell-type specificity and intercellular communication, by employing an attribution algorithm to dissect the visual input. Especially, eMCI has been applied to 3 cross-species datasets, including zebrafish melanomas, soybean nodule maturation, and human embryonic lung, which accurately and efficiently estimate per-spot cell composition and infer proximal and distal cellular interactions within the spatial and temporal context. In summary, eMCI serves as an integrative analytical framework to better resolve the spatial transcriptome based on existing single-cell datasets and elucidate proximal and distal intercellular signal transduction mechanisms over spatial domains without requirement of biological prior reference. This approach is expected to facilitate the discovery of spatial expression patterns of potential biomolecules with cell type and cell–cell communication specificity.

Work Together: Correlation-Identity Reconstruction Hashing for Unsupervised Cross-Modal Retrieval

Unsupervised cross-modal hashing has attracted considerable attention to support large-scale cross-modal retrieval. Although promising progresses have been made so far, existing methods still suffer from limited capability on excavating and preserving the intrinsic multi-modal semantics. In this paper, we propose a Correlation-Identity Reconstruction Hashing (CIRH) method to alleviate this challenging problem. We develop a new unsupervised deep cross-modal hash learning framework to model and preserve the heterogeneous multi-modal correlation semantics into both hash codes and functions, and simultaneously, we involve both the hash codes and functions with the descriptive identity semantics. Specifically, we construct a multi-modal collaborated graph to model the heterogeneous multi-modal correlations, and jointly perform the intra-modal and cross-modal semantic aggregation on homogeneous and heterogeneous graph networks to generate a multi-modal complementary representation with correlation reconstruction. Furthermore, an identity semantic reconstruction process is designed to involve the generated representation with identity semantics by reconstructing the input modality representations. Finally, we propose a correlation-identity consistent hash function learning strategy to transfer the modelled multi-modal semantics into the neural networks of modality-specific deep hash functions. Experiments demonstrate the superior performance of the proposed method on both retrieval accuracy and efficiency. We provide our source codes and experimental datasets at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/XizeWu/CIRH</uri> .

Multi-Modality, Multi-Dimensional Characterization of Pediatric Non-Alcoholic Fatty Liver Disease.

Non-alcoholic fatty liver disease is a multifaceted disease that progresses through multiple phases; it involves metabolic as well as structural changes. These alterations can be measured directly or indirectly through blood, non-invasive imaging, and/or tissue analyses. While some studies have evaluated the correlations between two sets of measurements (e.g., histopathology with cross-sectional imaging or blood biomarkers), the interrelationships, if any, among histopathology, clinical blood profiles, cross-sectional imaging, and metabolomics in a pediatric cohort remain unknown. We created a multiparametric clinical MRI-histopathologic NMR network map of pediatric NAFLD through multimodal correlation networks, in order to gain insight into how these different sets of measurements are related. We found that leptin and other blood markers were correlated with many other measurements; however, upon filtering out the blood biomarkers, the network was decomposed into three independent hubs centered around histopathological features, each with associated MRI and plasma metabolites. These multi-modality maps could serve as a framework for characterizing disease status and progression and could potentially guide medical interventions.

Multi-modal cognitive computing

The human brain perceives its surroundings through multiple sensory organs and integrates these multi-sensory perceptions to generate a comprehensive understanding. Inspired by synaesthesia, multi-modal cognitive computing endows machines with multi-sensory capabilities and has become the key to general artificial intelligence. With the explosion of multi-modal data such as image, video, text, and audio, a large number of methods have been developed to address this topic. However, the theoretical basis of multi-modal cognitive computing is still unclear. From the perspective of information theory, this paper establishes an information transmission model to profile the cognitive process. Based on the theory of information capacity, this study finds out that multi-modal cognitive computing helps machines extract more information. In this way, multi-modal cognitive computing research is unified by the same theoretical basis. Then, the development of typical tasks is reviewed and discussed, including multi-modal correlation, cross-modal generation, and multi-modal collaboration. Finally, focusing on the opportunities and challenges faced by multi-modal cognitive computing, some potential directions are discussed in depth, and several open-ended questions are considered.

Excavating multimodal correlation for representation learning

A majority of previous methods for multimodal representation learning ignore the rich correlation information inherently stored in each sample, leading to a lack of robustness when trained on small datasets. Although a few contrastive learning frameworks leverage that information in a self-supervised manner, they generally encourage the intra-sample unimodal representations to be identical, neglecting the modality-specific information carried by individual modalities. In contrast, we propose a novel algorithm that learns the correlations between modalities to facilitate downstream multimodal tasks by leveraging the prior information across samples, and we explore the feasibility of the proposed method on elaborately designed unsupervised and supervised auxiliary learning tasks. Specifically, we construct the positive and negative sets for correlation learning as unimodal embeddings from the same sample and from different samples, respectively. A weak predictor is employed on the concatenated unimodal embeddings to learn the correspondence relationship for each set. In this way, the model can correlate unimodal features and discover the shared information across modalities. In contrast to contrastive learning methods, the proposed framework is compatible with any number of modalities and can retain modality-specific information, enabling multimodal representation to capture richer information. Moreover, in the supervised version, one of the main novelties is that the sample labels are further utilized to learn more discriminative features, where the assigned correlation scores of negative sets vary according to the label variations between the associated samples. Extensive experiments suggest that the proposed method reaches state-of-the-art performance on the tasks of multimodal sentiment analysis, emotion recognition, and humor detection, and can improve the performance of various fusion approaches.

Temporal sentence grounding in videos with fine-grained multimodal correlation

Videos have become a new way of communication among Internet users with the proliferation of sensor-rich mobile devices. Due to the redundant background information in video data, people usually spend much time browsing and analyzing video content. This necessity motivates us to investigate the temporal sentence grounding task in videos. Formally, given an untrimmed video and a natural language sentence query, the task is to identify the start and end points of the video segment in response to the given sentence query. With such a technique, people can quickly find specific content of interest in the video by providing a clear and concise text description, thereby improving users’ video browsing experience and search efficiency. Previous methods often formulate the temporal grounding task as a multimodal matching problem. Doing so ignores the important sentence details for grounding and neglects the important guiding role of sentences to compose and correlate video contents over time, causing limited temporal grounding accuracy. To solve the above problems, we first propose a multimodal co-attention mechanism to mine important semantic details for temporal grounding in the given query and finely construct the semantic correlation between each word in the sentence and the video content. On this basis, we then propose a semantic condition dynamic normalization mechanism to tightly compose the sentence-related video content over time, including a clip-level actionness prediction module for fine-grained temporal boundary adjustment, thus making the temporal grounding results in the video clearer, more flexible, and more accurate than usual. Experiments on public datasets also verify our effectiveness and superiority over the state-of-the-arts. Last but not least, we present our insights on future research directions that deserve further investigations in the areas of audio-enabled temporal grounding techniques, weakly supervised grounding problem formulation, and debiased temporal grounding dataset construction.

Multimodal MRI demonstrates task-related cortical hyper-activation and neuro- chemical alteration in amyotrophic lateral sclerosis

BackgroundThere is a paucity of reliable markers of disease activity and progression in ALS. Better biomarkers would also reduce clinical trial duration and cost by providing more sensitive measures of target engagement. Cortical hyperexcitability with transcranial magnetic stimulation, while promising, has not yet been clinically translatable. Multimodal correlation of cortical hyperexcitability could yield more robust composite biomarkers.MethodsThis project applies multimodal non-invasive neuroimaging (MRI, MEG) and neurophysiol- ogy (TMS) to explore cortical hyperexcitability in ALS. 11 affected ALS patients, 9 age matched healthy controls and 13 asymptomatic relatives at-risk of inheriting the disease-causing C9orf72 hexanucleotide repeat expansion were studied. MRI findings are reported here (diffusion-tensor imaging [DTI], functional MRI and MR spectroscopy).ResultsRight cortical activation with contralateral finger movement was significantly increased in patients (p = 0.04), while right cortical de-activation with ipsilateral movement was lost (p = 0.01). N-acetyl-aspar- tate (p=0.04) and glutamate (p = 0.01) are significantly reduced in patients. At-risk relatives occupy an intermediate profile.DiscussionBilateral motor cortex hyperexcitability was found in ALS. Asymmetry in task-related activa- tion is consistent with previously reported loss of cortical inhibition in early ALS. Integration of multimodal imaging and neurophysiological data could explain variable phenotype and disease mechanisms in ALS.eedmond@gmail.com

Personalized Smart Clothing Design Based on Multimodal Visual Data Detection.

In the traditional clothing customization system, only the designer participates in the clothing design, and the style is single. In the face of numerous styles, the user just repeatedly arranges and combines the styles, but does not realize the user's innovative design. In this paper, we propose a novel multitask deep convolutional neural network training method for task-by-task transfer learning, and learn deep image features for image retrieval tasks on noisy user click data. Image retrieval model based on image-text multimodal correlation features: this paper uses image-text multimodal correlation features to calculate the correlation between query keywords and images, and calculates the correlation between images and images. In this paper, the method of automatic generation of clothing style is researched, and parameterized coding is designed for it. Taking the typical style of suits as the initial population, through the human-computer interaction interface, the scoring value is assigned to the fitness value to carry out the evolution process. The binary string of the suit style generated by the genetic algorithm is decoded through the decoding algorithm and rules, decoded into a visual style diagram of the style of the suit style, and the style diagram of the suit style is automatically drawn. From the perspective of clothing design, this paper summarizes the general design methods of outdoor sports smart clothing, follows the integration of people-clothing-environment, and proposes a wearer-centered design concept to deeply explore the rationality of outdoor sports smart clothing design methods. This paper further solves the key problems in the design process, and takes the fashion of clothing as the core principle to design the structure of clothing modeling. The fabric selection of suits is based on the principle of clothing comfort. The key is to realize the outdoor sports monitoring function of suits through sensing technology. The design uses Arduino as an electronic prototype platform, so as to detect the heart rate of the human body during exercise and the microclimate temperature under the clothes. This kind of suit with monitoring function is ultimately a combination of sensing device and clothing. It not only has monitoring function, but also has the aesthetic concept of clothing design and conforms to the performance of human body structure, which will provide reference and reference for the design of outdoor sports smart clothing.

Neighbor-consistent multi-modal canonical correlations for feature fusion

Locality neighbor relationships of raw high-dimensional data are usually utilized in multi-modal feature fusion methods, and intrinsic locality neighbor relationships hidden in the raw data are beneficial to the discriminative power of fused low-dimensional features. However, since the raw data contains a lot of redundant information and noises, the neighbor relationships will deviate from intrinsic neighbor relationships and the deviation can weaken the discriminative power of the fused features. Aiming at this issue, we construct cross-modal neighbor-consistent scatters of all the modalities by explicitly embedding the neighbor complementarity of different modalities. Then we constrain the scatters in the multi-modal correlation analysis framework and further develop a novel neighbor-consistent correlation feature fusion method, i.e. neighbor-consistent multi-modal canonical correlations (NcMCC). The fused correlation features of our method preserve the intrinsic neighbor relationships with cross-modal neighbor complementarity as many as possible and possess the well discriminative power. Extensive experimental results on several categories of images such as thermal images have demonstrated the effectiveness and robustness of our method in image recognition.

Construction and applications of knowledge graph of porphyry copper deposits

A knowledge graph is becoming popular due to its ability to describe the real world by using a graph language that can be understood by both humans and machines using computer technologies. A case study to construct the knowledge graph of porphyry copper deposits is presented in this paper. First of all, the raw text data is collected and integrated from selected porphyry copper deposits and porphyry-skarn copper deposits in the Qinzhou Bay – Hangzhou Bay metallogenic belt, South China. Second, the text's entities, relations, and attributes are labeled and extracted with reference to the conceptual model of porphyry copper deposits in the study area. The third, a knowledge graph of porphyry copper deposits, was constructed using Neo4j 4.3. The resulted knowledge graph of porphyry copper deposit has the basic functions of an application. Furthermore, as part of a planned integrated knowledge graph from a single deposit, through an upper-geared metallogenic series, to a high-top metallogenic province, the understanding from the present study may be extended to mineral resource prospectivity and assessment beyond today. The interrelationship between the earth system, the metallogenic system, the exploration system, and the prospectivity and assessment (ES-MS-ES-PS) should be completely understood, and a knowledge graph system for ES-MS-ES-PS is needed. The key scientific and technological problems for achieving the ES-MS-ES-PS knowledge graph system are included in the progressively relative system of the domain ontology and knowledge graph of ES-MS-ES-PS, the automatic construction technology of complicated ESMS-ES-PS domain ontology and knowledge graph, the self-evolution and complementary techniques for multi-modal correlation data embedding in the ES-MS-ES-PS knowledge graph, and the knowledge graph, big data mining and artificial intelligence based on ES-resource prospectivity, and assessment theory, and methods.

Increased brain atrophy and lesion load is associated with stronger lower alpha MEG power in multiple sclerosis patients

In multiple sclerosis, the interplay of neurodegeneration, demyelination and inflammation leads to changes in neurophysiological functioning. This study aims to characterize the relation between reduced brain volumes and spectral power in multiple sclerosis patients and matched healthy subjects.During resting-state eyes closed, we collected magnetoencephalographic data in 67 multiple sclerosis patients and 47 healthy subjects, matched for age and gender. Additionally, we quantified different brain volumes through magnetic resonance imaging (MRI).First, a principal component analysis of MRI-derived brain volumes demonstrates that atrophy can be largely described by two components: one overall degenerative component that correlates strongly with different cognitive tests, and one component that mainly captures degeneration of the cortical grey matter that strongly correlates with age. A multimodal correlation analysis indicates that increased brain atrophy and lesion load is accompanied by increased spectral power in the lower alpha (8–10 Hz) in the temporoparietal junction (TPJ). Increased lower alpha power in the TPJ was further associated with worse results on verbal and spatial working memory tests, whereas an increased lower/upper alpha power ratio was associated with slower information processing speed.In conclusion, multiple sclerosis patients with increased brain atrophy, lesion and thalamic volumes demonstrated increased lower alpha power in the TPJ and reduced cognitive abilities.

Multi‐level Deep Correlative Networks for Multi‐modal Sentiment Analysis

Multi-modal sentiment analysis (MSA) is increasingly becoming a hotspot because it extends the conventional Sentiment analysis (SA) based on texts to multi-modal content which can provide richer affective information. However, compared with textbased sentiment analysis, multi-modal sentiment analysis has much more challenges, because the joint learning process on multi-modal data requires both fine-grained semantic matching and effective heterogeneous feature fusion. Existing approaches generally infer sentiment type from splicing features extracted from different modalities but neglect the strong semantic correlation among cooccurrence data of different modalities. To solve the challenges, a multi-level deep correlative network for multimodal sentiment analysis is proposed, which can reduce the semantic gap by analyzing simultaneously the middlelevel semantic features of images and the hierarchical deep correlations. First, the most relevant cross-modal feature representation is generated with Multi-modal Deep and discriminative correlation analysis (Multi-DDCA) while keeping those respective modal feature representations to be discriminative. Second, the high-level semantic outputs from multi-modal deep and discriminative correlation analysis are encoded into attention-correlation cross-modal feature representation through a co-attention-based multimodal correlation submodel, and then they are further merged by multi-layer neural network to train a sentiment classifier for predicting sentimental categories. Extensive experimental results on five datasets demonstrate the effectiveness of the designed approach, which outperforms several state-of-the-art fusion strategies for sentiment analysis.

Semantic ranking structure preserving for cross-modal retrieval

Cross-modal retrieval not only needs to eliminate the heterogeneity of modalities, but also needs to constrain the return order of retrieval results. Accordingly, we propose a novel common representation space learning method, called Semantic Ranking Structure Preserving (SRSP) for Cross-modal Retrieval in this paper. First, the dependency relationship between labels is used to minimize the discriminative loss of multi-modal data and mine potential relationships between samples to get richer semantic information in the common space. Second, we constrain the correlation ranking of representations in common space, so as to break the modal gap and promote the multi-modal correlation learning. The comprehensive experimental comparison results show that our algorithm substantially enhances the performance and consistently outperforms very recent algorithms in terms of widely used cross-modal benchmark datasets.

Adversarial Learning With Multi-Modal Attention for Visual Question Answering.

Visual question answering (VQA) has been proposed as a challenging task and attracted extensive research attention. It aims to learn a joint representation of the question-image pair for answer inference. Most of the existing methods focus on exploring the multi-modal correlation between the question and image to learn the joint representation. However, the answer-related information is not fully captured by these methods, which results that the learned representation is ineffective to reflect the answer of the question. To tackle this problem, we propose a novel model, i.e., adversarial learning with multi-modal attention (ALMA), for VQA. An adversarial learning-based framework is proposed to learn the joint representation to effectively reflect the answer-related information. Specifically, multi-modal attention with the Siamese similarity learning method is designed to build two embedding generators, i.e., question-image embedding and question-answer embedding. Then, adversarial learning is conducted as an interplay between the two embedding generators and an embedding discriminator. The generators have the purpose of generating two modality-invariant representations for the question-image and question-answer pairs, whereas the embedding discriminator aims to discriminate the two representations. Both the multi-modal attention module and the adversarial networks are integrated into an end-to-end unified framework to infer the answer. Experiments performed on three benchmark data sets confirm the favorable performance of ALMA compared with state-of-the-art approaches.

Clustering adaptive canonical correlations for high-dimensional multi-modal data

Multi-modal canonical correlation analysis (MCCA) is an important joint dimension reduction method and has been widely applied to clustering tasks of multi-modal data. MCCA-based clustering is usually dimension reduction of high-dimensional data followed by clustering of low-dimensional data. However, the two-stage clustering is difficult to ensure the adaptability of dimension reduction and clustering, which will affect the final clustering performance. To solve the issue, we propose a novel clustering adaptive multi-modal canonical correlations (CAMCCs) method, which constructs a unified optimization model of multi-modal correlation learning and clustering. The method not only realizes discriminant learning of correlation projection directions under unsupervised cases, but also is able to directly obtain class labels of multi-modal data. Additionally, the method also realizes out-of-sample extension in class labels. Solutions of CAMCCs are optimized by an iterative way, and we analyze its convergence. Extensive experimental results on various datasets have demonstrated the effectiveness of the method.

Sequential Extraction of Late Exponentials (SELE): A technique for deconvolving multimodal correlation curves in Dynamic Light Scattering

Abstract:In techniques such as Dynamic Light Scattering (DLS), Fluorescence Correlation Spectroscopy, and image mining, motion is tracked by the autocorrelation of a signal over logarithmic time scales. For instance the tracking signal in DLS is the scattered light intensity; it remains correlated at time scales where scant changes in the arrangement of the scattering particles occur, but decays exponentially at the time scales of their diffusion. When there are multiple time scales of motion (for instance due to scatterers of different sizes), the correlation curve has more than one exponential fall. Extracting the decay constants or hydrodynamic sizes due to each exponential fall in a multi-species field correlation curve becomes an ill-conditioned mathematical problem. We describe a new algorithm to invert a multi-modal correlation curve by Sequential Extraction of the Late Exponentials (SELE). The idea is that while the inversion of a multi-exponential equation may be ill posed, that of a single exponential is not. So we fit data windows towards to base of the correlation curve to extract the largest contribution species, remove the species contribution from the correlation curve, and repeat the process with the remnant curve. The single exponent can be robustly fitted by least-square minimization with initial guesses generated by an adapted cumutant technique (power-series) that includes stretch coefficients (measure of sample dispersity). The proposed algorithm resolves particle sizes separated by 3X, and is reliable against fluctuations in the correlation curve and to localized regions of suboptimal data. The algorithm can be used to track particle dynamics in solution in multi-species problems such as self-assembly.

Prognostic role of EEG identical bursts in patients after cardiac arrest: Multimodal correlation

AimsEEG burst-suppression (BS) heralds poor outcome after cardiac arrest (CA). Within this pattern, identical bursts (IB) have been suggested to be absolutely specific, in isolation. We assessed IB prevalence and their added predictive value for poor outcome in a multimodal prognostic approach. MethodsWe retrospectively analyzed a registry of comatose adults with CA (April 2011–February 2019), undergoing EEG at 5–36 h and 36–72 h. SB and IB were visually assessed. Cerebral Performance Categories (CPC) at 3 months were dichotomized as “good” (CPC 1–2), or “poor” (CPC 3–5). Sensitivity, specificity, positive, negative predictive values of BS and IB for poor outcome were calculated. A multimodal prognostic score was created assigning one point each to unreactive and epileptiform EEG, pupillary light reflex and SSEPs absence, NSE > 75 μg/l. In a second score, IB were added; predictive performances were compared using Receiver Operating Characteristic (ROC) curves. ResultsOf 522 patients, 147 (28%) had BS in any EEG (10 [7%] had good outcome and 129 [88%] died). Of them, 53/147 (36%, 10% of total) showed IB, 47/53 (89%) of which within 36 h. IB were 100% specific for poor outcome, and associated with higher serum NSE than BS. However, there was no significant difference between the scores with and without IB for CPC 3–5 (p = 0.116). ConclusionIB occur in 10% of patients after CA. In our multimodal context, IB, albeit being very specific for poor outcome, seem redundant with other predictors.