Representation Fusion Research Articles

Temporal Knowledge Graph Reasoning Based on Dynamic Fusion Representation Learning

ABSTRACTRecently, significant progress has been made in completing static knowledge graphs. However, knowledge tends to evolve with time, and static knowledge graph completion (KGC) methods struggle to capture the changes. Therefore, temporal knowledge graph (TKG) reasoning has become a focus of research. Most existing TKG methods incorporate temporal information into triplets and transform them into KGC tasks, ignoring the important influence of time information and implicit relationships between entities. In this paper, we propose a new method called TD‐RKG, which addresses the challenges of temporal variability and implicit entity correlations based on a dynamic fusion representation learning approach. The method consists of four modules: dynamic local recurrent encoding layer, dynamic implicit encoding layer, dynamic global information attention layer and decoding layer. Experimental results on three benchmark datasets demonstrate substantial improvements in TD‐RKG across multiple evaluation metrics.

IPCT-Net: Parallel information bottleneck modality fusion network for obstructive sleep apnea diagnosis

Obstructive sleep apnea (OSA) is a common sleep breathing disorder and timely diagnosis helps to avoid the serious medical expenses caused by related complications. Existing deep learning (DL)-based methods primarily focus on single-modal models, which cannot fully mine task-related representations. This paper develops a modality fusion representation enhancement (MFRE) framework adaptable to flexible modality fusion types with the objective of improving OSA diagnostic performance, and providing quantitative evidence for clinical diagnostic modality selection. The proposed parallel information bottleneck modality fusion network (IPCT-Net) can extract local-global multi-view representations and eliminate redundant information in modality fusion representations through branch sharing mechanisms. We utilize large-scale real-world home sleep apnea test (HSAT) multimodal data to comprehensively evaluate relevant modality fusion types. Extensive experiments demonstrate that the proposed method significantly outperforms existing methods in terms of participant numbers and OSA diagnostic performance. The proposed MFRE framework delves into modality fusion in OSA diagnosis and contributes to enhancing the screening performance of artificial intelligence (AI)-assisted diagnosis for OSA.

VTNet: A multi-domain information fusion model for long-term multi-variate time series forecasting with application in irrigation water level

Time series forecasting is intricately tied to production and life, garnering widespread attention over an extended period. Enhancing the performance of long-term multivariate time series forecasting (MTSF) poses a highly challenging task, as it requires mining complicated and obscure temporal patterns in many aspects. For this reason, this paper proposes a long-term forecasting model based on multi-domain fusion (VTNet) to adaptively capture and refine multi-scale intra- and inter-variate dependencies. In contrast to previous techniques, we devise a dual-stream learning architecture. Firstly, the fast Fourier Transform (FFT) is adopted to extract frequency domain information. The original sequences are then transformed into 2D visual features in the temporal-frequency domain, and a 2D-TBlock is designed for multi-scale dynamic learning. Secondly, a combination of convolution and recurrent networks continues to explore the local temporal features and preserve the global trend. Finally, multi-modal circulant fusion is applied to achieve a more comprehensive and enriched feature fusion representation, further promoting overall performance. Extensive experiments are conducted on 9 public benchmark datasets and the real-world irrigation water level to showcase VTNet’s promoted performance and generalization. Moreover, VTNet yields 46.93% and 25.36% relative improvements for water level forecasting, revealing its potential application value in water-saving planning and extreme event early warning.

Deep Multiview Adaptive Clustering With Semantic Invariance.

Multiview clustering has attracted significant attention in various fields, due to the superiority in mining patterns of multiview data. However, previous methods are still confronted with two challenges. First, they do not fully consider the semantic invariance of multiview data in aggregating complementary information, degrading semantic robustness of fusion representations. Second, they rely on predefined clustering strategies to mine patterns, lacking adequate explorations of data structures. To address the challenges, deep multiview adaptive clustering via semantic invariance (DMAC-SI) is proposed, which learns an adaptive clustering strategy on semantics-robust fusion representations to fully explore structures in mining patterns. Specifically, a mirror fusion architecture is devised to explore interview invariance and intrainstance invariance hidden in multiview data, which captures invariant semantics of complementary information to learn semantics-robust fusion representations. Then, a Markov decision process of multiview data partitions is proposed within the reinforcement learning framework, which learns an adaptive clustering strategy on semantics-robust fusion representations to guarantee the structure explorations in mining patterns. The two components seamlessly collaborate in an end-to-end manner to accurately partition multiview data. Finally, extensive experiment results on five benchmark datasets demonstrate that DMAC-SI outperforms the state-of-the-art methods.

Multi-spectral image fusion for moving object detection

This study delves into the problem of moving object detection in infrared and visible images. While existing approaches primarily focus on single-task detection using single-spectral image data, such as thermal infrared or visible images, they often ignore the differences between different spectral images and the interconnectedness of related tasks such as image fusion and segmentation. To tackle these problems, we present a novel multi-spectral image fusion network with quality and semantic awareness for moving object detection (MOD), particularly in scenarios where ground truth labels for both infrared and visible images are unavailable. Our approach fuses multi-spectral images, incorporating additional subtasks in the image fusion to obtain content and quality perception of infrared and visible images. In addition, we design a novel residual global perception module (RGPM) and multi-spectral fusion loss, which can capture more hidden features and contextual information across various scales. This enhanced capability leads to more precise detection and tracking of moving objects, particularly in challenging situations involving occlusions and dynamic backgrounds. Compared with single-spectral moving object detection optimization, the hurdles of utilizing deep learning for multi-spectral image fusion, e.g., without ground truth labels and harmful noise, are significantly mitigated. Extensive quantitative and qualitative comparative experiments demonstrate its effectiveness, robustness, and superior performance compared to contemporary methods. Concisely, the proposed fusion representation learning has gained 44.2%,31.2%,36.3%,41.6%,5.3%,31.2%,76.2% on EI, SF, DF, AG, MI, SD and Nabf metrics compared with the best competitors.

ICKA: An instruction construction and Knowledge Alignment framework for Multimodal Named Entity Recognition

Multimodal Named Entity Recognition (MNER) aims to identify entities of predefined types in text by leveraging information from multiple modalities, most notably textual and visual information. Most efforts concentrate on improving cross-modality attention mechanisms to facilitate guidance between modalities. However, they still suffer from certain limitations: (1) it is difficult to establish a unified representation to bridge the semantic gap among different modalities; (2) mining the implicit relationships between text and image is crucial yet challenging. In this paper, we propose an Instruction Construction and Knowledge Alignment Framework for MNER named ICKA to address these issues. Specifically, we first employ a multi-head cross-modal attention mechanism to obtain the cross-modal fusion representation by fusing features from text–image pairs. Then, we integrate external knowledge from the pre-trained vision-language model (VLM) to facilitate semantic alignment between text and image and obtain inter-modality connections. Next, we construct the multimodal instruction that consists of the modal features and uses the inter-modality connections as a bridge between them. We then integrate the instruction into the language model to effectively incorporate multimodal knowledge. Finally, we perform sequence labeling using a Conditional Random Fields (CRF) decoder with a gating mechanism. The proposed method achieves F1 scores of 75.42% on the Twitter2015 dataset and 87.12% on the Twitter2017 dataset, demonstrating the competitiveness of our method.

TCHFN: Multimodal sentiment analysis based on Text-Centric Hierarchical Fusion Network

Multimodal sentiment analysis (MSA) has become a popular field of research in recent years. The aim is to combine the three modalities of text, video, and audio to obtain comprehensive emotional information. However, current research often treats these three modalities equally, downplaying the crucial role of text modality in MSA and ignoring the redundant information generated during multimodal fusion. To address these problems, we propose the Text-Centric Hierarchical Fusion Network (TCHFN), employing a hierarchical fusion strategy. In this framework, low-level fusion involves cross-modal interactions between pairs of modalities, while high-level fusion extends these interactions to involve all three modalities. Through the design of the Cross-modal Reinforced Transformer (CRT), we achieve cross-modal enhancement of the target modality, facilitating a nuanced fusion process with text serving as its core. Additionally, we design Text-Centric Contrastive Learning (TCCL) to align non-text modalities with the text modality, emphasising the central role of text in the fusion process. After fusion, a multimodal fusion output gate is employed to mitigate redundant information within the multimodal fusion representation, which is subsequently processed by a linear layer for prediction. Simultaneously, to fully leverage limited labelled datasets, we introduced knowledge distillation. This approach involves preserving the model parameters that yield the best performance during training as a teacher model. The teacher model aids in capturing rich emotional information, enabling the model to transcend local optima and discover more optimal parameters, thereby enhancing overall model performance. Extensive experiments on the CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets demonstrate the superiority of our model over state-of-the-art methods in MSA tasks.

Progressive Neighbor-masked Contrastive Learning for Fusion-style Deep Multi-view Clustering

Fusion-style Deep Multi-view Clustering (FDMC) can efficiently integrate comprehensive feature information from latent embeddings of multiple views and has drawn much attention recently. However, existing FDMC methods suffer from the interference of view-specific information for fusion representation, affecting the learning of discriminative cluster structure. In this paper, we propose a new framework of Progressive Neighbor-masked Contrastive Learning for FDMC (PNCL-FDMC) to tackle the aforementioned issues. Specifically, by using neighbor-masked contrastive learning, PNCL-FDMC can explicitly maintain the local structure during the embedding aggregation, which is beneficial to the common semantics enhancement on the fusion view. Based on the consistent aggregation, the fusion view is further enhanced by diversity-aware cluster structure enhancement. In this process, the enhanced cluster assignments and cluster discrepancies are employed to guide the weighted neighbor-masked contrastive alignment of semantic structure between individual views and the fusion view. Extensive experiments validate the effectiveness of the proposed framework, revealing its ability in discriminative representation learning and improving clustering performance.

A novel multimodal depression diagnosis approach utilizing a new hybrid fusion method

In recent years, research has found that the impact of depression status primarily lies in patients’ language expression and facial expressions. Furthermore, facial expressions and intonation in speech exhibit a natural coexistence, making facial and vocal information core recognition indicators in depression identification. It is imperative to explore the effective use of deep learning methods for multimodal depression detection. We have proposed a novel trilateral bimodal encoding model (MEN), attentional decision fusion (ADF), and feature extraction fusion strategy. We employed a hybrid fusion approach that combines early intra-modality fusion with late inter-modality fusion, for multimodal depression diagnosis. In the feature extraction fusion component, we combine different representations of the same modality before inputting them into the network for training, enhancing features relevant to depression in the data. Through our multimodal encoding network, we extract frame-level information using Convolutional Neural Networks (CNN) while considering long-term context information and dependencies with Bidirectional Long Short-Term Memory (BiLSTM). Finally, the three streams of information were effectively integrated through attention fusion representation in our Attention Decision Fusion module (ADF), for depression score regression prediction. Extensive experiments were conducted on two public datasets, AVEC2013 and AVEC2014. The average absolute error/ root mean squared error (MAE/RMSE) scores for predicting depression scores were 6.48/8.91 and 7.01/9.38, respectively. This demonstrated that our hybrid fusion method outperforms traditional early or late fusion methods in terms of performance.

Emotion Recognition From Multimodal Physiological Signals via Discriminative Correlation Fusion With a Temporal Alignment Mechanism.

Modeling correlations between multimodal physiological signals [e.g., canonical correlation analysis (CCA)] for emotion recognition has attracted much attention. However, existing studies rarely consider the neural nature of emotional responses within physiological signals. Furthermore, during fusion space construction, the CCA method maximizes only the correlations between different modalities and neglects the discriminative information of different emotional states. Most importantly, temporal mismatches between different neural activities are often ignored; therefore, the theoretical assumptions that multimodal data should be aligned in time and space before fusion are not fulfilled. To address these issues, we propose a discriminative correlation fusion method coupled with a temporal alignment mechanism for multimodal physiological signals. We first use neural signal analysis techniques to construct neural representations of the central nervous system (CNS) and autonomic nervous system (ANS). respectively. Then, emotion class labels are introduced in CCA to obtain more discriminative fusion representations from multimodal neural responses, and the temporal alignment between the CNS and ANS is jointly optimized with a fusion procedure that applies the Bayesian algorithm. The experimental results demonstrate that our method significantly improves the emotion recognition performance. Additionally, we show that this fusion method can model the underlying mechanisms in human nervous systems during emotional responses, and our results are consistent with prior findings. This study may guide a new approach for exploring human cognitive function based on physiological signals at different time scales and promote the development of computational intelligence and harmonious human-computer interactions.

Modeling refined differences of cortical folding patterns via spatial, morphological, and temporal fusion representations.

The gyrus, a pivotal cortical folding pattern, is essential for integrating brain structure-function. This study focuses on 2-Hinge and 3-Hinge folds, characterized by the gyral convergence from various directions. Existing voxel-level studies may not adequately capture the precise spatial relationships within cortical folding patterns, especially when relying solely on local cortical characteristics due to their variable shapes and homogeneous frequency-specific features. To overcome these challenges, we introduced a novel model that combines spatial distribution, morphological structure, and functional magnetic resonance imaging data. We utilized spatio-morphological residual representations to enhance and extract subtle variations in cortical spatial distribution and morphological structure during blood oxygenation, integrating these with functional magnetic resonance imaging embeddings using self-attention for spatio-morphological-temporal representations. Testing these representations for identifying cortical folding patterns, including sulci, gyri, 2-Hinge, and 2-Hinge folds, and evaluating the impact of phenotypic data (e.g. stimulus) on recognition, our experimental results demonstrate the model's superior performance, revealing significant differences in cortical folding patterns under various stimulus. These differences are also evident in the characteristics of sulci and gyri folds between genders, with 3-Hinge showing more variations. Our findings indicate that our representations of cortical folding patterns could serve as biomarkers for understanding brain structure-function correlations.

Graph-infused hybrid vision transformer: Advancing GeoAI for enhanced land cover classification

Hyperspectral Image Classification (HSIC) is a challenging task due to the high-dimensional nature of Hyperspectral Imaging (HSI) data and the complex relationships between spectral and spatial information. This paper proposes a Graph-Infused Hybrid spatial–spectral Transformer (GFormer) for HSIC. The GFormer combines the power of graph and spatial–spectral transformer to capture both spectral relationships and spatial context. We represent the HSI data as a graph, where nodes represent pixels and edges capture spectral similarities. By incorporating an attention mechanism, the GFormer learns spatial–spectral fusion representations, allowing it to effectively discriminate between different classes. The model can capture long-range dependencies among spectral bands, enabling it to understand complex interactions in the HSI data. Moreover, the GFormer adapts to different spectral resolutions by dynamically adjusting attention weights for each spectral band. Experimental results on benchmark HSI datasets demonstrate that the GFormer outperforms state-of-the-art (SOTA) methods, achieving superior classification accuracy.

DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency

The combination of electronic health records (EHR) and medical images is crucial for clinicians in making diagnoses and forecasting prognoses. Strategically fusing these two data modalities has great potential to improve the accuracy of machine learning models in clinical prediction tasks. However, the asynchronous and complementary nature of EHR and medical images presents unique challenges. Missing modalities due to clinical and administrative factors are inevitable in practice, and the significance of each data modality varies depending on the patient and the prediction target, resulting in inconsistent predictions and suboptimal model performance. To address these challenges, we propose DrFuse to achieve effective clinical multi-modal fusion. It tackles the missing modality issue by disentangling the features shared across modalities and those unique within each modality. Furthermore, we address the modal inconsistency issue via a disease-wise attention layer that produces the patient- and disease-wise weighting for each modality to make the final prediction. We validate the proposed method using real-world large-scale datasets, MIMIC-IV and MIMIC-CXR. Experimental results show that the proposed method significantly outperforms the state-of-the-art models.

Multimodal consistency-specificity fusion based on information bottleneck for sentiment analysis

Sentiment analysis, a subtask of affective computing, endows machines with the ability to sense and comprehend emotions. Recently, research attention has shifted from traditional isolated modality to ubiquitous multi-modalities, requiring to model the complex relationships between modalities and extract the task-relevant information. However, current methods only focus on modality consistency neglecting the complementary relationship and specificity information within each modality. Furthermore, most studies regard multimodal fusion as a mere process of information integration without considering the control of information flow. We propose a variational model that explicitly decomposes unimodal representations into two types, capturing consistency and specificity information, respectively. Following the information bottleneck principle, the implementation optimizes the fusion representation by maximizing its mutual information with consistency and specificity representations while minimizing mutual information with raw inputs. With information integrity constraints and task label supervision, the fusion representation preserves task-relevant information and discards irrelevant noise. Finally, quantitative and qualitative experiments on two benchmark datasets show that our method achieves competitive performance compared to recent baselines, for multimodal sentiment analysis.

A Two-Stage Stacked Transformer Framework for Multimodal Sentiment Analysis

Transformer-based methods have achieved superior performance in multimodal sentiment analysis (MSA). However, recent studies have used it more to focus on the potential mutual adaptation between unimodal modalities, while ignoring intra-modal interactions and complementarity between potential fusion representations. To address this problem, this study proposes a two-stage stacked transformer framework for MSA. The framework decomposes the fusion into two stages, each of which concentrates on a subset of multimodal signals to simultaneously capture the communication information between the unimodal modalities and the interaction information between fusion representations. Further, stacked transformers are the core components of the framework. Two transformer layers are used to model the cross-modal interaction and intra-modal interaction of multimodal input. For cross-modal attention, we propose an attention weight accumulation mechanism to further improve the ability of the framework. Experimental results on 3 benchmark datasets (MOSI [Multimodal Opinion-level Sentiment Intensity], MOSEI [Multimodal Opinion Sentiment and Emotion Intensity], and SIMS) show performance superior or comparable to the state-of-the-art models, thus demonstrating the effectiveness of the proposed framework.

Coupling Local–Nonlocal Feature Representation for SAR and Multispectral Image Fusion

Coupling Local–Nonlocal Feature Representation for SAR and Multispectral Image Fusion

Learning Temporal–Spectral Feature Fusion Representation for Radio Signal Classification

Learning Temporal–Spectral Feature Fusion Representation for Radio Signal Classification

Trial Selection Tensor Canonical Correlation Analysis (TSTCCA) for Depression Recognition with Facial Expression and Pupil Diameter.

Facial expressions have been widely used for depression recognition because it is intuitive and convenient to access. Pupil diameter contains rich emotional information that is already reflected in facial video streams. However, the spatiotemporal correlation between pupillary changes and facial behavior changes induced by emotional stimuli has not been explored in existing studies. This paper presents a novel multimodal fusion algorithm - Trial Selection Tensor Canonical Correlation Analysis (TSTCCA) to optimize the feature space and build a more robust depression recognition model, which innovatively combines the spatiotemporal relevance and complementarity between facial expression and pupil diameter features. TSTCCA explores the interaction between trials and obtains an effective fusion representation of two modalities from a trial subset related to depression. The experimental results show that TSTCCA achieves the highest accuracy of 78.81% with the subset of 25 trials.

DER-GCN: Dialog and Event Relation-Aware Graph Convolutional Neural Network for Multimodal Dialog Emotion Recognition.

With the continuous development of deep learning (DL), the task of multimodal dialog emotion recognition (MDER) has recently received extensive research attention, which is also an essential branch of DL. The MDER aims to identify the emotional information contained in different modalities, e.g., text, video, and audio, and in different dialog scenes. However, the existing research has focused on modeling contextual semantic information and dialog relations between speakers while ignoring the impact of event relations on emotion. To tackle the above issues, we propose a novel dialog and event relation-aware graph convolutional neural network (DER-GCN) for multimodal emotion recognition method. It models dialog relations between speakers and captures latent event relations information. Specifically, we construct a weighted multirelationship graph to simultaneously capture the dependencies between speakers and event relations in a dialog. Moreover, we also introduce a self-supervised masked graph autoencoder (SMGAE) to improve the fusion representation ability of features and structures. Next, we design a new multiple information Transformer (MIT) to capture the correlation between different relations, which can provide a better fuse of the multivariate information between relations. Finally, we propose a loss optimization strategy based on contrastive learning to enhance the representation learning ability of minority class features. We conduct extensive experiments on the benchmark datasets, Interactive Emotional Dyadic Motion Capture (IEMOCAP) and Multimodal EmotionLines Dataset (MELD), which verify the effectiveness of the DER-GCN model. The results demonstrate that our model significantly improves both the average accuracy and the F1 value of emotion recognition. Our code is publicly available at https://github.com/yuntaoshou/DER-GCN.

Preoperative Prediction of Microvascular Invasion in Hepatocellular Carcinoma from Multi-sequence Magnetic Resonance Imaging based on Deep Fusion Representation Learning.

Recent studies have identified microvascular invasion (MVI) as the most vital independent biomarker associated with early tumor recurrence. With advancements in medical technology, several computational methods have been developed to predict preoperative MVI using diverse medical images. These existing methods rely on human experience, attribute selection or clinical trial testing, which is often time-consuming and labor-intensive. Leveraging the advantages of deep learning, this study presents a novel end-to-end algorithm for predicting MVI prior to surgery. We devised a series of data preprocessing strategies to fully extract multi-view features from the data while preserving peritumoral information. Notably, a new multi-branch deep fused feature algorithm based on ResNet (DFFResNet) is introduced, which combines Magnetic Resonance Images (MRI) from different sequences to enhance information complementarity and integration. We conducted prediction experiments on a dataset from the Radiology Department of the First Hospital of Lanzhou University, comprising 117 individuals and seven MRI sequences. The model was trained on 80% of the data using 10-fold cross-validation, and the remaining 20% were used for testing. This evaluation was processed in two cases: CROI, containing samples with a complete region of interest (ROI), and PROI, containing samples with a partial ROI region. The robustness results from repeated experiments at both image and patient levels demonstrate the superior performance and improved generalization of the proposed method compared to alternative models. Our approach yields highly competitive prediction results even when the ROI region outline is incomplete, offering a novel and effective multi-sequence fused strategy for predicting preoperative MVI.