Hierarchical Fusion Research Articles

HF2TNet: A Hierarchical Fusion Two-stage Training Network for Infrared and Visible Image Fusion

HF2TNet: A Hierarchical Fusion Two-stage Training Network for Infrared and Visible Image Fusion

HiSIF-DTA: A Hierarchical Semantic Information Fusion Framework for Drug-Target Affinity Prediction.

Accurately identifying drug-target affinity (DTA) plays a significant role in promoting drug discovery and has attracted increasing attention in recent years. Exploring appropriate protein representation methods and increasing the abundance of protein information is critical in enhancing the accuracy of DTA prediction. Recently, numerous deep learning-based models have been proposed to utilize the sequential or structural features of target proteins. However, these models capture only the low-order semantics that exist in a single protein, while the high-order semantics abundant in biological networks are largely ignored. In this article, we propose HiSIF-DTA'a hierarchical semantic information fusion framework for DTA prediction. In this framework, a hierarchical protein graph is constructed that includes not only contact maps as low-order structural semantics but also protein-rotein interaction (PPI) networks as high-order functional semantics. Particularly, two distinct hierarchical fusion strategies (i.e., Top-down and Bottom-Up) are designed to integrate the different protein semantics, therefore contributing to a richer protein representation. Comprehensive experimental results demonstrate that HiSIF-DTA outperforms current state-of-the-art methods for prediction on the benchmark datasets of the DTA task. Further validation on binary tasks and visualization analysis demonstrates the generalization and interpretation abilities of the proposed method.

Automated Diagnosis of Major Depressive Disorder With Multi-Modal MRIs Based on Contrastive Learning: A Few-Shot Study.

Depression ranks among the most prevalent mood-related psychiatric disorders. Existing clinical diagnostic approaches relying on scale interviews are susceptible to individual and environmental variations. In contrast, the integration of neuroimaging techniques and computer science has provided compelling evidence for the quantitative assessment of major depressive disorder (MDD). However, one of the major challenges in computer-aided diagnosis of MDD is to automatically and effectively mine the complementary cross-modal information from limited datasets. In this study, we proposed a few-shot learning framework that integrates multi-modal MRI data based on contrastive learning. In the upstream task, it is designed to extract knowledge from heterogeneous data. Subsequently, the downstream task is dedicated to transferring the acquired knowledge to the target dataset, where a hierarchical fusion paradigm is also designed to integrate features across inter- and intra-modalities. Lastly, the proposed model was evaluated on a set of multi-modal clinical data, achieving average scores of 73.52% and 73.09% for accuracy and AUC, respectively. Our findings also reveal that the brain regions within the default mode network and cerebellum play a crucial role in the diagnosis, which provides further direction in exploring reproducible biomarkers for MDD diagnosis.

Indoor Point Cloud Segmentation Based on Hierarchical Fusion of Structural Features

Indoor Point Cloud Segmentation Based on Hierarchical Fusion of Structural Features

HFA-GTNet: Hierarchical Fusion Adaptive Graph Transformer network for dance action recognition

Dance action recognition is a hot research topic in computer vision. However, current skeleton-based action recognition methods face difficulties in capturing the adequate spatial structure and temporal variations of dance actions, resulting in lower recognition accuracy. In this paper, we propose a Hierarchical Fusion Adaptive Graph Transformer network (HFA-GTNet) for dance action recognition. A Hierarchical Spatial Attention (HSAtt) module is designed to extract different levels of spatial feature information from joint to parts to group, it can effectively learn high-order dependency relationships from local joints to global poses in dance actions. Secondly, to extract the joint variations in dance actions at different speeds, we have designed a Temporal Fusion Attention (TFAtt) module. This module learns the short-term and long-term temporal dependencies among joints across frames. Additionally, to capture the variations in motion patterns and dance styles among different dancers, we introduce an Adaptive Component (AdaptC). Finally, we evaluate our model on two self-built dance datasets, MSDanceAction and InDanceAction, and demonstrate its superior performance compared to other state-of-the-art methods in dance action recognition.

Multimodal Mutual Attention-Based Sentiment Analysis Framework Adapted to Complicated Contexts

Sentiment analysis has broad application prospects in the field of social opinion mining. The openness and invisibility of the internet makes users’ expression styles more diverse and thus results in the blooming of complicated contexts in which different unimodal data have inconsistent sentiment tendencies. However, most sentiment analysis algorithms only focus on designing multimodal fusion methods without preserving the individual semantics of each unimodal data. To avoid misunderstandings caused by ambiguity and sarcasm in complicated contexts, we propose a multimodal mutual attention-based sentiment analysis (MMSA) framework adapted to complicated contexts, which consists of three levels of subtasks to preserve the unimodal unique semantics and enhance the common semantics, to mine the association between unique semantics and common semantics and to balance decisions from unique and common semantics. In the framework, a multiperspective and hierarchical fusion (MHF) module is developed to fully fuse multimodal data, in which different modalities are mutually constrained and the fusion order is adjusted in the next step to enhance cross-modal complementarity. To balance the data, we calculate the loss by applying different weights to positive and negative samples. The experimental results on the CH-SIMS multimodal dataset show that our method outperforms existing multimodal sentiment analysis algorithms.The code of this work is available at https://gitee.com/viviziqing/mmsacode.

Space-time multi-level modeling for zooplankton abundance employing double data fusion and calibration

An important objective for marine biologists is to forecast the distribution and abundance of planktivorous marine predators. To do so, it is critically important to understand the spatiotemporal dynamics of their prey. Here, the prey we study are zooplankton and we build a novel space-time hierarchical fusion model to describe the distribution and abundance of zooplankton species in Cape Cod Bay (CCB), MA, USA. The data were collected irregularly in space and time at sites within the first half of the year over a 17 year period, using two different sampling methods. We focus on sea surface zooplankton abundance and incorporate sea surface temperature as a primary driver, also collected with two different sampling methods. So, with two sources for each, we observe true abundance or true sea surface temperature with measurement error. To account for such error, we apply calibrations to align the data sources and use the fusion model to develop a prediction of daily spatial zooplankton abundance surfaces throughout CCB. To infer average abundance on a given day within a given year in CCB, we present a marginalization of the zooplankton abundance surface. We extend the inference to consider abundance averaged to a bi-weekly or annual scale as well as to make an annual comparison of abundance.

Hierarchical multimodal-fusion of physiological signals for emotion recognition with scenario adaption and contrastive alignment

The lack of complementary affective responses from both the central and peripheral nervous systems could limit the performance of emotion recognition with the single-modal physiological signal. However, when integrating multimodalities, a direct fusion may ignore the heterogeneous nature of multiple feature domains from one modality to another. Besides, there is a risk that the distribution of the multimodal physiological responses may vary across different affective scenarios for stimulating an identical emotional category. The inter-individual variation may also increase due to the superposition of the biometric information from the multimodal features. To tackle these issues, we present a hierarchical multimodal network for robust heterogeneous physiological representations (RHPRNet). First, we applied a spatial-frequency pattern extractor to identify the electroencephalogram (EEG) representations in both the spatial and frequency domains. Next, inter-domain and inter-modality affective encoders are separately applied to the statistic-complexity EEG features and multimodal peripheral features, respectively. All the learned representations are integrated via a hierarchical fusion module. To model the multi-peak patterns stimulated by different affective scenarios, we designed a scenario-adapting pretraining stage. A random contrastive training loss was also applied to mitigate the inter-individual variance. In the end, we performed adequate experiments to examine the performance of the RHPRNet based on three publicly available multimodal databases combined with two validation approaches.

Intangible cultural heritage image classification with multimodal attention and hierarchical fusion

Designing an efficient Intangible Cultural Heritage (ICH) image classification is beneficial for the public to recognize the ICH and fostering the preservation and spread of that. Currently, related ICH image classification researches mainly focus on the visual features of ICH images, ignoring attached textual descriptions. However, attached textual descriptions can provide crucial clues for ICH images classification. Therefore, in this study, we propose to combine attached textual descriptions to perform ICH image classification in a multimodal way. Additionally, to capture intra- and inter-interactions between ICH images and attached textual descriptions, we propose a novel model named MICMLF, mainly consisted of multimodal attention and hierarchical fusion. Multimodal attention is employed to make the model focus on “important regions” and “important words” in ICH image and attached textual descriptions respectively. Hierarchical fusion is utilized to capture inter-modal dynamics interactions. Extensive experiments are conducted on two Chinese nation-level ICH lists datasets, New Year Print (年画) and Clay Figurine (泥塑). Experimental results demonstrate the superiority of MICMLF, compared with several state-of-the-art methods. Also, the proposed model can handle the situation where ICH images and textual descriptions are incomplete. To the best of our knowledge, we are the first to propose to combine textual descriptions to perform ICH image classification in a multimodal way.

Hierarchical fusion evaluation and optimization of radar intelligent tracking algorithm via hybrid weight design mechanism

In this paper, a hybrid weight design and optimization mechanism is proposed to ensure a better application effect in the multi-level weighted evaluation/optimization process of intelligent neural network (NN) guidance algorithm. According to different target tracking scenarios, the tracking index are divided into different periods, a single period score is given according to a linear-nonlinear hybrid scoring mechanism. Furthermore, in a single index, the internal scores of different time periods are weighted and fused for the second time to obtain the index overall score. Finally, the third weighted fusion of the multi-index scores is used to obtain the comprehensive score of the algorithm. Design the algorithm parameter evaluation case sets, repeat the above compound weighting, obtain the case with the highest comprehensive score. Finally, the algorithm is optimized by the highest score case. In the hybrid weight design mechanism, the objective grey correlation first defines the weight range, then the subjective expert decision method with local extreme expert elimination and standard deviation global test for double optimization finally ensures the weight reliability of the hierarchical evaluation optimization. The evaluation optimization experiment using fuzzy NN radar seeker verifies the effectiveness of the method.

A joint hierarchical cross‐attention graph convolutional network for multi‐modal facial expression recognition

AbstractEmotional recognition in conversations (ERC) is increasingly being applied in various IoT devices. Deep learning‐based multimodal ERC has achieved great success by leveraging diverse and complementary modalities. Although most existing methods try to adopt attention mechanisms to fuse different information, these methods ignore the complementarity between modalities. To this end, the joint cross‐attention model is introduced to alleviate this issue. However, multi‐scale feature information on different modalities is not utilized. Moreover, the context relationship plays an important role in feature extraction in the expression recognition task. In this paper, we propose a novel joint hierarchical graph convolution network (JHGCN) which exploits different layer features and context relationships for facial expression recognition based on audio‐visual (A‐V) information. Specifically, we adopt different deep networks to extract features from different modalities individually. For V modality, we construct V graph data based on patch embeddings which are extracted from the transformer encoder. Moreover, we embed the graph convolution which can leverage the intra‐modality relationships with the transformer encoder. Then, the deep feature from different layers is fed to the hierarchical fusion module to enhance feature representation. At last, we use the joint cross‐attention mechanism to exploit the complementary inter‐modality relationships. To validate the proposed model, we have conducted various experiments on the AffWild2 and CMU‐MOSI datasets. All results confirm that our proposed model achieves highly promising performance compared to the joint cross‐attention model and other methods.

MIA-Net: Multi-Modal Interactive Attention Network for Multi-Modal Affective Analysis

When a multi-modal affective analysis model generalizes from a bimodal task to a trimodal or multi-modal task, it is usually transformed into a hierarchical fusion model based on every two pairwise modalities, similar to a binary tree structure. This easily leads to large growth in model parameters and computation as the number of modalities increases, which limits the model's generalization. Moreover, many multi-modal fusion methods ignore that different modalities contribute differently to affective analysis. To tackle these challenges, this paper proposes a general multi-modal fusion model that supports trimodal or multi-modal affective analysis tasks, called Multi-modal Interactive Attention Network (MIA-Net). Instead of treating different modalities equally, MIA-Net takes the modality that contributes the most to emotion as the main modality and the others as auxiliary modalities. MIA-Net introduces multi-modal interactive attention modules to adaptively select the important information of each auxiliary modality one by one to improve the main-modal representation. Moreover, MIA-Net enables quick generalization to trimodal or multi-modal tasks through stacking multiple MIA modules, which maintains efficient training and only requires linear computation and stable parameter counts. Experimental results of the transfer, generalization, and efficiency experiments on the widely-used datasets demonstrate the effectiveness and generalization of the proposed method.

Knowledge Graph Augmented Network Towards Multiview Representation Learning for Aspect-Based Sentiment Analysis

Aspect-based sentiment analysis (ABSA) is a fine-grained task of sentiment analysis. To better comprehend long complicated sentences and obtain accurate aspect-specific information, linguistic and commonsense knowledge are generally required in this task. However, most current methods employ complicated and inefficient approaches to incorporate external knowledge, e.g., directly searching the graph nodes. Additionally, the complementarity between external knowledge and linguistic information has not been thoroughly studied. To this end, we propose a knowledge graph augmented network KGAN, which aims to effectively incorporate external knowledge with explicitly syntactic and contextual information. In particular, KGAN captures the sentiment feature representations from multiple different perspectives, i.e., context-, syntax- and knowledge-based. First, KGAN learns the contextual and syntactic representations in parallel to fully extract the semantic features. Then, KGAN integrates the knowledge graphs into the embedding space, based on which the aspect-specific knowledge representations are further obtained via an attention mechanism. Last, we propose a hierarchical fusion module to complement these multi-view representations in a local-to-global manner. Extensive experiments on five popular ABSA benchmarks demonstrate the effectiveness and robustness of our KGAN. Notably, with the help of the pretrained model of RoBERTa, KGAN achieves a new record of state-of-the-art performance among all datasets.

Full stage networks with auxiliary focal loss and multi-attention module for submarine garbage object detection

Submarine garbage is constantly destroying the marine ecological environment and polluting the ocean. It is critical to use detection methods to quickly locate and identify submarine garbage. The background of submarine garbage images is much more complex than that of natural scene images, with object deformation and missing contours putting higher demands on the detection network. To solve the problem of low accuracy under complex backgrounds, full stage networks with auxiliary focal loss and multi-attention module are proposed for submarine garbage object detection based on YOLO. To maximize the gradient combination, a hierarchical fusion feature mechanism and a segmentation and merging strategy are used in this paper to optimize the difference in gradient combination to obtain full-stage features. Then the criss-cross attention module is used to precisely extract multi-scale features of small object dense regions while removing noise information from complex backgrounds. Finally, the auxiliary focal loss function addresses the issue of unbalanced positive and negative samples, focusing on the learning of difficult samples while improving overall detection precision. Based on comparative experiments and ablation experiments, the FSA networks achieved state-of-the-art performance, and is applicable to the real-time object detection of submarine garbage in complex backgrounds.

Multi-modal hierarchical fusion network for fine-grained paper classification

Multi-modal hierarchical fusion network for fine-grained paper classification

YOLACTFusion: An instance segmentation method for RGB-NIR multimodal image fusion based on an attention mechanism

The tomato plant’s main-stem is a feasible lead for robotic searching the grows discretely-growing targets of harvesting, pruning or pollinating. Owing to the highlighted reflection characteristics of the main-stem in the near-infrared (NIR) waveband, this study proposes a multimodal hierarchical fusion method (YOLACTFusion) based on the attention mechanism, to achieve an instance segmentation of the main-stem from similar-colored differentiation (i.e., green leaf and green fruit) in robotic vision systems. The model inputs RGB images and 900–1100 nm NIR images into two ResNet50 backbone networks and uses a parallel attention mechanism to fuse feature maps of various scales together into the head network, to improve the segmentation performance of the main-stem of RGB images. The loss function for the multimodal image weights the original loss on the RGB image and the position offset loss and classification loss on the NIR image. Furthermore, the local depthwise separable convolution is used for the backbone network, and Conv-BN layers are merged to reduce the computational complexity. The results show that the precision and recall of YOLACTFusion of the main-stem detection, respectively reached 93.90 % and 62.60 %; and the precision and recall of instance segmentation reached 95.12 % and 63.41 %, respectively. Compared to YOLACT, the mean average precision (mAP) of YOLACTFusion is increased from 39.20 % to 46.29 %, the model size is reduced from 199.03 MB to 165.52 MB, while the image processing efficiency remains similar. The overall results show that the multimodal instance segmentation method proposed in this study significantly improves the detection and segmentation of tomato main-stems under a similar-colored background, which would be a potential method for improving agricultural robot’s visual perception.

Hierarchical Fusion Network with Enhanced Knowledge and Contrastive Learning for Multimodal Aspect-Based Sentiment Analysis on Social Media.

Aspect-based sentiment analysis (ABSA) is a task of fine-grained sentiment analysis that aims to determine the sentiment of a given target. With the increased prevalence of smart devices and social media, diverse data modalities have become more abundant. This fuels interest in multimodal ABSA (MABSA). However, most existing methods for MABSA prioritize analyzing the relationship between aspect-text and aspect-image, overlooking the semantic gap between text and image representations. Moreover, they neglect the rich information in external knowledge, e.g., image captions. To address these limitations, in this paper, we propose a novel hierarchical framework for MABSA, known as HF-EKCL, which also offers perspectives on sensor development within the context of sentiment analysis. Specifically, we generate captions for images to supplement the textual and visual features. The multi-head cross-attention mechanism and graph attention neural network are utilized to capture the interactions between modalities. This enables the construction of multi-level aspect fusion features that incorporate element-level and structure-level information. Furthermore, for this paper, we integrated modality-based and label-based contrastive learning methods into our framework, making the model learn shared features that are relevant to the sentiment of corresponding words in multimodal data. The results, based on two Twitter datasets, demonstrate the effectiveness of our proposed model.

Multimodal Sentiment Analysis in Realistic Environments Based on Cross-Modal Hierarchical Fusion Network

In the real world, multimodal sentiment analysis (MSA) enables the capture and analysis of sentiments by fusing multimodal information, thereby enhancing the understanding of real-world environments. The key challenges lie in handling the noise in the acquired data and achieving effective multimodal fusion. When processing the noise in data, existing methods utilize the combination of multimodal features to mitigate errors in sentiment word recognition caused by the performance limitations of automatic speech recognition (ASR) models. However, there still remains the problem of how to more efficiently utilize and combine different modalities to address the data noise. In multimodal fusion, most existing fusion methods have limited adaptability to the feature differences between modalities, making it difficult to capture the potential complex nonlinear interactions that may exist between modalities. To overcome the aforementioned issues, this paper proposes a new framework named multimodal-word-refinement and cross-modal-hierarchy (MWRCMH) fusion. Specifically, we utilized a multimodal word correction module to reduce sentiment word recognition errors caused by ASR. During multimodal fusion, we designed a cross-modal hierarchical fusion module that employed cross-modal attention mechanisms to fuse features between pairs of modalities, resulting in fused bimodal-feature information. Then, the obtained bimodal information and the unimodal information were fused through the nonlinear layer to obtain the final multimodal sentiment feature information. Experimental results on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets demonstrated that the proposed approach outperformed other comparative methods, achieving Has0-F1 scores of 76.43%, 80.15%, and 81.93%, respectively. Our approach exhibited better performance, as compared to multiple baselines.

FURSformer: Semantic Segmentation Network for Remote Sensing Images with Fused Heterogeneous Features

Semantic segmentation of remote sensing images poses a formidable challenge within this domain. Our investigation commences with a pilot study aimed at scrutinizing the advantages and disadvantages of employing a Transformer architecture and a CNN architecture in remote sensing imagery (RSI). Our objective is to substantiate the indispensability of both local and global information for RSI analysis. In this research article, we harness the potential of the Transformer model to establish global contextual understanding while incorporating an additional convolution module for localized perception. Nonetheless, a direct fusion of these heterogeneous information sources often yields subpar outcomes. To address this limitation, we propose an innovative hierarchical fusion feature information module that this model can fuse Transformer and CNN features using an ensemble-to-set approach, thereby enhancing information compatibility. Our proposed model, named FURSformer, amalgamates the strengths of the Transformer architecture and CNN. The experimental results clearly demonstrate the effectiveness of this approach. Notably, our model achieved an outstanding accuracy of 90.78% mAccuracy on the DLRSD dataset.

STATE: Learning structure and texture representations for novel view synthesis

Novel viewpoint image synthesis is very challenging, especially from sparse views, due to large changes in viewpoint and occlusion. Existing image-based methods fail to generate reasonable results for invisible regions, while geometry-based methods have difficulties in synthesizing detailed textures. In this paper, we propose STATE, an end-to-end deep neural network, for sparse view synthesis by learning structure and texture representations. Structure is encoded as a hybrid feature field to predict reasonable structures for invisible regions while maintaining original structures for visible regions, and texture is encoded as a deformed feature map to preserve detailed textures. We propose a hierarchical fusion scheme with intra-branch and inter-branch aggregation, in which spatio-view attention allows multi-view fusion at the feature level to adaptively select important information by regressing pixel-wise or voxel-wise confidence maps. By decoding the aggregated features, STATE is able to generate realistic images with reasonable structures and detailed textures. Experimental results demonstrate that our method achieves qualitatively and quantitatively better results than state-of-the-art methods. Our method also enables texture and structure editing applications benefiting from implicit disentanglement of structure and texture. Our code is available at http://cic.tju.edu.cn/faculty/likun/projects/STATE.