Multi-view Feature Fusion Research Articles

Multi-View Feature Fusion and Rich Information Refinement Network for Semantic Segmentation of Remote Sensing Images

Semantic segmentation is currently a hot topic in remote sensing image processing. There are extensive applications in land planning and surveying. Many current studies combine Convolutional Neural Networks (CNNs), which extract local information, with Transformers, which capture global information, to obtain richer information. However, the fused feature information is not sufficiently enriched and it often lacks detailed refinement. To address this issue, we propose a novel method called the Multi-View Feature Fusion and Rich Information Refinement Network (MFRNet). Our model is equipped with the Multi-View Feature Fusion Block (MAFF) to merge various types of information, including local, non-local, channel, and positional information. Within MAFF, we introduce two innovative methods. The Sliding Heterogeneous Multi-Head Attention (SHMA) extracts local, non-local, and positional information using a sliding window, while the Multi-Scale Hierarchical Compressed Channel Attention (MSCA) leverages bar-shaped pooling kernels and stepwise compression to obtain reliable channel information. Additionally, we introduce the Efficient Feature Refinement Module (EFRM), which enhances segmentation accuracy by interacting the results of the Long-Range Information Perception Branch and the Local Semantic Information Perception Branch. We evaluate our model on the ISPRS Vaihingen and Potsdam datasets. We conducted extensive comparison experiments with state-of-the-art models and verified that MFRNet outperforms other models.

End-to-end multiview fusion for building mapping from aerial images

In the domain of photogrammetry, the fusion of information from multiple views holds the potential to significantly enhance the accuracy and robustness of building mapping. While multiview observation and stereoscopic imaging form the bedrock of photogrammetric projects, current deep learning methodologies predominantly focus on orthophotos and digital surface models (DSMs), often sidelining the rich multiview information inherent in original images. Addressing this gap, we present Multiview Mapper (MVMapper), an end-to-end learning framework explicitly crafted to harness and fuse the rich semantic information from original multiview images with object-space features. MVMapper uses stereo labels for supervised building segmentation in a dual-space, encompassing both image and object domains. Additionally, it incorporates a novel piecewise affine projection method for ensuring a robust image-to-object feature transformation. Experimental results on an aerial photogrammetric dataset with a resolution of 30 cm demonstrate MVMapper’s superiority over state-of-the-art multiview data fusion methods, yielding significant improvements in segmentation and contour accuracy. Notably, the proposed piecewise affine projection method mitigates misalignment issues caused by DSM noise, enabling the effective fusion of multiview features with object-space features. Further experimentation on a separate open-source dataset demonstrates MVMapper’s substantial advantages in transferability to other contexts.

Parallel Hierarchical Multi-View Feature Fusion Based on Canonical Correlation Analysis for Mammogram Retrieval

Due to the diversity of image sources, content-based multi-source image fusion and retrieval have shown promising capabilities in computer vision tasks, and especially when applied in Computer-Aided Diagnosis (CAD) to automate and improve the accuracy of medical image analysis. The combination of computer vision and CAD systems has the potential to revolutionize healthcare by augmenting the expertise of clinicians, improving overall diagnostic accuracy and helping experts in the clinical decision-making process by classifying and retrieving similar annotated clinical images to a given query. In the context of multi-view mammography interpretation, the concept of multi-view feature fusion has recently been studied to improve retrieval performance while effectively guaranteeing the complementarity of both MLO and CC views. However, conventional multi-view feature fusion makes descriptors long and lacks to take into consideration the relationship between descriptors. To deal with this issue, we propose two hierarchical multi-view feature fusion methods, for multi-view mammogram retrieval, based on the Canonical Correlation Analysis (CCA), which is the most commonly used multivariate parametric test. In fact, we have adapted CCA to determine the relationship between two descriptors by processing latent correlation factors. Moreover, after extracting descriptors for each view, a comparative study of texture and shape fusion descriptors is proposed in order to identify the more discriminative features for multi-view mammogram retrieval. Then, a query-dependent distance metric preserving both visual resemblance and semantic similarity is carried out to dynamically determine the more appropriate distance measure for each query image. Extensive experiments on the challenging Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM) have demonstrated the effectiveness of the proposed hierarchical multi-view feature fusion for mammogram retrieval, which outperforms the performance achieved either by conventional fused information or by single view information. To improve the transparency of our paper, the source code of the proposed method and the related dataset (including readme files) are publicly accessible through the following GitHub link: https://github.com/ABDERRAHIMMAR/Multi-View-Feature-Fusion-for-Mammogram-Retrieval . This open-access resource empowers researchers and practitioners to delve deeper into our methodology, fostering collaboration and advancements in the field of computer-aided diagnosis and medical image analysis.

Accurate classification of power quality disturbance based on 3D visualized spiral curve and hybrid ER-MVCNN model

This work proposes a new method for accurate power quality disturbance (PQD) classification. Firstly, 3D-visualized spiral curve (3D-VSC) is innovatively proposed to convert 1D PQD signal into 3D spiral curve. It can display time-frequency information in multi-dimensional perspectives and the disturbance characteristics more intuitively in image visual sense, which can realize PQD features enhancement. Secondly, efficient channel attention-ResNet multi-view convolution neural network (ER-MVCNN) hybrid model is creatively proposed by introducing ECA-ResNet as feature extraction module. It can solve limitations of multi-view convolution neural network (MVCNN) in single-view feature extraction and defects of multi-view feature fusion in the weighting problem, which can effectively achieve classification accuracy to 99.68 %. Comparison experiments with MVCNN, AlexNet-MVCNN, VGGNet-MVCNN, ResNet18-MVCNN, and SENet-ResNet-MVCNN show that ER-MVCNN has higher accuracy by 12.4 %, 6.79 %, 6.14 %, 3.86 %, and 0.88 % respectively. Finally, through establishing PQD experiment platform and IEEE PES datasets to verify high accuracy and robustness of proposed method.

A Multi-View Temporal Knowledge Graph Reasoning Framework with Interpretable Logic Rules and Feature Fusion

A temporal knowledge graph represents temporal information between entities in a multi-relational graph. Its reasoning aims to infer and predict potential links among entities. Predicting time-aware entities is a challenging task due to significant differences in entity appearances over time, such as different timestamps and frequencies. Current embedding-based similarity-matching methods have been introduced for predicting temporal facts. However, they lack deterministic logical explainability and cannot model the dynamic evolution of entities over time. To address these challenges, we propose a novel framework for temporal knowledge graph reasoning based on multi-view feature fusion (MVFF). First, MVFF extracts logical rules and uses the Gumbel-Softmax trick to sample high-quality rules. Second, it integrates logical rules, temporal quadruples, and factual triples to capture semantic features, temporal information, and structural information to solve link prediction tasks. Through experiments on four benchmark datasets, we show that MVFF outperforms state-of-the-art methods, providing not only better performance but also interpretable results.

Research on 3D Animation Scene Plane Design Based on Deep Learning

Abstract At present, most graphic design methods of 3D animation scenes only get a small part of 3D animation scenes, and all of them are in a 3D coordinate system, with observers as the core, so it is difficult to express the depth information of 3D animation scenes. This project intends to study a plane design method for 3D animation scenes based on deep learning. CNN (Convolutional Neural Network) is used to build a multi-view 3D animation scene generation network, and 3D geometry and structure of objects are reconstructed through multiple or a group of images. On this basis, the feature extraction method of 3D animation scenes is studied, and the collaborative learning model of multiple networks is established to improve the modeling accuracy of 3D animation scenes. The experimental results show that the network model is superior to the method based on multi-view and 0-1 voxel in detecting retrieval performance, and the accuracy rate can reach 91.725%. The multi-view 3D animation scene generation method in this paper has achieved better results than the current advanced methods, which proves that the multi-view feature fusion network proposed in this paper is a more reasonable method to fuse multi-view image features.

Bistatic SAR Automatic Target Recognition with Multichannel Multiview Feature Fusion Network

Bistatic SAR Automatic Target Recognition with Multichannel Multiview Feature Fusion Network

AFSleepNet: Attention-based multi-view feature fusion framework for pediatric sleep staging.

The widespread prevalence of sleep problems in children highlights the importance of timely and accurate sleep staging in the diagnosis and treatment of pediatric sleep disorders. However, most existing sleep staging methods rely on one-dimensional raw polysomnograms or two-dimensional spectrograms, which omit critical details due to single-view processing. This shortcoming is particularly apparent in pediatric sleep staging, where the lack of a specialized network fails to meet the needs of precision medicine. Therefore, we introduce AFSleepNet, a novel attention-based multi-view feature fusion network tailored for pediatric sleep analysis. The model utilizes multimodal data (EEG, EOG, EMG), combining one-dimensional convolutional neural networks to extract time-invariant features and bidirectional-long-short-term memory to learn the transition rules among sleep stages, as well as employing short-time Fourier transform to generate two-dimensional spectral maps. This network employs a fusion method with self-attention mechanism and innovative pre-training strategy. This strategy can maintain the feature extraction capabilities of AFSleepNet from different views, enhancing the robustness of the multi-view model while effectively preventing model overfitting, thereby achieving efficient and accurate automatic sleep stage analysis. A "leave-one-subject-out" cross-validation on CHAT and clinical datasets demonstrated the excellent performance of AFSleepNet, with mean accuracies of 87.5% and 88.1%, respectively. Superiority over existing methods improves the accuracy and reliability of pediatric sleep staging.

Spatial Feature Integration in Multidimensional Electromyography Analysis for Hand Gesture Recognition

Enhancing information representation in electromyography (EMG) signals is pivotal for interpreting human movement intentions. Traditional methods often concentrate on specific aspects of EMG signals, such as the time or frequency domains, while overlooking spatial features and hidden human motion information that exist across EMG channels. In response, we introduce an innovative approach that integrates multiple feature domains, including time, frequency, and spatial characteristics. By considering the spatial distribution of surface electromyographic electrodes, our method deciphers human movement intentions from a multidimensional perspective, resulting in significantly enhanced gesture recognition accuracy. Our approach employs a divide-and-conquer strategy to reveal connections between different muscle regions and specific gestures. Initially, we establish a microscopic viewpoint by extracting time-domain and frequency-domain features from individual EMG signal channels. We subsequently introduce a macroscopic perspective and incorporate spatial feature information by constructing an inter-channel electromyographic signal covariance matrix to uncover potential spatial features and human motion information. This dynamic fusion of features from multiple dimensions enables our approach to provide comprehensive insights into movement intentions. Furthermore, we introduce the space-to-space (SPS) framework to extend the myoelectric signal channel space, unleashing potential spatial information within and between channels. To validate our method, we conduct extensive experiments using the Ninapro DB4, Ninapro DB5, BioPatRec DB1, BioPatRec DB2, BioPatRec DB3, and Mendeley Data datasets. We systematically explore different combinations of feature extraction techniques. After combining multi-feature fusion with spatial features, the recognition performance of the ANN classifier on the six datasets improved by 2.53%, 2.15%, 1.15%, 1.77%, 1.24%, and 4.73%, respectively, compared to a single fusion approach in the time and frequency domains. Our results confirm the substantial benefits of our fusion approach, emphasizing the pivotal role of spatial feature information in the feature extraction process. This study provides a new way for surface electromyography-based gesture recognition through the fusion of multi-view features.

Multi-View Feature Fusion Model for Software Bug Repair Pattern Prediction

Many search-based Automatic Program Repair (APR) techniques employ a set of repair patterns to generate candidate patches. Regarding repair pattern selection, existing search-based APR techniques either randomly select a repair pattern from the repair pattern set to apply or prioritize all repair patterns based on the bug's context information. In this paper, we introduce PatternNet, a multi-view feature fusion model capable of predicting the repair pattern for a reported software bug. To accomplish this task, PatternNet first extracts multi-view features from the pair of buggy code and bug report using different models. Specifically, a transformer-based model (i.e., UniXcoder) is utilized to obtain the bimodal feature representation of the buggy code and bug report. Additionally, an Abstract Syntax Tree (AST)-based neural model (i.e., ASTNN) is employed to learn the feature representation of the buggy code. Second, a co-attention mechanism is adopted to capture the dependencies between the statement trees in the AST of the buggy code and the textual tokens of the reported bug, resulting in co-attentive features between statement trees and reported bug's textual tokens. Finally, these multi-view features are combined into a unified representation using a feature fusion network. We quantitatively demonstrate the effectiveness of PatternNet and the feature fusion network for predicting software bug repair patterns.

SMIFormer: Learning Spatial Feature Representation for 3D Object Detection from 4D Imaging Radar via Multi-View Interactive Transformers

4D millimeter wave (mmWave) imaging radar is a new type of vehicle sensor technology that is critical to autonomous driving systems due to its lower cost and robustness in complex weather. However, the sparseness and noise of point clouds are still the main problems restricting the practical application of 4D imaging radar. In this paper, we introduce SMIFormer, a multi-view feature fusion network framework based on 4D radar single-modal input. SMIFormer decouples the 3D point cloud scene into 3 independent but interrelated perspectives, including bird’s-eye view (BEV), front view (FV), and side view (SV), thereby better modeling the entire 3D scene and overcoming the shortcomings of insufficient feature representation capabilities under single-view built from extremely sparse point clouds. For multi-view features, we proposed multi-view feature interaction (MVI) to exploit the inner relationship between different views by integrating features from intra-view interaction and cross-view interaction. We evaluated the proposed SMIFormer on the View-of-Delft (VoD) dataset. The mAP of our method reached 48.77 and 71.13 in the fully annotated area and the driving corridor area, respectively. This shows that 4D radar has great development potential in the field of 3D object detection.

Multi-view feature fusion and density-based minority over-sampling technique for amyloid protein prediction under imbalanced data

Amyloid fibers formed by the aggregation of amyloid proteins can lead to various neurological diseases. Accurate prediction of amyloid proteins can provide scientific basis for exploring the pathogenesis of related neurological diseases, and developing targeted treatment plans. In view of the limitations of the synthetic minority over-sampling technique (SMOTE), an improved SMOTE method based on density clustering is proposed in this paper, and combined with multi-view features to construct an amyloid protein prediction model. The benefits of this enhanced SMOTE method include eliminating problematic samples, accomplishing an evenly distributed minority sample pool, and utilizing all available boundary sample information. Compared with other over-sampling methods, this method can effectively improve the authenticity and representativeness of the synthesized samples, and has a strong ability to process imbalanced data. After the dataset is over-sampling, different prediction algorithms are trained respectively to construct the baseline models. To fully characterize proteins, the probability and category features are extracted from the baseline models and combined with the sequence features selected by CFS-GSSS (Correlation-based Feature Selection Combined with Greedy Stepwise Search Strategy) to generate the multi-view features. Experimental results demonstrate the complementarity and feasibility of the multi-view features. On the independent test set, compared with the existing best model (ECAmyloid), the proposed amyloid protein prediction model improved the sensitivity and geometric mean by 0.0706 and 0.0129, respectively, far superior to other existing methods.

Multi-view feature fusion fault diagnosis method based on an improved temporal convolutional network

This paper addresses the problem of fault identification in rotating machinery by analysing vibration data using a neural network approach. Temporal convolutional networks (TCNs) have attracted a lot of focus in the domain of fault identification; however, TCN convolution kernels are small and susceptible to high-frequency noise interference. Furthermore, the default weight coefficient of the internal residual connection is 1. When there are few residual blocks, the residual block characteristic extraction ability is suppressed and only the vibration signal collected at a single location is utilised for fault diagnosis as it contains incomprehensive fault information. To tackle the above issues, this paper proposes a multi-view feature fusion fault diagnosis algorithm with an adaptive residual coefficient assignment TCN with wide first-layer kernels (WD-ARCATCN). Firstly, a WD-ARCATCN feature extraction network is designed to extract deep state features from different views and the first layer of the TCN is set as a wide-kernel (WD) convolutional layer to suppress high-frequency noise. An adaptive residual coefficient assignment (ARCA) unit is designed in the residual connection to increase the characteristic learning capability of the residual blocks and the residual blocks with ARCA units are stacked to further extract multi-view deep fault features. In this paper, acceleration signals collected at different positions are used as the multi-view feature source for the first time and the fault information contained is more comprehensive. Then, based on a self-attention mechanism, the multi-view feature fusion method is improved and the view weights are adaptively assigned to effectively fuse different view characteristics and enhance the identification of the fault characteristics. Finally, the mapping between the multi-view fusion features and the labels is achieved using a softmax classifier. The algorithm has been tested using experimental data from the bearing vibration database at Case Western Reserve University (CWRU) and it performed much better compared to other diagnostic algorithms.

IoT-Enabled Few-Shot Image Generation for Power Scene Defect Detection Based on Self-Attention and Global-Local Fusion.

Defect detection in power scenarios is a critical task that plays a significant role in ensuring the safety, reliability, and efficiency of power systems. The existing technology requires enhancement in its learning ability from large volumes of data to achieve ideal detection effect results. Power scene data involve privacy and security issues, and there is an imbalance in the number of samples across different defect categories, all of which will affect the performance of defect detection models. With the emergence of the Internet of Things (IoT), the integration of IoT with machine learning offers a new direction for defect detection in power equipment. Meanwhile, a generative adversarial network based on multi-view fusion and self-attention is proposed for few-shot image generation, named MVSA-GAN. The IoT devices capture real-time data from the power scene, which are then used to train the MVSA-GAN model, enabling it to generate realistic and diverse defect data. The designed self-attention encoder focuses on the relevant features of different parts of the image to capture the contextual information of the input image and improve the authenticity and coherence of the image. A multi-view feature fusion module is proposed to capture the complex structure and texture of the power scene through the selective fusion of global and local features, and improve the authenticity and diversity of generated images. Experiments show that the few-shot image generation method proposed in this paper can generate real and diverse defect data for power scene defects. The proposed method achieved FID and LPIPS scores of 67.87 and 0.179, surpassing SOTA methods, such as FIGR and DAWSON.

Delving Deep into Pixel Alignment Feature for Accurate Multi-View Human Mesh Recovery

Regression-based methods have shown high efficiency and effectiveness for multi-view human mesh recovery. The key components of a typical regressor lie in the feature extraction of input views and the fusion of multi-view features. In this paper, we present Pixel-aligned Feedback Fusion (PaFF) for accurate yet efficient human mesh recovery from multi-view images. PaFF is an iterative regression framework that performs feature extraction and fusion alternately. At each iteration, PaFF extracts pixel-aligned feedback features from each input view according to the reprojection of the current estimation and fuses them together with respect to each vertex of the downsampled mesh. In this way, our regressor can not only perceive the misalignment status of each view from the feedback features but also correct the mesh parameters more effectively based on the feature fusion on mesh vertices. Additionally, our regressor disentangles the global orientation and translation of the body mesh from the estimation of mesh parameters such that the camera parameters of input views can be better utilized in the regression process. The efficacy of our method is validated in the Human3.6M dataset via comprehensive ablation experiments, where PaFF achieves 33.02 MPJPE and brings significant improvements over the previous best solutions by more than 29%. The project page with code and video results can be found at https://kairobo.github.io/PaFF/.

Multi-View Joint Learning and BEV Feature-Fusion Network for 3D Object Detection

Traditional 3D object detectors use BEV (bird’s eye view) feature maps to generate 3D object proposals, but in a single BEV feature map, there are inevitably the problems of high compression and information loss. To solve this problem, we propose a multi-view joint learning and BEV feature-fusion network. In this network, we mainly propose two fusion modules: the multi-view feature-fusion module and the multi-BEV feature-fusion module. The multi-view feature fusion module performs joint learning from multiple views, fusing features learned from multiple views, and supplementing missing information in the BEV feature map. The multi-BEV feature-fusion module fuses BEV feature map outputs from different feature extractors to further enrich the feature information in the BEV feature map, in order to generate better quality 3D object proposals. We conducted experiments on a widely used KITTI dataset. The results show that our method has significantly improved the detection accuracy of the cyclist category.For cyclist detection tasks at the easy, moderate, and hard levels of the KITTI test dataset, our method improves by 1.57%, 2.03%, and 0.67%, respectively, compared to PV-RCNN.

Radar Emitter Identification with Multi-View Adaptive Fusion Network (MAFN)

Radar emitter identification (REI) aims to extract the fingerprint of an emitter and determine the individual to which it belongs. Although many methods have used deep neural networks (DNNs) for an end-to-end REI, most of them only focus on a single view of signals, such as spectrogram, bi-spectrum, signal waveforms, and so on. When the electromagnetic environment varies, the performance of DNN will be significantly degraded. In this paper, a multi-view adaptive fusion network (MAFN) is proposed by simultaneously exploring the signal waveform and ambiguity function (AF). First, the original waveform and ambiguity function of the radar signals are used separately for feature extraction. Then, a multi-scale feature-level fusion module is constructed for the fusion of multi-view features from waveforms and AF, via the Atrous Spatial Pyramid Pooling (ASPP) structure. Next, the class probability is modeled as Dirichlet distribution to perform adaptive decision-level fusion via evidence theory. Extensive experiments are conducted on two datasets, and the results show that the proposed MAFN can achieve accurate classification of radar emitters and is more robust than its counterparts.

Multi-view feature representation and fusion for drug-drug interactions prediction

BackgroundDrug-drug interactions (DDIs) prediction is vital for pharmacology and clinical application to avoid adverse drug reactions on patients. It is challenging because DDIs are related to multiple factors, such as genes, drug molecular structure, diseases, biological processes, side effects, etc. It is a crucial technology for Knowledge graph to present multi-relation among entities. Recently some existing graph-based computation models have been proposed for DDIs prediction and get good performance. However, there are still some challenges in the knowledge graph representation, which can extract rich latent features from drug knowledge graph (KG).ResultsIn this work, we propose a novel multi-view feature representation and fusion (MuFRF) architecture to realize DDIs prediction. It consists of two views of feature representation and a multi-level latent feature fusion. For the feature representation from the graph view and KG view, we use graph isomorphism network to map drug molecular structures and use RotatE to implement the vector representation on bio-medical knowledge graph, respectively. We design concatenate-level and scalar-level strategies in the multi-level latent feature fusion to capture latent features from drug molecular structure information and semantic features from bio-medical KG. And the multi-head attention mechanism achieves the optimization of features on binary and multi-class classification tasks. We evaluate our proposed method based on two open datasets in the experiments. Experiments indicate that MuFRF outperforms the classic and state-of-the-art models.ConclusionsOur proposed model can fully exploit and integrate the latent feature from the drug molecular structure graph (graph view) and rich bio-medical knowledge graph (KG view). We find that a multi-view feature representation and fusion model can accurately predict DDIs. It may contribute to providing with some guidance for research and validation for discovering novel DDIs.

A Protein-DNA Binding Site Prediction Method Based on Multi-View Feature Fusion of Adjacent Residue

A Protein-DNA Binding Site Prediction Method Based on Multi-View Feature Fusion of Adjacent Residue

Multiview Feature Fusion Attention Convolutional Recurrent Neural Networks for EEG‐Based Emotion Recognition

Emotion recognition is essential for computers to understand human emotions. Traditional EEG emotion recognition methods have significant limitations. To improve the accuracy of EEG emotion recognition, we propose a multiview feature fusion attention convolutional recurrent neural network (multi‐aCRNN) model. Multi‐aCRNN combines CNN, GRU, and attention mechanisms to fuse features from multiple perspectives deeply. Specifically, multiscale CNN can unite elements in the frequency and spatial domains through the convolution of different scales. The role of the attention mechanism is to weigh the frequency domain and spatial domain information of different periods to find more valuable temporal perspectives. Finally, the implicit feature representation is learned from the time domain through the bidirectional GRU to achieve the profound fusion of features from multiple perspectives in the time domain, frequency domain, and spatial domain. At the same time, for the noise problem, we use label smoothing to reduce the influence of label noise to achieve a better emotion recognition classification effect. Finally, the model is validated on the EEG data of 32 subjects on a public dataset (DEAP) by fivefold cross‐validation. Multi‐aCRNN achieves an average classification accuracy of 96.43% and 96.30% in arousal and valence classification tasks, respectively. In conclusion, multi‐aCRNN can better integrate EEG features from different angles and provide better classification results for emotion recognition.