Multi-view Feature Fusion Research Articles

Cattle Facial Matching Recognition Algorithm Based on Multi-View Feature Fusion

In the process of collecting facial images of cattle in the field, some features of the collected images end up going missing due to the changeable posture of the cattle, which makes the recognition accuracy decrease or impossible to recognize. This paper verifies the practical effects of the classical matching algorithms ORB, SURF, and SIFT in bull face matching recognition. The experimental results show that the traditional matching algorithms perform poorly in terms of matching accuracy and matching time. In this paper, a new matching recognition model is constructed. The model inputs the target cattle facial data from different angles into the feature extraction channel, combined with GMS (grid-based motion statistics) algorithm and random sampling consistent algorithm, to achieve accurate recognition of individual cattle, and the recognition process is simple and fast. The recognition accuracy of the model was 85.56% for the Holstein cow face dataset, 82.58% for the Simmental beef cattle, and 80.73% for the mixed Holstein and Simmental beef cattle dataset. The recognition model constructed in the study can achieve individual recognition of cattle in complex environments, has good robustness to matching data, and can effectively reduce the effects of data angle changes and partial features missing in cattle facial recognition.

A multi-view feature fusion approach for effective malware classification using Deep Learning

The number of malware infected machines from all over the world has been growing day by day. New malware variants appear in the wild to evade the malware detection and classification systems and may infect with ransomware or crypto miners for adversary financial gain. A recent colonial pipeline ransomware attack is an example of these attacks that impacted daily human activities, and the victim had to pay ransom to restore their operations. Windows-based systems are the most adopted systems across different industries for running applications. They are prone to get targeted by installing the malware. In this paper, we propose a Deep Learning (DL)-based Convolutional Neural Network (CNN) model to perform the malware classification on Portable Executable (PE) binary files using the fusion feature set approach. We present an extensive performance evaluation of various DL model architecture and Machine Learning (ML) classifier i.e. Support Vector Machine (SVM), on multi-aspect feature sets covering the static, dynamic, and image features to select the proposed CNN model. We further leverage the CNN-based architecture for effective classification of the malware using different combinations of feature sets and compare the results with the best-performed individual feature set. Our performance evaluation of the proposed model shows that the model classifies the malware or benign files with an accuracy of 97% when using fusion feature sets. The proposed model is robust and generalizable and showed similar performances on completely unseen two malware datasets. In addition, the embedding features of the CNN model are visualized, and various visualization methods are employed to understand the characteristics of the datasets. Further, large-scale learning and stacked classifiers were employed after the penultimate layer to enhance the CNN classification performance.

Cascaded multi-3D-view fusion for 3D-oriented object detection

Currently, multi-view fusion methods fuse point- or proposal-level features from different views at the end stage of the backbone. This once-end-fusion method is not conducive to the timely adjustment of spatial misalignment for features from different views. Consequently, the discriminative depth and orientation details of the 3D oriented point cloud object may be filtered. To enhance the feature capture capability of the network, we introduce a cascaded multi-3D-view fusion method (CM3DV) to learn the implicit representation of object orientation. In particular, the proposed CM3DV method incorporates the cylindrical front view projection into a voxelised 3D bird’s-eye-view representation in a cascaded manner, and vice versa. Through the learning of 3D-regulated instance representation, this bi-directional mutual fusion module, called cascaded multi-view feature fusion module, alleviates the spatial misalignment of the two views. Furthermore, to learn the rotation- and shape-invariant features of objects, modulated rotation head (MRH) develops a direction-guided adjustment instead of an axis-aligned structure to extract instance-consistent features. By excluding the irrelevant content using MRH, this instance-consistent feature will benefit the object classification and orientation regression. Extensive experiments on the KITTI dataset show that the proposed method achieves a significant improvement over existing advanced methods, especially for orientation estimation.

Semantic Segmentation Algorithm of Rice Small Target Based on Deep Learning

At present, rice is generally in a state of dense adhesion and small granular volume during processing, resulting in no effective semantic segmentation method for rice to extract complete rice. Aiming at the above problems, this paper designs a small object semantic segmentation network model based on multi-view feature fusion. The overall structure of the network is divided into a multi-view feature extraction module, a super-resolution feature building module and a semantic segmentation module. The extraction ability of small target features is improved by super-resolution construction of small target detail features, and the learning ability of the network for small target features is enhanced and expanded through multi-view. At the same time, a dataset of quality inspection during rice processing was constructed. We train and test the model on this dataset. The results show that the average segmentation accuracy of the semantic segmentation model in this paper reaches 87.89%. Compared with the semantic segmentation models such as SegNet, CBAM, RefineNet, DeepLabv3+ and G-FRNet, it has obvious advantages in various indicators, which can provide rice quality detection and an efficient method of rice grain extraction.

ADHD Children Identification With Multiview Feature Fusion of fNIRS Signals

Attention-deficit/hyperactivity disorder (ADHD) is one of highly prevalent neurodevelopmental disorders in children. It is of great value to find an objective, simple and non-invasive diagnosis method to compliment current diagnostic criteria. A discriminant correlation analysis (DCA) based multi-view feature fusion method combined with support vector machine (SVM) was proposed to identify ADHD children. The functional near-infrared spectroscopy signals from twenty-five ADHD children and twenty-five typically developing children were recorded during an n-back task. The DCA method was used to find a joint feature by fusing the features extracted in 1-back condition and 0-back condition. Then a SVM model was trained with the joint feature to identify the ADHD children. The results showed that the DCA-SVM method could identify the ADHD children with a classification accuracy of 88.0%. Meanwhile, part of left dorsolateral prefrontal cortex and left medial prefrontal cortex, posterior superior frontal cortex and bilateral temporal cortex were more discriminative in 1-back condition, left dorsolateral prefrontal cortex and part of left medial prefrontal cortex were more discriminative in 0-back condition. Our findings indicated that the DCA-SVM method is promising in clinical diagnosis of ADHD children.

MSFM: Multi-view Semantic Feature Fusion Model for Chinese Named Entity Recognition

MSFM: Multi-view Semantic Feature Fusion Model for Chinese Named Entity Recognition

Fault detection of overhead contact systems based on multi-view Faster R-CNN

Overhead contact systems are important power supply equipments for electrified railway locomotives. If there are faults in the equipment, it will threaten the safety of train operation. Dropper faults and foreign matter are two common types of Overhead contact systems faults, the fault regions of which occupy a relatively small area in the whole image, so it is difficult to detect the faults accurately by capturing detailed features using original Faster R-CNN. To solve the above problem, an improved Faster R-CNN based on convolutional neural networks, named multi-view Faster R-CNN, is proposed in order to extract more details of the fault regions by combining deep and shallow feature maps. Experiments on the images collected from the Lanzhou-Xinjiang Railway line show that the feature fusion can significantly enhance the mean average precision. The precision of our model on the test set was 89.53%, increasing by 3.5%, 9.65%, and 5.8% compared with YOLOv3, SSD, and Faster R-CNN, and the detection accuracies of our model for unforced droppers and foreign matter were 88.17% and 90.60%, respectively, under a recall rate of almost 1.0. Because the multi-view feature fusion model in our method can flexibly detect faults with various sizes, it has an important application value in the fault detection of an overhead contact system.

An Enhanced Intrusion Detection System for IoT Networks Based on Deep Learning and Knowledge Graph

Nowadays, the intrusion detection system (IDS) plays a crucial role in the Internet of Things (IoT) networks, which could effectively protect sensitive data from various attacks. However, the existing works have not considered multiview features fusion and failed to capture the semantic relationships among the anomalous requests. They are not robust and cannot detect the attack types in real-time. This paper proposes a lightweight intrusion detection system based on deep learning and knowledge graph. First, our system extracts semantic relationships and key features by knowledge graph and statistical analysis. Then, IoT network requests are converted into word vectors through multiview feature fusion and feature alignment. Finally, an attention-based CNN-BiLSTM model is designed to identify malicious request attacks, which can capture long-distance dependence and contextual semantic information. Experiment results show that the proposed model significantly outperforms the existing solution in the robustness of the model. Moreover, it can select more critical features for IDS to achieve better accuracy and lower the false alarm rate. Compared with the state-of-the-art systems, the proposed IDS achieves a higher detection accuracy of 90.01%. In addition, our system can detect various stealthy attack types (including DoS, Probe, R2L, and U2L) and extract semantic relationships among features.

Compositional action recognition with multi-view feature fusion.

Most action recognition tasks now treat the activity as a single event in a video clip. Recently, the benefits of representing activities as a combination of verbs and nouns for action recognition have shown to be effective in improving action understanding, allowing us to capture such representations. However, there is still a lack of research on representational learning using cross-view or cross-modality information. To exploit the complementary information between multiple views, we propose a feature fusion framework, and our framework is divided into two steps: extraction of appearance features and fusion of multi-view features. We validate our approach on two action recognition datasets, IKEA ASM and LEMMA. We demonstrate that multi-view fusion can effectively generalize across appearances and identify previously unseen actions of interacting objects, surpassing current state-of-the-art methods. In particular, on the IKEA ASM dataset, the performance of the multi-view fusion approach improves 18.1% over the performance of the single-view approach on top-1.

Dual-attention EfficientNet based on multi-view feature fusion for cervical squamous intraepithelial lesions diagnosis

Cervicograms are widely used in cervical cancer screening but exhibit a high misdiagnosis rate. Even senior experts show only 48% specificity on clinical examinations. Most existing methods only use single-view images applied with acetic acid or Lugol’s iodine solution as their input data, ignoring the fact that non-pathological tissues may show false-positive reactions in these single-view images. This can lead to misdiagnosis in clinical diagnosis. Therefore, it is essential to extract features from multi-view colposcopy images (including the original images) as inputs, because three-view cervicograms provide complementary information. In this work, we propose an improved EfficientNet based on multi-view feature fusion for the automatic diagnosis of cervical squamous intraepithelial lesions. Specifically, EfficientNet-B0 is employed as the backbone network, and three-view images are taken as inputs by channel cascading to reduce misclassification. Additionally, we propose a dual-attention mechanism that implements the feature selection function based on Convolution Block Attention Module (CBAM) and Coordinate Attention (CA). These two attention mechanisms assist each other to enhance the feature representation of HSIL. We leverage a dataset of 3294 clinical cervigrams and obtain 90.0% accuracy with recall, specificity, and F1-Score of 87.1%, 93.0%, and 89.7%, respectively. Experimental results prove that this method can help clinicians with precise disease classification and diagnosis, and outperforms known related works.

Multiview Feature Fusion Representation for Interictal Epileptiform Spikes Detection.

Interictal epileptiform spikes (IES) of scalp electroencephalogram (EEG) signals have a strong relation with the epileptogenic region. Since IES are highly unlikely to be detected in scalp EEG signals, the primary diagnosis depends heavily on the visual evaluation of IES. However, visual inspection of EEG signals, the standard IES detection procedure is time-consuming, highly subjective, and error-prone. Furthermore, the highly complex, nonlinear, and nonstationary characteristics of EEG signals lead to the incomplete representation of EEG signals in existing computer-aided methods and consequently unsatisfactory detection performance. Therefore, a novel multiview feature fusion representation (MVFFR) method was developed and combined with a robustness classifier to detect EEG signals with/without IES. MVFFR comprises two steps: First, temporal, frequency, temporal-frequency, spatial, and nonlinear domain features are transformed by the IES to express the latent information effectively. Second, the unsupervised infinite feature-selection method determines the most distinct feature fusion representations. Experimental results using a balanced dataset of six patients showed that MVFFR achieved the optimal detection performance (accuracy: 89.27%, sensitivity: 89.01%, specificity: 89.54%, and precision: 89.82%) compared with other feature ranking methods, and the MVFFR-related method were complementary and indispensable. Additionally, in an independent test, MVFFR maintained excellent generalization capacity with a false detection rate per minute of 0.15 on the unbalanced dataset of one patient.

A Multi-View Image Feature Fusion Network Applied in Analysis of Aeration Velocity for WWTP

The instability of the aeration system brings a significant challenge to the management of wastewater treatment plants (WWTP). Using image recognition methods to monitor aeration conditions accurately and enhance management efficiency is a promising way to solve this problem. To improve the efficiency of aeration condition identification and provide support for troubleshooting, we propose a method for aeration velocity condition identification based on a multi-view image feature fusion network (MVNN). Firstly, an experimental platform for simulating aeration tanks is established, and two cameras are used to acquire aeration images from different perspectives. Secondly, an image data set with 10 aeration velocity gradients is constructed and applied to the network’s training. Finally, the MVNN is used to extract and fuse the features of aeration images, and the model’s performance is evaluated on the dataset. Experiments show that the average accuracy of the method is over 98.3%, and the AUC of aeration identification is above 0.98, which indicates that the model has the potential for practical application in WWTP.

MVFFNet: Multi-view feature fusion network for imbalanced ship classification

The accurate classification of moving ships is of fundamental importance to maritime authorities for ensuring the safety and security of shipping operations. With the wide use of automatic identification systems (AISs), which allow ships to receive identification/location information from nearby ships, it is feasible to classify the types of ships by analysing ship behaviour from AIS-based trajectories. However, the imbalanced features of AIS data make it difficult to achieve satisfactory classification results in the presence of several different types of ships. To overcome these potential limitations, we propose a multi-view feature fusion network (MVFFNet) to achieve accurate ship classification with imbalanced data. To guarantee the powerful representation and generalization abilities of MVFFNet, we first extract several multi-view features (i.e., motion features and morphological features) from AIS-based ship trajectories. Several kinematic variables related to ship behaviour are empirically adopted as motion features. The morphological features are automatically extracted via convolutional auto-encoder (CAE) networks. CAE networks are capable of optimally learning the features from informative trajectory images, which are strictly related to the original ship trajectories. The bidirectional gated recurrent unit (BiGRU) network is then proposed to combine multi-view features to generate the ship classification results. In addition, a hybrid loss function is presented to handle the imbalance problem of ship types, potentially leading to enhanced robustness and accuracy of ship classification. Comprehensive experiments on two realistic datasets have demonstrated that our proposed MVFFNet consistently outperforms other competing methods in terms of classification accuracy and robustness.

Local2Global: Unsupervised multi-view deep graph representation learning with Nearest Neighbor Constraint

Multi-view feature fusion is a vital phase for multi-view representation learning. Recently, most Graph Auto-Encoders (GAEs) and their variants focus on multi-view learning. However, most of them ignore deep representation fusion of features of each multi-view. Furthermore, there are scarcely unsupervised constraints guiding to enhance the graph representation capability in training process. In this paper, we propose a novel unsupervised Multi-view Deep Graph Representation Learning (MDGRL) framework on multi-view data which is based on the Graph Auto-Encoders (GAEs) for local feature leaning, a feature fusion module for producing global representation and a valid variant of Variational Graph Auto-Encoder (VGAE) for global deep graph representation learning. To fuse Nearest Neighbor Constraint (NNC) between the maximal degree nodes which represents the most close joining node and their adjacent nodes into VGAE, we propose a new Nearest Neighbor Constraint Variational Graph Auto-Encoder (NNC-VGAE) to enhance the global deep graph representation capability for multi-view data. In the training process of NNC-VGAE, NNC makes the adjacent nodes gradually close to the maximal degree node. Hence, the proposed MDGRL has excellent deep graph representation capability for multi-view data. Experiments on eight non-medical benchmark multi-view data sets and four medical data sets confirm the effectiveness of our MDGRL compared with other state-of-the-art methods for unsupervised clustering.

Multi-view feature fusion for person re-identification

Person re-identification (ReID) suffers from camera view variants. Existing works, which typically learn a feature for each image, share a limitation that the learned features are single-view: each feature only contains information in one camera view. Thus, view bias occurs when matching pedestrians across camera views. In this paper, we seek to mitigate the view bias by generating multi-view features (fusion of features from a fixed number of cameras). To this end, we define the complementary-view features (complementary features to generate multi-view features with single-view features) and perform in-depth analysis. Based on this insight, we alleviate the view bias in testing and training, respectively. In testing, we present Multi-view Message Passing (MVMP), which generates multi-view features by aggregating single-view features from the neighborhood. In training, we propose Multi-view Feature Fusion Network (MFFN), which involves the single-view feature extractor and the complementary-view feature aggregator. MFFN makes the network sensitive to view-specific cues by adding constraints on multi-view features rather than single-view features. In addition, MVMP and MFFN have two key advantages: (1) They are parameter-free. (2) They can be applied to any Convolutional Neural Networks (CNNs) readily without extra supervision. Extensive experiments are conducted to validate the superiority of our method for person ReID over state-of-the-art methods on four benchmark datasets (Market-1501, DukeMTMC-reID, CUHK03, and MSMT17). The code is available at https://github.com/Yinsongxu/MVMP_MFFN.

DLFF-ACP: prediction of ACPs based on deep learning and multi-view features fusion.

An emerging type of therapeutic agent, anticancer peptides (ACPs), has attracted attention because of its lower risk of toxic side effects. However process of identifying ACPs using experimental methods is both time-consuming and laborious. In this study, we developed a new and efficient algorithm that predicts ACPs by fusing multi-view features based on dual-channel deep neural network ensemble model. In the model, one channel used the convolutional neural network CNN to automatically extract the potential spatial features of a sequence. Another channel was used to process and extract more effective features from handcrafted features. Additionally, an effective feature fusion method was explored for the mutual fusion of different features. Finally, we adopted the neural network to predict ACPs based on the fusion features. The performance comparisons across the single and fusion features showed that the fusion of multi-view features could effectively improve the model’s predictive ability. Among these, the fusion of the features extracted by the CNN and composition of k-spaced amino acid group pairs achieved the best performance. To further validate the performance of our model, we compared it with other existing methods using two independent test sets. The results showed that our model’s area under curve was 0.90, which was higher than that of the other existing methods on the first test set and higher than most of the other existing methods on the second test set. The source code and datasets are available at https://github.com/wame-ng/DLFF-ACP.

Tensor dispersion-based multi-view feature embedding for dimension reduction

With the development of feature extraction technique, one image object can be represented by multiple heterogeneous features from different views that locate in high-dimensional space. Multiple features can reflect various characteristics of the same object; they contain compatible and complementary information among each other, integrating them together used in the special image processing application that can obtain better performance. However, facing these multi-view features, most dimensionality reduction methods fail to completely achieve the desired effect. Therefore, how to construct an uniform low-dimensional embedding subspace, which exploits useful information from multi-view features is still an important and urgent issue to be solved. So, we propose an innovative fusion dimension reduction method named tensor dispersion-based multi-view feature embedding (TDMvFE). TDMvFE reconstructs a feature subspace of each object by utilizing its k nearest neighbors, which preserves the underlying neighborhood structure of the original manifold in the lower dimensional mapping space. The new method fully exploits the channel correlations and spatial complementarities from the multi-view features by tensor dispersion analysis model. Furthermore, the method constructs an optimization model and derives an iterative procedure to generate the unified low dimensional embedding. Various evaluations based on the applications of image classification and retrieval demonstrate the effectiveness of our proposed method on multi-view feature fusion dimension reduction.

A multiview feature fusion model for heartbeat classification

Objective. An electrocardiogram (ECG) is one of the most common means to diagnose arrhythmia according to different waveforms clinically. Although there are advanced classification methods such as deep learning, the single view feature cannot meet the demand of classification accuracy for new individuals. To this end, a classification model based on multiview fusion was proposed. Approach. First, handcrafted view features were extracted from heartbeats and then deep view features were obtained from the deep learning model. The features of two different perspectives were fused in the fully connected layer, and the random forest classifier was used instead of the Softmax classifier for classification. Notably, Bayesian optimization was utilized in the hyper-parameter tuning of the classifier. The proposed method employed the MIT-BIH database to classify five classes: normal heartbeat (N), left bundle branch block heartbeat (LB), right bundle branch block heartbeat (RB), atrial premature contraction (APC) and premature ventricular contraction (PVC). Main results. The experimental results achieved a higher average accuracy of 98.93%, average precision of 96.92%, average sensitivity of 96.46%, and average specificity of 99.33% in five types of heartbeat classification for inter-patient. Significance. The proposed framework improves the performance of ECG detection for new individuals. And it provides an feasible algorithmic model for single-lead wearable devices with multiview fusion.

Unsupervised Multi-View Clustering by Squeezing Hybrid Knowledge From Cross View and Each View

Multi-view clustering methods have been a focus in recent years because of their superiority in clustering performance. However, typical traditional multi-view clustering algorithms still have shortcomings in some aspects, such as removal of redundant information, utilization of various views and fusion of multi-view features. In view of these problems, this paper proposes a new multi-view clustering method, low-rank subspace multi-view clustering based on adaptive graph regularization. We construct two new data matrix decomposition models into a unified optimization model. In this framework, we address the significance of the common knowledge shared by the cross view and the unique knowledge of each view by presenting new low-rank and sparse constraints on the sparse subspace matrix. To ensure that we achieve effective sparse representation and clustering performance on the original data matrix, adaptive graph regularization and unsupervised clustering constraints are also incorporated in the proposed model to preserve the internal structural features of the data. Finally, the proposed method is compared with several state-of-the-art algorithms. Experimental results for five widely used multi-view benchmarks show that our proposed algorithm surpasses other state-of-the-art methods by a clear margin.

A Multi-View Fusion Method via Tensor Learning and Gradient Descent for Image Features

In many computer vision applications, one image can be represented by multiple heterogeneous features from different views, most of them commonly locate in high-dimensional space. These features can reflect different characteristics of one same object, they contain compatible and complementary information among each other. How to construct an uniform low-dimensional embedding features which represent useful information of multi-view features is still an important and urgent issue to be solved. Therefore, we propose a multi-view fusion method via tensor learning and gradient descent (MvF-TG) in this paper. MvF-TG reconstructs a lowdimensional mapping subspace of each object by utilizing its k nearest neighbors, which preserves the underlying neighborhood structure of the original local manifold. The new method can effectively exploit the spatial correlation information from the multi-view features by tensor learning. Furthermore, the method constructs a gradient descent optimization model to generate the better unified low dimensional embedding. The proposed method is compared with several single-view and multi-view dimensional reduction methods in these indicators of P, R, MAP and F-measure. In the retrieval experiments, the P values of the newmethod respectively are 86.80%, 52.00%, 68.56% and 78.80% on datasets of Corel1k, Corel5k, Corel10k and Holidays. In the classification experiments, the mean accuracies of it respectively are 47.94% and 87.58% on datasets of Caltech101 and Coil. These values are higher than those obtained by other comparison methods, various evaluations based on the applications of image classification and retrieval demonstrates the effectiveness of our proposed method on multi-view feature fusion dimension reduction.