Feature Fusion Technique Research Articles

CNN-GRU-FF: a double-layer feature fusion-based network intrusion detection system using convolutional neural network and gated recurrent units

Identifying and preventing malicious network behavior is a challenge for establishing a secure network communication environment or system. Malicious activities in a network system can seriously threaten users’ privacy and potentially jeopardize the entire network infrastructure and functions. Furthermore, cyber-attacks have grown in complexity and number due to the ever-evolving digital landscape of computer and network devices in recent years. Analyzing network traffic using network intrusion detection systems (NIDSs) has become an integral security measure in modern networks to identify malicious and suspicious activities. However, most intrusion detection datasets contain imbalance classes, making it difficult for most existing classifiers to achieve good performance. In this paper, we propose a double-layer feature extraction and feature fusion technique (CNN-GRU-FF), which uses a modified focal loss function instead of the traditional cross-entropy to handle the class imbalance problem in the IDS datasets. We use the NSL-KDD and UNSW-NB15 datasets to evaluate the effectiveness of the proposed model. From the research findings, it is evident our CNN-GRU-FF method obtains a detection rate of 98.22% and 99.68% using the UNSW-NB15 and NSL-KDD datasets, respectively while maintaining low false alarm rates on both datasets. We compared the proposed model’s performance with seven baseline algorithms and other published methods in literature. It is evident from the performance results that our proposed method outperforms the state-of-the-art network intrusion detection methods.

Fusion of Coherent and Non-Coherent Pol-SAR Features for Land Cover Classification

Remote Sensing plays a fundamental role in acquiring crucial information about the Earth’s surface from a distance, especially through fully polarimetric data, which offers a rich source of information for diverse applications. However, extracting meaningful insights from this intricate data necessitates sophisticated techniques. In addressing this challenge, one predominant trend that has emerged is known as target decomposition techniques. These techniques can be broadly classified into coherent and non-coherent methods. Each of these methods provides high-quality information using different procedures. In this context, this paper introduces innovative feature fusion techniques, amalgamating coherent and non-coherent information. While coherent techniques excel in detailed exploration and specific feature extraction, non-coherent methods offer a broader perspective. Our feature fusion techniques aim to harness the strengths of both approaches, providing a comprehensive and high-quality fusion of information. In the first approach, features derived from Pauli coherent decomposition, Freeman–Durden non-coherent technique, and the Symmetry criterion from Cameron’s stepwise algorithm are combined to construct a sophisticated feature vector. This fusion is achieved using the well-established Fisher Linear Discriminant Analysis algorithm. In the second approach, the Symmetry criterion serves as the basis for fusing coherent and non-coherent coefficients, resulting in the creation of a new feature vector. Both approaches aim to exploit information simultaneously extracted from coherent and non-coherent methods in feature extraction from Remote Sensing data through fusion at the feature level. To evaluate the effectiveness of the feature generated by the proposed fusion techniques, we employ a land cover classification procedure. This involves utilizing a basic classifier, achieving overall accuracies of approximately 82% and 86% for each of the two proposed techniques. Furthermore, the accuracy in individual classes surpasses 92%. The evaluation aims to gauge the effectiveness of the fusion methods in enhancing feature extraction from fully polarimetric data and opens avenues for further exploration in the integration of coherent and non-coherent features for remote sensing applications.

Development and validation of a multi-modality fusion deep learning model for differentiating glioblastoma from solitary brain metastases.

Glioblastoma (GBM) and brain metastases (BMs) are the two most common malignant brain tumors in adults. Magnetic resonance imaging (MRI) is a commonly used method for screening and evaluating the prognosis of brain tumors, but the specificity and sensitivity of conventional MRI sequences in differential diagnosis of GBM and BMs are limited. In recent years, deep neural network has shown great potential in the realization of diagnostic classification and the establishment of clinical decision support system. This study aims to apply the radiomics features extracted by deep learning techniques to explore the feasibility of accurate preoperative classification for newly diagnosed GBM and solitary brain metastases (SBMs), and to further explore the impact of multimodality data fusion on classification tasks. Standard protocol cranial MRI sequence data from 135 newly diagnosed GBM patients and 73 patients with SBMs confirmed by histopathologic or clinical diagnosis were retrospectively analyzed. First, structural T1-weight, T1C-weight, and T2-weight were selected as 3 inputs to the entire model, regions of interest (ROIs) were manually delineated on the registered three modal MR images, and multimodality radiomics features were obtained, dimensions were reduced using a random forest (RF)-based feature selection method, and the importance of each feature was further analyzed. Secondly, we used the method of contrast disentangled to find the shared features and complementary features between different modal features. Finally, the response of each sample to GBM and SBMs was predicted by fusing 2 features from different modalities. The radiomics features using machine learning and the multi-modal fusion method had a good discriminatory ability for GBM and SBMs. Furthermore, compared with single-modal data, the multimodal fusion models using machine learning algorithms such as support vector machine (SVM), Logistic regression, RF, adaptive boosting (AdaBoost), and gradient boosting decision tree (GBDT) achieved significant improvements, with area under the curve (AUC) values of 0.974, 0.978, 0.943, 0.938, and 0.947, respectively; our comparative disentangled multi-modal MR fusion method performs well, and the results of AUC, accuracy (ACC), sensitivity (SEN) and specificity(SPE) in the test set were 0.985, 0.984, 0.900, and 0.990, respectively. Compared with other multi-modal fusion methods, AUC, ACC, and SEN in this study all achieved the best performance. In the ablation experiment to verify the effects of each module component in this study, AUC, ACC, and SEN increased by 1.6%, 10.9% and 15.0%, respectively after 3 loss functions were used simultaneously. A deep learning-based contrast disentangled multi-modal MR radiomics feature fusion technique helps to improve GBM and SBMs classification accuracy.

A novel feature fusion technique for robust hand gesture recognition

A novel feature fusion technique for robust hand gesture recognition

Contact Defect Detection of Gas-Insulated Line Via Thermal–Vibration Feature Fusion and Deep Neural Network Technique

Contact Defect Detection of Gas-Insulated Line Via Thermal–Vibration Feature Fusion and Deep Neural Network Technique

Toward Unified AI Drug Discovery with Multimodal Knowledge.

Background: In real-world drug discovery, human experts typically grasp molecular knowledge of drugs and proteins from multimodal sources including molecular structures, structured knowledge from knowledge bases, and unstructured knowledge from biomedical literature. Existing multimodal approaches in AI drug discovery integrate either structured or unstructured knowledge independently, which compromises the holistic understanding of biomolecules. Besides, they fail to address the missing modality problem, where multimodal information is missing for novel drugs and proteins. Methods: In this work, we present KEDD, a unified, end-to-end deep learning framework that jointly incorporates both structured and unstructured knowledge for vast AI drug discovery tasks. The framework first incorporates independent representation learning models to extract the underlying characteristics from each modality. Then, it applies a feature fusion technique to calculate the prediction results. To mitigate the missing modality problem, we leverage sparse attention and a modality masking technique to reconstruct the missing features based on top relevant molecules. Results: Benefiting from structured and unstructured knowledge, our framework achieves a deeper understanding of biomolecules. KEDD outperforms state-of-the-art models by an average of 5.2% on drug-target interaction prediction, 2.6% on drug property prediction, 1.2% on drug-drug interaction prediction, and 4.1% on protein-protein interaction prediction. Through qualitative analysis, we reveal KEDD's promising potential in assisting real-world applications. Conclusions: By incorporating biomolecular expertise from multimodal knowledge, KEDD bears promise in accelerating drug discovery.

WECNN-PDP: Weighted Ensemble Convolutional Neural Networks Models to Improve the Plant Disease Prediction

As an agricultural country, Indonesia’s agricultural production is essential. However, crop failure will occur if diseases and other factors, such as natural disasters, attack many plant fields. These problems can be minimized by early detection of plant diseases. However, detection will be challenging if done conventionally. Prior research has shown that deep learning algorithms can perform detection with promising results. In this study, we propose a new weighted deep learning ensemble method as a solution for better performance in plant disease detection. We ensemble the model by considering the combination of two and three pre-trained convolutional neural networks (CNNs). Initially, we perform transfer learning on individual CNN models by prioritizing high-dimensional features through weight updates on the last few layers. Finally, we ensemble the models by finding the best weights for each model using grid search. Experimental results on the Plant Village dataset indicate that our model has improved the classification of 38 plant diseases. Based on metrics, the three-model ensemble performed better than the two-model ensemble. The best accuracy results of the ensemble MobileNetV2-DenseNet121 and MobileNetV2-Xception-DenseNet121 models are 99.49% and 99.56%, respectively. In addition, these models are also better than the state-of-the-art models and previous feature fusion techniques we proposed in LEMOXINET. Based on these results, the ensemble technique improved the detection performance, and it is expected to be applied to real-world conditions and can be a reference to be developed further in future research.

Pattern recognition study of musical rhythm perception based on multimodal deep learning

Abstract Rhythm perception is becoming more and more important in the field of music information processing and music understanding. The study first adopts signal processing methods to extract musical features, then uses feature fusion techniques to integrate features of different modalities into a single feature vector. Based on this model, the study identifies the rhythmic activation function of music and combines it with the hidden Markov model to infer the rhythm of the music. One of the key points of the study is to perform rhythm recognition on music containing drums, to explore the recognition effect. One of the focuses of the study is to recognize the rhythm of music containing drums to explore the recognition effect.In addition, the study also analyzes the Softmax output values of the music and compares the recognition effect of different models. The results show that the rhythm recognition of music using the multimodal deep learning method performs the best in terms of the F-Measure value, the Cemgil value, the Goto value, and the P-score value, with the respective 65.65%, 66.76%, 36.75%, and 36.75%. 66.76%, 36.75%, and 75.68%.Especially in the drum music recognition, the position of each drum music is accurately recognized, proving the model’s effectiveness in this paper. The research provides a new feasible method for the recognition and understanding of music rhythms and a valuable reference for the research in this field.

A novel automated system to detect breast cancer from ultrasound images using deep fused features with super resolution

Cancer patients can benefit from early detection and diagnosis. This study proposes a machine vision strategy for detecting breast cancer in ultrasound images and correcting several ultrasound difficulties such artifacts, speckle noise, and blurring. In quantitative evolution, edge preservation criteria were discovered to be superior to standard ones for hybrid anisotropic diffusion. A learnable super-resolution (SR) is inserted in the deep CNN to dig for extra possible information. The feature is fused with a pre-trained deep CNN model utilizing Gabor Wavelet Transform (GWT) and Local Binary Pattern (LBP). Machine learning (ML) techniques that are used to create these recommendation systems require well-balanced data in terms of class distribution, however most datasets in the real world are imbalanced. Imbalanced data forces a classifier to concentrate on the majority class while ignoring other classes of interest, lowering the predicted performance of any classification model. We propose a generative adversarial networks (GAN) strategy to overcome the data imbalance problem and improve the performance of recommendation systems in this research. Standard data is used to train this model, which assures a high level of resolution. In the testing phase, generalized data of varied resolution is used to evaluate the model's performance. It is discovered through cross-validation that a 5-fold method can successfully eliminate the overfitting problem. With an accuracy of 99.48 %, this suggested feature fusion technique indicates satisfactory performance when compared to current related works. Finally finding cancer region, researcher used U-Net architecture and extract cancer region from BC ultrasound images.

Cucumber Disease Classification with Ensemble Learning Method for Complex Datasets

Many researchers are taking into account the algorithm's ability to detect diseases in plants since it can save expenses and deliver more accurate results. However, there are various obstacles in detecting diseases, particularly in cucumber plants, such as disease similarities and the ability of models to adapt to the information they have. To address this issue, we propose an ensemble learning strategy based on the averaging method to improve the model's ability to generalize to different cucumber plant environments. According to the results, the ensemble learning approach outperforms the feature fusion method with a test accuracy of 94.20% and a loss of 0.01105. Feature fusion and ensemble learning techniques, in general, have the potential to increase the model's capacity to classify difficult data.

MSER: Multimodal speech emotion recognition using cross-attention with deep fusion

In human–computer interaction (HCI) and especially speech signal processing, emotion recognition is one of the most important and challenging tasks due to multi-modality and limited data availability. Nowadays, an intelligent system is required for real-world applications to efficiently process and understand the speaker's emotional state and to enhance the analytical abilities to assist communication by a human-machine interface (HMI). Designing a reliable and robust Multimodal Speech Emotion Recognition (MSER) to efficiently recognize emotions through multi-modality such as speech and text is necessary. This paper propose a novel MSER model with a deep feature fusion technique using a multi-headed cross-attention mechanism. The proposed model utilize audio and text cues to predict the emotion label accordingly. Our proposed model process the raw speech signal and text by CNN and feeds to corresponding encoders for discriminative and semantic feature extractions. The cross-attention mechanism is applied to both features to enhance the interaction between text and audio cues by crossway to extract the most relevant information for emotion recognition. Finally, combining the region-wise weights from both encoders enables interaction among different layers and paths by the proposed deep feature fusion scheme. The authors evaluate the proposed system using the IEMOCAP and MELD datasets and conduct extensive experiments that obtain state-of-the-art (SOTA) results and show a 4.5% improved recognition rate, respectively. Our model secured a significant improvement over SOTA methods, which shows the robustness and effectiveness of the proposed MSER model.

Spatio-Temporal Information Fusion and Filtration for Human Action Recognition

Human action recognition (HAR) as the most representative human-centred computer vision task is critical in human resource management (HRM), especially in human resource recruitment, performance appraisal, and employee training. Currently, prevailing approaches to human action recognition primarily emphasize either temporal or spatial features while overlooking the intricate interplay between these two dimensions. This oversight leads to less precise and robust action classification within complex human resource recruitment environments. In this paper, we propose a novel human action recognition methodology for human resource recruitment environments, which aims at symmetrically harnessing temporal and spatial information to enhance the performance of human action recognition. Specifically, we compute Depth Motion Maps (DMM) and Depth Temporal Maps (DTM) from depth video sequences as space and time descriptors, respectively. Subsequently, a novel feature fusion technique named Center Boundary Collaborative Canonical Correlation Analysis (CBCCCA) is designed to enhance the fusion of space and time features by collaboratively learning the center and boundary information of feature class space. We then introduce a spatio-temporal information filtration module to remove redundant information introduced by spatio-temporal fusion and retain discriminative details. Finally, a Support Vector Machine (SVM) is employed for human action recognition. Extensive experiments demonstrate that the proposed method has the ability to significantly improve human action recognition performance.

A Study of Vietnamese Sentiment Classification with Ensemble Pre-Trained Language Models

Sentiment Analysis (SA) has attracted increasing research attention in recent years. Most existing works tackle the SA task by fine-tuning single pre-trained language models combined with specific layers. Despite their effectiveness, the previous studies overlooked the utilization of feature representations from various contextual language models. Ensemble learning techniques have garnered increasing attention within the field of Natural Language Processing (NLP). However, there is still room for improvement in ensemble models for the SA task, particularly in the aspect-level SA task. Furthermore, the utilization of heterogeneous ensembles, which combine various pre-trained transformer-based language models, may prove beneficial in enhancing overall performance by incorporating diverse linguistic representations. This paper introduces two ensemble models that leverage soft voting and feature fusion techniques by combining individual pre-trained transformer-based language models for the SA task. The latest transformer-based models, including PhoBERT, XLM, XLM-Align, InfoXLM, and viBERT_FPT, are employed to integrate knowledge and representations using feature fusion and a soft voting strategy. We conducted extensive experiments on various Vietnamese benchmark datasets, encompassing sentence-level, document-level, and aspect-level SA. The experimental results demonstrate that our approaches outperform most existing methods, achieving new state-of-the-art results with F1-weighted scores of 94.03%, 95.65%, 75.36%, and 76.23% on the UIT_VSFC, Aivivn, UIT_ABSA for the restaurant domain, and UIT_ViSFD datasets, respectively.

Multi-view feature fusion and density-based minority over-sampling technique for amyloid protein prediction under imbalanced data

Amyloid fibers formed by the aggregation of amyloid proteins can lead to various neurological diseases. Accurate prediction of amyloid proteins can provide scientific basis for exploring the pathogenesis of related neurological diseases, and developing targeted treatment plans. In view of the limitations of the synthetic minority over-sampling technique (SMOTE), an improved SMOTE method based on density clustering is proposed in this paper, and combined with multi-view features to construct an amyloid protein prediction model. The benefits of this enhanced SMOTE method include eliminating problematic samples, accomplishing an evenly distributed minority sample pool, and utilizing all available boundary sample information. Compared with other over-sampling methods, this method can effectively improve the authenticity and representativeness of the synthesized samples, and has a strong ability to process imbalanced data. After the dataset is over-sampling, different prediction algorithms are trained respectively to construct the baseline models. To fully characterize proteins, the probability and category features are extracted from the baseline models and combined with the sequence features selected by CFS-GSSS (Correlation-based Feature Selection Combined with Greedy Stepwise Search Strategy) to generate the multi-view features. Experimental results demonstrate the complementarity and feasibility of the multi-view features. On the independent test set, compared with the existing best model (ECAmyloid), the proposed amyloid protein prediction model improved the sensitivity and geometric mean by 0.0706 and 0.0129, respectively, far superior to other existing methods.

Synthetic 3D Spinal Vertebrae Reconstruction from Biplanar X-rays Utilizing Generative Adversarial Networks.

Computed tomography (CT) offers detailed insights into the internal anatomy of patients, particularly for spinal vertebrae examination. However, CT scans are associated with higher radiation exposure and cost compared to conventional X-ray imaging. In this study, we applied a Generative Adversarial Network (GAN) framework to reconstruct 3D spinal vertebrae structures from synthetic biplanar X-ray images, specifically focusing on anterior and lateral views. The synthetic X-ray images were generated using the DRRGenerator module in 3D Slicer by incorporating segmentations of spinal vertebrae in CT scans for the region of interest. This approach leverages a novel feature fusion technique based on X2CT-GAN to combine information from both views and employs a combination of mean squared error (MSE) loss and adversarial loss to train the generator, resulting in high-quality synthetic 3D spinal vertebrae CTs. A total of n = 440 CT data were processed. We evaluated the performance of our model using multiple metrics, including mean absolute error (MAE) (for each slice of the 3D volume (MAE0) and for the entire 3D volume (MAE)), cosine similarity, peak signal-to-noise ratio (PSNR), 3D peak signal-to-noise ratio (PSNR-3D), and structural similarity index (SSIM). The average PSNR was 28.394 dB, PSNR-3D was 27.432, SSIM was 0.468, cosine similarity was 0.484, MAE0 was 0.034, and MAE was 85.359. The results demonstrated the effectiveness of this approach in reconstructing 3D spinal vertebrae structures from biplanar X-rays, although some limitations in accurately capturing the fine bone structures and maintaining the precise morphology of the vertebrae were present. This technique has the potential to enhance the diagnostic capabilities of low-cost X-ray machines while reducing radiation exposure and cost associated with CT scans, paving the way for future applications in spinal imaging and diagnosis.

Unsupervised multi-source domain adaptation for person re-identification via feature fusion and pseudo-label refinement

The objective of unsupervised domain adaptation (UDA) for person re-identification (re-ID) is to associate person in images captured from heterogeneous camera perspectives. Currently, mainstream UDA methods for person re-ID are mainly conducted in single-source and single-target domain scenarios. Moreover, most of these methods do not take the repercussions of pseudo-label noise on model performance into consideration. Therefore, we put forward an unsupervised multi-source domain adaptation (UMDA) method for person re-ID via feature fusion and pseudo-label refinement. Firstly, our method is designed for scenarios where there exist several source domains and only one target domain. We suggest using feature fusion techniques to minimize the domain disparity among the source domains, and employ pseudo-label refinement techniques to ameliorate the ramifications of label noise on model predictions. To substantiate the effectiveness of the recommended methodology, we carry out a succession of experiments on multiple datasets. The advantage and preeminence of our proposed method can be manifested by the experimental outcomes.

Automatic segmentation of mandibular canal using transformer based neural networks.

Accurate 3D localization of the mandibular canal is crucial for the success of digitally-assisted dental surgeries. Damage to the mandibular canal may result in severe consequences for the patient, including acute pain, numbness, or even facial paralysis. As such, the development of a fast, stable, and highly precise method for mandibular canal segmentation is paramount for enhancing the success rate of dental surgical procedures. Nonetheless, the task of mandibular canal segmentation is fraught with challenges, including a severe imbalance between positive and negative samples and indistinct boundaries, which often compromise the completeness of existing segmentation methods. To surmount these challenges, we propose an innovative, fully automated segmentation approach for the mandibular canal. Our methodology employs a Transformer architecture in conjunction with cl-Dice loss to ensure that the model concentrates on the connectivity of the mandibular canal. Additionally, we introduce a pixel-level feature fusion technique to bolster the model's sensitivity to fine-grained details of the canal structure. To tackle the issue of sample imbalance and vague boundaries, we implement a strategy founded on mandibular foramen localization to isolate the maximally connected domain of the mandibular canal. Furthermore, a contrast enhancement technique is employed for pre-processing the raw data. We also adopt a Deep Label Fusion strategy for pre-training on synthetic datasets, which substantially elevates the model's performance. Empirical evaluations on a publicly accessible mandibular canal dataset reveal superior performance metrics: a Dice score of 0.844, click score of 0.961, IoU of 0.731, and HD95 of 2.947mm. These results not only validate the efficacy of our approach but also establish its state-of-the-art performance on the public mandibular canal dataset.

Keypoint regression strategy and angle loss based YOLO for object detection

The YOLOv4 approach has gained significant popularity in industrial object detection due to its impressive real-time processing speed and relatively favorable accuracy. However, it has been observed that YOLOv4 faces challenges in accurately detecting small objects. Its bounding box regression strategy is rigid and fails to effectively leverage the asymmetric characteristics of objects, limiting its ability to enhance object detection accuracy. This paper proposes an enhanced version of YOLOv4 called KR–AL–YOLO (keypoint regression strategy and angle loss based YOLOv4). The KR–AL–YOLO approach introduces two customized modules: an keypoint regression strategy and an angle-loss function. These modules contribute to improving the algorithm’s detection accuracy by enabling more precise localization of objects. Additionally, KR–AL–YOLO adopts an improved feature fusion technique, which facilitates enhanced information flow within the network, thereby further enhancing accuracy performance. Experimental evaluations conducted on the COCO2017 dataset demonstrate the effectiveness of the proposed method. KR–AL–YOLO achieves an average precision of 45.6%, surpassing both YOLOv4 and certain previously developed one-stage detectors. The utilization of keypoint regression strategy and the incorporation of robust feature fusion contribute to superior object detection accuracy in KR–AL–YOLO compared to YOLOv4.

Distributed status evaluation method based on evidence revision fusion

BackgroundComplex process industrial process is large in scale, with numerous variables and complex variable relationship. It is difficult to construct an overall status evaluation model to accurately evaluate the operation status. Compared with holistic modeling status evaluation, distributed status evaluation can make better use of potentially valuable local details. Besides, the integration of result from multiple status evaluation models using decision fusion method such as Dempster-Shafer (DS) evidence theory can make the model more robust. However, DS evidence theory produces counterintuitive result when dealing with high conflict evidence. MethodsIn this paper, a distributed status evaluation method based on evidence revision fusion is proposed. Firstly, a partition method combining qualitative and quantitative knowledge is proposed, which improves the ability to capture local effective information. Secondly, a new status fusion evaluation method is proposed. Feature fusion technique is used to process the original process data, and the evaluation results of multiple status evaluation models are fused by the decision fusion method. Finally, a conflict management strategy with parallel supervised and unsupervised revision is proposed to revise high-conflict evidence from different status evaluation models. FindingsThe proposed method shows good performance in Tennessee-Eastman process, which demonstrates the effectiveness of the method.

BEMF-Net: Semantic Segmentation of Large-Scale Point Clouds via Bilateral Neighbor Enhancement and Multi-Scale Fusion

The semantic segmentation of point clouds is a crucial undertaking in 3D reconstruction and holds great importance. However, achieving precise semantic segmentation represents a significant hurdle. In this paper, we present BEMF-Net, an efficient method for large-scale environments. It starts with an effective feature extraction method. Unlike images, 3D data comprise not only geometric relations but also texture information. To accurately depict the scene, it is crucial to take into account the impacts of texture and geometry on the task, and incorporate modifications to improve feature description. Additionally, we present a multi-scale feature fusion technique that effectively promotes the interaction between features at different resolutions. The approach mitigates the problem of the smoothing of detailed information caused by downsampling mechanisms, while ensuring the integrity of features across different layers, allowing a more comprehensive representation of the point cloud. We confirmed the effectiveness of this method by testing it on benchmark datasets such as S3DIS, SensatUrban, and Toronto3D.