Feature Fusion Model Research Articles

Multi-feature Fusion Based on RV Correlation Coefficient for Facial Expression Recognition

Extracting effective features of expressions becomes a hot research topic, and a single feature pattern cannot reflect the diversity of expressions. Therefore, to obtain rich information feature data and raise the expression recognition performance, we propose a feature fusion model of multiple feature selection by the measure of the RV correlation coefficient. In the proposed feature fusion model, the feature patterns are firstly selected by RV correlation coefficient from various expression texture features. And then according to rank the values of the RV correlation coefficient, we build a CCA subspace and PCA subspace respectively to fuse selected features. Finally, a new facial expression feature presentation is constructed through weighting and combining the two fusion features from the subspaces. The new features are fed to SVM classifier for expression recognition. Experimental verification shows that our proposed model has a superior performance than the existing algorithms.

A Multi-Layer Feature Fusion Model Based on Convolution and Attention Mechanisms for Text Classification

Text classification is one of the fundamental tasks in natural language processing and is widely applied in various domains. CNN effectively utilizes local features, while the Attention mechanism performs well in capturing content-based global interactions. In this paper, we propose a multi-layer feature fusion text classification model called CAC, based on the Combination of CNN and Attention. The model adopts the idea of first extracting local features and then calculating global attention, while drawing inspiration from the interaction process between membranes in membrane computing to improve the performance of text classification. Specifically, the CAC model utilizes the local feature extraction capability of CNN to transform the original semantics into a multi-dimensional feature space. Then, global attention is computed in each respective feature space to capture global contextual information within the text. Finally, the locally extracted features and globally extracted features are fused for classification. Experimental results on various public datasets demonstrate that the CAC model, which combines CNN and Attention, outperforms models that solely rely on the Attention mechanism. In terms of accuracy and performance, the CAC model also exhibits significant improvements over other models based on CNN, RNN, and Attention.

Gastric polyp detection module based on improved attentional feature fusion

Gastric cancer is a deadly disease and gastric polyps are at high risk of becoming cancerous. Therefore, the timely detection of gastric polyp is of great importance which can reduce the incidence of gastric cancer effectively. At present, the object detection method based on deep learning is widely used in medical images. However, as the contrast between the background and the polyps is not strong in gastroscopic image, it is difficult to distinguish various sizes of polyps from the background. In this paper, to improve the detection performance metrics of endoscopic gastric polyps, we propose an improved attentional feature fusion module. First, in order to enhance the contrast between the background and the polyps, we propose an attention module that enables the network to make full use of the target location information, it can suppress the interference of the background information and highlight the effective features. Therefore, on the basis of accurate positioning, it can focus on detecting whether the current location is the gastric polyp or background. Then, it is combined with our feature fusion module to form a new attentional feature fusion model that can mitigate the effects caused by semantic differences in the processing of feature fusion, using multi-scale fusion information to obtain more accurate attention weights and improve the detection performance of polyps of different sizes. In this work, we conduct experiments on our own dataset of gastric polyps. Experimental results show that the proposed attentional feature fusion module is better than the common feature fusion module and can improve the situation where polyps are missed or misdetected.

An efficient dual feature fusion model of feature extraction for hyperspectral images

ABSTRACT Conventional feature extraction (FE) and spatial or context-preserving filters have been extensively studied when applying hyperspectral images (HSIs). However, there still exists some issues to resolve, such as the destruction of the complete structural information because of unfolded 2-D matrix before extracting, the unchanged or even increased number of resulting features from the additional context or spatial-preserving filters, and overreliance on experience to choose retained dimensionality that significantly improves the operating time. This article presents an efficient FE framework, i.e., a dual feature fusion model (DFFM), to address these issues. Specifically, a novel two-order feature fusion (FF) weighted on partial Shannon’s entropy is proposed to hold low dimensional characteristics. Then, a valid three-order FF using constrained Tucker compression is performed on the resulting elements, containing the intact spatial structure and saving computing costs. It can also automatically select a suitable number of features to keep and is robust to noise and training sets. Comparative experiments on three benchmark HSIs were performed to verify the efficiency of DFFM in cases of different training sizes. All the experimental results show that this framework is robust and effective, outperforming several state-of-the-art techniques in classification precision and execution time.

A Skin Cancer Detector Based on Transfer Learning and Feature Fusion

Background: With the rapid development of advanced artificial intelligence technologies which have been applied in varying types of applications, especially in the medical field. Cancer is one of the biggest problems in medical sciences. If cancer can be detected and treated early, the possibility of a cure will be greatly increased. Malignant skin cancer is one of the cancers with the highest mortality rate, which cannot be diagnosed in time only through doctors’ experience. We can employ artificial intelligence algorithms to detect skin cancer at an early stage, for example, patients are determined whether suffering from skin cancer by detecting skin damage or spots. Objective: We use the real HAM10000 image dataset to analyze and predict skin cancer. Methods: (1) We introduce a lightweight attention module to discover the relationships between features, and we fine-tune the pre-trained model (i.e., ResNet-50) on the HAM10000 dataset to extract the hidden high-level features from the images; (2) we integrate these high-level features with generic statistical features, and use the SMOTE oversampling technique to augment samples from the minority classes; and (3) we input the augmented samples into the XGBoost model for training and predicting. Results: The experimental results show that the accuracy, sensitivity, and specificity of the proposed SkinDet (Skin cancer detector based on transfer learning and feature fusion) model reached 98.24%, 97.84%, and 98.13%. The proposed model has stronger classification capability for the minority classes, such as dermato fibroma and actinic keratoses. Conclusion: SkinDet contains a lightweight attention module and can extract the hidden high-level features of the images by fine-tuning the pretrained model on the skin cancer dataset. In particular, SkinDet integrates high-level features with statistical features and augments samples of these minority classes. Importantly, SkinDet can be applied to classify the samples into minority classes.

Cross-Attention Based Multi-Resolution Feature Fusion Model for Self-Supervised Cervical OCT Image Classification.

Cervical cancer seriously endangers the health of the female reproductive system and even risks women's life in severe cases. Optical coherence tomography (OCT) is a non-invasive, real-time, high-resolution imaging technology for cervical tissues. However, since the interpretation of cervical OCT images is a knowledge-intensive, time-consuming task, it is tough to acquire a large number of high-quality labeled images quickly, which is a big challenge for supervised learning. In this study, we introduce the vision Transformer (ViT) architecture, which has recently achieved impressive results in natural image analysis, into the classification task of cervical OCT images. Our work aims to develop a computer-aided diagnosis (CADx) approach based on a self-supervised ViT-based model to classify cervical OCT images effectively. We leverage masked autoencoders (MAE) to perform self-supervised pre-training on cervical OCT images, so the proposed classification model has a better transfer learning ability. In the fine-tuning process, the ViT-based classification model extracts multi-scale features from OCT images of different resolutions and fuses them with the cross-attention module. The ten-fold cross-validation results on an OCT image dataset from a multi-center clinical study of 733 patients in China indicate that our model achieved an AUC value of 0.9963 ± 0.0069 with a 95.89 ± 3.30% sensitivity and 98.23 ± 1.36 % specificity, outperforming some state-of-the-art classification models based on Transformers and convolutional neural networks (CNNs) in the binary classification task of detecting high-risk cervical diseases, including high-grade squamous intraepithelial lesion (HSIL) and cervical cancer. Furthermore, our model with the cross-shaped voting strategy achieved a sensitivity of 92.06% and specificity of 95.56% on an external validation dataset containing 288 three-dimensional (3D) OCT volumes from 118 Chinese patients in a different new hospital. This result met or exceeded the average of four medical experts who have used OCT for over one year. In addition to promising classification performance, our model has a remarkable ability to detect and visualize local lesions using the attention map of the standard ViT model, providing good interpretability for gynecologists to locate and diagnose possible cervical diseases.

A Cross-Scale and Illumination Invariance-Based Model for Robust Object Detection in Traffic Surveillance Scenarios

Robust object detection methods in traffic surveillance scenarios often encounters challenges due to large-scale deformations and illumination variations in outdoor scenes. To enhance the tolerance of such methods against these variations, we design a cross-scale and illumination-invariant detection model (CSIM) based on the You Only Look Once (YOLO) architecture. A main cause of false detection in large-scale detection tasks is the inconsistency between various feature scales. To address this issue, we introduce an adaptive cross-scale feature fusion model to ensure the consistency of the constructed feature pyramid. To overcome the influence of uneven light, we build an illumination-invariant chromaticity space on the CSIM model, which is independent of the correlated color temperature. In addition, we adopt spatial attention modules, K-means clustering and the Mish activation function for further model optimization. The obtained experimental results show that the proposed CSIM produces excellent detection results for addressing the challenges derived from large-scale deformations and the illumination changes encountered during traffic surveillance. Compared with state-of-the-art object detection methods on public datasets, our proposed model has achieved competitive results in robust object detection tasks in traffic surveillance scenarios.

Deep learning and ultrasound feature fusion model predicts the malignancy of complex cystic and solid breast nodules with color Doppler images

This study aimed to evaluate the performance of traditional-deep learning combination model based on Doppler ultrasound for diagnosing malignant complex cystic and solid breast nodules. A conventional statistical prediction model based on the ultrasound features and basic clinical information was established. A deep learning prediction model was used to train the training group images and derive the deep learning prediction model. The two models were validated, and their accuracy rates were compared using the data and images of the test group, respectively. A logistic regression method was used to combine the two models to derive a combination diagnostic model and validate it in the test group. The diagnostic performance of each model was represented by the receiver operating characteristic curve and the area under the curve. In the test cohort, the diagnostic efficacy of the deep learning model was better than traditional statistical model, and the combined diagnostic model was better and outperformed the other two models (combination model vs traditional statistical model: AUC: 0.95 > 0.70, P = 0.001; combination model vs deep learning model: AUC: 0.95 > 0.87, P = 0.04). A combination model based on deep learning and ultrasound features has good diagnostic value.

Mental health analysis for college students based on pattern recognition and reinforcement learning

AbstractMental health education for college students is an important part of ideological and political work in colleges and universities, which is related to the physical and mental health and long‐term development of college students. In particular, the nature of the work of counselors endows them with unique advantages in psychological education. Based on the insufficiency of unimodal data features, we propose a method for analyzing the mental health of college students based on multimodal social‐affective classification. At the same time, we design a multimodal data fusion model, which takes text data as the main body and uses text and images to jointly classify the main body's emotion. First, we use the Bidirectional Encoder Representations from Transformers (BERT) pre‐training model to extract text features and obtain corresponding text vectors. Second, we utilize the Visual Geometry Group (VGG16) model trained on the ImageNet dataset as a pre‐training model to obtain image features. Third, we combine the modality features extracted by the two models to complete the final mental health classification task. Experimental results show that our proposed multimodal feature fusion model exhibits good performance on both constructed and public datasets.

SOH estimation and RUL prediction of lithium batteries based on multidomain feature fusion and CatBoost model

AbstractIn this paper, a lithium‐ion battery State of Health (SOH) estimation algorithm is proposed based on the fusion of multidomain features and the application of a CatBoost model. The aim is to address the issue of low prediction accuracy in SOH caused by the utilization of single‐feature extraction techniques. The algorithm encompasses the extraction of various features from the original charge–discharge data, including time‐domain, frequency‐domain, entropy, and time‐series features. Following the evaluation of feature importance, a feature selection process is conducted to eliminate redundant features that provide a limited contribution to the predictive results. Subsequently, a multiple‐set discriminative correlation analysis is employed to integrate high‐dimensional features. To attain accurate predictions, the CatBoost model is further optimized through the utilization of a sparrow search algorithm. Experimental results demonstrate that the proposed algorithm achieves accurate SOH estimations within individual batteries, as evidenced by mean square error values consistently below 4e−4 and goodness‐of‐fit values exceeding or equal to 0.98. Additionally, the algorithm exhibits reliable prediction capabilities across different batteries operating under the same charge/discharge strategy. Comparative analysis indicates that the adoption of the multidomain feature fusion approach yields improved prediction accuracy in contrast to the utilization of a single feature extraction method.

ASFL-YOLOX: an adaptive spatial feature fusion and lightweight detection method for insect pests of the Papilionidae family.

Insect pests from the family Papilionidae (IPPs) are a seasonal threat to citrus orchards, causing damage to young leaves, affecting canopy formation and fruiting. Existing pest detection models used by orchard plant protection equipment lack a balance between inference speed and accuracy. To address this issue, we propose an adaptive spatial feature fusion and lightweight detection model for IPPs, called ASFL-YOLOX. Our model includes several optimizations, such as the use of the Tanh-Softplus activation function, integration of the efficient channel attention mechanism, adoption of the adaptive spatial feature fusion module, and implementation of the soft Dlou non-maximum suppression algorithm. We also propose a structured pruning curation technique to eliminate unnecessary connections and network parameters. Experimental results demonstrate that ASFL-YOLOX outperforms previous models in terms of inference speed and accuracy. Our model shows an increase in inference speed by 29 FPS compared to YOLOv7-x, a higher mAP of approximately 10% than YOLOv7-tiny, and a faster inference frame rate on embedded platforms compared to SSD300 and Faster R-CNN. We compressed the model parameters of ASFL-YOLOX by 88.97%, reducing the number of floating point operations per second from 141.90G to 30.87G while achieving an mAP higher than 95%. Our model can accurately and quickly detect fruit tree pest stress in unstructured orchards and is suitable for transplantation to embedded systems. This can provide technical support for pest identification and localization systems for orchard plant protection equipment.

Deep radiomic model based on the sphere–shell partition for predicting treatment response to chemotherapy in lung cancer

BackgroundThe prognosis of chemotherapy is important in clinical decision-making for non-small cell lung cancer (NSCLC) patients. ObjectivesTo develop a model for predicting treatment response to chemotherapy in NSCLC patients from pre-chemotherapy CT images. Materials and MethodsThis retrospective multicenter study enrolled 485 patients with NSCLC who received chemotherapy alone as a first-line treatment. Two integrated models were developed using radiomic and deep-learning-based features. First, we partitioned pre-chemotherapy CT images into spheres and shells with different radii around the tumor (0–3, 3–6, 6–9, 9–12, 12–15 mm) containing intratumoral and peritumoral regions. Second, we extracted radiomic and deep-learning-based features from each partition. Third, using radiomic features, five sphere–shell models, one feature fusion model, and one image fusion model were developed. Finally, the model with the best performance was validated in two cohorts. ResultsAmong the five partitions, the model of 9–12 mm achieved the highest area under the curve (AUC) of 0.87 (95% confidence interval: 0.77–0.94). The AUC was 0.94 (0.85–0.98) for the feature fusion model and 0.91 (0.82–0.97) for the image fusion model. For the model integrating radiomic and deep-learning-based features, the AUC was 0.96 (0.88–0.99) for the feature fusion method and 0.94 (0.85–0.98) for the image fusion method. The best-performing model had an AUC of 0.91 (0.81–0.97) and 0.89 (0.79–0.93) in two validation sets, respectively. ConclusionsThis integrated model can predict the response to chemotherapy in NSCLC patients and assist physicians in clinical decision-making.

Sentiment Analysis of Medium and Long Text Based on Feature Fusion Model

Sentiment Analysis of Medium and Long Text Based on Feature Fusion Model

MCFF-MTDDI: multi-channel feature fusion for multi-typed drug-drug interaction prediction.

Adverse drug-drug interactions (DDIs) have become an increasingly serious problem in the medical and health system. Recently, the effective application of deep learning and biomedical knowledge graphs (KGs) have improved the DDI prediction performance of computational models. However, the problems of feature redundancy and KG noise also arise, bringing new challenges for researchers. To overcome these challenges, we proposed a Multi-Channel Feature Fusion model for multi-typed DDI prediction (MCFF-MTDDI). Specifically, we first extracted drug chemical structure features, drug pairs' extra label features, and KG features of drugs. Then, these different features were effectively fused by a multi-channel feature fusion module. Finally, multi-typed DDIs were predicted through the fully connected neural network. To our knowledge, we are the first to integrate the extra label information into KG-based multi-typed DDI prediction; besides, we innovatively proposed a novel KG feature learning method and a State Encoder to obtain target drug pairs' KG-based features which contained more abundant and more key drug-related KG information with less noise; furthermore, a Gated Recurrent Unit-based multi-channel feature fusion module was proposed in an innovative way to yield more comprehensive feature information about drug pairs, effectively alleviating the problem of feature redundancy. We experimented with four datasets in the multi-class and the multi-label prediction tasks to comprehensively evaluate the performance of MCFF-MTDDI for predicting interactions of known-known drugs, known-new drugs and new-new drugs. In addition, we further conducted ablation studies and case studies. All the results fully demonstrated the effectiveness of MCFF-MTDDI.

Prediction of brain tumor recurrence location based on multi-modal fusion and nonlinear correlation learning.

Brain tumor is one of the leading causes of cancer death. The high-grade brain tumors are easier to recurrent even after standard treatment. Therefore, developing a method to predict brain tumor recurrence location plays an important role in the treatment planning and it can potentially prolong patient's survival time. There is still little work to deal with this issue. In this paper, we present a deep learning-based brain tumor recurrence location prediction network. Since the dataset is usually small, we propose to use transfer learning to improve the prediction. We first train a multi-modal brain tumor segmentation network on the public dataset BraTS 2021. Then, the pre-trained encoder is transferred to our private dataset for extracting the rich semantic features. Following that, a multi-scale multi-channel feature fusion model and a nonlinear correlation learning module are developed to learn the effective features. The correlation between multi-channel features is modeled by a nonlinear equation. To measure the similarity between the distributions of original features of one modality and the estimated correlated features of another modality, we propose to use Kullback-Leibler divergence. Based on this divergence, a correlation loss function is designed to maximize the similarity between the two feature distributions. Finally, two decoders are constructed to jointly segment the present brain tumor and predict its future tumor recurrence location. To the best of our knowledge, this is the first work that can segment the present tumor and at the same time predict future tumor recurrence location, making the treatment planning more efficient and precise. The experimental results demonstrated the effectiveness of our proposed method to predict the brain tumor recurrence location from the limited dataset.

Emotion classification using EEG signal for women safety application based on deep learning

An emotion is a conscious logical response that varies for different situations in women’s life. These mental responses are caused by physiological, cognitive, and behavioral changes. Gender-based violence undermines the participation of women in decision-making, resulting in a decline in their quality of life. More accurate and automatic classification of women’s emotions can enhance human-computer interfaces and security in real time. There are some wearable technologies and mobile applications that claim to ensure the safety of women. However, they rely on limited social action and are ineffective at ensuring women’s safety when and where it is needed. In this work, a novel CDB-LSTM network has been proposed to accurately classify the emotions of women in seven different classes. The electroencephalogram (EEG) offers non-radioactive methods of identifying emotions. Initially, the EEG signals are preprocessed and they are converted into images via Time-Frequency Representation (TPR). A smoothed pseudo-Wigner-Ville distribution (SPWVD) is employed to convert the EEG time-domain signals into input images. Consequently, these converted images are given as input to the Convolutional Deep Belief Network (CDBN) for extracting the most relevant features. Finally, Bi-directional LSTM is used for classifying the emotions of women into seven classes namely: happy, relax, sad, fear, anxiety, anger, and stress. The proposed CDB-LSTM network preserves the high accuracy range of 97.27% in the validation phase. The proposed CDB-LSTM network improves the overall accuracy by 6.20% 32.98% 6.85% and 3.30% better than CNN-LSTM, Multi-domain feature fusion model, GCNN-LSTM and CNN with SVM and DT respectively.

Data association for maneuvering targets through a combined siamese network and XGBoost model

Data association plays an important role in forming target tracks when false alarms exist. Its accuracy is key to reducing the computational burden of the combinatorial explosion problem inherent to target tracking in environments where false alarms are densely distributed. Traditional methods are usually developed according to some assumed motion models and suffer performance degradation when there is mismatch between assumed models and the actual motion, which may often be the case for maneuvering targets. To cope with this problem, we propose a data-driven method that learns the association criteria directly from data. By use of the domain knowledge and a convolutional Siamese network, features of different attributes present in the observations are extracted. Then the features are fused through the use of XGBoost. Simulation results show that the proposed method performs better than the traditional model-based methods in correlating observations of maneuvering targets and avoiding correlations of false alarms. By visualizing the decision process of the feature fusion model, it is found that the proposed method learns from the data to make comprehensive use of multiple features and does not simply set hard thresholds for the features. The computational complexity is also analyzed both theoretically and experimentally.

Deep learning for large-scale point cloud segmentation in tunnels considering causal inference

This work designs an integrated deep learning approach to accomplish a multi-label segmentation of various objects including seepage from 3D point clouds of tunnels. 3D laser scanning is first utilized to collect raw point clouds from the operating tunnels. An integration approach based on projection strategy and point-based method is developed. A causal inference-based data augmentation approach is proposed to enhance the segmentation performance between the seepage and the segment class. A Dempster–Shafer (D-S) evidence-based feature fusion model is established to combine the local feature with global features from different CNN models. A real cross-river tunnel section in China is applied as a case study for illustration. Results verify that (1) the integration framework shows an outperforming performance of seepage segmentation, and the Intersection over Union (IOU), Precision, Recall, and F1 score for seepage segmentation achieve 0.70, 0.896, 0.762, and 0.824, respectively; (2) the segmentation model achieves competitive results in multiple classes, and the segmentation F1 score of the cable, segment, pipe, powertrack and the track classes obtain 0.913, 0.911, 0.993, 0.898, and 0.963, respectively; (3) the segmentation model is superior to other SOTA point cloud segmentation algorithms in terms of evaluation metrics.

A multi-kernel and multi-scale learning based deep ensemble model for predicting recurrence of non-small cell lung cancer.

Predicting recurrence in patients with non-small cell lung cancer (NSCLC) before treatment is vital for guiding personalized medicine. Deep learning techniques have revolutionized the application of cancer informatics, including lung cancer time-to-event prediction. Most existing convolutional neural network (CNN) models are based on a single two-dimensional (2D) computational tomography (CT) image or three-dimensional (3D) CT volume. However, studies have shown that using multi-scale input and fusing multiple networks provide promising performance. This study proposes a deep learning-based ensemble network for recurrence prediction using a dataset of 530 patients with NSCLC. This network assembles 2D CNN models of various input slices, scales, and convolutional kernels, using a deep learning-based feature fusion model as an ensemble strategy. The proposed framework is uniquely designed to benefit from (i) multiple 2D in-plane slices to provide more information than a single central slice, (ii) multi-scale networks and multi-kernel networks to capture the local and peritumoral features, (iii) ensemble design to integrate features from various inputs and model architectures for final prediction. The ensemble of five 2D-CNN models, three slices, and two multi-kernel networks, using 5 × 5 and 6 × 6 convolutional kernels, achieved the best performance with an accuracy of 69.62%, area under the curve (AUC) of 72.5%, F1 score of 70.12%, and recall of 70.81%. Furthermore, the proposed method achieved competitive results compared with the 2D and 3D-CNN models for cancer outcome prediction in the benchmark studies. Our model is also a potential adjuvant treatment tool for identifying NSCLC patients with a high risk of recurrence.

Feature fusion model based health indicator construction and self-constraint state-space estimator for remaining useful life prediction of bearings in wind turbines

Feature fusion model based health indicator construction and self-constraint state-space estimator for remaining useful life prediction of bearings in wind turbines