Feature Extraction Fusion Research Articles

Multi-scale fusion and efficient feature extraction for enhanced sonar image object detection

Sonar imaging is an underwater detection technology that relies on the transmission and reception of acoustic pulse waves. This technique plays a crucial role in various domains including underwater archaeology, energy exploration, and oceanographic surveying. A primary challenge in sonar imaging is the low signal-to-noise ratio and significant noise interference, issues that are influenced by the constraints of equipment performance and the underwater environment. Traditional object detection techniques have been inadequate in effectively extracting deep features from sonar images possessing targets with complex structures and have also demonstrated shortcomings in the fusion processing of target features at multiple scales, thereby affecting the robustness and accuracy of object detection. To improve the performance of sonar image object detection, we propose an advanced detection framework that integrates efficient feature extraction with multi-scale feature fusion. We employ EfficientNet as the backbone network. EfficientNet exhibits excellent feature extraction capabilities through comprehensive adjustments of depth, width, and resolution. We introduce a dual-channel attention module that blends Squeeze-and-Excitation (SE) and Efficient Channel Attention (ECA) mechanisms to amplify the expression of crucial feature channels and suppress the lesser ones. Additionally, we utilize a modified bidirectional feature pyramid network (BiFPN) to strengthen the integration of features across different layers. Employing these methods, we amalgamate the features into a shared weights classification network and bounding box prediction network for accurate target class discernment and localization. The experimental outcomes provide evidence of the notable superlative nature of the proposed framework in sonar image object detection, effectively ameliorating detection performance amidst noise interference.

Multi-class plaque segmentation in Intravascular Ultrasound via inter-frame feature fusion and contrast feature extraction

Intravascular Ultrasound (IVUS) is an effective aid in diagnosing cardiovascular diseases, with plaque segmentation in IVUS images being a pivotal step in assessing vascular plaque risk. However, IVUS plaque segmentation encounters two major challenges: (1) the variability in plaque shapes makes it difficult to locate plaques accurately, and (2) distinguishing plaque types is challenging due to the similarity in visual features among different types. To tackle these challenges, we propose a novel framework for IVUS plaque segmentation. Our approach incorporates a segmentation network featuring a dual feature attention(DFA) module and a cross feature fusion(CFF) module. Additionally, we introduce contrastive learning to enhance plaque sensitivity and extract contrast features within our network. We evaluate our method using a private IVUS dataset, achieving the state-of-the-art performance with a Dice score of 84.79% and a 14.17 HD value. Our approach represents significant progress in the field of multi-class IVUS plaque segmentation, making it a more effective tool to assist doctors in diagnosing vascular plaques.

MSFNet‐2SE: A multi‐scale fusion convolutional network for Alzheimer's disease classification on magnetic resonance images

AbstractAlzheimer's disease (AD) is an irreversible neurodegenerative disease, and the early diagnosis and effective intervention of AD is essential for patients and doctors. Intelligence diagnosis based on magnetic resonance imaging and deep learning has become one of the useful methods for AD identification. To improve the diagnosis effect of AD in early stages, a novel multi‐scale classification model of AD, called MSFNet‐2SE, is proposed. First, a new multi‐scale fusion (MSF) feature extraction module is designed based on the idea of the feature maps split into feature subsets of Res2Net. Second, a channel attention module is embedded into the MSF module through integrating two Squeeze‐and‐Excitation (SE) blocks. Finally, a gradient centralization Adam optimizer is used to improve the model classification performance. Experimental results illustrate that, compared with other available state‐of‐the‐art classification models of AD, the proposed model has excellent classification performance. It is helpful to improve the clinical diagnosis efficiency.

Isolated sign language recognition through integrating pose data and motion history images

This article presents an innovative approach for the task of isolated sign language recognition (SLR); this approach centers on the integration of pose data with motion history images (MHIs) derived from these data. Our research combines spatial information obtained from body, hand, and face poses with the comprehensive details provided by three-channel MHI data concerning the temporal dynamics of the sign. Particularly, our developed finger pose-based MHI (FP-MHI) feature significantly enhances the recognition success, capturing the nuances of finger movements and gestures, unlike existing approaches in SLR. This feature improves the accuracy and reliability of SLR systems by more accurately capturing the fine details and richness of sign language. Additionally, we enhance the overall model accuracy by predicting missing pose data through linear interpolation. Our study, based on the randomized leaky rectified linear unit (RReLU) enhanced ResNet-18 model, successfully handles the interaction between manual and non-manual features through the fusion of extracted features and classification with a support vector machine (SVM). This innovative integration demonstrates competitive and superior results compared to current methodologies in the field of SLR across various datasets, including BosphorusSign22k-general, BosphorusSign22k, LSA64, and GSL, in our experiments.

A cardiac MRI motion artifact reduction method based on edge enhancement network

Cardiac magnetic resonance imaging (MRI) usually requires a long acquisition time. The movement of the patients during MRI acquisition will produce image artifacts. Previous studies have shown that clear MR image texture edges are of great significance for pathological diagnosis. In this paper, a motion artifact reduction method for cardiac MRI based on edge enhancement network is proposed. Firstly, the four-plane normal vector adaptive fractional differential mask is applied to extract the edge features of blurred images. The four-plane normal vector method can reduce the noise information in the edge feature maps. The adaptive fractional order is selected according to the normal mean gradient and the local Gaussian curvature entropy of the images. Secondly, the extracted edge feature maps and blurred images are input into the de-artifact network. In this network, the edge fusion feature extraction network and the edge fusion transformer network are specially designed. The former combines the edge feature maps with the fuzzy feature maps to extract the edge feature information. The latter combines the edge attention network and the fuzzy attention network, which can focus on the blurred image edges. Finally, extensive experiments show that the proposed method can obtain higher peak signal-to-noise ratio and structural similarity index measure compared to state-of-art methods. The de-artifact images have clear texture edges.

Semantics-enhanced discriminative descriptor learning for LiDAR-based place recognition

LiDAR-based place recognition (LPR) aims to localize autonomous vehicles and mobile robots relative to pre-built maps or retrieve previously visited places. However, the complexity of real-world scenes and changes in viewpoint are significant challenges for place recognition. As high-level information, semantics makes it easier to distinguish geometrically similar scene situations. Unlike most existing methods that rely solely on a single type of information (geometric or semantic) to construct scene descriptors, we consider the complementary nature of the semantic and geometric information and propose a semantics-enhanced discriminative feature learning method for LPR. Specifically, we first develop a Multi-layer Fusion Feature Extraction Network (MFFEN) based on the transformer encoder to hierarchically fuse local geometric and semantic information and utilize contextual information for extracting discriminative local features. To obtain semantic information, we introduce the dynamic graph convolution network to extract local semantic features with local relations. In addition, to weaken the interference of redundancy and dynamic objects in the scene, we design a semantics-guided local attention network (SLAN) to focus on salient local features that are helpful for recognizing scenes, thereby enhancing the descriptive ability of the global descriptor. Extensive experiments on public datasets KITTI and KITTI-360 demonstrate that the proposed method performs better than recent LiDAR-based methods on the 3D place recognition task. For instance, it achieves the mean F1max score of 96.9% on the KITTI dataset, surpassing the strongest prior model by 2.7%.

The multi-channel signals based tensor sparse representation classification method for fault diagnosis of high-speed train

The transmission system is a key component to ensure the stable operation of high-speed trains. Thus, it is significant to monitor its condition to ensure the operation safety. Nowadays sparse representation is widely used in fault diagnosis. However, as the number of sensors is increasing, the existing method destroys the internal structure of multi-channel signals and cannot effectively deal with the fault diagnosis of multi-channel signals in parallel. Therefore, this article extends the existing sparse representation method to tensor space to extract the coupling information between channels and realize the fault diagnosis of multi-channel. First, a tensor sparse representation model is proposed to achieve data-level multi-channel signal fusion and complete inter-channel fault feature extraction. Then, a multimodal dictionary learning algorithm is proposed to adaptively design the data-driven dictionary to achieve data-driven feature extraction. Finally, a tensor sparse representation classification method is proposed to achieve the purpose of intelligent diagnosis. Fault experiments verify the effectiveness and superiority of the method.

A Multichannel-Based Deep Learning Framework for Ocean SAR Scene Classification

High-resolution synthetic aperture radars (SARs) are becoming an indispensable environmental monitoring system to capture the important geophysical phenomena on the earth and sea surface. However, there is a lack of comprehensive models that can orchestrate such large-scale datasets from numerous satellite missions such as GaoFen-3 and Sentinel-1. In addition, these SAR images of different ocean scenes need to convey a variety of high-level classification features in oceanic and atmospheric phenomena. In this study, we propose a multichannel neural network (MCNN) that supports oceanic SAR scene classification for limited oceanic data samples according to multi-feature fusion, data augmentation, and multichannel feature extraction. To exploit the multichannel semantics of SAR scenes, the multi-feature fusion module effectively combines and reshapes the spatiotemporal SAR images to preserve their structural properties. This fine-grained feature augmentation policy is extended to improve the data quality so that the classification model is less vulnerable to both small- and large-scale data. The multichannel feature extraction also aggregates different oceanic features convolutionally extracted from ocean SAR scenes to improve the classification accuracy of oceanic phenomena with different scales. Through extensive experimental analysis, our MCNN framework has demonstrated a commendable classification performance, achieving an average precision rate of 96%, an average recall rate of 95%, and an average F-score of 95% across ten distinct oceanic phenomena. Notably, it surpasses two state-of-the-art classification techniques, namely, AlexNet and CMwv, by margins of 23.7% and 18.3%, respectively.

Robust indoor localization based on multi-modal information fusion and multi-scale sequential feature extraction

Magnetic-assisted indoor localization has attracted significant attention because of its commercial and social values. However, it is challenging to construct a robust and accurate system due to the severe feature ambiguity caused by different users, mobiles, attitudes, and moving speeds. In order to cope with this issue, we first propose to fuse magnetic with the multi-modal features, including Bluetooth low energy (BLE), and context information of continuous predictions, to improve the feature discrimination with more valuable localization clues. Then, we extract more orientation-insensitive magnetic features and remove the Direct Current (DC) component of the sequence can reduce the feature ambiguity caused by different holding attitudes, devices, and users. After that, we propose an online data augmentation algorithm to automatically generate a sufficient amount of various-speed sequences based on only one dense sampling benchmark sequence, thus reducing the influence of multi-scale sequences caused by different moving speeds. Finally, we propose a multi-branch and attention mechanism-based end-to-end localization model to extract and efficiently fuse the significant features of the multi-modal data for accurate localization. We evaluate the performance of the proposed localization system (DamLoc) in a typical indoor environment based on extensive experiments. Evaluation results showcase that DamLoc more robust for diverse heterogeneous factors, and can support about 63% improvement compared to state-of-the-art methods. It is worth pointing out that the BLE in our work can be replaced with other signals, such as Wireless Fidelity (Wi-Fi), which is more general than other fusion-based localization.

Robustness defect detection: Improving the performance of surface defect detection in interference environment

In industrial production environments, the performance of surface defect detection methods for products is affected by disturbances in the equipment, manufacturing processes, and the environment, resulting in poor robustness of the training model. In this paper, an improved defect detection method based on feature extraction fusion and dynamic label assignment is proposed. Three improved network structures, i.e., the adaptive feature recognition convolution (AFRC), path aggregation balanced feature pyramid, and dynamic label assignment, are proposed for the feature extraction, feature fusion, and detection phases of the existing Faster RCNN model, respectively. The AFRC is employed to improve the feature extraction ability of the model for irregular defects, the path balance feature pyramid is utilized to enhance the recognition ability of the model for different levels of feature maps, and the dynamic label assignment is employed to improve the quality of the candidate frames. Under disturbed environments, the results of the proposed improvements outperform the benchmark faster RCNN model in terms of average precision (AP) by 4.1%, 3.7%, and 4.2% on an aluminum surface defect dataset and by 1.8%, 2.2%, and 3.7% AP on a magnetic tile surface defect dataset. The experimental results demonstrate that the proposed scheme effectively improves the robustness of the baseline model in environments with various disturbances.

An investigation of methods to improve the accuracy of classical dance steps based on machine vision recognition technology

Abstract Methods that use motion capture data to analyze human behavior are highly interpretable and offer significant advantages in vision-based dance step analysis. In this study, we propose a fusion feature extraction method that describes the rotational information embedded in the skeleton and combines the advantages of single-skeleton feature extraction and Li group feature extraction methods for recognition. The feasibility and validity of the proposed model are verified by conducting recognition experiments on BVH data of classical dance steps performed by subjects. In the ablation experiments, the average accuracy of the classical dance step recognition model based on fused features for step movement recognition is improved by 3.01% and 1.86% compared with the neural network model and the Lie group network model, respectively. It has been proven that adding rotation information to skeletal features can effectively differentiate 3D motion trajectories in similar dance steps. Furthermore, the dance movement trajectories derived from this model are very clear and can be utilized to direct the correct joint point positions throughout the movement. The recognition accuracy of the fusion feature extraction-based recognition method for all seven classical dance basic foot positions is greater than 90%, which reflects the accuracy of the proposed machine vision model in recognizing classical dance steps.

Fusion of Audio and Vibration Signals for Bearing Fault Diagnosis Based on a Quadratic Convolution Neural Network.

In this paper, a quadratic convolution neural network (QCNN) using both audio and vibration signals is utilized for bearing fault diagnosis. Specifically, to make use of multi-modal information for bearing fault diagnosis, the audio and vibration signals are first fused together using a 1 × 1 convolution. Then, a quadratic convolution neural network is applied for the fusion feature extraction. Finally, a decision module is designed for fault classification. The proposed method utilizes the complementary information of audio and vibration signals, and is insensitive to noise. The experimental results show that the accuracy of the proposed method can achieve high accuracies for both single and multiple bearing fault diagnosis in the noisy situations. Moreover, the combination of two-modal data helps improve the performance under all conditions.

Improvement of the YOLOv5 Model in the Optimization of the Brown Spot Disease Recognition Algorithm of Kidney Bean

The kidney bean is an important cash crop whose growth and yield are severely affected by brown spot disease. Traditional target detection models cannot effectively screen out key features, resulting in model overfitting and weak generalization ability. In this study, a Bi-Directional Feature Pyramid Network (BiFPN) and Squeeze and Excitation (SE) module were added to a YOLOv5 model to improve the multi-scale feature fusion and key feature extraction abilities of the improved model. The results show that the BiFPN and SE modules show higher heat in the target location region and pay less attention to irrelevant environmental information in the non-target region. The detection Precision, Recall, and mean average Precision (mAP@0.5) of the improved YOLOv5 model are 94.7%, 88.2%, and 92.5%, respectively, which are 4.9% higher in Precision, 0.5% higher in Recall, and 25.6% higher in the mean average Precision compared to the original YOLOv5 model. Compared with the YOLOv5-SE, YOLOv5-BiFPN, FasterR-CNN, and EfficientDet models, detection Precision improved by 1.8%, 3.0%, 9.4%, and 9.5%, respectively. Moreover, the rate of missed and wrong detection in the improved YOLOv5 model is only 8.16%. Therefore, the YOLOv5-SE-BiFPN model can more effectively detect the brown spot area of kidney beans.

Intelligent fault diagnosis of rotating machinery under variable working conditions based on deep transfer learning with fusion of local and global time–frequency features

With the development of deep learning methods, the data-driven fault diagnosis methods have attracted a great deal of interest. However, as for the data-driven fault diagnosis methods, technology has to overcome various difficulties in the practical industrial scenarios, such as variable working conditions, insufficient effective samples, and environmental noise interference. Combining with the time–frequency analysis of vibration signals, a domain adaptation fault diagnosis model based on ResNet and Transformer (DAFDMRT) is proposed in this work, aiming to solve the problems encountered by current rotating machinery fault diagnosis methods in the field of application. Firstly, the vibration signal is processed by wavelet packet transform and the time–frequency information grayscale maps is constructed. Next, a deep fusion feature extraction network combining ResNet and Transformer encoder, is designed for the extraction and fusion of the local and global features of multi-scale time–frequency information. Finally, the multi-kernel maximum mean discrepancy is applied to measure and minimize the distribution difference between the deep features of source and target domain, thereby improving the diagnostic performance of the diagnosis model in variable working conditions. In this work, comparative experiments are conducted as for bearing and gearbox datasets under variable working conditions. The results indicate that DAFDMRT can show excellent performances in terms of fault diagnosis and generalization ability.

Multiple Distilling-based spatial-temporal attention networks for unsupervised human action recognition

Unsupervised action recognition based on spatiotemporal fusion feature extraction has attracted much attention in recent years. However, existing methods still have several limitations: (1) The long-term dependence relationship is not effectively extracted at the time level. (2) The high-order motion relationship between non-adjacent nodes is not effectively captured at the spatial level. (3) The model complexity is too high when the cascade layer input sequence is long, or there are many key points. To solve these problems, a Multiple Distilling-based spatial-temporal attention (MD-STA) networks is proposed in this paper. This model can extract temporal and spatial features respectively and fuse them. Specifically, we first propose a Screening Self-attention (SSA) module; this module can find long-term dependencies in distant frames and high-order motion patterns between non-adjacent nodes in a single frame through a sparse metric on dot product pairs. Then, we propose the Frames and Keypoint-Distilling (FKD) module, which uses extraction operations to halve the input of the cascade layer to eliminate invalid key points and time frame features, thus reducing time and memory complexity. Finally, the Dim-reduction Fusion (DRF) module is proposed to reduce the dimension of existing features to further eliminate redundancy. Numerous experiments were conducted on three distinct datasets: NTU-60, NTU-120, and UWA3D, showing that MD-STA achieves state-of-the-art standards in skeleton-based unsupervised action recognition.

Dual-band polarimetric HRRP recognition via a brain-inspired multi-channel fusion feature extraction network.

Radar high-resolution range profile (HRRP) provides geometric and structural information of target, which is important for radar automatic target recognition (RATR). However, due to the limited information dimension of HRRP, achieving accurate target recognition is challenging in applications. In recent years, with the rapid development of radar components and signal processing technology, the acquisition and use of target multi-frequency and polarization scattering information has become a significant way to improve target recognition performance. Meanwhile, deep learning inspired by the human brain has shown great promise in pattern recognition applications. In this paper, a Multi-channel Fusion Feature Extraction Network (MFFE-Net) inspired by the human brain is proposed for dual-band polarimetric HRRP, aiming at addressing the challenges faced in HRRP target recognition. In the proposed network, inspired by the human brain's multi-dimensional information interaction, the similarity and difference features of dual-frequency HRRP are first extracted to realize the interactive fusion of frequency features. Then, inspired by the human brain's selective attention mechanism, the interactive weights are obtained for multi-polarization features and multi-scale representation, enabling feature aggregation and multi-scale fusion. Finally, inspired by the human brain's hierarchical learning mechanism, the layer-by-layer feature extraction and fusion with residual connections are designed to enhance the separability of features. Experiments on simulated and measured datasets verify the accurate recognition capability of MFFE-Net, and ablative studies are conducted to confirm the effectiveness of components of network for recognition.

Modality preserving U-Net for segmentation of multimodal medical images.

Recent advances in artificial intelligence and digital image processing have inspired the use of deep neural networks for segmentation tasks in multimodal medical imaging. Unlike natural images, multimodal medical images contain much richer information regarding different modal properties and therefore present more challenges for semantic segmentation. However, there is no report on systematic research that integrates multi-scaled and structured analysis of single-modal and multimodal medical images. We propose a deep neural network, named as Modality Preserving U-Net (MPU-Net), for modality-preserving analysis and segmentation of medical targets from multimodal medical images. The proposed MPU-Net consists of a modality preservation encoder (MPE) module that preserves the feature independency among the modalities and a modality fusion decoder (MFD) module that performs a multiscale feature fusion analysis for each modality in order to provide a rich feature representation for the final task. The effectiveness of such a single-modal preservation and multimodal fusion feature extraction approach is verified by multimodal segmentation experiments and an ablation study using brain tumor and prostate datasets from Medical Segmentation Decathlon (MSD). The segmentation experiments demonstrated the superiority of MPU-Net over other methods in the segmentation tasks for multimodal medical images. In the brain tumor segmentation tasks, the Dice scores (DSCs) for the whole tumor (WT), the tumor core (TC) and the enhancing tumor (ET) regions were 89.42%, 86.92%, and 84.59%, respectively. In the meanwhile, the 95% Hausdorff distance (HD95) results were 3.530, 4.899 and 2.555, respectively. In the prostate segmentation tasks, the DSCs for the peripheral zone (PZ) and the transitional zone (TZ) of the prostate were 71.20% and 90.38%, respectively. In the meanwhile, the 95% HD95 results were 6.367 and 4.766, respectively. The ablation study showed that the combination of single-modal preservation and multimodal fusion methods improved the performance of multimodal medical image feature analysis. In the segmentation tasks using brain tumor and prostate datasets, the MPU-Net method has achieved the improved performance in comparison with the conventional methods, indicating its potential application for other segmentation tasks in multimodal medical images.

FCAN–XGBoost: A Novel Hybrid Model for EEG Emotion Recognition

In recent years, artificial intelligence (AI) technology has promoted the development of electroencephalogram (EEG) emotion recognition. However, existing methods often overlook the computational cost of EEG emotion recognition, and there is still room for improvement in the accuracy of EEG emotion recognition. In this study, we propose a novel EEG emotion recognition algorithm called FCAN–XGBoost, which is a fusion of two algorithms, FCAN and XGBoost. The FCAN module is a feature attention network (FANet) that we have proposed for the first time, which processes the differential entropy (DE) and power spectral density (PSD) features extracted from the four frequency bands of the EEG signal and performs feature fusion and deep feature extraction. Finally, the deep features are fed into the eXtreme Gradient Boosting (XGBoost) algorithm to classify the four emotions. We evaluated the proposed method on the DEAP and DREAMER datasets and achieved a four-category emotion recognition accuracy of 95.26% and 94.05%, respectively. Additionally, our proposed method reduces the computational cost of EEG emotion recognition by at least 75.45% for computation time and 67.51% for memory occupation. The performance of FCAN–XGBoost outperforms the state-of-the-art four-category model and reduces computational costs without losing classification performance compared with other models.

Automatic detection of microaneurysms in fundus images based on multiple preprocessing fusion to extract features

Diabetic retinopathy (DR) is the main cause of new cases of blindness, and retinal microaneurysm (MA) is one of the early clinical manifestations of DR. However, it is difficult to directly observe MA with naked eyes because of their small size and poor contrast with the surrounding background. This paper presents a new method for automatic detection of MAs from fundus images. It mainly includes four steps: image preprocessing, candidate region extraction, feature extraction and classification. At the stage of feature extraction, different from traditional methods, we propose a feature extraction method based on multi-preprocessing fusion. In this paper, features are extracted from the original green channel image and the original gray image respectively, and then different processing methods are used to extract the same features from multiple pre-processed images. Finally, the extracted features are classified by multi-layer perceptron (MLP) classifier to determine whether the candidate object is microaneurysm (MA). The method is evaluated on e-ophtha-MA database and achieved high sensitivity. Using the FROC (free-response ROC) indicator, the detection achieves the F-score, accuracy, sensitivity, specificity and AUC (area under the curve) of 0.507, 0.940, 0.870, 0.938 and 0.9821, respectively, compared to the most advanced methods available. The proposed method of multi-preprocessing and fusion feature extraction is beneficial to classification and improves the detection performance.

A novel residual block: replace Conv1× 1 with Conv3×3 and stack more convolutions.

The residual structure has an important influence on the design of the neural network model. The neural network model based on residual structure has excellent performance in computer vision tasks. However, the performance of classical residual networks is restricted by the size of receptive fields, channel information, spatial information and other factors. In this article, a novel residual structure is proposed. We modify the identity mapping and down-sampling block to get greater effective receptive field, and its excellent performance in channel information fusion and spatial feature extraction is verified by ablation studies. In order to further verify its feature extraction capability, a non-deep convolutional neural network (CNN) was designed and tested on Cifar10 and Cifar100 benchmark platforms using a naive training method. Our network model achieves better performance than other mainstream networks under the same training parameters, the accuracy we achieved is 3.08 percentage point higher than ResNet50 and 1.38 percentage points higher than ResNeXt50. Compared with SeResNet152, it is 0.29 percentage point higher in the case of 50 epochs less training.