Feature Fusion Convolutional Neural Network Research Articles

Wind Turbine Bearing Failure Diagnosis Using Multi-Scale Feature Extraction and Residual Neural Networks with Block Attention

Wind turbine rolling bearings are crucial components for ensuring the reliability and stability of wind power systems. Their failure can lead to significant economic losses and equipment downtime. Therefore, the accurate diagnosis of bearing faults is of great importance. Although existing deep learning fault diagnosis methods have achieved certain results, they still face limitations such as inadequate feature extraction capabilities, insufficient generalization to complex working conditions, and ineffective multi-scale feature capture. To address these issues, this paper proposes an advanced fault diagnosis method named the two-stream feature fusion convolutional neural network (TSFFResNet-Net). Firstly, the proposed method combines the advantages of one-dimensional convolutional neural networks (1D-ResNet) and two-dimensional convolutional neural networks (2D-ResNet). It transforms one-dimensional vibration signals into two-dimensional images through the empirical wavelet transform (EWT) method. Then, parallel convolutional kernels in 1D-ResNet and 2D-ResNet are used to extract multi-scale features, respectively. Next, the Convolutional Block Attention Module (CBAM) is introduced to enhance the network’s ability to capture key features by focusing on important features in specific channels or spatial areas. After feature fusion, CBAM is introduced again to further enhance the effect of feature fusion, ensuring that the features extracted by different network branches can be effectively integrated, ultimately providing more accurate input features for the classification task of the fully connected layer. The experimental results demonstrate that the proposed method outperforms other traditional methods and advanced convolutional neural network models on different datasets. Compared with convolutional neural network models such as LeNet-5, AlexNet, and ResNet, the proposed method achieves a significantly higher accuracy on the test set, with a stable accuracy of over 99%. Compared with other models, it shows better generalization and stability, effectively improving the overall performance of rolling bearing vibration signal fault diagnosis. The method provides an effective solution for the intelligent fault diagnosis of wind turbine rolling bearings.

Impact Load Localization Based on Multi-Scale Feature Fusion Convolutional Neural Network.

In order to achieve impact load localization of complex structures such as ships, this paper proposes a multi-scale feature fusion convolutional neural network (MSFF-CNN) method for impact load localization. An end-to-end machine learning model is used, where the raw vibration signals of impact loads are directly fed into the network model to avoid the process of feature extraction. Automatic feature learning and feature concatenation of the signal are achieved through four independent convolutional layers, each using a different size of convolutional kernel. Data normalization and L2 regularization techniques are introduced to enhance the data and prevent overfitting. Classification and localization of impact loads are accomplished using a softmax classification layer. Validation experiments are carried out using a ship's stern compartment model. Our results show that the classification and localization accuracy of the impact load sample group of MSFF-CNN reaches 94.29% compared with a traditional CNN. The method further improves the ability of the network to extract state features, takes local perception and global vision into account, effectively improves the classification ability of the model, and has good prospects for engineering applications.

NKDFF-CNN: A convolutional neural network with narrow kernel and dual-view feature fusion for multitype gesture recognition based on sEMG

Deep learning algorithms have been widely applied to gesture recognition based on multi-channel surface electromyography (sEMG). However, the limitations in feature extraction capabilities of existing algorithms have restricted the performance of multitype gesture recognition. To address this challenge, we propose a novel sEMG-based gesture recognition algorithm, namely, Narrow Kernel and Dual-view Feature Fusion Convolutional Neural Network (NKDFF-CNN). Firstly, to overcome the issue of traditional square kernel convolution operation, which loses channel independence features, we employ the narrow kernel convolution in the model to learn time-related features in each independent channel of sEMG, resulting in obtaining representative correlation information between specific muscles and gestures. Then, the dual-view structure is used to capture both shallow and deep features, which are fused at the decision level. Thus, the multi-dimensional feature information is extracted. The NKDFF-CNN is further extended to ACCNKDFF-CNN by introducing acceleration signals for multimodal feature integration. Experimental validation on the NinaPro DB2 dataset demonstrates the superior classification performance of NKDFF-CNN, achieving 88.03 % accuracy for 49 hand gestures, outperforming other state-of-the-art MSFF-net. In addition, the ACCNKDFF-CNN model with multimodal feature information significantly improved the accuracy to 95.25 %. We also validated the proposed NKDFF-CNN on NinaPro DB3 with the disabled subjects and the NinaPro DB4 with healthy subjects. The results showcased that the NKDFF-CNN achieved advanced accuracies of 70.58 % and 85.91 % for the multitype hand gestures classification, respectively, showing the high generalization ability of the proposed model. As a consequence, the proposed NKDFF-CNN method achieved superior recognition performance in both accuracy and generality compared to other advanced models. Thus, it provides a reliable algorithm for research in fields such as rehabilitative medicine.

A novel multimodal feature fusion convolutional neural network to predict the mechanical properties of magnesium alloys

A novel multimodal feature fusion convolutional neural network (MFFCNN) model, based on the combined effects of texture, grain size, and grain morphology, is established to predict the mechanical properties of AZ31 alloys. Utilizing an image-based approach, texture is reconstructed and further optimized with a reconstruction coefficient, n. When n = 41, the model has strong predictive capabilities for tensile yield strength, ultimate tensile strength, and elongation, achieving goodness of fit (R2) values of approximately 0.95, 0.94, and 0.90, respectively. Therefore, this model offers new insights into quantitatively analyzing the microstructure-property relationship of Mg alloys.

Hyperspectral imaging combined with spectral-imagery feature fusion convolutional neural network to discriminate different geographical origins of wolfberries

Geographical origin significantly influences the composition of Chinese wolfberries. Establishing a rapid and effective model for origin traceability is crucial. In this study, visible-near infrared (Vis-NIR) hyperspectral imaging technology combined with the proposed Spectral-Imagery Feature Fusion Convolutional Neural Network (S-IFCNN) was developed to identify the origin of wolfberries for the first time. The process began with segmenting adherent wolfberries using a combination of minimum transformation and the watershed algorithm, enabling the extraction of both spectral and image data. We compared various preprocessing methods, input data types, and convolutional kernel sizes to assess their impact on the performance of 1DCNN and 2DCNN models. The optimal configurations were then integrated into the S-IFCNN for a comprehensive evaluation. The study demonstrates the S-IFCNN model's capability in leveraging spectral and image data from hyperspectral imaging for accurately identifying the geographical origins of wolfberries, achieving an accuracy of 91.99%. This research not only highlights the potential applications of the S-IFCNN model but also provides a theoretical framework for the use of hyperspectral imaging in tracing the geographical origins of wolfberries.

A bimodal feature fusion convolutional neural network for detecting obstructive sleep apnea/hypopnea from nasal airflow and oximetry signals

The most prevalent sleep-disordered breathing condition is Obstructive Sleep Apnea (OSA), which has been linked to various health consequences, including cardiovascular disease (CVD) and even sudden death. Therefore, early detection of OSA can effectively help patients prevent the diseases induced by it. However, many existing methods have low accuracy in detecting hypopnea events or even ignore them altogether. According to the guidelines provided by the American Academy of Sleep Medicine (AASM), two modal signals, namely nasal pressure airflow and pulse oxygen saturation (SpO2), offer significant advantages in detecting OSA, particularly hypopnea events. Inspired by this notion, we propose a bimodal feature fusion CNN model that primarily comprises of a dual-branch CNN module and a feature fusion module for the classification of 10-second-long segments of nasal pressure airflow and SpO2. Additionally, an Efficient Channel Attention mechanism (ECA) is incorporated into the second module to adaptively weight feature map of each channel for improving classification accuracy. Furthermore, we design an OSA Severity Assessment Framework (OSAF) to aid physicians in effectively diagnosing OSA severity. The performance of both the bimodal feature fusion CNN model and OSAF is demonstrated to be excellent through per-segment and per-patient experimental results, based on the evaluation of our method using two real-world datasets consisting of polysomnography (PSG) recordings from 450 subjects.

Multiscale Feature Fusion Convolutional Neural Network for Surface Damage Detection in Retired Steel Shafts

Abstract The detection of surface damage is an important part of the process before remanufacturing a retired steel shaft (RSS). Traditional damage detection is mainly done manually, which is time-consuming and error-prone. In recent years, computer vision methods have been introduced into the community of surface damage detection. However, some advanced typical object detection methods perform poorly in the detection of surface damage on RSS due to the complex surface background and rich diversity of damage patterns and scales. To address these issues, we propose a Faster R-CNN–based surface damage detection method for RSS. To improve the adaptability of the network, we endow it with a feature pyramid network (FPN) as well as adaptable multiscale information modifications to the region proposal network (RPN). In this paper, a detailed study of an FPN-based feature extraction network and the multiscale object detection network is conducted. Experimental results show that our method improves the mean average precision (mAP) score by 8.9% compared with the original Faster R-CNN for surface damage detection of RSS, and the average detection accuracy for small objects is improved by 18.2%. Compared with the current advanced object detection methods, our method is more advantageous for the detection of multiscale objects.

Low-Dose CT Image Denoising with a Residual Multi-scale Feature Fusion Convolutional Neural Network and Enhanced Perceptual Loss

Low-Dose CT Image Denoising with a Residual Multi-scale Feature Fusion Convolutional Neural Network and Enhanced Perceptual Loss

A multi‐scale feature fusion convolutional neural network for facial expression recognition

AbstractThis paper designs a new facial Expression recognition network called a multi‐scale feature Fusion Convolutional neural Network (EFCN). This network is proposed to solve two problems in the facial expression recognition task. First, there are many commonalities between faces of different expression categories, and the recognition task cannot be precisely performed when the commonality is greater than the individuality. Secondly, facial detail features have a significant impact on the final results of expression recognition, while the image detail features extracted by traditional convolutional neural networks are not sufficient. In order to address the above issues, the feature enhancement network (FEN) and the detail information enhancement module (DEM) are designed. The FEN fuses deep and shallow features. Accordingly, the feature map contains richer information, making it easy to identify the samples. The DEM extracts and fuses the features passed by the backbone network with multi‐scale features to enhance the network's ability to extract features from small regions of the face. We validated the proposed method on three datasets, RAF‐DB, CK+, and JAFFE, and achieved 84.50%, 97.86%, and 91.05% accuracy, respectively, and the experimental results showed the effectiveness of the proposed method in this paper. For example, on the JAFFE dataset, the recognition accuracy of this method surpasses the MLT method by 1.87%.

Deep transfer learning rolling bearing fault diagnosis method based on convolutional neural network feature fusion

Current deep-learning methods are often based on significantly large quantities of labeled fault data for supervised training. In practice, it is difficult to obtain samples of rolling bearing failures. In this paper, a transfer learning-based feature fusion convolutional neural network approach for bearing fault diagnosis is proposed. Specifically, the raw vibration signal features and the corresponding time-frequency image features of the input data are extracted by a one-dimensional convolutional neural network and a pre-trained ConvNeXt, respectively, and connected by a feature fusion strategy. Then, the fine-tuning method based on transfer learning can effectively reduce the reliance on labeled samples in the target domain. A wide convolution kernel is introduced in the time-domain signal feature extraction to increase the receptive field, which is combined with the channel attention mechanism to further optimize the feature quality. Finally, two common bearing datasets are utilized for fault diagnosis experiments. The experimental results show that the proposed model achieves an average accuracy of more than 98.63% in both cross-working conditions and cross-device diagnosis tasks. Meanwhile, anti-noise experiments and ablation experiments further validate the accuracy and robustness of the proposed method.

Deep learning-based end-to-end 3D depth recovery from a single-frame fringe pattern with the MSUNet++ network.

Deep learning (DL)-based single-frame fringe pattern to 3D depth reconstruction methods have aroused extensive research interest. The goal is to estimate high-precision 3D shape from a single frame of fringe pattern with limited information. Therefore, the purpose of this work attempts to propose an end-to-end DL-based 3D reconstruction method from the single fringe pattern with excellent capability of achieving high accuracy depth recovery and geometry details preservation of tested objects. We construct a multi-scale feature fusion convolutional neural network (CNN) called MSUNet++, which incorporates discrete wavelet transform (DWT) in data preprocessing for extracting high-frequency signals of fringe patterns as input of the network. Additionally, a loss function that combines structural similarity with edge perception is established. Through these measures, high-frequency geometry details of the reconstruction results can be obviously enhanced, while the geometric shape can be effectively maintained. Ablation experiments are involved in validating the effectiveness of our proposed solution. 3D reconstructed results and analysis of generalization experiments on different tested samples imply that the proposed method in this research enjoys capabilities of higher accuracy, better detail preservation, and robustness in comparison with the compared methods.

Facial feature fusion convolutional neural network for driver fatigue detection

Driver fatigue detection is critical for preventing traffic accidents. However, in real driving scenarios, low-quality input situations often occur due to occlusion, large-turn head deformation, and inconsistent eye states, which introduce noise and degrade the performance of single-stream networks. To address this, we propose a multi-stream facial feature fusion convolution neural network (FFF–CNN) that enhances performance on low-quality inputs. The FFF-CNN includes a global face stream and two local eye streams, with feature fusion used for eye state classification. To facilitate feature fusion, we implemented two preprocessing modules, the feature attention module (FAM) and the stream interaction module (SIM). FAM emphasizes important features in the global face stream, while SIM corrects information from the two local eye streams. The proposed FFF-CNN achieved good performance on both our dataset and the public Closed Eyes in the Wild (CEW) dataset. Compared to single-stream networks, the FFF-CNN improved performance by 4.95% and 4.27% on our dataset and the CEW dataset, respectively. Furthermore, it achieved state-of-the-art performance on the CEW dataset with an accuracy of 98.35%. The effectiveness of the proposed FAM and SIM has been verified by experiments. Notably, our experimental results show that the proposed FFF-CNN effectively improved fatigue detection performance on low-quality input, approaching the performance on normal situations.

Multi-Scale Feature Fusion Convolutional Neural Networks for Fault Diagnosis of Electromechanical Actuator

Airborne electromechanical actuators (EMAs) play a key role in the flight control system, and their health condition has a considerable impact on the flight status and safety of aircraft. Considering the multi-scale feature of fault signals and the fault diagnosis reliability for EMAs under complex working conditions, a novel fault diagnosis method of multi-scale feature fusion convolutional neural network (MSFFCNN) is proposed. Leveraging the multiple different scales’ learning structure and attention mechanism-based feature fusion, the fault-related information can be effectively captured and learned, thereby improving the recognition ability and diagnostic performance of the network. The proposed method was evaluated by experiments and compared with the other three fault-diagnosis algorithms. The results show that the proposed MSFFCNN approach has a better diagnostic performance compared with the state-of-the-art fault diagnosis methods, which demonstrates the effectiveness and superiority of the proposed method.

Real-Time Quality Inspection of Motor Rotor Using Cost-Effective Intelligent Edge System

Induction motors (IMs) are used extensively as driving actuators in electric vehicles. Motor rotors are prone to defects in the die casting procedure, which can significantly reduce the production quality. Benefitting from the development of Internet of Things (IoT) techniques and edge computing, this study designed an instrumentation system for the fast inspection of rotor defects to meet the objectives of efficient and high-quality rotor production. First, an electromagnetic sensing device is designed to acquire the induced voltage signal of the rotor under investigation. Second, a residual multiscale feature fusion convolutional neural network model is designed to extract the hierarchical features of the signal, to facilitate defect recognition. The developed algorithm is deployed into a cost-effective edge computing node that includes a signal acquisition circuit and a Raspberry Pi microcontroller. The conducted experimental studies show that this implementation can achieve an inference time of less than 200 ms and accuracy of more than 99%. It is shown that the designed system exhibits superior performance when compared with conventional methods. The developed, compact and flexible handheld solution with enhanced deep learning techniques shows outstanding potential for use in real-time rotor defect detection.

MGFFCNN: Two‐dimensional matrix spectroscopy combined with multi‐channel gradient feature fusion convolutional neural network means to diagnose glioma and esophageal cancer patients

AbstractCurrently, glioma and esophageal cancer are common malignant tumors worldwide with low cure rate and high mortality rate, and they endanger human health seriously. In this study, we analyzed the correlation and difference between glioma and esophageal cancer through serum mid‐infrared and Raman spectra and established a multi‐channel gradient feature fusion convolutional neural network to achieve rapid diagnosis of glioma and esophageal cancer patients. We transformed the spectra from one‐dimensional matrix to two‐dimensional matrix form separately as the input of the network and fused the features extracted from the Flatten layer of the network. First, we fused the features of mid‐infrared and Raman spectra and constructed a two‐channel gradient feature fusion idea. Then, in order to enrich the learning of features further, we took the first‐order derivative of mid‐infrared and Raman original spectra, respectively, and used the derivative spectra as two channels as well. The mid‐infrared and Raman spectra in two‐dimensional matrix form were fused with their derivative spectral features, respectively, and the fused features were fused again to construct a four‐channel gradient feature fusion network model. Finally, compared with the single original spectrum and the one‐dimensional matrix feature fusion spectrum, the two‐dimensional matrix feature fused spectrum was more advantageous, and the classification accuracy of the model was as high as 99.2% ± 0.7%. This study showed that two‐dimensional matrix spectra combined with multi‐channel gradient feature fusion technique had great potential for rapid and accurate identification of patients with glioma and esophageal cancer.

Mixed gas concentration inversion based on the hierarchical feature fusion convolutional neural network

Recently, the detection of gas mixtures has attracted considerable attention due to its promising application in disease diagnosis, environmental monitoring and so on. However, the detection of gas mixtures is inevitably subject to the cross-interference between different components. Herein, we propose a hierarchical feature fusion convolutional neural network (CNN) model for mixed gas concentration inversion. In our experiment, mixtures of SO2, NO2 and NH3 were analyzed. SO2 and NO2 were the detected gases while NH3 was the interfering gas. The proposed model can be applied to low-resolution spectra. Compared with the model without hierarchical structure and feature fusion, the mean absolute errors of SO2 and NO2 decreased by 80.3% and 87.0% in the simulated spectral samples. The experimental results also demonstrate that the proposed model could effectively reduce the cross-interference and improve the accuracy of gas concentration inversion. This model would have a promising application in the field of gas detection and low-resolution spectra.

A double-layer feature fusion convolutional neural network for infrared small target detection

ABSTRACT Infrared small target detection is critical in remote sensing, military, and other fields. However, the low resolution of most infrared images and the lack of texture and detailed information could cause the target to be lost in a relatively noisy background. Therefore, in recent years, researchers have paid particular attention to the problem of small infrared target detection. In this paper, we propose a double-layer feature fusion convolutional neural network for infrared small target detection (DLFF), consisting of a simultaneous upsampling two-layer network module and a ‘T’-type fusion structure. First, the upsampling double-layer network module shares detection information while synchronizing detection, suppressing the background noise and enhancing the detection of the target. In addition, for the small target detection task, since the direct fusion of shallow spatial information and deep semantic information may lose only some small target features, we propose a ‘T’-type fusion structure to solve this problem. Furthermore, we collate an infrared small target dataset (MDFA_SIRIST) and design a pre-processing method for pre-detection images. The experimental results show that our network outperforms the other six state-of-the-art methods in combined evaluation metrics ( -score) and mean intersection ratio (mIou).

A Deep Learning-Based Target Recognition Method for Entangled Optical Quantum Imaging System

Quantum imaging has the characteristics of non-locality and strong anti-interference ability, and it has received much attention. However, the target after entangled optical quantum imaging loses its local appearance structure, and it is difficult to be recognized accurately. Therefore, this paper proposes a denoise and target recognition method for entangled optical quantum imaging system. Specifically, we design the restore conditional generative adversarial network (RestoreCGAN) to restore and reconstruct the missing edge contour structure of the target. Then, we design the two-stream feature fusion convolutional neural network (TSFFCNet) to extract deep semantic features and shallow features for target recognition. The experimental results show that RestoreCGAN outperforms the state-of-the-art methods in terms of both PSNR and SSIM. Moreover, the recognition accuracy of the RestoreCGAN in combination with the TSFFCNet reaches 97.42%. This proves that the deep learning method is effective for denoise and target recognition of entangled optical quantum imaging system.

A convolutional neural network model of multi-scale feature fusion: MFF-Net

MFF-Net (a multi-scale feature fusion convolutional neural network) was designed to improve the recognition rate of handwritten digits. The low-level, middle-level and high-level features of the image were first extracted through the convolution operation, and then the low-level and intermediate features were further extracted through different convolutional layers, later directly fused with the high-level features of the image with a certain weight, and then processed by the full connection layer. By adding a batch normalization layer before the activation layer, and a dropout layer between the full connection layers, the accuracy and generalization capacity of the network are improved. At the same time, a dynamic learning rate algorithm was designed, with which, the trained network accuracy was significantly improved as shown in the experiments on the MNIST data set. The accurate rate could reach 99.66% through only 30 epochs training. The comparison indicated that the accuracy of the network model is significantly higher than that of others.

Convolutional neural network method for damage detection of CFRP in electrical impedance tomography

Damage detection is vitally important for carbon fiber reinforced polymer (CFRP) laminates. When CFRP laminates are damaged, its impedance property is changed. Based on the electrical properties of CFRP laminates, the changed conductivity distribution can be reconstructed with the electrical impedance tomography (EIT) method. The detection method is attractive due to its simple equipment, low cost, and easy operation. However, image reconstruction of EIT faces a serious ill-conditioned nonlinear inverse problem. In order to solve this problem, a feature fusion convolutional neural network based on the dense connection (FF-D) method is applied in EIT to establish the mapping relationship between voltage measurement and conductivity distribution in this paper. The optimization can extract and utilize features to a greater degree and improve reconstruction accuracy and robustness. For the purpose of simulating the electrical properties of CFRP better, the conductivity values measured by an impedance analyzer are used as the data set. The correlation coefficient (CC) and root mean square error (RMSE) are used as indicators to evaluate the quality of image reconstruction. The simulation and experimental results suggest that the FF-D method can reconstruct images better than typical algorithms based on deep learning and conventional algorithms of EIT.