Fusion Convolutional Neural Network Research Articles

Automatic Diagnosis of Melanoma Based on EfficientNet and Patch Strategy

Melanoma is a fatal skin disease, and there are many challenging tasks in the detection of melanoma through neural network at this stage. We propose a new method for melanoma diagnosis based on EfficientNet and patch strategy. The diagnosis method has three stages of operation. First, Cyclegan is applied offline to synthesize the under-represented category samples from the over-represented category samples, and the original image and the synthesized samples are combined into a new training set to complete the conditional image synthesis task; second, we creatively propose the patch strategy and implement the patch algorithm, and apply the patch strategy offline to obtain the patch image of the newly merged training set; finally, the newly merged training set and the obtained patch image are sent to the classification network. Where, the model we proposed is composed of three parts: Basic Convolutional Neural Network, Auxiliary Convolutional Neural Network, and Fusion Convolutional Neural Network, and applies a weighted integration strategy. We evaluated the proposed method on the ISIC 2016 Skin Injury Challenge classification dataset. Experiments show that the patch strategy plays an important role in the field of melanoma classification, and the melanoma detection method proposed in this paper obtains an accuracy of 0.852 and an AUC value of 0.854 on the test set. This method can focus the attention of the classification network on the meaningful area of the skin lesion image through manual intervention, and can effectively solve the problem of category imbalance, thereby improving the performance of skin lesion classification.

Gearbox Fault Diagnosis Method Based on Multidomain Information Fusion

Traditional methods of gearbox fault diagnosis rely heavily on manual experience. To address this problem, our study proposes a gearbox fault diagnosis method based on multidomain information fusion. An experimental platform consisting of a JZQ250 fixed-axis gearbox was built. An acceleration sensor was used to obtain the vibration signal of the gearbox. Singular value decomposition (SVD) was used to preprocess the signal in order to reduce noise, and the processed vibration signal was subjected to short-time Fourier transform to obtain a two-dimensional time–frequency map. A multidomain information fusion convolutional neural network (CNN) model was constructed. Channel 1 was a one-dimensional convolutional neural network (1DCNN) model that input a one-dimensional vibration signal, and channel 2 was a two-dimensional convolutional neural network (2DCNN) model that input short-time Fourier transform (STFT) time–frequency images. The feature vectors extracted using the two channels were then fused into feature vectors for input into the classification model. Finally, support vector machines (SVM) were used to identify and classify the fault types. The model training performance used multiple methods: training set, verification set, loss curve, accuracy curve and t-SNE visualization (t-SNE). Through experimental verification, the method proposed in this paper was compared with FFT-2DCNN, 1DCNN-SVM and 2DCNN-SVM in terms of gearbox fault recognition performance. The model proposed in this paper had the highest fault recognition accuracy (98.08%).

A comprehensive diagnosis method of valve leakage faults based on bi-sensor information fusion

Acoustic emission (AE) signals caused by valve leakage exhibit obvious nonlinearity and nonstationarity characteristics. Due to the limitations of traditional valve leakage diagnosis methods, it is difficult to distinguish between internal and external valve leakage failures effectively. Recognizing this challenge, a comprehensive valve leakage diagnosis method based on a multichannel fusion convolutional neural network (MCFCNN) is proposed. First, AE signals are converted from one-dimensional time-domain signals to two-dimensional time–frequency images by the time–frequency analysis method. Then, the time–frequency images are used as model inputs, and MCFCNN fuses the features of time–frequency images from two different position. Hence, a new comprehensive diagnosis method for the bi-sensor fusion contains the time–frequency information, modal information, and position information of valve leakage is proposed. Subsequently, the effectiveness of the proposed method was verified through valve leakage simulation experiments. Furthermore, in order to study the impact of modal information on identifying internal and external valve leakage faults, the fault prediction performance of MCFCNN was compared and analyzed using short-time Fourier transform (STFT) and time-reassigned synchrosqueezing transform (TSST). Finally, according to the needs of engineering practice, the impact of sampling length on different methods is studied. The results show that compared to STFT–MCFCNN, TSST–MCFCNN required a shorter sampling length with the same diagnostic accuracy, which means that the method proposed in this study can achieve faster response time for ball valve leakage under conventional leakage flow rates.

Power Quality Disturbance Classification Based on Parallel Fusion of CNN and GRU

Effective identification of complex power quality disturbances (PQDs) is the premise and key to improving power quality issues in the current complex power grid environment. However, with the increasing application of solid-state switches, nonlinear devices, and multi-energy system generation, the power grid disturbance signals are distorted and complicated. This increases the difficulty of PQDs identification. To address this issue, this paper presents a novel method for power quality disturbance classification using a convolutional neural network (CNN) and gated recurrent unit (GRU). The CNN consists of convolutional blocks, some of which come with a squeeze-and-excitation block (SE), and is used to extract the short-term features from PQDs, where the convolutional block is used to capture the spatial information from PQDs and the SE is used to enhance the feature extraction capability of the convolutional neural network. The GRU network is designed to capture the long-term feature from PQDs, and an attention mechanism connected to GRU’s hidden states at different times is proposed to improve the GRU’s feature capture ability in long-term sequences. The CNN and GRU are parallelly arranged to perceive the same PQDs in two different views, and the feature information extracted from them is fused and transmitted to the Softmax activation layer for classification. Based on MATLAB-Simulink, a typical multi-energy-source system is constructed to analyze PQDs, and twelve PQDs are simulated to validate the proposed method. The simulation results show that the proposed method has higher classification accuracy in both single and hybrid disturbances and significant advantages in noise immunity.

Fusion-Extracted Features by Deep Networks for Improved COVID-19 Classification with Chest X-ray Radiography.

Convolutional neural networks (CNNs) have shown promise in accurately diagnosing coronavirus disease 2019 (COVID-19) and bacterial pneumonia using chest X-ray images. However, determining the optimal feature extraction approach is challenging. This study investigates the use of fusion-extracted features by deep networks to improve the accuracy of COVID-19 and bacterial pneumonia classification with chest X-ray radiography. A Fusion CNN method was developed using five different deep learning models after transferred learning to extract image features (Fusion CNN). The combined features were used to build a support vector machine (SVM) classifier with a RBF kernel. The performance of the model was evaluated using accuracy, Kappa values, recall rate, and precision scores. The Fusion CNN model achieved an accuracy and Kappa value of 0.994 and 0.991, with precision scores for normal, COVID-19, and bacterial groups of 0.991, 0.998, and 0.994, respectively. The results indicate that the Fusion CNN models with the SVM classifier provided reliable and accurate classification performance, with Kappa values no less than 0.990. Using a Fusion CNN approach could be a possible solution to enhance accuracy further. Therefore, the study demonstrates the potential of deep learning and fusion-extracted features for accurate COVID-19 and bacterial pneumonia classification with chest X-ray radiography.

Using CNN with Multi-Level Information Fusion for Image Denoising

Deep convolutional neural networks (CNN) with hierarchical architectures have obtained good results for image denoising. However, in some cases where the noise level is unknown and the image background is complex, it is challenging to obtain robust information through CNN. In this paper, we present a multi-level information fusion CNN (MLIFCNN) in image denoising containing a fine information extraction block (FIEB), a multi-level information interaction block (MIIB), a coarse information refinement block (CIRB), and a reconstruction block (RB). In order to adapt to more complex image backgrounds, FIEB uses parallel group convolution to extract wide-channel information. To enhance the robustness of the obtained information, a MIIB uses residual operations to act in two sub-networks for implementing the interaction of wide and deep information to adapt to the distribution of different noise levels. To enhance the stability of the training denoiser, CIRB stacks common and group convolutions to refine the obtained information. Finally, RB uses a residual operation to act in a single convolution in order to obtain the resultant clean image. Experimental results show that our method is better than many other excellent methods, both in terms of quantitative and qualitative aspects.

DFNet: Dense fusion convolution neural network for plant leaf disease classification

AbstractThe early identification of plant diseases is crucial for preventing the loss of crop production. Recently, the advancement of deep learning has significantly improved the identification of plant leaf diseases. However, most approaches depend on a single convolutional neural network (CNN) to extract the leaf features, ignoring the opportunity to take full advantage of the feature richness available in the images. This paper explores a novel CNN model with multiple automated feature extractors, namely, dense fusion CNN (DFNet), for classifying plant leaf diseases. DFNet aims to increase the diversity of extracted features in order to improve discrimination. Instead of using a single‐CNN model, DFNet relies on a double‐pretrained CNN model, MobileNetV2 and NASNetMobile, as the feature extractor. The features extracted from each CNN are fused in the fusion layer using a fully connected network. The proposed method was evaluated using corn (Zea mays L.) and coffee (Coffea canephora) leaf disease datasets and compared to the existing models. The experiment showed that DFNet is superior and consistent to other CNN methods by achieving an accuracy of 97.53% for corn leaf diseases and 94.65% for coffee leaf diseases.

Heterogeneous Fusion of Camera and mmWave Radar Sensor of Optimizing Convolutional Neural Networks for Parking Meter System.

In this article, a novel heterogeneous fusion of convolutional neural networks that combined an RGB camera and an active mmWave radar sensor for the smart parking meter is proposed. In general, the parking fee collector on the street outdoor surroundings by traffic flows, shadows, and reflections makes it an exceedingly tough task to identify street parking regions. The proposed heterogeneous fusion convolutional neural networks combine an active radar sensor and image input with specific geometric area, allowing them to detect the parking region against different tough conditions such as rain, fog, dust, snow, glare, and traffic flow. They use convolutional neural networks to acquire output results along with the individual training and fusion of RGB camera and mmWave radar data. To achieve real-time performance, the proposed algorithm has been implemented on a GPU-accelerated embedded platform Jetson Nano with a heterogeneous hardware acceleration methodology. The experimental results exhibit that the accuracy of the heterogeneous fusion method can reach up to 99.33% on average.

Attention-Embedded Triple-Fusion Branch CNN for Hyperspectral Image Classification

Hyperspectral imaging (HSI) is widely used in various fields owing to its rich spectral information. Nonetheless, the high dimensionality of HSI and the limited number of labeled data remain significant obstacles to HSI classification technology. To alleviate the above problems, we propose an attention-embedded triple-branch fusion convolutional neural network (AETF-Net) for an HSI classification. The network consists of a spectral attention branch, a spatial attention branch, and a multi-attention fusion branch (MAFB). The spectral branch introduces the cross-channel attention to alleviate the band redundancy problem in high dimensions, while the spatial branch preserves the location information of features and eliminates interfering image elements by a bi-directional spatial attention module. These pre-extracted spectral and spatial attention features are then embedded into a novel MAFB with large kernel decomposition technique. The proposed AETF-Net achieves multi-attention features reuse and extracts more representative and discriminative features. Experimental results on three well-known datasets demonstrate the superiority of the method AETF-Net.

Real-Time Quality Inspection of Motor Rotor Using Cost-Effective Intelligent Edge System

Induction motors (IMs) are used extensively as driving actuators in electric vehicles. Motor rotors are prone to defects in the die casting procedure, which can significantly reduce the production quality. Benefitting from the development of Internet of Things (IoT) techniques and edge computing, this study designed an instrumentation system for the fast inspection of rotor defects to meet the objectives of efficient and high-quality rotor production. First, an electromagnetic sensing device is designed to acquire the induced voltage signal of the rotor under investigation. Second, a residual multiscale feature fusion convolutional neural network model is designed to extract the hierarchical features of the signal, to facilitate defect recognition. The developed algorithm is deployed into a cost-effective edge computing node that includes a signal acquisition circuit and a Raspberry Pi microcontroller. The conducted experimental studies show that this implementation can achieve an inference time of less than 200 ms and accuracy of more than 99%. It is shown that the designed system exhibits superior performance when compared with conventional methods. The developed, compact and flexible handheld solution with enhanced deep learning techniques shows outstanding potential for use in real-time rotor defect detection.

Objective rating method for fabric pilling based on LSNet network

Fabric fuzzing and pilling have long been a problem in the textile industry, and consumer complaints about fabric fuzzing and pilling account for about 29.5% of the total textile product quality problems every year. At present, more and more subjective rating methods are used at home and abroad to rate fabric pilling, but there are still certain limitations, such as slow speed and subjectivity, which prompted the exploration of an accurate and objective automatic rating method to become a very urgent need. In response to these problems, this work is based on the multi-feature fusion convolutional neural network (CNN) Light-Smart-Network (LSNet) that has the capability to objectively grade fabric pilling. The fuzzing and pilling data is first set for six kinds of knitted fabrics, woven fabrics, and non-woven fabrics; next, the LSNet network determines the best parameters for objective rating. The network model is composed of a SqueezeNet branch (S1 Branch) and a ShuffleNet branch (S2 Branch), where the S1 Branch uses deep separable convolution (DSC) to improve the Fire module of the SqueezeNet network and adds an attention mechanism that makes the network favor contextual information. An S2 Branch adds short connections to the ShuffleNet network to combine feature maps of different sizes, perform fusion, enhance the network’s ability to extract feature map information, and optimize some details. Experiments show that the accuracy of LSNet’s objective rating for fabric fuzzing and pilling is 97.386% with a model size of only 5.25 M. Finally, the LSNet model is verified via a heat and feature map, which proves that LSNet has an objective rating for fabric fuzzing and pilling reliability.

A century and a half precipitation oxygen isoscape for China generated using data fusion and bias correction

The precipitation oxygen isotopic composition is a useful environmental tracer for climatic and hydrological studies. However, accurate and high-resolution precipitation oxygen isoscapes are currently lacking in China. In this study, a precipitation oxygen isoscape in China for a period of 148 years is built by integrating observed and iGCMs-simulated isotope compositions using an optimal hybrid approach of three data fusion and two bias correction methods. The temporal and spatial resolutions of the isoscape are monthly and 50–60 km, respectively. Results show that the Convolutional Neural Networks (CNN) fusion method performs the best (correlation coefficient larger than 0.95 and root mean square error smaller than 1‰), and the other two data fusion methods perform slightly better than the bias correction methods. Thus, the isoscape is generated by using the CNN fusion method for the common 1969–2007 period and by using the bias correction methods for remaining years. The generated isoscape, which shows similar spatio-temporal distributions to observations, is reliable and useful for providing strong support for tracking atmospheric and hydrological processes.

A Marine Organism Detection Framework Based on Dataset Augmentation and CNN-ViT Fusion

Underwater vision-based detection plays an important role in marine resources exploration, marine ecological protection and other fields. Due to the restricted carrier movement and the clustering effect of some marine organisms, the size of some marine organisms in the underwater image is very small, and the samples in the dataset are very unbalanced, which aggravate the difficulty of vision detection of marine organisms. To solve these problems, this study proposes a marine organism detection framework with a dataset augmentation strategy and Convolutional Neural Networks (CNN)-Vision Transformer (ViT) fusion model. The proposed framework adopts two data augmentation methods, namely, random expansion of small objects and non-overlapping filling of scarce samples, to significantly improve the data quality of the dataset. At the same time, the framework takes YOLOv5 as the baseline model, introduces ViT, deformable convolution and trident block in the feature extraction network, and extracts richer features of marine organisms through multi-scale receptive fields with the help of the fusion of CNN and ViT. The experimental results show that, compared with various one-stage detection models, the mean average precision (mAP) of the proposed framework can be improved by 27%. At the same time, it gives consideration to both performance and real-time, so as to achieve high-precision real-time detection of the marine organisms on the underwater mobile platform.

MGFFCNN: Two‐dimensional matrix spectroscopy combined with multi‐channel gradient feature fusion convolutional neural network means to diagnose glioma and esophageal cancer patients

AbstractCurrently, glioma and esophageal cancer are common malignant tumors worldwide with low cure rate and high mortality rate, and they endanger human health seriously. In this study, we analyzed the correlation and difference between glioma and esophageal cancer through serum mid‐infrared and Raman spectra and established a multi‐channel gradient feature fusion convolutional neural network to achieve rapid diagnosis of glioma and esophageal cancer patients. We transformed the spectra from one‐dimensional matrix to two‐dimensional matrix form separately as the input of the network and fused the features extracted from the Flatten layer of the network. First, we fused the features of mid‐infrared and Raman spectra and constructed a two‐channel gradient feature fusion idea. Then, in order to enrich the learning of features further, we took the first‐order derivative of mid‐infrared and Raman original spectra, respectively, and used the derivative spectra as two channels as well. The mid‐infrared and Raman spectra in two‐dimensional matrix form were fused with their derivative spectral features, respectively, and the fused features were fused again to construct a four‐channel gradient feature fusion network model. Finally, compared with the single original spectrum and the one‐dimensional matrix feature fusion spectrum, the two‐dimensional matrix feature fused spectrum was more advantageous, and the classification accuracy of the model was as high as 99.2% ± 0.7%. This study showed that two‐dimensional matrix spectra combined with multi‐channel gradient feature fusion technique had great potential for rapid and accurate identification of patients with glioma and esophageal cancer.

Recognition of single upper limb motor imagery tasks from EEG using multi-branch fusion convolutional neural network.

Motor imagery-based brain-computer interfaces (MI-BCI) have important application values in the field of neurorehabilitation and robot control. At present, MI-BCI mostly use bilateral upper limb motor tasks, but there are relatively few studies on single upper limb MI tasks. In this work, we conducted studies on the recognition of motor imagery EEG signals of the right upper limb and proposed a multi-branch fusion convolutional neural network (MF-CNN) for learning the features of the raw EEG signals as well as the two-dimensional time-frequency maps at the same time. The dataset used in this study contained three types of motor imagery tasks: extending the arm, rotating the wrist, and grasping the object, 25 subjects were included. In the binary classification experiment between the grasping object and the arm-extending tasks, MF-CNN achieved an average classification accuracy of 78.52% and kappa value of 0.57. When all three tasks were used for classification, the accuracy and kappa value were 57.06% and 0.36, respectively. The comparison results showed that the classification performance of MF-CNN is higher than that of single CNN branch algorithms in both binary-class and three-class classification. In conclusion, MF-CNN makes full use of the time-domain and frequency-domain features of EEG, can improve the decoding accuracy of single limb motor imagery tasks, and it contributes to the application of MI-BCI in motor function rehabilitation training after stroke.

Single-trial P300 classification algorithm based on centralized multi-person data fusion CNN.

Currently, it is still a challenge to detect single-trial P300 from electroencephalography (EEG) signals. In this paper, to address the typical problems faced by existing single-trial P300 classification, such as complex, time-consuming and low accuracy processes, a single-trial P300 classification algorithm based on multiplayer data fusion convolutional neural network (CNN) is proposed to construct a centralized collaborative brain-computer interfaces (cBCI) for fast and highly accurate classification of P300 EEG signals. In this paper, two multi-person data fusion methods (parallel data fusion and serial data fusion) are used in the data pre-processing stage to fuse multi-person EEG information stimulated by the same task instructions, and then the fused data is fed as input to the CNN for classification. In building the CNN network for single-trial P300 classification, the Conv layer was first used to extract the features of single-trial P300, and then the Maxpooling layer was used to connect the Flatten layer for secondary feature extraction and dimensionality reduction, thereby simplifying the computation. Finally batch normalisation is used to train small batches of data in order to better generalize the network and speed up single-trial P300 signal classification. In this paper, the above new algorithms were tested on the Kaggle dataset and the Brain-Computer Interface (BCI) Competition III dataset, and by analyzing the P300 waveform features and EEG topography and the four standard evaluation metrics, namely Accuracy, Precision, Recall and F1-score,it was demonstrated that the single-trial P300 classification algorithm after two multi-person data fusion CNNs significantly outperformed other classification algorithms. The results show that the single-trial P300 classification algorithm after two multi-person data fusion CNNs significantly outperformed the single-person model, and that the single-trial P300 classification algorithm with two multi-person data fusion CNNs involves smaller models, fewer training parameters, higher classification accuracy and improves the overall P300-cBCI classification rate and actual performance more effectively with a small amount of sample information compared to other algorithms.

Mixed gas concentration inversion based on the hierarchical feature fusion convolutional neural network

Recently, the detection of gas mixtures has attracted considerable attention due to its promising application in disease diagnosis, environmental monitoring and so on. However, the detection of gas mixtures is inevitably subject to the cross-interference between different components. Herein, we propose a hierarchical feature fusion convolutional neural network (CNN) model for mixed gas concentration inversion. In our experiment, mixtures of SO2, NO2 and NH3 were analyzed. SO2 and NO2 were the detected gases while NH3 was the interfering gas. The proposed model can be applied to low-resolution spectra. Compared with the model without hierarchical structure and feature fusion, the mean absolute errors of SO2 and NO2 decreased by 80.3% and 87.0% in the simulated spectral samples. The experimental results also demonstrate that the proposed model could effectively reduce the cross-interference and improve the accuracy of gas concentration inversion. This model would have a promising application in the field of gas detection and low-resolution spectra.

Research on Weld Defect Detection and Evaluation Technology based on Deep Learning

Aiming at the problems of low efficiency and strong subjectivity in the current detection of weld defects by radiographic imaging technology, an object detection method of weld defects based on multi-channel fusion convolutional neural network is proposed. In this method, the images of weld defects are encoded and input into multiple feature extraction channels formed by parallel fusion of CNN. After that, the extracted features are fused with full connection layer and the feature vectors are output. Finally, the final output is obtained by Softmax for classification. The proposed method is verified by weld defect images in actual production. The experimental results indicate that the mAP of the multi-channel fusion convolutional neural network reaches 76.37%, and the detection accuracy of weld defects is higher than that of other network such as ResNet-50 and VGG-16. The proposed method can be applied to X-ray intelligent detection of weld defects and other scenarios.

CFCNN: A novel convolutional fusion framework for collaborative fault identification of rotating machinery

Sensor techniques and emerging CNN models have greatly facilitated the development of collaborative fault diagnosis. Existing CNN models apply different fusion schemes to achieve reliable fault identification based on multisensor data. Few CNN models, however, take into account both the intrinsic correlations and the distribution gap between different signals, which may result in a limited exploration of multisource data. To address this issue, a novel convolutional fusion framework called a collaborative fusion convolutional neural network (CFCNN) is developed in this paper. More specifically, a multiscale shrinkage denoising module (MSDM) is developed first to extract multilevel modality-specific features from different mechanical signals. Then, drawing inspiration from the intermediate fusion scheme, a central fusion module (CFM) is introduced to explore the intrinsic correlations and integrate cross-modal features. Moreover, an online label smoothing training (OLST) strategy is applied to reduce overfitting and promote better classification performance of CFCNN. The developed CFCNN is expected to shed new light on collaborative fault diagnosis using the intermediate fusion scheme. The efficacy of the developed CFCNN is verified through the cylindrical rolling bearing dataset and the planetary gearbox dataset.

An incremental learning method with hybrid data over/down-sampling for sEMG-based gesture classification

Surface electromyography(sEMG)-based gesture classification methods have been widely developed in neural decoding. However, these decoding methods are usually constrained to a fixed set of gestures, which hinders flexibility in practical application. This paper contributes to an incremental learning framework to make classifiers learn different gesture sets (new tasks) gradually without catastrophic forgetting. First, this study analyzes the existing neural decoding methods with deep learning, and introduces an early and late fusion convolutional neural network (ELFCNN) structure based on frequency spectrum. Then, a sEMG-based gesture classification with incremental learning is demonstrated, which modifies the end to end learning method with hybrid data over/down-sampling (HDOD) method. By combining ELFCNN and the HDOD progress, the incremental learning method can make a comparable performance without data dimension reduction, and mitigate catastrophic forgetting while reducing data storage. Experimental results on both small and large size situations show consistent classification accuracy improvement from 0.47% to 0.71% compared with other popular incremental learning methods.