Fusion Convolutional Neural Network Research Articles

A Multi-defect detection system for sewer pipelines based on StyleGAN-SDM and fusion CNN

With the development of deep learning, convolutional neural networks (CNN) have been gradually used in pipeline defeats detection. However, due to the complex environment inside the pipeline, few defeat images are not enough for the training of CNN. A multi-defect detection system based on StyleGAN-SDM and fusion CNN for sewer pipelines is proposed in this paper. First, aiming at the problem of data acquisition and small data volume, raw images are preprocessed by StyleGAN-SDM, which integrates StyleGAN v2 and sharpness discrimination model (SDM) to generate multi-defect images and automatically select clear images. The indexes of Inception-Residual score (IRS), accuracy and macro-F1 score to evaluate the quality of the images generated are 2.968 ± 0.024, 99.64%, and 0.997, respectively. Second, to improve the detection accuracy, a multi-defect classification model (MDCM) based on fusion CNN, which combines Inception network and Residual network, is proposed to classify the on-site images into four categories. Third, compared with conventional deep-learning methods, the mean accuracy and macro-F1 score of the proposed model reach 95.64% and 0.955, which are increased by 1.51% and 0.015 by StyleGAN-SDM, respectively. Finally, to solve the timeliness problem of on-site detection, a real-time multi-defeat detection system for sewer pipelines is established with the computer vision library of OpenCV. Some on-site videos are detected with the mean speed of 24.11 FPS and these results could aid the staff.

Grasp Detection Based on Light-Weight Hierarchical Fusion Convolutional Neural Network

This paper presents a light-weight Hierarchical Fusion Convolutional Neural Network (HF-CNN) which can be used for grasping detection. The network mainly employs residual structures, atrous spatial pyramid pooling (ASPP) and coding-decoding based feature fusion. Compared with the usual grasping detection, the network in this paper greatly improves the robustness and generalizability on detecting tasks by extensively extracting feature information of the images. In our test with the Cornell University dataset, we achieve 85% accuracy when detecting the unknown objects.

Automated Classification of Retinal OCT Images Using a Deep Multi-Scale Fusion CNN

The automated analysis of optical coherence tomography (OCT) images can play a crucial role in the diagnosis and management of retinal diseases. The wide variations of the retinal disease manifestations in terms of shape, size, texture and spatial location pose a huge challenge in designing reliable and efficient automated methods. Existing methods mostly use single-scale deep frameworks for encoding features from the OCT images to make a diagnosis decision. Such approaches potentially ignore the useful discriminative information in different scales. Therefore, in this paper, we propose a Deep Multi-scale Fusion Convolutional Neural Network (DMF-CNN) that can encode the multi-scaled disease characteristics and effectively combine them for reliable classification. Specifically, multiple CNNs with different receptive fields are used to obtain scale-specific feature representations from the OCT images. These representations are fused to mine the cross-scale powerful discriminative features for classification. A joint multi-loss optimization strategy is designed to collectively learn the scale-specific and cross-scale complementary information during training. The method is evaluated on two publicly available OCT databases (the UCSD and the NEH) and delivers state-of-the-art performance. An impressive overall accuracy of 96.03% and 99.60% is obtained for the UCSD and the NEH datasets, respectively. The outstanding performance and the improved generalization makes the method a reliable diagnostic aid for medical practitioners.

Two-stream graph convolutional neural network fusion for weakly supervised temporal action detection

Two-stream graph convolutional neural network fusion for weakly supervised temporal action detection

Dual-Domain Fusion Convolutional Neural Network for Contrast Enhancement Forensics.

Contrast enhancement forensics techniques have always been of great interest for the image forensics community, as they can be an effective tool for recovering image history and identifying tampered images. Although several contrast enhancement forensic algorithms have been proposed, their accuracy and robustness against some kinds of processing are still unsatisfactory. In order to attenuate such deficiency, in this paper, we propose a new framework based on dual-domain fusion convolutional neural network to fuse the features of pixel and histogram domains for contrast enhancement forensics. Specifically, we first present a pixel-domain convolutional neural network to automatically capture the patterns of contrast-enhanced images in the pixel domain. Then, we present a histogram-domain convolutional neural network to extract the features in the histogram domain. The feature representations of pixel and histogram domains are fused and fed into two fully connected layers for the classification of contrast-enhanced images. Experimental results show that the proposed method achieves better performance and is robust against pre-JPEG compression and antiforensics attacks, obtaining over 99% detection accuracy for JPEG-compressed images with different QFs and antiforensics attack. In addition, a strategy for performance improvements of CNN-based forensics is explored, which could provide guidance for the design of CNN-based forensics tools.

Using Artificial Intelligence to Achieve Auxiliary Training of Table Tennis Based on Inertial Perception Data.

Compared with optical sensors, wearable inertial sensors have many advantages such as low cost, small size, more comprehensive application range, no space restrictions and occlusion, better protection of user privacy, and more suitable for sports applications. This article aims to solve irregular actions that table tennis enthusiasts do not know in actual situations. We use wearable inertial sensors to obtain human table tennis action data of professional table tennis players and non-professional table tennis players, and extract the features from them. Finally, we propose a new method based on multi-dimensional feature fusion convolutional neural network and fine-grained evaluation of human table tennis actions. Realize ping-pong action recognition and evaluation, and then achieve the purpose of auxiliary training. The experimental results prove that our proposed multi-dimensional feature fusion convolutional neural network has an average recognition rate that is 0.17 and 0.16 higher than that of CNN and Inception-CNN on the nine-axis non-professional test set, which proves that we can better distinguish different human table tennis actions and have a more robust generalization performance. Therefore, on this basis, we have better realized the enthusiast of table tennis the purpose of the action for auxiliary training.

Skeleton-based human activity recognition using ConvLSTM and guided feature learning

Human activity recognition aims to determine actions performed by a human in an image or video. Examples of human activity include standing, running, sitting, sleeping, etc. These activities may involve intricate motion patterns and undesired events such as falling. This paper proposes a novel deep convolutional long short-term memory (ConvLSTM) network for skeletal-based activity recognition and fall detection. The proposed ConvLSTM network is a sequential fusion of convolutional neural networks (CNNs), long short-term memory (LSTM) networks, and fully connected layers. The acquisition system applies human detection and pose estimation to pre-calculate skeleton coordinates from the image/video sequence. The ConvLSTM model uses the raw skeleton coordinates along with their characteristic geometrical and kinematic features to construct the novel guided features. The geometrical and kinematic features are built upon raw skeleton coordinates using relative joint position values, differences between joints, spherical joint angles between selected joints, and their angular velocities. The novel spatiotemporal-guided features are obtained using a trained multi-player CNN-LSTM combination. Classification head including fully connected layers is subsequently applied. The proposed model has been evaluated on the KinectHAR dataset having 130,000 samples with 81 attribute values, collected with the help of a Kinect (v2) sensor. Experimental results are compared against the performance of isolated CNNs and LSTM networks. Proposed ConvLSTM have achieved an accuracy of 98.89% that is better than CNNs and LSTMs having an accuracy of 93.89 and 92.75%, respectively. The proposed system has been tested in realtime and is found to be independent of the pose, facing of the camera, individuals, clothing, etc. The code and dataset will be made publicly available.

Multi-Focus Network to Decode Imaging Phenotype for Overall Survival Prediction of Gastric Cancer Patients.

Gastric cancer (GC) is the third leading cause of cancer-associated deaths globally. Accurate risk prediction of the overall survival (OS) for GC patients shows significant prognostic value, which helps identify and classify patients into different risk groups to benefit from personalized treatment. Many methods based on machine learning algorithms have been widely explored to predict the risk of OS. However, the accuracy of risk prediction has been limited and remains a challenge with existing methods. Few studies have proposed a framework and pay attention to the low-level and high-level features separately for the risk prediction of OS based on computed tomography images of GC patients. To achieve high accuracy, we propose a multi-focus fusion convolutional neural network. The network focuses on low-level and high-level features, where a subnet to focus on lower-level features and the other enhanced subnet with lateral connection to focus on higher-level semantic features. Three independent datasets of 640 GC patients are used to assess our method. Our proposed network is evaluated by metrics of the concordance index and hazard ratio. Our network outperforms state-of-the-art methods with the highest concordance index and hazard ratio in independent validation and test sets. Our results prove that our architecture can unify the separate low-level and high-level features into a single framework, and can be a powerful method for accurate risk prediction of OS.

Laplacian pyramid networks: A new approach for multispectral pansharpening

Pansharpening is about fusing a high spatial resolution panchromatic image with a simultaneously acquired multispectral image with lower spatial resolution. In this paper, we propose a Laplacian pyramid pansharpening network architecture for accurately fusing a high spatial resolution panchromatic image and a low spatial resolution multispectral image, aiming at getting a higher spatial resolution multispectral image. The proposed architecture considers three aspects. First, we use the Laplacian pyramid method whose blur kernels are designed according to the sensors’ modulation transfer functions to separate the images into multiple scales for fully exploiting the crucial spatial information at different spatial scales. Second, we develop a fusion convolutional neural network (FCNN) for each scale, combining them to form the final multi-scale network architecture. Specifically, we use recursive layers for the FCNN to share parameters across and within pyramid levels, thus significantly reducing the network parameters. Third, a total loss consisting of multiple across-scale loss functions is employed for training, yielding higher accuracy. Extensive experimental results based on quantitative and qualitative assessments exploiting benchmarking datasets demonstrate that the proposed architecture outperforms state-of-the-art pansharpening methods. Code is available at https://github.com/ChengJin-git/LPPN.

Fusion of convolutional neural networks based on Dempster–Shafer theory for automatic pneumonia detection from chest X‐ray images

AbstractDeep learning‐based applications for disease detection are essential tools for experts to effectively diagnose diseases at different stages. In this article, a new approach based on an evidence based fusion theory is proposed, allowing the combination of a set of deep learning classifiers to provide more accurate disease detection results. The main contribution of this work is the application of the Dempster–Shafer theory for the fusion of five pre trained convolutional neural networks including VGG16, Xception, InceptionV3, ResNet50, and DenseNet201 for the diagnosis of pneumonia from chest X‐ray images. To evaluate this approach, experiments are conducted using a publicly available dataset containing more than 5800 chest X‐ray images. The obtained results demonstrate that our approach provides excellent detection performance compared to other state‐of‐the‐art methods; it achieves a precision of 97.5%, a recall of 98%, an f1‐score of 97.8%, and an accuracy of 97.3%.

Automatic feature extraction and fusion recognition of motor imagery EEG using multilevel multiscale CNN.

A motor imagery EEG (MI-EEG) signal is often selected as the driving signal in an active brain computer interface (BCI) system, and it has been a popular field to recognize MI-EEG images via convolutional neural network (CNN), which poses a potential problem for maintaining the integrity of the time-frequency-space information in MI-EEG images and exploring the feature fusion mechanism in the CNN. However, information is excessively compressed in the present MI-EEG image, and the sequential CNN is unfavorable for the comprehensive utilization of local features. In this paper, a multidimensional MI-EEG imaging method is proposed, which is based on time-frequency analysis and the Clough-Tocher (CT) interpolation algorithm. The time-frequency matrix of each electrode is generated via continuous wavelet transform (WT), and the relevant section of frequency is extracted and divided into nine submatrices, the longitudinal sums and lengths of which are calculated along the directions of frequency and time successively to produce a 3 × 3 feature matrix for each electrode. Then, feature matrix of each electrode is interpolated to coincide with their corresponding coordinates, thereby yielding a WT-based multidimensional image, called WTMI. Meanwhile, a multilevel and multiscale feature fusion convolutional neural network (MLMSFFCNN) is designed for WTMI, which has dense information, low signal-to-noise ratio, and strong spatial distribution. Extensive experiments are conducted on the BCI Competition IV 2a and 2b datasets, and accuracies of 92.95% and 97.03% are yielded based on 10-fold cross-validation, respectively, which exceed those of the state-of-the-art imaging methods. The kappa values and p values demonstrate that our method has lower class skew and error costs. The experimental results demonstrate that WTMI can fully represent the time-frequency-space features of MI-EEG and that MLMSFFCNN is beneficial for improving the collection of multiscale features and the fusion recognition of general and abstract features for WTMI.

Improved stereo matching algorithm based on multi-scale fusion

Aiming at the low matching accuracy of local stereo matching algorithm in weak texture or discontinuous disparity areas, a stereo matching algorithm combining multi-scale fusion of convolutional neural network (CNN) and feature pyramid structure (FPN) is proposed. The feature pyramid is applied on the basis of the convolutional neural network to realize the multi-scale feature extraction and fusion of the image, which improves the matching similarity of the image blocks. The guide graph filter is used to quickly and effectively complete the cost aggregation. The disparity selection stage adapts the improvement dynamic programming algorithm to obtain the initial disparity map. The initial disparity map is refined so as to obtain the final disparity map. The algorithm is trained and tested on the image provided by Middlebury data set, and the result shows that the disparity map obtained by the algorithm has good effect.

Breast ultrasound image segmentation: A coarse-to-fine fusion convolutional neural network.

Breast ultrasound (BUS) image segmentation plays a crucial role in computer-aided diagnosis systems for BUS examination, which are useful for improved accuracy of breast cancer diagnosis. However, such performance remains a challenging task owing to the poor image quality and large variations in the sizes, shapes, and locations of breast lesions. In this paper, we propose a new convolutional neural network with coarse-to-fine feature fusion to address the aforementioned challenges. The proposed fusion network consists of an encoder path, a decoder path, and a core fusion stream path (FSP). The encoder path is used to capture the context information, and the decoder path is used for localization prediction. The FSP is designed to generate beneficial aggregate feature representations (i.e., various-sized lesion features, aggregated coarse-to-fine information, and high-resolution edge characteristics) from the encoder and decoder paths, which are eventually used for accurate breast lesion segmentation. To better retain the boundary information and alleviate the effect of image noise, we input the superpixel image along with the original image to the fusion network. Furthermore, a weighted-balanced loss function was designed to address the problem of lesion regions having different sizes. We then conducted exhaustive experiments on three public BUS datasets to evaluate the proposed network. The proposed method outperformed state-of-the-art (SOTA) segmentation methods on the three public BUS datasets, with average dice similarity coefficients of 84.71(±1.07), 83.76(±0.83), and 86.52(±1.52), average intersection-over-union values of 76.34(±1.50), 75.70(±0.98), and 77.86(±2.07), average sensitivities of 86.66(±1.82), 85.21(±1.98), and 87.21(±2.51), average specificities of 97.92(±0.46), 98.57(±0.19), and 99.42(±0.21), and average accuracies of 95.89(±0.57), 97.17(±0.3), and 98.51(±0.3). The proposed fusion network could effectively segment lesions from BUS images, thereby presenting a new feature fusion strategy to handle challenging task of segmentation, while outperforming the SOTA segmentation methods. The code is publicly available at https://github.com/mniwk/CF2-NET.

NFCNN: toward a noise fusion convolutional neural network for image denoising

Deep learning-based methods have achieved the state-of-the-art performance in image denoising. In this paper, a deep learning-based denoising method is proposed and a module called fusion block is introduced in the convolutional neural network. For this so-called noise fusion convolutional neural network (NFCNN), there are two branches in its multistage architecture. One branch aims to predict the latent clean image, while the other one predicts the residual image. A fusion block is contained between every two stages by taking the predicted clean image and the predicted residual image as a part of inputs, and it outputs a fused result to the next stage. NFCNN has an attractive texture-preserving ability because of the fusion block. To train NFCNN, a stage-wise supervised training strategy is adopted to avoid the vanishing gradient and exploding gradient problems. Experimental results show that NFCNN is able to perform competitive denoising results when compared with some state-of-the-art algorithms.

Fast detection of cumin and fennel using NIR spectroscopy combined with deep learning algorithms

Cumin and fennel are two types of food raw materials that have similar appearances but great differences in efficacy. Therefore, the efficient identification of these two types of food is very important to the related food processing industry and the development of food detection technology. In this study, a deep learning algorithm was designed to process Fourier NIR spectral data for the classification of cumin and fennel. The method was based on the difference in the signal intensity in NIR spectra between cumin and fennel, and the identification was performed by combining the improved multiscale fusion convolutional neural network (MCNN) and the bidirectional long-short-term memory network (BILSTM) based on NIR spectra. The classification accuracies of the MCNN and BILSTM models were 100% and 98.57%, respectively. The models have high sensitivity and fast analysis speed, which directly promote the development of real-time identification technology of cumin and fennel and verify the feasibility of deep learning algorithms in the field of food detection.

3D Skeletal Human Action Recognition Using a CNN Fusion Model

Smart homes have become central in the sustainability of buildings. Recognizing human activity in smart homes is the key tool to achieve home automation. Recently, two-stream Convolutional Neural Networks (CNNs) have shown promising performance for video-based human action recognition. However, such models cannot act directly on the 3D skeletal sequences due to its limitation to the 2D image video inputs. Considering the powerful effect of 3D skeletal data for describing human activity, in this study, we present a novel method to recognize the skeletal human activity in sustainable smart homes using a CNN fusion model. Our proposed method can represent the spatiotemporal information of each 3D skeletal sequence into three images and three image sequences through gray value encoding, referred to as skeletal trajectory shape images (STSIs) and skeletal pose image (SPI) sequences, and build a CNNs’ fusion model with three STSIs and three SPI sequences as input for skeletal activity recognition. Such three STSIs and three SPI sequences are, respectively, generated in three orthogonal planes as complementary to each other. The proposed CNN fusion model allows the hierarchical learning of spatiotemporal features, offering better action recognition performance. Experimental results on three public datasets show that our method outperforms the state-of-the-art methods.

Hierarchical fusion convolutional neural networks for SAR image segmentation

Abstract Convolutional neural network (CNN) has achieved promising results in image segmentation recently. However, for the segmentation of synthetic aperture radar (SAR) images with complicated scene, the single receptive field in CNN has a limited ability to effectively capture structural and regional information at the same time. In this paper, we propose a hierarchical fusion CNN (HIFCNN) model for SAR image segmentation. At each convolutional layer, HIFCNN sets several different-sized receptive fields, and thus extracts hierarchical features. Concretely, the larger-sized receptive field captures regional information and is robust against speckle, while the smaller one preserves the structural information well. Then, based on the Dempster-Shafer evidential theory, the proposed hierarchical network, HIFCNN, implements a decision-level fusion to integrate these hierarchical features. In this way, the structural and regional information can be accurately captured by different receptive fields, which is beneficial for edge location, structure preservation and region homogeneity in SAR image segmentation. The effectiveness of HIFCNN model is demonstrated by the application to the segmentation of the simulated images and real SAR images.

Facial Expression Recognition Algorithm Based on Fusion of Transformed Multilevel Features and Improved Weighted Voting SVM

In allusion to the shortcomings of traditional facial expression recognition (FER) that only uses a single feature and the recognition rate is not high, a FER method based on fusion of transformed multilevel features and improved weighted voting SVM (FTMS) is proposed. The algorithm combines the transformed traditional shallow features and convolutional neural network (CNN) deep semantic features and uses an improved weighted voting method to make a comprehensive decision on the results of the four trained SVM classifiers to obtain the final recognition result. The shallow features include local Gabor features, LBP features, and joint geometric features designed in this study, which are composed of distance and deformation characteristics. The deep feature of CNN is the multilayer feature fusion of CNN proposed in this study. This study also proposes to use a better performance SVM classifier with CNN to replace Softmax since the poor distinction between facial expressions. Experiments on the FERPlus database show that the recognition rate of this method is 17.2% higher than that of the traditional CNN, which proves the effectiveness of the fusion of the multilayer convolutional layer features and SVM. FTMS-based facial expression recognition experiments are carried out on the JAFFE and CK+ datasets. Experimental results show that, compared with the single feature, the proposed algorithm has higher recognition rate and robustness and makes full use of the advantages and characteristics of different features.

Fusion of convolution neural network, support vector machine and Sobel filter for accurate detection of COVID-19 patients using X-ray images

The coronavirus (COVID-19) is currently the most common contagious disease which is prevalent all over the world. The main challenge of this disease is the primary diagnosis to prevent secondary infections and its spread from one person to another. Therefore, it is essential to use an automatic diagnosis system along with clinical procedures for the rapid diagnosis of COVID-19 to prevent its spread. Artificial intelligence techniques using computed tomography (CT) images of the lungs and chest radiography have the potential to obtain high diagnostic performance for Covid-19 diagnosis. In this study, a fusion of convolutional neural network (CNN), support vector machine (SVM), and Sobel filter is proposed to detect COVID-19 using X-ray images. A new X-ray image dataset was collected and subjected to high pass filter using a Sobel filter to obtain the edges of the images. Then these images are fed to CNN deep learning model followed by SVM classifier with ten-fold cross validation strategy. This method is designed so that it can learn with not many data. Our results show that the proposed CNN-SVM with Sobel filter (CNN-SVM + Sobel) achieved the highest classification accuracy, sensitivity and specificity of 99.02%, 100% and 95.23%, respectively in automated detection of COVID-19. It showed that using Sobel filter can improve the performance of CNN. Unlike most of the other researches, this method does not use a pre-trained network. We have also validated our developed model using six public databases and obtained the highest performance. Hence, our developed model is ready for clinical application.

Fusion of CNN and sparse representation for threat estimation near power lines and poles infrastructure using aerial stereo imagery

Fires or electrical hazards and accidents can occur if vegetation is not controlled or cleared around overhead power lines, resulting in serious risks to people and property and significant costs to the community. There are numerous blackouts due to interfering the trees with the power transmission lines in hilly and urban areas. Power distribution companies are facing a challenge to monitor the vegetation to avoid blackouts and flash-over threats. Recently, several methods have been developed for vegetation monitoring; however, existing methods are either not accurate or could not provide better disparity map in the textureless region. Moreover, are not able to handle depth discontinuity in stereo thus are not able to find a feasible solution in the smooth areas to compute the disparity map. This study presents a cost-effective framework based on UAV and satellite Stereo images to monitor the trees and vegetation, which provide better disparity. We present a novel approach based on the fusion of the convolutional neural network (CNN) and sparse representation that handled textureless region, depth discontinuity and smooth region to produce better disparity map that further used for threat estimation using height and distance of vegetation/trees near power lines and poles. Extensive experimental evaluation on real time powerline monitoring showed considerable imporvemnt in vegetation threat estimation with accuracy of 90.3% in comparison to graph-cut, dynamic programming, belief propagation, and area-based methods.