Hierarchical Convolutional Neural Network Research Articles

Research on Rotating Machinery Fault Diagnosis Method Based on Energy Spectrum Matrix and Adaptive Convolutional Neural Network

Traditional intelligent fault diagnosis methods focus on distinguishing different fault modes, but ignore the deterioration of fault severity. This paper proposes a new two-stage hierarchical convolutional neural network for fault diagnosis of rotating machinery bearings. The failure mode and failure severity are modeled as a hierarchical structure. First, the original vibration signal is transformed into an energy spectrum matrix containing fault-related information through wavelet packet decomposition. Secondly, in the model training method, an adaptive learning rate dynamic adjustment strategy is further proposed, which adaptively extracts robust features from the spectrum matrix for fault mode and severity diagnosis. To verify the effectiveness of the method, the bearing fault data was collected using a rotating machine test bench. On this basis, the diagnostic accuracy, convergence performance and robustness of the model under different signal-to-noise ratios and variable load environments are evaluated, and the feature learning ability of the method is verified by visual analysis. Experimental results show that this method has achieved satisfactory results in both fault pattern recognition and fault severity evaluation, and is superior to other machine learning and deep learning methods.

A deep CNN‐based approach for predicting MCI to AD conversion

AbstractBackgroundDeep convolutional neural networks (CNNs) have demonstrated a considerable ability to decode various biomedical signals, including 2D and 3D neuroimaging data. Magnetoencephalography‐based functional connectivity (MEG‐FC) is proven to be a useful tool in predicting the conversion from mild cognitive impairment (MCI) to Alzheimer's disease (AD), since early signs of AD may be hidden in the way neuronal activity is spatially distributed and coordinated among different brain regions. In the present work, we propose a deep learning approach for the early diagnosis of AD, able to capture the electrophysiological anomalies occurring before the conversion by exploit an image‐based coding of MEG‐FC data. The proposed approach has been tested on a set of resting‐state MEG recordings, from a longitudinal study (24 months between 2 MEG scans for each subject) involving 54 participants: 27 'progressive' patients (i.e., MCI at the time of the 1st scan converted to AD at the time of the 2nd scan), and 27 'static' patients (i.e., MCI at the time of the 1st scan remained MCI at the time of the 2nd scan).MethodMEG and MRI data have been gathered from the 54 patients using an Elekta Vectorview system and a General Electric 1.5T MRI scanner, respectively. Source reconstruction was performed with LCMV beamformer. The obtained neural MEG sources were anatomically parcellated into 90 ROIs according to the AAL atlas. Considering the signals stemming from the 90 ROIs we extracted the MEG‐FC measures (PLV and COR), for different frequency bands [Hz]: θ(4‐8), α(8‐12), β(12‐30), and γ(30‐55). MEG‐FC has been then coded into image‐based representations and hierarchically decomposed by means of the CNN layers of 'AlexNet'. Finally, different classification techniques have been tested as predictive modules.ResultWith the proposed approach we obtained an accuracy of 0.9 in discriminating ‘static’ from ‘progressive’ patients using MEG‐FC data from the first scan. The best classification results have been obtained using the Linear Discriminant Analysis and Support Vector Machine techniques.ConclusionThe classification of FC profiles mediated by hierarchical CNNs decomposition is a promising method for detecting the early neurophysiological anomalies characterizing the progression to AD.

Sequence-to-Sequence Video Prediction by Learning Hierarchical Representations

Video prediction which maps a sequence of past video frames into realistic future video frames is a challenging task because it is difficult to generate realistic frames and model the coherent relationship between consecutive video frames. In this paper, we propose a hierarchical sequence-to-sequence prediction approach to address this challenge. We present an end-to-end trainable architecture in which the frame generator automatically encodes input frames into different levels of latent Convolutional Neural Network (CNN) features, and then recursively generates future frames conditioned on the estimated hierarchical CNN features and previous prediction. Our design is intended to automatically learn hierarchical representations of video and their temporal dynamics. Convolutional Long Short-Term Memory (ConvLSTM) is used in combination with skip connections so as to separately capture the sequential structures of multiple levels of hierarchy of features. We adopt Scheduled Sampling for training our recurrent network in order to facilitate convergence and to produce high-quality sequence predictions. We evaluate our method on the Bouncing Balls, Moving MNIST, and KTH human action dataset, and report favorable results as compared to existing methods.

A Feature Inherited Hierarchical Convolutional Neural Network (FI-HCNN) for Motor Fault Severity Estimation Using Stator Current Signals

Motors, which are one of the most widely used machines in the manufacturing field, take charge of a key role in precision machining. Therefore, it is important to accurately estimate the health state of the motor that affects the quality of the product. The research outlined in this paper aims to improve motor fault severity estimation by suggesting a novel deep learning method, specifically, feature inherited hierarchical convolutional neural network (FI-HCNN). FI-HCNN consists of a fault diagnosis part and a severity estimation part, arranged hierarchically. The main novelty of the proposed FI-HCNN is the special inherited structure between the hierarchy; the severity estimation part utilizes the latent features to exploit the fault-related representations in the fault diagnosis task. FI-HCNN can improve the accuracy of the fault severity estimation because the level-specific abstraction is supported by the latent features. Also, FI-HCNN has ease in practical application because it is developed based on stator current signals which are usually acquired for a control purpose. Experimental studies of mechanical motor faults, including eccentricity, broken rotor bars, and unbalanced conditions, are used to corroborate the high performance of FI-HCNN, as compared to both conventional methods and other hierarchical deep learning methods.

Deep learning-based method for reducing residual motion effects in diffusion parameter estimation.

Conventional motion-correction techniques for diffusion MRI can introduce motion-level-dependent bias in derived metrics. To address this challenge, a deep learning-based technique was developed to minimize such residual motion effects. The data-rejection approach was adopted in which motion-corrupted data are discarded before model-fitting. A deep learning-based parameter estimation algorithm, using a hierarchical convolutional neural network (H-CNN), was combined with motion assessment and corrupted volume rejection. The method was designed to overcome the limitations of existing methods of this kind that produce parameter estimations whose quality depends strongly on a proportion of the data discarded. Evaluation experiments were conducted for the estimation of diffusion kurtosis and diffusion-tensor-derived measures at both the individual and group levels. The performance was compared with the robust approach of iteratively reweighted linear least squares (IRLLS) after motion correction with and without outlier replacement. Compared with IRLLS, the H-CNN-based technique is minimally sensitive to motion effects. It was tested at severe motion levels when 70% to 90% of the data are rejected and when random motion is present. The technique had a stable performance independent of the numbers and schemes of data rejection. A further test on a data set from children with attention-deficit hyperactivity disorder shows the technique can potentially ameliorate spurious group-level difference caused by head motion. This method shows great potential for reducing residual motion effects in motion-corrupted diffusion-weighted-imaging data, bringing benefits that include reduced bias in derived metrics in individual scans and reduced motion-level-dependent bias in population studies employing diffusion MRI.

Automatic multi-organ segmentation in computed tomography images using hierarchical convolutional neural network.

Purpose: Accurate segmentation of treatment planning computed tomography (CT) images is important for radiation therapy (RT) planning. However, low soft tissue contrast in CT makes the segmentation task challenging. We propose a two-step hierarchical convolutional neural network (CNN) segmentation strategy to automatically segment multiple organs from CT. Approach: The first step generates a coarse segmentation from which organ-specific regions of interest (ROIs) are produced. The second step produces detailed segmentation of each organ. The ROIs are generated using UNet, which automatically identifies the area of each organ and improves computational efficiency by eliminating irrelevant background information. For the fine segmentation step, we combined UNet with a generative adversarial network. The generator is designed as a UNet that is trained to segment organ structures and the discriminator is a fully convolutional network, which distinguishes whether the segmentation is real or generator-predicted, thus improving the segmentation accuracy. We validated the proposed method on male pelvic and head and neck (H&N) CTs used for RT planning of prostate and H&N cancer, respectively. For the pelvic structure segmentation, the network was trained to segment the prostate, bladder, and rectum. For H&N, the network was trained to segment the parotid glands (PG) and submandibular glands (SMG). Results: The trained segmentation networks were tested on 15 pelvic and 20H&N independent datasets. The H&N segmentation network was also tested on a public domain dataset ( ) and showed similar performance. The average dice similarity coefficients ( ) of pelvic structures are (prostate), (bladder), (rectum), and H&N structures are (PG) and (SMG). The segmentation for each CT takes on average. Conclusions: Experimental results demonstrate that the proposed method can produce fast, accurate, and reproducible segmentation of multiple organs of different sizes and shapes and show its potential to be applicable to different disease sites.

Robust RGB-D tracking via compact CNN features

Feature representation is at the core of visual tracking. This paper presents a robust tracking method in RGB-D videos. Firstly, the RGB and depth images are separately encoded using a hierarchical convolutional neural network (CNN) features. Secondly, in order to reduce computation cost, we exploit random projection to compress the CNN features. The high dimensional CNN features are randomly projected into a low dimensional feature space. The correlation filter tracking framework is then independently carried out in RGB and depth images. And backward tracking scheme is adopted to evaluate the tracking results in these two images. The final position is determined according to the tracked location in the two image channels. In addition, model updating is implemented adaptively. Our tracker is evaluated on two RGB-D benchmark datasets and achieves comparable results to the other state-of-the-art RGB-D tracking methods.

Difficulty-aware hierarchical convolutional neural networks for deformable registration of brain MR images.

The aim of deformable brain image registration is to align anatomical structures, which can potentially vary with large and complex deformations. Anatomical structures vary in size and shape, requiring the registration algorithm to estimate deformation fields at various degrees of complexity. Here, we present a difficulty-aware model based on an attention mechanism to automatically identify hard-to-register regions, allowing better estimation of large complex deformations. The difficulty-aware model is incorporated into a cascaded neural network consisting of three sub-networks to fully leverage both global and local contextual information for effective registration. The first sub-network is trained at the image level to predict a coarse-scale deformation field, which is then used for initializing the subsequent sub-network. The next two sub-networks progressively optimize at the patch level with different resolutions to predict a fine-scale deformation field. Embedding difficulty-aware learning into the hierarchical neural network allows harder patches to be identified in the deeper sub-networks at higher resolutions for refining the deformation field. Experiments conducted on four public datasets validate that our method achieves promising registration accuracy with better preservation of topology, compared with state-of-the-art registration methods.

SAR Image Classification Using Greedy Hierarchical Learning With Unsupervised Stacked CAEs

Synthetic aperture radar (SAR) can provide stable data source for earth observation due to its advantages of all day and night, all-weather, and strong penetration. SAR image classification as a fundamental procedure has been proved its great value in plenty of remote sensing applications. Conventional classification algorithms mainly rely on hand-designed features, which are susceptible to widespread coherent speckle noise and geometric distortion in high-resolution SAR images. Inspired by the recent impressive success in data mining and deep learning, a greedy hierarchical convolutional neural network (GHCNN) is developed. It aims at obtaining optimized feature representation, relieving the effect of speckle noise, and promoting the local pattern recognition of geometric distortion in single-polarized SAR image classification. First, a series of convolutional autoencoders (CAEs) is trained in the greedy layer-wise unsupervised strategy. This step provides an unbiased regularizer and <i>a priori</i> distribution derived from large volumes of unlabeled SAR patches. Then, to optimize multiple parameter subspaces globally, several CAEs are coupled together to form a deeper hierarchical structure in a stacked and unsupervised fashion. Afterward, a convolutional network with identical topology inherits the pretrained weights. After supervised finetuning, it realizes class prediction. Synchronously, t-distributed stochastic neighbor embedding (t-SNE) algorithm is applied to monitor the efficiency of feature representation during the training period. Experimental results demonstrate that the proposed method has competitive advantages over involved contrast methods.

A deep learning–based method for improving reliability of multicenter diffusion kurtosis imaging with varied acquisition protocols

Multicenter magnetic resonance imaging is gaining more popularity in large-sample projects. Since both varying hardware and software across different centers cause unavoidable data heterogeneity across centers, its impact on reliability in study outcomes has also drawn much attention recently. One fundamental issue arises in how to derive model parameters reliably from image data of varying quality. This issue is even more challenging for advanced diffusion methods such as diffusion kurtosis imaging (DKI). Recently, deep learning–based methods have been demonstrated with their potential for robust and efficient computation of diffusion-derived measures. Inspired by these approaches, the current study specifically designed a framework based on a three-dimensional hierarchical convolutional neural network, to jointly reconstruct and harmonize DKI measures from multicenter acquisition to reformulate these to a state-of-the-art hardware using data from traveling subjects. The results from the harmonized data acquired with different protocols show that: 1) the inter-scanner variation of DKI measures within white matter was reduced by 51.5% in mean kurtosis, 65.9% in axial kurtosis, 53.7% in radial kurtosis, and 61.5% in kurtosis fractional anisotropy, respectively; 2) data reliability of each single scanner was enhanced and brought to the level of the reference scanner; and 3) the harmonization network was able to reconstruct reliable DKI values from high data variability. Overall the results demonstrate the feasibility of the proposed deep learning–based method for DKI harmonization and help to simplify the protocol setup procedure for multicenter scanners with different hardware and software configurations.

Decidual Vasculopathy Identification in Whole Slide Images Using Multiresolution Hierarchical Convolutional Neural Networks

After a child is born, the examination of the placenta by a pathologist for abnormalities, such as infection or maternal vascular malperfusion, can provide important information about the immediate and long-term health of the infant. Detection of the pathologic placental blood vessel lesion decidual vasculopathy (DV) has been shown to predict adverse pregnancy outcomes, such as preeclampsia, which can lead to mother and neonatal morbidity in subsequent pregnancies. However, because of the high volume of deliveries at large hospitals and limited resources, currently a large proportion of delivered placentas are discarded without inspection. Furthermore, the correct diagnosis of DV often requires the expertise of an experienced perinatal pathologist. We introduce a hierarchical machine learning approach for the automated detection and classification of DV lesions in digitized placenta slides, along with a method of coupling learned image features with patient metadata to predict the presence of DV. Ultimately, the approach will allow many more placentas to be screened in a more standardized manner, providing feedback about which cases would benefit most from more in-depth pathologic inspection. Such computer-assisted examination of human placentas will enable real-time adjustment to infant and maternal care and possible chemoprevention (eg, aspirin therapy) to prevent preeclampsia, a disease that affects 2% to 8% of pregnancies worldwide, in women identified to be at risk with future pregnancies.

A hierarchical deep convolutional neural network and gated recurrent unit framework for structural damage detection

Structural damage detection has become an interdisciplinary area of interest for various engineering fields, while the available damage detection methods are being in the process of adapting machine learning concepts. Most machine learning based methods heavily depend on extracted “hand-crafted” features that are manually selected in advance by domain experts and then, fixed. Recently, deep learning has demonstrated remarkable performance on traditional challenging tasks, such as image classification, object detection, etc., due to the powerful feature learning capabilities. This breakthrough has inspired researchers to explore deep learning techniques for structural damage detection problems. However, existing methods have considered either spatial relation (e.g., using convolutional neural network (CNN)) or temporal relation (e.g., using long short term memory network (LSTM)) only. In this work, we propose a novel Hierarchical CNN and Gated recurrent unit (GRU) framework to model both spatial and temporal relations, termed as HCG, for structural damage detection. Specifically, CNN is utilized to model the spatial relations and the short-term temporal dependencies among sensors, while the output features of CNN are fed into the GRU to learn the long-term temporal dependencies jointly. Extensive experiments on IASC-ASCE structural health monitoring benchmark and scale model of three-span continuous rigid frame bridge structure datasets have shown that our proposed HCG outperforms other existing methods for structural damage detection significantly.

Hierarchical convolutional neural network via hierarchical cluster validity based visual tree learning

In multi-category classification task, some categories have strong inter-categories similarity, while others do not. Therefore, it is unreasonable to treat all these categories equally. One possible way is to organize all categories into a hierarchical structure and train a hierarchical classifier based on it. The general convolutional neural networks (CNN) can be seen as a flat classifier on hierarchical feature representations. Therefore, it is natural to combine the hierarchical structure and deep neural networks. However, for hierarchical classification, one open issue is how to build a reasonable hierarchical structure which characterizes the inter-relations between categories. An effective approach is to utilize hierarchical clustering to build a visual tree structure, but the critical issue is how to determine the number of clusters in hierarchical clustering. In this paper, a hierarchical cluster validity index (HCVI) is developed for supporting visual tree learning. Before clustering of each level begins, we will measure the impact of different numbers of clusters on visual tree building and select the most suitable number of clusters. Based on this visual tree, a hierarchical convolutional neural network (HCNN) can be trained for achieving more discriminative capability. Our experimental results have demonstrated that the proposed hierarchical cluster validity index (HCVI) can guide the building of a more reasonable visual tree structure, so that the hierarchical convolutional neural network can achieve better results on classification accuracy.

Time-resolved correspondences between deep neural network layers and EEG measurements in object processing

The ventral visual stream is known to be organized hierarchically, where early visual areas processing simplistic features feed into higher visual areas processing more complex features. Hierarchical convolutional neural networks (CNNs) were largely inspired by this type of brain organization and have been successfully used to model neural responses in different areas of the visual system. In this work, we aim to understand how an instance of these models corresponds to temporal dynamics of human object processing. Using representational similarity analysis (RSA) and various similarity metrics, we compare the model representations with two electroencephalography (EEG) data sets containing responses to a shared set of 72 images. We find that there is a hierarchical relationship between the depth of a layer and the time at which peak correlation with the brain response occurs for certain similarity metrics in both data sets. However, when comparing across layers in the neural network, the correlation onset time did not appear in a strictly hierarchical fashion. We present two additional methods that improve upon the achieved correlations by optimally weighting features from the CNN and show that depending on the similarity metric, deeper layers of the CNN provide a better correspondence than shallow layers to later time points in the EEG responses. However, we do not find that shallow layers provide better correspondences than those of deeper layers to early time points, an observation that violates the hierarchy and is in agreement with the finding from the onset-time analysis. This work makes a first comparison of various response features—including multiple similarity metrics and data sets—with respect to a neural network.

Detection and diagnosis of pancreatic tumor using deep learning-based hierarchical convolutional neural network on the internet of medical things platform

Pancreatic cancer is one of the deadliest types of cancer and its prognosis is extremely poor in the present scenario. Automatic pancreatic tumor image segmentation is often provided by computer-aided screening (CAD), diagnosis and quantitative evaluations in radiology images such as CT and MRI. Tumor classification through these methods can also help to track, predict and endorse customized therapy as a part of effective treatment, without invasions of cancer. Nowadays, neural networks (NN) have shown promising results for precise pancreatic image segmentation. This paper presents a deep learning-based Hierarchical Convolutional Neural Network (HCNN) for pancreatic tumor detection. A recurrent neural network (RNN) is provided to meet the issue of spatial discrepancy segmentation across slices of adjacent images. The recurrent neural network generates CNN results and fine-tunes its segmentation by enhancing the smoothness and shape. Furthermore, the HCNN training and configurations objectives have been illustrated to the performance of pancreatic tumor image segmentation. The experimental results demonstrate that the proposed approach can improve the performance of the classifier and reduce the cost on the Internet of medical things platform (IoMT).

Robust fusion for RGB-D tracking using CNN features

Recently, RGB-D sensors have become popular. Many computer vision problems can be better dealt with depth data. It is a challenging problem to integrate depth data into a visual object tracker to address the problems such as scale change and occlusion. In this paper, we propose a robust fusion based RGB-D tracking method. Specifically, hierarchical convolutional neural network (CNN) features are first adopted to encode RGB and depth images separately. Next, target is tracked based on correlation filter tracking framework. Then the results of each CNN feature are localized according to the tracking results in a short period of time. Finally, the target is localized by jointly fusing the results of RGB and depth images. Model updating is finally carried out according to the differences between RGB and depth images. Experiments on the University of Birmingham RGB-D Tracking Benchmark (BTB) and the Princeton RGB-D Tracking Benchmark (PTB) achieve comparable results to state-of-the-art methods.

A Novel Electronic Component Classification Algorithm Based on Hierarchical Convolution Neural Network

In the paper, the author proposes a recognition and classification algorithm of electronic components based on hierarchical convolutional neural network (NH-CNN) that reduces the computational complexity on the drawbacks of traditional image classification method based on deep learning such as the failure of effectively combining multiple deep characteristics, the poor performance of classifier, the difficulty of parameter adjustment and the long training time. The algorithm is trained through the Convolutional Automatic Coding (CAE) layer to obtain relevant feature maps, which reduces the input parameters of the designed convolutional neural network. Since the CAE model and the convolutional neural network have similar convolution and pooling operations, the feature maps obtained from the CAE model are put into the designed neural network based on the transfer learning method. Finally, the feature fusion method is used to output the obtained features to the fully connected layer, which is used to better express the depth information contained in the electronic component images and improve the accuracy of classification. The experimental results show that the proposed algorithm can effectively extract depth features with high precision of 94.26% and less complexity and overcome the defects of traditional image classification algorithms such as manual image extraction and low classification efficiency.

Real-time license plate detection and recognition using deep convolutional neural networks

Automatic License Plate Recognition (ALPR) is an important task with many applications in Intelligent Transportation and Surveillance systems. This work presents an end-to-end ALPR method based on a hierarchical Convolutional Neural Network (CNN). The core idea of the proposed method is to identify the vehicle and the license plate region using two passes on the same CNN, and then to recognize the characters using a second CNN. The recognition CNN massively explores the use of synthetic and augmented data to cope with limited training datasets, and our results show that the augmentation process significantly increases the recognition rate. In addition, we present a novel temporal coherence technique to better stabilize the OCR output in videos. Our method was tested with publicly available datasets containing Brazilian and European license plates, achieving accuracy rates better than competitive academic methods and a commercial system.

Traffic Sign Recognition in Harsh Environment Using Attention Based Convolutional Pooling Neural Network

Convolutional neural networks (CNNs) have achieved significant progress in computer vision systems, helping to efficiently obtain feature information by sliding filters on the input images. However, CNNs have difficulty capturing specific properties when the images are affected by various noises. This paper proposes an attention based convolutional pooling neural network (ACPNN) where an attention-mechanism is applied to feature maps to obtain key features, and max pooling is replaced with convolutional pooling to improve recognition accuracy in harsh environments. The ACPNN with attention mechanism and convolutional pooling structure is robust against external noises and maintains classification performance under such conditions. The proposed ACPNN was validated on the German traffic sign recognition benchmark with various cases. Considering the traffic signs are suffered from various noises, the recognition performances were demonstrated with conventional CNN and state-of-the art CNNs such as multi-scale CNN, committee of CNN, hierarchical CNN, and multi-column deep neural network. Under such harsh conditions, the proposed ACPNN shows 66.981% and 83.198% respectively, which are the best performances compared to other CNNs.

CNN tracking based on data augmentation

Correlation filter based tracking methods have aroused increasing attention due to the appealing performance on tracking benchmark datasets. For each frame, a filter is trained to separate the object from its background. Considering that the object always undergoes challenging situations, the trained filter should consider both external and internal distractions. In this paper, we propose a data augmentation based robust visual tracking algorithm to better generalize the training data. Specifically, data augmentation technique is utilized to generate training samples to improve the robustness of the training filter. Then hierarchical convolutional neural network (CNN) features are utilized to encode the target and the augmented sample. Different from previous work, we exploit to use a hash matrix to reduce the dimension of the CNN features. Next, the correlation filter tracking method is employed. The tracking results of multiple hash features are combined to locate the target. Extensive experiments on five large scale datasets show that the proposed method achieves comparable results to state-of-the-art trackers.