Multi-layer Deep Neural Network Research Articles

Medical image analysis of abdominal X-ray CT images by deep multi-layered GMDH-type neural network

In this study, a deep multi-layered group method of data handling (GMDH)-type neural network is applied to the medical image analysis of the abdominal X-ray computed tomography (CT) images. The deep neural network architecture which has many hidden layers are automatically organized using the deep multi-layered GMDH-type neural network algorithm so as to minimize the prediction error criterion defined as Akaike’s information criterion (AIC) or prediction sum of squares (PSS). The characteristics of the medical images are very complex and therefore the deep neural network architecture is very useful for the medical image diagnosis and medical image recognition. In this study, it is shown that this deep multi-layered GMDH-type neural network is useful for the medical image analysis of abdominal X-ray CT images.

FoodNet: Recognizing Foods Using Ensemble of Deep Networks

In this work we propose a methodology for an automatic food classification system which recognizes the contents of the meal from the images of the food. We developed a multi-layered deep convolutional neural network (CNN) architecture that takes advantages of the features from other deep networks and improves the efficiency. Numerous classical handcrafted features and approaches are explored, among which CNNs are chosen as the best performing features. Networks are trained and fine-tuned using preprocessed images and the filter outputs are fused to achieve higher accuracy. Experimental results on the largest real-world food recognition database ETH Food-101 and newly contributed Indian food image database demonstrate the effectiveness of the proposed methodology as compared to many other benchmark deep learned CNN frameworks.

Deepgender: real-time gender classification using deep learning for smartphones

Face recognition, expression identification, age determination, racial binding and gender classification are common examples of image processing computerization. Gender classification is very straightforward for us like we can tell by the person’s hair, nose, eyes, mouth and skin whether that person is a male or female with a relatively high degree of confidence and accuracy; however, can we program a computer to perform just as well at gender classification? The very problem is the main focus of this research. The conventional sequence for recent real-time facial image processing consists of five steps: face detection, noise removal, face alignment, feature representation and classification. With the aim of human gender classification, face alignment and feature vector extraction stages have been re-examined keeping in view the application of the system on smartphones. Face alignment has been made by spotting out 83 facial landmarks and 3-D facial model with the purpose of applying affine transformation. Furthermore, ‘feature representation’ is prepared through proposed modification in multilayer deep neural network, and hence we name it Deepgender. This convolutional deep neural network consists of some locally connected hidden layers without common weights of kernels as previously followed in legacy layered architecture. This specific case study involves deep learning as four convolutional layers, three max-pool layers (for downsizing of unrelated data), two fully connected layers (connection of outcome to all inputs) and a single layer of ‘multinomial logistic regression.’ Training has been made using CAS-PEAL and FEI which contain 99,594 face images of 1040 people and 2800 face images of 200 individuals, respectively. These images are either in different poses or taken under uncontrolled conditions which are close to real-time input facial image for gender classification application. The proposed system ‘Deepgender’ has registered 98% accuracy by combined use of both databases with the specific preprocess procedure, i.e., exhibiting alignment before resizing. Experiments suggest that accuracy is nearly 100% with frontal and nonblurred facial images. State-of-the-art steps have been taken to overcome memory and battery constraints in mobiles.

Deep Belief Networks for Multimodal, Images-based Non Contact Material Characterization

Our growing cognisance about the chemo-physical properties of electromagnetic waves and of their interaction with various materials provides expanding range of possibilities for quantification and non contact material characterization even in difficult to computation, « non-standard » domains as this of the cultural heritage. This review - thematically part of the IFIDA project dedicated to digitization of archaeometric and conservation-restoration praxis - deals with the possibilities latest neuroinformatic methods offer for a more efficient and fast interrogation, understanding and classification of multi-modal spectral records of paintings on various supports. The workflow followed during their routine non destructive (ND) analysis by human experts is described in terms of specific technical characteristics, research objectives and concrete application for the artworks' characterization as a prototype to simulate artificially. References to techniques for high semantic level information extraction from low-level features are suggested. The strengths and limitations in application of Deep Belief Networks (DBNs) for compression, visualization and data recognition are also considered. Examples of most appropriate architectures and topological maps of Deep Learning Networks for interpretation of UV, IR, XR, β, γ and CT records performing simple characterization and classification tasks - from single layer perceptron to multi-layer deep belief neural networks for unlabelled data - are presented and explained.

Deep multi-layered GMDH-type neural network using revised heuristic self-organization and its application to medical image diagnosis of liver cancer

In this study, the deep multi-layered group method of data handling (GMDH)-type neural network algorithm using revised heuristic self-organization method is proposed and applied to medical image diagnosis of liver cancer. The deep GMDH-type neural network can automatically organize the deep neural network architecture which has many hidden layers. The structural parameters such as the number of hidden layers, the number of neurons in hidden layers and useful input variables are automatically selected to minimize prediction error criterion defined as Akaike’s information criterion (AIC) or prediction sum of squares (PSS). The architecture of the deep neural network is automatically organized using the revised heuristic self-organization method which is a type of the evolutionary computation. This new neural network algorithm is applied to the medical image diagnosis of the liver cancer and the recognition results are compared with the conventional 3-layered sigmoid function neural network.

Deep learning based monitoring of furnace combustion state and measurement of heat release rate

Effective and efficient monitoring of furnace combustion state and measurement of heat release rate are important and pressing problems in the power industry. However, traditional methods including image segmentation based methods, feature based methods and shallow classifier based methods cannot meet the requirements of highly accurate. These methods are composed with several separating steps, i.e. feature selection and recognition. This paper proposes a novel deep learning based method to identify furnace combustion state and measure heat release rate. With an end-to-end network, feature extraction and classification are integrated into one framework. The deep learning model takes flame images into a multi-layer DNN (Deep Neural Network) or CNN (Convolutional Neural Network) to predict combustion state and heat release rate simultaneously. We also implement smooth and adjustment techniques which can get a trade-off between stability and sensitivity, ensuring both accurate prediction of burner state and fast detection of unstable combustion. The proposed system achieved state-of-the-art 99.9% accuracy in predicting combustion state with a speed of 1 ms per image. Experimental results show that this method has great potential for practical applications on power plants.

Speech Bandwidth Extension Using Recurrent Temporal Restricted Boltzmann Machines

In this paper, we present a new speech bandwidth extension method (BWE) using recurrent temporal restricted Boltzmann machine (RTRBM). The conventional Gaussian mixture model (GMM)-based and deep neural networks (DNNs)-based BWE methods perform stably and effectively. However, the mapping function of GMM-based methods is a piecewise linear transformation, which is insufficient to model the complex nonlinear mapping relationship between the spectral envelope features of low frequency (LF) and high frequency (HF). In the conventional DNNs methods, temporal correlations across speech frames are ignored, resulting in spectral detail loss of the reconstructed speech by BWE. To counter these issues, a multilayer DNN which is composed of two RTRBMs and a feedforward neural network (NN) is employed to obtain time information and model deep nonlinear relationships between the spectral envelope features of LF and HF. The proposed method takes advantage of the strong ability of RTRBM in discovering the temporal correlation in the high-order space and modeling deep nonlinear relationships between input and output. Both the objective and subjective evaluations indicate that our proposed method outperforms the conventional GMM-based methods and other NN-based methods.

Sparseness Analysis in the Pretraining of Deep Neural Networks.

A major progress in deep multilayer neural networks (DNNs) is the invention of various unsupervised pretraining methods to initialize network parameters which lead to good prediction accuracy. This paper presents the sparseness analysis on the hidden unit in the pretraining process. In particular, we use the L1 -norm to measure sparseness and provide some sufficient conditions for that pretraining leads to sparseness with respect to the popular pretraining models-such as denoising autoencoders (DAEs) and restricted Boltzmann machines (RBMs). Our experimental results demonstrate that when the sufficient conditions are satisfied, the pretraining models lead to sparseness. Our experiments also reveal that when using the sigmoid activation functions, pretraining plays an important sparseness role in DNNs with sigmoid (Dsigm), and when using the rectifier linear unit (ReLU) activation functions, pretraining becomes less effective for DNNs with ReLU (Drelu). Luckily, Drelu can reach a higher recognition accuracy than DNNs with pretraining (DAEs and RBMs), as it can capture the main benefit (such as sparseness-encouraging) of pretraining in Dsigm. However, ReLU is not adapted to the different firing rates in biological neurons, because the firing rate actually changes along with the varying membrane resistances. To address this problem, we further propose a family of rectifier piecewise linear units (RePLUs) to fit the different firing rates. The experimental results show that the performance of RePLU is better than ReLU, and is comparable with those with some pretraining techniques, such as RBMs and DAEs.

Assumed Density Filtering Methods for Learning Bayesian Neural Networks

Buoyed by the success of deep multilayer neural networks, there is renewed interest in scalable learning of Bayesian neural networks. Here, we study algorithms that utilize recent advances in Bayesian inference to efficiently learn distributions over network weights. In particular, we focus on recently proposed assumed density filtering based methods for learning Bayesian neural networks -- Expectation and Probabilistic backpropagation. Apart from scaling to large datasets, these techniques seamlessly deal with non-differentiable activation functions and provide parameter (learning rate, momentum) free learning. In this paper, we first rigorously compare the two algorithms and in the process develop several extensions, including a version of EBP for continuous regression problems and a PBP variant for binary classification. Next, we extend both algorithms to deal with multiclass classification and count regression problems. On a variety of diverse real world benchmarks, we find our extensions to be effective, achieving results competitive with the state-of-the-art.

A protein-protein interaction extraction approach based on deep neural network

Protein-Protein Interactions PPIs information extraction from biomedical literature helps unveil the molecular mechanisms of biological processes. Machine learning methods have been the most popular ones in PPI extraction area. However, these methods are still feature engineering-based, which means that their performances are also heavily dependent on the appropriate feature selection which is still a skill-dependent task. This paper presents a deep neural network-based approach which can learn complex and abstract features automatically from unlabelled data by unsupervised representation learning methods. This approach first employs the training algorithm of auto-encoders to initialise the parameters of a deep multilayer neural network. Then the gradient descent method using back propagation is applied to train this deep multilayer neural network model. Experimental results on five public PPI corpora show that our method can achieve better performance than can a multilayer neural network: on two 'toughest handling' corpora AImed and BioInfer, the former outperforms the latter with the improvements of 3.10 and 2.89 percentage units in F-score, respectively. In addition, the performance comparison with APG also verifies the effectiveness of our method.

Multimodal Deep Autoencoder for Human Pose Recovery.

Video-based human pose recovery is usually conducted by retrieving relevant poses using image features. In the retrieving process, the mapping between 2D images and 3D poses is assumed to be linear in most of the traditional methods. However, their relationships are inherently non-linear, which limits recovery performance of these methods. In this paper, we propose a novel pose recovery method using non-linear mapping with multi-layered deep neural network. It is based on feature extraction with multimodal fusion and back-propagation deep learning. In multimodal fusion, we construct hypergraph Laplacian with low-rank representation. In this way, we obtain a unified feature description by standard eigen-decomposition of the hypergraph Laplacian matrix. In back-propagation deep learning, we learn a non-linear mapping from 2D images to 3D poses with parameter fine-tuning. The experimental results on three data sets show that the recovery error has been reduced by 20%-25%, which demonstrates the effectiveness of the proposed method.

Exploring Strategies for Training Deep Neural Networks

Deep multi-layer neural networks have many levels of non-linearities allowing them to compactly represent highly non-linear and highly-varying functions. However, until recently it was not clear how to train such deep networks, since gradient-based optimization starting from random initialization often appears to get stuck in poor solutions. Hinton et al. recently proposed a greedy layer-wise unsupervised learning procedure relying on the training algorithm of restricted Boltzmann machines (RBM) to initialize the parameters of a deep belief network (DBN), a generative model with many layers of hidden causal variables. This was followed by the proposal of another greedy layer-wise procedure, relying on the usage of autoassociator networks. In the context of the above optimization problem, we study these algorithms empirically to better understand their success. Our experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy helps the optimization by initializing weights in a region near a good local minimum, but also implicitly acts as a sort of regularization that brings better generalization and encourages internal distributed representations that are high-level abstractions of the input. We also present a series of experiments aimed at evaluating the link between the performance of deep neural networks and practical aspects of their topology, for example, demonstrating cases where the addition of more depth helps. Finally, we empirically explore simple variants of these training algorithms, such as the use of different RBM input unit distributions, a simple way of combining gradient estimators to improve performance, as well as on-line versions of those algorithms.

Exploring Strategies for Training Deep Neural Networks

Deep multi-layer neural networks have many levels of non-linearities allowing them to compactly represent highly non-linear and highly-varying functions. However, until recently it was not clear ho...