Multi-layer Deep Neural Network Research Articles

Rapid quantitative imaging of high intensity ultrasonic pressure fields.

High intensity focused ultrasound (FUS) is a noninvasive technique for treatment of tissues that can lie deep within the body. There is a need for methods to rapidly and quantitatively map FUS pressure beams for quality assurance and accelerate development of FUS systems and techniques. However, conventional ultrasound pressure beam mapping instruments, including hydrophones and optical techniques, are slow, not portable, and expensive, and most cannot map beams at actual therapeutic pressure levels. Here, a rapid projection imaging method to quantitatively map FUS pressure beams based on continuous-wave background-oriented schlieren (CW-BOS) imaging is reported. The method requires only a water tank, a background pattern, and a camera and uses a multi-layer deep neural network to reconstruct two-dimensional root-mean-square (RMS) projected pressure maps that resolve the ultrasound propagation dimension and one lateral dimension. In this work, the method was applied to collect beam maps over a 3 × 1 cm2 field-of-view with 0.425 mm resolution for focal pressures up to 9 MPa. Results at two frequencies and comparisons to hydrophone measurements show that CW-BOS imaging produces high-resolution quantitative RMS projected FUS pressure maps in under 10 s, the technique is linear and robust to beam rotations and translations, and it can map aberrated beams.

Inference With Deep Generative Priors in High Dimensions

Deep generative priors offer powerful models for complex-structured data, such as images, audio, and text. Using these priors in inverse problems typically requires estimating the input and/or hidden signals in a multi-layer deep neural network from observation of its output. While these approaches have been successful in practice, rigorous performance analysis is complicated by the non-convex nature of the underlying optimization problems. This paper presents a novel algorithm, Multi-Layer Vector Approximate Message Passing (ML-VAMP), for inference in multi-layer stochastic neural networks. ML-VAMP can be configured to compute maximum a priori (MAP) or approximate minimum mean-squared error (MMSE) estimates for these networks. We show that the performance of ML-VAMP can be exactly predicted in a certain high-dimensional random limit. Furthermore, under certain conditions, ML-VAMP yields estimates that achieve the minimum (i.e., Bayes-optimal) MSE as predicted by the replica method. In this way, ML-VAMP provides a computationally efficient method for multi-layer inference with an exact performance characterization and testable conditions for optimality in the large-system limit.

Bidirectional Backpropagation

We extend backpropagation (BP) learning from ordinary unidirectional training to bidirectional training of deep multilayer neural networks. This gives a form of backward chaining or inverse inference from an observed network output to a candidate input that produced the output. The trained network learns a bidirectional mapping and can apply to some inverse problems. A bidirectional multilayer neural network can exactly represent some invertible functions. We prove that a fixed three-layer network can always exactly represent any finite permutation function and its inverse. The forward pass computes the permutation function value. The backward pass computes the inverse permutation with the same weights and hidden neurons. A joint forward–backward error function allows BP learning in both directions without overwriting learning in either direction. The learning applies to classification and regression. The algorithms do not require that the underlying sampled function has an inverse. A trained regression network tends to map an output back to the centroid of its preimage set.

DEEP MULTILAYER NEURAL NETWORK FOR PREDICTING THE WINNER OF FOOTBALL MATCHES

In this work, we draw attention to prediction of football (soccer) match winner. We propose the deep multilayer neural network based on elastic net regularization that predicts the winner of the English Premier League football matches. Our main interest is to predict the match result (win, loss or draw). In our experimental study, we prove that using open access limited data such as team shots, shots on target, yellow and red cards, etc. the system has a good prediction accuracy and profitability. The proposed approach should be considered as a basis of Oracle engine for predicting the match outcomes.

DEEP MULTILAYER NEURAL NETWORK FOR PREDICTING THE WINNER OF FOOTBALL MATCHES

In this work, we draw attention to prediction of football (soccer) match winner. We propose the deep multilayer neural network based on elastic net regularization that predicts the winner of the English Premier League football matches. Our main interest is to predict the match result (win, loss or draw). In our experimental study, we prove that using open access limited data such as team shots, shots on target, yellow and red cards, etc. the system has a good prediction accuracy and profitability. The proposed approach should be considered as a basis of Oracle engine for predicting the match outcomes.

SACNN: Self-Attention Convolutional Neural Network for Low-Dose CT Denoising With Self-Supervised Perceptual Loss Network.

Computed tomography (CT) is a widely used screening and diagnostic tool that allows clinicians to obtain a high-resolution, volumetric image of internal structures in a non-invasive manner. Increasingly, efforts have been made to improve the image quality of low-dose CT (LDCT) to reduce the cumulative radiation exposure of patients undergoing routine screening exams. The resurgence of deep learning has yielded a new approach for noise reduction by training a deep multi-layer convolutional neural networks (CNN) to map the low-dose to normal-dose CT images. However, CNN-based methods heavily rely on convolutional kernels, which use fixed-size filters to process one local neighborhood within the receptive field at a time. As a result, they are not efficient at retrieving structural information across large regions. In this paper, we propose a novel 3D self-attention convolutional neural network for the LDCT denoising problem. Our 3D self-attention module leverages the 3D volume of CT images to capture a wide range of spatial information both within CT slices and between CT slices. With the help of the 3D self-attention module, CNNs are able to leverage pixels with stronger relationships regardless of their distance and achieve better denoising results. In addition, we propose a self-supervised learning scheme to train a domain-specific autoencoder as the perceptual loss function. We combine these two methods and demonstrate their effectiveness on both CNN-based neural networks and WGAN-based neural networks with comprehensive experiments. Tested on the AAPM-Mayo Clinic Low Dose CT Grand Challenge data set, our experiments demonstrate that self-attention (SA) module and autoencoder (AE) perceptual loss function can efficiently enhance traditional CNNs and can achieve comparable or better results than the state-of-the-art methods.

Bone mineral density value evaluation based on photoacoustic spectral analysis combined with deep learning method

The diagnosis of osteoporosis is eventually converted to the measurement of bone mineral density (BMD) in clinical trials. Since our previous work had proved the ability of using photoacoustic spectral analysis (PASA) to efficiently detect osteoporosis, in this contribution, we proposed a fully connected multi-layer deep neural network combined with PASA to semi-quantify BMD values corresponding to varying degrees of bone loss and to further evaluate the degree of osteoporosis. Experiments were carried out on swine femur heads, and the performance of our proposed method is satisfying for future clinical screening.

Voice Conversion System Based on Deep Neural Networks

The paper focuses on usage of deep neural networks for converting a person’s voice to another person’s voice, analogous to a mimic. The work in this paper introduces the concept of neural networks and deploys multi-layer deep neural networks for building a framework for voice conversion. The spectral Mel-Frequency Cepstral Coefficients (MFCCs) are converted using a 10-layer deep network while fundamental frequency (F0) conversion is accomplished by logarithmic Gaussian normalized transformation. MFCCs are subjected to inverse cepstral filtering while changes in F0 are incorporated using Pitch Synchronous OverLap Add (PSOLA) algorithm for re-synthesis. The results obtained are compared using Mel Cepstral Distortion (MCD) for objective evaluation while ABX-listening test is conducted for subjective assessment. Maximum improvement in MCD of 13.87% is obtained for female-to-male conversion while ABX-listening test indicates that female-to-male is closest to target with an agreement of 76.2%. The method achieves reasonably good performance compared to state-of-the-art using optimal resources and avoids requirement of highly complex computations.

Real-time watermark reconstruction for the identification of source information based on deep neural network

A novel deep neural network-based image watermarking method is presented to identify the source of digital data that is shared/forwarded on the internet using various messenger apps. The app that is used to share/communicate the image at the very first time is also identified in the proposed method. The ten-digit mobile number of the source (user) and identification data of particular messenger app (i.e. WhatsApp, Snapchat, Kik, Facebook messenger, etc.) is combined to get the text watermark signal. The part of the watermark signal representing specific mobile-based messenger application is obtained by randomizing the Walsh orthogonal codes using secret keys. To embed the watermark, the host image (shared/forwarded) is divided into blocks of equal size and then, slantlet transform is applied on each block. To get high reliability, three copies of the source information (user and app) are embedded during watermark embedding. Watermark extraction is performed using trained multilayer deep neural network. Furthermore, an optimal block selection logic is used to get improved results for real-time applications. The method is examined against various signal-processing attacks and high robustness with significant imperceptibility is attained. The method is also found to be fast enough for real-time applications. The prime objective of identifying the first user (source) and the shared/forwarded status (app detection) is successfully accomplished.

Improving the Professional Level of Managers Through Individualized Recommendation to Enhance the Quality of Air Pollutant Management in China

With the rapid development of the economy, and fossil fuel consumption lacking systematic emission controls, China has experienced substantially elevated concentrations of air pollutants, which not only degrades regional air quality but also poses significant impacts on public health. However, faced with the demand for a large number of experts in air pollution protection, people with real expertise for air pollutant management are difficult to find. Therefore, individualized recommendation is an effective and sustainable method for enhancing the professional level of managers and is good for improving the quality of air pollutant management. Thus, this paper initially proposes a novel framework to recommend strengths in air pollutant management. This framework comprises four stages: data preprocessing is the first stage; then, after constructing ability classifications and ability assessment strategies, activity experiences are transformed into corresponding ability values; next, a multilayer perceptron deep neural network (MLP-DNN) is used to predict potential types according to their ability values; finally, a hybrid system is constructed to recommend suitable and sustainable potential managers for air pollutant management. The experiments indicate that the proposed method can assess the full picture of people’s strengths, which can recommend suggestions for building a scientific and rational specialties recommendation system for governments and schools. This method can have significant effects on pollutant emission reduction by enhancing the professional level of managers with regard to air pollutant management.

AI hardware acceleration with analog memory: Microarchitectures for low energy at high speed

In this article, we present innovative microarchitectural designs for multilayer deep neural networks (DNNs) implemented in crossbar arrays of analog memories. Data is transferred in a fully parallel manner between arrays without explicit analog-to-digital converters. Design ideas including source follower-based readout, array segmentation, and transmit-by-duration are adopted to improve the circuit efficiency. The execution energy and throughput, for both DNN training and inference, are analyzed quantitatively using circuit simulations of a full CMOS design in the 90-nm technology node. We find that our current design could achieve up to 12–14 TOPs/s/W energy efficiency for training, while a projected scaled design could achieve up to 250 TOPs/s/W. Key challenges in realizing analog AI systems are discussed.

Real-Time Event Classification in Power System With Renewables Using Kernel Density Estimation and Deep Neural Network

Real-time classification of events facilitates corrective control strategies, supervisory protection schemes, and on-line transient stability assessment of a power system. The synchrophasor-based event classification techniques face challenges like similar responses for different classes of events, i.e., inter-class similarity (ICS), applicability to limited classes of events, and moderate real-time performance for a large power system. In addition, the enhanced ICS effect of increased renewable penetration on events classification needs to be addressed. This paper proposes a kernel density estimation approach for accurate real-time classification of events in a power system with renewables using synchrophasor data. The proposed method uses a diffusion type kernel density estimator (DKDE) to characterize the shape of 3-D voltage and frequency distribution along time in terms of probability density functions (PDFs). That have distinct scale, shape, and orientation for different classes of events. Thereafter, a set of statistical features is derived from PDFs to train a multi-layered deep neural network for event classification. The proposed method is validated for renewables in IEEE-39 bus system and real transmission system of India grid using DIgSILENT/PowerFactory and also on a real phasor measurement unit data for India grid, where it showed better performance for ICS and renewable integration cases.

Few-shot hypercolumn-based mitochondria segmentation in cardiac and outer hair cells in focused ion beam-scanning electron microscopy (FIB-SEM) data

We present a novel AI-based approach to the few-shot automated segmentation of mitochondria in large-scale electron microscopy images. Our framework leverages convolutional features from a pre-trained deep multilayer convolutional neural network, such as VGG-16. We then train a binary gradient boosting classifier on the resulting high-dimensional feature hypercolumns. We extract VGG-16 features from the first four convolutional blocks and apply bilinear upsampling to resize the obtained maps to the input image size. This procedure yields a 2688-dimensional feature hypercolumn for each pixel in a 224 × 224 input image. We then apply L1-regularized logistic regression for supervised active feature selection to reduce dependencies among the features, to reduce overfitting, as well as to speed-up gradient boosting-based training. During inference we block process 1728 × 2022 large microscopy images. Our experiments show that in such a formulation of transfer learning our processing pipeline is able to achieve high-accuracy results on very challenging datasets containing a large number of irregularly shaped mitochondria in cardiac and outer hair cells. Our proposed few-shot training approach gives competitive performance with the state-of-the-art using far less training data.

Advanced Deep Learning Framework for Stock Value Prediction

The main attractive feature to stock market is speedy growth of stock economic value in short yoke of time. The investor analyses the demonstration, estimated value and growth of organizations before investing money in market. The analysis may not be enough by using conventional process or some available methods suggested by different researches. In present days large number of stocks are available in market it is very difficult to study each stock by help of very few suggested foretelling methods. To know the anticipated stock value we need some advanced prediction technology for stock market. This paper introduce an advanced skillful method to plan and analyze the different organizers stock execution in market and prognosticate best suitable stock by predicting close price of stock. The projected arrangement is based on multilayer deep learning neural Network optimized by Adam optimizer. Recent 6 years (2010-2016) data of different organizations are applied to the model to demonstrate the skillfulness of the projected proficient method. From result it has been ascertained that the projected framework is best suited to all different data set of various sectors. The prediction error is very minimal as visible from outcome graph of framework.

An ultra-compact leaky-integrate-and-fire model for building spiking neural networks

We introduce an ultra-compact electronic circuit that realizes the leaky-integrate-and-fire model of artificial neurons. Our circuit has only three active devices, two transistors and a silicon controlled rectifier (SCR). We demonstrate the implementation of biologically realistic features, such as spike-frequency adaptation, a refractory period and voltage modulation of spiking rate. All characteristic times can be controlled by the resistive parameters of the circuit. We built the circuit with out-of-the-shelf components and demonstrate that our ultra-compact neuron is a modular block that can be associated to build multi-layer deep neural networks. We also argue that our circuit has low power requirements, as it is normally off except during spike generation. Finally, we discuss the ultimate ultra-compact limit, which may be achieved by further replacing the SCR circuit with Mott materials.

Packaging Domain-Based Named Entity Recognition with Multi-Layer Neural Networks

Artificial neural networks (ANNs) are the greatest success story that inspired by biological neural networks and neuroscience; ANNs model realistic problems by a network of neurons which are designed by simulating biological neurons. This paper attempts to design a multi-layer neural network to recognize the named entities in packaging domain. For this purpose, a neural network language model was designed to automatically learn the distributed word features and partial speech features. Based on these distributed features, a multi-layer deep neural network model was constructed for the NER of packaging products. The experiments prove that our model can automatically extract more and better advanced features than traditional methods, thus minimizing the workloads of manual feature selection. The results show that the model outperformed the traditional sequence labelled CRF model by 10% in precision and 6% in recall, and that the four-layer neural network with two hidden layers boasted the best NER of packaging products.

Deep Flare Net (DeFN) Model for Solar Flare Prediction

Abstract We developed a solar flare prediction model using a deep neural network (DNN) named Deep Flare Net (DeFN). This model can calculate the probability of flares occurring in the following 24 hr in each active region, which is used to determine the most likely maximum classes of flares via a binary classification (e.g., ≥M class versus <M class or ≥C class versus <C class). From 3 × 105 observation images taken during 2010–2015 by the Solar Dynamic Observatory, we automatically detected sunspots and calculated 79 features for each region, to which flare occurrence labels of X-, M-, and C-class were attached. We adopted the features used in Nishizuka et al. (2017) and added some features for operational prediction: coronal hot brightening at 131 Å (T ≥ 107 K) and the X-ray and 131 Å intensity data 1 and 2 hr before an image. For operational evaluation, we divided the database into two for training and testing: the data set in 2010–2014 for training, and the one in 2015 for testing. The DeFN model consists of deep multilayer neural networks formed by adapting skip connections and batch normalizations. To statistically predict flares, the DeFN model was trained to optimize the skill score, i.e., the true skill statistic (TSS). As a result, we succeeded in predicting flares with TSS = 0.80 for ≥M-class flares and TSS = 0.63 for ≥C-class flares. Note that in usual DNN models, the prediction process is a black box. However, in the DeFN model, the features are manually selected, and it is possible to analyze which features are effective for prediction after evaluation.

Medical image analysis of abdominal X-ray CT images by deep multi-layered GMDH-type neural network

In this study, a deep multi-layered group method of data handling (GMDH)-type neural network is applied to the medical image analysis of the abdominal X-ray computed tomography (CT) images. The deep neural network architecture which has many hidden layers are automatically organized using the deep multi-layered GMDH-type neural network algorithm so as to minimize the prediction error criterion defined as Akaike’s information criterion (AIC) or prediction sum of squares (PSS). The characteristics of the medical images are very complex and therefore the deep neural network architecture is very useful for the medical image diagnosis and medical image recognition. In this study, it is shown that this deep multi-layered GMDH-type neural network is useful for the medical image analysis of abdominal X-ray CT images.

FoodNet: Recognizing Foods Using Ensemble of Deep Networks

In this work we propose a methodology for an automatic food classification system which recognizes the contents of the meal from the images of the food. We developed a multi-layered deep convolutional neural network (CNN) architecture that takes advantages of the features from other deep networks and improves the efficiency. Numerous classical handcrafted features and approaches are explored, among which CNNs are chosen as the best performing features. Networks are trained and fine-tuned using preprocessed images and the filter outputs are fused to achieve higher accuracy. Experimental results on the largest real-world food recognition database ETH Food-101 and newly contributed Indian food image database demonstrate the effectiveness of the proposed methodology as compared to many other benchmark deep learned CNN frameworks.

Deepgender: real-time gender classification using deep learning for smartphones

Face recognition, expression identification, age determination, racial binding and gender classification are common examples of image processing computerization. Gender classification is very straightforward for us like we can tell by the person’s hair, nose, eyes, mouth and skin whether that person is a male or female with a relatively high degree of confidence and accuracy; however, can we program a computer to perform just as well at gender classification? The very problem is the main focus of this research. The conventional sequence for recent real-time facial image processing consists of five steps: face detection, noise removal, face alignment, feature representation and classification. With the aim of human gender classification, face alignment and feature vector extraction stages have been re-examined keeping in view the application of the system on smartphones. Face alignment has been made by spotting out 83 facial landmarks and 3-D facial model with the purpose of applying affine transformation. Furthermore, ‘feature representation’ is prepared through proposed modification in multilayer deep neural network, and hence we name it Deepgender. This convolutional deep neural network consists of some locally connected hidden layers without common weights of kernels as previously followed in legacy layered architecture. This specific case study involves deep learning as four convolutional layers, three max-pool layers (for downsizing of unrelated data), two fully connected layers (connection of outcome to all inputs) and a single layer of ‘multinomial logistic regression.’ Training has been made using CAS-PEAL and FEI which contain 99,594 face images of 1040 people and 2800 face images of 200 individuals, respectively. These images are either in different poses or taken under uncontrolled conditions which are close to real-time input facial image for gender classification application. The proposed system ‘Deepgender’ has registered 98% accuracy by combined use of both databases with the specific preprocess procedure, i.e., exhibiting alignment before resizing. Experiments suggest that accuracy is nearly 100% with frontal and nonblurred facial images. State-of-the-art steps have been taken to overcome memory and battery constraints in mobiles.