Trained Deep Neural Network Model Research Articles

Design of a Broadband Solar Thermal Absorber Using a Deep Neural Network and Experimental Demonstration of Its Performance

In using nanostructures to design solar thermal absorbers, computational methods, such as rigorous coupled-wave analysis and the finite-difference time-domain method, are often employed to simulate light-structure interactions in the solar spectrum. However, those methods require heavy computational resources and CPU time. In this study, using a state-of-the-art modeling technique, i.e., deep learning, we demonstrate significant reduction of computational costs during the optimization processes. To minimize the number of samples obtained by actual simulation, only regulated amounts are prepared and used as a data set to train the deep neural network (DNN) model. Convergence of the constructed DNN model is carefully examined. Moreover, several analyses utilizing an evolutionary algorithm, which require a remarkable number of performance calculations, are performed using the trained DNN model. We show that deep learning effectively reduces the actual simulation counts compared to the case of a design process without a neural network model. Finally, the proposed solar thermal absorber is fabricated and its absorption performance is characterized.

Deep Neural Network for Ore Production and Crusher Utilization Prediction of Truck Haulage System in Underground Mine

A new method using a deep neural network (DNN) model is proposed to predict the ore production and crusher utilization of a truck haulage system in an underground mine. An underground limestone mine was selected as the study area, and the DNN model input/output nodes were designed to reflect the truck haulage system characteristics. Big data collected on-site for 1 month were processed to create learning datasets. To select the optimal DNN learning model, the numbers of hidden layers and hidden layer nodes were set to various values for analyzing the training and test data. The optimal DNN model structure for ore production prediction was set to five hidden layers and 40 hidden layer nodes. The test data exhibited a coefficient of determination of 0.99 and mean absolute percentage error (MAPE) of 2.80%. The optimal configuration for the crusher utilization prediction was set to four hidden layers and 40 hidden layer nodes, and the test data exhibited a coefficient of determination of 0.99 and MAPE of 2.49%. The trained DNN model was used to predict the ore production and crusher utilization, which were similar to the actual observed values.

Neural Sign Language Translation Based on Human Keypoint Estimation

We propose a sign language translation system based on human keypoint estimation. It is well-known that many problems in the field of computer vision require a massive dataset to train deep neural network models. The situation is even worse when it comes to the sign language translation problem as it is far more difficult to collect high-quality training data. In this paper, we introduce the KETI (Korea Electronics Technology Institute) sign language dataset, which consists of 14,672 videos of high resolution and quality. Considering the fact that each country has a different and unique sign language, the KETI sign language dataset can be the starting point for further research on the Korean sign language translation. Using the KETI sign language dataset, we develop a neural network model for translating sign videos into natural language sentences by utilizing the human keypoints extracted from the face, hands, and body parts. The obtained human keypoint vector is normalized by the mean and standard deviation of the keypoints and used as input to our translation model based on the sequence-to-sequence architecture. As a result, we show that our approach is robust even when the size of the training data is not sufficient. Our translation model achieved 93.28% (55.28%, respectively) translation accuracy on the validation set (test set, respectively) for 105 sentences that can be used in emergency situations. We compared several types of our neural sign translation models based on different attention mechanisms in terms of classical metrics for measuring the translation performance.

Crop Yield Prediction Using Deep Neural Networks.

Crop yield is a highly complex trait determined by multiple factors such as genotype, environment, and their interactions. Accurate yield prediction requires fundamental understanding of the functional relationship between yield and these interactive factors, and to reveal such relationship requires both comprehensive datasets and powerful algorithms. In the 2018 Syngenta Crop Challenge, Syngenta released several large datasets that recorded the genotype and yield performances of 2,267 maize hybrids planted in 2,247 locations between 2008 and 2016 and asked participants to predict the yield performance in 2017. As one of the winning teams, we designed a deep neural network (DNN) approach that took advantage of state-of-the-art modeling and solution techniques. Our model was found to have a superior prediction accuracy, with a root-mean-square-error (RMSE) being 12% of the average yield and 50% of the standard deviation for the validation dataset using predicted weather data. With perfect weather data, the RMSE would be reduced to 11% of the average yield and 46% of the standard deviation. We also performed feature selection based on the trained DNN model, which successfully decreased the dimension of the input space without significant drop in the prediction accuracy. Our computational results suggested that this model significantly outperformed other popular methods such as Lasso, shallow neural networks (SNN), and regression tree (RT). The results also revealed that environmental factors had a greater effect on the crop yield than genotype.

Hardware-Software Co-design to Accelerate Neural Network Applications

Many applications, such as machine learning and data sensing, are statistical in nature and can tolerate some level of inaccuracy in their computation. A variety of designs have been put forward exploiting the statistical nature of machine learning through approximate computing. With approximate multipliers being the main focus due to their high usage in machine-learning designs. In this article, we propose a novel approximate floating point multiplier, called CMUL, which significantly reduces energy and improves performance of multiplication while allowing for a controllable amount of error. Our design approximately models multiplication by replacing the most costly step of the operation with a lower energy alternative. To tune the level of approximation, CMUL dynamically identifies the inputs that produces the largest approximation error and processes them in precise mode. To use CMUL for deep neural network (DNN) acceleration, we propose a framework that modifies the trained DNN model to make it suitable for approximate hardware. Our framework adjusts the DNN weights to a set of “ potential weights ” that are suitable for approximate hardware. Then, it compensates the possible quality loss by iteratively retraining the network. Our evaluation with four DNN applications shows that, CMUL can achieve 60.3% energy efficiency improvement and 3.2× energy-delay product (EDP) improvement as compared to the baseline GPU, while ensuring less than 0.2% quality loss. These results are 38.7% and 2.0× higher than energy efficiency and EDP improvement of the CMUL without using the proposed framework.

EMI Radiation Prediction and Structure Optimization of Packages by Deep Learning

With a rapid increase in operating frequency and package complexity, conventional analysis methods cannot efficiently cope with current complex electromagnetic interference (EMI) issues. In this paper, a new method built on a deep neural network (DNN) model is proposed to accurately and rapidly predict the maximum radiated electric field at 3-meters of a wire-bond ball grid array (WB-BGA) package. The key hyper-parameters of the DNN model, such as the learning rate and type of optimizers, are discussed in depth so as to attain optimal performance. Predicted radiation results by the DNN model and the results of the full-wave simulation show good agreement. Once DNN is trained, the prediction time is in the order of milliseconds and the model size is in megabytes, which can acquire the predicted radiation quickly and accurately and save storage space. Furthermore, to prevent the radiation from exceeding requirements, package structures are optimized by adjusting those parameters sensitive to radiation and disregarding insensitive parameters. The sensitivity of these WB-BGA package structural parameters to EMI radiation can be analyzed quickly based on data with different deviations generated by the trained DNN model. The sensitive parameters are adjusted according to their correlation with EMI radiation. The effectiveness and feasibility of the optimization method are verified by the WB-BGA package with two sets of different structural parameters.

Deep Learning-Based Assessment of Cognitive Impairment in Elderly Using General Blood Test

Background: Development of a mass screening test for cognitive impairment is important for preventing onset of dementia. Based on the results of previous studies that cognitive impairment may be caused by systemic disorders, we developed a method of early detection using a deep neural network (DNN) trained on blood test data from health examinations. Methods: We studied a total of 304 subjects (age 72.1± 12.8 years); the cognitive function was assessed by the Mini Mental State Examination (MMSE), while systemic disorders were evaluated by a general blood test. First, we developed trained a DL-based algorithm DNN for predicting the subject's MMSE score based on the blood test data and age of subject in the algorism group in 202 patients with systemic disorders (73.5 ± 13.1years). The accuracy of the trained DL model DNN was first evaluated by leave-one-out cross validation Leave-one-out Cross Validation (LOO CV) in the Algorism group. Next, the trained DL model DNN was validated in the test a second group including 65 patients with systemic disorders (73.6 ± 11.0 year) and 37 healthy subjects (62.0 ± 8.6 years). Results: In the Training Set's LOO CV the leave-one out cross validation, the predicted MMSE scores exhibited a significant positive correlation with ground truth (the measured) -MMSE scores (p<0.001). The trained DNN model exhibited a both high prediction sensitivity and (90%) and specificity scores of (90% ) for two-class binary classification (23/24 cut off) (normal ≥ 24, cognitive impairment ≤23). In Finally, for the Out-of-Sample Set, upon which no training was performed, Test-group, the predicted MMSE scores exhibited a significant positive correlation with the ground truth measured MMSE scores (p<0.001). The DNN model exhibited a high prediction accuracy for the two-classbinary classification, with a sensitivity of 75% and a specificity of 87%. Interpretation: The DL-based algorithm DNN predicted cognitive impairment expressed by MMSE scores with high accuracy. Going forward, The DL-based such AI-based assessment of cognitive impairment may become a mass screening test of cognitive impairment, and contribute to prevention strategies for dementia. Funding: This work was supported in part by Grant-in-Aids from the MEXT (S1411017), JSPS Grant-in-Aid for Young Scientists (B, 16K16077). Declaration of Interest: There are no competing interests in this study. Ethical Approval: The ethical committee in the hospital approved this study.

Deep neural network based i-vector mapping for speaker verification using short utterances

Text-independent speaker recognition using short utterances is a highly challenging task due to the large variation and content mismatch between short utterances. I-vector and probabilistic linear discriminant analysis (PLDA) based systems have become the standard in speaker verification applications, but they are less effective with short utterances. In this paper, we first compare two state-of-the-art universal background model (UBM) training methods for i-vector modeling using full-length and short utterance evaluation tasks. The two methods are Gaussian mixture model (GMM) based (denoted I-vector_GMM) and deep neural network (DNN) based (denoted as I-vector_DNN) methods. The results indicate that the I-vector_DNN system outperforms the I-vector_GMM system under various durations (from full length to 5 s). However, the performances of both systems degrade significantly as the duration of the utterances decreases. To address this issue, we propose two novel nonlinear mapping methods which train DNN models to map the i-vectors extracted from short utterances to their corresponding long-utterance i-vectors. The mapped i-vector can restore missing information and reduce the variance of the original short-utterance i-vectors. The proposed methods both model the joint representation of short and long utterance i-vectors: the first method trains an autoencoder first using concatenated short and long utterance i-vectors and then uses the pre-trained weights to initialize a supervised regression model from the short to long version; the second method jointly trains the supervised regression model with an autoencoder reconstructing the short utterance i-vector itself. Experimental results using the NIST SRE 2010 dataset show that both methods provide significant improvement and result in a 24.51% relative improvement in Equal Error Rates (EERs) from a baseline system. In order to learn a better joint representation, we further investigate the effect of a deep encoder with residual blocks, and the results indicate that the residual network can further improve the EERs of a baseline system by up to 26.47%. Moreover, in order to improve the short i-vector mapping to its long version, an additional vector, which represents the average value of phoneme posteriors across frames, is also added to the input, and results in a 28.43% improvement. When further testing the best-validated models of SRE10 on the Speaker In The Wild (SITW) dataset, the methods result in a 23.12% improvement on arbitrary-duration (1–5 s) short-utterance conditions.

Human Splice-Site Prediction with Deep Neural Networks.

Accurate splice-site prediction is essential to delineate gene structures from sequence data. Several computational techniques have been applied to create a system to predict canonical splice sites. For classification tasks, deep neural networks (DNNs) have achieved record-breaking results and often outperformed other supervised learning techniques. In this study, a new method of splice-site prediction using DNNs was proposed. The proposed system receives an input sequence data and returns an answer as to whether it is splice site. The length of input is 140 nucleotides, with the consensus sequence (i.e., "GT" and "AG" for the donor and acceptor sites, respectively) in the middle. Each input sequence model is applied to the pretrained DNN model that determines the probability that an input is a splice site. The model consists of convolutional layers and bidirectional long short-term memory network layers. The pretraining and validation were conducted using the data set tested in previously reported methods. The performance evaluation results showed that the proposed method can outperform the previous methods. In addition, the pattern learned by the DNNs was visualized as position frequency matrices (PFMs). Some of PFMs were very similar to the consensus sequence. The trained DNN model and the brief source code for the prediction system are uploaded. Further improvement will be achieved following the further development of DNNs.

Identification of human circadian genes based on time course gene expression profiles by using a deep learning method

Circadian genes express periodically in an approximate 24-h period and the identification and study of these genes can provide deep understanding of the circadian control which plays significant roles in human health. Although many circadian gene identification algorithms have been developed, large numbers of false positives and low coverage are still major problems in this field. In this study we constructed a novel computational framework for circadian gene identification using deep neural networks (DNN) – a deep learning algorithm which can represent the raw form of data patterns without imposing assumptions on the expression distribution. Firstly, we transformed time-course gene expression data into categorical-state data to denote the changing trend of gene expression. Two distinct expression patterns emerged after clustering of the state data for circadian genes from our manually created learning dataset. DNN was then applied to discriminate the aperiodic genes and the two subtypes of periodic genes. In order to assess the performance of DNN, four commonly used machine learning methods including k-nearest neighbors, logistic regression, naïve Bayes, and support vector machines were used for comparison. The results show that the DNN model achieves the best balanced precision and recall. Next, we conducted large scale circadian gene detection using the trained DNN model for the remaining transcription profiles. Comparing with JTK_CYCLE and a study performed by Möller-Levet et al. (doi: https://doi.org/10.1073/pnas.1217154110), we identified 1132 novel periodic genes. Through the functional analysis of these novel circadian genes, we found that the GTPase superfamily exhibits distinct circadian expression patterns and may provide a molecular switch of circadian control of the functioning of the immune system in human blood. Our study provides novel insights into both the circadian gene identification field and the study of complex circadian-driven biological control. This article is part of a Special Issue entitled: Accelerating Precision Medicine through Genetic and Genomic Big Data Analysis edited by Yudong Cai & Tao Huang.

Intelligent fault diagnosis approach with unsupervised feature learning by stacked denoising autoencoder

Condition monitoring and fault diagnosis are important for maintaining the system performance and guaranteeing the operational safety. The traditional data-driven approaches mostly incorporate well-defined features and methodologies such as supervised artificial intelligence algorithms. Prior knowledge of possible features and a large quantity of labelled condition data are needed. Besides, many traditional approaches require rebuilding or a retraining of the original model to diagnosis new conditions. The present study proposes an intelligent fault diagnosis approach that uses a deep neural network (DNN) based on stacked denoising autoencoder. Representative features are learned by applying the denoising autoencoder to the unlabelled data in an unsupervised manner. A DNN is then constructed and fine-tuned with just a few items of labelled data. The trained DNN achieves high performance in fault classification. Furthermore, new conditions can be correctly classified by simply fine-tuning the trained DNN model using a small amount of labelled data under the new conditions. The effectiveness of the proposed approach is evaluated using a case study of fault diagnosis of a bearing unit. The results indicate that the proposed method can extract representative features from massive unlabelled data on the system condition and achieve high performance in fault diagnosis.

Packet Loss Concealment Based on Deep Neural Networks for Digital Speech Transmission

In this paper, we propose the regression-based packet loss concealment (PLC) for digital speech transmission by using deep neural networks (DNNs) with a multiple-layer deep architecture. For the DNN training, log-power spectra and phases are employed as features in the input layer for the large training set, which ensures non-linear mapping the frames from the last correctly received frame to the missing frame. Once the training is accomplished by the restricted Boltzmann machine (RBM)-based pre-training to initialize the DNN, minimum mean square error (MMSE)-based fine tuning is then performed based on the back-propagation algorithm. In the reconstruction stage, the trained DNN model is fed with the features of the previous frames in order to estimate the log-power spectra and phases of the missing frames. Reconstruction is further improved by using the cross-fading technique to mitigate discontinuity between the reconstruction signal and good frame signal in the time-domain. To demonstrate the performance of the proposed algorithm, hidden Markov model (HMM)-based PLC algorithm and the PLC algorithm standardized in adaptive multi-rate wideband (AMR-WB) Appendix I were used for comparison. The experimental results show that the proposed approach provides better speech quality and speech recognition accuracy than the conventional approaches.