Deep Feedforward Neural Network Research Articles

Deep Neural Network Based Linearization and Cold Junction Compensation of Thermocouple

In this proposed work, temperature sensors, namely, a thermocouple and thermistor were linearized using deep neural networks. The deep feedforward neural network (DFNN) technique was proposed to linearize the K-type thermocouple’s output, in the given temperature range −100 °C to 1372 °C, while nonlinearity was reduced from 2.03% to 0.002% full scale span (FSS). Deep layer recurrent neural network (DLR-NN) was used to reduce the nonlinearity of a negative temperature coefficient (NTC) thermistor from 84.63% to 0.13% FSS. The linearized thermistor was used for cold junction compensation (CJC) of the thermocouple. In both the thermocouple and thermistor, linearization was achieved in a single stage for a wide range digitally using deep neural networks alone. There were no analog pre-signal conditioning circuits, unlike the existing neural network-based linearization techniques in literature. A hardware setup of a stand-alone module for linearization was designed using the Raspberry pi microcontroller consisting of two soft modules, one for thermocouple linearization and the other for thermistor linearization. The proposed system was experimentally tested using a K-type thermocouple on a thermal calibrator in a 0 °C–300 °C range. The cold junction compensated output of the thermocouple had a maximum absolute error of 0.34 °C when ambient temperature varied from 0 °C to 40 °C. The results were satisfactory and better than the existing National Institute of Standards and Technology (NIST) standard. This linearization technique can be extended to other thermocouple types as well as other nonlinear sensors.

Preliminary experimental results of a stacked 3D cyclic chaotic neural network reservoir integrated circuit

A cyclic chaotic neural network reservoir (CNNR) circuit is implemented on a stacked three-dimensional (3D) cyclic neural network integrated circuit (IC) system, which was originally designed and fabricated for a deep feedforward neural network. We create a circuit emulator for a neuron chip used in the 3D integrated system according to the measurement results obtained from the chip. We preliminary evaluate the performance of the cyclic CNNR circuit for a waveform classification task using the emulator.

Comparative Features of MIA GMDH and Deep Feed-Forward Neural Networks

Comparative Features of MIA GMDH and Deep Feed-Forward Neural Networks

Recurrent processing improves occluded object recognition and gives rise to perceptual hysteresis.

Over the past decades, object recognition has been predominantly studied and modelled as a feedforward process. This notion was supported by the fast response times in psychophysical and neurophysiological experiments and the recent success of deep feedforward neural networks for object recognition. Recently, however, this prevalent view has shifted and recurrent connectivity in the brain is now believed to contribute significantly to object recognition — especially under challenging conditions, including the recognition of partially occluded objects. Moreover, recurrent dynamics might be the key to understanding perceptual phenomena such as perceptual hysteresis. In this work we investigate if and how artificial neural networks can benefit from recurrent connections. We systematically compare architectures comprised of bottom-up, lateral, and top-down connections. To evaluate the impact of recurrent connections for occluded object recognition, we introduce three stereoscopic occluded object datasets, which span the range from classifying partially occluded hand-written digits to recognizing three-dimensional objects. We find that recurrent architectures perform significantly better than parameter-matched feedforward models. An analysis of the hidden representation of the models suggests that occluders are progressively discounted in later time steps of processing. We demonstrate that feedback can correct the initial misclassifications over time and that the recurrent dynamics lead to perceptual hysteresis. Overall, our results emphasize the importance of recurrent feedback for object recognition in difficult situations.

Deep learning systems for automatic diagnosis of infant cry signals

Nowadays, deep learning architectures are promising artificial intelligence systems in various applications of biomedical engineering. For instance, they can be combined with signal processing techniques to build computer-aided diagnosis systems used to help physician making appropriate decision related to the diagnosis task. The goal of the current study is to design and validate various deep learning systems to improve diagnosis of infant cry records. Specifically, deep feedforward neural networks (DFFNN), long short-term memory (LSTM) neural networks, and convolutional neural networks (CNN) are designed, implemented and trained with cepstrum analysis-based coefficients as inputs to distinguish between healthy and unhealthy infant cry records. All deep learning systems are validated on expiration and inspiration sets separately. The number of convolutional layers and number of neurons in hidden layers are respectively varied in CNN and DFFNN. It is found that CNN achieved the highest accuracy and sensitivity, followed by DFFNN. The latter, obtained the highest specificity. Compared to similar work in the literature, it is concluded that deep learning systems trained with cepstrum analysis-based coefficients are powerful machines that can be employed for accurate diagnosis of infant cry records so as to distinguish between healthy and pathological signals.

Multi-Objective Optimization of Laminated Functionally Graded Carbon Nanotube-Reinforced Composite Plates Using Deep Feedforward Neural Networks-NSGAII Algorithm

This paper proposes a novel and efficient DNN-NSGAII approach which is an integration of the deep feedforward neural network (DNN) and the nondominated sorting genetic algorithm II (NSGAII) to solve multi-objective optimization (MOO) problems of laminated functionally graded carbon nanotube-reinforced composite (FG-CNTRC) quadrilateral plates. The core idea of the proposed approach is to use the DNN as an analyzer to evaluate the objective and constraint functions instead of using the time-consuming finite element analysis methods (FEAMs) during the optimization process, while the NSGAII is employed to find a set of Pareto-optimal solutions of the MOO problems. Accordingly, the proposed DNN-NSGAII remarkably saves the computational cost, but still provides a high accurate optimal solution. The precision, efficiency, and capability of the proposed method are demonstrated through two different MOO problems of the FG-CNTRC quadrilateral plates. Obtained results of the DNN-NSGAII are compared with those of other methods to investigate the reliability and efficiency of proposed method. Moreover, the effects of various boundary conditions and carbon nanotube (CNT) distributions on the Pareto-optimal solutions of MOO problems are also examined.

Using Artificial Neural Network Condensation to Facilitate Adaptation of Machine Learning in Medical Settings by Reducing Computational Burden: Model Design and Evaluation Study.

BackgroundMachine learning applications in the health care domain can have a great impact on people’s lives. At the same time, medical data is usually big, requiring a significant number of computational resources. Although this might not be a problem for the wide adoption of machine learning tools in high-income countries, the availability of computational resources can be limited in low-income countries and on mobile devices. This can limit many people from benefiting from the advancement in machine learning applications in the field of health care.ObjectiveIn this study, we explore three methods to increase the computational efficiency and reduce model sizes of either recurrent neural networks (RNNs) or feedforward deep neural networks (DNNs) without compromising their accuracy.MethodsWe used inpatient mortality prediction as our case analysis upon review of an intensive care unit dataset. We reduced the size of RNN and DNN by applying pruning of “unused” neurons. Additionally, we modified the RNN structure by adding a hidden layer to the RNN cell but reducing the total number of recurrent layers to accomplish a reduction of the total parameters used in the network. Finally, we implemented quantization on DNN by forcing the weights to be 8 bits instead of 32 bits.ResultsWe found that all methods increased implementation efficiency, including training speed, memory size, and inference speed, without reducing the accuracy of mortality prediction.ConclusionsOur findings suggest that neural network condensation allows for the implementation of sophisticated neural network algorithms on devices with lower computational resources.

Deep Learning in Water Resources Management: Τhe Case Study of Kastoria Lake in Greece

The effects of climate change on water resources management have drawn worldwide attention. Water quality predictions that are both reliable and precise are critical for an effective water resources management. Although nonlinear biological and chemical processes occurring in a lake make prediction complex, advanced techniques are needed to develop reliable models and effective management systems. Artificial intelligence (AI) is one of the most recent methods for modeling complex structures. The applications of machine learning (ML), as a part of AI, in hydrology and water resources management have been increasing in recent years. In this paper, the ability of deep neural networks (DNNs) to predict the quality parameter of dissolved oxygen (DO), in Lake Kastoria, Greece, is tested. The available dataset from 11 November 2015, to 15 March 2018, on an hourly basis, from four telemetric stations located in the study area consists of (1) Chl-a (μg/L), (2) pH, (3) temperature—Tw (°C), (4) conductivity (μS/cm), (5) turbidity (NTU), (6) ammonia (NH4, mg/L), (7) nitrate nitrogen (N–NO3, mg/L), and (8) dissolved oxygen (DO) (mg/L). Feed-forward deep neural networks (FF-DNNs) of DO, with different structures, are tested for all stations. All the well-trained DNNs give satisfactory results. The optimal selected FF-DNNs of DO for each station with a high efficiency (NSE > 0.89 for optimal selected structures/station) constitute a good choice for modeling dissolved oxygen. Moreover, they provide information in real time and comprise a powerful decision support system (DSS) for preventing accidental and emergency conditions that may arise from both natural and anthropogenic hazards.

Movie Genre Classification from RGB Movie Poster Image Using Deep Feed-forward Network

A movie poster image is one of the important media in the filmmaking process, providing valuable information about the movie, such as movie titles, characters, and genres. Identifying a movie genre from a poster can be a daunting task, as it can relate to multiple genres. To solve this problem, this paper uses a deep feedforward neural network to classify movie genres from movie poster images. In this regard, we used and trained a state-of-the-art InceptionV3 deep neural network. The network is trained on our dataset consisting of 36,423 movie poster images taken from the IMDB website, which is categorized into 28 genres. The model predicts the top three classes with the highest probability of a particular movie poster.

A deep feed-forward neural network for damage detection in functionally graded carbon nanotube-reinforced composite plates using modal kinetic energy

A deep feed-forward neural network for damage detection in functionally graded carbon nanotube-reinforced composite plates using modal kinetic energy

A Quantitative Enhancement Mechanism of University Students’ Employability and Entrepreneurship Based on Deep Learning in the Context of the Digital Era

This paper adopts a deep learning approach to analyze and study the mechanism of quantitative enhancement of college students’ employment and entrepreneurial abilities in the context of the digital era. The deep learning connotation is predetermined as five abilities, which are metacognitive ability, active communication and cooperation ability, deep processing ability, creative practice ability, and learning empathy experience, and, based on this, the deep learning questionnaire is designed, and it is reclassified by exploratory factor analysis to reduce the dimensionality, and the specific indicators and scientific connotation dimensions of the deep learning questionnaire are determined; and, through the deep learning of each dimension, the problems of deep learning of college students are examined and in-depth analysis is conducted, and the inner relationship and correlation among the dimensions of deep learning of college students are derived through correlation analysis. The success of innovation and entrepreneurship depends on the innovation and entrepreneurial ability of college students, and the formation of the ability influenced various factors. Therefore, not only is studying the influencing factors of college students’ innovation and entrepreneurship ability in line with the requirements of the times and social development, but also it can solve real problems. This thesis adopts a combination of two methods, qualitative research and quantitative research, to study the influencing factors of college students’ innovation and entrepreneurship ability and tries to ensure the scientificity, accuracy, and comprehensiveness of the conclusion. In this paper, we analyzed the requirements of the employment prediction system for graduating secondary school students, carried out the software framework and database design of the employment analysis and prediction system for secondary school students, and designed the system modularly based on the analysis results. By applying the proposed deep feedforward neural network prediction model to the prediction system, a software system applicable to the employment prediction and guidance of secondary school students is implemented.

RBPSpot: Learning on appropriate contextual information for RBP binding sites discovery

SummaryIdentifying the factors determining the RBP-RNA interactions remains a big challenge. It involves sparse binding motifs and a suitable sequence context for binding. The present work describes an approach to detect RBP binding sites in RNAs using an ultra-fast inexact k-mers search for statistically significant seeds. The seeds work as an anchor to evaluate the context and binding potential using flanking region information while leveraging from Deep Feed-forward Neural Network. The developed models also received support from MD-simulation studies. The implemented software, RBPSpot, scored consistently high for all the performance metrics including average accuracy of ∼90% across a large number of validated datasets. It outperformed the compared tools, including some with much complex deep-learning models, during a comprehensive benchmarking process. RBPSpot can identify RBP binding sites in the human system and can also be used to develop new models, making it a valuable resource in the area of regulatory system studies.

A deep-learning-based prediction method of the estimated ultimate recovery (EUR) of shale gas wells

The estimated ultimate recovery (EUR) of shale gas wells is influenced by many factors, and the accurate prediction still faces certain challenges. As an artificial intelligence algorithm, deep learning yields notable advantages in nonlinear regression. Therefore, it is feasible to predict the EUR of shale gas wells based on a deep-learning algorithm. In this paper, according to geological evaluation data, hydraulic fracturing data, production data and EUR evaluation results of 282 wells in the WY shale gas field, a deep-learning-based algorithm for EUR evaluation of shale gas wells was designed and realized. First, the existing EUR evaluation methods of shale gas wells and the deep feedforward neural network algorithm was systematically analyzed. Second, the technical process of a deep-learning-based algorithm for EUR prediction of shale gas wells was designed. Finally, by means of real data obtained from the WY shale gas field, several different cases were applied to testify the validity and accuracy of the proposed approach. The results show that the EUR prediction with high accuracy. In addition, the results are affected by the variety and number of input parameters, the network structure and hyperparameters. The proposed approach can be extended to other shale fields using the similar technic process.

Investigating deep feedforward neural networks for classification of transposon-derived piRNAs

PIWI-interacting RNAs (piRNAS) form an important class of non-coding RNAs that play a key role in gene expression regulation and genome integrity by silencing transposable elements. However, despite the importance of piRNAs and the large application of deep learning in computational biology, there are few studies of deep learning for piRNAs prediction. Still, current methods focus on using advanced architectures like CNN and variations. This paper presents an investigation on deep feedforward network models for classification of human transposon-derived piRNAs. We developed a lightweight predictor (when compared to other deep learning methods) and we show by practical evidence that simple neural networks can perform as well as better than complex neural networks when using the appropriate hyperparameters. For that, we train, analyze and compare the results of a multilayer perceptron with different hyperparameter choices, such as numbers of hidden layers, activation functions and optimizers, clarifying the advantages and disadvantages of each choice. Our proposed predictor reached a F-score of 0.872, outperforming other state-of-the-art methods for human transposon-derived piRNAs classification. In addition, to better access the generalization of our proposal, we also showed it achieved competitive results when classifying piRNAs of other species.

Analytical derivatives of neural networks

We propose a simple recursive algorithm that allows the computation of the first- and second-order derivatives with respect to the inputs of an arbitrary deep feed forward neural network (DFNN). The algorithm naturally incorporates the derivatives with respect to the network parameters. To test the algorithm, we apply it to the study of the quantum mechanical variational problem for few cases of simple potentials, modeling the ground-state wave function in terms of a DFNN.

Deep feed forward neural network-based screening system for diabetic retinopathy severity classification using the lion optimization algorithm.

Diabetic Retinopathy (DR) has become a major cause of blindness in recent years. Diabetic patients should be screened on a regular basis for early detection, which can help them avoid blindness. Furthermore, the number of diabetic patients undergoing these screening procedures is rapidly increasing, resulting in increased workload for ophthalmologists. An efficient screening system that assists ophthalmologists in DR diagnosis saves ophthalmologists a lot of time and effort. To address this issue, an automatic DR detection screening system is required to improve diagnosis speed and detection accuracy. Appropriate treatment can be provided to patients to prevent vision loss if the severity levels of DR are accurately diagnosed in the early stages. A growing number of screening systems for DR diagnosis have been developed in recent years using various deep learning models, and the majority of the published work did not include any optimization algorithm in the neural network for severity classification. The use of an optimization algorithm with the necessary hyper parameter tuning will improve the model's performance. Considering this as motivation, we proposed a five-phase DFNN-LOA model. The DFNN-LOA algorithm presented here has five phases: (i) pre-processing, (ii) optic disc detection, (iii) segmentation, (iv) feature extraction, and (v) severity classification. The proposed model's experimental analysis is carried out on the MESSIDOR dataset. The experimental results show that the proposed DFNN-LOA model has superior characteristics, with maximum accuracy, sensitivity, specificity, F1-score, PPV, and NPV of 97.6%, 98.4%, 90.7%, 96.5%, 94.6%, and 97.1%, respectively.

Deep Tobit networks: A novel machine learning approach to microeconometrics

Tobit models (also called as “censored regression models” or classified as “sample selection models” in microeconometrics) have been widely applied to microeconometric problems with censored outcomes. However, due to their linear parametric settings and restrictive normality assumptions, the traditional Tobit models fail to capture the pervading nonlinearities and thus may be inadequate for microeconometric analysis with large-scale datasets. This paper proposes two novel deep neural networks for Tobit problems and explores machine learning approaches in the context of microeconometric modeling. We connect the censored outputs in Tobit models with some deep learning techniques, which are thought to be unrelated to microeconometrics, and use the rectified linear unit activation and a particularly designed network structure to implement the censored output mechanisms and realize the underlying econometric conceptions. The benchmark Tobit-I and Tobit-II models are then reformulated as two carefully designed deep feedforward neural networks named deep Tobit-I network and deep Tobit-II network, respectively. A novel significance testing method is developed based on the proposed networks. Compared with the traditional models, our networks with deep structures can effectively describe the underlying highly nonlinear relationships and achieve considerable improvements in fitting and prediction. With the novel testing method, the proposed networks enable highly accurate and sophisticated econometric analysis with minimal random assumptions. The encouraging numerical experiments on synthetic and realistic datasets demonstrate the utility and advantages of the proposed method.

Deep Learning for Classifying Physical Activities from Accelerometer Data.

Physical inactivity increases the risk of many adverse health conditions, including the world’s major non-communicable diseases, such as coronary heart disease, type 2 diabetes, and breast and colon cancers, shortening life expectancy. There are minimal medical care and personal trainers’ methods to monitor a patient’s actual physical activity types. To improve activity monitoring, we propose an artificial-intelligence-based approach to classify physical movement activity patterns. In more detail, we employ two deep learning (DL) methods, namely a deep feed-forward neural network (DNN) and a deep recurrent neural network (RNN) for this purpose. We evaluate the two models on two physical movement datasets collected from several volunteers who carried tri-axial accelerometer sensors. The first dataset is from the UCI machine learning repository, which contains 14 different activities-of-daily-life (ADL) and is collected from 16 volunteers who carried a single wrist-worn tri-axial accelerometer. The second dataset includes ten other ADLs and is gathered from eight volunteers who placed the sensors on their hips. Our experiment results show that the RNN model provides accurate performance compared to the state-of-the-art methods in classifying the fundamental movement patterns with an overall accuracy of 84.89% and an overall F1-score of 82.56%. The results indicate that our method provides the medical doctors and trainers a promising way to track and understand a patient’s physical activities precisely for better treatment.

A Survey on Sentiment Analysis and Opinion Mining in Greek Social Media

As the amount of content that is created on social media is constantly increasing, more and more opinions and sentiments are expressed by people in various subjects. In this respect, sentiment analysis and opinion mining techniques can be valuable for the automatic analysis of huge textual corpora (comments, reviews, tweets etc.). Despite the advances in text mining algorithms, deep learning techniques, and text representation models, the results in such tasks are very good for only a few high-density languages (e.g., English) that possess large training corpora and rich linguistic resources; nevertheless, there is still room for improvement for the other lower-density languages as well. In this direction, the current work employs various language models for representing social media texts and text classifiers in the Greek language, for detecting the polarity of opinions expressed on social media. The experimental results on a related dataset collected by the authors of the current work are promising, since various classifiers based on the language models (naive bayesian, random forests, support vector machines, logistic regression, deep feed-forward neural networks) outperform those of word or sentence-based embeddings (word2vec, GloVe), achieving a classification accuracy of more than 80%. Additionally, a new language model for Greek social media has also been trained on the aforementioned dataset, proving that language models based on domain specific corpora can improve the performance of generic language models by a margin of 2%. Finally, the resulting models are made freely available to the research community.

A Deep Neural Network-Based Multi-Frequency Path Loss Prediction Model from 0.8 GHz to 70 GHz.

Large-scale fading models play an important role in estimating radio coverage, optimizing base station deployments and characterizing the radio environment to quantify the performance of wireless networks. In recent times, multi-frequency path loss models are attracting much interest due to their expected support for both sub-6 GHz and higher frequency bands in future wireless networks. Traditionally, linear multi-frequency path loss models like the ABG model have been considered, however such models lack accuracy. The path loss model based on a deep learning approach is an alternative method to traditional linear path loss models to overcome the time-consuming path loss parameters predictions based on the large dataset at new frequencies and new scenarios. In this paper, we proposed a feed-forward deep neural network (DNN) model to predict path loss of 13 different frequencies from 0.8 GHz to 70 GHz simultaneously in an urban and suburban environment in a non-line-of-sight (NLOS) scenario. We investigated a broad range of possible values for hyperparameters to search for the best set of ones to obtain the optimal architecture of the proposed DNN model. The results show that the proposed DNN-based path loss model improved mean square error (MSE) by about 6 dB and achieved higher prediction accuracy R2 compared to the multi-frequency ABG path loss model. The paper applies the XGBoost algorithm to evaluate the importance of the features for the proposed model and the related impact on the path loss prediction. In addition, the effect of hyperparameters, including activation function, number of hidden neurons in each layer, optimization algorithm, regularization factor, batch size, learning rate, and momentum, on the performance of the proposed model in terms of prediction error and prediction accuracy are also investigated.