Non-recurrent Neural Networks Research Articles

BiGRU-attention for Continuous blood pressure trends estimation through single channel PPG

BackgroundPhysiological parameter monitoring based on photoplethysmography (PPG) detection has the advantage of fast, portable, and non-invasive. Changes in the morphology of the PPG waveform can reflect the effect of arterial elasticity changes on blood pressure (BP). However, machine learning models and non-recurrent neural network models typically ignore the time-dependency of continuous PPG data, leading to the decrease of accuracy or the increased calibration frequency. ObjectiveThis paper proposes a BiGRU model with attention to estimate BP trends, which uses a single-channel PPG signal combined with demographic information to estimate continuous BP trends point-by-point and to discuss the impact of calibration cycle. MethodsThis paper selects 15 typical subjects from two groups with/without cardiovascular disease (CVD) and evaluates the model performance. Regarding the calibration frequency problem, we set two modes of non-calibration and calibration to validate the results of blood pressure trends estimation. ResultsIn the calibration mode, the estimation errors (ME ± STD) of SBP for CVD/non-CVD groups are 0.91 ± 10.58 mmHg/0.17 ± 10.06 mmHg respectively, and DBP are 0.34 ± 5.28 mmHg/-0.19 ± 5.36 mmHg; in the non-calibration mode, the estimation errors of SBP for CVD/non-CVD groups are 0.27 ± 9.87 mmHg/-0.82 ± 9.92 mmHg respectively, and DBP are −0.63 ± 3.28 mmHg/0.80 ± 4.93 mmHg. ConclusionsThe results show that the proposed model has high accuracy in estimating BP levels, which is expected to achieve real-time, long-term continuous BP trends monitoring in wearable devices.

Predicting academic performance grades of control student using a LSTM neural network

The evolution of artificial intelligent (AI) allows the development of new neural networks models associated with academic process such as: research, teachinglearning, and evaluation. Typically, in evaluation, the teacher verifies the correct use of the competency by the students. Deep Neural Networks has a significant impact to automate grading process for certain types of subjects. In skills learning process, especially in control and automation, the student must develop professional tasks. In this paper, a LSTM recurrent neural model is presented as an evaluation tool to classify the level of the skill in sufficient, prominent, or automated, using a temporal series associated with task performance. The model contains 5 layers, with SoftMax activation function and Adam optimizer. A non-recurrent Neural Network (NN) with 5 layers, RELU activation function and Adam optimizer, is developed to compare results. The results show an improvement in the LSTM model with 91.7% accuracy, 92.4% precision and 60% less requirement in the training and validation process. The results allow the proposed LSTM model to be a toolassistant for the evaluation of control and automation skills.

Estimating battery state of charge using recurrent and non-recurrent neural networks

Battery state of charge estimation is critical for determining the remaining range of electrified vehicles and the runtime of battery-powered equipment. Neural network algorithms which learn the relation between battery measurements and state of charge are a promising alternative to estimators based on models with adaptive filters. In this work, two types of neural networks are studied: recurrent networks, which have inherent memory of the past, and non-recurrent networks, which can effectively have memory added through exogenous filtered inputs. An extensive and comprehensive study is performed for these network types, with learnable parameters ranging from 20 to 3000. Network performance is compared for two different battery types, multiple temperatures, drive cycles, and training repetitions. Compared to a recurrent neural network, a non-recurrent feedforward neural network with filtered inputs is found to be up to 23% more accurate, require less training time (76% less using a CPU and 60% less using a GPU), and execute in about 1/3 the amount of time on an NXP S32K142 microprocessor.

Automatic Construction of General Non-Recurrent Neural Network Using Genetic Programming

An Artificial Neural Network (ANNs) is a model that has been studied for many years in the hope of achieving human like performance. It was used in many applications like (pattern recognition, signal processing, vision, speech recognition, and decisions making aids and robotics).There are many types of ANNs, most of these types are suffering from some problems such as Convergence, Design and some of these ANNs cannot solve the non-linear problems.Since Genetic Programming (GP) is a machine learning technique used in the automatic induction of computer programs, therefore we used it as automatic system for designing and implementing ANNs trying to overcome the previous problems.In this paper we proposed an approach to build ANNs based on evolutionary computation, which uses the GP to evolve both the architecture and weights of General non-recurrent NN simultaneously. New GP modification operations were proposed which are (structure-preserving crossover, structure-preserving mutation, and structure-preserving permutation), these modification operations have clear effects in improving the results of the proposed system.The proposed system has been used in solving five simple problems which are (AND, OR, NOT, Exclusive-OR, and Half-Adder) problems. In addition, we compare the results of the proposed system with Koza work and with three major types of supervised ANNs, which are (Perceptron, Back-Propagation, and Adaline).

Prediction of Real-World Slope Movements via Recurrent and Non-recurrent Neural Network Algorithms: A Case Study of the Tangni Landslide

Prediction of Real-World Slope Movements via Recurrent and Non-recurrent Neural Network Algorithms: A Case Study of the Tangni Landslide

Analyzing the Short-Term Dependency in Ultra-High Magnetic Response Systems - Modeling Sequential Data with Non-Recurrent Neural Networks

Recurrent neural network (RNN) is a popular modeling choice for sequential data. However, empirical experience shows RNNs are often difficult and time-consuming to tune and customize. It drives practitioners to replace RNNs with non-recurrent neural networks which presented comparable performances in some cases. The success of using non-recurrent neural networks to model sequential data indicates the short-term dependency among sequential data. In this paper, we systematically analyze the short-term dependency in ultra-high magnetic response systems (UHMR) with partial system knowledge based on the observability criterion of the dynamic systems. Moreover, we show that the sequential data in the UHMR system only have 2-step dependency. This result indicates that any consecutive three steps in an experiment form a datum to train a feed-forward neural network (FFNN). Therefore, sufficient data can be collected within a small number of experiments. Based on the analysis, we train a feed-forward neural network to model the UHMR system based on the sequential data from four experiments. Through proper data pre-processing, the FFNN model can predict the system performance with bounded mean absolute error.

Structured Semantic Knowledge Can Emerge Automatically from Predicting Word Sequences in Child-Directed Speech.

Previous research has suggested that distributional learning mechanisms may contribute to the acquisition of semantic knowledge. However, distributional learning mechanisms, statistical learning, and contemporary “deep learning” approaches have been criticized for being incapable of learning the kind of abstract and structured knowledge that many think is required for acquisition of semantic knowledge. In this paper, we show that recurrent neural networks, trained on noisy naturalistic speech to children, do in fact learn what appears to be abstract and structured knowledge. We trained two types of recurrent neural networks (Simple Recurrent Network, and Long Short-Term Memory) to predict word sequences in a 5-million-word corpus of speech directed to children ages 0–3 years old, and assessed what semantic knowledge they acquired. We found that learned internal representations are encoding various abstract grammatical and semantic features that are useful for predicting word sequences. Assessing the organization of semantic knowledge in terms of the similarity structure, we found evidence of emergent categorical and hierarchical structure in both models. We found that the Long Short-term Memory (LSTM) and SRN are both learning very similar kinds of representations, but the LSTM achieved higher levels of performance on a quantitative evaluation. We also trained a non-recurrent neural network, Skip-gram, on the same input to compare our results to the state-of-the-art in machine learning. We found that Skip-gram achieves relatively similar performance to the LSTM, but is representing words more in terms of thematic compared to taxonomic relations, and we provide reasons why this might be the case. Our findings show that a learning system that derives abstract, distributed representations for the purpose of predicting sequential dependencies in naturalistic language may provide insight into emergence of many properties of the developing semantic system.

Investigating very deep highway networks for parametric speech synthesis

Deep neural networks are powerful tools for classification and regression tasks. While a network with more than 100 hidden layers has been reported for image classification, how such a non-recurrent neural network with more than 10 hidden layers will perform for speech synthesis is as yet unknown. This work investigates the performance of deep networks on statistical parametric speech synthesis, particularly the question of whether different acoustic features can be better generated by a deeper network. To answer this question, this work examines a multi-stream highway network that separately generates spectral and F0 acoustic features based on the highway architecture. Experiments on the Blizzard Challenge 2011 corpus show that the accuracy of the generated spectral features consistently improves as the depth of the network increases from 2 to 40, but the F0 trajectory can be generated equally well by either a deep or a shallow network. Additional experiments on a single-stream highway and normal feedforward network, both of which generate spectral and F0 features from a single network, show that these networks must be deep enough to generate both kinds of acoustic features well. The difference in the performance of multi- and single-stream highway networks is further analyzed on the basis of the networks’ activation and sensitivity to input features. In general, the highway network with more than 10 hidden layers, either multi- or single-stream, performs better on the experimental corpus than does a shallow network.

Emphasizing Essential Words for Sentiment Classification Based on Recurrent Neural Networks

With the explosion of online communication and publication, texts become obtainable via forums, chat messages, blogs, book reviews and movie reviews. Usually, these texts are much short and noisy without sufficient statistical signals and enough information for a good semantic analysis. Traditional natural language processing methods such as Bow-of-Word (BOW) based probabilistic latent semantic models fail to achieve high performance due to the short text environment. Recent researches have focused on the correlations between words, i.e., term dependencies, which could be helpful for mining latent semantics hidden in short texts and help people to understand them. Long short-term memory (LSTM) network can capture term dependencies and is able to remember the information for long periods of time. LSTM has been widely used and has obtained promising results in variants of problems of understanding latent semantics of texts. At the same time, by analyzing the texts, we find that a number of keywords contribute greatly to the semantics of the texts. In this paper, we establish a keyword vocabulary and propose an LSTM-based model that is sensitive to the words in the vocabulary; hence, the keywords leverage the semantics of the full document. The proposed model is evaluated in a short-text sentiment analysis task on two datasets: IMDB and SemEval-2016, respectively. Experimental results demonstrate that our model outperforms the baseline LSTM by 1%~2% in terms of accuracy and is effective with significant performance enhancement over several non-recurrent neural network latent semantic models (especially in dealing with short texts). We also incorporate the idea into a variant of LSTM named the gated recurrent unit (GRU) model and achieve good performance, which proves that our method is general enough to improve different deep learning models.

Trajectory generation and modulation using dynamic neural networks

Generation of desired trajectory behavior using neural networks involves a particularly challenging spatio-temporal learning problem. This paper introduces a novel solution, i.e., designing a dynamic system whose terminal behavior emulates a prespecified spatio-temporal pattern independently of its initial conditions. The proposed solution uses a dynamic neural network (DNN), a hybrid architecture that employs a recurrent neural network (RNN) in cascade with a nonrecurrent neural network (NRNN). The RNN generates a simple limit cycle, which the NRNN reshapes into the desired trajectory. This architecture is simple to train. A systematic synthesis procedure based on the design of relay control systems is developed for configuring an RNN that can produce a limit cycle of elementary complexity. It is further shown that a cascade arrangement of this RNN and an appropriately trained NRNN can emulate any desired trajectory behavior irrespective of its complexity. An interesting solution to the trajectory modulation problem, i.e., online modulation of the generated trajectories using external inputs, is also presented. Results of several experiments are included to demonstrate the capabilities and performance of the DNN in handling trajectory generation and modulation problems.

Tool Wear Monitoring in Turning: A Neural Network Application

Tool wear monitoring is the most difficult task in the area of tool condition monitoring for metal-cutting manufacturing processes. The main objective is to improve the process reliability, but the production costs need to be reduced as well. This article summarises a new approach for online and indirect tool wear estimation or classification in turning using neural networks. This technique uses a physical process model describing the influence of cutting conditions (such as the feed rate) on measured process parameters (here: cutting force signals) in order to separate signal changes caused by variable cutting conditions from signal changes caused by tool wear. Features extracted from the normalised process parameters are taken as inputs of a dynamic, but nonrecurrent neural network that estimates the current state of the tool. It is shown that the estimation error can be reduced significantly with this combination of a hard computing and a soft computing technique. The article represents an extended summary of the author's investigations and publications in the area of online and indirect tool wear monitoring in turning by means of artificial neural networks.