Recurrent Artificial Neural Network Research Articles

Optimal deep recurrent neural network for sentiment grade classification

Purpose Internet has endorsed a tremendous change with the advancement of the new technologies. The change has made the users of the internet to make comments regarding the service or product. The Sentiment classification is the process of analyzing the reviews for helping the user to decide whether to purchase the product or not. Design/methodology/approach A rider feedback artificial tree optimization-enabled deep recurrent neural networks (RFATO-enabled deep RNN) is developed for the effective classification of sentiments into various grades. The proposed RFATO algorithm is modeled by integrating the feedback artificial tree (FAT) algorithm in the rider optimization algorithm (ROA), which is used for training the deep RNN classifier for the classification of sentiments in the review data. The pre-processing is performed by the stemming and the stop word removal process for removing the redundancy for smoother processing of the data. The features including the sentiwordnet-based features, a variant of term frequency-inverse document frequency (TF-IDF) features and spam words-based features are extracted from the review data to form the feature vector. Feature fusion is performed based on the entropy of the features that are extracted. The metrics employed for the evaluation in the proposed RFATO algorithm are accuracy, sensitivity, and specificity. Findings By using the proposed RFATO algorithm, the evaluation metrics such as accuracy, sensitivity and specificity are maximized when compared to the existing algorithms. Originality/value The proposed RFATO algorithm is modeled by integrating the FAT algorithm in the ROA, which is used for training the deep RNN classifier for the classification of sentiments in the review data. The pre-processing is performed by the stemming and the stop word removal process for removing the redundancy for smoother processing of the data. The features including the sentiwordnet-based features, a variant of TF-IDF features and spam words-based features are extracted from the review data to form the feature vector. Feature fusion is performed based on the entropy of the features that are extracted.

Hill Climb Modular Assembler Encoding: Evolving Modular Neural Networks of fixed modular architecture

The paper presents a novel generative Neuro-Evolutionary (NE) method called Hill Climb Modular Assembler Encoding (HCMAE). The target application of the HCMAE is to evolve modular Artificial Neural Networks (ANNs) whose modular structure is known in advance.Different variants of HCMAE were tested on two well-known ANN benchmarks, i.e. the Two-Spiral problem (feed-forward ANNs), and the Inverted-Pendulum problem (recurrent ANNs), for four different modular neural architectures. Particle Swarm Optimization and Differential Evolution were selected as rivals for HCMAE. Both rival methods were tested in two variants, i.e. a classical one-population variant and cooperative co-evolutionary multi-population variant. The paper presents the proposed method and reports all the experiments.

Neuromorphic metamaterial structures

Computerized structural optimization methods are often used to design the shape of structures to achieve a desired function, such as a specific compliance. In the simplest case of a structure built from a material with small deformations obeying Cauchy elasticity, the compliance is constant and the relationship between the forces applied on the structure and its deformation is linear. This linearity severely limits the types of functions which can be achieved by such materials. Here we study mechanical metamaterials made of repeating unit cells, each with specific dimensional parameters and with one-sided contact non-linearities. We show that the force and displacement equilibrium configurations of these metamaterials are mathematically equivalent to the fixed points of a recurrent artificial neural network. By exploiting this equivalence, we demonstrate mechanical metamaterials that can be designed (trained) to implement complex non-linear functions, using a gradient descent algorithm as in artificial neural networks. One of our metamaterial structures has a higher compliance when it is pressed against a pattern of raised bumps corresponding to the vowels in the Braille alphabet, than when it is pressed against patterns for six consonants. As artificial neural networks are known to be efficient models for numerous problems in machine learning, our results reveal that beneficial features of neural networks can be transferred to physical objects (mechanical structures). These features include the design of systems with complex input–output relationships by using generic methods that only rely on the repetitive processing of pairs of inputs and desired outputs, as well as remarkable generalization capabilities. We anticipate our methodology to be a starting point for the transfer of some of the breakthroughs in the rapidly advancing field of machine learning to highly functional physical devices in applications that are constrained by energy, volume, data processing or response time.

Primate neuronal connections are sparse in cortex as compared to mouse.

Detailing how primate and mouse neurons differ is critical for creating generalized models of how neurons process information. We reconstruct 15,748 synapses in adult Rhesus macaques and mice and ask how connectivity differs on identified cell types in layer 2/3 of primary visual cortex. Primate excitatory and inhibitory neurons receive 2-5 times fewer excitatory and inhibitory synapses than similar mouse neurons. Primate excitatory neurons have lower excitatory-to-inhibitory (E/I) ratios than mouse but similar E/I ratios in inhibitory neurons. In both species, properties of inhibitory axons such as synapse size and frequency are unchanged, and inhibitory innervation of excitatory neurons is local and specific. Using artificial recurrent neural networks (RNNs) optimized for different cognitive tasks, we find that penalizing networks for creating and maintaining synapses, as opposed to neuronal firing, reduces the number of connections per node as the number of nodes increases, similar to primate neurons compared with mice.

Extreme events in globally coupled chaotic maps

Understanding and predicting uncertain things are the central themes of scientific evolution. Human beings revolve around these fears of uncertainties concerning various aspects like a global pandemic, health, finances, to name but a few. Dealing with this unavoidable part of life is far tougher due to the chaotic nature of these unpredictable activities. In the present article, we consider a global network of identical chaotic maps, which splits into two different clusters, despite the interaction between all nodes are uniform. The stability analysis of the spatially homogeneous chaotic solutions provides a critical coupling strength, before which we anticipate such partial synchronization. The distance between these two chaotic synchronized populations often deviates more than eight times of standard deviation from its long-term average. The probability density function of these highly deviated values fits well with the generalized extreme value distribution. Meanwhile, the distribution of recurrence time intervals between extreme events resembles the Weibull distribution. The existing literature helps us to characterize such events as extreme events using the significant height. These extremely high fluctuations are less frequent in terms of their occurrence. We determine numerically a range of coupling strength for these extremely large but recurrent events. On-off intermittency is the responsible mechanism underlying the formation of such extreme events. Besides understanding the generation of such extreme events and their statistical signature, we furnish forecasting these events using the powerful deep learning algorithms of an artificial recurrent neural network. This long short-term memory (LSTM) can offer handy one-step forecasting of these chaotic intermittent bursts. We also ensure the robustness of this forecasting model with two hundred hidden cells in each LSTM layer.

Model of the Austenite Decomposition during Cooling of the Medium Carbon Steel Using LSTM Recurrent Neural Network.

The motivation of the presented paper is the desire to create a universal tool to analyse the process of austenite decomposition during the cooling process of various steel grades. The presented analysis concerns the application of Recurrent Artificial Neural Networks (RANN) of the Long Short-Term Memory (LSTM) type for the analysis of the transition path of the cooling curve. This type of network was selected due to its ability to predict events in time sequences. The proposed generalisation allows for the determination of the austenite transformation during the continuous cooling process for various cooling curves. As training data for the neural network, values determined from the macroscopic model based on the analysis of Continuous Cooling Transformation (CCT) diagrams were used. All relations and analyses used to build training/testing or validation sets are presented in the paper. The modelling with the use of LSTM network gives the possibility to determine the incremental changes of phase transformation (in a given time step) with the assumed changes of temperature resulting from the considered cooling rate.

Predicción y modelado del comportamiento de corrientes armónicas en sistemas fotovoltaicos conectados a red basados en redes NARX

This research presents the modeling and prediction of the harmonic behavior of current in an electric power supply grid with integration of photovoltaic power by inverters. The methodology used was based on the use of recurrent artificial neural networks of the nonlinear autoregressive with external input type. Work data was obtained from experimental sources through the use of a test bench, measurement, acquisition and monitoring equipment. The input-output parameters for the neural network were the current values in the inverter and in the supply grid respectively. The results showed that the neural network can capture the dynamics of the analyzed system. The generated model presented flexibility in data handling, allowing to represent and predict the behavior of the harmonic phenomenon. The obtained algorithm can be transferred to physical or virtual systems for the control or reduction of harmonic distortion.

Modeling of Harmonic Current in Electrical Grids with Photovoltaic Power Integration Using a Nonlinear Autoregressive with External Input Neural Networks

This research presents the modeling and prediction of the harmonic behavior of current in an electric power supply grid with the integration of photovoltaic power by inverters using artificial neural networks to determine if the use of the proposed neural network is capable of capturing the harmonic behavior of the photovoltaic energy integrated into the user’s electrical grids. The methodology used was based on the use of recurrent artificial neural networks of the nonlinear autoregressive with external input type. Work data were obtained from experimental sources through the use of a test bench, measurement, acquisition, and monitoring equipment. The input–output parameters for the neural network were the current values in the inverter and the supply grid, respectively. The results showed that the neural network can capture the dynamics of the analyzed system. The generated model presented flexibility in data handling, allowing to represent and predict the behavior of the harmonic phenomenon. The obtained algorithm can be transferred to physical or virtual systems for the control or reduction of harmonic distortion.

Wearable motion sensor-based chewing side detection

Chewing side preference means a tendency to use one side to chew food more frequently than the other. Medical studies show that chewing side preference can result in lateral facial asymmetry, teeth abrasion, temporomandibular disorders, malocclusion, and stomach illness. To continuously detect chewing side preference and quantify its severity in daily life, several wearable sensor-based methods have been proposed in recent years. However, these methods are either intrusive or not fine-grained enough. In this paper, we propose a wearable motion sensor-based chewing side detection method. We observe that chewing activity generates mastication muscle bulge and skull vibration, which can be sensed by motion sensors worn on the mastication muscles. In addition, the muscle bulge and skull vibration of the chewing side are different from those of the non-chewing side. These observations motivate us to deploy motion sensors on the left and right temporalis muscles to detect chewing sides. We propose a heuristic-rules based method to exclude non-chewing data and segment each chew accurately. The relative difference series of the left and right sensors are then calculated to characterize the difference of muscle bulge and skull vibration between the chewing side and the non-chewing side. A two-class classifier is trained using long short-term memory (LSTM), an artificial recurrent neural network, to model the data samples and classify chewing sides. A real-world evaluation dataset of eight food types is collected from eight human subjects. The average detection accuracy reaches 84.8%. The highest detection accuracy for a single subject is up to 97.4%.

Battery monitoring system using machine learning

Battery life prediction helps in smooth and uniform functioning of the battery-operated systems. Although, the capacity of the battery can be monitored by some devices, they cannot estimate how long the battery can work before a failure occurs. The technique that we have proposed here, estimates the life span of a battery using Long Short Term-Memory (LSTM), an artificial Recurrent Neural Network (RNN) architecture in Machine Learning (ML). The battery life is measured by considering each cell voltage, load voltage, temperature of the battery and charge-discharge cycle. The voltage and temperature of the battery cells are measured by a thermistor and microcontroller. The voltage values fetched by the micro controller are sent to a web server and these values are displayed on a web based mobile phone application. Balance charging of the battery cells and over charge protection is provided by constantly monitoring the battery status. This paper includes explanation on different circuit parts, algorithm used in training the model, Graphical User Interface (GUI) and the test results.

Predicting temperature of Erbil City applying deep learning and neural network

<span>One of the most significant and daunting activities in today's world is temperature prediction. The meteorologists traditionally predict temperature via some statistical models aimed to forecast the fluctuations that might have happened to atmospheric parameters such as temperature, humidity, etc. The main objective of this paper is to build an intelligent temperature prediction model of Erbil city in KRG/ Iraq based on a historical dataset from 1992 to 2016 in each year there are twelve months’ average temperature readings from (January to December). Hence to resolve this prediction problem an up-to-date deep learning neural network has been used, the network model is based on <span id="docs-internal-guid-850bd062-7fff-c6c8-9146-ba6427eb24e0" style="font-size: 9pt; font-family: 'Times New Roman'; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">long short-term memory</span> (LSTM) as an artificial recurrent neural network (RNN) architecture which employed to estimate the future average temperature. The implementing model uses the dataset from real-time 30 weather stations deployed in the area of the city. The prediction performance of the proposed recurrent neural network model has been compared with some state of art algorithms like Adeline neural network, Autoregressive neural network (NAR), and <span id="docs-internal-guid-14d37b98-7fff-0f76-848f-ad9f89224f77" style="font-size: 9pt; font-family: 'Times New Roman'; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> generalized regression neural network</span> (GRNN). The results show that the proposed model based on deep learning gives minimum prediction error.</span>

Forecasting solar radiation using a deep long short-term memory artificial neural network

Solar systems are widely used to mitigate the environmental impact of the energy sector and their importance has constantly increased due to the recent EU’s strategy to lower the CO2 emissions. Moreover, the newest Energy of Buildings Directive empathises the importance of producing energy from renewable sources to decrease the overall impact of buildings over the total end-use energy consumption. Generally, the systems’ performances are highly correlated with the incident solar radiation and outdoor air temperature. Thus, being able to accurately forecast these two parameters represents a vital step in dimensioning and maximizing the overall energy production. This Paper presents the results obtained by implementing a deep recurrent artificial neural network (ANN) trained with on year solar radiation data harvested from the UPB campus. The time series data was modelled using a special ANN architecture – the LSTM (Long Short-Term Memory) – due to its special designed internal ‘memory’ which increases its capabilities of predicting temporal sequence data. The model uses sequences of 24 hours and the resulted mean squared error (mse) for both training and validation data is under 30%.

Recognition of human locomotion on various transportations fusing smartphone sensors

Recognition of daily human activities in various locomotion and transportation modes has numerous applications like coaching users for behavior modification and maintaining a healthy lifestyle. Besides, applications and user interfaces aware of user mobility through their smartphones can also aid in urban transportation planning, smart parking, and vehicular traffic monitoring. In this paper, we explored smartphone sensor-based two benchmark datasets (Sussex Huawei Locomotion (SHL) and Transportation Mode Detection (TMD)). Firstly, we demonstrated preprocesssing of sensor data, window length optimization based on Akaike Information Criteria (AIC), and introduced smartphone orientation independent features. We also provided an in-depth analysis of different smartphone sensors’ importance for classifying daily activities and transportation modes. We justified the sensor relevance by showing the variation of performances with the number of sensors explored. For refining classifier predictions, we also proposed a post-processing approach named “Mode technique”. This method primarily concentrates on the statistical analysis of transportation modes and improves the activity recognition rate in statistical classifiers: Decision Tree, K-Nearest Neighbors, Linear Discriminant Analysis, Logistic Regression, Support Vectors Machine with RBF kernel, Random Forest, and deep learning-based methods: Artificial Neural Network and Recurrent Neural Network by smoothing the outputs of these classifiers. Besides, we showed the use of magnitude and jerk-based features to overcome the overfitting problem due to smartphone orientation. We obtained 97.2% accuracy in the SHL dataset and 99.13% accuracy in the TMD dataset. These results demonstrate that our approach can profoundly recognize various activities in advanced locomotion and transportation modes compared to existing methods in two large-scale datasets.

Groundwater level forecasting with artificial neural networks: a comparison of long short-term memory (LSTM), convolutional neural networks (CNNs), and non-linear autoregressive networks with exogenous input (NARX)

Abstract. It is now well established to use shallow artificial neural networks (ANNs) to obtain accurate and reliable groundwater level forecasts, which are an important tool for sustainable groundwater management. However, we observe an increasing shift from conventional shallow ANNs to state-of-the-art deep-learning (DL) techniques, but a direct comparison of the performance is often lacking. Although they have already clearly proven their suitability, shallow recurrent networks frequently seem to be excluded from the study design due to the euphoria about new DL techniques and its successes in various disciplines. Therefore, we aim to provide an overview on the predictive ability in terms of groundwater levels of shallow conventional recurrent ANNs, namely non-linear autoregressive networks with exogenous input (NARX) and popular state-of-the-art DL techniques such as long short-term memory (LSTM) and convolutional neural networks (CNNs). We compare the performance on both sequence-to-value (seq2val) and sequence-to-sequence (seq2seq) forecasting on a 4-year period while using only few, widely available and easy to measure meteorological input parameters, which makes our approach widely applicable. Further, we also investigate the data dependency in terms of time series length of the different ANN architectures. For seq2val forecasts, NARX models on average perform best; however, CNNs are much faster and only slightly worse in terms of accuracy. For seq2seq forecasts, mostly NARX outperform both DL models and even almost reach the speed of CNNs. However, NARX are the least robust against initialization effects, which nevertheless can be handled easily using ensemble forecasting. We showed that shallow neural networks, such as NARX, should not be neglected in comparison to DL techniques especially when only small amounts of training data are available, where they can clearly outperform LSTMs and CNNs; however, LSTMs and CNNs might perform substantially better with a larger dataset, where DL really can demonstrate its strengths, which is rarely available in the groundwater domain though.

Supervised Distributed Multi-Instance and Unsupervised Single-Instance Autoencoder Machine Learning for Damage Diagnostics with High-Dimensional Data—A Hybrid Approach and Comparison Study

Structural health monitoring (SHM) is a promising technique for in-service inspection of technical structures in a broad field of applications in order to reduce maintenance efforts as well as the overall structural weight. SHM is basically an inverse problem deriving physical properties such as damages or material inhomogeneity (target features) from sensor data. Often models defining the relationship between predictable features and sensors are required but not available. The main objective of this work is the investigation of model-free distributed machine learning (DML) for damage diagnostics under resource and failure constraints by using multi-instance ensemble and model fusion strategies and featuring improved scaling and stability compared with centralised single-instance approaches. The diagnostic system delivers two features: A binary damage classification (damaged or non-damaged) and an estimation of the spatial damage position in case of a damaged structure. The proposed damage diagnostics architecture should be able to be used in low-resource sensor networks with soft real-time capabilities. Two different machine learning methodologies and architectures are evaluated and compared posing low- and high-resolution sensor processing for low- and high-resolution damage diagnostics, i.e., a dedicated supervised trained low-resource and an unsupervised trained high-resource deep learning approach, respectively. In both architectures state-based recurrent artificial neural networks are used that process spatially and time-resolved sensor data from experimental ultrasonic guided wave measurements of a hybrid material (carbon fibre laminate) plate with pseudo defects. Finally, both architectures can be fused to a hybrid architecture with improved damage detection accuracy and reliability. An extensive evaluation of the damage prediction by both systems shows high reliability and accuracy of damage detection and localisation, even by the distributed multi-instance architecture with a resolution in the order of the sensor distance.

The structure dilemma in biological and artificial neural networks

Brain research up to date has revealed that structure and function are highly related. Thus, for example, studies have repeatedly shown that the brains of patients suffering from schizophrenia or other diseases have a different connectome compared to healthy people. Apart from stochastic processes, however, an inherent logic describing how neurons connect to each other has not yet been identified. We revisited this structural dilemma by comparing and analyzing artificial and biological-based neural networks. Namely, we used feed-forward and recurrent artificial neural networks as well as networks based on the structure of the micro-connectome of C. elegans and of the human macro-connectome. We trained these diverse networks, which markedly differ in their architecture, initialization and pruning technique, and we found remarkable parallels between biological-based and artificial neural networks, as we were additionally able to show that the dilemma is also present in artificial neural networks. Our findings show that structure contains all the information, but that this structure is not exclusive. Indeed, the same structure was able to solve completely different problems with only minimal adjustments. We particularly put interest on the influence of weights and the neuron offset value, as they show a different adaption behaviour. Our findings open up new questions in the fields of artificial and biological information processing research.

Forecasting Bitcoin Price Based on Blockchain Information Using Long-Short Term Method

Since its founding in 2008, Bitcoin (financial code: BTC) has emerged as a digital currency in market cap and continues to attract investors and policymakers' attention. In recent years, BTC has high price volatility, a substantial increase in 2016, followed by a significant decline in 2018. Unlike stock markets, BTC is open for 24x7 dan has no closing period. It means everyone can trade it for any time. However, this flexibility carries investment risk. This research attempts to forecast BTC's price by considering the blockchain's information to minimize the risk. We employ Long-Short Term Memory (LSTM), the artificial Recurrent Neural Network (RNN) architecture. Its model can avoid long-term problems. The data used is BTC's price and blockchain information data from August 4, 2018, to January 21, 2020. The model with 20 neurons and 500 epochs has the smallest MSE value. Then a prediction has an accuracy rate of 91.07%.

Relationship between the brain and the central pattern generator based on recurrent neural network

In animals, the command centers in the brain can drive the locomotion. However, it remains unclear how the brain modulates the locomotor central pattern generator (CPG). In this paper, a novel model is established to describe the relation between the brain and the CPG with time delay. The artificial recurrent neural network (RNN) consists of various computational modules that are used to model the brain. The brain synchronization under amplitude and frequency variations of the CPG and the effect of the RNN parameters variations on the CPG are investigated. In the paper, the excitatory neuron probability and average connections number are parameter space of RNN and the parameter space of CPG, which include frequency and amplitude. According to the simulation results, the best RNN synchronization could be obtained by finding the optimum parameters space between the RNN and the CPG. I propose that the parameter space of some CPGs is related to the parameter space of the brain. This leads to a brain load decrement that facilities the control action. The results are meaningful to investigate how to study the relationship between the brain and the locomotion.

Artificial Neural Network Hysteresis Operators for the Identification of Hammerstein Hysteretic Systems

Abstract This paper introduces explicit neural representations of fundamental hysteresis operators such as the play and stop operators. The hysteresis neurons are represented by recurrent artificial neural networks (ANN) using classical activation functions and are trained with gradient-based learning algorithms. The hysteresis neurons are combined into a single ANN hysteretic layer which, from a mathematical point of view, is equivalent to a classical Prandtl-Ishlinskii model expression, but has all the advantages of ANNs. One such benefit is the flexibility it offers to be combined with other ANN layers to obtain various complex model structures. In this paper, this is illustrated by combining it with a linear recurrent neural network to obtain a Hammerstein model structure, which is often considered for the characterization of systems driven by a class of smart material-based actuators. The input nonlinearity in the Hammerstein model is represented by a Prandtl-Ishlinskii ANN hysteretic layer while the linear dynamics are captured using a linear recurrent ANN representing a single-input multiple-output linear time-invariant state-space model. The identification approach of the Hammerstein model with a hysteretic layer is illustrated both numerically and experimentally.

New method for wind potential prediction using recurrent artificial neural networks

The aim of the study is to find the right architecture of the NARX neural network, in order to perform the daily prediction of the maximum wind speed of Laayoune city. We relied on the Levenberg-Marquardt optimization algorithm. The RMSE error metric showed that NARX-SP outperforms NARX-P.