Basic Recurrent Neural Network Research Articles

Hybrid Intelligent Control System for Adaptive Microgrid Optimization: Integration of Rule-Based Control and Deep Learning Techniques

Microgrids (MGs) have evolved as critical components of modern energy distribution networks, providing increased dependability, efficiency, and sustainability. Effective control strategies are essential for optimizing MG operation and maintaining stability in the face of changing environmental and load conditions. Traditional rule-based control systems are extensively used due to their interpretability and simplicity. However, these strategies frequently lack the flexibility for complex and changing system dynamics. This paper provides a novel method called hybrid intelligent control for adaptive MG that integrates basic rule-based control and deep learning techniques, including gated recurrent units (GRUs), basic recurrent neural networks (RNNs), and long short-term memory (LSTM). The main target of this hybrid approach is to improve MG management performance by combining the strengths of basic rule-based systems and deep learning techniques. These deep learning techniques readily enhance and adapt control decisions based on historical data and domain-specific rules, leading to increasing system efficiency, stability, and resilience in adaptive MG. Our results show that the proposed method optimizes MG operation, especially under demanding conditions such as variable renewable energy supply and unanticipated load fluctuations. This study investigates special RNN architectures and hyperparameter optimization techniques with the aim of predicting power consumption and generation within the adaptive MG system. Our promising results show the highest-performing models indicating high accuracy and efficiency in power prediction. The finest-performing model accomplishes an R2 value close to 1, representing a strong correlation between predicted and actual power values. Specifically, the best model achieved an R2 value of 0.999809, an MSE of 0.000002, and an MAE of 0.000831.

Regular paper Comparative Analysis of Deep Learning Models for Sentiment Analysis on IMDB Reviews

Within the domain of natural language processing, sentiment analysis assumes a fundamental position, facilitating the comprehension of societal perspectives and opinions, thereby playing a pivotal role in understanding public sentiment. In this study, an examination and comparison of deep learning architectures was conducted on IMDB movie reviews. We evaluated the performance of Basic Recurrent Neural Networks, Long Short-Term Memory (LSTM), Gated Recurrent Neural Unit (GRU), Bidirectional LSTM, Bidirectional GRU and 1D Convolutional Neural Networks (Conv1D) based on their training, validation, and testing accuracies. Our results indicate that while LSTM achieved the highest accuracy of 99.91% on the training data, GRU demonstrated superior performance (88.28%) on the validation dataset. Interestingly, Bidirectional GRU emerged as the top-performing model (87.54%) on the testing data, showcasing its robustness in generalizing to unseen instances. These findings highlight the importance of evaluating model performance across multiple datasets to assess their real-world effectiveness. Furthermore, our comparative analysis provides valuable understanding of the advantages and limitations of each model, offering practical guidance for selecting the optimal framework for sentiment analysis endeavors. Overall, this research contributes to the progress of such methodologies and deep learning approaches in natural language processing.

Stock Price Prediction using Multi-Layered Sequential LSTM

Stock markets are frequently among the most volatile locations to invest in. The choice to buy or sell stocks is heavily influenced by statistical analysis of prior stock performance and external circumstances. All these variables are employed to maximize profitability. Stock value prediction is a hard undertaking that necessitates a solid computational foundation to compute longer-term share values. Stock prices are connected inside the market, making it harder to forecast expenses. Financial data is a category that includes past data from time series that provides a lot of knowledge and is frequently employed in data analysis tasks. This research provides a unique optimisation strategy for stock price prediction based on a Multi-Layer Sequential Long Short Term Memory (MLS LSTM) model and the adam optimizer in this context. Furthermore, to make reliable predictions, the MLS LSTM algorithm uses normalised time series data separated into time steps to assess the relationship between past and future values. Furthermore, it solves the vanishing gradient problem that plagues basic recurrent neural networks.

Sediment load forecasting of Gobindsagar reservoir using machine learning techniques

With ever advancing computer technology in machine learning, sediment load prediction inside the reservoirs has been computed using various artificially intelligent techniques. The sediment load in the catchment region of Gobindsagar reservoir of India is forecasted in this study utilizing the data collected for years 1971–2003 using several models of intelligent algorithms. Firstly, multi-layered perceptron artificial neural network (MLP-ANN), basic recurrent neural network (RNN), and other RNN based models including long-short term memory (LSTM), and gated recurrent unit (GRU) are implemented to validate and predict the sediment load inside the reservoir. The proposed machine learning models are validated for Gobindsagar reservoir using three influencing factors on yearly basis [rainfall (Ra), water inflow (Iw), and the storage capacity (Cr)]. The results demonstrate that the suggested MLP-ANN, RNN, LSTM, and GRU models produce better results with maximum errors reduced from 24.6% to 8.05%, 7.52%, 1.77%, and 0.05% respectively. For future prediction of the sediment load for next 22 years, the influencing factors were first predicted for next 22 years using ETS forecasting model with the help of data collected for 33 years. Additionally, it was noted that each prediction’s error was lower than that of the reference model. Furthermore, it was concluded that the GRU model predicts better results than the reference model and its alternatives. Secondly, by comparing the prediction precision of all the machine learning models established in this study, it can be evidently shown that the LSTM and GRU models were superior to the MLP-ANN and RNN models. It is also observed that among all, the GRU took the best precision due to the highest R of 0.9654 and VAF of 91.7689%, and the lowest MAE of 0.7777, RMSE of 1.1522 and MAPE of 0.3786%. The superiority of GRU can also be ensured from Taylor’s diagram. Lastly, Garson’s algorithm and Olden’s algorithm for MLP-ANN, as well as the perturbation method for RNN, LSTM, and GRU models, are used to test the sensitivity analysis of each influencing factor in sediment load forecasting. The sediment load was discovered to be most sensitive to the annual rainfall.

Multivariate time series prediction by RNN architectures for energy consumption forecasting

Households and buildings have been utilizing the traditional electric network structure for the last decade, relying on energy supplied by manufacturing centers based on fossil fuels. Large energy use places a burden on such centers. In this perspective, smart grids are a new technology and a new generation of traditional electric networks that provide increased efficiency, dependability, and energy management based on demand optimization. The importance of smart grids can also be seen in the possibility of integrating communication systems for energy demand forecasting, to provide an optimal management of the combination of renewable energies and production centers energy. The authors present a comparative analysis of several deep learning models, notably Recurrent Neural Network (RNN) architectures such as basic RNN, Long Short Term Memory (LSTM), and Gated Recurrent Unit (GRU), in this paper. These architectures are trained and tested on the Smart Grid Smart City (SGSC) project’s energy datasets (2010–2014) and assessed using a variety of indicators such as Root Mean Square Error (RMSE), Mean Absolute Error MAE, and R2 scores in order to analyze, compare and ultimately choose the most efficient model. As expected from the literature of RNN architectures, with the lowest value of RMSE error and the highest value of R2 Score among the three architectures, GRU outperformed both of basic RNN and LSTM, this result can be explained by several reasons the most important one is the ability of the GRU model to deal with the vanishing gradient problem and the impact of the number of parameters, used in building such a model, on the same problem.

In-depth understanding of LSTM and its recent advances in lung disease diagnosis

Of late, long short-term memory (LSTM) has proven its worth in medical diagnosis. Hence, there is a need to explore this special version of recurrent neural network (RNN), which can learn long-term dependencies. LSTM addresses the short-term memory problem of basic RNNs. In this paper, an in-depth study of LSTM is done with the help of a few real-life examples. Some of the recent advances of LSTM in COVID-19 and other lung disease diagnoses have also been discussed.

Real-time twitter data analytics of mental illness in COVID-19: sentiment analysis using deep neural network

The World Health Organization (WHO) <span>states that the COVID-19 epidemic is being treated as a pandemic, with thousands of individuals infected and dead worldwide. School and college students are suffering from their online classes without any physical activities. Working men and women are also suffering from their working situations, as lots of people have lost their jobs and unemployment rates have become high due to the pandemic, and people are also losing physical contact with other family members, friends, and colleagues. The main objective of the proposed model is to monitor and analyse the real-time Twitter data-related tweets, such as coronavirus mental illness that are commonly used while referencing the pandemic. We have compared three deep learning approaches to sentiment analysis and found them to be useful. The first deep learning technique is to use a basic recurrent neural network (RNN), and the second is to use a deep learning RRN with long short-term memory (LSTM), followed by a gated recurrent unit (GRU). The experiment results indicate that the recurrent neural network built using GRU has the maximum accuracy of 99.47% for positive, negative, and neutral words and statements in Twitter data.</span>

ECG-Based Arrhythmia Classification using Recurrent Neural Networks in Embedded Systems

Cardiac arrhythmia is one of the most important cardiovascular diseases (CVDs), causing million deaths every year. Moreover it is difficult to diagnose because it occurs intermittently and as such requires the analysis of large amount of data, collected during the daily life of patients. An important tool for CVD diagnosis is the analysis of electrocardiogram (ECG), because of its non-invasive nature and simplicity of acquisition. In this work we propose a classification algorithm for arrhythmia based on recurrent neural networks (RNNs) that operate directly on ECG data, exploring the effectiveness and efficiency of several variations of the general RNN, in particular using different types of layers implementing the network memory. We use the MIT-BIH arrhythmia database and the evaluation protocol recommended by the Association for the Advancement of Medical Instrumentation (AAMI). After designing and testing the effectiveness of the different networks, we then test its porting to an embedded platform, namely the STM32 microcontroller architecture from ST, using a specific framework to port a pre-built RNN to the embedded hardware, convert it to optimized code for the platform and evaluate its performance in terms of resource usage. Both in binary and multiclass classification, the basic RNN model outperforms the other architectures in terms of memory storage (∼117 KB), number of parameters (∼5 k) and inference time (∼150 ms), while the RNN LSTM-based achieved the best accuracy (∼90%).

Time series modelling and forecasting for predicting Covid19 Case Load using LSTM

The epidemic of the Novel Coronavirus across the globe has influenced the globe overall and caused a large number of death results. This remains as an unfavorable admonition to general wellbeing and will be set apart as perhaps the inordinate pandemics in the history. Inorder to validate and analyse, the details was taken from COVID-19. The detail contains daily tallies of confirmed, relieved and demise cases. Likewise, it includes extra data with respect to patients testing present in various states and the outcomes isolated in confirmed and invalidated cases. With the data provided it allows infected person to get the proper treatment and timely quarantine. The proposed paper utilizes Long Short Term Memory (LSTM ) networks for sequential prediction of data. The networks are viable apparatuses in short-term time series gauge the COVID-19 confirmed cases. It is a complex gated memory unit made to disappearing gradient issues restricting the effectiveness of a basic Recurrent Neural Network (RNN). Here Neural Network is used to solve the complex operation on the dataset. The result demonstrate that the LSTM Network is executed with various activation functions by utilizing a exponential linear unit brought about better execution for determining the complete number of COVID-19 cases. With the timely observations the corona virus state can be effectively monitored and the proper treatment can be assigned for the infected ones.

Online dynamic prediction of potassium concentration in biomass fuels through flame spectroscopic analysis and recurrent neural network modelling

Biomass fuels are widely used as a renewable source for heat and power generation. Alkali metals in a biomass fuel have an significant impact on furnace safety as such metals lead to fouling and slagging in the furnace and corrosion of water pipes. This paper presents a technique for dynamic predicting Potassium (K) concentration in a biomass fuel based on spectroscopic analysis and different recurrent neural networks. A miniature spectrometer is employed to acquire the spectroscopic signals of K in different biomass fuels, including peanut shell, willow, corn cob, corn straw and wheat straw, and their blends. The spectroscopic features of K are extracted. The factors that influence the spectral intensity of K in the biomass fuels are investigated. A basic recurrent neural network (RNN), and its variants, i.e., long short-term memory neural network (LSTM-NN) and deep recurrent neural network (DRNN), are constructed using the spectroscopic signal of K from the spectrometer. The performances of the neural networks for the dynamic prediction of K concentration are compared and analysed theoretically and experimentally. It is found that the relative error in the K concentration prediction through the use of the DRNN model is within 6.34% whilst the LSTM-NN and RNN models give errors slightly greater than this.

Optical Proximity Correction Using Bidirectional Recurrent Neural Network With Attention Mechanism

Recurrent neural network (RNN) is employed as a machine learning model for fast optical proximity correction (OPC). RNN consists of a number of neural network instances which are serially connected, with each instance in charge of one segment. RNN thus allows some localized segments to be corrected together in one execution, which offers higher accuracy. A basic RNN is extended by introducing gated recurrent unit (GRU) cells in recurrent hidden layers, which are then connected in both forward and backward directions; attention layer is adopted to help output layer predict mask bias more accurately. The proposed RNN structure is used for OPC implementation. A choice of input features, sampling of training data, an algorithm of mapping segments to neural network instances, and final mask bias calculation are addressed toward efficient implementation. Experiments demonstrate that the proposed OPC method corrects a mask layout with 36% lower EPE compared to state-of-the-art OPC method using artificial neural network structure.

Improving Radar-Based Rainfall Forecasts by Long Short-Term Memory Network in Urban Basins

Radar-based rainfall forecasts are widely used extrapolation algorithms that are popular in systems of precipitation for predicting up to six hours in lead time. Nevertheless, the reliability of rainfall forecasts gradually declines for heavy rain events with lead time due to the lack of predictability. Recently, data-driven approaches were commonly implemented in hydrological problems. In this research, the data-driven models were developed based on the data obtained from a radar forecasting system named McGill Algorithm for Precipitation nowcasting by Lagrangian Extrapolation (MAPLE) and ground rain gauges. The data included thirteen urban stations in the five metropolitan cities located in South Korea. The twenty-five data points of MAPLE surrounding each rain station were utilized as the model input, and the observed rainfall at the corresponding gauges were used as the model output. The results showed superior capabilities of long short-term memory (LSTM) network in improving 180-min rainfall forecasts at the stations based on a comparison of five different data-driven models, including multiple linear regression (MLR), multivariate adaptive regression splines (MARS), multi-layer perceptron (MLP), basic recurrent neural network (RNN), and LSTM. Although the model still produced an underestimation of extreme rainfall values at some examined stations, this study proved that the LSTM could provide reliable performance. This model can be an optional method for improving rainfall forecasts at the stations for urban basins.

A Prediction Method of Trend-Type Capacity Index Based on Recurrent Neural Network

Due to the increase in the types of business and equipment in telecommunications companies, the performance index data collected in the operation and maintenance process varies greatly. The diversity of index data makes it very difficult to perform high-precision capacity prediction. In order to improve the forecasting efficiency of related indexes, this paper designs a classification method of capacity index data, which divides the capacity index data into trend type, periodic type and irregular type. Then for the prediction of trend data, it proposes a capacity index prediction model based on Recurrent Neural Network (RNN), denoted as RNN-LSTM-LSTM. This model includes a basic RNN, two Long Short-Term Memory (LSTM) networks and two Fully Connected layers. The experimental results show that, compared with the traditional HoltWinters, Autoregressive Integrated Moving Average (ARIMA) and Back Propagation (BP) neural network prediction model, the mean square error (MSE) of the proposed RNN-LSTM-LSTM model are reduced by 11.82% and 20.34% on the order storage and data migration, which has greatly improved the efficiency of trend-type capacity index prediction.

Forecasting Copper Electrorefining Cathode Rejection by Means of Recurrent Neural Networks With Attention Mechanism

Electrolytic refining is the last step of pyrometallurgical copper production. Here, smelted copper is converted into high-quality cathodes through electrolysis. Cathodes that do not meet the physical quality standards are rejected and further reprocessed or sold at a minimum profit. Prediction of cathodic rejection is therefore of utmost importance to accurately forecast the electrorefining cycle economic production. Several attempts have been made to estimate this process outcomes, mostly based on physical models of the underlying electrochemical reactions. However, they do not stand the complexity of real operations. Data-driven methods, such as deep learning, allow modeling complex non-linear processes by learning representations directly from the data. We study the use of several recurrent neural network models to estimate the cathodic rejection of a cathodic cycle, using a series of operational measurements throughout the process. We provide an ARMAX model as a benchmark. Basic recurrent neural network models are analyzed first: a vanilla RNN and an LSTM model provide an initial approach. These are further composed into an Encoder-Decoder model, that uses an attention mechanism to selectively weight the input steps that provide most information upon inference. This model obtains 5.45% relative error, improving by 81.4% the proposed benchmark. Finally, we study the attention mechanism’s output to distinguish the most relevant electrorefining process steps. We identify the initial state as critical in predicting cathodic rejection. This information can be used as an input for decision support systems or control strategies to reduce cathodic rejection and improve electrolytic refining’s profitability.

Venue-Popularity Prediction Using Social Data Participatory Sensing Systems and RNNs

Analyzing social data as a participatory sensing system (PSS) provides a deep understanding of city dynamics, such as people's mobility patterns, social patterns, and events detection. In a PSS, individuals with mobile devices sense their environment, collect, and share data. For smart cities, intelligent city dynamics analysis has many applications such as for urban planning, transportation systems, city environment, energy consumption, public safety, and city economy. This study aimed to develop an intelligent application to predict the potential number of visitors for specific venues based on the analysis of mobility patterns of individuals. The ability to accurately predict the number of visitors to a venue allows authorities to better understand the behavior of the people and allocate recourses accordingly. We formulated the venue-popularity problem as a sequence-based regression and classification problem. We employed three recurrent neural network (RNN)-based models to predict the locations of popular venues on a city map. The proposed models include basic RNN, long short-term memory (LSTM), and gated recurrent unit (GRU). We constructed several social datasets for Riyadh city using Twitter and Foursquare as the PSS. Our results revealed that modeling venue-popularity prediction as a sequence regression problem yields better results than modeling it as a sequence classification problem. For the city-popularity map prediction problem, the vector autoregression baseline model achieved better performance than the RNN-family models.

Efficient two-microphone speech enhancement using basic recurrent neural network cell for hearing and hearing aids.

This work presents a two-microphone speech enhancement (SE) framework based on basic recurrent neural network (RNN) cell. The proposed method operates in real-time, improving the speech quality and intelligibility in noisy environments. The RNN model trained using a simple feature set-real and imaginary parts of the short-time Fourier transform (STFT) are computationally efficient with a minimal input-output processing delay. The proposed algorithm can be used in any stand-alone platform such as a smartphone using its two inbuilt microphones. The detailed operation of the real-time implementation on the smartphone is presented. The developed application works as an assistive tool for hearing aid devices (HADs). Speech quality and intelligibility test results are used to compare the proposed algorithm to existing conventional and neural network-based SE methods. Subjective and objective scores show the superior performance of the developed method over several conventional methods in different noise conditions and low signal to noise ratios (SNRs).

Recurrent Neural Networks and its variants in Remaining Useful Life prediction

Data-driven techniques, especially artificial intelligence (AI) based deep learning (DL) techniques, have attracted more and more attention in the manufacturing sector because of the rapid growth of the industrial Internet of Things (IoT) and Big Data. Tremendous researches of DL techniques have been applied in machine health monitoring, but still very limited works focus on the application of DL on the Remaining Useful Life (RUL) prediction. Precise RUL prediction can significantly improve the reliability and operational safety of industrial components or systems, avoid fatal breakdown and reduce the maintenance costs. This paper reviews and compares the state-of-the-art DL approaches for RUL prediction focusing on Recurrent Neural Networks (RNN) and its variants. It has been observed from the results for a publicly available dataset that Long Short-Term Memory (LSTM) networks and Gated Recurrent Unit (GRU) networks outperform the basic RNNs, and the number of the network layers affects the performance of the prediction.

Bayesian Recurrent Neural Network Models for Forecasting and Quantifying Uncertainty in Spatial-Temporal Data.

Recurrent neural networks (RNNs) are nonlinear dynamical models commonly used in the machine learning and dynamical systems literature to represent complex dynamical or sequential relationships between variables. Recently, as deep learning models have become more common, RNNs have been used to forecast increasingly complicated systems. Dynamical spatio-temporal processes represent a class of complex systems that can potentially benefit from these types of models. Although the RNN literature is expansive and highly developed, uncertainty quantification is often ignored. Even when considered, the uncertainty is generally quantified without the use of a rigorous framework, such as a fully Bayesian setting. Here we attempt to quantify uncertainty in a more formal framework while maintaining the forecast accuracy that makes these models appealing, by presenting a Bayesian RNN model for nonlinear spatio-temporal forecasting. Additionally, we make simple modifications to the basic RNN to help accommodate the unique nature of nonlinear spatio-temporal data. The proposed model is applied to a Lorenz simulation and two real-world nonlinear spatio-temporal forecasting applications.

Enhancing recurrent neural network-based language models by word tokenization

Different approaches have been used to estimate language models from a given corpus. Recently, researchers have used different neural network architectures to estimate the language models from a given corpus using unsupervised learning neural networks capabilities. Generally, neural networks have demonstrated success compared to conventional n-gram language models. With languages that have a rich morphological system and a huge number of vocabulary words, the major trade-off with neural network language models is the size of the network. This paper presents a recurrent neural network language model based on the tokenization of words into three parts: the prefix, the stem, and the suffix. The proposed model is tested with the English AMI speech recognition dataset and outperforms the baseline n-gram model, the basic recurrent neural network language models (RNNLM) and the GPU-based recurrent neural network language models (CUED-RNNLM) in perplexity and word error rate. The automatic spelling correction accuracy was enhanced by approximately 3.5% for Arabic language misspelling mistakes dataset.

Optimization of RNN-Based Speech Activity Detection

Speech activity detection (SAD) is an essential component of automatic speech recognition systems impacting the overall system performance. This paper investigates an optimization process for recurrent neural network (RNN) based SAD. This process optimizes all system parameters including those used for feature extraction, the NN weights, and the back-end parameters. Three cost functions are considered for SAD optimization: the frame error rate, the NIST detection cost function, and the word error rate of a downstream speech recognizer. Different types of RNN models and optimization methods are investigated. Three types of RNNs are compared: a basic RNN, long short-term memory (LSTM) network with peepholes, and a coordinated-gate LSTM (CG-LSTM) network introduced by Gelly and Gauvain. Well suited for nondifferentiable optimization problems, quantum-behaved particle swarm optimization is used to optimize feature extraction and posterior smoothing, as well as for the initial training of the neural networks. Experimental SAD results are reported on the NIST 2015 SAD evaluation data as well as REPERE and AMI meeting corpora. Speech recognition results are reported on the OpenKWS’13 test data. For all tasks and conditions, the proposed optimization method significantly improves the SAD performance and among all the tested SAD methods the CG-LSTM model gives the best results.