Long Short-term Memory Recurrent Neural Network Research Articles

An Interpretation of Long Short-Term Memory Recurrent Neural Network for Approximating Roots of Polynomials

This paper aims to present a flexible method for interpreting the Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) for the relational structure between the roots and the coefficients of a polynomial. A database is first developed for randomly selected inputs based on the degrees of the univariate polynomial which is then used to approximate the polynomial roots through the proposed LSTM-RNN model. Furthermore, an adaptive learning optimization algorithm is used specifically to update the network weights iteratively based on training deep neural networks data. Thus, the method can exploit the ability to find the individual learning rates for each variable through adaptive learning rate strategies to effectively prevent the weights from fluctuating in a wide spectrum. Finally, several experimental results are performed which shows that the proposed LSTM-RNN model can be used as an alternative approach to compute an approximation of each root for a given polynomial. Furthermore, the results are compared with the conventional feedforward neural network based artificial neural network model. The results clearly demonstrate the superiority of the proposed LSTM-RNN model for roots approximation in terms of accuracy, mean square error and faster convergence.

Predicting COVID-19 cases in various scenarios using RNN-LSTM models aided by adaptive linear regression to identify data anomalies.

The evolution of the Sars-CoV-2 (COVID-19) virus pandemic has revealed that the problems of social inequality, poverty, public and private health systems guided by controversial public policies are much more complex than was conceived before the pandemic. Therefore, understanding how COVID-19 evolves in society and looking at the infection spread is a critical task to support efficient epidemiological actions capable of suppressing the rates of infections and deaths. In this article, we analyze daily COVID-19 infection data with two objectives: (i) to test the predictive power of a Recurrent Neural Network - Long Short Term Memory (RNN-LSTM) on the daily stochastic fluctuation in different scenarios, and (ii) analyze, through adaptive linear regression, possible anomalies in the reported data to provide a more realistic and reliable scenario to support epidemic control actions. Our results show that the approach is even more suitable for countries, states or cities where the rate of testing, diagnosis and prevention were low during the virus dissemination. In this sense, we focused on investigating countries and regions where the disease evolved in a severe and poorly controlled way, as in Brazil, highlighting the favelas in Rio de Janeiro as a regional scenario.

Accurate storm surge forecasting using the encoder–decoder long short term memory recurrent neural network

The encoder–decoder LSTM (long short term memory) recurrent neural network is proposed to predict storm surge in Florida. Two types of hurricanes with six events are collected for training and testing. The previously observed meteorological data including the storm surge, the wind speed, the wind gust, the barometric pressure, and the air temperature are chosen as the inputs of model, while the future storm surge is designated as the output. The predicted results for 1, 3, 6, 9, and 12 h-lead time are obtained. Four indices are introduced to evaluate the accuracy and stability of the proposed model. Through comparing the predicted results by both models with the observed data, it is found that the encoder–decoder LSTM approach is more accurate than the convolutional neural network model for all considered cases. Furthermore, we notice that the combinations of storm surge, wind speed, wind gust, barometric pressure, and air temperature, and storm surge and wind speed give the best prediction for the first and the second types of hurricanes, respectively. Our work suggests that the encoder–decoder LSTM model has great potential in storm surge prediction. It is expected that the accuracy of this model can be further improved by introducing more observed data and considering more physical factors.

Macroscopic Big Data Analysis and Prediction of Driving Behavior With an Adaptive Fuzzy Recurrent Neural Network on the Internet of Vehicles

Dangerous driving behaviors are diverse and complex. Determining how to analyze the driving behavior of public drivers objectively and accurately has always been a research challenge. This research proposes a macroscopic and dynamic method for evaluating drivers’ dangerous driving degree based on a fuzzy inference system. It also designs fuzzy-macro long short-term memory (LSTM), a variant of LSTM recurrent neural networks, which can predict drivers’ dangerous driving behaviors and risk degree. We elucidate how a macroscopic fuzzy inference dangerous driving behavior system is designed based on various driving behavior factors and the neuron architecture of the fuzzy-macro LSTM network. We collect real driving behavior data of drivers on the road and conduct a series of experimental analyses. Compared with five other commonly used time-series forecasting neural network models, our fuzzy-macro LSTM model performs best in terms of prediction error. Experimental results verify the effectiveness of the proposed method for macroanalysis and prediction of dangerous driving behavior.

Long Short-Term Memory Recurrent Neural Network for Automatic Speech Recognition

Automatic speech recognition (ASR) is one of the most demanding tasks in natural language processing owing to its complexity. Recently, deep learning approaches have been deployed for this task and have been proven to outperform traditional machine learning approaches such as Artificial Neural Network (ANN). In particular, deep-learning methods such as long short-term memory (LSTM) have achieved improved ASR performance. However, this method is limited to processing continuous input streams. Traditional LSTM requires four (4) linear layers (multilayer perceptron (MLP) layer) per cell with a large memory bandwidth for each sequence time step. LSTM cannot accommodate the many computational units required for processing continuous input streams because the system does not have sufficient memory bandwidth to feed the computational units. In this study, an enhanced deep learning LSTM recurrent neural network (RNN) model was proposed to resolve this shortcoming. In the proposed model, the RNN is incorporated as a “forget gate” to the memory block to allow the resetting of cell states at the beginning of the sub-sequences. This enables the system to process continuous input streams efficiently without necessarily increasing the required bandwidths. In the proposed model, the standard architecture of the LSTM network is modified to effectively use the model parameters. Some CNN-based and sequential models were used on the same dataset, and the models were compared with the proposed model. LSTM-RNN outperformed the other deep learning models with an accuracy of 99.36% on the well-established public benchmark spoken English digit dataset.

Prediction of Lower Extremity Multi-Joint Angles during Overground Walking by Using a Single IMU with a Low Frequency Based on an LSTM Recurrent Neural Network.

The joint angle during gait is an important indicator, such as injury risk index, rehabilitation status evaluation, etc. To analyze gait, inertial measurement unit (IMU) sensors have been used in studies and continuously developed; however, they are difficult to utilize in daily life because of the inconvenience of having to attach multiple sensors together and the difficulty of long-term use due to the battery consumption required for high data sampling rates. To overcome these problems, this study propose a multi-joint angle estimation method based on a long short-term memory (LSTM) recurrent neural network with a single low-frequency (23 Hz) IMU sensor. IMU sensor data attached to the lateral shank were measured during overground walking at a self-selected speed for 30 healthy young persons. The results show a comparatively good accuracy level, similar to previous studies using high-frequency IMU sensors. Compared to the reference results obtained from the motion capture system, the estimated angle coefficient of determination () is greater than 0.74, and the root mean square error and normalized root mean square error (NRMSE) are less than 7° and 9.87%, respectively. The knee joint showed the best estimation performance in terms of the NRMSE and among the hip, knee, and ankle joints.

Application of Machine Learning to Predict Acute Kidney Disease in Patients With Sepsis Associated Acute Kidney Injury.

Background: Sepsis-associated acute kidney injury (AKI) is frequent in patients admitted to intensive care units (ICU) and may contribute to adverse short-term and long-term outcomes. Acute kidney disease (AKD) reflects the adverse events developing after AKI. We aimed to develop and validate machine learning models to predict the occurrence of AKD in patients with sepsis-associated AKI.Methods: Using clinical data from patients with sepsis in the ICU at Beijing Friendship Hospital (BFH), we studied whether the following three machine learning models could predict the occurrence of AKD using demographic, laboratory, and other related variables: Recurrent Neural Network-Long Short-Term Memory (RNN-LSTM), decision trees, and logistic regression. In addition, we externally validated the results in the Medical Information Mart for Intensive Care III (MIMIC III) database. The outcome was the diagnosis of AKD when defined as AKI prolonged for 7–90 days according to Acute Disease Quality Initiative-16.Results: In this study, 209 patients from BFH were included, with 55.5% of them diagnosed as having AKD. Furthermore, 509 patients were included from the MIMIC III database, of which 46.4% were diagnosed as having AKD. Applying machine learning could successfully achieve very high accuracy (RNN-LSTM AUROC = 1; decision trees AUROC = 0.954; logistic regression AUROC = 0.728), with RNN-LSTM showing the best results. Further analyses revealed that the change of non-renal Sequential Organ Failure Assessment (SOFA) score between the 1st day and 3rd day (Δnon-renal SOFA) is instrumental in predicting the occurrence of AKD.Conclusion: Our results showed that machine learning, particularly RNN-LSTM, can accurately predict AKD occurrence. In addition, Δ SOFAnon−renal plays an important role in predicting the occurrence of AKD.

An Evaluation of Speech-Based Recognition of Emotional and Physiological Markers of Stress

Life in modern societies is fast-paced and full of stress-inducing demands. The development of stress monitoring methods is a growing area of research due to the personal and economic advantages that timely detection provides. Studies have shown that speech-based features can be utilised to robustly predict several physiological markers of stress, including emotional state, continuous heart rate, and the stress hormone, cortisol. In this contribution, we extend previous works by the authors, utilising three German language corpora including more than 100 subjects undergoing a Trier Social Stress Test protocol. We present cross-corpus and transfer learning results which explore the efficacy of the speech signal to predict three physiological markers of stress—sequentially measured saliva-based cortisol, continuous heart rate as beats per minute (BPM), and continuous respiration. For this, we extract several features from audio as well as video and apply various machine learning architectures, including a temporal context-based Long Short-Term Memory Recurrent Neural Network (LSTM-RNN). For the task of predicting cortisol levels from speech, deep learning improves on results obtained by conventional support vector regression—yielding a Spearman correlation coefficient (ρ) of 0.770 and 0.698 for cortisol measurements taken 10 and 20 min after the stress period for the two corpora applicable—showing that audio features alone are sufficient for predicting cortisol, with audiovisual fusion to an extent improving such results. We also obtain a Root Mean Square Error (RMSE) of 38 and 22 BPM for continuous heart rate prediction on the two corpora where this information is available, and a normalised RMSE (NRMSE) of 0.120 for respiration prediction (−10: 10). Both of these continuous physiological signals show to be highly effective markers of stress (based on cortisol grouping analysis), both when available as ground truth and when predicted using speech. This contribution opens up new avenues for future exploration of these signals as proxies for stress in naturalistic settings.

Shared Blocks-Based Ensemble Deep Learning for Shallow Landslide Susceptibility Mapping

Natural disaster impact assessment is of the utmost significance for post-disaster recovery, environmental protection, and hazard mitigation plans. With their recent usage in landslide susceptibility mapping, deep learning (DL) architectures have proven their efficiency in many scientific studies. However, some restrictions, including insufficient model variance and limited generalization capabilities, have been reported in the literature. To overcome these restrictions, ensembling DL models has often been preferred as a practical solution. In this study, an ensemble DL architecture, based on shared blocks, was proposed to improve the prediction capability of individual DL models. For this purpose, three DL models, namely Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Long Short-Term Memory (LSTM), together with their ensemble form (CNN–RNN–LSTM) were utilized to model landslide susceptibility in Trabzon province, Turkey. The proposed DL architecture produced the highest modeling performance of 0.93, followed by CNN (0.92), RNN (0.91), and LSTM (0.86). Findings proved that the proposed model excelled the performance of the DL models by up to 7% in terms of overall accuracy, which was also confirmed by the Wilcoxon signed-rank test. The area under curve analysis also showed a significant improvement (~4%) in susceptibility map accuracy by the proposed strategy.

A Scheme with Acoustic Emission Hit Removal for the Remaining Useful Life Prediction of Concrete Structures.

In this study, a scheme of remaining useful lifetime (RUL) prognosis from raw acoustic emission (AE) data is presented to predict the concrete structure’s failure before its occurrence, thus possibly prolong its service life and minimizing the risk of accidental damage. The deterioration process is portrayed by the health indicator (HI), which is automatically constructed from raw AE data with a deep neural network pretrained and fine-tuned by a stacked autoencoder deep neural network (SAE-DNN). For the deep neural network structure to perform a more accurate construction of health indicator lines, a hit removal process with a one-class support vector machine (OC-SVM), which has not been investigated in previous studies, is proposed to extract only the hits which matter the most to the portrait of deterioration. The new set of hits is then harnessed as the training labels for the deep neural network. After the completion of the health indicator line construction, health indicators are forwarded to a long short-term memory recurrent neural network (LSTM-RNN) for the training and validation of the remaining useful life prediction, as this structure is capable of capturing the long-term dependencies, even with a limited set of data. Our prediction result shows a significant improvement in comparison with a similar scheme but without the hit removal process and other methods, such as the gated recurrent unit recurrent neural network (GRU-RNN) and the simple recurrent neural network.

Short-Term Nacelle Orientation Forecasting Using Bilinear Transformation and ICEEMDAN Framework

To maximize energy extraction, the nacelle of a wind turbine follows the wind direction. Accurate prediction of wind direction is vital for yaw control. A tandem hybrid approach to improve the prediction accuracy of the wind direction data is developed. The proposed approach in this paper includes the bilinear transformation, effective data decomposition techniques, long-short-term-memory recurrent neural networks (LSTM-RNNs), and error decomposition correction methods. In the proposed approach, the angular wind direction data is firstly transformed into time-series to accommodate the full range of yaw motion. Then, the continuous transformed series are decomposed into a group of subseries using a novel decomposition technique. Next, for each subseries, the wind directions are predicted using LSTM-RNNs. In the final step, it decomposed the errors for each predicted subseries to correct the predicted wind direction and then perform inverse bilinear transformation to obtain the final wind direction forecasting. The robustness and effectiveness of the proposed approach are verified using data collected from a wind farm located in Huitengxile, Inner Mongolia, China. Computational results indicate that the proposed hybrid approach outperforms the other single approaches tested to predict the nacelle direction over short-time horizons. The proposed approach can be useful for practical wind farm operations.

Research on maintenance spare parts requirement prediction based on LSTM recurrent neural network

Abstract The aim of this study was to improve the low accuracy of equipment spare parts requirement predicting, which affects the quality and efficiency of maintenance support, based on the summary and analysis of the existing spare parts requirement predicting research. This article introduces the current latest popular long short-term memory (LSTM) algorithm which has the best effect on time series data processing to equipment spare parts requirement predicting, according to the time series characteristics of spare parts consumption data. A method for predicting the requirement for maintenance spare parts based on the LSTM recurrent neural network is proposed, and the network structure is designed in detail, the realization of network training and network prediction is given. The advantages of particle swarm algorithm are introduced to optimize the network parameters, and actual data of three types of equipment spare parts consumption are used for experiments. The performance comparison of predictive models such as BP neural network, generalized regression neural network, wavelet neural network, and squeeze-and-excitation network prove that the new method is effective and provides an effective method for scientifically predicting the requirement for maintenance spare parts and improving the quality of equipment maintenance.

Using Noise Pollution Data for Traffic Prediction in Smart Cities: Experiments Based on LSTM Recurrent Neural Networks

Traffic prediction is one of the most important use cases for smart cities. Accurate traffic information is key to managing traffic issues. Many approaches that use traffic time series data to predict traffic flow have been proposed. In addition to traffic- specific parameters, some other features (called signatures) may be associated with road traffic, i.e., air and noise pollution. In this paper, we show how noise pollution and traffic time-series data were used to train Long-Short Term Memory (LSTM) Recurrent Neural Networks (RNNs), which led to better traffic prediction on major roads in Madrid. This approach has already been used with pollution signatures. This work addresses a new potential investigation path closely related to the use of signature profiles and Artificial Intelligent techniques as a way to reduce the specialization of sensing infrastructure.

Spam text classification using LSTM Recurrent Neural Network

Sequence Classification is one of the on-demand research projects in the field of Natural Language Processing (NLP). Classifying a set of images or text into an appropriate category or class is a complex task that a lot of Machine Learning (ML) models fail to accomplish accurately and end up under-fitting the given dataset. Some of the ML algorithms used in text classification are KNN, Naïve Bayes, Support Vector Machines, Convolutional Neural Networks (CNNs), Recursive CNNs, Recurrent Neural Networks (RNNs), Long Short Term Memory (LSTM), etc. For this experimental study, LSTM and a few other algorithms were chosen for a more comparative study. The dataset used is the SMS Spam Collection Dataset from Kaggle and 150 more entries were additionally added from different sources. Two possible class labels for the data points are spam and ham. Each entry consists of the class label, a few sentences of text followed by a few useless features that are eliminated. After converting the text to the required format, the models are run and then evaluated using various metrics. In experimental studies, the LSTM gives much better classification accuracy than the other machine learning models. F1-Scores in the high nineties were achieved using LSTM for classifying the text. The other models showed very low F1-Scores and Cosine Similarities indicating that they had underperformed on the dataset. Another interesting observation is that the LSTM had reduced the number of false positives and false negatives than any other model.

Li-ion battery state of health Prediction based on Long Short-Term Memory Recurrent Neural Network

The state of health (SOH) prediction of lithium-ion battery is essential for the health management of batteries. At present, the prediction method combined with the extraction of health indicators in charge-discharge process has received extensive attention, however, many studies ignored that the extraction of battery discharge data will be affected by the actual operating conditions, which will affect the effectiveness of health indicators extraction. In this work, a type of recurrent neural network (RNN), which is long short-term memory-RNN(LSTM-RNN), is proposed to prediction the SOH of Li-ion batteries through the data of charging process and capacity. Because the different choice of network parameters will also affect the performance of the model, particle swarm optimization (PSO) algorithm is used to optimize LSTM model. The test results show that this method can effectively predict SOH of battery, and the maximum RMSE is less than 0.01. Compared with the traditional LSTM algorithm, it has higher accuracy.

GAMPAL: an anomaly detection mechanism for Internet backbone traffic by flow size prediction with LSTM-RNN

This paper proposes a general-purpose anomaly detection mechanism for Internet backbone traffic named GAMPAL (General-purpose Anomaly detection Mechanism using Prefix Aggregate without Labeled data). GAMPAL does not require labeled data to achieve general-purpose anomaly detection. For scalability to the number of entries in the BGP RIB (Border Gateway Protocol Routing Information Base), GAMPAL introduces prefix aggregate. The BGP RIB entries are classified into prefix aggregates, each of which is identified with the first three AS (Autonomous System) numbers in the AS_PATH attribute. GAMPAL establishes a prediction model for traffic sizes based on past traffic sizes. It adopts a LSTM-RNN (Long Short-Term Memory Recurrent Neural Network) model that focuses on the periodicity of the Internet traffic patterns at a weekly scale. The validity of GAMPAL is evaluated using real traffic information, BGP RIBs exported from the WIDE backbone network (AS2500), a nationwide backbone network for research and educational organizations in Japan, and the dataset of an ISP (Internet Service Provider) in Spain. As a result, GAMPAL successfully detects anomalies such as increased traffic due to an event, DDoS (Distributed Denial of Service) attacks targeted at a stub organization, a connection failure, an SSH (Secure Shell) scan attack, and anomaly spam.

Model of the Austenite Decomposition during Cooling of the Medium Carbon Steel Using LSTM Recurrent Neural Network.

The motivation of the presented paper is the desire to create a universal tool to analyse the process of austenite decomposition during the cooling process of various steel grades. The presented analysis concerns the application of Recurrent Artificial Neural Networks (RANN) of the Long Short-Term Memory (LSTM) type for the analysis of the transition path of the cooling curve. This type of network was selected due to its ability to predict events in time sequences. The proposed generalisation allows for the determination of the austenite transformation during the continuous cooling process for various cooling curves. As training data for the neural network, values determined from the macroscopic model based on the analysis of Continuous Cooling Transformation (CCT) diagrams were used. All relations and analyses used to build training/testing or validation sets are presented in the paper. The modelling with the use of LSTM network gives the possibility to determine the incremental changes of phase transformation (in a given time step) with the assumed changes of temperature resulting from the considered cooling rate.

Recurrent neural network FPGA hardware accelerator for delay-tolerant indoor optical wireless communications.

The optical wireless communication (OWC) system has been widely studied as a promising solution for high-speed indoor applications. The transmitter diversity scheme has been proposed to improve the performance of high-speed OWC systems. However, the transmitter diversity is vulnerable to the delay of multiple channels. Recently neural networks have been studied to realize delay-tolerant indoor OWC systems, where long-short term memory (LSTM) and attention-augmented LSTM (ALSTM) recurrent neural networks (RNNs) have shown their capabilities. However, they have high computation complexity and long computation latency. In this paper, we propose a low complexity delay-tolerant RNN scheme for indoor OWC systems. In particular, an RNN with parallelized structure is proposed to reduce the computation cost. The proposed RNN schemes show comparable capability to the more complicated ALSTM, where a bit-error-rate (BER) performance within the forward-error-correction (FEC) limit is achieved for up to 5.5 symbol periods delays. In addition, previously studied LSTM/ALSTM schemes are implemented using high-end GPUs, which have high cost, high power consumption, and long processing latency. To solve these practical limitations, in this paper we further propose and demonstrate the FPGA-based RNN hardware accelerator for delay-tolerant indoor OWC systems. To optimize the processing latency and power consumption, we also propose two optimization methods: the parallel implementation with triple-phase clocking and the stream-in based computation with additive input data insertion. Results show that the FPGA-based RNN hardware accelerator with the proposed optimization methods achieves 96.75% effective latency reduction and 90.7% lower energy consumption per symbol compared with the FPGA-based RNN hardware accelerator without optimization. Compared to the GPU implementation, the latency is reduced by about 61% and the power consumption is reduced by about 58.1%.

Development of high-precision wind, wave and current forecast system for offshore wind energy industry in Taiwan: a two-stage method of numerical simulation and AI correction

ABSTRACT At the Changhwa Fuhai wind farm, the high-resolution numerical models for wind, wave and current simulations are crucial to provide daily forecasts as a decision-support system for Taiwan’s offshore wind energy industry. In this study, the two-stage method, consisting of numerical simulation and deep-learning correction based on the observed data at Fuhai wind farm site, is implemented to further improve the meteorological and marine forecast accuracy. The recurrent neural network-long short-term memory (RNN-LSTM) model is adopted in the deep learning analytics. Currently, this system is capable of daily providing the next 4-day wind field forecast and the next 7-day wave and current field forecast. Through the testing, it is found that the error correction of wind simulations can reduce root mean square error (RMSE) by up to 36.7%. Furthermore, through adopting the SQL and HBase hybrid database system, a wide range of historical data as well as daily forecast data can be quickly queried and displayed on the web interface to provide important decision support to the relevant wind power companies in scheduling offshore construction plans, forecasting wind energy production, and arranging wind turbines’ shutdown and maintenance sequence.

Day-Ahead Solar Irradiance Forecasting Using Hybrid Recurrent Neural Network with Weather Classification for Power System Scheduling

At the present time, power-system planning and management is facing the major challenge of integrating renewable energy resources (RESs) due to their intermittent nature. To address this problem, a highly accurate renewable energy generation forecasting system is needed for day-ahead power generation scheduling. Day-ahead solar irradiance (SI) forecasting has various applications for system operators and market agents such as unit commitment, reserve management, and biding in the day-ahead market. To this end, a hybrid recurrent neural network is presented herein that uses the long short-term memory recurrent neural network (LSTM-RNN) approach to forecast day-ahead SI. In this approach, k-means clustering is first used to classify each day as either sunny or cloudy. Then, LSTM-RNN is used to learn the uncertainty and variability for each type of cluster separately to predict the SI with better accuracy. The exogenous features such as the dry-bulb temperature, dew point temperature, and relative humidity are used to train the models. Results show that the proposed hybrid model has performed better than a feed-forward neural network (FFNN), a support vector machine (SVM), a conventional LSTM-RNN, and a persistence model.