Long Short-term Memory Recurrent Neural Network Research Articles

Inter-Sentence Segmentation of YouTube Subtitles Using Long-Short Term Memory (LSTM)

Recently, with the development of Speech to Text, which converts voice to text, and machine translation, technologies for simultaneously translating the captions of video into other languages have been developed. Using this, YouTube, a video-sharing site, provides captions in many languages. Currently, the automatic caption system extracts voice data when uploading a video and provides a subtitle file converted into text. This method creates subtitles suitable for the running time. However, when extracting subtitles from video using Speech to Text, it is impossible to accurately translate the sentence because all sentences are generated without periods. Since the generated subtitles are separated by time units rather than sentence units, and are translated, it is very difficult to understand the translation result as a whole. In this paper, we propose a method to divide text into sentences and generate period marks to improve the accuracy of automatic translation of English subtitles. For this study, we use the 27,826 sentence subtitles provided by Stanford University’s courses as data. Since this lecture video provides complete sentence caption data, it can be used as training data by transforming the subtitles into general YouTube-like caption data. We build a model with the training data using the LSTM-RNN (Long-Short Term Memory – Recurrent Neural Networks) and predict the position of the period mark, resulting in prediction accuracy of 70.84%. Our research will provide people with more accurate translations of subtitles. In addition, we expect that language barriers in online education will be more easily broken by achieving more accurate translations of numerous video lectures in English.

Real-Time Crash Risk Prediction using Long Short-Term Memory Recurrent Neural Network

With the help of traffic detectors widely deployed along arterial roads and intersections, real-time traffic data are collected and updated in a very short time period, which makes it possible to conduct real-time analysis at signalized intersections. Among them, real-time crash risk prediction is one of the most promising and challenging research topics. This study attempts to predict real-time crash risk by considering time series dependency with the employment of a long short-term memory recurrent neural network (LSTM-RNN) algorithm. Also, the synthetic minority over-sampling technique (SMOTE) was utilized in this study to generate a balanced training dataset for algorithm training. In comparison, a conditional logistic model was developed based on matched case control design. Both models were evaluated based on the real-world unbalanced test dataset rather than an artificially balanced dataset. The comparison results indicate that the LSTM-RNN with SMOTE outperforms the conditional logistic model. The methods and findings of this study attempt to verify the feasibility of real-time crash risk prediction by using LSTM-RNN with over-sampled dataset (SMOTE).

Twitter User’s Hobby Estimation Based on Sequential Statements Using Deep Neural Networks

With more and more frequency, users communicate with each other on social media. Many users start on Twitter or Facebook to find friends who have the same hobby. Our study proposes a method to estimate the users’ interests (hobby) based on tweets on Twitter. One tweet does not, in and of itself, contain a lot of information, and some tweets are not related to the user’s hobby. Therefore, we propose a reliable hobby estimation method by extracting features from multiple, sequential tweets. The proposed method uses Recurrent Neural Networks (RNN) which can accommodate time-series information. We also used a Convolutional Neural Networks (CNN) which can treat contextual information. We used an averaged vector of word distributed representation as a feature. Using the proposed method based on Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN), we obtained a 23.72% improvement as compared with a baseline method using a Random Forest (RF) regression as a machine learning algorithm.

Machine Health Monitoring Using Adaptive Kernel Spectral Clustering and Deep Long Short-Term Memory Recurrent Neural Networks

Machine health monitoring is of great importance in industrial informatics field. Recently, deep learning methods applied to machine health monitoring have been proven effective. However, the existing methods face enormous difficulties in extracting heterogeneous features indicating the variation until failure and revealing the inherent high-dimensional features of massive signals, which affect the accuracy and efficiency of machine health monitoring. In this paper, a novel data-driven machine health monitoring method is proposed using adaptive kernel spectral clustering (AKSC) and deep long short-term memory recurrent neural networks (LSTM-RNN). This method include three steps: First, features in the time domain, frequency domain, and time-frequency domain are, respectively, extracted from massive measured signals. And, an Euclidean distance based algorithm is designed to select degradation features. Second, the AKSC algorithm is introduced to adaptively identify machine anomaly behaviors from multiple degradation features. Third, a new deep learning model (LSTM-RNN) is constructed to update and predict the failure time of the machine. The effectiveness of the proposed method is validated using a set of test-to-failure experimental data. The results show that the performance of the proposed method is competitive with other existing methods.

Spatial granularity analysis on electricity consumption prediction using LSTM recurrent neural network

The building sector takes a large proportion of electricity consumption and carbon emission in high-density urban areas. To reduce the carbon emissions and use energy more efficiently in the building sector, home energy management system (HEMS) is proposed and used. In the HEMS, the prediction of electricity consumption in the short-term future is used to support the decision makings in the HEMS. Although there existed a number of studies in the prediction of electricity consumption in buildings, there lacks a spatial analysis in the prediction performance, especially on the appliance or sub-meter level and household level. The authors made an assumption that by the performance of household energy consumption prediction can be significantly improved if the prediction is aggregated from the prediction data at the appliance or sub-meter level. Next, two typical datasets are used to validate the assumption by comparing the prediction performance of aggregating the prediction data at appliance level and the one of making direct prediction at the household level. The models used for the prediction are standard stateful long short-term memory (LSTM) neural networks, which have been proofed to be promising in load prediction by previous studies. The results from the comparison validated the assumption, showing that the prediction performance can be significantly improved if prediction is made at the appliance-level first and then aggregated to get the household-level prediction. Therefore, the authors conclude that prediction at the finer appliance granularity level can significantly improve the performance of household-level electricity prediction.

A Hybrid Auto scaling for Cloud applications using an Efficient Predictive technique in Private Cloud

Background/Objective: To provide an efficient predictive technique to foresee future workload as well as to handle the resources efficiently by performing hybrid auto scaling for Cloud applications. Cloud applications might expertise completely different workload at different times, automatic provisioning has to work with efficiency at any point of time. Auto scaling is a feature of cloud computing that potentially scale the resources in line on demand. Considering this expectation, they are generally categorized into Reactive scaling which adds or reduces resources based on a fixed threshold value. The predictive scaling is used provide necessary scaling actions beforehand. Methods/Statistical Analysis: To perform the hybrid auto scaling (reactive plus predictive auto scaling), a time series technique should be used. Auto-regressive Moving Average (ARMA) model, the Exponential Smoothing (ES) model, the Autoregressive model (AR), the Moving Average model (MA) and the Trend- Adjusted Exponential Smoothing (TAES), Auto Regressive Integrated Moving Average (ARIMA) Time-series model, Naïve bayes algorithm, Recurrent Neural Network- Long Short Term Memory (RNN-LSTM), Independent Recurrent Neural Network (IndRNN) are time series techniques used to foresee the future workload. To find the effectiveness of predictive techniques, Mean Absolute Error (MAE), Mean Squared Error (MSE) and Mean Absolute Percentage Error (MAPE) performance metrics are evaluated. Findings: Based on the evaluation, IndRNN gives the minimum error rate. IndRNN is used to predict the future resource requisites in order to ascertain adequate resource are available ahead of time. Application: The predicted result from IndRNN method is integrated on private cloud to autoscale the resources for cloud applications. Keywords: Cloud Applications, Hybrid Autoscaling, Independent Recurrent Neural Network (IndRNN), Private Cloud, Workload

Cloud based health monitoring and abnormality detection using smart mirror

Background/Objective: To provide an efficient predictive technique to foresee future workload as well as to handle the resources efficiently by performing hybrid auto scaling for Cloud applications. Cloud applications might expertise completely different workload at different times, automatic provisioning has to work with efficiency at any point of time. Auto scaling is a feature of cloud computing that potentially scale the resources in line on demand. Considering this expectation, they are generally categorized into Reactive scaling which adds or reduces resources based on a fixed threshold value. The predictive scaling is used provide necessary scaling actions beforehand. Methods/Statistical Analysis: To perform the hybrid auto scaling (reactive plus predictive auto scaling), a time series technique should be used. Auto-regressive Moving Average (ARMA) model, the Exponential Smoothing (ES) model, the Autoregressive model (AR), the Moving Average model (MA) and the Trend- Adjusted Exponential Smoothing (TAES), Auto Regressive Integrated Moving Average (ARIMA) Time-series model, Naïve bayes algorithm, Recurrent Neural Network- Long Short Term Memory (RNN-LSTM), Independent Recurrent Neural Network (IndRNN) are time series techniques used to foresee the future workload. To find the effectiveness of predictive techniques, Mean Absolute Error (MAE), Mean Squared Error (MSE) and Mean Absolute Percentage Error (MAPE) performance metrics are evaluated. Findings: Based on the evaluation, IndRNN gives the minimum error rate. IndRNN is used to predict the future resource requisites in order to ascertain adequate resource are available ahead of time. Application: The predicted result from IndRNN method is integrated on private cloud to autoscale the resources for cloud applications. Keywords: Abnormality, Cloud, Health Monitoring, Raspberry Pi, Sensor, Smart Mirror, Two Way Mirror

Cascaded LSTM recurrent neural network for automated sleep stage classification using single-channel EEG signals

Automated evaluation of a subject's neurocognitive performance (NCP) is a relevant topic in neurological and clinical studies. NCP represents the mental/cognitive human capacity in performing a specific task. It is difficult to develop the study protocols as the subject's NCP changes in a known predictable way. Sleep is time-varying NCP and can be used to develop novel NCP techniques. Accurate analysis and interpretation of human sleep electroencephalographic (EEG) signals is needed for proper NCP assessment. In addition, sleep deprivation may cause prominent cognitive risks in performing many common activities such as driving or controlling a generic device; therefore, sleep scoring is a crucial part of the process. In the sleep cycle, the first stage of non-rapid eye movement (NREM) sleep or stage N1 is the transition between wakefulness and drowsiness and becomes relevant for the study of NCP.In this study, a novel cascaded recurrent neural network (RNN) architecture based on long short-term memory (LSTM) blocks, is proposed for the automated scoring of sleep stages using EEG signals derived from a single-channel. Fifty-five time and frequency-domain features were extracted from the EEG signals and fed to feature reduction algorithms to select the most relevant ones. The selected features constituted as the inputs to the LSTM networks. The cascaded architecture is composed of two LSTM RNNs: the first network performed 4-class classification (i.e. the five sleep stages with the merging of stages N1 and REM into a single stage) with a classification rate of 90.8%, and the second one obtained a recognition performance of 83.6% for 2-class classification (i.e. N1 vs REM). The overall percentage of correct classification for five sleep stages is found to be 86.7%. The objective of this work is to improve classification performance in sleep stage N1, as a first step of NCP assessment, and at the same time obtain satisfactory classification results in the other sleep stages.

Addressing noise and pitch sensitivity of speech recognition system through variational mode decomposition based spectral smoothing

In this paper, we propose a novel front-end speech parameterization technique for automatic speech recognition (ASR) that is less sensitive towards ambient noise and pitch variations. First, using variational mode decomposition (VMD), we break up the short-time magnitude spectrum obtained by discrete Fourier transform into several components. In order to suppress the ill-effects of noise and pitch variations, the spectrum is then sufficiently smoothed. The desired spectral smoothing is achieved by discarding the higher-order variational mode functions and reconstructing the spectrum using the first-two modes only. As a result, the smoothed spectrum closely resembles the spectral envelope. Next, the Mel-frequency cepstral coefficients (MFCC) are extracted using the VMD-based smoothed spectra. The proposed front-end acoustic features are observed to be more robust towards ambient noise and pitch variations than the conventional MFCC features as demonstrated by the experimental evaluations presented in this study. For this purpose, we developed an ASR system using speech data from adult speakers collected under relatively clean recording conditions. State-of-the-art acoustic modeling techniques based on deep neural networks (DNN) and long short-term memory recurrent neural networks (LSTM-RNN) were employed. The ASR systems were then evaluated under noisy test conditions for assessing the noise robustness of the proposed features. To assess robustness towards pitch variations, experimental evaluations were performed on another test set consisting of speech data from child speakers. Transcribing children's speech helps in simulating an ASR task where pitch differences between training and test data are significantly large. The signal domain analyses as well as the experimental evaluations presented in this paper support our claims.

Deep learning PM2.5 concentrations with bidirectional LSTM RNN

A better understanding of spatiotemporal distribution of PM2.5 (particulate matter with diameter less than 2.5 micrometer) concentrations in a continuous space-time domain is critical for risk assessment and epidemiologic studies. Existing spatiotemporal interpolation algorithms are usually based on strong assumptions by restricting the interpolation models to the ones with explicit and simple mathematical descriptions, thus neglecting plenty of hidden yet critical influencing factors. In this study, we developed a novel deep-learning-based spatiotemporal interpolation model, which includes the bidirectional Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN) as the main ingredient. Our model is able to take into account both spatial and temporal hidden influencing factors automatically. To the best of our knowledge, it is the first time of applying the bidirectional LSTM RNN in the spatiotemporal interpolation of air pollutants concentrations. We evaluated our novel method using a dataset that contains daily PM2.5 measurements in 2009 over the contiguous southeast region of the USA. Results demonstrate a good performance of our model. We also conducted simulations to explore the properties of spatiotemporal correlations. In particular, we found the temporal correlation is stronger than the spatial correlation.

Sequence-to-sequence Bangla Sentence Generation with LSTM Recurrent Neural Networks

Sequence to sequence text generation is the most efficient approach for automatically converting the script of a word from a source sequence to a target sequence. Text generation is the application of natural language generation which is useful in sequence modeling like the machine translation, speech recognition, image captioning, language identification, video captioning and much more. In this paper we have discussed about Bangla text generation, using deep learning approach, Long Short-term Memory (LSTM), a special kind of RNN (Recurrent Neural Network). LSTM networks are suitable for analyzing sequences of text data and predicting the next word. LSTM could be a respectable solution if you want to predict the very next point of a given time sequence. In this article we proposed a artificial Bangla Text Generator with LSTM, which is very early for this language and also this model is validated with satisfactory accuracy rate.

Short-Term Residential Load Forecasting Based on LSTM Recurrent Neural Network

As the power system is facing a transition toward a more intelligent, flexible, and interactive system with higher penetration of renewable energy generation, load forecasting, especially short-term load forecasting for individual electric customers plays an increasingly essential role in the future grid planning and operation. Other than aggregated residential load in a large scale, forecasting an electric load of a single energy user is fairly challenging due to the high volatility and uncertainty involved. In this paper, we propose a long short-term memory (LSTM) recurrent neural network-based framework, which is the latest and one of the most popular techniques of deep learning, to tackle this tricky issue. The proposed framework is tested on a publicly available set of real residential smart meter data, of which the performance is comprehensively compared to various benchmarks including the state-of-the-arts in the field of load forecasting. As a result, the proposed LSTM approach outperforms the other listed rival algorithms in the task of short-term load forecasting for individual residential households.

A MEMS Gyroscope Noise Suppressing Method Using Neural Architecture Search Neural Network

Inertial measurement unit (IMU) (an IMU usually contains three gyroscopes and accelerometers) is the key sensor to construct a self‐contained inertial navigation system (INS). IMU manufactured through the Micromechanics Electronics Manufacturing System (MEMS) technology becomes more popular, due to its smaller column, lower cost, and gradually improved accuracy. However, limited by the manufacturing technology, the MEMS IMU raw measurement signals experience complicated noises, which cause the INS navigation solution errors diverge dramatically over time. For addressing this problem, an advanced Neural Architecture Search Recurrent Neural Network (NAS‐RNN) was employed in the MEMS gyroscope noise suppressing. NAS‐RNN was the recently invented artificial intelligence method for time series problems in data science community. Different from conventional method, NAS‐RNN was able to search a more feasible architecture for selected application. In this paper, a popular MEMS IMU STIM300 was employed in the testing experiment, and the sampling frequency was 125 Hz. The experiment results showed that the NAS‐RNN was effective for MEMS gyroscope denoising; the standard deviation values of denoised three‐axis gyroscope measurements decreased by 44.0%, 34.1%, and 39.3%, respectively. Compared with the Long Short‐Term Memory Recurrent Neural Network (LSTM‐RNN), the NAS‐RNN obtained further decreases by 28.6%, 3.7%, and 8.8% in standard deviation (STD) values of the signals. In addition, the attitude errors decreased by 26.5%, 20.8%, and 16.4% while substituting the LSTM‐RNN with the NAS‐RNN.

Acceleration of LSTM With Structured Pruning Method on FPGA

This paper focuses on accelerating long short-term memory (LSTM), which is one of the popular types of recurrent neural networks (RNNs). Because of the large number of weight memory accesses and high computation complexity with the cascade-dependent structure, it is a big challenge to efficiently implement the LSTM on field-programmable gate arrays (FPGAs). To speed up the inference on FPGA, considering its limited resource, a structured pruning method that can not only reduce the LSTM model's size without loss of prediction accuracy but also eliminate the imbalance computation and irregular memory accesses is proposed. Besides that, the hardware architecture of the compressed LSTM is designed to pursue high performance. As a result, the implementation of an LSTM language module on Stratix V GXA7 FPGA can achieve 85.2 GOPS directly on the sparse LSTM network by our method, corresponding to 681.6-GOPS effective throughput on the dense one, which shows that the proposed structured pruning algorithm makes 7.82 times speedup when only 1/8 parameters are reserved. We hope that our method can give an efficient way to accelerate the LSTM and similar recurrent neural networks when the resource-limited environment is emphasized.

Multi-layer Long Short-term Memory based Condenser Vacuum Degree Prediction Model on Power Plant

A multi-layer LSTM (Long short-term memory) model is proposed for condenser vacuum degree prediction of power plants. Firstly, Min-max normalization is used to pre-process the input data. Then, the model proposes the two-layer LSTM architecture to identify the time series pattern effectively. ADAM（Adaptive moment）optimizer is selected to find the optimum parameters for the model during training. Under the proposed forecasting framework, experiments illustrates that the two-layer LSTM model can give a more accurate forecast to the condenser vacuum degree compared with other simple RNN (Recurrent Neural Network) and one-layer LSTM model.

Robust Feature Selection With LSTM Recurrent Neural Networks for Artificial Immune Recognition System

Stability and robustness of feature selection techniques have great importance in the high dimensional and small sample data. The neglected subject in the feature selection is solving the instability problem. Therefore, an ensemble gene selection framework is used in order to provide stable and accurate results of feature selection algorithms. Sequence modeling from high-dimensional data is an important research area for the discovery of biomarkers. Identifying biomarkers requires robust gene selection methods, which makes it possible to find important tumor-related genes with high accuracy. The main issue of this paper is creating a model in order to learn long sequences with the artificial immune recognition system (AIRS) for robust feature selection. Long short-term memory (LSTM) recurrent neural networks are trained with the AIRS in order to obtain the long-lived unit cells for use in the feature selection process. LSTM was used to be better understanding the mechanisms involving the “remember” feature of the immunological behavior of the immune response. We tried to apply a theory suggested by immunologists in order to develop stable associative memory, which capable of solving robustness and optimization tasks. We examined the initial gene selection step based on the different types of group formation algorithm for analysis of the most informative selected features. Microarray datasets are showing remarkable increases in their robustness and classification accuracy. The suggested framework is evaluated on six commonly used microarray datasets.

Bearing performance degradation assessment using long short-term memory recurrent network

Bearing is commonly used in rotating machinery, and it is significant to monitor bearing running states to ensure machine safety. Performance degradation assessment is an important work in bearing condition-based maintenance (CBM) and predictive maintenance. In this paper, a data-driven bearing performance degradation assessment method based on long short-term memory (LSTM) recurrent neural network (RNN) is proposed to comprehensively utilize the fault propagation information. Firstly, universal degradation simulation model based on vibration response mechanism is constructed for feature verification. A new proposed indicator "waveform entropy (WFE)" is developed and validated by this simulation model. Then waveform entropy and other conventional indicators are input into the LSTM RNN to identify the bearing running state, while particle swarm optimization method is applied to optimize the network structure parameters simultaneously. Experimental results demonstrate that LSTM RNN can effectively identify the bearing degradation states and accurately predict the remaining useful life.

Modeling Speech Acts in Asynchronous Conversations: A Neural-CRF Approach

Participants in an asynchronous conversation (e.g., forum, e-mail) interact with each other at different times, performing certain communicative acts, called speech acts (e.g., question, request). In this article, we propose a hybrid approach to speech act recognition in asynchronous conversations. Our approach works in two main steps: a long short-term memory recurrent neural network (LSTM-RNN) first encodes each sentence separately into a task-specific distributed representation, and this is then used in a conditional random field (CRF) model to capture the conversational dependencies between sentences. The LSTM-RNN model uses pretrained word embeddings learned from a large conversational corpus and is trained to classify sentences into speech act types. The CRF model can consider arbitrary graph structures to model conversational dependencies in an asynchronous conversation. In addition, to mitigate the problem of limited annotated data in the asynchronous domains, we adapt the LSTM-RNN model to learn from synchronous conversations (e.g., meetings), using domain adversarial training of neural networks. Empirical evaluation shows the effectiveness of our approach over existing ones: (i) LSTM-RNNs provide better task-specific representations, (ii) conversational word embeddings benefit the LSTM-RNNs more than the off-the-shelf ones, (iii) adversarial training gives better domain-invariant representations, and (iv) the global CRF model improves over local models.

Short‐Term Precipitation Forecast Based on the PERSIANN System and LSTM Recurrent Neural Networks

AbstractShort‐term Quantitative Precipitation Forecasting is important for flood forecasting, early flood warning, and natural hazard management. This study proposes a precipitation forecast model by extrapolating Cloud‐Top Brightness Temperature (CTBT) using advanced Deep Neural Networks, and applying the forecasted CTBT into an effective rainfall retrieval algorithm to obtain the Short‐term Quantitative Precipitation Forecasting (0–6 hr). To achieve such tasks, we propose a Long Short‐Term Memory (LSTM) and the Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks (PERSIANN), respectively. The precipitation forecasts obtained from our proposed framework, (i.e., LSTM combined with PERSIANN) are compared with a Recurrent Neural Network (RNN), Persistency method, and Farneback optical flow each combined with PERSIANN algorithm and the numerical model results from the first version of Rapid Refresh (RAPv1.0) over three regions in the United States, including the states of Oregon, Oklahoma, and Florida. Our experiments indicate better statistics, such as correlation coefficient and root‐mean‐square error, for the CTBT forecasts from the proposed LSTM compared to the RNN, Persistency, and the Farneback method. The precipitation forecasts from the proposed LSTM and PERSIANN framework has demonstrated better statistics compared to the RAPv1.0 numerical forecasts and PERSIANN estimations from RNN, Persistency, and Farneback projections in terms of Probability of Detection, False Alarm Ratio, Critical Success Index, correlation coefficient, and root‐mean‐square error, especially in predicting the convective rainfalls. The proposed method shows superior capabilities in short‐term forecasting over compared methods, and has the potential to be implemented globally as an alternative short‐term forecast product.

Remaining useful life prediction of PEMFC based on long short-term memory recurrent neural networks

To solve the prediction problem of proton exchange membrane fuel cell (PEMFC) remaining useful life (RUL), a novel RUL prediction approach of PEMFC based on long short-term memory (LSTM) recurrent neural networks (RNN) has been developed. The method uses regular interval sampling and locally weighted scatterplot smoothing (LOESS) to realize data reconstruction and data smoothing. Not only the primary trend of the original data can be preserved, but noise and spikes can be effectively removed. The LSTM RNN is adopted to estimate the remaining life of test data. 1154-hour experimental aging analysis of PEMFC shows that the prediction accuracy of the novel method is 99.23%, the root mean square error (RMSE) and mean absolute error (MAE) is 0.003 and 0.0026 respectively. The comparison analysis shows that the prediction accuracy of the novel method is 28.46% higher than that of back propagation neural network (BPNN). Root mean square error, relative error (RE) and mean absolute error are all much smaller than that of BPNN. Therefore, the novel method can quickly and accurately forecast the residual service life of the fuel cell.