Long Short-term Memory Recurrent Neural Network Research Articles

Multiscale LSTM-Based Deep Learning for Very-Short-Term Photovoltaic Power Generation Forecasting in Smart City Energy Management

Photovoltaic power generation forecasting (PVGF) is an attractive research topic for efficient energy management in smart city. In addition, the long short-term memory recurrent neural network (LSTM/RNN) has been actively utilized for predicting various time series tasks in recent years due to its outstanding ability to learn the feature of sequential time-series data. Although the existing forecasting models were obtained from learning the sequential PVGF data, it is observed that irregular factors made adverse effects on the forecasting results of very-short-term PVGF tasks, thus, the entire forecasting performance was deteriorated. In this regard, multiscale LSTM-based deep learning which is capable for forecasting very-short-term PVGF is proposed for efficient management. The model concatenates on two different scaled LSTM modules to overcome the deterioration that is originated from the irregular factors. Lastly, experimental results present the proposed framework can assist to forecast the tendency of PVGF amount steadily.

Riboflow: Using Deep Learning to Classify Riboswitches With ∼99% Accuracy.

Riboswitches are cis-regulatory genetic elements that use an aptamer to control gene expression. Specificity to cognate ligand and diversity of such ligands have expanded the functional repertoire of riboswitches to mediate mounting apt responses to sudden metabolic demands and signal changes in environmental conditions. Given their critical role in microbial life, riboswitch characterisation remains a challenging computational problem. Here we have addressed the issue with advanced deep learning frameworks, namely convolutional neural networks (CNN), and bidirectional recurrent neural networks (RNN) with Long Short-Term Memory (LSTM). Using a comprehensive dataset of 32 ligand classes and a stratified train-validate-test approach, we demonstrated the accurate performance of both the deep learning models (CNN and RNN) relative to conventional hyperparameter-optimized machine learning classifiers on all key performance metrics, including the ROC curve analysis. In particular, the bidirectional LSTM RNN emerged as the best-performing learning method for identifying the ligand-specificity of riboswitches with an accuracy >0.99 and macro-averaged F-score of 0.96. An additional attraction is that the deep learning models do not require prior feature engineering. A dynamic update functionality is built into the models to factor for the constant discovery of new riboswitches, and extend the predictive modeling to new classes. Our work would enable the design of genetic circuits with custom-tuned riboswitch aptamers that would effect precise translational control in synthetic biology. The associated software is available as an open-source Python package and standalone resource for use in genome annotation, synthetic biology, and biotechnology workflows.

Deep learning-based BCI for gait decoding from EEG with LSTM recurrent neural network

Objective. Mobile Brain/Body Imaging (MoBI) frameworks allowed the research community to find evidence of cortical involvement at walking initiation and during locomotion. However, the decoding of gait patterns from brain signals remains an open challenge. The aim of this work is to propose and validate a deep learning model to decode gait phases from Electroenchephalography (EEG). Approach. A Long-Short Term Memory (LSTM) deep neural network has been trained to deal with time-dependent information within brain signals during locomotion. The EEG signals have been preprocessed by means of Artifacts Subspace Reconstruction (ASR) and Reliable Independent Component Analysis (RELICA) to ensure that classification performance was not affected by movement-related artifacts. Main results. The network was evaluated on the dataset of 11 healthy subjects walking on a treadmill. The proposed decoding approach shows a robust reconstruction (AUC > 90%) of gait patterns (i.e. swing and stance states) of both legs together, or of each leg independently. Significance. Our results support for the first time the use of a memory-based deep learning classifier to decode walking activity from non-invasive brain recordings. We suggest that this classifier, exploited in real time, can be a more effective input for devices restoring locomotion in impaired people.

Improving Road Traffic Forecasting Using Air Pollution and Atmospheric Data: Experiments Based on LSTM Recurrent Neural Networks.

Traffic flow forecasting is one of the most important use cases related to smart cities. In addition to assisting traffic management authorities, traffic forecasting can help drivers to choose the best path to their destinations. Accurate traffic forecasting is a basic requirement for traffic management. We propose a traffic forecasting approach that utilizes air pollution and atmospheric parameters. Air pollution levels are often associated with traffic intensity, and much work is already available in which air pollution has been predicted using road traffic. However, to the best of our knowledge, an attempt to improve forecasting road traffic using air pollution and atmospheric parameters is not yet available in the literature. In our preliminary experiments, we found out the relation between traffic intensity, air pollution, and atmospheric parameters. Therefore, we believe that addition of air pollutants and atmospheric parameters can improve the traffic forecasting. Our method uses air pollution gases, including and . We chose these gases because they are associated with road traffic. Some atmospheric parameters, including pressure, temperature, wind direction, and wind speed have also been considered, as these parameters can play an important role in the dispersion of the above-mentioned gases. Data related to traffic flow, air pollution, and the atmosphere were collected from the open data portal of Madrid, Spain. The long short-term memory (LSTM) recurrent neural network (RNN) was used in this paper to perform traffic forecasting.

An 8.93 TOPS/W LSTM Recurrent Neural Network Accelerator Featuring Hierarchical Coarse-Grain Sparsity for On-Device Speech Recognition

Long short-term memory (LSTM) is a type of recurrent neural networks (RNNs), which is widely used for time-series data and speech applications, due to its high accuracy on such tasks. However, LSTMs pose difficulties for efficient hardware implementation because they require a large amount of weight storage and exhibit computation complexity. Prior works have proposed compression techniques to alleviate the storage/computation requirements of LSTMs but elementwise sparsity schemes incur sizable index memory overhead and structured compression techniques report limited compression ratios. In this article, we present an energy-efficient LSTM RNN accelerator, featuring an algorithm-hardware co-optimized memory compression technique called hierarchical coarse-grain sparsity (HCGS). Aided by the HCGS-based blockwise recursive weight compression, we demonstrate LSTM networks with up to 16 $\times $ fewer weights while achieving minimal error rate degradation. The prototype chip fabricated in 65-nm LP CMOS achieves up to 8.93 TOPS/W for real-time speech recognition using compressed LSTMs based on HCGS. HCGS-based LSTMs have demonstrated energy-efficient speech recognition with low error rates for TIMIT, TED-LIUM, and LibriSpeech data sets.

Forecasting of solar and wind power using LSTM RNN for load frequency control in isolated microgrid

ABSTRACT Renewable sources such as solar PV and wind are stochastic in nature, hence their integration with emerging isolated microgrid (MG) is challenging especially with regards to stability issues. An accurate prediction model of wind and solar sources is necessary to analyze the uncertainty in MG system and to encourage the reliable participation of wind and solar power in the energy market. The advancement in deep learning methods has made it possible to develop a multi-step forecasting model unlike shallow neural networks (SNNs). The time series forecasting using SNN and Recurrent Neural Network (RNN) suffers from the problem of vanishing/exploding gradient while training. To eliminate this problem the long short-term memory (LSTM) RNN has been used in this study for wind speed and solar irradiance prediction. The forecasted solar and wind power is applied to analyze the load frequency behavior and the response of nonrenewable sources for sudden rise and fall in load power demand and PI controller is used to mitigate frequency deviation to ensure the stability of the MG power system.

Link quality prediction in wireless community networks using deep recurrent neural networks

Wireless community networks (WCNs) are large, heterogeneous, dynamic, and decentralized networks. Such complex characteristics raise different challenges, such as the effect of wireless communications on the performance of networks and routing protocols. The prediction approaches of link quality (LQ) can improve the performance of routing algorithms of WCNs while avoiding weak links. The prediction of LQ in WCNs can be a complex task because of the fluctuated nature of LQ measurements due to the dynamic wireless environment. In this paper, a deep learning based approach is proposed to accurately predict LQ in WCNs. Specifically, we propose the use of two variants of deep recurrent neural network (RNN): long short-term memory recurrent neural networks (LSTM-RNN) and gated recurrent unit (GRU). The positive feature of the proposed variants is that they can handle the fluctuating nature of LQ due to their ability to learn and exploit the context in LQ time-series. The experimental results on data collected from a real-world WCN show that the proposed LSTM-RNN and GRU models accurately predict LQ in WCNs compared to related methods. The proposed approach could be a helpful tool for accurately predicting LQ, thereby improving the performance of routing protocols of WCNs.

Text-independent speaker recognition using LSTM-RNN and speech enhancement

Speaker recognition revolution has lead to the inclusion of speaker recognition modules in several commercial products. Most published algorithms for speaker recognition focus on text-dependent speaker recognition. In contrast, text-independent speaker recognition is more advantageous as the client can talk freely to the system. In this paper, text-independent speaker recognition is considered in the presence of some degradation effects such as noise and reverberation. Mel-Frequency Cepstral Coefficients (MFCCs), spectrum and log-spectrum are used for feature extraction from the speech signals. These features are processed with the Long-Short Term Memory Recurrent Neural Network (LSTM-RNN) as a classification tool to complete the speaker recognition task. The network learns to recognize the speakers efficiently in a text-independent manner, when the recording circumstances are the same. The recognition rate reaches 95.33% using MFCCs, while it is increased to 98.7% when using spectrum or log-spectrum. However, the system has some challenges to recognize speakers from different recording environments. Hence, different speech enhancement techniques, such as spectral subtraction and wavelet denoising, are used to improve the recognition performance to some extent. The proposed approach shows superiority, when compared to the algorithm of R. Togneri and D. Pullella (2011).

A new methodology for kick detection during petroleum drilling using long short-term memory recurrent neural network

Kick is a downhole phenomenon which can lead to blowout, and so early detection is important. In addition to early detection, the need to prevent false alarm is also useful in order to minimize wastage of operation time. A major challenge in ensuring early detection is that it increases the chances of false alarm. While several data-driven approaches have been used in the past, there is also ongoing research on the use of derived indicators such as d-exponent for kick detection. This article presents a data-driven approach which uses d-exponent and standpipe pressure for kick detection. The data-driven approach presented in this article serves as a complementary methodology to other stand-alone kick detection equations, and uses d-exponent and standpipe pressure as inputs.This paper proposes a methodology which uses the long short-term memory recurrent neural network (LSTM-RNN) to capture temporal relationships between time series data comprising of d-exponent data and standpipe pressure data with the aim of increasing the chances of achieving early kick detection without false alarm. The methodology involves obtaining the slope trend of the d-exponent data and the peak reduction in the standpipe pressure data for training the LSTM-RNN for kick detection. Field data is used for training and testing. Early detection was achieved without false alarm.

Predicting optical spectra for optoelectronic polymers using coarse-grained models and recurrent neural networks

Coarse-grained modeling of conjugated polymers has become an increasingly popular route to investigate the physics of organic optoelectronic materials. While ultraviolet (UV)-vis spectroscopy remains one of the key experimental methods for the interrogation of these materials, a rigorous bridge between simulated coarse-grained structures and spectroscopy has not been established. Here, we address this challenge by developing a method that can predict spectra of conjugated polymers directly from coarse-grained representations while avoiding repetitive procedures such as ad hoc back-mapping from coarse-grained to atomistic representations followed by spectral computation using quantum chemistry. Our approach is based on a generative deep-learning model: the long-short-term memory recurrent neural network (LSTM-RNN). The latter is suggested by the apparent similarity between natural languages and the mathematical structure of perturbative expansions of, in our case, excited-state energies perturbed by conformational fluctuations. We also use this model to explore the level of sensitivity of spectra to the coarse-grained representation back-mapping protocol. Our approach presents a tool uniquely suited for improving postsimulation analysis protocols, as well as, potentially, for including spectral data as input in the refinement of coarse-grained potentials.

Attention-Based SeriesNet: An Attention-Based Hybrid Neural Network Model for Conditional Time Series Forecasting

Traditional time series forecasting techniques can not extract good enough sequence data features, and their accuracies are limited. The deep learning structure SeriesNet is an advanced method, which adopts hybrid neural networks, including dilated causal convolutional neural network (DC-CNN) and Long-short term memory recurrent neural network (LSTM-RNN), to learn multi-range and multi-level features from multi-conditional time series with higher accuracy. However, they didn’t consider the attention mechanisms to learn temporal features. Besides, the conditioning method for CNN and RNN is not specific, and the number of parameters in each layer is tremendous. This paper proposes the conditioning method for two types of neural networks, and respectively uses the gated recurrent unit network (GRU) and the dilated depthwise separable temporal convolutional networks (DDSTCNs) instead of LSTM and DC-CNN for reducing the parameters. Furthermore, this paper presents the lightweight RNN-based hidden state attention module (HSAM) combined with the proposed CNN-based convolutional block attention module (CBAM) for time series forecasting. Experimental results show our model is superior to other models from the viewpoint of forecasting accuracy and computation efficiency.

A Statistical Data-Filtering Method Proposed for Short-Term Load Forecasting Models

Reliability assessment of the SCADA-system based load data is necessary for improving accuracy of short-term load forecasting (STLF) methods in a distribution network (DN). Specifically, the reliability evaluation of the load data is to properly eliminate noise/outliers caused by random power consumption behaviors or the sudden change in load demand from industrial and residential customers in the DN. Thus, this paper proposes a novel statistical data-filtering method, working at an input data pre-processing stage, which will evaluate the reliability of input load data by analyzing all possible data confidence levels in order to filter-out the noise/outliers for accuracy improvement of different short-term load forecasting models. The proposed statistical data-filtering method is also compared to other existing data-filtering methods (such as Kalman Filter, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), Discrete Wavelet Transform (DWT) and Singular Spectrum Analysis (SSA)). Moreover, several case studies of short-term load forecasting for a typical 22 kV distribution network in Vietnam are conducted with an Artificial Neural Network (ANN) model, a Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) model, a combined model of Long Short-Term Memory Network and Convolutional Neural Network (LSTM-CNN), and a conventional Autoregressive Integrated Moving Average (ARIMA) model to validate the statistical data-filtering method proposed. The achieved results demonstrate which the STLF using ANN, LSTM-RNN, LSTM-CNN, and ARIMA models with the statistical data-filtering method can all outperform those with the existing data-filtering methods. Additionally, the numerical results also indicate that in case the SCADA-based load data is normally distributed, time-series forecasting models should be more preferred than neural network models; otherwise, when the SCADA-based load data contains multiple normally distributed sub-datasets, neural network-based prediction models are highly recommended.

Deep learning spatiotemporal air pollution data in China using data fusion

An efficient and effective spatiotemporal prediction algorithm for PM2.5 (i.e. particulate matter with a diameter of less than 2.5 micrometers) is urgently needed to study the distribution of PM2.5 over a continuous spatiotemporal domain, which not only helps to make scientific decisions on the prevention and control of PM2.5 pollution but also promotes meaningful assessment of the quantitative relationship between adverse health effects and PM2.5 concentrations over time. Existing spatiotemporal interpolation algorithms are usually based on the assumption that interpolation models follow explicit and simple mathematical descriptions. Unfortunately, the real world does not really follow these perfect mathematical models. Combining data fusion techniques and a Long Short-Term Memory (LSTM) recurrent neural network (RNN), we present a novel spatiotemporal interpolation model, which is able to achieve high estimation accuracies over a long time period and a large area. By fusing the daily PM2.5 data, meteorological data, elevation data, and land-use data collected from China in 2016, four experiments were conducted in this study to evaluate the efficiency and effectiveness of the proposed approach. Results showed that applying LSTM RNN on the fused dataset can achieve consistent and high accuracy in different geographies.

RETRACTED ARTICLE: Multi-edge optimized LSTM RNN for video summarization

Video summarization is an inevitable process in this developed communication world. The improvements in digital communication and filmless video recording technologies triggered the tremendous growth of storing and sharing variety of videos. Video summarization is used to optimize the searching and organizing process of different types of videos. Precision, Recall, F-Score and Processing time are the primary evaluation metrics of a video summarization procedure. A frequency domain multi-edge detection process and Multi-Edge optimized Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN) are proposed and integrated in this work. The frequency domain Multi-Edge detection is introduced to improve the precision, recall and F-Score whereas, Multi-Edge Optimized LSTM is used to reduce the processing time of the summarization process. Discrete Wavelet Transformation based multi-edge detection algorithm is introduced and integrated with the optimized LSTM to achieve the betterment of the summarization process. The proposed method named as Multi-Edge optimized LSTM RNN for Video Summarization (MOLRVS) is indented to perform the video summarization process in real time video streaming environments to reduce a significant amount of manual interventions.

Using a Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) to Classify Network Attacks

An intrusion detection system (IDS) identifies whether the network traffic behavior is normal or abnormal or identifies the attack types. Recently, deep learning has emerged as a successful approach in IDSs, having a high accuracy rate with its distinctive learning mechanism. In this research, we developed a new method for intrusion detection to classify the NSL-KDD dataset by combining a genetic algorithm (GA) for optimal feature selection and long short-term memory (LSTM) with a recurrent neural network (RNN). We found that using LSTM-RNN classifiers with the optimal feature set improves intrusion detection. The performance of the IDS was analyzed by calculating the accuracy, recall, precision, f-score, and confusion matrix. The NSL-KDD dataset was used to analyze the performances of the classifiers. An LSTM-RNN was used to classify the NSL-KDD datasets into binary (normal and abnormal) and multi-class (Normal, DoS, Probing, U2R, and R2L) sets. The results indicate that applying the GA increases the classification accuracy of LSTM-RNN in both binary and multi-class classification. The results of the LSTM-RNN classifier were also compared with the results using a support vector machine (SVM) and random forest (RF). For multi-class classification, the classification accuracy of LSTM-RNN with the GA model is much higher than SVM and RF. For binary classification, the classification accuracy of LSTM-RNN is similar to that of RF and higher than that of SVM.

NeuroVAD: Real-Time Voice Activity Detection from Non-Invasive Neuromagnetic Signals.

Neural speech decoding-driven brain-computer interface (BCI) or speech-BCI is a novel paradigm for exploring communication restoration for locked-in (fully paralyzed but aware) patients. Speech-BCIs aim to map a direct transformation from neural signals to text or speech, which has the potential for a higher communication rate than the current BCIs. Although recent progress has demonstrated the potential of speech-BCIs from either invasive or non-invasive neural signals, the majority of the systems developed so far still assume knowing the onset and offset of the speech utterances within the continuous neural recordings. This lack of real-time voice/speech activity detection (VAD) is a current obstacle for future applications of neural speech decoding wherein BCI users can have a continuous conversation with other speakers. To address this issue, in this study, we attempted to automatically detect the voice/speech activity directly from the neural signals recorded using magnetoencephalography (MEG). First, we classified the whole segments of pre-speech, speech, and post-speech in the neural signals using a support vector machine (SVM). Second, for continuous prediction, we used a long short-term memory-recurrent neural network (LSTM-RNN) to efficiently decode the voice activity at each time point via its sequential pattern-learning mechanism. Experimental results demonstrated the possibility of real-time VAD directly from the non-invasive neural signals with about 88% accuracy.

A day-ahead PV power forecasting method based on LSTM-RNN model and time correlation modification under partial daily pattern prediction framework

Photovoltaic (PV) power generation is an effective means to realize solar energy utilization. Due to the natural characteristics of random fluctuations in solar energy, the applications of PV power such as grid-connected PV power plant, distributed PVs, and building integrated PVs will introduce new characteristics to the generation and load side of the power grid. Therefore, accurate day-ahead PV power forecasting is of great significance for enabling grid manager to achieve PV power output data in advance and mitigate the influence of random fluctuations. To tackle the deficiencies of conventional artificial intelligence (AI) modeling methods such as overfitting problem and insufficient generalization ability to complex nonlinear modeling, a day-ahead PV power forecasting model assembled by fusing deep learning modeling and time correlation principles under a partial daily pattern prediction (PDPP) framework is proposed. First, an independent day-ahead PV power forecasting model based on long-short-term memory recurrent neural network (LSTM-RNN) is established. Second, a modification method is proposed to update the forecasting results of LSTM-RNN model based on time correlation principles regarding different patterns of PV power in the forecasting day. Third, a partial daily pattern prediction (PDPP) framework is proposed to provide accurate daily pattern prediction information of particular days, which is used to guide the modification parameters. Simulation results show that the proposed forecasting method with time correlation modification (TCM) is more accurate than the individual LSTM-RNN model, and the performance of the forecasting model can be further improved for those days with accurate daily pattern predictions under the proposed PDPP framework.

Bidirectional LSTM Recurrent Neural Network Plus Hidden Markov Model for Wearable Sensor-Based Dynamic State Estimation

Abstract Behavior of animals living in the wild is often studied using visual observations made by trained experts. However, these observations tend to be used to classify behavior during discrete time periods and become more difficult when used to monitor multiple individuals for days or weeks. In this work, we present automatic tools to enable efficient behavior and dynamic state estimation/classification from data collected with animal borne bio-logging tags, without the need for statistical feature engineering. A combined framework of an long short-term memory (LSTM) network and a hidden Markov model (HMM) was developed to exploit sequential temporal information in raw motion data at two levels: within and between windows. Taking a moving window data segmentation approach, LSTM estimates the dynamic state corresponding to each window by parsing the contiguous raw data points within the window. HMM then links all of the individual window estimations and further improves the overall estimation. A case study with bottlenose dolphins was conducted to demonstrate the approach. The combined LSTM–HMM method achieved a 6% improvement over conventional methods such as K-nearest neighbor (KNN) and support vector machine (SVM), pushing the accuracy above 90%. In addition to performance improvements, the proposed method requires a similar amount of training data to traditional machine learning methods, making the method easily adaptable to new tasks.

Deep Learning Based Multi-Channel Intelligent Attack Detection for Data Security

Deep learning methods, e.g., convolutional neural networks (CNNs) and Recurrent Neural Networks (RNNs), have achieved great success in image processing and natural language processing especially in high level vision applications such as recognition and understanding. However, it is rarely used to solve information security problems such as attack detection studied in this paper. Here, we move forward a step and propose a novel multi-channel intelligent attack detection method based on long short term memory recurrent neural networks (LSTM-RNNs). To achieve high detection rate, data preprocessing, feature abstraction, and multi-channel training and detection are seamlessly integrated into an end-to-end detection framework. Data preprocessing provides high-quality data for subsequent processing, then different types of features are extracted from the processed data. Multi-channel processing is used to generate classifiers by training neural networks with different types of features, which preserve attack features of input vectors and classify the attack from normal data. With the results of the classifier's attack detection, we introduce a voting algorithm to decide whether the input data is an attack or not. Experimental results validate that the proposed attack detection method greatly outperforms several attack detection methods that use feature detection and Bayesian or SVM classifiers.

The Art of Natural Language Processing: Classical, Modern and Contemporary Approaches to Text Document Classification

In this tutorial we introduce three approaches to preprocess text data with Natural Language Processing (NLP) and perform text document classification using machine learning. The first approach is based on 'bag-of-' models, the second one on word embeddings, while the third one introduces the two most popular Recurrent Neural Networks (RNNs), i.e. the Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) architectures. We apply all approaches on a case study where we classify movie reviews using Python and Tensorflow 2.0. The results of the case study show that extreme gradient boosting algorithms outperform adaptive boosting and random forests on bag-of-words and word embedding models, as well as LSTM and GRU RNNs, but at a steep computational cost. Finally, we provide the reader with comments on NLP applications for the insurance industry.