Deep LSTM Research Articles

Deep residual LSTM with domain-invariance for remaining useful life prediction across domains

Currently developed unsupervised domain adaptation (UDA) methods have somewhat improved the prognostic performance of cross-domain RUL prediction, but only optimizing one single metric (MMD or adversarial mechanism) to reduce the domain discrepancy has limited further improvement. Moreover, learning a set of good features has been a long-standing issue in RUL prediction. To address these issues, an effective UDA method namely deep residual LSTM with Domain-invariance (DIDRLSTM) is investigated to improve the prognostic performance. First, the DRLSTM is designed as the feature extractor to learn high-level features from both source and target domains. The introduction of residual connections allows DRLSTM to add more nonlinear layers to learn the more representative degradation features. Second, two modules are integrated to further reduce the domain discrepancy. One is domain adaptation, which reduces the domain discrepancy by adding MK-MMD constraints to map the features to RHKS. The other is domain confusion, which reduces the domain discrepancy through minimizing the domain discriminative ability of the domain classifier trained under adversarial optimization strategy. Finally, the outstanding performance of DIDRLSTM is validated on C-MAPSS dataset and FEMTO-ST dataset. The experimental results show that the DIDRLSTM outperforms five state-of-the-art UDA methods.

Combining Residual and LSTM Recurrent Networks for Transportation Mode Detection Using Multimodal Sensors Integrated in Smartphones

In recent years, with the rapid development of public transportation, the ways people travel has become more diversified and complicated. Transportation mode detection, as a significant branch of human activity recognition (HAR), is of great importance in analyzing human travel patterns, traffic prediction and planning. Though many works have been devoted to transportation mode detection, there remains challenge for accurate and robust transportation pattern identification. In this paper, we propose a residual and LSTM recurrent networks-based transportation mode detection algorithm using multiple light-weight sensors integrated in commodity smartphones. Feature representation learning is adopted separately on multiple preprocessed sensor data using deep residual and LSTM network, which can enhance the identification accuracy and support one or more sensors. Residual units are introduced to accelerate the learning speed and enhance the accuracy of transportation mode detection. Furthermore, we also leverage the attention model to learn the significance of different features and different timesteps to enhance the recognition accuracy. Extensive experimental results on three datasets indicate that using our proposed model can achieve the best recognition accuracy for eight transportation modes including being stationary, walking, running, cycling, taking a car, taking a bus, taking a subway and taking a train, which outperforms other benchmark algorithms.

Deep long short term memory based minimum variance kernel random vector functional link network for epileptic EEG signal classification

In this paper, the efficiently extracted and reduced features using deep long short-term memory (DLSTM) of the epileptic EEG signal integrated with minimum variance kernel random vector functional link net (MVKRVFLN) classifier are used to identify the seizure and non-seizure productively. Our methodology uses Children Hospital Boston-Massachusetts Institute of Technology (CHB-MIT) multi-channel scalp EEG data, Bonn university single-channel intracranial EEG data, and two-channel Bern Barcelona intracranial EEG data recordings to assess the performance. The non-stationary, non-linear, complex, and chaotic type EEG signal is directly applied to DLSTM to obtain compressed significant features. The scatter plots of the DLSTM output features signify this compressed information are unique in nature. The excellent generalization ability, faster learning rate, simpler network-based MVKRVFLN classifier is formulated to well identify the seizure and non-seizure epochs precisely by applying the deep LSTM extracted discriminative features as input. The type of kernel function selection and choice of regularization coefficient are added information to improve the performance of the proposed approach. The suggested technique provides excellent classification accuracy, superior detection ability, faster speed, and insignificant false positive rate per hour, simpler structure, robustness to classify the seizure and non-seizure signals.

Taylor–HHO algorithm: A hybrid optimization algorithm with deep long short‐term for malicious JavaScript detection

The security of information has become a major issue due to the development of network information-based technologies. The malicious script, like, JavaScript, is a major threat to computer networks in terms of network security. Here, the JavaScript allows the programmers not only to build advanced client-side interfaces for web-based applications but also utilized for carrying out attacks that may steal the user's confidential data. In addition, the attackers can easily induce malicious JavaScript into webpages for implanting attacks, like, phishing, spreading viruses, and Trojan horses. This paper devises a novel method, namely, Taylor–Harris Hawks Optimization driven deep long short-term memory (Taylor–HHO-based Deep LSTM) for malicious JavaScript discovery. Initially, the JavaScript is subjected as input to feature extraction in which certain features, such as time of execution, function calls, condition statement, break statement, loop statements, Boolean, number of lines, and number of O(N2) loops, are extracted. The obtained features are fed to transformation, wherein log transformation is applied for data transformation. The obtained transformed features are fused using information gain and Deep LSTM. Furthermore, the proposed Taylor–HHO-based Deep LSTM is employed for discovering malevolent JavaScript. The proposed Taylor–HHO-based Deep LSTM provided enhanced performance with the highest accuracy of 0.955, minimal FPR of 0.059, and highest TPR of 0.967.

A fully integrated violence detection system using CNN and LSTM

Recently, the number of violence-related cases in places such as remote roads, pathways, shopping malls, elevators, sports stadiums, and liquor shops, has increased drastically which are unfortunately discovered only after it’s too late. The aim is to create a complete system that can perform real-time video analysis which will help recognize the presence of any violent activities and notify the same to the concerned authority, such as the police department of the corresponding area. Using the deep learning networks CNN and LSTM along with a well-defined system architecture, we have achieved an efficient solution that can be used for real-time analysis of video footage so that the concerned authority can monitor the situation through a mobile application that can notify about an occurrence of a violent event immediately.

Improved Soil Moisture and Electrical Conductivity Prediction of Citrus Orchards Based on IoT Using Deep Bidirectional LSTM

In order to create an irrigation scheduling plan for use in large-area citrus orchards, an environmental information collection system of citrus orchards was established based on the Internet of Things (IoT). With the environmental information data, deep bidirectional long short-term memory (Bid-LSTM) networks are proposed to improve soil moisture (SM) and soil electrical conductivity (SEC) predictions, providing a meaningful reference for the irrigation and fertilization of citrus orchards. The IoT system contains SM, SEC, air temperature and humidity, wind speed, and precipitation sensors, while the mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R2) were calculated to evaluate the performance of the models. The performance of the deep Bid-LSTM model was compared with a multi-layer neural network (MLNN). The results for the performance criteria reveal that the proposed deep Bid-LSTM networks perform better than the MLNN model, according to many of the evaluation indicators of this study.

Chronological Harris hawks‐based deep LSTM classifier in wireless sensor network for aqua status prediction

AbstractAquaculture becomes very popular in economic where aquatic organisms, like fishes and prawns, are mainly dependent on the quality of water in aquaculture pond. Also, the water quality constraints, which include turbidity, carbon dioxide, temperature, pH level, dissolved oxygen and phosphorus, are considered for achieving better performance. Hence, this paper presents an approach for aqua status prediction based on Deep Long Short‐Term Memory (Deep LSTM) classifier. The sensor nodes are placed in the aqua pond for measuring the parameters of water quality, and then the cell network transformation is done using the Voronoi partition. After that, the Cluster Head (CH) selection is carried out using Piecewise Fuzzy C‐means clustering (piFCM). Once the clusters are selected, the Chronological Harris Hawks (Chronological HH) optimization algorithm is introduced for optimal sink placement where the constraints for enabling the optimal sink placement are the distance and energy of the nodes. Finally, the aqua status is predicted using Deep LSTM. The performance of the Chronological HH‐based Deep LSTM is computed in terms of accuracy, energy and the number of dead nodes. The proposed Chronological HH‐based Deep LSTM outperformed other methods with maximal accuracy of 92.65%, maximal energy of 0.976 and the minimal dead nodes of 32.

Estimation of Frequency-Dependent Impedances in Power Grids by Deep LSTM Autoencoder and Random Forest

This paper proposes a deep-learning-based method for frequency-dependent grid impedance estimation. Through measurement of voltages and currents at a specific system bus, the estimate of the grid impedance was obtained by first extracting the sequences of the time-dependent features for the measured data using a long short-term memory autoencoder (LSTM-AE) followed by a random forest (RF) regression method to find the nonlinear map function between extracted features and the corresponding grid impedance for a wide range of frequencies. The method was trained via simulation by using time-series measurements (i.e., voltage and current) for different system parameters and verified through several case studies. The obtained results show that: (1) extracting the time-dependent features of the voltage/current data improves the performance of the RF regression method; (2) the RF regression method is robust and allows grid impedance estimation within 1.5 grid cycles; (3) the proposed method can effectively estimate the grid impedance both in steady state and in case of large transients like electrical faults.

Application of time series model and deep learning method in measuring the impact of COVID-19 on agriculture in Hubei, China

Taking the agricultural situation of Hubei Province as the research object, this paper uses the time series ARIMA model, ARMA model and deep learning LSTM neural network model to explore the impact of COVID-19 on the agriculture of Hubei Province. Three main indicators are screened out to measure the development of agricultural economy, that is, gross regional product, gross output value of agriculture, forestry, animal husbandry and fishery, agricultural product production price index. Based on the quarterly data of indicators from 2001 to 2019 from the National Bureau of Statistics, three indicators in Hubei Province in the first quarter and the second quarter of 2020 are predicted by using the deep learning-based time series ARIMA model, ARMA model and LSTM neural network model. By comparing the predicted data with the real data, the impact of COVID-19 is measured on the agricultural situation of Hubei Province. It was found that COVID-19 had a great impact on the agricultural situation of Hubei Province in both of the first and second quarters of 2020, with the impact in the first quarter being greater than that in the second quarter. At the same time, the prediction accuracy of the two methods is compared to find that the time series model is more effective and reliable in predicting the agricultural product price index. The LSTM neural network model with a long and short term memory has a good prediction effect on the regional gross product and the total output value of agriculture, forestry, animal husbandry and fishery.

Energy consumption prediction model with deep inception residual network inspiration and LSTM

Predicting electricity consumption is not an easy task depending on many factors that affect energy consumption. Therefore, electricity utilities and governments are always searching for intelligent models to improve the accuracy of prediction and recently, deep learning becomes the most used field in prediction. In this paper, we introduce a deep learning model based on deep feedforward neural networks and Long Short-Term Memory. The deep feedforward neural networks architecture was inspired by the Inception Residual Network v2, which achieved the highest accuracy in image classification. We compared our proposed model to other recent deep learning models in two different datasets: dataset from the Distribution Network Station of Tetouan city in Morocco and dataset from the North American Utility. The proposed model achieved the smallest error of Root Mean Square Error comparing to its counterparts.

Deep LSTM Recurrent Neural Networks for Arterial Traffic Volume Data Imputation

Deep LSTM Recurrent Neural Networks for Arterial Traffic Volume Data Imputation

A Deep Neural Network Model for Speaker Identification

Speaker identification is a classification task which aims to identify a subject from a given time-series sequential data. Since the speech signal is a continuous one-dimensional time series, most of the current research methods are based on convolutional neural network (CNN) or recurrent neural network (RNN). Indeed, these methods perform well in many tasks, but there is no attempt to combine these two network models to study the speaker identification task. Due to the spectrogram that a speech signal contains, the spatial features of voiceprint (which corresponds to the voice spectrum) and CNN are effective for spatial feature extraction (which corresponds to modeling spectral correlations in acoustic features). At the same time, the speech signal is in a time series, and deep RNN can better represent long utterances than shallow networks. Considering the advantage of gated recurrent unit (GRU) (compared with traditional RNN) in the segmentation of sequence data, we decide to use stacked GRU layers in our model for frame-level feature extraction. In this paper, we propose a deep neural network (DNN) model based on a two-dimensional convolutional neural network (2-D CNN) and gated recurrent unit (GRU) for speaker identification. In the network model design, the convolutional layer is used for voiceprint feature extraction and reduces dimensionality in both the time and frequency domains, allowing for faster GRU layer computation. In addition, the stacked GRU recurrent network layers can learn a speaker’s acoustic features. During this research, we tried to use various neural network structures, including 2-D CNN, deep RNN, and deep LSTM. The above network models were evaluated on the Aishell-1 speech dataset. The experimental results showed that our proposed DNN model, which we call deep GRU, achieved a high recognition accuracy of 98.96%. At the same time, the results also demonstrate the effectiveness of the proposed deep GRU network model versus other models for speaker identification. Through further optimization, this method could be applied to other research similar to the study of speaker identification.

An improved pollution forecasting model with meteorological impact using multiple imputation and fine-tuning approach

Air pollution forecasting is a significant step for air quality pollution management to mitigate pollution’s negative impact on the environment and people’s health. The data-driven forecasting model can help a better understanding of environmental air quality. The existing data-driven forecasting models usually ignore missing values, the correlations between the pollutant and meteorological factors and fail to perform temporal modeling effectively, affecting prediction accuracy. In response to these issues, we present a deep learning-based Convolutional LSTM–SDAE (CLS) model to forecast the particulate matter level, revealing the correlation between particulate matter and meteorological factors. In the proposed architecture, the k nearest neighbor (KNN) imputation technique is employed to recover the air quality dataset’s missing values. The Convolutional Long Short Term Memory (CNN–LSTM) unit identifies the vast dataset’s hidden features and performs pollutants’ temporal modeling. In addition, Bidirectional Gatted Recurrent Unit (BIGRU) is implemented as both encoder and decoder in Sparse Denoising Autoencoder, which reconstructs the CNN–LSTM model’s output in the dynamic fine-tuning layer to get robust prediction results. The experimental results in Talcher, India, and Beijing, China indicate that the model can improve forecasting accuracy and outperforms the other state of art and baseline models.

Stock Market Prediction Using LSTM

This paper aims to develop an innovative neural network approach to achieve better stock market predictions. Data were obtained from the live stock market for real-time and off-line analysis and results of visualizations and analytics to demonstrate Internet of Multimedia of Things for stock analysis. To study the influence of market characteristics on stock prices, traditional neural network algorithms may incorrectly predict the stock market, since the initial weight of the random selection problem can be easily prone to incorrect predictions. Based on the development of word vector in deep learning, we demonstrate the concept of “stock vector.” The input is no longer a single index or single stock index, but multi-stock high-dimensional historical data. We propose the deep long short-term memory neural network (LSTM) with embedded layer and the long short-term memory neural network with automatic encoder to predict the stock market. In these two models, we use the embedded layer and the automatic encoder, respectively, to vectorize the data, in a bid to forecast the stock via long short-term memory neural network. The experimental results show that the deep LSTM with embedded layer is better.

A Deep Graph-Embedded LSTM Neural Network Approach for Airport Delay Prediction

Due to the strong propagation causality of delays between airports, this paper proposes a delay prediction model based on a deep graph neural network to study delay prediction from the perspective of an airport network. We regard airports as nodes of a graph network and use a directed graph network to construct airports’ relationship. For adjacent airports, weights of edges are measured by the spherical distance between them, while the number of flight pairs between them is utilized for airports connected by flights. On this basis, a diffusion convolution kernel is constructed to capture characteristics of delay propagation between airports, and it is further integrated into the sequence-to-sequence LSTM neural network to establish a deep learning framework for delay prediction. We name this model as deep graph-embedded LSTM (DGLSTM). To verify the model’s effectiveness and superiority, we utilize the historical delay data of 325 airports in the United States from 2015 to 2018 as the model training set and test set. The experimental results suggest that the proposed method is superior to the existing mainstream methods in terms of accuracy and robustness.

Sparse Deep LSTMs with Convolutional Attention for Human Action Recognition

Deep learning has recently gained remarkable results in action recognition. In this paper, an architecture is proposed for action recognition, including ResNet feature extractor, Conv-Attention-LSTM, BiLSTM, and fully connected layers. Furthermore, a sparse layer after each LSTM layer is added to overcome overfitting. In addition to RGB images, optical flow is also used to incorporate motion information into our architecture. Due to similarities of consecutive frames, video sequences are divided into equal parts. Frames of successive parts are used as consecutive frames to obtain the flow. Furthermore, to find the significant regions, the convolutional attention network is applied. The proposed method is evaluated using two popular datasets, UCF-101, and HMDB-51 and its accuracy for these datasets is 95.24 and 71.62, respectively. Overfitting is reduced due to using a sparse layer instead of a dropout based on the results achieved. Moreover, a deep LSTM network leads to a higher recognition rate than one-layer LSTM.

Deep sequence to sequence Bi-LSTM neural networks for day-ahead peak load forecasting

The power industry is currently facing the problem of an electricity supply–demand imbalance. The most inexpensive and efficient solution to alleviate this imbalance is to decrease electricity demand. Local electrical utilities should deploy demand response programs (DRP), and short-term peak demand forecasting (STPDF) plays a crucial role in their successful deployment. In residential sectors, peak demand forecasting is also critical because the energy policies, technological growth, and changing climate are further increasing the peak demand. Therefore, an accurate peak demand forecasting will help utility companies in avoiding blackouts and secure a continuous power supply by implementing subsidy schemes such as DRP. However, daily peak load is volatile, nonstationary, and nonlinear in nature, and hence it is hard to predict it accurately. This research work for the first time has attempted to design, implement, and test deep bidirectional long short-term memory based sequence to sequence (Bi-LSTM S2S) regression approach for “day-ahead” peak demand forecasting and has accomplished preliminary success. The day-ahead peak electricity demand forecasting model is designed and tested using the MATLAB software. For performance comparison, shallow Bi-LSTM S2S, shallow LSTM S2S, deep LSTM S2S, Levenberg-Marquardt backpropagation artificial neural networks (LMBP-ANN), and medium Gaussian support vector regression (MG-SVR) forecasting models are also developed and tested. Mean absolute percentage error (MAPE) and Root Mean Squared Error (RMSE) are used as performance metrics. It has been found out that in terms of both performance metrics, the proposed deep Bi-LSTM S2S day-ahead “peak” forecasting model has outperformed all the other models on both public holidays and normal days. The load pattern on public holidays is always different than on normal days, and there is always less data available in contrast to the normal days. Therefore, it is hard to accurately forecast their load.

Combining CNN and LSTM for activity of daily living recognition with a 3D matrix skeleton representation

In socially assistive robotics, human activity recognition plays a central role when the adaptation of the robot behavior to the human one is required. In this paper, we present an activity recognition approach for activities of daily living based on deep learning and skeleton data. In the literature, ad hoc features extraction/selection algorithms with supervised classification methods have been deployed, reaching an excellent classification performance. Here, we propose a deep learning approach, combining CNN and LSTM, that exploits both the learning of spatial dependencies correlating the limbs in a skeleton 3D grid representation and the learning of temporal dependencies from instances with a periodic pattern that works on raw data and so without requiring an explicit feature extraction process. These models are proposed for real-time activity recognition, and they are tested on the CAD-60 dataset. Results show that the proposed model behaves better than an LSTM model thanks to the automatic features extraction of the limbs’ correlation. “New Person” results show that the CNN-LSTM model achieves 95.4% of precision and 94.4% of recall, while the “Have Seen” results are 96.1% of precision and 94.7% of recall.

Decomposition-based hybrid wind speed forecasting model using deep bidirectional LSTM networks

The goal of sustainable development can be attained by the efficient management of renewable energy resources. Wind energy is attracting attention worldwide due to its renewable and sustainable nature. Accurate wind speed prediction is essential for the stable functioning of wind turbines to generate wind power. However, the flexible and intermittent nature of wind speed makes accurate wind speed forecasting a challenging task. The proposed wind speed forecasting framework combines the features of various data decomposition techniques and Bidirectional Long Short Term Memory (BiDLSTM) networks. Presently, Data Decomposition models such as the Wavelet Transform are extensively employed for wind speed forecasting to improve the accuracy of the forecasting models. Hence, in this paper, various data decomposition techniques that can denoise the signal are investigated and applied to partition the input time series into several high and low-frequency signals. The data decomposition methods, namely, Wavelet Transform, Empirical Model Decomposition, Ensemble Empirical Mode Decomposition, and Empirical Wavelet Transform, have been applied to denoise the dataset. The low and high-frequency sub-series are forecasted separately using Bidirectional LSTM networks, and the forecasting outcomes of low and high-frequency signals are aggregated to get the final forecasting results. The empirical results establish that the proposed EWT- based hybrid model outperforms other decomposition-based models in accuracy and stability. The performance of the EWT-BiDLSTM model is further compared with Bidirectional LSTM networks with skip connections. The experimental results substantiate that the proposed decomposition-based hybrid deep BiDLSTM models with skip connections exhibit better prediction accuracy than other models.

Online Tuning of Hyperparameters in Deep LSTM for Time Series Applications

Deep learning is one of the most remarkable artificial intelligence trends. It stands behind numerous recent achievements in several domains, such as speech processing, and computer vision, to mention a few. Accordingly, these achievements have sparked great attention to employing deep learning in time series modelling and forecasting. It is known that the deep learning algorithms built on neural networks contain multiple hidden layers, which make the computation of deep neural network challenging and, sometimes, complex. The reason for this complexity is that obtaining an outstanding and consistent result from such deep architecture requires optimizing many parameters known as hyperparameters. Doubtless, hyperparameter tuning plays a critical role in improving the performance of deep learning. This paper proposes an online tuning approach for the hyperparameters of deep long short-term memory (DLSTM) model in a dynamic fashion. The proposed approach adapts to learn any time series based application, particularly the applications that contain streams of data. The experimental results show that the dynamic tuning of the DLSTM hyperparameters performs better than the original static tuning fashion.