Long Short-term Memory Units Research Articles

Spatio–Temporal–Spectral Collaborative Learning for Spatio–Temporal Fusion with Land Cover Changes

Spatio-temporal fusion by combining the complementary spatial and temporal advantages of multi-source remote sensing images to obtain time-series high spatial resolution images is highly desirable in monitoring surface dynamics. Currently, deep learning (DL)-based fusion methods have received extensive attention. However, existing DL-based spatio-temporal fusion methods are generally limited in fusing the images with land cover changes. In this paper, we propose a spatio-temporal-spectral collaborative learning framework for spatio-temporal fusion to alleviate this problem. Specifically, the proposed method integrates the convolutional neural network and recurrent neural network into a unified framework, consisting of three sub-networks: multi-scale siamese convolutional neural network, multi-layer convolutional recurrent neural network, and adaptive weighting fusion network. The multi-scale siamese convolutional neural network has a flexible weight-sharing network to extract multi-scale spatial-spectral features from multi-source remote sensing images. The multi-layer convolutional recurrent neural network is constructed on the convolutional long-short term memory units to comprehensively learn the land cover changes by spatial, spectral, and temporal joint features. The adaptive weighting fusion network with a spatio-temporal-spectral change loss is proposed to further improve the interpretability and robustness. The experiments were performed on the publicly available benchmark datasets featured by phenology and land cover type changes, respectively. The experimental results demonstrated the competitive performance of the proposed method than other state-of-the-art fusion methods.

Temperature Forecasting for Stored Grain: A Deep Spatiotemporal Attention Approach

The development of Internet-of-Things (IoT) technology promotes the advances of grain condition detection and analysis systems. Temperature monitoring is a main element to maintain grain quality, and effective control of grain temperature is crucial to safe storage of grain. In this article, an encoder–decoder model with attention mechanism is proposed to accurately forecast the temperature of stored grain. Considering that the points on the gradient direction of the temperature surface have a great influence on the temperature of the target point, the Sobel operator is used to extract the local characteristics of the target point. In addition, considering the correlation structure in the sensory data, the attention mechanism is used to extract the global features of the target point. The extracted spatial features are fed into long short-term memory (LSTM) networks to obtain the long-term state information of spatial factors. LSTM unit and convolutional neural network are used to encode the spatial features of the target points. Taking meteorological factors as the external input of the decoder, temporal attention mechanism and LSTM unit are used to complete the decoding process and realize the prediction of grain temperature in the future. The results with real grain storage data show that the proposed model outperforms several schemes, including Kalman-modified the least absolute shrinkage and selection operator (Kalman-modified LASSO), temporal graph convolutional network (T-GCN), LSTM, CNN-LSTM, and convolutional LSTM (Conv-LSTM), with considerable gains.

A SIA-LSTM based virtual metrology for quality variables in irregular sampled time sequence of industrial processes

In industrial processes, there are usually strongly dynamic temporal relationship between process data sequence. Hence, dynamic modeling methods are popular for soft sensing of key quality variables. For the real-life industrial processes, the data series are usually sampled with different intervals for the samples. However, most of the mainstream dynamic models, represented by long short-term memory (LSTM), is difficult to directly handle the time series with irregularly sampled data. Hence, a sampling-interval-attention LSTM (SIA-LSTM) is developed to deal with it, in which an attention network is introduced to adaptively learn the dynamics between two consecutive samples and transfer it into an attention weight of the sampling interval. Next, the previous hidden state and the attention weight are multiplied. The weighted previous hidden state is further sent to the control gates to compute the current hidden state in LSTM unit. Hence, the dynamic relationship is calculated adaptively according to the sampling interval. The effectiveness of SIA-LSTM is evaluated for quality prediction on a refining hydrocracking process.

Automated atrial fibrillation prediction using a hybrid long short-term memory network with enhanced whale optimization algorithm on electrocardiogram datasets

Background: Cardiac arrhythmias are one of the leading causes of heart failure. In particular, atrial fibrillation (AFib) is a kind of arrhythmia that can lead to heart stroke and myocardial infarction. It is very important and crucial to predict AFib at an early stage to prevent heart disease. Electrocardiogram is one of the premium diagnostic tools which is used by most of the researchers for predicting irregular heartbeats. There are many works carried out in finding heart disease using machine learning classifiers. Aims and Objectives: Deep learning based hybrid Long Short Term Memory (LSTM) network is hybridized with Enhanced Whale Optimization (EWO) to minimize the network optimization and configuration issues faced in the existing models and proposed to increases the accuracy of predicting AFib. Materials and Methods: The proposed LSTM network is hybridized with a EWO technique for predicting AFib. This study uses a hybrid LSTM EWO network for classifying the various output labels of heart disease. EWO is used to predict the most relevant features from the raw dataset. Then, the LSTM model is used to predict the AFib of a patient from normal ECG data. Results: The DL based LSTM EWO achieves better results in all the performance metrics by analyzing the optimized features in feature space, training, and testing phase and successfully obtains better performance in an effective manner. LSTM improves the accuracy by reducing the number of units in the hidden layer which optimizes the network configuration. The proposed model achieves 96.12% accuracy which is 12.81% higher than RF, 15.01% higher than GB, 28.04% higher than CART, and 16.92% higher than SVM. Conclusion: The proposed model hybrid LSTM network integrated EWO for predicting the AFib. The EWO is applied for selecting the most appropriate features needed for the model to learn and produce improvised performance. The optimization and network configuration problems faced in the existing studies are avoided by choosing the suitable number of LSTM units and the size of the time window. This has been implemented through LSTM units and their window size. In addition, we made a statistical examination to prove the importance of proposed work against other models. It is observed that the experimental results attained with 96% of accuracy, better than conventional models.

Predicting the success rate of a start-up using LSTM with a swish activation function

The researchers emphasised more small-scale start-ups which required improvement in the economy decreased the failure rate by avoiding the valuable resources. The resources were required to be used without wasting them lead to the development of the economy and reduces the rate of unemployment. Therefore, to minimise resource wastage and to avoid the risk of failure the researchers considered certain factors that affected the failure and success of small-scale industries. The present research uses the Crunch Base dataset that predicts the success or failure of a start-up by using the Long Short Term Memory (LSTM). The LSTM unit Recurrent Neural Network (RNN) uses the Swish activation function in Feed Forward Neural (FFN) Network for the classification. The proposed LSTM obtained better accuracy of 71.64% when compared with existing methods such as RNN that attained 65.67% and Artificial Neural Network (ANN) of 69.7%.

Generation of surgical reports using keyword-augmented next sequence prediction

Abstract The documentation of a surgical procedure remains a time-consuming task that surgeons must incorporate into their daily routine. However, since a surgical report should be produced immediately after the operation with all impressions of the procedure in mind, a means of automation assistance should be provided. We, therefore, propose a method that generates surgical reports based on keywords stated during the procedure. Our report generation is based on a sequence-tosequence model that is trained on sentence pairs of two consecutive sentences in a surgical report. The known sentence is augmented with a keyword based on the following surgical action to be documented and is then passed into a language model to generate the next sentence. In this way, the complexity of predicting a vast number of possible surgical report phrasings is reduced to a next sentence prediction task. For the language model, an encoder-decoder structure was used with bidirectional 2-layer Long-Short Term Memory (LSTM) units for both components and an attention layer between input and output sentences. The training data consisted of 50 ear-,nose- and throat surgery (ENT) reports with 1500 sentences. The model training was performed in a k-fold cross-validation study with k = 10 and cross-entropy loss as the objective function. The generated reports were investigated using NIST, ROUGE, and METEOR metrics. Additionally, three medical experts identified the report content regarding plausibility and text errors. The trained models reached an accuracy of 0.82 for the next sentence predictions. The generated reports show consistent sentence structures and keyword correspondence for about 70 % of provided keyword sequences. The NIST, ROUGE, and METEOR metrics reached 0.65, 0.71, and 0.64, respectively. The model underperformed for not yet known keyword sequences and shows signs of overfitting when keyword sequences deviate from the baseline of the training set. Our approach for the keyword-augmented generation of surgical reports shows the potential of reducing the text generation complexity by providing a sequence of anchor words. However, the automated generation of surgical reports remains a difficult task due to individual report phrasings and the high variance in keyword sequences.

Action Recognition Algorithm of Spatio–Temporal Differential LSTM Based on Feature Enhancement

The Long Short-Term Memory (LSTM) network is a classic action recognition method because of its ability to extract time information. Researchers proposed many hybrid algorithms based on LSTM for human action recognition. In this paper, an improved Spatio–Temporal Differential Long Short-Term Memory (ST-D LSTM) network is proposed, an enhanced input differential feature module and a spatial memory state differential module are added to the network. Furthermore, a transmission mode of ST-D LSTM is proposed; this mode enables ST-D LSTM units to transmit the spatial memory state horizontally. Finally, these improvements are added into classical Long-term Recurrent Convolutional Networks (LRCN) to test the new network’s performance. Experimental results show that ST-D LSTM can effectively improve the accuracy of LRCN.

Early prediction and analysis of corona pandemic outbreak using deep learning technique

PurposeThe purpose of this paper is to analyze and build a deep learning model that can furnish statistics of COVID-19 and is able to forecast pandemic outbreak using Kaggle open research COVID-19 data set. As COVID-19 has an up-to-date data collection from the government, deep learning techniques can be used to predict future outbreak of coronavirus. The existing long short-term memory (LSTM) model is fine-tuned to forecast the outbreak of COVID-19 with better accuracy, and an empirical data exploration with advanced picturing has been made to comprehend the outbreak of coronavirus.Design/methodology/approachThis research work presents a fine-tuned LSTM deep learning model using three hidden layers, 200 LSTM unit cells, one activation function ReLu, Adam optimizer, loss function is mean square error, the number of epochs 200 and finally one dense layer to predict one value each time.FindingsLSTM is found to be more effective in forecasting future predictions. Hence, fine-tuned LSTM model predicts accurate results when applied to COVID-19 data set.Originality/valueThe fine-tuned LSTM model is developed and tested for the first time on COVID-19 data set to forecast outbreak of pandemic according to the authors’ knowledge.

A Fusion Method of Local Path Planning for Mobile Robots Based on LSTM Neural Network and Reinforcement Learning

Due to the limitation of mobile robots’ understanding of the environment in local path planning tasks, the problems of local deadlock and path redundancy during planning exist in unknown and complex environments. In this paper, a novel algorithm based on the combination of a long short-term memory (LSTM) neural network, fuzzy logic control, and reinforcement learning is proposed, and uses the advantages of each algorithm to overcome the other’s shortcomings. First, a neural network model including LSTM units is designed for local path planning. Second, a low-dimensional input fuzzy logic control (FL) algorithm is used to collect training data, and a network model (LSTM_FT) is pretrained by transferring the learned method to learn the basic ability. Then, reinforcement learning is combined to learn new rules from the environments autonomously to better suit different scenarios. Finally, the fusion algorithm LSTM_FTR is simulated in static and dynamic environments, and compared to FL and LSTM_FT algorithms, respectively. Numerical simulations show that, compared to FL, LSTM_FTR can significantly improve decision-making efficiency, improve the success rate of path planning, and optimize the path length. Compared to the LSTM_FT, LSTM_FTR can improve the success rate and learn new rules.

The effect of phased recurrent units in the classification of multiple catalogues of astronomical light curves

ABSTRACT In the new era of very large telescopes, where data are crucial to expand scientific knowledge, we have witnessed many deep learning applications for the automatic classification of light curves. Recurrent neural networks (RNNs) are one of the models used for these applications, and the Long Short-Term Memory (LSTM) unit stands out for being an excellent choice for the representation of long time series. In general, RNNs assume observations at discrete times, which may not suit the irregular sampling of light curves. A traditional technique to address irregular sequences consists of adding the sampling time to the network’s input, but this is not guaranteed to capture sampling irregularities during training. Alternatively, the Phased LSTM (PLSTM) unit has been created to address this problem by updating its state using the sampling times explicitly. In this work, we study the effectiveness of the LSTM- and PLSTM-based architectures for the classification of astronomical light curves. We use seven catalogues containing periodic and non-periodic astronomical objects. Our findings show that LSTM outperformed PLSTM on six of seven data sets. However, the combination of both units enhances the results in all data sets.

Integrating big data driven sentiments polarity and ABC-optimized LSTM for time series forecasting

Stock market is a dynamic and volatile market that is considered as time series data. The growth of financial data exposed the computational efficiency of the conventional systems. This paper proposed a hybrid deep learning model based on Long Short- Term Memory (LSTM) and Artificial Bee Colony (ABC) algorithm. ABC is best fit for hyper parameter selection for deep LSTM models and maintains the equilibrium of exploitation and exploration issues. Handling a large volume of multidimensional reviews from social media is a major challenge. This paper evolves the multiple aspects of market sentiments and uses the reliable Big data platform Hadoop ecosystem and its services to compute sentiment polarity index. The ABC-LSTM hybrid model is validated with other core and hybrid models with evolutionary algorithms as Differential Evolution (DE) and Genetic Algorithm (GA). For the experiments, 10 years of historical datasets and social media reviews of IT sector funds Apple Inc. (AAPL), Microsoft corporation (MSFT) and Intel corporation (INTL) from NASDAQ GS, an American stock exchange are considered to validate hybrid forecasting models. Proposed algorithm ABC-LSTM is used to tune the hyperparameters (window size, LSTM units, dropout probability, epochs, batch size and learning rate) and evaluated through Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE) as loss function. Performance analysis proves that with sentiment polarity, ABC optimized LSTM obtains improved forecasting accuracy over its counterpart models.

Prediction of time series using an analysis filter bank of LSTM units

Time series emerge in various applications such as financial data and production data, however, most of the generated data exhibit nonlinear inter-dependency between samples and noise, making necessary the development of methods capable of handling such nonlinearities and other abnormalities. In this paper we present an architecture for prediction of time series embedded in noise. The proposed architecture combines a convolutional and long short term memory (LSTM) layers into a structure similar to an analysis filterbank of two channels. The first element of each channel is a convolutional layer followed by a LSTM, which is able to find temporal dependencies of the signal. Finally the channels are summed to obtain a prediction. We found that the frequency response of the filters resemble a complementary filter bank response, with each channel having a maximum at different bands which could suggest that it characterizes the incoming signal in frequency. Comparisons with other methods demonstrate that the proposed method offer much better results in terms of different error measures.

Health condition monitoring of machines based on long short-term memory convolutional autoencoder

Machine health assessment is crucial to prognostics and health management (PHM), which can increase reliability of machines while reducing operation cost. However, the data collected from machines in real-world cases are noised and high-dimensional, so that it is very difficult to detect the early defects of machines. Moreover, it is still a challenging problem to capture sequential information from multi-sensor signals for machine health assessment. This paper proposes a novel autoencoder (AE), called long short-term memory convolutional autoencoder (LSTMCAE), where LSTM and convolutional units are embedded in this specific network for feature learning from sensor signals based on unsupervised-learning. A long short-term memory (LSTM) unit in LSTMCAE is utilized to capture sequential information from multi-sensor time series data. A convolutional and deconvolutional unit is further embedded after the LSTM unit to filter noise and extract features corresponding to health state of machines. Residual learning is employed to reduce the training difficulty and improve the feature learning performance of LSTMCAE. Multivariate Gaussian distribution (MGD) is adopted to generate health index (HI) based on reconstruction errors of LSTMCAE for quantifying machines health state. A contribution analysis-based feature selection method is proposed to select effective features for machinery health assessment. Experimental results on turbofan engines demonstrate the effectiveness of LSTMCAE for machine health assessment. Compared with other unsupervised learning methods, the HIs generated by LSTMCAE show better tendency. LSTMCAE can detect earlier slight degradation than LSTM-AE. Moreover, contribution analysis results indicate that the sensitive variables can be selected for machine health monitoring. Residual learning significantly improves the feature learning performance of LSTMCAE. Thus, LSTMCAE can well quantify the degradation trend of machines. The experimental results on engines show that the proposed method will be an effective tool for machine health assessment.

Convolutional and recurrent neural network based model for short-term load forecasting

The consumed electrical load is affected by many external factors such as weather, season of the year, weekday or weekend and holiday. In this paper, it is tried to provide a high accurate forecasting model for hourly load consumption with considering these external variables. At first, the electrical load and temperature time series are rearranged into separate two-dimensional matrices. Convolutional neural networks (CNNs) are utilized to extract the load and temperature features. The autocorrelation coefficients of the load and temperature sequences are used to determine the kernel size of the convolutional layers. At this stage, the convolutional layers specifically convert the univariate data to multidimensional features by applying two-dimensional convolutional kernels, which potentially increase the forecasting capability of recurrent neural networks. On the other hand, long short term memory (LSTM) and gated recurrent unit (GRU) are able to hold short-term and long-term memories. Therefore, in the next stage, the multidimensional features extracted by 2-D CNNs are fed as input to the bidirectional propagating GRU and LSTM units to perform hourly electrical load forecasting. The results of experiments on two datasets show the superiority of the proposed method compared to some recent works in the field of short-term load forecasting.

New Results for Prediction of Chaotic Systems Using Deep Recurrent Neural Networks

Prediction of nonlinear and dynamic systems is a challenging task, however with the aid of machine learning techniques, particularly neural networks, is now possible to accomplish this objective. Most common neural networks used are the multilayer perceptron (MLP) and recurrent neural networks (RNN) using long-short term memory units (LSTM-RNN). In recent years, deep learning neural network models have become more relevant due the improved results they show for various tasks. In this paper the authors compare these neural network models with deep learning neural network models such as long-short term memory deep recurrent neural network (LSTM-DRNN) and gate recurrent unit deep recurrent neural network (GRU-DRNN) when presented with the task of predicting three different chaotic systems such as the Lorenz system, Rabinovich–Fabrikant and the Rossler System. The results obtained show that the deep learning neural network model GRU-DRNN has better results when predicting these three chaotic systems in terms of loss and accuracy than the two other models using less neurons and layers. These results can be very helpful to solve much more complex problems such as the control and synchronization of these chaotic systems.

COVID-19 Patient Count Prediction Using LSTM.

In December 2019, a pandemic named COVID-19 broke out in Wuhan, China, and in a few weeks, it spread to more than 200 countries worldwide. Every country infected with the disease started taking necessary measures to stop the spread and provide the best possible medical facilities to infected patients and take precautionary measures to control the spread. As the infection spread was exponential, there arose a need to model infection spread patterns to estimate the patient volume computationally. Such patients' estimation is the key to the necessary actions that local governments may take to counter the spread, control hospital load, and resource allocations. This article has used long short-term memory (LSTM) to predict the volume of COVID-19 patients in Pakistan. LSTM is a particular type of recurrent neural network (RNN) used for classification, prediction, and regression tasks. We have trained the RNN model on Covid-19 data (March 2020 to May 2020) of Pakistan and predict the Covid-19 Percentage of Positive Patients for June 2020. Finally, we have calculated the mean absolute percentage error (MAPE) to find the model's prediction effectiveness on different LSTM units, batch size, and epochs. Predicted patients are also compared with a prediction model for the same duration, and results revealed that the predicted patients' count of the proposed model is much closer to the actual patient count.

Hierarchical Long Short-Term Concurrent Memory for Human Interaction Recognition.

In this work, we aim to address the problem of human interaction recognition in videos by exploring the long-term inter-related dynamics among multiple persons. Recently, Long Short-Term Memory (LSTM) has become a popular choice to model individual dynamic for single-person action recognition due to its ability to capture the temporal motion information in a range. However, most existing LSTM-based methods focus only on capturing the dynamics of human interaction by simply combining all dynamics of individuals or modeling them as a whole. Such methods neglect the inter-related dynamics of how human interactions change over time. To this end, we propose a novel Hierarchical Long Short-Term Concurrent Memory (H-LSTCM) to model the long-term inter-related dynamics among a group of persons for recognizing human interactions. Specifically, we first feed each person's static features into a Single-Person LSTM to model the single-person dynamic. Subsequently, at one time step, the outputs of all Single-Person LSTM units are fed into a novel Concurrent LSTM (Co-LSTM) unit, which mainly consists of multiple sub-memory units, a new cell gate, and a new co-memory cell. In the Co-LSTM unit, each sub-memory unit stores individual motion information, while this Co-LSTM unit selectively integrates and stores inter-related motion information between multiple interacting persons from multiple sub-memory units via the cell gate and co-memory cell, respectively. Extensive experiments on several public datasets validate the effectiveness of the proposed H-LSTCM by comparing against baseline and state-of-the-art methods.

Adaptive Online State of Charge Estimation of EVs Lithium-Ion Batteries with Deep Recurrent Neural Networks

The State of Charge (SOC) estimation is a significant issue for safe performance and the lifespan of Lithium-ion (Li-ion) batteries. In this paper, a Robust Adaptive Online Long Short-Term Memory (RoLSTM) method is proposed to extract SOC estimation for Li-ion Batteries in Electric Vehicles (EVs). This real-time, as its name suggests, method is based on a Recurrent Neural Network (RNN) containing Long Short-Term Memory (LSTM) units and using the Robust and Adaptive online gradient learning method (RoAdam) for optimization. In the proposed architecture, one sequential model is defined for each of the three inputs: voltage, current, and temperature of the battery. Therefore, the three networks work in parallel. With this approach, the number of LSTM units are reduced. Using this suggested method, one is not dependent on precise battery models and can avoid complicated mathematical methods. In addition, unlike the traditional recursive neural network where content is re-written at any time, the LSTM network can decide on preserving the current memory through the proposed gateways. In that case, it can easily transfer this information over long paths to receive and maintain long-term dependencies. Using real databases, the experiment results illustrate the better performance of RoLSTM applied to SOC estimation of Li-Ion batteries in comparison with a neural network modeling and unscented Kalman filter method that have been used thus far.

Context-aware network for RGB-D salient object detection

Convolutional neural networks (CNNs) have shown unprecedented success in object representation and detection. Nevertheless, CNNs lack the capability to model context dependencies among objects, which are crucial for salient object detection. As the long short-term memory (LSTM) is advantageous in propagating information, in this paper, we propose two variant LSTM units for the exploration of contextual dependencies. By incorporating these units, we present a context-aware network (CAN) to detect salient objects in RGB-D images. The proposed model consists of three components: feature extraction, context fusion of multiple modalities and context-dependent deconvolution. The first component is responsible for extracting hierarchical features in color and depth images using CNNs, respectively. The second component fuses high-level features by a variant LSTM to model multi-modal spatial dependencies in contexts. The third component, embedded with another variant LSTM, models local hierarchical context dependencies of the fused features at multi-scales. Experimental results on two public benchmark datasets show that the proposed CAN can achieve state-of-the-art performance for RGB-D stereoscopic salient object detection.

Deep learning models for bridge deck evaluation using impact echo

Impact echo (IE) is a common nondestructive evaluation (NDE) method to detect subsurface defects in concrete bridge decks. The conventional approach for analyzing the IE data requires user expertise to define analysis parameters that could hinder broad field implementation. In this paper, the feasibility of using deep learning models (DLMs) for autonomous subsurface defect detection in bridge decks using IE has been investigated. A set of eight laboratory-made reinforced concrete bridge specimens with artificial defects were constructed at the Federal Highway Administration Advanced Sensing Technology NDE laboratory. A total number of 2016 of IE data was collected from these specimens. A one-dimensional (1D) convolutional neural network (CNN), and a 1D recurrent neural network using bidirectional long-short term memory units, were developed and applied on the IE data. In addition, two-dimensional (2D) world renowned CNNs were applied on the 2D representatives of the IE data, i.e., spectrograms. The proposed 1D CNN was the most accurate model achieving an overall accuracy of 0.88 by classifying 0.70 of the defects and 0.95 of the sound regions correctly. The proposed 1DCNN was superior to previous machine learning models that were previously used for IE classification. The results of this study showed the feasibility and the potentials of the DLMs for subsurface defect detection.