Long Short-term Memory Variants Research Articles

Forecasting Indian Trade Trends through LSTM- based Predictive Modeling

The efficacy of Long Short-Term Memory (LSTM) neural networks and attention-based models in predicting next-day closing prices of the MSFT 500index is meticulously examined. A comprehensive suite of nine carefully chosen predictors spanning fundamental market data, macroeconomic indicators, and technical metrics is amalgamated, fostering a holistic comprehension of market behavior. Through rigorous analysis, the research evaluates single-layer and multilayer LSTM architectures alongside attention- based LSTM variants, juxtaposed against traditional ARIMA models. Surprisingly, the single-layer LSTM consistently outperforms its multilayer counterpart, demonstrating superior accuracy and model fit. The integration of corporate accounting statistics augments predictive capabilities, enriching the models' efficacy. Notably, attention-based LSTM models, particularly the Attention-LSTM variant, exhibit markedly lower prediction errors and higherreturns in trading strategies compared to other methodologies. However, the heightened complexity of stacked-LSTM structures fails to surpass the predictive acumen of simpler LSTM architectures.This inquiry underscores the paramount importance of leveraging advanced AI techniques and comprehensive datasets in navigating the intricate nuances of modern financialmarkets, offering invaluable insights for both researchers and practitioners engaged in stock priceforecasting endeavors.

Analysis & Stock Price Prediction and Forecasting Using Different LSTM Models

The objective of this research is to develop a Deep Learning model to forecast the stock price, by using the variant of Long Short-Term Memory. This model predicts the close price of the stock for the future selected date, choosing as inputs the following data: open, high, low, adj close and close prices. This model shows a comparative analysis between three different LSTM networks: Long Short-Term Memory (LSTM), Stacked Long Short-Term Memory (Stacked LSTM), and Stacked Bi-directional Long Short-Term Memory (Stacked Bidirectional LSTM) concluding which one is the best and implementing the model using that variant. We have used the historical stock prices data from yahoo’s financial website over 5 years, by choosing multiple datasets: Apple, Amazon, Google, Meta, Microsoft and Tesla (daily values). In order to get effective in the forecasting model, we have tested the network with different iterations and epochs. The model represents Multiple Graphs for data visualization in different comparisons. We have estimated the effectiveness of our proposed model by using the following performance indicators: the Mean Square Error (MSE), the Root Mean Square Error (RMSE), and the R-Squared of the model. The experimental results clearly show that our Stacked Bi-LSTM model has the highest accuracy values when comparing with the LSTM and Stacked LSTM Models. Hence, we can conclude that our Stacked Bi-LSTM Model is suitable for accurate prediction of the stock market time series. Key Words: stock price prediction, Machine Learning, stacked LSTM, Bi-directional LSTM, Deep Learning, Data pre-processing techniques, Data normalization, Data Visualization, Training and Testing Set, Financial Time Series, Future prediction

Exploring the development and application of LSTM variants

Long Short-Term Memory (LSTM) is receiving increasing attention as the development of deep learning technology. The gate structure of LSTM enhances long-term memory, forming its superior capacity to complete tasks that challenge traditional RNN. However, considering the wide variety of applications, a comprehensive understanding of the development and application of the model, which is vital for future research, is comparatively lacking. Therefore, this paper is produced with the hope of offering an overview of the development of LSTM. It shows the process of development from RNN to LSTM and explains the aim and necessity of LSTMs birth. After that it introduces the structure of LSTM, analyses its advantages over RNN, and discusses the application of some popular LSTM variants, such as peephole LSTM, bidirectional LSTM, and GRU. Hopefully, this work can provide a more profound knowledge of LSTM's benefits and potential, identifying worthwhile avenues or fields of future research.

A Novel Variant of LSTM Stock Prediction Method Incorporating Attention Mechanism

Long Short-Term Memory (LSTM) is an effective method for stock price prediction. However, due to the nonlinear and highly random nature of stock price fluctuations over time, LSTM exhibits poor stability and is prone to overfitting, resulting in low prediction accuracy. To address this issue, this paper proposes a novel variant of LSTM that couples the forget gate and input gate in the LSTM structure, and adds a “simple” forget gate to the long-term cell state. In order to enhance the generalization ability and robustness of the variant LSTM, the paper introduces an attention mechanism and combines it with the variant LSTM, presenting the Attention Mechanism Variant LSTM (AMV-LSTM) model along with the corresponding backpropagation algorithm. The parameters in AMV-LSTM are updated using the Adam gradient descent method. Experimental results demonstrate that the variant LSTM alleviates the instability and overfitting issues of LSTM, effectively improving prediction accuracy. AMV-LSTM further enhances accuracy compared to the variant LSTM, and compared to AM-LSTM, it exhibits superior generalization ability, accuracy, and convergence capability.

Quantum Convolutional Long Short-Term Memory Based on Variational Quantum Algorithms in the Era of NISQ

In the era of noisy intermediate-scale quantum (NISQ) computing, the synergistic collaboration between quantum and classical computing models has emerged as a promising solution for tackling complex computational challenges. Long short-term memory (LSTM), as a popular network for modeling sequential data, has been widely acknowledged for its effectiveness. However, with the increasing demand for data and spatial feature extraction, the training cost of LSTM exhibits exponential growth. In this study, we propose the quantum convolutional long short-term memory (QConvLSTM) model. By ingeniously integrating classical convolutional LSTM (ConvLSTM) networks and quantum variational algorithms, we leverage the variational quantum properties and the accelerating characteristics of quantum states to optimize the model training process. Experimental validation demonstrates that, compared to various LSTM variants, our proposed QConvLSTM model outperforms in terms of performance. Additionally, we adopt a hierarchical tree-like circuit design philosophy to enhance the model’s parallel computing capabilities while reducing dependence on quantum bit counts and circuit depth. Moreover, the inherent noise resilience in variational quantum algorithms makes this model more suitable for spatiotemporal sequence modeling tasks on NISQ devices.

FlightForecast: A Comparative Analysis of Stack LSTM and Vanilla LSTM Models for Flight Prediction

The Coronavirus was first reported in China in the city of Wuhan in December 2019, after a couple of months, it was widespread around the world. The whole world was in a state of lockdown. This hazardous disease affects the normal daily life of every individual and the tourism industry, especially the airline business was at a greater loss. Considering the airline business, this study contains data on commercial flights from 2019 to 2020. The conducted research analyzed the rise and fall of different flights in the lockdown period. The research is based on the variants of Long Short-Term Memory (LSTM) such as standard Recurrent Neural Network (RNN) and stack LSTM. The comparative research shows that the prediction of the stack LSTM model is better than the standard RNN keeping view of taking a considerable amount of time to train.

Emergency sign language recognition from variant of convolutional neural network (CNN) and long short term memory (LSTM) models

Sign language is the primary communication tool used by the deaf community and people with speaking difficulties, especially during emergencies. Numerous deep learning models have been proposed to solve the sign language recognition problem. Recently. Bidirectional LSTM (BLSTM) has been proposed and used in replacement of Long Short-Term Memory (LSTM) as it may improve learning long-team dependencies as well as increase the accuracy of the model. However, there needs to be more comparison for the performance of LSTM and BLSTM in LRCN model architecture in sign language interpretation applications. Therefore, this study focused on the dense analysis of the LRCN model, including 1) training the CNN from scratch and 2) modeling with pre-trained CNN, VGG-19, and ResNet50. Other than that, the ConvLSTM model, a special variant of LSTM designed for video input, has also been modeled and compared with the LRCN in representing emergency sign language recognition. Within LRCN variants, the performance of a small CNN network was compared with pre-trained VGG-19 and ResNet50V2. A dataset of emergency Indian Sign Language with eight classes is used to train the models. The model with the best performance is the VGG-19 + LSTM model, with a testing accuracy of 96.39%. Small LRCN networks, which are 5 CNN subunits + LSTM and 4 CNN subunits + BLSTM, have 95.18% testing accuracy. This performance is on par with our best-proposed model, VGG + LSTM. By incorporating bidirectional LSTM (BLSTM) into deep learning models, the ability to understand long-term dependencies can be improved. This can enhance accuracy in reading sign language, leading to more effective communication during emergencies.

AI-Based System Automates Textual Classification of Daily Drilling Reports

_ This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper OTC 32978, “Development and Implementation of an AI-Based System To Automate Textual Classification on Daily Drilling Reports,” by Stephan Perrout, Aliel F. Riente, and Guilherme S.F. Vanni, SPE, Petrobras, et al. The paper has not been peer reviewed. Copyright 2023 Offshore Technology Conference. Reproduced by permission. _ Structured daily drilling reports (DDRs) are a rich source of information that allows better planning, more-accurate risk analysis, and improved key performance indicators and contracts. However, such information is originally stored in a free-text and unstructured format, which becomes difficult for efficient data mining. With the advance of artificial intelligence (AI) technologies, particularly AI language models, applying such techniques over unstructured data has become critical to digital transformation. The complete paper presents an approach for automatic DDR classification that incorporates new techniques of AI. Introduction This work addresses the complex task of automatic classification of DDRs according to a newly proposed ontology. The ontology follows a hierarchical model that classifies actions into three or four levels depending on the intervention, considering drilling, completion, and abandonment. Each event has an ontology built and reviewed by experts in oil and gas. Classifying DDR constitutes a demanding task, and effectively exploiting AI-based models represents a promising solution. This work bridges the gap by proposing a classifier based on transformers along with recurrent neural networks (RNNs) to classify reported events described in unstructured text related to drilling, completion, and abandonment interventions. A large number of DDRs was used for training and validation of the proposed classifier, yielding promising results for key processes in the company. Neural-Network Techniques Bidirectional Long Short-Term Memory. Early neural-network models are characterized by inputs of fixed length. This is a drawback when working with texts, however, because sentences vary in their number of words. To overcome such an issue and to process data sequentially, RNNs were proposed. The RNNs are characterized by a set of parameters inherent from the early models plus an internal memory (a hidden or internal state) responsible for storing the context of the sequence being processed. Long short-term memory (LSTM) is a variation of RNN proposed to mitigate two problems: Information can be easily lost when processing very long sequences, and the gradient can become quite low because of the high number of mathematical operations performed during the processing while remaining far from reaching the threshold. LSTM consists of a set of parameters called the input gate, forget gate, and output gate that control information flow through the network. This set of additional parameters helps to maintain only what is important for the internal state of the network besides controlling the output. BiLSTM is a variant of LSTM that comprises two LSTMs. One processes texts from left to right, and the second one processes texts from right to left. This feature allows “future” elements to be part of the model’s decision process for “past” elements. The final classification is the combination of the output of both LSTMs.

Stratification of Depressed and Non-Depressed Texts from Social Media using LSTM and its Variants

This work examines the performance of various LSTM (long short-term memory) variants on social media text data. This study evaluates the performance of LSTM models with different architectures, namely, classic LSTM, Bidirectional LSTM, Stacked LSTM, gated recurrent unit (GRU), and bidirectional GRU, on a social network dataset comprising texts extracted from multiple social media platforms. We aim to identify the most effective LSTM variant of the five considered LSTM models for text analysis through a comparative study of the models’ precision, recall, F1-score, and accuracy. The research findings show that the Classic LSTM and the GRU model perform better than the other models in accuracy. In contrast, the bidirectional models (Bidirectional LSTM and Bidirectional GRU) provide better precision scores than their respective primitive models. This research has significant implications for developing more efficient models for natural language processing applications. It offers beneficial insights into the implications involving the scrutiny of depression on social media platforms through text data analysis.

Long Short-Term Memory-based Models for Sleep Quality Prediction from Wearable Device Time Series Data

Several studies suggest that sleep quality is associated with physical activities. Moreover, deep sleep time can be used to determine the sleep quality of an individual. In this work, we aim to find the association between physical activities and deep sleep time by modeling the time series data such as heart rate and a number of steps captured from a commercial wearable device. Our previous study demonstrates that deep learning-based time series modeling is well suited for our problem since the temporal patterns in the two physical parameters need to be captured to obtain more accurate results. We first preprocess our series data to have a time-step size of 10 minutes. To improve our previous effort in this modeling, we compare four different variants of Long Short-Term Memory (LSTM)-based models, ranging from single input to dual input models. Our result shows that the simple stacked LSTM model performs better for our data because the remaining models suffer from overfitting due to a larger number of the trained parameters.

Deep learning-based algorithms for long-term prediction of chlorophyll-a in catchment streams

Accurate estimation of harmful algal blooms is imperative for the protection of surface water. Chlorophyll-a (Chl-a) which is used as a proxy for estimating the algal concentration, is affected by a wide range of weather and physicochemical factors that act at varying spatial and temporal scales. Deep learning (DL) based models such as Long-Short Term Memory (LSTM) and Convolution Neural Networks (CNNs) have shown promising results for hydrological and Chl-a simulations. Recently several variants of LSTM and CNNs have been developed which can model highly non-linear relationships between input and target data. Therefore, these advanced DL methods have the potential for long-term simulation of Chl-a. Previous DL-based studies on Chl-a simulation have developed site-dependent models. This indicates that the DL models were trained and evaluated using data from the same site, making it difficult to apply these models to other sites. Development of site-independent models requires a more robust training strategy which can result in DL models that can be evaluated in new novel situations. To address these issues, we propose a DL-based framework which can incorporate irregularly measured water quality observations, static physical features, and climate data measured at constant time steps. In this framework, we compared the performance of six state of the art DL methods which include (1) LSTM, (2) CNN, (3) Temporal Convolution Networks (TCN), (4) CNN-LSTM, (5) LSTM based autoencoder, and (6) input-attention LSTM (IA-LSTM). The IA-LSTM is an explainable DL method which can select important hydrologic, climatic and water quality parameters for Chl-a prediction. Our results indicate that the IA-LSTM exhibited the best performance, with an R2 value of 0.85 at the training site and 0.52 at the test site. We showed that attention-based deep learning models improve the prediction performance and make the black-box deep learning models interpretable and explainable. The attention-based deep learning models indicated that Chl-a concentration in the Nakdong River was strongly affected by climate factors during the previous three days. The proposed DL framework can be adopted to develop regional water quality models using deep learning.

Remaining Useful-Life Prediction of the Milling Cutting Tool Using Time–Frequency-Based Features and Deep Learning Models

The milling machine serves an important role in manufacturing because of its versatility in machining. The cutting tool is a critical component of machining because it is responsible for machining accuracy and surface finishing, impacting industrial productivity. Monitoring the cutting tool’s life is essential to avoid machining downtime caused due to tool wear. To prevent the unplanned downtime of the machine and to utilize the maximum life of the cutting tool, the accurate prediction of the remaining useful life (RUL) cutting tool is essential. Different artificial intelligence (AI) techniques estimate the RUL of cutting tools in milling operations with improved prediction accuracy. The IEEE NUAA Ideahouse dataset has been used in this paper for the RUL estimation of the milling cutter. The accuracy of the prediction is based on the quality of feature engineering performed on the unprocessed data. Feature extraction is a crucial phase in RUL prediction. In this work, the authors considers the time–frequency domain (TFD) features such as short-time Fourier-transform (STFT) and different wavelet transforms (WT) along with deep learning (DL) models such as long short-term memory (LSTM), different variants of LSTN, convolutional neural network (CNN), and hybrid models that are a combination of CCN with LSTM variants for RUL estimation. The TFD feature extraction with LSTM variants and hybrid models performs well for the milling cutting tool RUL estimation.

Hybrid deep neural network with dimension attention for state-of-health estimation of Lithium-ion Batteries

Lithium-ion batteries (LIBs) are widely used and became the main energy storage medium for many devices. Accurate estimation of LIBs state-of-health (SOH) is crucial for safe and reliable operation of devices. This study designs an end-to-end multi-battery shared hybrid neural network (NN) prognostic framework that combines a convolutional neural network (CNN), a multi-layer variant long-short-term memory (VLSTM) NN and a dimensional attention mechanism (CNN-VLSTM-DA) to SOH estimation for LIBs. First, feature extraction and selection on the raw input data are performed by using a CNN. Second, a suitable VLSTM is designed. The network adds a “peephole connection” to the forget gate and output gate, respectively, which enhances the network’s ability to distinguish subtle features between input sequences. Besides, the forget gate and the input gate are coupled, so that, together, they determine the information that needs to be forgotten and the new data that needs to be added. Then, the output data of the CNN layer are fed into a multi-layer VLSTM NN to further capture the temporal correlation of these data. Finally, the attention mechanism is applied to the output of the VLSTM, to assign different weights to the features of each dimension and to give the prediction results. Several experiments are carried out on three datasets from NASA, CALCE and Oxford. These include full charge/discharge data, charge/discharge data in different SOC ranges, and non-fixed discharge current data. The results verify the effectiveness of the proposed method.

Incorporating Heterogeneous User Behaviors and Social Influences for Predictive Analysis

Behavior prediction based on historical behavioral data have practical real-world significance. It has been applied in recommendation, predicting academic performance, etc. With the refinement of user data description, the development of new functions, and the fusion of multiple data sources, heterogeneous behavioral data which contain multiple types of behaviors become more and more common. In this paper, we aim to incorporate heterogeneous user behaviors and social influences for behavior predictions. To this end, this paper proposes a variant of Long-Short Term Memory (LSTM) which can consider context information while modeling a behavior sequence, a projection mechanism which can model multi-faceted relationships among different types of behaviors, and a multi-faceted attention mechanism which can dynamically find out informative periods from different facets. Many kinds of behavioral data belong to spatio-temporal data. An unsupervised way to construct a social behavior graph based on spatio-temporal data and to model social influences is proposed. Moreover, a residual learning-based decoder is designed to automatically construct multiple high-order cross features based on social behavior representation and other types of behavior representations. Qualitative and quantitative experiments on real-world datasets have demonstrated the effectiveness of this model.

Sentiment classification using bidirectional LSTM-SNP model and attention mechanism

Aspect-level sentiment classification still remains a challenge: how to capture contextual semantic correlation between aspect word and content words more effectively. LSTM-SNP is a variant of long short-term memory (LSTM), inspired from nonlinear spiking mechanisms in nonlinear spiking neural P systems. To address the challenge, we develop a modification of LSTM-SNP and design a bidirectional LSTM-SNP based on the modification, termed as BiLSTM-SNP. Based on BiLSTM-SNP and attention mechanism, we propose a novel method for aspect-level sentiment classification. BiLSTM-SNP is used to capture semantic correlation between aspect word and content words, while attention mechanism is utilized to generate appropriate attention weights for hidden states of BiLSTM-SNP. Experiments on English and Chinese data sets are conducted on the proposed model and several baseline models. Experiment results on English and Chinese data sets demonstrate the effectiveness of the proposed model for aspect-level sentiment classification.

A Comparative Study of LSTM Variants in Prediction for Tesla’s Stock Price

Long short-term memory (LSTM) is widely used in the stock market to train the prediction model and forecast future stock prices. Applying the LSTM method to research may incur some problems and facilitate the improvement of the method. Therefore, many LSTM variants are put forward under different circumstances. This paper surveys four LSTM variants, including Vanilla, Stacked, Bi-directional, and CNN LSTM on two different data sets regarding Tesla's stock price. Two data sets mentioned in this paper represent different stock types. To be more specific, data set 1 refers to stocks with a single long-term trend, while data set 2 can be seen as an example of stocks with more complexity. The result shows that the Vanilla LSTM reaches the highest prediction accuracy on the data set without any irregular shift in the long-term trend. CNN LSTM also provides decent predictions for the stock price. Otherwise, the Stacked LSTM performs the best for stock prediction. Bi-LSTM and CNN LSTM are also suitable for stock forecasting in more complicated situations. The change in preference for model selection proves that a company's operation situation and market circumstances also influence the prediction performance of LSTM variants.

Climatological parameters estimation based on artificial intelligence techniques with particle swarm optimization and deep neural networks

Climate forecasting plays an important role for human life in many areas such as water management, agriculture, natural hazards including drought and flood, tourism, business, and regional investment. Estimating these data is a difficult task as the time series climate parameter values vary monthly and seasonally. Therefore, predicting climate parameters based on learning and artificial intelligence is important to long-term efficient results in these fields. For this purpose, in this study, a time-series based Long Short-Term Memory (LSTM) deep neural network is proposed to predict future climate in Çankırı and Adıyaman cities in Turkey. With the help of this network, the average temperature, relative humidity, and precipitation values, which are known as the most effective climate parameters, have been estimated. An improved Particle Swarm Optimization (PSO) technique is also proposed to optimize input weight values of the LSTM deep network, and reduce the estimation errors. The proposed algorithm is compared with deep models of LSTM variants based on Root Mean Square Error (RMSE), Mean Absolute Deviation (MADE), and Mean Absolute Percentage Error (MAPE) metrics. The proposed adaptive LSTM-PSO and non-adaptive LSTM-PSO models achieved at RMSE 0.98 and 1.05 for temperature, 1.19 and 1.27 for relative humidity, and 4.21 and 4.67 for precipitation estimation, respectively. The RMSE is %7 lower with the proposed adaptive LSTM-PSO method than proposed non-adaptive LSTM-PSO method.

Warehouse LSTM-SVM-Based ECG Data Classification With Mitigated Device Heterogeneity

Device heterogeneity is a social concern, especially in healthcare domain. In this work, we mitigate the problem of device heterogeneity and further classify the healthcare electrocardiogram (ECG) data with improved performance using a proposed variant of long short-term memory (LSTM). ECG data sensed from different devices are used in this work for experimentation. Device heterogeneity is addressed using the proposed multiplicative convergence-based heterogeneity mitigation (MCHM) method. The proposed warehouse LSTM, in addition to support vector machine (SVM), is leveraged in this work for healthcare data classification. The warehouse LSTM keeps a track of the data that are considered as insignificant in the initial epoch. We mitigate heterogeneity in medical devices and reduce the root-mean-squared error to the order of 10−6–10−5. Using the warehouse, the LSTM-SVM attains the classification accuracies of 98.34% and 96.27% during training on the MIT-BIH and PTB datasets, respectively. The proposed MCHM method increases the reliability on the usage of devices from multiple manufacturers. The novel warehouse LSTM-SVM model also outperforms the existing methods for classification of data.

Deep learning rainfall–runoff predictions of extreme events

Abstract. The most accurate rainfall–runoff predictions are currently based on deep learning. There is a concern among hydrologists that the predictive accuracy of data-driven models based on deep learning may not be reliable in extrapolation or for predicting extreme events. This study tests that hypothesis using long short-term memory (LSTM) networks and an LSTM variant that is architecturally constrained to conserve mass. The LSTM network (and the mass-conserving LSTM variant) remained relatively accurate in predicting extreme (high-return-period) events compared with both a conceptual model (the Sacramento Model) and a process-based model (the US National Water Model), even when extreme events were not included in the training period. Adding mass balance constraints to the data-driven model (LSTM) reduced model skill during extreme events.

Deep fake detection using a sparse auto encoder with a graph capsule dual graph CNN.

Deepfake (DF) is a kind of forged image or video that is developed to spread misinformation and facilitate vulnerabilities to privacy hacking and truth masking with advanced technologies, including deep learning and artificial intelligence with trained algorithms. This kind of multimedia manipulation, such as changing facial expressions or speech, can be used for a variety of purposes to spread misinformation or exploitation. This kind of multimedia manipulation, such as changing facial expressions or speech, can be used for a variety of purposes to spread misinformation or exploitation. With the recent advancement of generative adversarial networks (GANs) in deep learning models, DF has become an essential part of social media. To detect forged video and images, numerous methods have been developed, and those methods are focused on a particular domain and obsolete in the case of new attacks/threats. Hence, a novel method needs to be developed to tackle new attacks. The method introduced in this article can detect various types of spoofs of images and videos that are computationally generated using deep learning models, such as variants of long short-term memory and convolutional neural networks. The first phase of this proposed work extracts the feature frames from the forged video/image using a sparse autoencoder with a graph long short-term memory (SAE-GLSTM) method at training time. The first phase of this proposed work extracts the feature frames from the forged video/image using a sparse autoencoder with a graph long short-term memory (SAE-GLSTM) method at training time. The proposed DF detection model is tested using the FFHQ database, 100K-Faces, Celeb-DF (V2) and WildDeepfake. The evaluated results show the effectiveness of the proposed method.