Short-term Memory Autoencoder Research Articles

Uncertainty-Aware Time Series Anomaly Detection

Traditional anomaly detection methods in time series data often struggle with inherent uncertainties like noise and missing values. Indeed, current approaches mostly focus on quantifying epistemic uncertainty and ignore data-dependent uncertainty. However, consideration of noise in data is important as it may have the potential to lead to more robust detection of anomalies and a better capability of distinguishing between real anomalies and anomalous patterns provoked by noise. In this paper, we propose LSTMAE-UQ (Long Short-Term Memory Autoencoder with Aleatoric and Epistemic Uncertainty Quantification), a novel approach that incorporates both aleatoric (data noise) and epistemic (model uncertainty) uncertainties for more robust anomaly detection. The model combines the strengths of LSTM networks for capturing complex time series relationships and autoencoders for unsupervised anomaly detection and quantifies uncertainties based on the Bayesian posterior approximation method Monte Carlo (MC) Dropout, enabling a deeper understanding of noise recognition. Our experimental results across different real-world datasets show that consideration of uncertainty effectively increases the robustness to noise and point outliers, making predictions more reliable for longer periodic sequential data.

Long Short-Term Memory Autoencoder for Anomaly Detection in Rails Using Laser Doppler Vibrometer Measurements

Abstract This study presents an application of Long Short-Term Memory Autoencoder (LSTM AE) for the detection of broken rails based on laser Doppler Vibrometer (LDV) measurements. This work is part of an ongoing project aimed at developing a non-contact damage detection system using LDV measurements. The damage detection system consists of two laser Doppler vibrometers (LDV) mounted on a moving rail car to measure vibrations induced on the rail head. Field tests were carried out at the Transportation Technology Center (TTC) in Pueblo, CO, to collect the vibrational data. This study focused on the detection of broken rails. To simulate the reflected and transmitted waves induced by the broken rail, a welded joint was used. The data was collected from moving LDV measurements, in which the train was operating at three different speeds: 16km/h (10mph), 32km/h (20mph), and 48km/h (30mph). After obtaining the data, filtering and signal processing were applied to obtain the signal features in time and frequency domains. Next, correlation analysis and principal component analysis were carried out for feature selection and dimension reduction to determine the input used to train and test the LSTM AE model. In this study, the LSTM AE models were trained based on different data sets for anomaly detection. Consequently, an automatic anomaly detection approach for anomaly detection based on the LSTM AE model was evaluated. The results show that the LSTM AE model can efficiently detect the anomaly based on the selected features at three different speeds.

Physical model learning based false data injection attack on power system state estimation

The cyber security of power system state estimation (PSSE) is crucial, and its robustness against evolving false data injection attacks (FDIA) is being rigorously assessed to develop advanced countermeasures. Existing FDIA methods have achieved satisfactory success rates but often fail to align with practical constraints such as the assumption of partial or complete knowledge of the power system network by the attacker, modifications in generator output measurements, and the sparsity of the attacks. This work proposes a near practical, stealthy approach using a deep generative adversarial network-long short-term memory autoencoder (GAN-LSTMAE) learning based sparse FDIA method against AC PSSE, leveraging only measurement data. To evade the bad data detection (BDD) mechanism effectively, an LSTMAE-based PSSE mimic is proposed, further optimizing the GAN-based attack generator to embed the physical laws of the system along with measurement residuals and temporal dependencies of states to the generated false data. The proposed modified training data preparation algorithm, coupled with the attack sub-graph method, defines the optimal attack region while keeping generator output measurements intact. The generated attack is validated extensively using IEEE 14 and 118 bus test benchmarks against various defense techniques, demonstrating high success rates.

Diagnosis of electrical submersible pump failure using deep learning model with sand-water flow experimental data

Reliability of electrical submersible pump (ESP) failure diagnosis is crucial because unexpected failures can lead to additional production costs. This study presents a deep-learning method based on a long short-term memory autoencoder (LSTM-AE) model with principal component analysis (PCA) for ESP failure diagnosis. To obtain data on variables related to ESP failure, a sand–water flow experiment was designed and conducted. An LSTM-AE model was then developed based on the PCA of the experimental data, demonstrating a failure diagnosis accuracy of 90.69%, which is higher than that of the LSTM-AE model without PCA. The failure-detection point was predicted, closely aligning with the initial point of failure. To assess its suitability for field applications, the proposed LSTM-AE model with PCA was tested with data from the Sandy 03 well in the Permian Basin, USA. The LSTM-AE model with PCA achieved a failure diagnosis accuracy of 81.81%, and the initial failure detection point was accurate. These results indicate that the LSTM-AE model with PCA can effectively capture long-term dependencies in time-series data and provide reliable ESP failure diagnosis. By accurately identifying potential failures, this approach offers significant potential for improving operational efficiency and reducing maintenance costs in ESP systems.

Joint Optimization-Based QoS and PAPR Reduction Technique for Energy-Efficient Massive MIMO System

Massive multiple input multiple output (MIMO) is an innovative wireless communication technology that significantly enhances the capacity and efficiency of data transmission in modern networks. Massive multiple input multiple output, despite benefits, faces difficulties in managing numerous antennas affecting signal quality, peak-to-average power ratio (PAPR), and energy efficiency due to its scale and complexity. However, overcoming these complexities is vital for unlocking massive multiple input multiple output’s potential in high-quality, energy-efficient wireless communication. A proposed joint optimization-based technique aims to address these issues in massive MIMO systems. In phase 1, joint optimization with quality of service maximizes capacity via power distribution, employing the hybrid spider wasp Fick’s law algorithm. In phase 2, zero-forcing with symbol-level linear precoding, especially the stacked convolutional sparse bidirectional long short-term memory autoencoder (SCS–BiLSTMAE) network, efficiently reduces peak-to-average power ratio, collaborating for reduced bit rate error and peak-to-average power ratio through adaptive mapping. Refining the model involves precise hyperparameter tuning using the enhanced influencer buddy optimization algorithm. The proposed framework not only demonstrates exceptional performance, but the metrics also underscore the model's effectiveness in addressing diverse challenges, establishing it as a robust solution for quality enhancement, peak-to-average power ratio reduction, and energy efficiency. Additionally, the proposed model attains 28.14% greater system capacity than existing methods.

Deep learning-based source term estimation of hydrogen leakages from a hydrogen fueled gas turbine

Hydrogen fueled gas turbine is an efficient and environmentally friendly energy conversion equipment. However, it is prone to gas leakages from corrosion-prone components and pipe connections. In order to address gas leakage problems, source term estimation (STE) can be applied to estimate the location and intensity of the leakage sources. Traditional STE methods are based on an optimization procedure using an atmospheric transport and dispersion model, which requires considerable computation efforts and cannot meet the need of real-time implementation. The complex flow field around the gas turbine, the possible existence of multiple leakage sources, and the high-dimensional time series data further increase the difficulty of the STE problem. To address these challenges, in the present study, a deep learning based STE approach is proposed. The basic idea is to use long short-term memory auto-encoder (LSTM-AE) network to extract the encoded features of multi-sensor data and then use a deep neural network (NN) to establish the relationship between the encoded features and the source term parameters of hydrogen leakages. The multi-sensor data are generated using computational fluid dynamics (CFD) analysis to simulate relevant hydrogen leakage scenarios of concern. The results show that compared with other machine learning algorithms, the proposed approach has an overall best performance in terms of the STE accuracy. Specifically, its hydrogen leakage source localization accuracy is 0.9798, and the leakage strength estimation R-squared (R2) can reach 0.9632. When the training data proportion is only 20–30%, the proposed approach can still perform well, indicating the ability of the model to effectively predict the location and strength of the leakage source even with relatively less training data.

Sensor Fault Detection and Classification Using Multi-Step-Ahead Prediction with an Long Short-Term Memoery (LSTM) Autoencoder

The Internet of Things (IoT) is witnessing a surge in sensor-equipped devices. The data generated by these IoT devices serve as a critical foundation for informed decision-making, real-time insights, and innovative solutions across various applications in everyday life. However, data reliability is often compromised due to the vulnerability of sensors to faults arising from harsh operational conditions that can adversely affect the subsequent operations that depend on the collected data. Hence, the identification of anomalies within sensor-derived data holds significant importance in the IoT context. This article proposes a sensor fault detection method using a Long Short-Term Memory autoencoder (LSTM-AE). The AE, trained on normal sensor data, predicts a 20-step window, generating three statistical features via SHapley Additive exPlanations from the estimated steps. These features aid in determining potential faults in the predicted steps using a machine learning classifier. A secondary classifier identifies the type of fault in the sensor signal. Experimentation on two sensor datasets showcases the method’s functionality, achieving fault detection accuracies of approximately 93% and 97%. It is possible to attain a perfect fault classification performance by slightly modifying the feature calculation approach. In a univariate prediction scenario, our proposed approach demonstrates good fault detection and classification performance.

Detection of Harmful H2S Concentration Range, Health Classification, and Lifespan Prediction of CH4 Sensor Arrays in Marine Environments

Underwater methane (CH4) detection technology is of great significance to the leakage monitoring and location of marine natural gas transportation pipelines, the exploration of submarine hydrothermal activity, and the monitoring of submarine volcanic activity. In order to improve the safety of underwater CH4 detection mission, it is necessary to study the effect of hydrogen sulfide (H2S) in leaking CH4 gas on sensor performance and harmful influence, so as to evaluate the health status and life prediction of underwater CH4 sensor arrays. In the process of detecting CH4, the accuracy decreases when H2S is found in the ocean water. In this study, we proposed an explainable sorted-sparse (ESS) transformer model for concentration interval detection under industrial conditions. The time complexity was decreased to O (n logn) using an explainable sorted-sparse block. Additionally, we proposed the Ocean X generative pre-trained transformer (GPT) model to achieve the online monitoring of the health of the sensors. The ESS transformer model was embedded in the Ocean X GPT model. When the program satisfied the special instructions, it would jump between models, and the online-monitoring question-answering session would be completed. The accuracy of the online monitoring of system health is equal to that of the ESS transformer model. This Ocean-X-generated model can provide a lot of expert information about sensor array failures and electronic noses by text and speech alone. This model had an accuracy of 0.99, which was superior to related models, including transformer encoder (0.98) and convolutional neural networks (CNN) + support vector machine (SVM) (0.97). The Ocean X GPT model for offline question-and-answer tasks had a high mean accuracy (0.99), which was superior to the related models, including long short-term memory–auto encoder (LSTM–AE) (0.96) and GPT decoder (0.98).

Detecting APS failures using LSTM-AE and anomaly transformer enhanced with human expert analysis

This study develops a novel semi-supervised approach for detecting Air Pressure System (APS) failures in Heavy-Duty Vehicles (HDVs) by exploiting two modern Machine Learning (ML) models: Long Short-Term Memory Autoencoder (LSTM-AE) and Transformer for Anomaly Detection (TranAD), and enhancing their performance with human expertise. To tackle the failure detection problem, a dataset comprising 30 days of operational time-series data from 110 healthy vehicles with no recorded APS issues and 30 vehicles that experienced APS failures requiring road assistance was acquired. Several preprocessing steps are proposed and three key features are extracted as APS health indicators. These features are then utilized both in human expert analysis (HEA) and training of ML models. When compared to HEA, both LSTM-AE and TranAD models exhibit superior performance individually in APS failure detection, achieving F1 scores of 0.75 and 0.79 respectively, and the same accuracy of 91.4%. Further, the integration of HEA with those ML models enhances model effectiveness in all experimental results, especially in reducing false alarms that cause customer dissatisfaction. The TranAD model combined with human expert analysis achieved the best performance with an unprecedented 0.82 F1 score and 92.8% accuracy. In addition to presenting a new methodology for failure detection, this paper suggests a way for more efficient and reliable predictive maintenance practices for HDVs.

Adaptive Modem Based on LSTM-AutoEncoder with Vector Quantization

Recently, researchers have achieved the goal of using unified architecture to achieve multiple modulation modes under specific conditions. However, existing research still suffers from the problem of large resource and time overhead. This paper proposes an adaptive modem based on a vector quantization (VQ) long short-term memory autoencoder (LSTM-AE), designed to implement modulation and demodulation of signals from the second to thirty-second order in a lightweight way. By leveraging the memory capacity of the LSTM module and the compression capability of the autoencoder, the model is able to support multi-order modulation methods. This study used the Adam optimizer for training and testing on a simple dataset extended by adding AWGN noise only and made modifications based on the MSE loss function to balance the accuracy and training speed of each part of the model. Experiments demonstrate that the proposed method not only achieves comparable modem performance to existing frameworks for second to thirty-second order signals, but also significantly reduces the number of parameters and training time. Experimental results indicate that the proposed methodology not only matches the performance of existing frameworks for signals ranging from the second to the thirty-second order, but also employs merely 79.6% of the average parameter count and a mere 7.4% of the average training duration. This represents a substantial reduction in resource expenditure.

RETRACTED: Enhancing Data Quality Management in Structural Health Monitoring through Irregular Time-Series Data Anomaly Detection Using IoT Sensors

The importance of monitoring in assessing structural safety and durability continues to grow. With recent technological advancements, Internet of Things (IoT) sensors have garnered attention for their complex scalability and varied detection capabilities, becoming essential devices for monitoring. However, during the data collection process of IoT sensors, anomalies arise due to network instability, sensor noise, and malfunctions, degrading data quality and compromising monitoring system reliability. In this study, Interquartile Range (IQR), Long Short-Term Memory Autoencoder (LSTM-AE), and time-series decomposition were employed for anomaly detection in Structural Health Monitoring (SHM) processes. IQR and LSTM-AE produce irregular patterns; however, time-series decomposition effectively detects such anomalies. In road monitoring influenced by weather and traffic, the time-series decomposition approach is expected to play a crucial role in enhancing monitoring accuracy.

Understanding Regulatory Changes: Deep Learning in Sustainable Finance and Banking

This paper examines the regulatory impact on the European Banking Sector using advanced deep learning techniques to analyze the relationship between Sustainable Finance guidelines and the SX7P Index from January 2012 to December 2023. Utilizing Long Short-Term Memory Auto-encoder (LSTM-AE), Variational Autoencoder (VAE), and Convolutional Neural Network (CNN) for anomaly detection, the study compares anomalies and investigates their correlation with European Banking Authority (EBA) events and Sustainable Finance guidelines from January 2020 to December 2023. Through the analysis of 43 pertinent EBA documents, the research identifies patterns and variations in anomalies, assessing their association with regulatory changes. The results reveal significant anomalies aligning with regulatory events, indicating a potential causal relationship. Notably, the VAE methodology shows the strongest correlation between EBA Sustainable Finance events and anomalies. This research advances the understanding of deep learning applications in financial markets and offers valuable insights for policymakers and financial institutions regarding regulatory shifts in Sustainable Finance.

Feature Engineering for a MIL-STD-1553B LSTM Autoencoder Anomaly Detector

The MIL-STD-1553B data bus protocol is used in both civilian and military aircraft to enable communications between subsystems. These interconnected subsystems are responsible for core services such as communications, flow of instrument data and aircraft control. With aircraft modernization, threat vectors are introduced through increased inter-connectivity internal and external to the aircraft. The resulting potential for exploitation introduces a requirement for an intrusion detection capability in order to maintain the integrity, availability and reliability of data transmitted using the MIL-STD-1553B protocol, safety of the aircraft and overall, to achieve mission assurance. Research in recent years has investigated signature, statistical and machine learning based solutions to detect attacks on MIL-STD-1553B buses. Of the different techniques, those based on machine learning have shown extremely good results. The aim of this research is to improve the performance of an existing Long Short-Term Memory Auto-Encoder by refining the feature engineering phase of its pipeline. The improvement in the detector’s overall effectiveness was accomplished through feature engineering focused on feature generation and selection. Five different attack datasets were used as the starting point, consisting of four different denial of service attacks and one data integrity attack. From initial feature extraction of 155 features, two feature generation techniques were employed to create over 38,000 features as a starting point. Using five different MIL-STD-1553B datasets and three feature selection techniques, fifteen different Long Short-Term Memory Auto-Encoder models were created, trained and evaluated using common performance metrics and compared to those of the original anomaly detector. This research demonstrated marked performance improvement achieved by the feature engineering refinements made in comparison to those of the original model. Equally important, this research also showed a significant reduction in the number of features required to achieve this performance gain. In the context of miliary air operations, the ability to improve detection capabilities with less data is important to the technical solutions that contribute to the achievement of cyber mission assurance.

A combined framework based on LSTM autoencoder and XGBoost with adaptive threshold classification for credit card fraud detection

The digital invasion of the banking and financial sectors made life simple and easy. Traditional machine learning models have been studied in credit card fraud detection, but these models are often difficult to find effective for unseen patterns. This study proposes a combined framework of deep learning and machine learning models. The long short term memory autoencoder (LSTMAE) with attention mechanism is developed to extract high-level features and avoid overfitting of the model. The extracted features serve as input to the powerful ensemble model XGBoost to classify legitimate and fraudulent transactions. As the focus of fraud detection is to increase the recall rate, an adaptive threshold technique is proposed to estimate an optimal threshold value to enhance performance. The experiment was done with the IEEE-CIS fraud detection dataset available in Kaggle. The proposed model with optimal threshold has an increase in predicting fraudulent transactions. The research findings are compared with conventional ensemble techniques to find the generalization of the model. The proposed LSTMAE-XGB w/ attention method attained a good precision and recall of 94.2 and 90.5%, respectively, at the optimal threshold of θ = 0.22. The experimental results proved that the proposed approach is better at finding fraudulent transactions than other cutting-edge models

Deep LSTM-based autoencoder for CFRP damage imaging using Lamb wave and correlation factor weighted delay-and-sum algorithm

The ultrasonic Lamb wave testing (ULWT) has proven valuable in non-destructive testing (NDT) due to its high sensitivity and wide coverage. However, the classical post-processing algorithm, the delay-and-sum (DAS) technique, suffers from notable artifacts and inadequate accuracy in Lamb wave inspection due to the existence of reflection and superposition, especially on composite materials. In this study, we proposed a novel algorithm, the correlation factor weighted DAS imaging algorithm based on the Long short-term memory-autoencoder (LSTM-AE), to address these deterioration issues. The LSTM-AE demonstrates the capability to extract potential features and accurately reconstruct input signals. By incorporating the concept of anomaly detection, an LSTM-AE trained with intact signals produces a significantly distorted output when exposed to damaged signals as input. We formulated a novel damage index (DI) based on the Pearson Correlation Coefficient (PCC) between the output and input signals as the weighting factor. The proposed method has undergone experimental validation, confirming its effectiveness in Lamb wave inspection.

An effective ensemble classification framework for intrusion detection system using LSTM autoencoder

In intrusion detection, the curse of dimensionality and the trade-off between maintaining a low false alarm rate and achieving a high detection rate are significant challenges. This research suggests a unique strategy based on dimensionality reduction methods to improve the performance of network intrusion detection systems (NIDS). Compressing high-dimensional network traffic data using a Long Short-Term Memory Autoencoder (LSTMAE) allows the reduced characteristics to be submitted to a classifier to identify anomalies that may indicate an attack. Using standard datasets, including Network Security Laboratory - Knowledge Discovery in Datasets (NSL-KDD), UNSW-NB15, and Canadian Institute for Cyber Security - Intrusion Detection Systems (CICIDS2017), the proposed model is tested with classifiers like Random Forest (RF) and LightGBM (Light Gradient Boosting Machines). It is hoped that by adopting this method, NIDS response times may be improved while costs associated with storing and processing data are minimized. Precision, recall, F-score, accuracy, detection rate (DR), and false alarm rate (FAR) are only a few of the performance measures used to assess the quality of the suggested models. The experimental findings show that the proposed LSTMAE model reduces prediction errors more effectively than classic machine learning techniques such as Random Forest (RF), Gradient Boosting (GB), Support Vector Machines (SVM), Deep Belief Networks (DBN), Deep Neural Networks (DNN), Autoencoder (AE), and Long Short-Term Memory (LSTM). The results also show that the proposed solution outperforms the state-of-the-art methods of detection accuracy and computing complexity using accuracy, precision, recall, F1_Score, detection rate, and FAR.

A novel industrial process situation awareness model based on multi‐time scale dynamic feature fusion with applications to float glass manufacturing

AbstractModern industrial manufacturing processes often focus on the changes of performance indicators and the evolution of operation conditions. However, the dynamics of variables are vague within processes and even more difficult to be characterised across different processes. The time‐series dynamic characteristics of the different scales among variables, processes, and production cycles develop and transfer with changes in parameters, equipment, and processes. It is difficult to accurately show the quality index and operating condition trend at the same time. To solve these problems, a situation awareness (SA) framework integrating multi‐time scale dynamic features for manufacturing processes is proposed. First, data are condensed and reconstructed through the denoising long short term memory autoencoder to reveal the time series dynamic characteristics to get neighbourhood features which contain features among neighbourhood samples. Second, the neighbourhood features divided into window blocks are fused into the stage features by statistical analysis. Finally, a multi‐scale isometric convolution network is designed to extract the local and global features, which can effectively show the development of dynamic features on a long time scale, and profoundly describe the influence of full cycle features on variables and operating conditions. The proposed model is verified on the real data set of a float glass manufacturing process, and the SA model can well predict the future trend of long time series.

Application of machine learning in determining and resolving state estimation anomalies in power systems

The state estimation (SE) process is one of the most important and efficient tools in achieving this goal. However, the occurrence of anomalies in the power grid, such as false data injection (FDI), can significantly impact the accuracy of the SE results. FDI results in reduced accuracy of SE results, potentially putting the power system in a critical situation. This paper addresses the limitations of simple residual-based algorithms in detecting FDI and proposes a scenario that utilizes machine learning (ML) models such as auto-encoder (AE), long short-term memory auto-encoder (LSTM AE), and 1-D convolutional neural network auto-encoder (1-D CNN AE) algorithms. The proposed method aims to offer a more effective FDI detection approach in the power grid, exhibiting higher detection accuracy. Also, the paper introduces the LSTM variational auto-encoder (LSTM VAE) algorithm for reconstructing anomalous data. By utilizing the LSTM VAE, anomalous data can be transformed to closely resemble the original data with an acceptable level of accuracy. Moreover, in critical situations involving FDI, the power system can maintain normal operation by employing the proposed method to reconstruct anomalous data. Finally, the performance of the presented methods is evaluated on the IEEE 14-bus, 30-bus, and 118-bus test networks. The results are then presented and discussed to demonstrate the effectiveness of the proposed method.

LSTM Auto Encoder 이용한 접근 이상 항적 탐지 모형

With the rise in aviation demand and the emergence of Urban Air Mobility, developing a safe aviation system in urban areas is becoming increasingly important. This study addresses the challenge of detecting anomalous flight trajectories, which can be influenced by environmental factors. We propose a novel Long Short-Term Memory-Auto Encoder (LSTM-AE) model that processes both environmental and trajectory data but only reconstructs trajectory data in its output. This approach was validated by assessing the average reconstruction error for specific trajectories. Additionally, the model's ability to identify anomalies was confirmed by evaluating the Area under the ROC curve (AUC) for typical anomalous trajectories, such as go-around maneuvers. Our findings indicate that the proposed LSTM-AE model effectively learns trajectory patterns in relation to environmental variables and shows enhanced anomaly detection capabilities compared to traditional AE and LSTM-AE models. These results contribute to the development of advanced models that incorporate a wider range of environmental factors, enhancing safety in urban air travel.

Dual-attention LSTM autoencoder for fault detection in industrial complex dynamic processes

Complex dynamic characteristics resulting from multi-system coupling and closed-loop control are ubiquitous in modern industrial process data, presenting significant challenges for process fault detection. However, conventional data-driven fault detection methods assume the data to be static or slightly dynamic. Addressing the complex dynamic characteristics and nonlinearity inherent in industrial processes, this paper proposes a dual-attention long short-term memory autoencoder (DALSTM-AE) for fault detection in dynamic processes. Long short-term memory (LSTM) and autoencoder (AE) are combined into a special encoder-decoder LSTM architecture to learn both dynamic features and deep representations of variables in an unsupervised manner. Then, a dual-attention module is embedded in the decoder to properly learn the temporal dependencies associated with long input sequences and retain the most critical information. In addition, based on the reconstruction results of the DALSTM-AE model, two monitoring statistics are designed for fault detection. Finally, the effectiveness and superiority of the proposed method are fully demonstrated through case studies on a numerical simulation example, the Tennessee Eastman (TE) benchmark process, and practical coal pulverizing systems in power plants.