Long-term Dependencies Research Articles

A dynamic snow depth retrieval model based on time-series clustering optimization for GPS-IR

Due to the influence of environmental factors (i.e., terrain and surface coverage) around the GPS receivers, the snow depth retrieval results obtained by the existing global positioning system interferometric reflection (GPS-IR) method show significant variability. The resulting loss of reliability and accuracy limits the broad application of this technology. Therefore, this paper proposes a dynamic snow depth retrieval model based on time-series clustering optimization for GPS-IR to fully leverage multi-source satellite observation data for automatic and high-precision snow depth retrieval. The model employs Dynamic Time Warping distance measurement combined with the K-Medoids clustering algorithm to categorize frequency sequences obtained from various satellite trajectories, facilitating effective integration of multi-constellation data and acquisition of optimal datasets. Additionally, Long Short-Term Memory networks are integrated to capture and process the long-term dependencies in snow depth data, enhancing the model’s adaptability in handling time-series data. Validated against SNOTEL measured data and standard machine learning algorithms (such as BP Neural Networks, RBF, and SVM), the model’s retrieval capability is confirmed. For P351 and AB39 sites, the correlation coefficients for L1 band data retrieval were both 0.996, with RMSEs of 0.051 and 0.018 m, respectively. The experiment results show that the proposed model demonstrates superior precision and robustness in snow depth retrieval compared to the previous method. Then, we analyze the accuracy loss caused by sudden snowfall events. The proposed model and methodology offer new insights into the in-depth study of snow depth monitoring.

Using experience classification for training non-Markovian tasks

Unlike standard Reinforcement Learning (RL) model, many real-world tasks are non-Markovian, which requires long-term memory and dependency. Hence solving a non-Markovian task is more difficult and challenging than solving a Markovian one. In this paper, we propose a novel RL approach for training non-Markovian tasks expressed in temporal logic LTLf (Linear Temporal Logic over Finite Traces). To this end, an encoding of linear complexity from LTLf into MDPs (Markov Decision Processes) is introduced in order to take advantage of advanced RL algorithms. We further propose an experience classification method based on the automaton structure (theoretically equivalent to LTLf specifications). An automatic reward shaping technique and a prioritized experience replay mechanism are developed to cooperate with the classification method to improve the performance of RL algorithms. We provide empirical evaluations on two widely used benchmark problems, Waterworld and Cartpole, both augmented with complex non-Markovian tasks. The evaluations are conducted with respect to the metrics of training speed, policy quality, convergence rates, computational efficiency and scalability etc. The experimental results show that our approach achieves superior performance over other relevant studies, specially with an average improvement of 133% in convergence rate and a reduction of 11% in training time.

DSnet: a new dual-branch network for hippocampus subfield segmentation

The hippocampus is a critical component of the brain and is associated with many neurological disorders. It can be further subdivided into several subfields, and accurate segmentation of these subfields is of great significance for diagnosis and research. However, the structures of hippocampal subfields are irregular and have complex boundaries, and their voxel values are close to surrounding brain tissues, making the segmentation task highly challenging. Currently, many automatic segmentation tools exist for hippocampal subfield segmentation, but they suffer from high time costs and low segmentation accuracy. In this paper, we propose a new dual-branch segmentation network structure (DSnet) based on deep learning for hippocampal subfield segmentation. While traditional convolutional neural network-based methods are effective in capturing hierarchical structures, they struggle to establish long-term dependencies. The DSnet integrates the Transformer architecture and a hybrid attention mechanism, enhancing the network’s global perceptual capabilities. Moreover, the dual-branch structure of DSnet leverages the segmentation results of the hippocampal region to facilitate the segmentation of its subfields. We validate the efficacy of our algorithm on the public Kulaga-Yoskovitz dataset. Experimental results indicate that our method is more effective in segmenting hippocampal subfields than conventional single-branch network structures. Compared to the classic 3D U-Net, our proposed DSnet improves the average Dice accuracy of hippocampal subfield segmentation by 0.57%.

Accurate short-term GHI forecasting using a novel temporal convolutional network model

Global energy demand is on the rise, driven by factors such as population growth and economic development.Utilizing renewable energy is crucial to meeting this energy demand in light of global warming and the depletion of fossil resources.One of the renewable energy sources that is extensively utilized in multiple countries worldwide is photovoltaic energy.While integrating photovoltaic (PV) energy into the grid offers substantial economic and environmental advantages, its intermittent nature presents challenges to maintaining grid stability at high penetration levels. Accurate solar energy estimations are essential for photovoltaic (PV) based energy facilities to enable early participation in energy markets and efficient resource planning. This study proposes and evaluates a Temporal Convolutional Network (TCN) model that can rapidly predict global horizontal irradiance (GHI) using univariate and multivariate approaches. To our knowledge, this is the first study to employ a TCN model for GHI forecasting. The univariate model solely considers GHI, while the multivariate time series models incorporate a combination of six variables: global horizontal irradiance, active power of a module plant, PV module temperature, wind speed, air temperature, and air pressure. The proposed model’s effectiveness is validated by comparing its performance to benchmark models, including MLP, RNN, GRU, and LSTM. The prediction models’ performance is evaluated using root mean square error (RMSE), coefficient of determination (R2), and mean absolute error (MAE). The results demonstrate that the effectiveness of univariate TCN models is evident in their performance for 5, 10, 15, and 20 min ahead short-term GHI forecasting, its mean absolute error (MAE) values achieved are 22.935 W/m2 33.47 W/m2, 36.9994 W/m2, and 41.9946 W/m2, respectively. The multivariate model, incorporating additional variables, yielded higher MAE values : 37.928 W/m2, 55.88 W/m2, 66.61 W/m2, and 74.82 W/m2, respectively. These results suggest that the TCN model’s capability to capture both long-term and short-term dependencies within the data contributes to its accuracy for GHI forecasting compared to other models.

A Residual Deep Learning Method for Accurate and Efficient Recognition of Gym Exercise Activities Using Electromyography and IMU Sensors

The accurate and efficient recognition of gym workout activities using wearable sensors holds significant implications for assessing fitness levels, tailoring personalized training regimens, and overseeing rehabilitation progress. This study introduces CNN-ResBiGRU, a novel deep learning architecture that amalgamates residual and hybrid methodologies, aiming to precisely categorize gym exercises based on multimodal sensor data. The primary goal of this model is to effectively identify various gym workouts by integrating convolutional neural networks, residual connections, and bidirectional gated recurrent units. Raw electromyography and inertial measurement unit data collected from wearable sensors worn by individuals during strength training and gym sessions serve as inputs for the CNN-ResBiGRU model. Initially, convolutional neural network layers are employed to extract unique features in both temporal and spatial dimensions, capturing localized patterns within the sensor outputs. Subsequently, the extracted features are fed into the ResBiGRU component, leveraging residual connections and bidirectional processing to capture the exercise activities’ long-term temporal dependencies and contextual information. The performance of the proposed model is evaluated using the Myogym dataset, comprising data from 10 participants engaged in 30 distinct gym activities. The model achieves a classification accuracy of 97.29% and an F1-score of 92.68%. Ablation studies confirm the effectiveness of the convolutional neural network and ResBiGRU components. The proposed hybrid model uses wearable multimodal sensor data to accurately and efficiently recognize gym exercise activity.

ArabBert-LSTM: improving Arabic sentiment analysis based on transformer model and Long Short-Term Memory.

Sentiment analysis also referred to as opinion mining, plays a significant role in automating the identification of negative, positive, or neutral sentiments expressed in textual data. The proliferation of social networks, review sites, and blogs has rendered these platforms valuable resources for mining opinions. Sentiment analysis finds applications in various domains and languages, including English and Arabic. However, Arabic presents unique challenges due to its complex morphology characterized by inflectional and derivation patterns. To effectively analyze sentiment in Arabic text, sentiment analysis techniques must account for this intricacy. This paper proposes a model designed using the transformer model and deep learning (DL) techniques. The word embedding is represented by Transformer-based Model for Arabic Language Understanding (ArabBert), and then passed to the AraBERT model. The output of AraBERT is subsequently fed into a Long Short-Term Memory (LSTM) model, followed by feedforward neural networks and an output layer. AraBERT is used to capture rich contextual information and LSTM to enhance sequence modeling and retain long-term dependencies within the text data. We compared the proposed model with machine learning (ML) algorithms and DL algorithms, as well as different vectorization techniques: term frequency-inverse document frequency (TF-IDF), ArabBert, Continuous Bag-of-Words (CBOW), and skipGrams using four Arabic benchmark datasets. Through extensive experimentation and evaluation of Arabic sentiment analysis datasets, we showcase the effectiveness of our approach. The results underscore significant improvements in sentiment analysis accuracy, highlighting the potential of leveraging transformer models for Arabic Sentiment Analysis. The outcomes of this research contribute to advancing Arabic sentiment analysis, enabling more accurate and reliable sentiment analysis in Arabic text. The findings reveal that the proposed framework exhibits exceptional performance in sentiment classification, achieving an impressive accuracy rate of over 97%.

Deep hybrid transformer network for robust modulation classification in wireless communications

Modulation classification is a research hot-spot in the field of machine learning and the knowledge-based systems of wireless communication, involving the identification of different types of wireless signals. However, classification targets like radio signals usually involve long-sequence and large-scale data, and the classification of modulation types is affected by environmental noises, resulting in unsatisfactory performance. To overcome these challenges, we propose a novel Deep Hybrid Transformer Network (DH-TR) that makes full use of the intrinsic properties of multi-head self-attention and different neural modules, facilitating the identification of modulation types from global to local perspectives. In particular, DH-TR uses a convolution stem to extract local features inherent in In-phase and Quadrature (IQ) data, based on which the Gated Recurrent Unit (GRU) is used to capture the step-wise sequential patterns of these signals. Afterward, the self-attention based Transformer branch is employed to learn the global and long-term dependencies among the signal patches. With this innovative hybrid design, DH-TR performs well in processing sequential signal data and can better capture the complex relationships between signals. We validate DH-TR’s effectiveness through extensive experiments on four benchmark datasets, demonstrating its superior performance in modulation classification with higher accuracy and robustness to noise compared to existing methods.

A generative adversarial network based on an efficient transformer for high-fidelity flow field reconstruction

The reconstruction of high-fidelity flow fields from low-fidelity data has attracted considerable attention in fluid dynamics but poses many challenges to existing deep learning methods due to the spatiotemporal complexity of flows and the lack of standardized benchmark datasets. In this study, we generate a low- and high-fidelity dataset containing 25 600 snapshots of four representative flow dynamics simulations using eight different numerical-precision and grid-resolution configurations. Using this dataset, we develop a physics-guided transformer-based generative adversarial network (PgTransGAN) for concurrently handling numerical-precision and grid-resolution enhancement. PgTransGAN leverages a dual-discriminator-based generative adversarial network for capturing continuous spatial and temporal dynamics of flows and applies a soft-constraint approach to enforce physical consistency in the reconstructed data using gradient information. An efficient transformer model is also developed to obtain the long-term temporal dependencies and further alleviate storage constraints. We compare the performance of PgTransGAN against standard linear interpolation and solutions based solely on convolutional neural networks or generative adversarial networks, and demonstrate that our method achieves better reconstruction quality at the data, image, and physics levels with an upscaling factor of 4 or even 8 in each grid dimension.

GLaLT: Global-Local Attention-Augmented Light Transformer for Scene Text Recognition.

Recent years have witnessed the growing popularity of connectionist temporal classification (CTC) and attention mechanism in scene text recognition (STR). CTC-based methods consume less time with few computational burdens, while they are not as effective as attention-based methods. To retain computational efficiency and effectiveness, we propose the global-local attention-augmented light Transformer (GLaLT), which adopts a Transformer-based encoder-decoder structure to orchestrate CTC and attention mechanism. The encoder integrates the self-attention module with the convolution module to augment the attention, where the self-attention module pays more attention to capturing long-term global dependencies and the convolution module focuses on local context modeling. The decoder consists of two parallel modules: one is the Transformer-decoder-based attention module and the other is the CTC module. The first one is removed in the testing phase and can guide the second one to extract robust features in the training phase. Extensive experiments on standard benchmarks demonstrate that GLaLT achieves state-of-the-art performance for both regular and irregular STR. In terms of tradeoffs, the proposed GLaLT is at or near the frontiers for maximizing speed, accuracy, and computational efficiency at the same time.

AEMNet: Unsupervised Video Anomaly Detection Method Based on Attention-Enhanced Memory Networks

Video anomaly detection has always been a challenging task in computer vision due to data imbalance and susceptibility to scene variations such as lighting and occlusions. In response to this challenge, this paper proposes an unsupervised video anomaly detection method based on an attention-enhanced memory network. The method utilizes a dual-stream network structure of autoencoders, enhancing the model’s learning ability for important features in appearance and motion by introducing coordinate attention mechanisms and variance attention mechanisms, emphasizing significant characteristics of static objects and rapidly moving regions. By adding memory modules to both the appearance and motion branches, the network structure’s memory information is reinforced, enabling it to capture long-term spatiotemporal dependencies in videos and thereby improving the accuracy of anomaly detection. Furthermore, by optimizing the network structure’s activation functions to handle negative inputs, it enhances its nonlinear modeling capabilities, enabling better adaptation to complex environments, including variations in lighting and occlusions, further improving the effectiveness of anomaly detection. The paper conducts comparative experiments and ablation studies using three public available datasets and various models. The results demonstrate that compared to baseline models, the AUC performance is improved by 3.9%, 4.7%, and 1.7% on UCSD Ped2, CHUK Avenue, and ShanghaiTech datasets, respectively. When compared with the other models, the average AUC performance is improved by 4.3%, 5.4%, and 6.2%, with an average improvement of 8.75% in the ERR metric, validating the effectiveness and adaptability of the proposed method. The code can be obtained at the following URL: https://github.com/AcademicWhite/AEMNet .

A unified European hydrogen infrastructure planning to support the rapid scale-up of hydrogen production

Hydrogen will become a key player in transitioning toward a net-zero energy system. However, a clear pathway toward a unified European hydrogen infrastructure to support the rapid scale-up of hydrogen production is still under discussion. This study explores plausible pathways using a fully sector-coupled energy system model. Here, we assess the emergence of hydrogen infrastructure build-outs connecting neighboring European nations through hydrogen import and domestic production centers with Western and Central European demands via four distinct hydrogen corridors. We identify a potential lock-in effect of blue hydrogen in the medium term, highlighting the risk of long-term dependence on methane. In contrast, we show that a self-sufficient Europe relying on domestic green hydrogen by 2050 would increase yearly expenses by around 3% and require 518 gigawatts of electrolysis capacity. This study emphasizes the importance of rapidly scaling up electrolysis capacity, building hydrogen networks and storage facilities, deploying renewable electricity generation, and ensuring coherent coordination across European nations.

Gated graph spiking neural P network for session-based recommendation

Session-based recommendation (SBR) is a recommendation system application scenario that focuses on user interests and behaviors in sessions formed within a short period. However, existing SBR methods do not fully extract user contextual information, and gated graph neural networks(GNNs) still face challenges in handling long-term dependencies. The nonlinear spiking neural P system is a novel spiking neural-like computing model in which the nonlinear spiking mechanism is a typical feature. Based on this nonlinear spiking mechanism, we propose a new recurrent-like model called the extended gated spiking neural P model or the EGSNP model, which can effectively capture temporal relationships in their sequence and obtain the user’s contextual information to alleviate information extraction problems in SBR. Based on this EGSNP model, we propose a new type of GNN called the extended gated graph spiking neural P network or termed the EGGSNP network. Finally, we use the EGSNP model and EGGSNP network to construct a new SBR model for the session recommendation task, termed the SR-EGGSNP model. This SR-EGGSNP model can effectively extract contextual information and flexibly capture the long-term evolution of interest while alleviating the long-distance dependency problem in GNNs. We conducted extensive comparative experiments on two public datasets, and the results demonstrated the effectiveness of the proposed model and accuracy of the recommendations.

A BERT-GRU Model for Measuring the Similarity of Arabic Text

Semantic Textual Similarity (STS) aims to assess the semantic similarity between two pieces of text. As a challenging task in natural language processing, various approaches for STS in high-resource languages, such as English, have been proposed. In this paper, we are concerned with STS in low resource languages such as Arabic. A baseline approach for STS is based on vector embedding of the input text and application of similarity metric on the embedding space. In this contribution, we propose a cross-encoder neural network (Cross-BERT-GRU) to handle semantic similarity of Arabic sentences that benefits from both the strong contextual understanding of BERT and the sequential modeling capabilities of GRU. The architecture begins by inputting the BERT word embeddings for each word into a GRU cell to model long-term dependencies. Then, max pooling and average pooling are applied to the hidden outputs of the GRU cell, serving as the sentence -pair encoder. Finally, a softmax layer is utilized to predict the degree of similarity. The experiment results show a Spearman correlation coefficient of around 0.9 and that Cross-BERT-GRU outperforms the other BERT models in predicting the semantic textual similarity of Arabic sentences. The experimentation results also indicate that the performance improves by integrating data augmentation techniques.

E-DNet: An End-to-End Dual-Branch Network for Driver Steering Intention Detection

An advanced driving assistant system (ADAS) is critical for improving traffic efficiency and ensuring driving safety. By anticipating the driver’s steering intentions in advance, the system can alert the driver in time to avoid a vehicle collision. This paper proposes a novel end-to-end dual-branch network (EDNet) that utilizes both in-cabin and out-of-cabin data. In this study, we designed an in-cabin driver intent feature extractor based on 3D residual networks and atrous convolution, which is applicable to video data and is capable of capturing a larger range of driver behavior. In order to capture the long-term dependency of temporal data, we designed the depthwise-separable max-pooling (DSMax) module and combined it with a convolutional LSTM to obtain the road environment feature extractor outside the cabin. In addition, to effectively fuse different features inside and outside the cockpit, we designed and propose the dynamic combined-feature attention fusion (D-CAF) module. EDNet employs a freeze-training method, which enables the creation of a lightweight model while simultaneously enhancing the final classification accuracy. Extensive experiments on the Brain4Cars dataset and the Zenodo dataset show that the proposed EDNet was able to recognize the driver’s steering intention up to 3 s in advance. It outperformed the existing state of the art in most driving scenarios.

A study of European wheat price series forecasting based on multiple variants of Holt-Winters and LSTM models

Given the importance of wheat in global food security and agricultural economy, accurate price forecasts are of significant practical significance for growers, the food chain, and government macro-control. Four different time series forecasting models were used in this study, including the traditional Holt-Winters three-parameter exponential smoothing method and three improved models based on the Long Short-Term Memory Network (LSTM): the CNN-LSTM, the GRU, and the VMD-LSTM. By comparing the forecasting effects of these models under the same training and test sets, this study aims to select the best models using the best model for predicting European wheat prices from 2010 to 2023 after training. The results show that the CNN-LSTM model performs the best in predicting the European wheat price series, and the future wheat price will show a slow downward trend, and the CNN-LSTM model combines the feature extraction ability of convolutional neural network and the long-term dependency learning ability of LSTM, which can effectively capture the fluctuation characteristics of the price series and the trend characteristics. The GRU model also shows a better prediction ability, especially in dealing with long-term dependencies. in dealing with long-term dependencies. However, the VMD-LSTM model has certain shortcomings in its prediction effect when facing time series data with large fluctuations in the later period. In addition, the Holt-Winters model predicts that after a period of slow decline, wheat prices may rebound, which may be related to the adjustment of market supply and demand and changes in external factors.

Sparse dynamic graph learning for district heat load forecasting

Accurate heat load forecasting is crucial for the efficient operation and management of district heating systems. This study introduces a novel Sparse Dynamic Graph Neural Network (SDGNN) framework designed to address the complexities of forecasting heat load in district heating networks. The proposed model represents the district heating network as a dynamic graph, with nodes corresponding to consumers or heat sources and edges denoting temporal dependencies. The SDGNN framework comprises three key components: (1) a sparse graph learning module that identifies the most relevant nodes and edges, (2) a spatio-temporal memory enhancement module that captures both short-term and long-term dependencies, and (3) a temporal fusion module that integrates node representations into a comprehensive global forecast. Evaluated on a real-world district heating dataset from Denmark, the SDGNN model demonstrates superior accuracy and efficiency compared to existing methods. The results indicate that the SDGNN framework effectively captures intricate spatio-temporal patterns in historical heat load data, achieving up to 5.7% improvement in RMSE, 7.4% in MAE, and 5.7% in CVRMSE over baseline models. Additionally, incorporating meteorological factors into the model further enhances its predictive performance. These findings suggest that the SDGNN framework is a robust and scalable solution for district heat load forecasting, with potential applications in other domains involving spatio-temporal graph data.

A New AI Approach by Acquisition of Characteristics in Human Decision-Making Process

Planning and decision making are closely interconnected processes that often occur in tandem, influence and informing each other. Planning usually precedes decision making in the chronological sequence, and it can be viewed as a strategy to make decisions. A comprehensive planning or decision strategy can facilitate effective decisions. Thus, understanding and learning human decision-making strategies has drawn intensive attention from the AI community. For example, applying planning algorithms into reinforcement leaning (RL) can simulate the consequence of different actions and select optimal decisions based on learned models, while inverse reinforcement learning (IRL) learns a reward function and policy from expert demonstration and applies them into new scenarios. Most of these methods work based on learning human decision strategies by using modeling of a Markovian decision-making process (MDP). In this paper, we argue that the property of MDP is not fit for human decision-making processes in the real-world and it is insufficient to capture human decision strategies. To tackle this challenge, we propose a new approach to identify the characteristics of human decision-making processes as a decision map, where the decision strategy is defined by the probability distribution of human decisions that are adaptive to the dynamic changes in the environment. The proposed approach was inspired by imitation learning (IL) but with fundamental differences: (a) Instead of aiming to learn an optimal policy based on expert’s demonstrations, we aimed to estimate the distribution of decisions of any group of people. (b) Instead of modeling the environment by an MDP, we used an ambiguity probability model to consider the uncertainty of each decision. (c) The participant trajectory was obtained by categorizing each decision of a participant to a certain cluster based on the commonness in the distribution of decisions. The result shows a feasible way to capture human long-term decision dependency, which provides a complement to the existing machine learning methods for understanding and learning human decision strategies.

Dynamic mechanical response and constitutive model of (Ti37.31Zr22.75Be26.39Al4.55Cu9)94Co6 high-entropy bulk metallic glass

In this work, the mechanical response and fracture characteristics of (Ti37.31Zr22.75Be26.39Al4.55Cu9)94Co6 high-entropy bulk metallic glass (HE-BMG) were investigated in detail over a wide range of strain rates (10−4–105 s−1). The HE-BMG exhibited a negative strain rate sensitivity under uniaxial compression, with the strength showing more significant rate dependence under dynamic conditions. The shear band behavior translated from the dominance of multiple shear bands propagations under quasi-static compression to the rapid propagation of a single shear band to form cracks under dynamic compression. Under dynamic loading, the shearing velocity increased along an arc-shaped displacement path, and the local stress state on the shear fracture surface shifted from compressive-shear to tensile-shear, accompanied by changes in fracture morphologies. The spall strength of HE-BMG decreased as flyer impact speed increased, while the long-term dependence of spall strength on strain rate may be positive. With the increase of impact speed, the main microstructural features of the spalling surface translated from flat regions accompanied by dimples to cup-cone structures. Furthermore, the complete parameters of the Johnson–Holmquist II (JH-2) model for HE-BMG were obtained based on experimental data. Numerical simulations of planar impact, penetration, and hypervelocity impact are in good agreement with experimental results, demonstrating the validity of the JH-2 model parameters. The current work has important guiding value for the application of HE-BMG in space debris protection.

AutoAMS: Automated attention-based multi-modal graph learning architecture search

Multi-modal attention mechanisms have been successfully used in multi-modal graph learning for various tasks. However, existing attention-based multi-modal graph learning (AMGL) architectures heavily rely on manual design, requiring huge effort and expert experience. Meanwhile, graph neural architecture search (GNAS) has made great progress toward automatically designing graph-based learning architectures. However, it is challenging to directly adopt existing GNAS methods to search for better AMGL architectures because of the search spaces that only focus on designing graph neural network architectures and the search objective that ignores multi-modal interactive information between modalities and long-term content dependencies within different modalities. To address these issues, we propose an automated attention-based multi-modal graph learning architecture search (AutoAMS) framework, which can automatically design the optimal AMGL architectures for different multi-modal tasks. Specifically, we design an effective attention-based multi-modal (AM) search space consisting of four sub-spaces, which can jointly support the automatic search of multi-modal attention representation and other components of multi-modal graph learning architecture. In addition, a novel search objective based on an unsupervised multi-modal reconstruction loss and task-specific loss is introduced to search and train AMGL architectures. The search objective can extract the global features and capture multi-modal interactions from multiple modalities. The experimental results on multi-modal tasks show strong evidence that AutoAMS is capable of designing high-performance AMGL architectures.

Anomaly detection for key performance indicators by fusing self-supervised spatio-temporal graph attention networks

With the development of Artificial Intelligence for IT Operations (AIOps), numerous software and services are monitored by Key Performance Indicators (KPIs) collection components. Multivariate KPIs, as a type of time series data, are essential for effective management of the entity's service quality. In recent years, deep learning methods have made great improvements in the anomaly detection of multivariate time series; however, existing methods have not fully considered how to explicitly capture the correlation between multivariate time series in the feature dimension and temporal dimension, resulting in inevitable abnormal false positives. Therefore, this paper proposes a self-supervised multivariate KPIs anomaly detection method MAD-STA that combines graph structure learning and spatio-temporal GAT (Graph Attention Network). In the feature dimension, MAD-STA introduces a node embedding mechanism for graph structure learning and then uses the feature-oriented GAT layer to compute the graph attention coefficient to obtain the correlation between different KPIs. In the temporal dimension, MAD-STA uses the time-oriented GAT layer to compute attention weights between correlated timestamps, and the GRU-based VAE encoder captures long-term dependence to extract more comprehensive temporal feature representations. Finally, MAD-STA uses GRU-based VAE decoder to reconstruct the captured high-level features and achieves efficient anomaly detection and localization by calculating the anomaly score of multiple KPIs. Compared with the baseline methods on multiple data sets, the experimental results show that the anomaly detection accuracy of MAD-STA is better than that of the baseline method. Especially on the KPI data sets of the two server clusters of SMD and CKM, MAD-STA improves the performance and the F1 comprehensive index compared with the best baseline method. In addition, MAD-STA performs well on anomaly false positive rate and has excellent interpretability, which can be used to assist anomaly diagnosis and root cause index analysis.