Time Series Domain Research Articles

Transformer-Based GAN with Multi-STFT for Rotating Machinery Vibration Data Analysis

Prognostics and health management of general rotating machinery have been studied over time to improve system stability. Recently, the excellent abnormal diagnosis performance of artificial intelligence (AI) was demonstrated, and therefore, AI-based intelligent diagnosis is now being implemented in these systems. AI models are trained using large volumes of data. Therefore, we propose a transformer-based generative adversarial network (GAN) model with a multi-resolution short-time Fourier transform (multi-STFT) loss function to augment the vibration data of rotating machinery to facilitate the successful learning of deep learning models. We constructed a model with a conditional GAN structure, which is transformer based, for learning the feature points of vibration data in the time-series domain. In addition, we applied the multi-STFT loss function to capture the frequency features of the vibration data. The generated data, which adequately captured the frequency features, were used to augment the training data to improve the performance of a deep learning classifier. Furthermore, by visualizing the generated vibration data and comparing the visualizations to those of the vibration data obtained from real machinery, we demonstrated that the generated data were indistinguishable from the actual data.

Multi-scale contrast approach for stock index prediction with adaptive stock fusion

The stock index summarizes the overall movement of a group of stocks, typically calculated as a weighted average of constituent stock prices. Stock Index Prediction (SIP) aids investors in assessing economic performance and making informed decisions. Traditional SIP methods rely heavily on historical index data, neglecting valuable insights from macro-financial indicators and stock prices. However, challenges arise when incorporating these data: (1) Including numerous unfiltered macro-financial indicators introduces noise. (2) Using non-correlated stock prices can be counterproductive. (3) Supervised training on stochastic time series can lead to overfitting. To overcome these issues, we propose a multi-scale contrast approach for stock index prediction with adaptive stock fusion (MCSIP). This method identifies highly correlated macro-financial indicators and stocks, optimizing the model through self-supervised multi-scale contrastive learning. MCSIP relies on contrastive learning in the time series domain to extract robust contextual information from stock index time series, mitigating stochastic data backpropagation and ensuring reliable representation. Experiments on US and Chinese stock datasets demonstrate superior performance compared to existing methods.

Rainforest: A three-stage distribution adaptation framework for unsupervised time series domain adaptation

Solving the unsupervised domain adaptation (UDA) task in time series is of great significance for practical applications, such as human activity recognition and machine fault diagnosis. Compared to UDA for computer vision, UDA in time series is more challenging due to the dynamics of time series data and the complex dependencies among different time steps. Existing UDA methods for time series fail to adequately capture the temporal dependencies, limiting their ability to learn domain-invariant temporal patterns. Furthermore, most UDA methods only focus on distribution adaptation on the backbone network without considering how the classifier adapts to the data distribution of the target domain. In this paper, we propose Rainforest, a three-stage UDA framework for time series. We first pre-train the backbone network through a self-supervised method called bidirectional autoregression, so that the model can comprehensively learn the temporal dependencies in time series. Next, we propose a novel meta-learning-based distribution adaptation method to achieve the joint alignment of the global and local distributions while encouraging the model to adaptively reduce the temporal dynamic differences among different domains. Finally, we design a pseudo-label-guided fine-tuning strategy to help the classifier estimate the data distribution of the target domain more accurately. Extensive experiments on four real-world time series datasets show that our Rainforest outperforms state-of-the-art methods, with an average improvement of 2.19% in accuracy and 2.41% in MF1-score.

Time-series domain adaptation via sparse associative structure alignment: Learning invariance and variance

Domain adaptation on time-series data, which is often encountered in the field of industry, like anomaly detection and sensor data forecasting, but received limited attention in academia, is an important but challenging task in real-world scenarios. Most of the existing methods for time-series data use the covariate shift assumption for non-time-series data to extract the domain-invariant representation, but this assumption is hard to meet in practice due to the complex dependence among variables and a small change of the time lags may lead to a huge change of future values. To address this challenge, we leverage the stableness of causal structures among different domains. To further avoid the strong assumptions in causal discovery like linear non-Gaussian assumption, we relax it to mine the stable sparse associative structures instead of discovering the causal structures directly. Besides the domain-invariant structures, we also find that some domain-specific information like the strengths of the structures is important for prediction. Based on the aforementioned intuition, we extend the sparse associative structure alignment model in the conference version to the Sparse Associative Structure Alignment model with domain-specific information enhancement (SASA2 in short), which aligns the invariant unweighted spare associative structures and considers the variant information for time-series unsupervised domain adaptation. Specifically, we first generate the segment set to exclude the obstacle of offsets. Second, we extract the unweighted sparse associative structures via sparse attention mechanisms. Third, we extract the domain-specific information via an autoregressive module. Finally, we employ a unidirectional alignment restriction to guide the transformation from the source to the target. Moreover, we further provide a generalization analysis to show the theoretical superiority of our method. Compared with existing methods, our method yields state-of-the-art performance, with a 5% relative improvement in three real-world datasets, covering different applications: air quality, in-hospital healthcare, and anomaly detection. Furthermore, visualization results of sparse associative structures illustrate what knowledge can be transferred, boosting the transparency and interpretability of our method.

Time-Series Anomaly Detection: Overview and New Trends

Anomaly detection is a fundamental data analytics task across scientific fields and industries. In recent years, an increasing interest has been shown in the application of anomaly detection techniques to time series. In this tutorial, we take a holistic view of anomaly detection in time series and comprehensively cover detection algorithms ranging from the 1980s to the most current state-of-the-art techniques. Importantly, the scope of this tutorial extends beyond algorithmic discussion, delving into the latest advancements in benchmarking and evaluation measures for this area. In particular, our interactive systems enable the exploration of detection algorithms and benchmarking results, thereby promoting user comprehension. Driven by the absence of a one-size-fits-all anomaly detector for various time series domains and applications, we review recent advancements in automated solutions and propose a new taxonomy to motivate further research.

Kernelized Bures metric: A framework for effective domain adaptation in sensor data analysis

Unsupervised Domain Adaptation (UDA) plays a crucial role in enabling the transfer of models trained on labeled source domains to unlabeled target domains. However, this transferability encounters significant challenges when dealing with intricate time series models due to the complex temporal dynamics that differ between domains. These divergent dynamics result in disparities in time and frequency representations, leading to misalignments and gaps. Furthermore, the absence of localized information capture in previous approaches undermines the performance potential of the models. We introduce the KBSDA (Kernel Bures Sub-Domain Adaptation) approach to tackle the issues above. Our methodology employs the fast Fourier transform to extract frequency features while encompassing the time features. Our approach’s key focus is capturing intricate attributes containing local information, a task accomplished by applying the Local Maximum Mean Discrepancy (LMMD) metric. Furthermore, we address the distribution gaps between the source and target domains, elevating the alignment process by incorporating the kernel Bures metric. This metric, which encompasses higher-order moments and effectively manages intricate non-linear relationships, significantly manages complex distribution shifts. To validate the effectiveness of our approach, we conducted comprehensive experiments on well-established time series domain adaptation datasets such as HAR, HHAR, WISDM, and SSC. Our approach shows average accuracy of 95.10%, 78.04%, 78.69%, and 74.25% for HAR, HHAR, WISDM, and SSC, respectively.

Research on A Ship Trajectory Classification Method Based on Deep Learning

The unrestricted development and utilization of marine resources have resulted in a series of practical problems, such as the destruction of marine ecology. The wide application of radar, satellites and other detection equipment has gradually led to a large variety of large-capacity marine spatiotemporal trajectory data from a vast number of sources. In the field of marine domain awareness, there is an urgent need to use relevant information technology means to control and monitor ships and accurately classify and identify ship behavior patterns through multisource data fusion analysis. In addition, the increase in the type and quantity of trajectory data has produced a corresponding increase in the complexity and difficulty of data processing that cannot be adequately addressed by traditional data mining algorithms. Therefore, this paper provides a deep learning-based algorithm for the recognition of four main motion types of the ship from automatic identification system (AIS) data: anchoring, mooring, sailing and fishing. A new method for classifying patterns is presented that combines the computer vision and time series domains. Experiments are carried out on a dataset constructed from the open AIS data of ships in the coastal waters of the United States, which show that the method proposed in this paper achieves more than 95\% recognition accuracy. The experimental results confirm that the method proposed in this paper is effective in classifying ship trajectories using AIS data and that it can provide efficient technical support for marine supervision departments.

A Historical Survey of Advances in Transformer Architectures

In recent times, transformer-based deep learning models have risen in prominence in the field of machine learning for a variety of tasks such as computer vision and text generation. Given this increased interest, a historical outlook at the development and rapid progression of transformer-based models becomes imperative in order to gain an understanding of the rise of this key architecture. This paper presents a survey of key works related to the early development and implementation of transformer models in various domains such as generative deep learning and as backbones of large language models. Previous works are classified based on their historical approaches, followed by key works in the domain of text-based applications, image-based applications, and miscellaneous applications. A quantitative and qualitative analysis of the various approaches is presented. Additionally, recent directions of transformer-related research such as those in the biomedical and timeseries domains are discussed. Finally, future research opportunities, especially regarding the multi-modality and optimization of the transformer training process, are identified.

Decoupling representation contrastive learning for carbon emission prediction and analysis based on time series

Carbon emissions have become particularly urgent with the acceleration of industrialization. Currently, AI-based carbon emission modeling provides essential support for solving the problem of predicting, managing, and optimizing carbon emissions. However, there are still challenges in accurately predicting carbon dioxide emissions and establishing an appropriate carbon emission regulatory model to manage carbon dioxide emissions effectively. Firstly, this paper introduces a framework for learning seasonal and trend representations called DeTF (Decoupling representation for Time and Frequency domain of time series). The framework is designed for time series data of CO2 emissions used for power generation from 1973 to 2023. Secondly, we use various data enhancement techniques to improve the robustness of the learned representations. Finally, the prediction results establish a tripartite evolutionary game relationship between the state, regulatory agencies, and enterprises. We use MSE and MAE as evaluation metrics, and our model overall outperforms current state-of-the-art end-to-end algorithms. It outperforms the best performing end-to-end prediction method by 27.08% (MSE) in the prediction results. Besides, the time series prediction results are innovatively combined with the tripartite game model to analyze the main factors determining the optimal strategy.

Transferable Time-Series Forecasting Under Causal Conditional Shift.

This paper focuses on the problem of semi-supervised domain adaptation for time-series forecasting, which is underexplored in literature, despite being often encountered in practice. Existing methods on time-series domain adaptation mainly follow the paradigm designed for static data, which cannot handle domain-specific complex conditional dependencies raised by data offset, time lags, and variant data distributions. In order to address these challenges, we analyze variational conditional dependencies in time-series data and find that the causal structures are usually stable among domains, and further raise the causal conditional shift assumption. Enlightened by this assumption, we consider the causal generation process for time-series data and propose an end-to-end model for the semi-supervised domain adaptation problem on time-series forecasting. Our method can not only discover the Granger-Causal structures among cross-domain data but also address the cross-domain time-series forecasting problem with accurate and interpretable predicted results. We further theoretically analyze the superiority of the proposed method, where the generalization error on the target domain is bounded by the empirical risks and by the discrepancy between the causal structures from different domains. Experimental results on both synthetic and real data demonstrate the effectiveness of our method for the semi-supervised domain adaptation method on time-series forecasting.

Domain Generalization in Time Series Forecasting

Domain generalization aims to design models that can effectively generalize to unseen target domains by learning from observed source domains. Domain generalization poses a significant challenge for time series data, due to varying data distributions and temporal dependencies. Existing approaches to domain generalization are not designed for time series data, which often results in suboptimal or unstable performance when confronted with diverse temporal patterns and complex data characteristics. We propose a novel approach to tackle the problem of domain generalization in time series forecasting. We focus on a scenario where time series domains share certain common attributes and exhibit no abrupt distribution shifts. Our method revolves around the incorporation of a key regularization term into an existing time series forecasting model: domain discrepancy regularization . In this way, we aim to enforce consistent performance across different domains that exhibit distinct patterns. We calibrate the regularization term by investigating the performance within individual domains and propose the domain discrepancy regularization with domain difficulty awareness . We demonstrate the effectiveness of our method on multiple datasets, including synthetic and real-world time series datasets from diverse domains such as retail, transportation, and finance. Our method is compared against traditional methods, deep learning models, and domain generalization approaches to provide comprehensive insights into its performance. In these experiments, our method showcases superior performance, surpassing both the base model and competing domain generalization models across all datasets. Furthermore, our method is highly general and can be applied to various time series models.

MDLR: A Multi-Task Disentangled Learning Representations for unsupervised time series domain adaptation

Unsupervised Time Series Domain Adaptation (UTSDA) is a method for transferring information from a labeled source domain to an unlabeled target domain. The majority of existing UTSDA approaches focus on learning a domain-invariant feature space by reducing the gap between domains. However, the single-task representation learning methods have limited expressive capability, while ignoring the distinctive season-related and trend-related domain-invariant mechanisms across different domains. To address this, we introduce a novel approach, distinct from existing methods, through a theoretical analysis of UTSDA from the perspective of causal inference. This analysis establishes a solid theoretical foundation for identifying and modeling such consistent domain-invariant mechanisms, which is a significant advancement in the field. As a solution, we introduce MDLR, a multi-task disentangled learning framework designed for UTSDA. MDLR utilizes a dual-tower architecture with a trend feature extractor (TFE) and a season feature extractor (SFE) to extract trend-related and season-related information. This approach ensures that domain-invariant features at different scales can be better represented. Additionally, MDLR is designed with two tasks: a label classifier and a domain classifier, enabling iterative training of the entire model. The experiments conducted on three datasets, namely UCIHAR, WISDM, and HHAR_SA, along with visualization results, have shown the effectiveness of the proposed approach. The source code for our MDLR model is available to the public at https://github.com/MoranCoder95/MDLR/.

Layerwise Quantum Deep Reinforcement Learning for Joint Optimization of UAV Trajectory and Resource Allocation

This study proposes a layerwise quantum-based deep reinforcement learning (LQ-DRL) method for optimizing continuous large space and time series problems using deep layer training. The actions in LQ-DRL are optimized using a layerwise quantum embedding that leverages the advantages of quantum computing to maximize reward and reduce training loss. Moreover, this study employs a local loss to minimize the occurrence of barren plateaus phenomena and further enhance performance. As a particular case, the proposed scheme is employed to jointly optimize: (1) UAV trajectory planning, (2) user grouping, and (3) power allocation for higher energy efficiency of a UAV as the reward. The combination of these optimized factors is referred to as action space in the presented LQ-DRL. The LQ-DRL is employed to solve the optimization problem due to its non-convexity, continuous and large action space, and time-series domain. In a practical view, LQ-DRL aims to solve the issue of energy consumption related to limited-battery energy of a UAV base station while maintaining quality-of-service (QoS) for users, by gaining maximum energy efficiency as the reward. One of real applications, as an example, LQ-DRL can be employed to maximize the energy efficiency of a UAV base station in UAV empowered disaster recovery networks scenario. The quantum circuits of layerwise quantum embedding are presented to show the practical implementation in noisy intermediate-scale quantum computers. Based on the results, LQ-DRL outperformed the classical DRL by achieving higher effective dimension, rewards, and lower learning losses. In addition, better performances were achieved using more layers.

Self-Supervised Autoregressive Domain Adaptation for Time Series Data.

Unsupervised domain adaptation (UDA) has successfully addressed the domain shift problem for visual applications. Yet, these approaches may have limited performance for time series data due to the following reasons. First, they mainly rely on the large-scale dataset (i.e., ImageNet) for source pretraining, which is not applicable for time series data. Second, they ignore the temporal dimension on the feature space of the source and target domains during the domain alignment step. Finally, most of the prior UDA methods can only align the global features without considering the fine-grained class distribution of the target domain. To address these limitations, we propose a SeLf-supervised AutoRegressive Domain Adaptation (SLARDA) framework. In particular, we first design a self-supervised (SL) learning module that uses forecasting as an auxiliary task to improve the transferability of source features. Second, we propose a novel autoregressive domain adaptation technique that incorporates temporal dependence of both source and target features during domain alignment. Finally, we develop an ensemble teacher model to align class-wise distribution in the target domain via a confident pseudo labeling approach. Extensive experiments have been conducted on three real-world time series applications with 30 cross-domain scenarios. The results demonstrate that our proposed SLARDA method significantly outperforms the state-of-the-art approaches for time series domain adaptation. Our source code is available at: https://github.com/mohamedr002/SLARDA.

Large Scale Time-Series Representation Learning via Simultaneous Low-and High-Frequency Feature Bootstrapping.

Learning representations from unlabeled time series data is a challenging problem. Most existing self-supervised and unsupervised approaches in the time-series domain fall short in capturing low-and high-frequency features at the same time. As a result, the generalization ability of the learned representations remains limited. Furthermore, some of these methods employ large-scale models like transformers or rely on computationally expensive techniques such as contrastive learning. To tackle these problems, we propose a noncontrastive self-supervised learning (SSL) approach that efficiently captures low-and high-frequency features in a cost-effective manner. The proposed framework comprises a Siamese configuration of a deep neural network with two weight-sharing branches which are followed by low-and high-frequency feature extraction modules. The two branches of the proposed network allow bootstrapping of the latent representation by taking two different augmented views of raw time series data as input. The augmented views are created by applying random transformations sampled from a single set of augmentations. The low-and high-frequency feature extraction modules of the proposed network contain a combination of multilayer perceptron (MLP) and temporal convolutional network (TCN) heads, respectively, which capture the temporal dependencies from the raw input data at various scales due to the varying receptive fields. To demonstrate the robustness of our model, we performed extensive experiments and ablation studies on five real-world time-series datasets. Our method achieves state-of-art performance on all the considered datasets.

Out-of-distribution Detection in Time-series Domain: A Novel Seasonal Ratio Scoring Approach

Safe deployment of time-series classifiers for real-world applications relies on the ability to detect the data that is not generated from the same distribution as training data. This task is referred to as out-of-distribution (OOD) detection. We consider the novel problem of OOD detection for the time-series domain. We discuss the unique challenges posed by time-series data and explain why prior methods from the image domain will perform poorly. Motivated by these challenges, this article proposes a novel Seasonal Ratio Scoring (SRS) approach. SRS consists of three key algorithmic steps. First, each input is decomposed into class-wise semantic component and remainder. Second, this decomposition is employed to estimate the class-wise conditional likelihoods of the input and remainder using deep generative models. The seasonal ratio score is computed from these estimates. Third, a threshold interval is identified from the in-distribution data to detect OOD examples. Experiments on diverse real-world benchmarks demonstrate that the SRS method is well-suited for time-series OOD detection when compared to baseline methods.

CALDA: Improving Multi-Source Time Series Domain Adaptation With Contrastive Adversarial Learning.

Unsupervised domain adaptation (UDA) provides a strategy for improving machine learning performance in data-rich (target) domains where ground truth labels are inaccessible but can be found in related (source) domains. In cases where meta-domain information such as label distributions is available, weak supervision can further boost performance. We propose a novel framework, CALDA, to tackle these two problems. CALDA synergistically combines the principles of contrastive learning and adversarial learning to robustly support multi-source UDA (MS-UDA) for time series data. Similar to prior methods, CALDA utilizes adversarial learning to align source and target feature representations. Unlike prior approaches, CALDA additionally leverages cross-source label information across domains. CALDA pulls examples with the same label close to each other, while pushing apart examples with different labels, reshaping the space through contrastive learning. Unlike prior contrastive adaptation methods, CALDA requires neither data augmentation nor pseudo labeling, which may be more challenging for time series. We empirically validate our proposed approach. Based on results from human activity recognition, electromyography, and synthetic datasets, we find utilizing cross-source information improves performance over prior time series and contrastive methods. Weak supervision further improves performance, even in the presence of noise, allowing CALDA to offer generalizable strategies for MS-UDA.

NarmViz: A novel method for visualization of time series numerical association rules for smart agriculture

AbstractNumerical association rule mining (NARM) is a popular method under the umbrella of data mining, focused on finding relationships between attributes in transaction databases. Numerical association rules for time series are a new paradigm that extends the applicability of NARM to the domain of time series. Association rule mining algorithms result in numerous rules, the interpretation of which is sometimes not easy for human experts. Therefore, various visualization methods have been developed to improve the explanation results of the rule mining process. This article is a novel contribution to the development of a new visualization method capable of presenting the association rules for time series developed according to the principles of explainable artificial intelligence. The experiments are conducted in the context of smart agriculture (i.e., agricultural time series data), and show the great potential of the proposed visualization method for the future.

Anomaly Detection Using an Ensemble of Multi-Point LSTMs.

As technologies for storing time-series data such as smartwatches and smart factories become common, we are collectively accumulating a great deal of time-series data. With the accumulation of time-series data, the importance of time-series abnormality detection technology that detects abnormal patterns such as Cyber-Intrusion Detection, Fraud Detection, Social Networks Anomaly Detection, and Industrial Anomaly Detection is emerging. In the past, time-series anomaly detection algorithms have mainly focused on processing univariate data. However, with the development of technology, time-series data has become complicated, and corresponding deep learning-based time-series anomaly detection technology has been actively developed. Currently, most industries rely on deep learning algorithms to detect time-series anomalies. In this paper, we propose an anomaly detection algorithm with an ensemble of multi-point LSTMs that can be used in three cases of time-series domains. We propose our anomaly detection model that uses three steps. The first step is a model selection step, in which a model is learned within a user-specified range, and among them, models that are most suitable are automatically selected. In the next step, a collected output vector from M LSTMs is completed by stacking ensemble techniques of the previously selected models. In the final step, anomalies are finally detected using the output vector of the second step. We conducted experiments comparing the performance of the proposed model with other state-of-the-art time-series detection deep learning models using three real-world datasets. Our method shows excellent accuracy, efficient execution time, and a good F1 score for the three datasets, though training the LSTM ensemble naturally requires more time.

SIDA

The coronavirus disease 2019 (COVID-19) pneumonia still persists and its chief complaint is dry cough. Physicians design wireless stethoscopes to facilitate diagnosis, however, lung sounds could be easily interfered with by external noises. To achieve lung sound enhancement, prior researches mostly assume the amount of clean and noisy data are the same. This assumption is hardly met due to extensive labor effort for data collection and annotation. The data imbalance across domains widely happens in real-world IoT systems, e.g. sound enhancement and WiFi-based human sensing. In this paper, we propose SIDA, a self-supervised imbalanced domain adaptation framework for sound enhancement and WiFi sensing, which makes it a generic time series domain adaptation solution for IoT systems. SIDA proposes a self-supervised imbalanced domain adaptation model that separately learns the representation of time series signals in a minority domain with limited samples, a majority domain with rich samples, and their mapping relations. For lung sound enhancement, we further proposes a phase correction model to sanitize the phase and a SNR prediction algorithm to recursively perform domain adaptation in an imbalanced noisy and clean lung sound dataset. Extensive experiments demonstrate SIDA increases noisy samples' SNR by 16.49dB and 4.06dB on a synthetic and a realistic imbalanced lung sound dataset, respectively. For WiFi-based human sensing, SIDA designs a cross-domain WiFi-based human identification model irrespective of walking trajectory. A specific trajectory where a group of people walks along in a realistic testing environment is considered the minority domain, and several other trajectories are stored at a server as the majority domain. Extensive experiments show SIDA could recognize individuals with an average accuracy of 94.72% and significantly outperform baselines on highly imbalanced WiFi dataset in cross-domain human identification tasks.