Large Time Series Datasets Research Articles

Lidar cloud and aerosol layer detection method based on point cloud filtering

A point cloud filtering method is presented for atmospheric layer detection from lidar data. The method involves rising edge event recognition based on a wavelet transform function. Density-based clustering was then utilized to separate the real boundary from the original noisy point clouds based on continuous distribution characteristics of cloud and aerosol layer. Tests were carried out to verify the performance of our algorithm with synthetic lidar signals with noise. The layer base detection error within ± 5 bins was achieved for signals with SNRs higher than 3. Even for SNRs higher than 1, high consistency was still observed between retrieved results with our method and a visual analysis. These results indicate that our algorithm is suitable for unsupervised detection with large time-series datasets, such as CALIOP.

DeepSketch: A Query Sketching Interface for Deep Time Series Similarity Search

By empowering domain experts to perform interactive exploration of large time series datasets, sketch-based query interfaces have revitalized interest in the well-studied problem of time series similarity search. In this new interaction paradigm, recent similarity algorithms (e.g., Qetch, Peax, LineNet) that attempt to capture perceptually relevant features have supplanted older, more straightforward distance measures (e.g., Euclidean, DTW). However, the downside of these algorithms is the resulting difficulty in designing corresponding index structures to support efficient similarity search over large datasets, thus necessitating brute-force search. This demo will showcase Deep Time Series Similarity Search (DTS3), our pluggable indexing pipeline for arbitrary distance measures. DTS3 can automatically train a foundation model for any custom, user-supplied distance measure with no strict constraints (e.g., differentiability), thus enabling fast retrieval via an off-the-shelf vector DBMS. Using our DeepSketch web interface, participants can compare DTS3 to the baseline brute-force versions of several similarity algorithms to see that our approach can achieve much lower latency without sacrificing accuracy when searching over large, real-world time series datasets.

Enhancing bridge damage assessment: Adaptive cell and deep learning approaches in time-series analysis

In recent years, the application of Deep Learning (DL) for damage detection in Structural Health Monitoring (SHM) using time-series data has garnered significant attention from the global scientific community. By incorporating numerous filters and units in their architecture, these models facilitate feature extraction making the processing of large time-series datasets in SHM more effective. However, optimizing of the number of filters and units is a challenge, often relying on scientists' experience or previous models without a systematic method for measurement or automatic selection. Therefore, this study proposes a novel framework to enhance the accuracy and efficiency of SHM for damage detection in bridge structures by using metaheuristic algorithms, Electric Eel Foraging Optimization (EEFO) algorithm, to optimize hyperparameters of DL model. This DL model combines the advantages of each traditional model. Specifically, 1D Convolutional Neural Network (1DCNN) is employed for features extraction, Gated Recurrent Units (GRU) for identifying long-term dependencies, and Residual Networks (ResNet) for avoiding vanishing gradient problem which often happens in DL model during training, referred to as 1DCNN-GRU-ResNet. The time-series dataset generated from Cua Rao bridge is used to demonstrate the effectiveness of the proposed method. 1DCNN-GRU-ResNet after optimization (1DCNN-GRU-ResNet-opt) achieves 91.6 %, significantly surpassing traditional 1DCNN (82.5 %), GRU (79.9 %), 1DCNN-GRU (85.7 %), and 1DCNN-GRU-ResNet (89.3 %) in the test set. The proposed 1DCNN-GRU-ResNet-opt approach demonstrates considerable potential in practical applications for SHM, offering high accuracy and efficiency.

Variance-Reduced Stochastic Optimization for Efficient Inference of Hidden Markov Models

Hidden Markov models (HMMs) are popular models to identify a finite number of latent states from sequential data. However, fitting them to large datasets can be computationally demanding because most likelihood maximization techniques require iterating through the entire underlying dataset for every parameter update. We propose a novel optimization algorithm that updates the parameters of an HMM without iterating through the entire dataset. Namely, we combine a partial E step with variance-reduced stochastic optimization within the M step. We prove the algorithm converges under certain regularity conditions. We test our algorithm empirically using a simulation study as well as a case study of kinematic data collected using suction-cup attached biologgers from eight northern resident killer whales (Orcinus orca) off the western coast of Canada. In both, our algorithm converges in fewer epochs, with less computation time, and to regions of higher likelihood compared to standard numerical optimization techniques. Our algorithm allows practitioners to fit complicated HMMs to large time-series datasets more efficiently than existing baselines. Supplemental materials are available online.

Reconstruction of stochastic dynamics from large streamed datasets.

The complex dynamics of physical systems can often be modeled with stochastic differential equations. However, computational constraints inhibit the estimation of dynamics from large time-series datasets. I present a method for estimating drift and diffusion functions from inordinately large datasets through the use of incremental, online, updating statistics. I demonstrate the validity and utility of this method by analyzing three large, varied synthetic datasets, as well as an empirical turbulence dataset. This method will hopefully facilitate the analysis of complex systems from exceedingly large, "big data" scientific datasets, as well as real-time streamed data.

Generating radar signals using one-dimensional GAN-based model for target classification in radar systems

Conventional radar systems are often unable to produce highly accurate results for target classification and identification via linear frequency modulation (LFM) signals. The potential of artificial intelligence, particularly deep learning, has been applied in various fields, which promotes utilizing them in the context of target classification in radar systems. However, to train deep learning models for this task, large datasets of LFM radar signals are required, which are practically difficult to obtain due to the time, effort, and involved high cost. Therefore, the presented work spots the light on utilizing the recent one-dimensional generative adversarial network (GAN) and Wasserstein GAN (WGAN) models to synthesize a large time-series LFM signal dataset from a reference smaller one. Moreover, the work fairly judges the generated LFM signals realistic via a decent qualitative and quantitative analysis, unlike other studies which rely solely on qualitative evaluation by human observers. The proposed study outcome reveals the WGAN’s efficiency in synthesizing high-quality LFM signals while reducing the training time and resource requirements.

Data-Driven Prognostics and Diagnostics of Industrial Machinery — A Turbofan Engine Case Study

A machine’s Remaining Useful Life (RUL) is the expected life or usage time remaining before the machine requires repair or replacement. In data-driven methods, typical RUL estimation is performed using models trained with health condition indicator values derived from measured system data. A significant challenge in developing an RUL estimation model is transforming large, multivariate, noisy sensor datasets into useful format(s) that make the data analysis and processing pipeline efficient and extract valuable condition indicators from the data. This work uses the N-CMAPSS dataset to explore options and implications for efficiently organizing and storing large time-series datasets to support prognostics and diagnostics applications. We extend the work to demonstrate a predictive maintenance workflow and solution to (1) detect and classify faults in a turbofan engine and (2) estimate the RUL once we detect performance degradation. Under data engineering, we investigate the impact of various file formats and file types on memory and execution time when dealing with large datasets like N-CMAPSS. We analyze, pre-process, and extract/engineer critical features from the transformed dataset by leveraging our understanding of gas turbines' operation (e.g., Brayton Cycle). We also analyze the performance of various engine submodules for different flight phases (climb, cruise, and descent). This work also explains an approach to down-sample the time series data without losing information relevant to our goals. Using the health condition indicators derived and synthesized in the data engineering stage, we train machine learning models for diagnostics (differentiate between healthy operation and seven different types of faults in the turbofan engine) and prognostics (RUL estimation).

An interactive time series image analysis software for dendritic spines

Live fluorescence imaging has demonstrated the dynamic nature of dendritic spines, with changes in shape occurring both during development and in response to activity. The structure of a dendritic spine correlates with its functional efficacy. Learning and memory studies have shown that a great deal of the information stored by a neuron is contained in the synapses. High precision tracking of synaptic structures can give hints about the dynamic nature of memory and help us understand how memories evolve both in biological and artificial neural networks. Experiments that aim to investigate the dynamics behind the structural changes of dendritic spines require the collection and analysis of large time-series datasets. In this paper, we present an open-source software called SpineS for automatic longitudinal structural analysis of dendritic spines with additional features for manual intervention to ensure optimal analysis. We have tested the algorithm on in-vitro, in-vivo, and simulated datasets to demonstrate its performance in a wide range of possible experimental scenarios.

Dynamic Time Warping Under Product Quantization, With Applications to Time-Series Data Similarity Search

The similarity search on sensor data generated by a myriad of sensing devices is a frequently encountered problem in the era of the Internet of Things (IoT). This sensor data generally appear in the form of time series, a temporally ordered sequence of real numbers obtained regularly in time. It has been widely accepted that the dynamic time warping (DTW) currently is the most prevalent similarity measure in the time-series mining community, mainly due to its flexibility and broad applicability. However, calculating DTW between two time series has quadratic time complexity, leading to unsatisfactory efficiency when performing the similarity search over the large time-series data set. The main contribution of this article is to propose a method called product quantization (PQ)-based DTW (PQDTW) for fast time-series approximate similarity search under DTW. The PQ, a well-known approximate nearest neighbor search approach, is used in PQDTW. Nevertheless, the conventional PQ is developed with the Euclidean distance and is not designed for DTW. To solve this problem, the DTW barycenter averaging (DBA) technique is utilized to adapt the PQ for DTW before using it. We employ PQDTW along with the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">filter-and-refine</i> framework to efficiently and accurately perform the time-series similarity search. Our method can reasonably reduce many DTW computations in the filtering phase; thus, the query process is accelerated. We compare PQDTW with related popular algorithms using public time-series data sets. Experimental results verify that the proposal achieves the best tradeoff between query efficiency and retrieval accuracy compared to the competitors.

TimeSpec4LULC: a global multispectral time series database for training LULC mapping models with machine learning

Abstract. Land use and land cover (LULC) mapping are of paramount importance to monitor and understand the structure and dynamics of the Earth system. One of the most promising ways to create accurate global LULC maps is by building good quality state-of-the-art machine learning models. Building such models requires large and global datasets of annotated time series of satellite images, which are not available yet. This paper presents TimeSpec4LULC (https://doi.org/10.5281/zenodo.5913554; Khaldi et al., 2022), a smart open-source global dataset of multispectral time series for 29 LULC classes ready to train machine learning models. TimeSpec4LULC was built based on the seven spectral bands of the MODIS sensors at 500 m resolution, from 2000 to 2021, and was annotated using spatial–temporal agreement across the 15 global LULC products available in Google Earth Engine (GEE). The 22-year monthly time series of the seven bands were created globally by (1) applying different spatial–temporal quality assessment filters on MODIS Terra and Aqua satellites; (2) aggregating their original 8 d temporal granularity into monthly composites; (3) merging Terra + Aqua data into a combined time series; and (4) extracting, at the pixel level, 6 076 531 time series of size 262 for the seven bands along with a set of metadata: geographic coordinates, country and departmental divisions, spatial–temporal consistency across LULC products, temporal data availability, and the global human modification index. A balanced subset of the original dataset was also provided by selecting 1000 evenly distributed samples from each class such that they are representative of the entire globe. To assess the annotation quality of the dataset, a sample of pixels, evenly distributed around the world from each LULC class, was selected and validated by experts using very high resolution images from both Google Earth and Bing Maps imagery. This smartly, pre-processed, and annotated dataset is targeted towards scientific users interested in developing various machine learning models, including deep learning networks, to perform global LULC mapping.

A method for finding anomalous astronomical light curves and their analogues

ABSTRACT Our understanding of the Universe has profited from deliberate targeted studies of known phenomena, as well as from serendipitous unexpected discoveries, such as the discovery of a complex variability pattern in the direction of KIC 8462852 (Boyajian’s star). Upcoming surveys such as the Vera C. Rubin Observatory Legacy Survey of Space and Time will explore the parameter space of astrophysical transients at all time-scales, and offer the opportunity to discover even more extreme examples of unexpected phenomena. We investigate strategies to identify novel objects and to contextualize them within large time-series data sets in order to facilitate the discovery of new classes of objects as well as the physical interpretation of their anomalous nature. We develop a method that combines tree-based and manifold-learning algorithms for anomaly detection in order to perform two tasks: 1) identify and rank anomalous objects in a time-domain data set; and 2) group those anomalies according to their similarity in order to identify analogues. We achieve the latter by combining an anomaly score from a tree-based method with a dimensionality manifold-learning reduction strategy. Clustering in the reduced space allows for the successful identification of anomalies and analogues. We also assess the impact of pre-processing and feature engineering schemes and investigate the astrophysical nature of the objects that our models identify as anomalous by augmenting the Kepler data with Gaia colour and luminosity information. We find that multiple models, used in combination, are a promising strategy to identify novel light curves and light curve families.

Analyzing and Modeling Special Offer Campaigns in Location-Based Social Networks

The proliferation of mobile handheld devices in combination with the technological advancements in mobile computing has led to a number of innovative services that make use of the location information available on such devices. Traditional yellow pages websites have now moved to mobile platforms, giving the opportunity to local businesses and potential, near-by, customers to connect. These platforms can offer an affordable advertisement channel to local businesses. One of the mechanisms offered by location-based social networks (LBSNs) allows businesses to provide special offers to their customers that connect through the platform. We collect a large time-series dataset from approximately 14 million venues on Foursquare and analyze the performance of such campaigns using randomization techniquesand (non-parametric) hypothesis testing with statistical bootstrapping. Our main finding indicates that this type of promotions are not as effective as anecdote success stories might suggest. Finally, we design classifiers by extracting three different types of features that are able to provide an educated decision on whether a special offer campaign for a local business will succeed or not both in short and long term.

Functional Data Representation of Inertial Sensor-based Torso-Thigh, Knee, and Ankle Movements during Lifting.

This study examined the goodness-of-fit of using a sigmoid function to characterize time-series angular displacement trajectories during two-handed anterior lifting. Twenty-six participants performed two-handed anterior lifting with a low (4.5 kg) vs. high (22.7 kg) load at floor vs. knee lifting height. A sigmoid function with three parameters was fit to the torso-thigh included angle, knee flexion-extension (F-E), and ankle F-E angles in the sagittal plane obtained from body-worn inertial sensors. Mean ± SD RMSE between measured vs. fitted trajectories were 3.6 ± 2.9°, 3.9 ± 4.2°, and 2.7 ± 2.8° for the torso-thigh included angle, knee F-E, and ankle F-E angles, respectively. Findings suggest that the sigmoid function adequately describes the trajectory shape of two-handed lifting kinematics. Functional representations facilitate data aggregation and feature extraction in large time-series datasets encountered in inertial-based motion analysis and machine learning applications.

Uncertainty Analysis on Electric Power Consumption

The analysis of large time-series datasets has profoundly enhanced our ability to make accurate predictions in many fields. However, unpredictable phenomena, such as extreme weather events or the novel coronavirus 2019 (COVID-19) outbreak, can greatly limit the ability of time-series analyses to establish reliable patterns. The present work addresses this issue by applying uncertainty analysis using a probability distribution function, and applies the proposed scheme within a preliminary study involving the prediction of power consumption for a single hotel in Seoul, South Korea based on an analysis of 53,567 data items collected by the Korea Electric Power Corporation using robotic process automation. We first apply Facebook Prophet for conducting time-series analysis. The results demonstrate that the COVID-19 outbreak seriously compromised the reliability of the time-series analysis. Then, machine learning models are developed in the TensorFlow framework for conducting uncertainty analysis based on modeled relationships between electric power consumption and outdoor temperature. The benefits of the proposed uncertainty analysis for predicting the electricity consumption of the hotel building are demonstrated by comparing the results obtained when considering no uncertainty, aleatory uncertainty, epistemic uncertainty, and mixed aleatory and epistemic uncertainty. The minimum and maximum ranges of predicted electricity consumption are obtained when using mixed uncertainty. Accordingly, the application of uncertainty analysis using a probability distribution function greatly improved the predictive power of the analysis compared to time-series analysis. © This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Robust archetypoids for anomaly detection in big functional data

Archetypoid analysis (ADA) has proven to be a successful unsupervised statistical technique to identify extreme observations in the periphery of the data cloud, both in classical multivariate data and functional data. However, two questions remain open in this field: the use of ADA for outlier detection and its scalability. We propose to use robust functional archetypoids and adjusted boxplot to pinpoint functional outliers. Furthermore, we present a new archetypoid algorithm for obtaining results from large data sets in reasonable time. Functional time series are occurring in many practical problems, so this paper focuses on functional data settings. The new algorithm for detecting functional anomalies, called CRO-FADALARA, can be used with both univariate and multivariate curves. Our proposal for outlier detection is compared with all the state-of-the-art methods in a controlled study, showing a good performance. Furthermore, CRO-FADALARA is applied to two large time series data sets, where outliers curves are discussed and the reduction in computational time is clearly stated. A third case study with a small ECG data set is discussed, given its importance in functional data scenarios. All data, R code and a new R package are freely available.

Efficient segmentation-based methods for anomaly detection in static and streaming time series under dynamic time warping

The problem of time series anomaly detection has attracted a lot of attention due to its usefulness in various application domains. However, most of the methods proposed so far used Euclidean distance to deal with this problem. Dynamic Time Warping (DTW) distance is more suitable than Euclidean distance because of its capability in shape-based similarity checking in many practical fields, for example those with multimedia data. In this paper, we propose two efficient anomaly detection methods, EP-Leader-DTW and SEP-Leader-DTW, for static and streaming time series under DTW, respectively. Our methods are based on time series segmentation, subsequence clustering, and anomaly scoring. For segmentation, the major extrema method is used to obtain subsequences. For clustering, we apply Leader algorithm to cluster the subsequences along with a lower bounding technique to accelerate DTW distance computation. Experimental results on several benchmark time series datasets reveal that our method for anomaly detection in static time series under DTW can perform very fast and accurately on large time series datasets. For streaming time series, our method can meet the instantaneous requirement with high accuracy. As a result, our anomaly detection methods are applicable to both static and streaming time series in practice.

Introducing time series snippets: a new primitive for summarizing long time series

The first question a data analyst asks when confronting a new dataset is often, “Show me some representative/typical data.” Answering this question is simple in many domains, with random samples or aggregate statistics of some kind. Surprisingly, it is difficult for large time series datasets. The major difficulty is not time or space complexity, but defining what it means to be representative data for this data type. In this work, we show that the obvious candidate definitions: motifs, shapelets, cluster centers, random samples etc., are all poor choices. We introduce time series snippets, a novel representation of typical time series subsequences. Informally, time series snippets can be seen as the answer to the following question. If a user, which could be a human or a higher-level algorithm, only has resources (including human time) to inspect k subsequences of a long time series, which k subsequences should be chosen? Beyond their utility for visualizing and summarizing massive time series collections, we show that time series snippets have utility for high-level comparison of large time series collections.

Entropy-Based Strategies for Rapid Pre-Processing and Classification of Time Series Data from Single-Molecule Force Experiments.

Recent advances in single-molecule science have revealed an astonishing number of details on the microscopic states of molecules, which in turn defined the need for simple, automated processing of numerous time-series data. In particular, large datasets of time series of single protein molecules have been obtained using laser optical tweezers. In this system, each molecular state has a separate time series with a relatively uneven composition from the point of view-point of local descriptive statistics. In the past, uncertain data quality and heterogeneity of molecular states were biased to the human experience. Because the data processing information is not directly transferable to the black-box-framework for an efficient classification, a rapid evaluation of a large number of time series samples simultaneously measured may constitute a serious obstacle. To solve this particular problem, we have implemented a supervised learning method that combines local entropic models with the global Lehmer average. We find that the methodological combination is suitable to perform a fast and simple categorization, which enables rapid pre-processing of the data with minimal optimization and user interventions.

Detecting and quantifying causal associations in large nonlinear time series datasets.

Identifying causal relationships and quantifying their strength from observational time series data are key problems in disciplines dealing with complex dynamical systems such as the Earth system or the human body. Data-driven causal inference in such systems is challenging since datasets are often high dimensional and nonlinear with limited sample sizes. Here, we introduce a novel method that flexibly combines linear or nonlinear conditional independence tests with a causal discovery algorithm to estimate causal networks from large-scale time series datasets. We validate the method on time series of well-understood physical mechanisms in the climate system and the human heart and using large-scale synthetic datasets mimicking the typical properties of real-world data. The experiments demonstrate that our method outperforms state-of-the-art techniques in detection power, which opens up entirely new possibilities to discover and quantify causal networks from time series across a range of research fields.

Enhanced time-aware QoS prediction in multi-cloud: a hybrid k-medoids and lazy learning approach (QoPC)

Cloud service providers should be able to predict the future states of their infrastructure in order to avoid any violation of Service Level Agreement. This becomes more complex when vendors have to deal with services from various providers in multi-clouds. As a result, QoS prediction can significantly support service providers in a better understanding of their resources future states. Users should also be very well aware of their resource needs, as well as the Quality of Service relative values. This paper proposes a hybrid approach to the prediction of the future value of the QoS features. The hybrid approach uses a modified version of k-medoids algorithm for the clustering of large time-series datasets, as well as a proposed algorithm inspired from the lazy learning and lower bound Dynamic Time Warping (LB-Keogh) for pruned DTW computations. The proposed method in this manuscript is a shape-based QoS prediction with a novel pre-processing method, which fulfills the missing data with statistically semi-real data. In order to solve the cold start problem, we proposed new algorithm based on the DTW Barycenter Averaging (DBA) algorithm. The results showed that our predicted values are very close to real values and achieve only 0.35 of the normalized mean absolute error rate, on average, for the WSDream dataset and 0.07 for the Alibaba dataset.