Outlier Detection In Time Series Research Articles

Probabilistic outlier detection for robust regression modeling of structural response for high-speed railway track monitoring

Outlier detection is an important procedure taken in structural health monitoring (SHM) to create clean and reliable data. A robust time series outlier detection method incorporating a Bayesian perspective and an extreme learning machine (ELM) neural network model is proposed, with application to long-term monitoring data of ballastless tracks for high-speed railway systems. A robust sparse Bayesian ELM (SBELM) model is first established by computing the posterior probability density function of the ELM weight parameters and then marginalizing over the prediction-error precision parameter to obtain a robust nonlinear regression model between the track temperature and structural response. Both the posterior mean and the associated uncertainties of the robust SBELM model are then taken into account to compute the outlier probability for each suspicious data point, which quantifies their degree of data “outlier-ness.” It effectively takes into account the prediction uncertainty of the SBELM regression model. The method is applied to long-term monitoring data for track temperatures, and track strain and relative displacement responses, from two high-speed rail track systems where there are both slight and serious outliers. The results demonstrate that the proposed method can reliably detect outliers by quantifying the outlier probability and that the final results are robust to the selection of the “thresholds.” It is also shown that our new algorithm produces significantly improved model prediction performance after the outliers are detected and removed.

Unsupervised TCN-AE-Based Outlier Detection for Time Series With Seasonality and Trend for Cellular Networks

Timely identification of outliers occurring in key performance indicators (KPIs) of mobile cellular networks is crucial for prompt action to unexpected events. The KPIs of cellular networks typically exhibit seasonality and trend effects and these make the detection of outliers challenging. In this paper, an online unsupervised deep machine learning (DML) algorithm for outlier detection for time series with seasonality and trend is proposed. The proposed algorithm utilizes a neural network based on a temporal convolution network (TCN) and an autoencoder (AE) to reconstruct a time series that captures the normal behavior of the input data and then the reconstruction errors between the input and reconstructed time series at the output of the TCN-AE network are used for outlier detection. To train the TCN-AE network to learn the normal behavior of the input time series, we propose a two-step training process to overcome the presence of outliers in the training data. In addition, a novel loss function specially designed to address the seasonality and trend effects of the time series is proposed. Furthermore, a pre-processing technique is used to combat the adverse trend effect which might cause false alarms in outlier detection. The performance of the proposed TCN-AE-based outlier detection algorithm is evaluated using synthetic time series, Yahoo Webscope dataset and real time series of KPIs of mobile cellular networks. The results show that the proposed TCN-AE-based outlier detection algorithm achieves better detection accuracy in terms of F-score than other DML-based algorithms such as long short-term memory (LSTM)-AE-based algorithm and the convolutional neural network (CNN) based algorithm.

Outlier Detection in Time-Series Receive Signal Strength Observation Using Z-Score Method with Sn Scale Estimator for Indoor Localization

Collecting time-series receive signal strength (RSS) observations and averaging them is a common method for dealing with RSS fluctuation. However, outliers in the time-series observations affect the averaging process, making this method less efficient. The Z-score method based on the median absolute deviation (MAD) scale estimator has been used to detect outliers, but it is only efficient with symmetrically distributed observations. Experimental analysis has shown that time-series RSS observations can have a symmetric or asymmetric distribution depending on the nature of the environment in which the measurement was taken. Hence, the use of the Z-score method with the MAD scale estimator will not be efficient. In this paper, the Sn scale estimator is proposed as an alternative to MAD to be used with the Z-score method in detecting outliers in time-series RSS observations. Performance comparison using an online RSS dataset shows that the Z-score with MAD and Sn as scale estimators falsely detected about 50% and 13%, respectively, of the RSS observations as outliers. Furthermore, the average absolute RSS median deviations between raw and outlier-free observations are 3 dB and 0.25 dB, respectively, for the MAD and Sn scale estimators, corresponding to a range error of about 2 m and 0.5 m.

Detection of Outliers in Time Series Power Data Based on Prediction Errors

The primary focus of smart grid power analysis is on power load forecasting and data anomaly detection. Efficient and accurate power load prediction and data anomaly detection enable energy companies to develop reasonable production and scheduling plans and reduce waste. Since traditional anomaly detection algorithms are typically for symmetrically distributed time series data, the distribution of energy consumption data features uncertainty. To this end, a time series outlier detection approach based on prediction errors is proposed in this paper, which starts by using an attention mechanism-based convolutional neural network (CNN)-gated recursive unit (GRU) method to obtain the residual between the measured value and its predicted value, and the residual data generally conform to a symmetric distribution. Subsequently, for these residual data, a random forest classification algorithm based on grid search optimization is used to identify outliers in the power consumption data. The model proposed in this paper is applied to both classical and real energy consumption datasets, and the performance is evaluated using different metrics. As shown in the results, the average accuracy of the model is improved by 25.2% and the average precision is improved by 17.2%, with an average recall improvement of 16.4% and an average F1 score improvement of 26.8% compared to the mainstream algorithms.

Anomaly detection by using a combination of generative adversarial networks and convolutional autoencoders

With the development of full digitalization, the amount of time series data generated by sensors is ever-increasing; thus, time series outlier detection has become crucial. Moreover, in practice, discovering and flagging anomalies is very time-consuming and expensive. To solve this problem, unsupervised anomaly detection methods have often been used in the past, in which the model is trained with normal data to learn its behavioral patterns. Generative adversarial networks (GANs) can simulate complex and high-dimensional distributions of data and can be used to learn the behavioral patterns of normal data for unsupervised anomaly detection. However, because of the problem of convergence, GANs are difficult to train. Thus, USADs (an unsupervised anomaly detection model) utilize an autoencoder (AE) to undertake the task of the generator and discriminator and enhance the stability during adversarial training by using the AE to alleviate the problem of non-convergence encountered in GANs. Therefore, in this study, we used the USAD’s generative adversarial training architecture combined with convolutional AEs to improve the model’s feature extraction capabilities. In addition, to reduce false-positive outcomes caused by the prominent sharp points in the reconstructed data, we used the exponential weighted moving average method to smooth the reconstruction error, thereby improving the anomaly detection accuracy of the model. Finally, we experimented with real-world time-series data (ECG and 2D gesture) and verified that our approach could improve accuracy. Compared to the best in the comparison method, our model improved by 0.028% in AUROC, 0.233% in AUPRC, and 0.187% in F1 on average.

Outliers in financial time series data: Outliers, margin debt, and economic recession

Outliers in financial time series data are different from that in cross-sectional data in terms of the treatment and the detection. First, outliers in time series can be the focus of analysis itself, such as outliers in margin debt to indicate an overheating market. Second, the outlier detection in time series should be accompanied by decomposition to exclude inherent patterns. Unfortunately, there is a lack of consensus on the best decomposition method. Thus, we propose an ensemble model that combines multiple decomposition methods. Using the approach, we found that the outliers in margin debt are strong predictors of a recession.

A framework for adapting online prediction algorithms to outlier detection over time series

This study introduces a novel framework that eases the adoption of any online prediction algorithm for outlier detection over time series data. The proposed framework comprises both streaming data normalization and online anomaly scoring and identification based on prediction errors. To demonstrate the utility of the proposed framework, a novel neural-network-based online time series anomaly detection algorithm called EORELM-AD is developed by implementing the steps of the proposed framework over an ensemble of online recurrent extreme learning machines. Extensive experiments on well-known benchmark datasets for time series outlier detection are presented and discussed, yielding two main conclusions. First, the performance of the proposed EORELM-AD detector is competitive in comparison to several state-of-the-art outlier detection algorithms. Second, the proposed framework is a useful tool for adapting an online time series prediction algorithm to outlier detection.

Global influenza surveillance systems to detect the spread of influenza-negative influenza-like illness during the COVID-19 pandemic: Time series outlier analyses from 2015-2020.

BackgroundSurveillance systems are important in detecting changes in disease patterns and can act as early warning systems for emerging disease outbreaks. We hypothesized that analysis of data from existing global influenza surveillance networks early in the COVID-19 pandemic could identify outliers in influenza-negative influenza-like illness (ILI). We used data-driven methods to detect outliers in ILI that preceded the first reported peaks of COVID-19.Methods and findingsWe used data from the World Health Organization’s Global Influenza Surveillance and Response System to evaluate time series outliers in influenza-negative ILI. Using automated autoregressive integrated moving average (ARIMA) time series outlier detection models and baseline influenza-negative ILI training data from 2015–2019, we analyzed 8,792 country-weeks across 28 countries to identify the first week in 2020 with a positive outlier in influenza-negative ILI. We present the difference in weeks between identified outliers and the first reported COVID-19 peaks in these 28 countries with high levels of data completeness for influenza surveillance data and the highest number of reported COVID-19 cases globally in 2020. To account for missing data, we also performed a sensitivity analysis using linear interpolation for missing observations of influenza-negative ILI. In 16 of the 28 countries (57%) included in this study, we identified positive outliers in cases of influenza-negative ILI that predated the first reported COVID-19 peak in each country; the average lag between the first positive ILI outlier and the reported COVID-19 peak was 13.3 weeks (standard deviation 6.8). In our primary analysis, the earliest outliers occurred during the week of January 13, 2020, in Peru, the Philippines, Poland, and Spain. Using linear interpolation for missing data, the earliest outliers were detected during the weeks beginning December 30, 2019, and January 20, 2020, in Poland and Peru, respectively. This contrasts with the reported COVID-19 peaks, which occurred on April 6 in Poland and June 1 in Peru. In many low- and middle-income countries in particular, the lag between detected outliers and COVID-19 peaks exceeded 12 weeks. These outliers may represent undetected spread of SARS-CoV-2, although a limitation of this study is that we could not evaluate SARS-CoV-2 positivity.ConclusionsUsing an automated system of influenza-negative ILI outlier monitoring may have informed countries of the spread of COVID-19 more than 13 weeks before the first reported COVID-19 peaks. This proof-of-concept paper suggests that a system of influenza-negative ILI outlier monitoring could have informed national and global responses to SARS-CoV-2 during the rapid spread of this novel pathogen in early 2020.

Distance- and Momentum-Based Symbolic Aggregate Approximation for Highly Imbalanced Classification.

Time-series representation is the most important task in time-series analysis. One of the most widely employed time-series representation method is symbolic aggregate approximation (SAX), which converts the results from piecewise aggregate approximation to a symbol sequence. SAX is a simple and effective method; however, it only focuses on the mean value of each segment in the time-series. Here, we propose a novel time-series representation method—distance- and momentum-based symbolic aggregate approximation (DM-SAX)—that can secure time-series distributions by calculating the perpendicular distance from the time-axis to each data point and consider the time-series trend by adding a momentum factor reflecting the direction of previous data points. Experimental results for 29 highly imbalanced classification problems on the UCR datasets revealed that DM-SAX affords the optimal area under the curve (AUC) among competing time-series representation methods (SAX, extreme-SAX, overlap-SAX, and distance-based SAX). We statistically verified that performance improvements resulted in significant differences in the rankings. In addition, DM-SAX yielded the optimal AUC for real-world wire cutting and crimping process dataset. Meaningful data points such as outliers could be identified in a time-series outlier detection framework via the proposed method.

An automated cryptocurrency trading system based on the detection of unusual price movements with a Time-Series Clustering-Based approach

The cryptocurrency market, which has a rapidly growing market size, attracts the increasing attention of individual and institutional investors. While this highly volatile market offers great profit opportunities to investors, it also brings risks due to its sensitivity to speculative news and the unpredictable behaviour of major investors that can cause unsual price movements. In this paper, we argue that rapid and high price fluctuations or unusual patterns that occur in this way may negatively affect the functionality of technical signals that constitute a basis for feature extraction in a machine learning (ML)-based trading system and this may cause the generalization of the model to deteriorate. To address this problem, we propose an end-to-end ML-based trading system including a time series outlier detection module that detects the periods in which unusual price formations are observed. The training of the classification algorithms for the price direction prediction task was performed on the remaining data. We present the results related to the accuracy of the classification models as well as the simulation results obtained using the proposed system for real time trading on the historical data. The findings showed that the outlier detection step significantly increases return on investment for the machine learning-based trading strategies. Besides, the results showed that during the highly volatile periods the trading system becomes more profitable compared to the baseline model and buy&hold strategy.

Performance evaluation of outlier detection techniques in production timeseries: A systematic review and meta-analysis

Time-series data have been extensively collected and analyzed in many disciplines, such as stock market, medical diagnosis, meteorology, and oil and gas industry. Numerous data in these disciplines are sequence of observations measured as functions of time, which can be further used for different applications via analytical or data analytics techniques (e.g., to forecast future price, climate change, etc.). However, presence of outliers can cause significant uncertainties to interpretation results; hence, it is essential to remove the outliers accurately and efficiently before conducting any further analysis. A total of 17 techniques that belong to statistical, regression-based, and machine learning (ML) based categories for outlier detection in timeseries are applied to the oil and gas production data analysis. 15 of these methods are utilized for production data analysis for the first time. Two state-of-the-art and high-performance techniques are then selected for data cleaning which require minimum control and time complexity. Moreover, performances of these techniques are evaluated based on several metrics including the accuracy, precision, recall, F1 score, and Cohen’s Kappa to rank the techniques. Results show that eight unsupervised algorithms outperform the rest of the methods based on the synthetic case study with known outliers. For example, accuracies of the eight shortlisted methods are in the range of 0.83–0.99 with a precision between 0.83 and 0.98, compared to 0.65–0.82 and 0.07–0.77 for the others. In addition, ML-based techniques perform better than statistical techniques. Our experimental results on real field data further indicate that the k-nearest neighbor (KNN) and Fulford-Blasingame methods are superior to other outlier detection frameworks for outlier detection in production data, followed by four others including density-based spatial clustering of applications with noise (DBSCAN), and angle-based outlier detection (ABOD). Even though the techniques are examined with oil and gas production data, but the same data cleaning workflow can be used to detect timeseries’ outliers in other disciplines.

Unsupervised time series outlier detection with diversity-driven convolutional ensembles

With the sweeping digitalization of societal, medical, industrial, and scientific processes, sensing technologies are being deployed that produce increasing volumes of time series data, thus fueling a plethora of new or improved applications. In this setting, outlier detection is frequently important, and while solutions based on neural networks exist, they leave room for improvement in terms of both accuracy and efficiency. With the objective of achieving such improvements, we propose a diversity-driven, convolutional ensemble. To improve accuracy, the ensemble employs multiple basic outlier detection models built on convolutional sequence-to-sequence autoencoders that can capture temporal dependencies in time series. Further, a novel diversity-driven training method maintains diversity among the basic models, with the aim of improving the ensemble's accuracy. To improve efficiency, the approach enables a high degree of parallelism during training. In addition, it is able to transfer some model parameters from one basic model to another, which reduces training time. We report on extensive experiments using real-world multivariate time series that offer insight into the design choices underlying the new approach and offer evidence that it is capable of improved accuracy and efficiency.

TODS: An Automated Time Series Outlier Detection System

We present TODS, an automated Time Series Outlier Detection System for research and industrial applications. TODS is a highly modular system that supports easy pipeline construction. The basic building block of TODS is primitive, which is an implementation of a function with hyperparameters. TODS currently supports 70 primitives, including data processing, time series processing, feature analysis, detection algorithms, and a reinforcement module. Users can freely construct a pipeline using these primitives and perform end- to-end outlier detection with the constructed pipeline. TODS provides a Graphical User Interface (GUI), where users can flexibly design a pipeline with drag-and-drop. Moreover, a data-driven searcher is provided to automatically discover the most suitable pipelines given a dataset. TODS is released under Apache 2.0 license at https://github.com/datamllab/tods. A video is available on YouTube (https://youtu.be/JOtYxTclZgQ)

Outlier Detection in Time Series via Mixed-Integer Conic Quadratic Optimization

We consider the problem of estimating the true values of a Wiener process given noisy observations corrupted by outliers. In this paper we show how to improve existing mixed-integer quadratic optim...

Demonetization and digital payments in India: perception and reality

ABSTRACT In the context of the demonetization experiment of November 2016 in the Indian economy, this paper aims at looking into its impact on digital payments. Using the major digital payment modes and following the methodology of time-series outlier detection proposed by Chen and Liu (1993), the impacts are found to be mostly transitory in nature.

The effects of the 2013 floods on Germany’s freight traffic

This article analyzes the spatio-temporal effect of the 2013 floods on freight traffic in Germany by using automatic traffic counter data. The methodology uses a proven time-series outlier detection and identification technique to endogenously determine if a counter was affected during the flood and estimate the magnitude and duration of the change in the number of vehicles passing through it. This is the first paper able to quantify climate-related variations in traffic across all the counters of a national network. Results show variations on 10% of all counters and 23% of all main roads. Results allow us to trace the configuration of disrupted and detour routes, recovery times, and the total effect on the network. Our findings serve as an input to other studies on the impact of exogenous events on the transport system and contribute towards the formulation of public policies to improve road resilience.

Characterizing Land Surface Phenology and Exotic Annual Grasses in Dryland Ecosystems Using Landsat and Sentinel-2 Data in Harmony

Invasive annual grasses, such as cheatgrass (Bromus tectorum L.), have proliferated in dryland ecosystems of the western United States, promoting increased fire activity and reduced biodiversity that can be detrimental to socio-environmental systems. Monitoring exotic annual grass cover and dynamics over large areas requires the use of remote sensing that can support early detection and rapid response initiatives. However, few studies have leveraged remote sensing technologies and computing frameworks capable of providing rangeland managers with maps of exotic annual grass cover at relatively high spatiotemporal resolutions and near real-time latencies. Here, we developed a system for automated mapping of invasive annual grass (%) cover using in situ observations, harmonized Landsat and Sentinel-2 (HLS) data, maps of biophysical variables, and machine learning techniques. A robust and automated cloud, cloud shadow, water, and snow/ice masking procedure (mean overall accuracy >81%) was implemented using time-series outlier detection and data mining techniques prior to spatiotemporal interpolation of HLS data via regression tree models (r = 0.94; mean absolute error (MAE) = 0.02). Weekly, cloud-free normalized difference vegetation index (NDVI) image composites (2016–2018) were used to construct a suite of spectral and phenological metrics (e.g., start and end of season dates), consistent with information derived from Moderate Resolution Image Spectroradiometer (MODIS) data. These metrics were incorporated into a data mining framework that accurately (r = 0.83; MAE = 11) modeled and mapped exotic annual grass (%) cover throughout dryland ecosystems in the western United States at a native, 30-m spatial resolution. Our results show that inclusion of weekly HLS time-series data and derived indicators improves our ability to map exotic annual grass cover, as compared to distribution models that use MODIS products or monthly, seasonal, or annual HLS composites as primary inputs. This research fills a critical gap in our ability to effectively assess, manage, and monitor drylands by providing a framework that allows for an accurate and timely depiction of land surface phenology and exotic annual grass cover at spatial and temporal resolutions that are meaningful to local resource managers.

A Time Convolutional Network Based Outlier Detection for Multidimensional Time Series in Cyber-Physical-Social Systems

With the development of the Cyber-Physical-Social Systems(CPSS), a large number of multidimensional time series have been generated in today's world, such as: sensor data for industrial equipment operation, vehicle driving data, and cloud server operation and maintenance data and so on. The key task of Cloud-Fog-Edge Computing in managing these systems is how to detect anomalous data in a specific time series to facilitate operator action to solve potential system problems. So multidimensional time series outlier detection become an important direction of CPSS data mining and Cloud-Fog-Edge Computing research, and it has a wide range of applications in industry, finance, medicine and other fields. This paper proposes a framework called Multidimensional time series Outlier detection based on a Time Convolutional Network AutoEncoder (MOTCN-AE), which can detect outliers in time series data, such as identifying equipment failures, dangerous driving behaviors of cars, etc. Specifically, this paper first uses a feature extraction method to transform the original time series into a feature-rich time series. Second, the proposed TCN-AE is used to reconstruct the feature-rich time series data, and the reconstruction error is used to calculate outlier scores. Finally, the MOTCN-AE framework is validated by multiple time series datasets to demonstrate its effectiveness in detecting time series outliers.

시계열 이상치 탐지 기법을 활용한 경부선 주요도시 철도 승객수의 이상치 탐색 연구

2004년 4월 1일, 국내 최초의 고속철도(HSR)인 KTX (Korea Train eXpress)가 경부선에 도 입 되었다. KTX의 등장은 경부선을 이용하는 철도 승객들의 운송수단 선택 및 도시구간별 이 용객 수 변화를 가져왔다. KTX의 등장과 같은 개입사건(Intervention events)의 영향은 개입사 건 전후 변화를 단순 통계량으로 분석하거나 개입 ARIMA 모델을 통해 분석 되었다. 개입 ARIMA 모델은 개입사건의 발생 시점(t)과 개입사건의 영향 형태(type) 등의 가정이 필요하다는 한계가 있었으며, 본 연구에서는 기존 연구에서의 한계점을 보완할 수 있는 시계열 이상치 탐지(time series outlier detection)를 활용하였다. 일반적으로 개입사건의 발생시기는 잘 알려져 있지 않으므로 시계열 이상치 탐지를 통해 개입사건에의 발생 시기를 추정할 수 있다. 시계열 이상치 탐지기법을 활용하여 개입의 시점과 영향 형태에 관한 가 정 없이 개입사건에 대한 영향을 분석할 수 있으며, 발생된 이상치의 시점을 개입사건의 시점, 이상치의 영향을 개입 사건의 영향으로 가정하였다. 데이터는 KTDB (Korea Transport Database)로 부터 KTX가 도입되기 이전인 2003 년부터 2014년까지 12년 동안의 경부선(4개의 주요 도시구간 합산)을 포함한 주요 도시구간 4개의 월별데이터를 수집하여 활용하였다. 경부선 도시 구간별 이상치를 탐지 하고 그 영향을 분석한 결과, 동일한 개입사건 임에도 그 영향의 형태의 정도가 도시구간마다 다르게 나타나거나 영향이 나타나지 않았으며, 기존 연구에서 분석되지 않은 개입사건을 찾을 수 있었다.

Semiparametric outlier detection in nonstationary times series: Case study for atmospheric pollution in Brno, Czech Republic

Large environmental datasets usually include outliers which can have significant effects on further analysis and modelling. There exist various outlier detection methods that depend on the distribution of the analysed variable. However quite often the distribution of environmental variables can not be estimated. This paper presents an approach for identification of outliers in environmental time series which does not impose restrictions on the distribution of observed variables. The suggested algorithm combines kernel smoothing and extreme value estimation techniques for stochastic processes within considerations of nonstationary expected value of the process. The nonstationarity in variance is evaded by change point analysis which precedes the proposed algorithm. Possible outliers are identified as observations with rare occurrence and, in correspondence to extreme value methodology, the confidence limits for high values of observed variables are constructed. The proposed methodology can be especially convenient for cases where validation of the data has to be carried out manually, since it significantly reduces the number of implausible observations. For a case study, the technique is applied for outlier detection in time series of hourly PM10 concentrations in Brno, Czech Republic. The methodology is derived on solid theoretical results and seems to perform well for the series of PM10. However its flexibility makes it generally applicable not only to series of atmospheric pollutants. On the other hand, the choice of return level turns out to be crucial in sensitivity to the outliers. This issue should be left to the practitioners to decide with respect to specific application conditions.