Multiple Change-point Detection Research Articles

Some clustering-based change-point detection methods applicable to high dimension, low sample size data

Detection of change-points in a sequence of high dimensional observations is a challenging problem, and this becomes even more challenging when the sample size (i.e., the sequence length) is small. In this article, we propose some change-point detection methods based on clustering, which can be conveniently used in such high dimension, low sample size situations. First, we consider the single change-point problem. Using k-means clustering based on a suitable dissimilarity measures, we propose some methods for testing the existence of a change-point and estimating its location. High dimensional behavior of these proposed methods are investigated under appropriate regularity conditions. Next, we extend our methods for detection of multiple change-points. We carry out extensive numerical studies and analyze a real data set to compare the performance of our proposed methods with some state-of-the-art methods.

Multiple change point detection for high-dimensional data

Multiple change point detection for high-dimensional data

A Selective Review on Information Criteria in Multiple Change Point Detection.

Change points indicate significant shifts in the statistical properties in data streams at some time points. Detecting change points efficiently and effectively are essential for us to understand the underlying data-generating mechanism in modern data streams with versatile parameter-varying patterns. However, it becomes a highly challenging problem to locate multiple change points in the noisy data. Although the Bayesian information criterion has been proven to be an effective way of selecting multiple change points in an asymptotical sense, its finite sample performance could be deficient. In this article, we have reviewed a list of information criterion-based methods for multiple change point detection, including Akaike information criterion, Bayesian information criterion, minimum description length, and their variants, with the emphasis on their practical applications. Simulation studies are conducted to investigate the actual performance of different information criteria in detecting multiple change points with possible model mis-specification for the practitioners. A case study on the SCADA signals of wind turbines is conducted to demonstrate the actual change point detection power of different information criteria. Finally, some key challenges in the development and application of multiple change point detection are presented for future research work.

Shifts in wetness pattern and periodicity across Tripura state in north east India

Region wetness variability was assessed across the Tripura state of North east India (1971 to 2016). Multiple Change point detection tests confirmed the high degree of spatiotemporal variability for the identified shifts in wetness pattern over study period. The periodicity of different wetness time-series varied between 2-128 months for the calculated SPI time scales over variable time series for the selected rain gauge stations. The periodicity pattern became more prominent with an increasing temporal domain of calculated SPI time series. Hierarchical clustering and Principle component analysis (PCA) accounted for the variability in randomness, trend and periodicity of all the SPI time series. Our present study identified the homogeneous clusters of raingauge stations suitable for real-time drought monitoring and reversible use of missing dataset on rainfall in near future across the Tripura state.

DDoS attack forecasting based on online multiple change points detection and time series analysis

DDoS attack forecasting based on online multiple change points detection and time series analysis

Arctic marginal ice zone interannual variability and change point detection using two definitions (1983–2022)

The ongoing decline in Arctic sea ice extent and thickness underscores the scientific significance of monitoring the marginal ice zone (MIZ), a transitional region between the open ocean and pack ice. In this study, we used Bootstrap sea ice concentration (SIC) to detect the trend and change point of the Arctic MIZ over 40 years (1983–2022) using two different MIZ definitions: SIC threshold-based (MIZ t ) and SIC anomaly-based (MIZ). This study marks the exploration of a SIC anomaly-based definition of the MIZ over the Arctic. While the two MIZ definitions yield comparable seasonal trends in marginal ice zone fraction (MIZF), the MIZ fraction values peak during the transition periods (e.g. freeze-up and break-up), while the MIZ t fraction values peak in August. The analysis also uncovers consistently higher MIZF values for the MIZ than for MIZ t across all seasons. Moreover, October and August show the fastest rate of increase in MIZ t fraction and MIZ fraction, reflecting the coinciding rapid decrease in sea ice extent during those particular months. Employing the pruned exact linear time, a multiple change point detection method, highlights a significant increase in the MIZ fraction in October (after 2005) and MIZ fraction in August (after 2007). This can be indicative of the recent climate change impacts in the Arctic region that may be linked with shifts in SIC and sea ice mobility for MIZ t and MIZ, respectively.

DiffSegR: an RNA-seq data driven method for differential expression analysis using changepoint detection

Abstract To fully understand gene regulation, it is necessary to have a thorough understanding of both the transcriptome and the enzymatic and RNA-binding activities that shape it. While many RNA-Seq-based tools have been developed to analyze the transcriptome, most only consider the abundance of sequencing reads along annotated patterns (such as genes). These annotations are typically incomplete, leading to errors in the differential expression analysis. To address this issue, we present DiffSegR - an R package that enables the discovery of transcriptome-wide expression differences between two biological conditions using RNA-Seq data. DiffSegR does not require prior annotation and uses a multiple changepoints detection algorithm to identify the boundaries of differentially expressed regions in the per-base log2 fold change. In a few minutes of computation, DiffSegR could rightfully predict the role of chloroplast ribonuclease Mini-III in rRNA maturation and chloroplast ribonuclease PNPase in (3′/5′)-degradation of rRNA, mRNA and tRNA precursors as well as intron accumulation. We believe DiffSegR will benefit biologists working on transcriptomics as it allows access to information from a layer of the transcriptome overlooked by the classical differential expression analysis pipelines widely used today. DiffSegR is available at https://aliehrmann.github.io/DiffSegR/index.html.

Genetic algorithm with a Bayesian approach for multiple change-point detection in time series of counting exceedances for specific thresholds

Although the applications of Non-Homogeneous Poisson Processes (NHPP) to model and study the threshold overshoots of interest in different time series of measurements have proven to provide good results, they needed to be complemented with an efficient and automatic diagnostic technique to establish the location of the change-points, which, when taken into account, make the estimated model fit poorly in regards of the information contained in the real one. Because of this, a new method is proposed to solve the segmentation uncertainty of the time series of measurements, where the generating distribution of exceedances of a specific threshold is the focus of investigation. One of the great contributions of the present algorithm is that all the days that trespassed are candidates to be a change-point, so all the possible configurations of overflow days under the heuristics of a genetic algorithm are the possible chromosomes, which will unite to produce new solutions. Also, such methods will be guarantee to non-local and the best possible one solution, reducing wasted machine time evaluating the least likely chromosomes to be a feasible solution. The analytical evaluation technique will be by means of the Minimum Description Length (MDL) as the objective function, which is the joint posterior distribution function of the parameters of the NHPP of each regime and the change-points that determines them and which account as well for the influence of the presence of said times. Thus, one of the practical implications of the present work comes in terms of overcoming the need of modeling the time series of measurements, where the distributions of exceedances of certain thresholds, or where the counting of certain events involving abrupt changes, is the main focus with applications in phenomena such as climate change, information security and epidemiology, to name a few.

Robust nonparametric multiple changepoint detection for multivariate variability

Two robust, nonparametric multiple changepoint detection algorithms are introduced: DWBS and MKWP. These algorithms can detect multiple changes in the variability of a sequence of independent multivariate observations, even when the number of changepoints is unknown. The algorithms DWBS and MKWP require minimal distributional assumptions and are robust to outlying observations and heavy tails. The DWBS algorithm uses a local search method based on depth-based ranks and wild binary segmentation. The MKWP algorithm estimates changepoints globally via maximizing a penalized version of the classical Kruskal–Wallis ANOVA test statistic. It is demonstrated that this objective function can be maximized via the well-known PELT algorithm. Under mild, nonparametric assumptions, both of these algorithms are shown to be consistent for the correct number of changepoints and the correct location(s) of the changepoint(s). A data driven thresholding method for multivariate data is introduced, based on the Schwartz information criteria. The robustness and accuracy of the new methods is demonstrated with a simulation study, where the algorithms are compared to several existing algorithms. These new methods can estimate the number of changepoints and their locations accurately when the data are heavy tailed or skewed and the sample size is large. Lastly, the proposed algorithms are applied to a four-dimensional sequence of European daily stock returns.

Detection of multiple change-points in high-dimensional panel data with cross-sectional and temporal dependence

Detection of multiple change-points in high-dimensional panel data with cross-sectional and temporal dependence

Multiple change point detection under serial dependence: Wild contrast maximisation and gappy Schwarz algorithm

We propose a methodology for detecting multiple change points in the mean of an otherwise stationary, autocorrelated, linear time series. It combines solution path generation based on the wild contrast maximisation principle, and an information criterion‐based model selection strategy termed gappy Schwarz algorithm. The former is well‐suited to separating shifts in the mean from fluctuations due to serial correlations, while the latter simultaneously estimates the dependence structure and the number of change points without performing the difficult task of estimating the level of the noise as quantified e.g. by the long‐run variance. We provide modular investigation into their theoretical properties and show that the combined methodology, named WCM.gSa, achieves consistency in estimating both the total number and the locations of the change points. The good performance of WCM.gSa is demonstrated via extensive simulation studies, and we further illustrate its usefulness by applying the methodology to London air quality data.

Generalized multiple change-point detection in the structure of multivariate, possibly high-dimensional, data sequences

The extensive emergence of big data techniques has led to an increasing interest in the development of change-point detection algorithms that can perform well in a multivariate, possibly high-dimensional setting. In the current paper, we propose a new method for the consistent estimation of the number and location of multiple generalized change-points in multivariate, possibly high-dimensional, noisy data sequences. The number of change-points is allowed to increase with the sample size and the dimensionality of the given data sequence. Having a number of univariate signals, which constitute the unknown multivariate signal, our algorithm can deal with general structural changes; we focus on changes in the mean vector of a multivariate piecewise-constant signal, as well as changes in the linear trend of any of the univariate component signals. Our proposed algorithm, labeled Multivariate Isolate–Detect (MID) allows for consistent change-point detection in the presence of frequent changes of possibly small magnitudes in a computationally fast way.

Vegetation photosynthetic phenology dataset in northern terrestrial ecosystems

Vegetation phenology can profoundly modulate the climate-biosphere interactions and thus plays a crucial role in regulating the terrestrial carbon cycle and the climate. However, most previous phenology studies rely on traditional vegetation indices, which are inadequate to characterize the seasonal activity of photosynthesis. Here, we generated an annual vegetation photosynthetic phenology dataset with a spatial resolution of 0.05 degrees from 2001 to 2020, using the latest gross primary productivity product based on solar-induced chlorophyll fluorescence (GOSIF-GPP). We combined smoothing splines with multiple change-point detection to retrieve the phenology metrics: start of the growing season (SOS), end of the growing season (EOS), and length of growing season (LOS) for terrestrial ecosystems above 30° N latitude (Northern Biomes). Our phenology product can be used to validate and develop phenology or carbon cycle models and monitor the climate change impacts on terrestrial ecosystems.

A Multiple change-point detection framework on linguistic characteristics of real versus fake news articles

Extracting information from textual data of news articles has been proven to be significant in developing efficient fake news detection systems. Pointedly, to fight disinformation, researchers concentrated on extracting information which focuses on exploiting linguistic characteristics that are common in fake news and can aid in detecting false content automatically. Even though these approaches were proven to have high performance, the research community proved that both the language as well as the word use in literature are evolving. Therefore, the objective of this paper is to explore the linguistic characteristics of fake news and real ones over time. To achieve this, we establish a large dataset containing linguistic characteristics of various articles over the years. In addition, we introduce a novel framework where the articles are classified in specified topics based on their content and the most informative linguistic features are extracted using dimensionality reduction methods. Eventually, the framework detects the changes of the extracted linguistic features on real and fake news articles over the time incorporating a novel change-point detection method. By employing our framework for the established dataset, we noticed that the linguistic characteristics which concern the article’s title seem to be significantly important in capturing important movements in the similarity level of “Fake” and “Real” articles.

Detecting and testing multiple change points in distributions of damage-sensitive feature data for data-driven structural condition assessment: A distributional time series change-point analytic approach

Structural change detection is a core component of structural health monitoring (SHM). One appealing aspect of SHM is that structural changes can be detected by detecting the changes in extracted damage-sensitive features (DSFs). The feature changes of the data are usually attributed to the changes of the underlying distributions; thus, developing reliable change-point detectors for automatically detecting and testing various potential changes in the distributions of DSF data are particularly beneficial for automatic structural health diagnosis. However, methodological researches devoted to detecting change points for probability distributions are rare either in statistics or engineering literature, and state-of-the-art approaches mainly focus on single change-point detection and have restrictive assumptions on the distributions or the types of change points. This motivates us to develop a more flexible multiple change-point detection method for the distributions of DSF data employed for data-driven structural condition assessment. To accommodate a large dataset, the DSF data are summarized by a distributional time series that consists of probability density functions (PDFs) estimated from blocks of DSF data. Then, detecting the distributional changes for the massive DSF data can be phrased as detecting the changes in the resulting distributional time series with much less data objects. To overcome the obstacle that the PDF-valued data are special functional data residing in nonlinear abstract spaces, this study employs two transformations to convert the PDFs into more tractable vector spaces. Then, by using the linear structures possessed by the vector spaces, a nonparametric multiple change-point detection method is presented for the distributional time series based on the techniques of functional principal component analysis and E-Divisive data segmentation. The proposed method possesses various nice features, such as being capable for multiple change-point detection, having fewer restrictive assumptions and scalability to large datasets. Moreover, it also shows superiority in revealing the modes of distributional variations. An application to distributional change detection involved in cable condition assessment for a long-span cable-stayed bridge demonstrates the applicability and versatility of our proposal, and some interesting results are also discovered.

Pattern Characterization of Meteorological Drought Using Multivariate Drought Index Over Mirzapur in Middle Gangetic Plains of India

Droughts and floods have been occurring at a higher frequency in recent decades. The rapid transition between them magnifies the socio-economic consequences of these catastrophes relative to the effects of the individual occurrences of the extreme event. This study examines the temporal variability of meteorological drought and wet event characteristics occurring over Mirzapur (Uttar Pradesh), India. The Standardized Precipitation Evapotranspiration Index (SPEI) is applied to monthly water balance at scales 3, 6, 9, and 12 months to estimate the meteorological drought and wet events from 1971 to 2018. Drought and wet event characteristics such as the number of drought/wet events, severity, duration, and intensity are estimated using run theory over SPEI output. While characterizing the pattern of trends over the historical time period, variable-sized cluster analysis (VSCA) allows the detection of multiple change points as opposed to the Mann-Kendall (MK) test, which produces a monotonic trend for the entire time period. The VSCA technique accounts for drought variability and depicts the pattern’s evolution across the period under consideration. Station-scale drought data from Mirzapur, Uttar Pradesh (UP), India, were used in the procedure. VSCA allows for the detection of many change points while describing the pattern of drought trend throughout a historical period, as opposed to the usual Mann-Kendall (MK) test, which provides a monotonic trend for the whole time. As a result, VSCA demonstrated the MK test compatibility.

Optimal multiple change-point detection for high-dimensional data

This manuscript makes two contributions to the field of change-point detection. In a general change-point setting, we provide a generic algorithm for aggregating local homogeneity tests into an estimator of change-points in a time series. Interestingly, we establish that the error rates of the collection of tests directly translate into detection properties of the change-point estimator. This generic scheme is then applied to various problems including covariance change-point detection, nonparametric change-point detection and sparse multivariate mean change-point detection. For the latter, we derive minimax optimal rates that are adaptive to the unknown sparsity and to the distance between change-points when the noise is Gaussian. For sub-Gaussian noise, we introduce a variant that is optimal in almost all sparsity regimes.

Change point detection for the intraday volatility using functional ARCH and conditional Copula

In this research, we are concerned with intraday volatilities computed by functional ARCH(1) (fARCH(1), for short) model for high-frequency financial time series. A conditional-Copula multiple change point detection (CPD) for intraday volatilities is proposed using fARCH(1), bivariate Gaussian Copula and t-Copula conditional distributions. We employ current available multivariate CPD models which include energy test based control chart (ETCC) and nonparametric multivariate change point model (NPMVCP) to implement the proposed CPD method for the intraday volatilities. A simulation study is conducted to demonstrate that the functional ARCH based conditional-Copula CPD for the intraday volatilities can be a useful econometrics method to detect abnormal intraday volatilities in the financial market. We analyze intraday volatilities of the Korea composite stock price index (KOSPI) and the Hyundai-Motor (HDM) company stock data with one minute high-frequency to illustrate our proposed CPD method.

Multiple Change Points Detection in High-Dimensional Multivariate Regression

Multiple Change Points Detection in High-Dimensional Multivariate Regression

Rough-Fuzzy CPD: a gradual change point detection algorithm

Changepoint detection is the problem of finding abrupt or gradual changes in time series data when the distribution of the time series changes significantly. There are many sophisticated statistical algorithms for solving changepoint detection problem, although there is not much work devoted towards gradual changepoints as compared to abrupt ones. Here we present a new approach to solve the changepoint detection problem using the fuzzy rough set theory which is able to detect such gradual changepoints. An expression for the rough-fuzzy estimate of changepoints is derived along with its mathematical properties concerning fast computation. In a statistical hypothesis testing framework, the asymptotic distribution of the proposed statistic on both single and multiple changepoints is derived under the null hypothesis enabling multiple changepoint detection. Extensive simulation studies have been performed to investigate how simple crude statistical measures of disparity can be subjected to improve their efficiency in the estimation of gradual changepoints. Also, the said rough-fuzzy estimate is robust to signal-to-noise ratio, a high degree of fuzziness in true changepoints, and also to hyperparameter values. Simulation studies reveal that the proposed method beats other methods of gradual changepoint detection (including MJPD, HSMUCE, fuzzy methods like FCP, FCMLCP etc) and also popular crisp methods like Binary Segmentation, PELT, and BOCD in detecting gradual changepoints. The applicability of the estimate is demonstrated using multiple real-life datasets including Covid-19. We have developed the python package roufcp for broader dissemination of the methods.