Discord Detection Research Articles

A generalizable unsupervised graph‐based approach for anomalous event detection in people living with dementia

AbstractBackgroundSensor‐based remote health monitoring of persons living with dementia (PLwD) can be used to monitor the progression of their condition with minimal intrusion. This helps minimize preventable hospital admissions, while allowing researchers to improve their understanding of dementia. Existing approaches for detecting anomalies in activity and behavior in PLwD are challenged by data noise, drift, label reliance, and poor explainability and generalizability due to wide inter‐individual variability.MethodWe propose and evaluate a solution that combines graph learning with the Contextual Matrix Profile (CMP), an ultra‐fast distance‐based discord detection technique. Daily household movement data collected via passive infrared sensors are used to generate multivariate CMP. This is encoded as a set of time‐evolving graphs. A self‐supervised graph model is trained to separate graphs by distance, so discordant time blocks show up as anomalous graphs. Differences between successive graph embeddings are subject to moving average thresholding to detect anomaly. We compare our method to four state‐of‐the‐art outlier detection algorithms, and demonstrate the generalizability of a distance‐based graph model trained on one cohort, to anomaly detection in unseen individuals.ResultGraph‐enhanced CMP models outperform state‐of‐the‐art algorithms, yielding 74% recall with a 5.7% alert rate and 88% household‐level validity when evaluated on 13,426 days’ data from 42 households with 145 agitation events, collected by the UK Dementia Research Institute between August 2019 and April 2022. Similar results are obtained for a 23‐participant, 5284‐day Falls cohort: 72% recall, 5.85% alert rate and 83% household‐level validity. Pretrained models yield similarly high performance for unseen households in both cohorts. Experiments on the publicly available PAMAP dataset reveal superior top‐k alert performance. Explainability is achieved via the CMP that is the source for graphs.ConclusionWe address the need for generalizable anomaly detection models for multivariate time series sensor data in remote health monitoring. While the CMP allows the ability to denoise and detect patterns at a household level, a graph model constructed in the distance space is generalizable across households. With higher sensitivity, fewer alerts than state‐of‐the‐art methods and inherent explainability, graph‐CMP models offer a clinically meaningful unsupervised anomaly detection technique for dementia and beyond.

Model-free detection of unique events in time series

Recognition of anomalous events is a challenging but critical task in many scientific and industrial fields, especially when the properties of anomalies are unknown. In this paper, we introduce a new anomaly concept called “unicorn” or unique event and present a new, model-free, unsupervised detection algorithm to detect unicorns. The key component of the new algorithm is the Temporal Outlier Factor (TOF) to measure the uniqueness of events in continuous data sets from dynamic systems. The concept of unique events differs significantly from traditional outliers in many aspects: while repetitive outliers are no longer unique events, a unique event is not necessarily an outlier; it does not necessarily fall out from the distribution of normal activity. The performance of our algorithm was examined in recognizing unique events on different types of simulated data sets with anomalies and it was compared with the Local Outlier Factor (LOF) and discord discovery algorithms. TOF had superior performance compared to LOF and discord detection algorithms even in recognizing traditional outliers and it also detected unique events that those did not. The benefits of the unicorn concept and the new detection method were illustrated by example data sets from very different scientific fields. Our algorithm successfully retrieved unique events in those cases where they were already known such as the gravitational waves of a binary black hole merger on LIGO detector data and the signs of respiratory failure on ECG data series. Furthermore, unique events were found on the LIBOR data set of the last 30 years.

The good, the bad, and the ugly: Data-driven load profile discord identification in a large building portfolio

Reducing the overall energy consumption and associated greenhouse gas emissions in the building sector is essential for meeting our future sustainability goals. Recently, smart energy metering facilities have been deployed to enable monitoring of energy consumption data with hourly or subhourly temporal resolution. This unprecedented data collection has created various opportunities for advanced data analytics involving load profiles (e.g., building energy benchmarking programs, building-to-grid integration, and calibration of urban-scale energy models). These applications often need preprocessing steps to detect daily load profile discords, such as: 1) outliers due to system malfunctions (the bad) and 2) irregular energy consumption patterns, such as those resulting from holidays (the ugly) compared to normal consumption patterns (the good). However, current preprocessing methods predominantly focus on filtering using statistical threshold values, which fail to capture the contextual discords of daily profiles. In addition, discord detection algorithms in building research are often aimed at finding individual building-level discords, which are not suitable at a large scale. Thus, in this paper, we develop a method for automated load profile discord identification (ALDI) in a large portfolio of buildings (more than 100 buildings). Specifically, ALDI 1) uses the matrix profile (MP) method to quantify the similarities of daily subsequences in time series meter data, 2) compares daily MP values with typical-day MP distributions using the Kolmogorov-Smirnov test, and 3) identifies daily load profile discords in a large building portfolio. We evaluate ALDI using the metering data of both an academic campus and a residential neighborhood. Our results demonstrate that ALDI efficiently discovers measurement errors by system malfunctions and low energy consumption days in the academic campus portfolio, and it detects unique load shape patterns likely driven by occupant behavior and extreme weather conditions in the residential neighborhood.

A parallel algorithm for robust fault detection in semiconductor manufacturing processes

The semiconductor manufacturing consists of a number of processes, and even a small fault occurring at any point can damage the product quality. The fast and accurate detection of such faults is essential to maintain high manufacturing yields. In this paper, we propose a parallel algorithm for fault detection in semiconductor manufacturing processes. The algorithm is a modification of the discord detection algorithm called HOT SAX, which adopted the SAX representation of time-series for efficient storage and computation. We first propose a sequential algorithm and then extend it to a parallel version. We evaluate our algorithm through experiments using the data obtained from a real-world semiconductor plasma etching process. As a result, our fault detection algorithm achieved 100 % accuracy without any false positive or false negative.

A Novel Never-ending Uncertain Top-k Discord Detection Method

A Novel Never-ending Uncertain Top-k Discord Detection Method

Weighted spherical 1-mean with phase shift and its application in electrocardiogram discord detection

Detecting discords in time series is a special novelty detection task that has found many interesting applications. Unlike the traditional novelty detection methods which can make use of a separate set of normal samples to build up the model, discord detection is often provided with mixed data containing both normal and abnormal data. The objective of this work is to present an effective method to detect discords in unsynchronized periodic time series data. The task of discord detection is considered as a problem of unsupervised learning with noise data. A new clustering algorithm named weighted spherical 1-mean with phase shift (PS-WS1M) is proposed in this work. It introduces a phase adjustment procedure into the iterative clustering process and produces a set of anomaly scores based upon which an unsupervised approach is employed to locate the discords automatically. A theoretical analysis on the robustness and convergence of PS-WS1M is also given. The proposed algorithm is evaluated via real-world electrocardiograms datasets extracted from the MIT-BIH database. The experimental results show that the proposed algorithm is effective and competitive for the problem of discord detection in periodic time series. Meanwhile, the robustness of PS-WS1M is also experimentally verified. As compared to some of the other discord detection methods, the proposed algorithm can always achieve ideal FScore values with most of which exceeding 0.98. The proposed PS-WS1M algorithm allows the integration of a phase adjustment procedure into the iterative clustering process and it can be successfully applied to detect discords in time series.

A novel discords detector for periodic time series based on weighted spherical single means with phase shift

The traditional one-class classifiers are not suitable for detecting discords in periodic time series. A novel one-class classifier PS-WS1M-OCC is proposed in this paper. In our method, the phase problem in time series is solved by introducing phase shift into the clustering procedure. Meanwhile, a novel criterion for adaptively choosing threshold is proposed. In this way, the proposed classifier is insensitive to noise in the training set. Experimental results show that our PS-WSKM-OCC is more robust than the existing one-class classifiers when it is applied to the problem of discord detection in the periodic time series.