Unsupervised Anomaly Detection Research Articles

Towards a unified approach for unsupervised brain MRI Motion Artefact Detection with few shot Anomaly Detection

Automated Motion Artefact Detection (MAD) in Magnetic Resonance Imaging (MRI) is a field of study that aims to automatically flag motion artefacts in order to prevent the requirement for a repeat scan. In this paper, we identify and tackle the three current challenges in the field of automated MAD; (1) reliance on fully-supervised training, meaning they require specific examples of Motion Artefacts (MA), (2) inconsistent use of benchmark datasets across different works and use of private datasets for testing and training of newly proposed MAD techniques and (3) a lack of sufficiently large datasets for MRI MAD. To address these challenges, we demonstrate how MAs can be identified by formulating the problem as an unsupervised Anomaly Detection (AD) task. We compare the performance of three State-of-the-Art AD algorithms DeepSVDD, Interpolated Gaussian Descriptor and FewSOME on two open-source Brain MRI datasets on the task of MAD and MA severity classification, with FewSOME achieving a MAD AUC >90% on both datasets and a Spearman Rank Correlation Coefficient of 0.8 on the task of MA severity classification. These models are trained in the few shot setting, meaning large Brain MRI datasets are not required to build robust MAD algorithms. This work also sets a standard protocol for testing MAD algorithms on open-source benchmark datasets. In addition to addressing these challenges, we demonstrate how our proposed ‘anomaly-aware’ scoring function improves FewSOME’s MAD performance in the setting where one and two shots of the anomalous class are available for training. Code available at https://github.com/niamhbelton/Unsupervised-Brain-MRI-Motion-Artefact-Detection/.

DWSP-MT: Discovery of workers sensing preferences to match tasks for improving data collection quality in MCS

Mobile CrowdSensing (MCS) is a new sensing paradigm that leverages the widespread use of mobile devices to collect data quickly and enable various key applications in the Internet of Things (IoT). Collecting high-quality data is essential for MCS to design and provide a high-quality service. However, it is difficult for the platform to detect dishonest workers who submit fake data. Therefore, MCS applications face a significant challenge. Although there are a number of studies on truth discovery, these studies mainly failed to consider the sensing preferences of workers, resulting in poor data quality enhancement. To overcome these issues, a Discovery of Workers Sensing Preferences to Match Tasks (DWSP-MT) scheme is proposed to improve the quality of MCS data collection. The DWSP-MT scheme primarily includes three components: 1) Unmanned Aerial Vehicles (UAVs) are employed to sense data as ground truth data and an unsupervised anomaly detection algorithm is adopted to identify the data truth of tasks submitted by unidentified workers. 2) The data truth of unidentified workers in historical tasks is evaluated to calculate their sensing preferences, which are used to identify trusted workers. 3) To improve the quality of data collection, a task assignment model based on the sensing preferences of trusted workers is proposed for the first time, which can ensure that tasks are assigned to their areas of most excellent expertise. At last, two open-access datasets were used to conduct experiments comparing the DWSP-MT scheme with existing truth discovery methods. The outcomes of these experiments indicate that our approach outperforms the state-of-the-art research and improves F1-score by 65.74 %, improves accuracy by 36.62 %, and reduces data bias by 60.61 %

An Unsupervised Method for Industrial Image Anomaly Detection with Vision Transformer-Based Autoencoder.

Existing industrial image anomaly detection techniques predominantly utilize codecs based on convolutional neural networks (CNNs). However, traditional convolutional autoencoders are limited to local features, struggling to assimilate global feature information. CNNs' generalizability enables the reconstruction of certain anomalous regions. This is particularly evident when normal and abnormal regions, despite having similar pixel values, contain different semantic information, leading to ineffective anomaly detection. Furthermore, collecting abnormal image samples during actual industrial production poses challenges, often resulting in data imbalance. To mitigate these issues, this study proposes an unsupervised anomaly detection model employing the Vision Transformer (ViT) architecture, incorporating a Transformer structure to understand the global context between image blocks, thereby extracting a superior representation of feature information. It integrates a memory module to catalog normal sample features, both to counteract anomaly reconstruction issues and bolster feature representation, and additionally introduces a coordinate attention (CA) mechanism to intensify focus on image features at both spatial and channel dimensions, minimizing feature information loss and thereby enabling more precise anomaly identification and localization. Experiments conducted on two public datasets, MVTec AD and BeanTech AD, substantiate the method's effectiveness, demonstrating an approximate 20% improvement in average AUROC% at the image level over traditional convolutional encoders.

A Holistic review and performance evaluation of unsupervised learning methods for network anomaly detection

Abstract The evolving cyber-attack landscape demands flexible and precise protection for information and networks. Network anomaly detection (NAD) systems play a crucial role in preventing and detecting abnormal activities on the network that may lead to catastrophic outcomes when undetected. This paper aims to provide a comprehensive analysis of NAD using unsupervised learning (UL) methods to evaluate the effectiveness of such systems. The paper presents a detailed overview of several UL techniques, lists the current developments and innovations in UL techniques for network anomaly and intrusion detection, and evaluates 13 unsupervised anomaly detection algorithms empirically on benchmark datasets such as NSL-KDD, UNSW-NB15, and CIC-IDS 2017 to analyze the performance of different classes of UL approaches for NAD systems. This study demonstrates the effectiveness of NAD algorithms, discusses UL approaches' research challenges, and unearths the potential drawbacks in the current network security environment.

MSC-AD: A Multiscene Unsupervised Anomaly Detection Dataset for Small Defect Detection of Casting Surface

MSC-AD: A Multiscene Unsupervised Anomaly Detection Dataset for Small Defect Detection of Casting Surface

SCA-GANomaly: an unsupervised anomaly detection model of high-speed railway catenary components

SCA-GANomaly: an unsupervised anomaly detection model of high-speed railway catenary components

Unsupervised Continual Anomaly Detection with Contrastively-Learned Prompt

Unsupervised Anomaly Detection (UAD) with incremental training is crucial in industrial manufacturing, as unpredictable defects make obtaining sufficient labeled data infeasible. However, continual learning methods primarily rely on supervised annotations, while the application in UAD is limited due to the absence of supervision. Current UAD methods train separate models for different classes sequentially, leading to catastrophic forgetting and a heavy computational burden. To address this issue, we introduce a novel Unsupervised Continual Anomaly Detection framework called UCAD, which equips the UAD with continual learning capability through contrastively-learned prompts. In the proposed UCAD, we design a Continual Prompting Module (CPM) by utilizing a concise key-prompt-knowledge memory bank to guide task-invariant 'anomaly' model predictions using task-specific 'normal' knowledge. Moreover, Structure-based Contrastive Learning (SCL) is designed with the Segment Anything Model (SAM) to improve prompt learning and anomaly segmentation results. Specifically, by treating SAM's masks as structure, we draw features within the same mask closer and push others apart for general feature representations. We conduct comprehensive experiments and set the benchmark on unsupervised continual anomaly detection and segmentation, demonstrating that our method is significantly better than anomaly detection methods, even with rehearsal training. The code will be available at https://github.com/shirowalker/UCAD.

Generating and Reweighting Dense Contrastive Patterns for Unsupervised Anomaly Detection

Recent unsupervised anomaly detection methods often rely on feature extractors pretrained with auxiliary datasets or on well-crafted anomaly-simulated samples. However, this might limit their adaptability to an increasing set of anomaly detection tasks due to the priors in the selection of auxiliary datasets or the strategy of anomaly simulation. To tackle this challenge, we first introduce a prior-less anomaly generation paradigm and subsequently develop an innovative unsupervised anomaly detection framework named GRAD, grounded in this paradigm. GRAD comprises three essential components: (1) a diffusion model (PatchDiff) to generate contrastive patterns by preserving the local structures while disregarding the global structures present in normal images, (2) a self-supervised reweighting mechanism to handle the challenge of long-tailed and unlabeled contrastive patterns generated by PatchDiff, and (3) a lightweight patch-level detector to efficiently distinguish the normal patterns and reweighted contrastive patterns. The generation results of PatchDiff effectively expose various types of anomaly patterns, e.g. structural and logical anomaly patterns. In addition, extensive experiments on both MVTec AD and MVTec LOCO datasets also support the aforementioned observation and demonstrate that GRAD achieves competitive anomaly detection accuracy and superior inference speed.

Personalized anomaly detection in PPG data using representation learning and biometric identification

Photoplethysmography (PPG) signals, typically acquired from wearable devices, hold significant potential for continuous fitness-health monitoring. In particular, heart conditions that manifest in rare and subtle deviating heart patterns may be interesting. However, robust and reliable anomaly detection within these data remains a challenge due to the scarcity of labeled data and high inter-subject variability. This paper introduces a two-stage framework leveraging representation learning and personalization to improve anomaly detection performance in PPG data. The proposed framework first employs representation learning to transform the original PPG signals into a more discriminative and compact representation. We then apply three different unsupervised anomaly detection methods for movement detection and biometric identification. We validate our approach using two different datasets in both generalized and personalized scenarios. Our results demonstrate significant improvements: for movement detection, in the generalized scenario, AUCs improved from barely 0.5 to above 0.9 with representation learning. Importantly, inter-subject variability was substantially reduced, from around 0.4 to below 0.1. In the personalized scenario, AUCs became close to 1.0, with variability further reduced to below 0.05, indicating the effectiveness of both representation learning and personalization for anomaly detection in PPG data. Similar enhancements were observed in biometric identification, emphasizing how our approach can minimize inter-subject variability and enhance PPG-based health monitoring systems.

ProMvSD: Towards unsupervised knowledge graph anomaly detection via prior knowledge integration and multi-view semantic-driven estimation

Knowledge graphs (KGs) have found extensive applications within intelligent systems, such as information retrieval. Much of the research has predominantly focused on completing missing knowledge, with little consideration given to examining errors. Unfortunately, during customizing KGs, diverse unpredictable errors are virtually unavoidable to be introduced, and these anomalies significantly impact the performance of applications. Detecting erroneous knowledge presents a formidable challenge due to the costly acquisition of ground-truth labels. In this work, we develop an unsupervised anomaly detection framework named ProMvSD, aiming to adapt KGs of varying scales via serialization components. To overcome the insufficient contextual information provided by the topological structure, we introduce the large language model as a reasoner to extract prior knowledge from extensive pre-trained textual data, thereby enhancing the understanding of KGs. Anomalous triple may result in a larger semantic gap between the head and tail neighborhoods. To uncover latent anomalies effectively, we propose a multi-view semantic-driven model (MvSD) based on the assumptions of self-consistency and information stability. MvSD jointly estimates the suspiciousness of triples from three hyperviews: node-view semantic contradiction, triple-view semantic gap, and pathway-view semantic gap. Extensive experiments on three English benchmark KGs and a Chinese medical KG demonstrate that, for the top 1% of the most suspicious triples, we can detect real anomalies with at most 99.9% accuracy. Furthermore, ProMvSD significantly outperforms state-of-the-art representation learning baselines, achieving a 29.2% improvement in detecting all anomalies.

An intelligent inspection method for body-in-white weld quality based on vibration excitation response signals

To enhance the assembly quality in Body-in-White (BIW) assembly, this paper proposes an intelligent detection method for the nugget quality of Resistance Spot Weld (RSW) based on weld joint vibration excitation response signals. The method proposes a novel deep learning model, the Local-Global Hierarchical Graph Neural Network (LGHGNN). LGHGNN can automatically construct graph structures and, by introducing a newly designed upgrade pooling operation, extends the traditional flat structure of graph networks into a hierarchical structure within three-dimensional space. Therefore, LGHGNN achieves layered interaction of local-global information, enabling the model to focus on local details while gaining a broader learning perspective. Additionally, this paper proposes a strategy for multi-label unsupervised anomaly detection that involves layered interaction and collaborative decision-making for local and global graphs. The effectiveness of LGHGNN is demonstrated through its application in the BIW right front door assembly, achieving a remarkable 97.5% average accuracy in multi-region parallel anomaly detection.

Encoder-Decoder Contrast for Unsupervised Anomaly Detection in Medical Images.

Unsupervised anomaly detection (UAD) aims to recognize anomalous images based on the training set that contains only normal images. In medical image analysis, UAD benefits from leveraging the easily obtained normal (healthy) images, avoiding the costly collecting and labeling of anomalous (unhealthy) images. Most advanced UAD methods rely on frozen encoder networks pre-trained using ImageNet for extracting feature representations. However, the features extracted from the frozen encoders that are borrowed from natural image domains coincide little with the features required in the target medical image domain. Moreover, optimizing encoders usually causes pattern collapse in UAD. In this paper, we propose a novel UAD method, namely Encoder-Decoder Contrast (EDC), which optimizes the entire network to reduce biases towards pre-trained image domain and orient the network in the target medical domain. We start from feature reconstruction approach that detects anomalies from reconstruction errors. Essentially, a contrastive learning paradigm is introduced to tackle the problem of pattern collapsing while optimizing the encoder and the reconstruction decoder simultaneously. In addition, to prevent instability and further improve performances, we propose to bring globality into the contrastive objective function. Extensive experiments are conducted across four medical image modalities including optical coherence tomography, color fundus image, brain MRI, and skin lesion image, where our method outperforms all current state-of-the-art UAD methods. Code is available at: https://github.com/guojiajeremy/EDC.

PKU-GoodsAD: A Supermarket Goods Dataset for Unsupervised Anomaly Detection and Segmentation

PKU-GoodsAD: A Supermarket Goods Dataset for Unsupervised Anomaly Detection and Segmentation

A Comparative Study of Deep Neural Networks and Support Vector Machines for Unsupervised Anomaly Detection in Cloud Computing Environments

Abstract: Cloud computing is now ubiquitous and provides convenient access to computing resources on demand. Cloud environments are complex and prone to faults, which can have a negative impact on service quality. Cloud providers must be able to detect issues in a proactive manner using unsupervised anomaly detection. This does not require labeled data. This paper presents a comparison of deep neural networks and support vector machine (SVMs), both used for unsupervised anomaly identification in cloud environments. On benchmark datasets provided by cloud providers, we evaluate the performance Autoencoders with LSTM models, One Class SVMs, and Isolation Forests. Our results show that shallow Autoencoders do not capture workload patterns well, but LSTMs or Convolutional Autoencoders can. SVMs are as good or better than Autoencoders. One-Class SVMs show robust performance in all workloads. Isolation Forests perform poorly on cloud data that is seasonal. One-Class SVMs are the most accurate and low latency option for anomaly detection. Our findings offer cloud providers guidance on how to select suitable unsupervised models based upon their performance, interpretability, and computational overhead. The results and comparative methodology will be used to inform future research into adapting unsupervised-learning for cloud anomaly detection.

Exploiting negative correlation for unsupervised anomaly detection in contaminated time series

Anomaly detection in time series data is crucial for many fields such as healthcare, meteorology, and industrial fault detection. However, traditional unsupervised time series anomaly detection methods suffer from biased anomaly measurement under contaminated training data. Most of existing methods employ hard strategies for contamination calibration by assigning pseudo-label to training data. These hard strategies rely on threshold selection and result in suboptimal performance. To address this problem, in this paper, we propose a novel unsupervised anomaly detection framework for contaminated time series (NegCo), which builds an effective soft contamination calibration strategy by exploiting the observed negative correlation between semantic representation and anomaly detection inherent within the autoencoder framework. We innovatively redefine anomaly detection in data contamination scenarios as an optimization problem rooted in this negative correlation. To model this negative correlation, we introduce a dual construct: morphological similarity captures semantic distinctions relevant to normality, while reconstruction consistency quantifies deviations indicative of anomalies. Firstly, the morphological similarity is effectively measured based on the representative normal samples generated from the center of the learned Gaussian distribution. Then, an anomaly measurement calibration loss function is designed based on negative correlation between morphological similarity and reconstruction consistency, to calibrate the biased anomaly measurement caused by contaminated samples. Extensive experiments on various time series datasets show that the proposed NegCo outperforms state-of-the-art baselines, achieving an improvement of 6.2% to 26.8% in Area Under the Receiver Operating Characteristics (AUROC) scores, particularly in scenarios with heavily contaminated training data.

(Predictable) performance bias in unsupervised anomaly detection

With the ever-increasing amount of medical imaging data, the demand for algorithms to assist clinicians has amplified. Unsupervised anomaly detection (UAD) models promise to aid in the crucial first step of disease detection. While previous studies have thoroughly explored fairness in supervised models in healthcare, for UAD, this has so far been unexplored. In this study, we evaluated how dataset composition regarding subgroups manifests in disparate performance of UAD models along multiple protected variables on three large-scale publicly available chest X-ray datasets. Our experiments were validated using two state-of-the-art UAD models for medical images. Finally, we introduced subgroup-AUROC (sAUROC), which aids in quantifying fairness in machine learning. Our experiments revealed empirical "fairness laws" (similar to "scaling laws" for Transformers) for training-dataset composition: Linear relationships between anomaly detection performance within a subpopulation and its representation in the training data. Our study further revealed performance disparities, even in the case of balanced training data, and compound effects that exacerbate the drop in performance for subjects associated with multiple adversely affected groups. Our study quantified the disparate performance of UAD models against certain demographic subgroups. Importantly, we showed that this unfairness cannot be mitigated by balanced representation alone. Instead, the representation of some subgroups seems harder to learn by UAD models than that of others. The empirical "fairness laws" discovered in our study make disparate performance in UAD models easier to estimate and aid in determining the most desirable dataset composition. European Research Council Deep4MI.

Real-time anomaly detection of short-term traffic disruptions in urban areas through adaptive isolation forest

The escalating congestion impacts of short-term traffic disruptions, such as double parking or short-duration work zones, are gaining increased attention. This study introduces an enhanced isolation forest-based unsupervised anomaly detection algorithm to detect short-term traffic disruptions in urban areas. To enable the detection in real-time, a sliding window approach is introduced to allow the streaming data to be read and processed according to the window size. An adaptive threshold that automatically adjusts itself to overcome the limitations of the miss rate on local anomalies is shown to further enhance the model predictions. The proposed algorithm is empirically validated on four study sites in Manhattan, New York City. The experimental results demonstrate that the proposed unsupervised algorithm can effectively detect different types of traffic anomalies including accidents, work zones/road closures, traffic jams, double parking events and police activities. On all sites, the average detection rate is 81.1% for traffic jams and 89.6% for police activities, respectively. For three out of the four sites, the detection rate ranges from 71.4% to 100% for accidents, work zones and double parking. An optimizer using high-pass filter is also presented to further improve off-line detection. The primary advantages of this proposed computationally-efficient method are that its only required data input is travel time information and that it does not need labeled data for training, which make it highly deployable for real-time operational applications and can be easily adopted by other cities.

Unsupervised Deep Anomaly Detection for Industrial Multivariate Time Series Data

With the rapid development of deep learning, researchers are actively exploring its applications in the field of industrial anomaly detection. Deep learning methods differ significantly from traditional mathematical modeling approaches, eliminating the need for intricate mathematical derivations and offering greater flexibility. Deep learning technologies have demonstrated outstanding performance in anomaly detection problems and gained widespread recognition. However, when dealing with multivariate data anomaly detection problems, deep learning faces challenges such as large-scale data annotation and handling relationships between complex data variables. To address these challenges, this study proposes an innovative and lightweight deep learning model—the Attention-Based Deep Convolutional Autoencoding Prediction Network (AT-DCAEP). The model consists of a characterization network based on convolutional autoencoders and a prediction network based on attention mechanisms. The AT-DCAEP exhibits excellent performance in multivariate time series data anomaly detection without the need for pre-labeling large-scale datasets, making it an efficient unsupervised anomaly detection method. We extensively tested the performance of AT-DCAEP on six publicly available datasets, and the results show that compared to current state-of-the-art methods, AT-DCAEP demonstrates superior performance, achieving the optimal balance between anomaly detection performance and computational cost.

The Credit Card Anti-fraud Detection Model in the Context of Dynamic Integration Selection Algorithm

Because of its close relationship with information technology, every change in the field of information technology will have an important impact on the means, time and space distribution of the final fraud of the credit card network. The rapid development and popularization of artificial intelligence (AI) software based on Deepfake technology has greatly promoted the intelligent transformation of credit card network fraud. Through the use of advanced hardware and software equipment, criminals have iterated on their own fraud tools, from the use of traditional phone calls and text messages to the use of the latest AI software. Therefore, for the problem of credit card fraud detection with missing data set labels and highly unbalanced category distribution, A credit card approval anomaly detection model DES-HBOS (Dynamic Ensemble Selection based on Histogram-based Outlier Score) is proposed. Firstly, the unsupervised anomaly detection algorithm is used to construct the false label of the training set customer. Then, the customer capability area to be measured is determined, and the classifier performance is evaluated according to Pearson correlation coefficient. Finally, a set of optimal classifiers is selected to integrate the test customers. Experiments on real credit card customer data sets show that DES-HBOS has a higher Recall and can identify more fraudulent customers than other 6 classical anomaly detection models. Comparative experiments were carried out on 4 unbalanced data sets, and the experimental results showed that DES-HBOS had stronger anomaly detection ability than HBOS.

MSFlow: Multiscale Flow-Based Framework for Unsupervised Anomaly Detection.

Unsupervised anomaly detection (UAD) attracts a lot of research interest and drives widespread applications, where only anomaly-free samples are available for training. Some UAD applications intend to locate the anomalous regions further even without any anomaly information. Although the absence of anomalous samples and annotations deteriorates the UAD performance, an inconspicuous, yet powerful statistics model, the normalizing flows, is appropriate for anomaly detection (AD) and localization in an unsupervised fashion. The flow-based probabilistic models, only trained on anomaly-free data, can efficiently distinguish unpredictable anomalies by assigning them much lower likelihoods than normal data. Nevertheless, the size variation of unpredictable anomalies introduces another inconvenience to the flow-based methods for high-precision AD and localization. To generalize the anomaly size variation, we propose a novel multiscale flow-based framework (MSFlow) composed of asymmetrical parallel flows followed by a fusion flow to exchange multiscale perceptions. Moreover, different multiscale aggregation strategies are adopted for image-wise AD and pixel-wise anomaly localization according to the discrepancy between them. The proposed MSFlow is evaluated on three AD datasets, significantly outperforming existing methods. Notably, on the challenging MVTec AD benchmark, our MSFlow achieves a new state-of-the-art (SOTA) with a detection AUORC score of up to 99.7%, localization AUCROC score of 98.8% and PRO score of 97.1%.