Anomaly Detection Algorithm Research Articles

Revisiting streaming anomaly detection: benchmark and evaluation

Anomaly detection in streaming data is an important task for many real-world applications, such as network security, fraud detection, and system monitoring. However, streaming data often exhibit concept drift, which means that the data distribution changes over time. This poses a significant challenge for many anomaly detection algorithms, as they need to adapt to the evolving data to maintain high detection accuracy. Existing streaming anomaly detection algorithms lack a unified evaluation framework that validly assesses their performance and robustness under different types of concept drifts and anomalies. In this paper, we conduct a systematic technical review of the state-of-the-art methods for anomaly detection in streaming data. We propose a new data generator, called SCAR (Streaming data generator with Customizable Anomalies and concept dRifts), that can synthesize streaming data based on synthetic and real-world datasets from different domains. Furthermore, we adapt four static anomaly detection models to the streaming setting using a generic reconstruction strategy as baselines, and then compare them systematically with 9 existing streaming anomaly detection algorithms on 76 synthesized datasets that have various types of anomalies and concept drifts. The challenges and future research directions for anomaly detection in streaming data are also presented.

Research on University Network Data Anomaly Detection and Security Protection Algorithm Based on edge computing

Research on University Network Data Anomaly Detection and Security Protection Algorithm Based on edge computing

AIoT-Based Visual Anomaly Detection in Photovoltaic Sequence Data via Sequence Learning

Anomaly detection is a common analytical task aimed at identifying rare cases that differ from the majority of typical cases in a dataset. In the management of photovoltaic (PV) power generation systems, it is essential for electric power companies to effectively detect anomalies in PV sequence data, as this helps operators and experts understand and interpret anomalies within PV arrays when making response decisions. However, traditional methods that rely on manual labor and regular data collection are difficult to monitor in real time, resulting in delays in fault detection and localization. Traditional machine learning algorithms are slow and cumbersome in processing data, which affects the operational safety of PV plants. In this paper, we propose a visual analytic approach for detecting and exploring anomalous sequences in a PV sequence dataset via sequence learning. We first compare the sequences with their reconstructions through an unsupervised anomaly detection algorithm (Long Short-Term Memory) based on AutoEncoders to identify anomalies. To further enhance the accuracy of anomaly detection, we integrate the artificial intelligence of things (AIoT) technology with a strict time synchronization data collection and real-time processing algorithm. This integration ensures that data from multiple sensors are synchronized and processed in real time. Then, we analyze the characteristics of the anomalies based on the visual comparison of different PV sequences and explore the potential correlation factors to analyze the possible causes of the anomalies. Case studies based on authentic enterprise datasets demonstrate the effectiveness of our method in the anomaly detection and exploration of PV sequence data.

Advancing Algorithmic Adaptability in Hyperspectral Anomaly Detection with Stacking-Based Ensemble Learning

Anomaly detection in hyperspectral imaging is crucial for remote sensing, driving the development of numerous algorithms. However, systematic studies reveal a dichotomy where algorithms generally excel at either detecting anomalies in specific datasets or generalizing across heterogeneous datasets (i.e., lack adaptability). A key source of this dichotomy may center on the singular and like biases frequently employed by existing algorithms. Current research lacks experimentation into how integrating insights from diverse biases might counteract problems in singularly biased approaches. Addressing this gap, we propose stacking-based ensemble learning for hyperspectral anomaly detection (SELHAD). SELHAD introduces the integration of hyperspectral anomaly detection algorithms with diverse biases (e.g., Gaussian, density, partition) into a singular ensemble learning model and learns the factor to which each bias should contribute so anomaly detection performance is optimized. Additionally, it introduces bootstrapping strategies into hyperspectral anomaly detection algorithms to further increase robustness. We focused on five representative algorithms embodying common biases in hyperspectral anomaly detection and demonstrated how they result in the previously highlighted dichotomy. Subsequently, we demonstrated how SELHAD learns the interplay between these biases, enabling their collaborative utilization. In doing so, SELHAD transcends the limitations inherent in individual biases, thereby alleviating the dichotomy and advancing toward more adaptable solutions.

Enhancing Hyperspectral Anomaly Detection Algorithm Comparisons: Leveraging Dataset and Algorithm Characteristics

Validating the contributions of new algorithms is a critical step in hyperspectral anomaly detection (HAD) research. Typically, validation involves comparing the performance of a proposed algorithm against other algorithms using a series of benchmark datasets. Despite the longstanding use of this comparison process, little attention has been paid to the characteristics of datasets and algorithms that ensure each algorithm has an equal opportunity of performing well. Characteristics of datasets and algorithms that inadvertently favor one algorithm can skew results, leading to misleading conclusions. To address this issue, this study introduces a feature-centric framework designed to assist in ensuring an unbiased comparison of HAD algorithms. The framework identifies significant correlations between datasets and algorithms by extracting distribution-related features from the datasets and statistically testing them against the algorithmic outcomes. The identified trends are then compared across datasets to ensure that all relevant trends are equally represented, thereby ensuring diversity and validating that no singular algorithm is afforded an inherent advantage. The framework was tested on five algorithms across 14 datasets. The results indicate that multiple measures of variance within the datasets are key drivers of diversity, and these measures accurately predicted algorithmic outcomes for 12 of the 14 datasets. This suggests that the identified trends effectively explain the algorithmic outcomes and highlights the importance of incorporating datasets with a diverse range of variances in comparisons of HAD algorithms.

Isolation Forest Anomaly Detection Algorithm Based On Multi-level Sub-subspace Partition

In the research of credit loan fraud detection, the isolation forest algorithm has attracted much attention because of its ability to efficiently process large-scale data sets. However, when facing high-dimensional data, the performance of the isolation forest algorithm is easily affected, resulting in deviation of the detection results. In order to solve the above problems, this paper proposes an isolation forest anomaly detection algorithm based on multi-level sub-subspace division. Firstly, the random forest algorithm is used to evaluate the importance of each feature, and the data is divided into different subspaces according to the importance of each feature, and the corresponding weight is assigned to each subspace. Then, the isolation forest algorithm is applied in each subspace for anomaly detection, and the anomaly score of each subspace is obtained. Finally, the anomaly score and weight of each subspace were combined to obtain the final anomaly detection score. In order to evaluate the effectiveness of the algorithm, the proposed algorithm was compared with other four algorithms on the credit loan fraud data set. The results show that the AUC index, accuracy, recall rate and F1 score of the proposed algorithm are higher than those of the comparison algorithms, showing high effectiveness.

Evaluation of Deep Isolation Forest (DIF) Algorithm for Mineral Prospectivity Mapping of Polymetallic Deposits

Mineral prospectivity mapping (MPM) is crucial for efficient mineral exploration, where prospective zones are identified in a cost-effective manner. This study focuses on generating prospectivity maps for hydrothermal polymetallic mineralization in the Feizabad area, in northeastern Iran, using unsupervised anomaly detection methods, i.e., isolation forest (IForest) and deep isolation forest (DIF) algorithms. As mineralization events are rare and complex, traditional approaches continue to encounter difficulties, despite advances in MPM. In this respect, unsupervised anomaly detection algorithms, which do not rely on ground truth samples, offer a suitable solution. Here, we compile geospatial datasets on the Feizabad area, which is known for its polymetallic mineralization showings. Fourteen evidence layers were created, based on the geology and mineralization characteristics of the area. Both the IForest and DIF algorithms were employed to identify areas with high mineralization potential. The DIF, which uses neural networks to handle non-linear relationships in high-dimensional data, outperformed the traditional decision tree-based IForest algorithm. The results, evaluated through a success rate curve, demonstrated that the DIF provided more accurate prospectivity maps, effectively capturing complex, non-linear relationships. This highlights the DIF algorithm’s suitability for MPM, offering significant advantages over the IForest algorithm. The present study concludes that the DIF algorithm, and similar unsupervised anomaly detection algorithms, are highly effective for MPM, making them valuable tools for both brownfield and greenfield exploration.

An integrated impact localization method for thermal protection structure based on hybrid kernel extreme learning machine

The thermal protection structure (TPS) of the aircraft is highly susceptible to impact events from foreign objects, which may cause significant risk to the safe flight of the aircraft. This paper presents an impact localization method based on a hybrid kernel extreme learning machine (HKELM). The impact signal is firstly processed by the Short Term/Long Term Average (STA/LTA) ratio method and the time of arrival (TOA) of the guided wave is further extracted by the Akaike information criterion (AIC) method, which is used as the input of the neural network. The impact position is used as the output of the neural network to train the HKELM model. The hyperparameters of the HKELM are optimized by using the immune particle swarm algorithm (IPSO). Meanwhile, the local outlier factor (LOF) algorithm is used to detect the abnormal data, and a data reconstruction method based on the correlation coefficient is proposed to correct abnormal data. Finally, the localization results demonstrate the validity of the anomaly detection algorithm and the IPSO-HKELM network model.

Unsupervised anomaly detection of multivariate time series based on multi-standard fusion

With the wide application of the Internet of Things, industrial systems generate a large amount of multivariate time series data every day. Industrial data often contains a wide variety of anomalies. The existing anomaly detection methods have some shortcomings in dealing with large-scale, high-dimensional, and real-time industrial data. Particularly in the absence of prior knowledge, unsupervised learning methods are easily influenced by noise. Therefore, we propose a new unsupervised anomaly detection algorithm of multivariate time series based on multi-standard fusion (MSAD). Specifically, MSAD first classifies the data by analyzing the data density and sample spacing, thereby converting unsupervised anomaly detection into weakly supervised anomaly detection. Then, based on the characteristics of the data and deep learning technology, the degree of abnormality in the sample is comprehensively measured from four perspectives: holistic, local, time, and deep feature learning. At this stage, MSAD 's ability to learn edge features of data has been improved. Finally, based on MSAD, we design an anomaly detection method for industrial data streams in combination with data filtering techniques. The experimental results show that MSAD can detect anomalies more accurately and show stronger robustness to the influence of noise.

Using Kan Extensions to Motivate the Design of a Surprisingly Effective Unsupervised Linear SVM on the Occupancy Dataset

Recent research has suggested that category theory can provide useful insights into the field of machine learning (ML). One example is improving the connection between an ML problem and the design of a corresponding ML algorithm. A tool from category theory called a Kan extension is used to derive the design of an unsupervised anomaly detection algorithm for a commonly used benchmark, the Occupancy dataset. Achieving an accuracy of 93.5% and an ROCAUC of 0.98, the performance of this algorithm is compared to state-of-the-art anomaly detection algorithms tested on the Occupancy dataset. These initial results demonstrate that category theory can offer new perspectives with which to attack problems, particularly in making more direct connections between the solutions and the problem’s structure.

Robust anomaly detection for particle physics using multi-background representation learning

Abstract Anomaly, or out-of-distribution, detection is a promising tool for aiding discoveries of new particles or processes in particle physics. In this work, we identify and address two overlooked opportunities to improve anomaly detection for high-energy physics. First, rather than train a generative model on the single most dominant background process, we build detection algorithms using representation learning from multiple background types, thus taking advantage of more information to improve the estimation of what is relevant for detection. Second, we generalize decorrelation to the multi-background setting, thus directly enforcing a more complete definition of robustness for anomaly detection. We demonstrate the benefit of the proposed robust multi-background anomaly detection algorithms on a high-dimensional dataset of particle decays at the Large Hadron Collider.

Hyperspectral Anomaly Detection Based on Spectral Similarity Variability Feature.

In the traditional method for hyperspectral anomaly detection, spectral feature mapping is used to map hyperspectral data to a high-level feature space to make features more easily distinguishable between different features. However, the uncertainty in the mapping direction makes the mapped features ineffective in distinguishing anomalous targets from the background. To address this problem, a hyperspectral anomaly detection algorithm based on the spectral similarity variability feature (SSVF) is proposed. First, the high-dimensional similar neighborhoods are fused into similar features using AE networks, and then the SSVF are obtained using residual autoencoder. Finally, the final detection of SSVF was obtained using Reed and Xiaoli (RX) detectors. Compared with other comparison algorithms with the highest accuracy, the overall detection accuracy (AUCODP) of the SSVFRX algorithm is increased by 0.2106. The experimental results show that SSVF has great advantages in both highlighting anomalous targets and improving separability between different ground objects.

Graph neural network-based lithium-ion battery state of health estimation using partial discharging curve

Data-driven methods have gained extensive attention in estimating the state of health (SOH) of lithium-ion batteries. Accurate SOH estimation requires degradation-relevant features and alignment of statistical distributions between training and testing datasets. However, current research often overlooks these needs and relies on arbitrary voltage segment selection. To address these challenges, this paper introduces an innovative approach leveraging spatio-temporal degradation dynamics via graph convolutional networks (GCNs). Our method systematically selects discharge voltage segments using the Matrix Profile anomaly detection algorithm, eliminating the need for manual selection and preventing information loss. These selected segments form a fundamental structure integrated into the GCN-based SOH estimation model, capturing inter-cycle dynamics and mitigating statistical distribution incongruities between offline training and online testing data. Validation with a widely accepted open-source dataset demonstrates that our method achieves precise SOH estimation, with a root mean squared error of less than 1%.

Multi-Class Hypersphere Anomaly Detection Based on Edge Outlier Exposure Set and Margin

Currently, the decision boundary of the multi-class anomaly detection algorithm based on deep learning does not sufficiently capture the positive class region, posing a risk of abnormal sample features falling into the domain of normal sample features and potentially leading to misleading outcomes in practical applications. In response to the above problems, this paper proposes a new method called multi-class hypersphere anomaly detection (MMHAD) based on the edge outlier exposure set and margin. The method aims to utilize convolutional neural networks for joint training of all normal object classes, identifying a shared set of outlier exposures, learning compact identification features, and setting appropriate edge parameters to guide the model in mapping outliers outside the hypersphere. This approach enables more comprehensive detection of various types of exceptions. The experiments demonstrate that the algorithm is superior to the most advanced baseline method, with an improvement of 26.0%, 8.2%, and 20.1% on CIFAR-10 and 14.8%, 12.0%, and 20.1% on FMNIST in the cases of (2/8), (5/5), and (9,1), respectively. Furthermore, we investigate the challenging (2/18) case on CIFAR-100, where our method achieves approximately 17.4% AUROC gain. Lastly, for a recycling waste dataset with the (4/1) case, our MMHAD yields a notable 22% enhancement in performance. Experimental results show the effectiveness of the proposed model in multi-classification anomaly detection.

Hyperspectral Anomaly Detection Based on Adaptive Low-Rank Transformed Tensor.

Hyperspectral anomaly detection, which is aimed at distinguishing anomaly pixels from the surroundings in spatial features and spectral characteristics, has attracted considerable attention due to its various applications. In this article, we propose a novel hyperspectral anomaly detection algorithm based on adaptive low-rank transform, in which the input hyperspectral image (HSI) is divided into a background tensor, an anomaly tensor, and a noise tensor. To take full advantage of the spatial-spectral information, the background tensor is represented as the product of a transformed tensor and a low-rank matrix. The low-rank constraint is imposed on frontal slices of the transformed tensor to depict the spatial-spectral correlation of the HSI background. Besides, we initialize a matrix with predefined size and then minimize its l2.1 -norm to adaptively derive an appropriate low-rank matrix. The anomaly tensor is constrained with the l2.1.1 -norm to depict the group sparsity of anomalous pixels. We integrate all regularization terms and a fidelity term into a non-convex problem and develop a proximal alternating minimization (PAM) algorithm to solve it. Interestingly, the sequence generated by the PAM algorithm is proven to converge to a critical point. Experimental results conducted on four widely used datasets demonstrate the superiority of the proposed anomaly detector over several state-of-the-art methods.

Unsupervised Data-Driven Approach for Scour Damage Detection in Offshore Monopile Foundations

Scour erosion around monopile foundations significantly impacts the structural integrity and operational performance of offshore wind turbines (OWTs). To deal with this problem, this paper proposes an innovative unsupervised data-driven methodology to detect foundation scour in OWTs. The proposed method employs acceleration data collected from the tower. A numerical model of the NERL 5 MW OWT installed in medium-dense sand is developed using OpenFAST software. The study employs the novelty detection method, where only data obtained from the healthy conditions are used for training and establishing a base model. In addition, the influence of operational conditions, e.g. different wind and wave parameters is also evaluated. The process consists of three steps: (a) data pre-processing to extract time domain features and isolate environmental variations, (b) training an unsupervised anomaly detection algorithm by employing the extracted features and (c) conducting real-time anomaly detection to assess the scour damage. The study results highlight the robustness and efficiency of the proposed method in detecting scour damage around monopiles under different environmental conditions. Keywords: Monopile, scour, unsupervised, Machine learning, damage detection, SHM.

Unsupervised Anomaly Detection of Industrial Images Based on Dual Generator Reconstruction Networks

At present, deep learning techniques are increasingly utilized in computer vision and anomaly detection. To address the limitations of inadequate reconstruction capability and subpar performance in reconstruction-based anomaly detection, this study enhances the existing algorithm and introduces an unsupervised anomaly detection of industrial images algorithm based on dual generator reconstruction networks-DGRNet. The network consists of two generators and a discriminator, introducing a widely recognized denoising diffusion probabilistic model (DDPM) as one of the generators, an autoencoder (AE) as the other generator, and a decoder as the discriminator. The model is tested on the MVTec AD dataset, and in the case of no additional training data,the anomaly detection AUC result of DGRNet exceeds the baseline method based on reconstruction by 19.6 percentage points. The experimental results show that DGRNet can improve the detection performance in the anomaly detection algorithm based on unsupervised and reconstructed networks.

Automatic fault detection in grid-connected photovoltaic systems via variational autoencoder-based monitoring

Anomaly detection is indispensable for ensuring the reliable operation of grid-connected photovoltaic (PV) systems. This study introduces a semi-supervised deep learning approach for fault detection in such systems. The method leverages a variational autoencoder (VAE) to extract features and identify anomalies. By training the VAE on normal operation data, a compact latent space representation is created. Abnormal observations, indicating faults, exhibit distinct feature vectors in this latent space. Multiple anomaly detection algorithms, including Isolation Forest, Epileptic Envelope, Local Outlier Factor, and One-Class SVM, are employed to discern normal and abnormal observations. This semi-supervised approach only requires fault-free data for training, without labeled faults, making it attractive in practice. A publicly available dataset, the Grid-connected PV System Faults (GPVS-Faults) dataset, which includes data from a PV plant operating in both maximum power point tracking (MPPT) and intermediate power point tracking (IPPT) switching modes, is used for evaluation. The proposed approach is assessed across various fault scenarios, such as partial shading, inverter faults, and MPPT/IPPT controller faults in boost converters. The outcomes underscore the effectiveness of VAE-based techniques in accurately identifying these faults, with accuracy rates reaching up to 92.90% for MPPT mode and 92.99% for IPPT mode, thus contributing to the robustness of fault detection in grid-connected PV systems.

Multivariate anomaly detection models enhance identification of errors in routine clinical chemistry testing.

Conventional autoverification rules evaluate analytes independently, potentially missing unusual patterns of results indicative of errors such as serum contamination by collection tube additives. This study assessed whether multivariate anomaly detection algorithms could enhance the detection of such errors. Multivariate Gaussian, k-nearest neighbours (KNN) distance, and one-class support vector machine (SVM) anomaly detection models, along with conventional limit checks, were developed using a training dataset of 127,451 electrolyte, urea, and creatinine (EUC) results, with a 5 % flagging rate targeted for all approaches. The models were compared with limit checks for their ability to detect atypical EUC results from samples spiked with additives from collection tubes: EDTA, fluoride, sodium citrate, or acid citrate dextrose (n=200 per contaminant). The study additionally assessed the ability of the models to identify 127,449 single-analyte errors, a potential weakness of multivariate models. The KNN distance and SVM models outperformed limit checks for detecting all contaminants (p-values <0.05). The multivariate Gaussian model did not surpass limit checks for detecting EDTA contamination but was superior for detecting the other additives. All models surpassed limit checks for identifying single-analyte errors, with the KNN distance model demonstrating the highest overall sensitivity. Multivariate anomaly detection models, particularly the KNN distance model, were superior to the conventional approach for detecting serum contamination and single-analyte errors. Developing multivariate approaches to autoverification is warranted to optimise error detection and improve patient safety.

Neural Network-Based Log Anomaly Detection Algorithm for 6G Wireless Integrated Cyber-Physical System

Neural Network-Based Log Anomaly Detection Algorithm for 6G Wireless Integrated Cyber-Physical System