Unsupervised Anomaly Detection Algorithm Research Articles

AIoT-Based Visual Anomaly Detection in Photovoltaic Sequence Data via Sequence Learning

Anomaly detection is a common analytical task aimed at identifying rare cases that differ from the majority of typical cases in a dataset. In the management of photovoltaic (PV) power generation systems, it is essential for electric power companies to effectively detect anomalies in PV sequence data, as this helps operators and experts understand and interpret anomalies within PV arrays when making response decisions. However, traditional methods that rely on manual labor and regular data collection are difficult to monitor in real time, resulting in delays in fault detection and localization. Traditional machine learning algorithms are slow and cumbersome in processing data, which affects the operational safety of PV plants. In this paper, we propose a visual analytic approach for detecting and exploring anomalous sequences in a PV sequence dataset via sequence learning. We first compare the sequences with their reconstructions through an unsupervised anomaly detection algorithm (Long Short-Term Memory) based on AutoEncoders to identify anomalies. To further enhance the accuracy of anomaly detection, we integrate the artificial intelligence of things (AIoT) technology with a strict time synchronization data collection and real-time processing algorithm. This integration ensures that data from multiple sensors are synchronized and processed in real time. Then, we analyze the characteristics of the anomalies based on the visual comparison of different PV sequences and explore the potential correlation factors to analyze the possible causes of the anomalies. Case studies based on authentic enterprise datasets demonstrate the effectiveness of our method in the anomaly detection and exploration of PV sequence data.

Evaluation of Deep Isolation Forest (DIF) Algorithm for Mineral Prospectivity Mapping of Polymetallic Deposits

Mineral prospectivity mapping (MPM) is crucial for efficient mineral exploration, where prospective zones are identified in a cost-effective manner. This study focuses on generating prospectivity maps for hydrothermal polymetallic mineralization in the Feizabad area, in northeastern Iran, using unsupervised anomaly detection methods, i.e., isolation forest (IForest) and deep isolation forest (DIF) algorithms. As mineralization events are rare and complex, traditional approaches continue to encounter difficulties, despite advances in MPM. In this respect, unsupervised anomaly detection algorithms, which do not rely on ground truth samples, offer a suitable solution. Here, we compile geospatial datasets on the Feizabad area, which is known for its polymetallic mineralization showings. Fourteen evidence layers were created, based on the geology and mineralization characteristics of the area. Both the IForest and DIF algorithms were employed to identify areas with high mineralization potential. The DIF, which uses neural networks to handle non-linear relationships in high-dimensional data, outperformed the traditional decision tree-based IForest algorithm. The results, evaluated through a success rate curve, demonstrated that the DIF provided more accurate prospectivity maps, effectively capturing complex, non-linear relationships. This highlights the DIF algorithm’s suitability for MPM, offering significant advantages over the IForest algorithm. The present study concludes that the DIF algorithm, and similar unsupervised anomaly detection algorithms, are highly effective for MPM, making them valuable tools for both brownfield and greenfield exploration.

Unsupervised anomaly detection of multivariate time series based on multi-standard fusion

With the wide application of the Internet of Things, industrial systems generate a large amount of multivariate time series data every day. Industrial data often contains a wide variety of anomalies. The existing anomaly detection methods have some shortcomings in dealing with large-scale, high-dimensional, and real-time industrial data. Particularly in the absence of prior knowledge, unsupervised learning methods are easily influenced by noise. Therefore, we propose a new unsupervised anomaly detection algorithm of multivariate time series based on multi-standard fusion (MSAD). Specifically, MSAD first classifies the data by analyzing the data density and sample spacing, thereby converting unsupervised anomaly detection into weakly supervised anomaly detection. Then, based on the characteristics of the data and deep learning technology, the degree of abnormality in the sample is comprehensively measured from four perspectives: holistic, local, time, and deep feature learning. At this stage, MSAD 's ability to learn edge features of data has been improved. Finally, based on MSAD, we design an anomaly detection method for industrial data streams in combination with data filtering techniques. The experimental results show that MSAD can detect anomalies more accurately and show stronger robustness to the influence of noise.

Using Kan Extensions to Motivate the Design of a Surprisingly Effective Unsupervised Linear SVM on the Occupancy Dataset

Recent research has suggested that category theory can provide useful insights into the field of machine learning (ML). One example is improving the connection between an ML problem and the design of a corresponding ML algorithm. A tool from category theory called a Kan extension is used to derive the design of an unsupervised anomaly detection algorithm for a commonly used benchmark, the Occupancy dataset. Achieving an accuracy of 93.5% and an ROCAUC of 0.98, the performance of this algorithm is compared to state-of-the-art anomaly detection algorithms tested on the Occupancy dataset. These initial results demonstrate that category theory can offer new perspectives with which to attack problems, particularly in making more direct connections between the solutions and the problem’s structure.

Unsupervised Data-Driven Approach for Scour Damage Detection in Offshore Monopile Foundations

Scour erosion around monopile foundations significantly impacts the structural integrity and operational performance of offshore wind turbines (OWTs). To deal with this problem, this paper proposes an innovative unsupervised data-driven methodology to detect foundation scour in OWTs. The proposed method employs acceleration data collected from the tower. A numerical model of the NERL 5 MW OWT installed in medium-dense sand is developed using OpenFAST software. The study employs the novelty detection method, where only data obtained from the healthy conditions are used for training and establishing a base model. In addition, the influence of operational conditions, e.g. different wind and wave parameters is also evaluated. The process consists of three steps: (a) data pre-processing to extract time domain features and isolate environmental variations, (b) training an unsupervised anomaly detection algorithm by employing the extracted features and (c) conducting real-time anomaly detection to assess the scour damage. The study results highlight the robustness and efficiency of the proposed method in detecting scour damage around monopiles under different environmental conditions. Keywords: Monopile, scour, unsupervised, Machine learning, damage detection, SHM.

Local outlier factor for anomaly detection in HPCC systems

Local Outlier Factor (LOF) is an unsupervised anomaly detection algorithm that finds anomalies by assessing the local density of a data point relative to its neighborhood. Anomaly detection is the process of finding anomalies in datasets. Anomalies in real-time datasets may indicate critical events like bank frauds, data compromise, network threats, etc. This paper deals with the implementation of the LOF algorithm in the HPCC Systems platform, which is an open-source distributed computing platform for big data analytics. Improved LOF is also proposed which efficiently detects anomalies in datasets rich in duplicates. The impact of varying hyperparameters on the performance of LOF is examined in HPCC Systems. This paper examines the performance of LOF with other algorithms like COF, LoOP, and kNN over several datasets in the HPCC Systems. Additionally, the efficacy of LOF is evaluated across big-data frameworks such as Spark, Hadoop, and HPCC Systems, by comparing their runtime performances.

DWSP-MT: Discovery of workers sensing preferences to match tasks for improving data collection quality in MCS

Mobile CrowdSensing (MCS) is a new sensing paradigm that leverages the widespread use of mobile devices to collect data quickly and enable various key applications in the Internet of Things (IoT). Collecting high-quality data is essential for MCS to design and provide a high-quality service. However, it is difficult for the platform to detect dishonest workers who submit fake data. Therefore, MCS applications face a significant challenge. Although there are a number of studies on truth discovery, these studies mainly failed to consider the sensing preferences of workers, resulting in poor data quality enhancement. To overcome these issues, a Discovery of Workers Sensing Preferences to Match Tasks (DWSP-MT) scheme is proposed to improve the quality of MCS data collection. The DWSP-MT scheme primarily includes three components: 1) Unmanned Aerial Vehicles (UAVs) are employed to sense data as ground truth data and an unsupervised anomaly detection algorithm is adopted to identify the data truth of tasks submitted by unidentified workers. 2) The data truth of unidentified workers in historical tasks is evaluated to calculate their sensing preferences, which are used to identify trusted workers. 3) To improve the quality of data collection, a task assignment model based on the sensing preferences of trusted workers is proposed for the first time, which can ensure that tasks are assigned to their areas of most excellent expertise. At last, two open-access datasets were used to conduct experiments comparing the DWSP-MT scheme with existing truth discovery methods. The outcomes of these experiments indicate that our approach outperforms the state-of-the-art research and improves F1-score by 65.74 %, improves accuracy by 36.62 %, and reduces data bias by 60.61 %

A Holistic review and performance evaluation of unsupervised learning methods for network anomaly detection

Abstract The evolving cyber-attack landscape demands flexible and precise protection for information and networks. Network anomaly detection (NAD) systems play a crucial role in preventing and detecting abnormal activities on the network that may lead to catastrophic outcomes when undetected. This paper aims to provide a comprehensive analysis of NAD using unsupervised learning (UL) methods to evaluate the effectiveness of such systems. The paper presents a detailed overview of several UL techniques, lists the current developments and innovations in UL techniques for network anomaly and intrusion detection, and evaluates 13 unsupervised anomaly detection algorithms empirically on benchmark datasets such as NSL-KDD, UNSW-NB15, and CIC-IDS 2017 to analyze the performance of different classes of UL approaches for NAD systems. This study demonstrates the effectiveness of NAD algorithms, discusses UL approaches' research challenges, and unearths the potential drawbacks in the current network security environment.

Real-time anomaly detection of short-term traffic disruptions in urban areas through adaptive isolation forest

The escalating congestion impacts of short-term traffic disruptions, such as double parking or short-duration work zones, are gaining increased attention. This study introduces an enhanced isolation forest-based unsupervised anomaly detection algorithm to detect short-term traffic disruptions in urban areas. To enable the detection in real-time, a sliding window approach is introduced to allow the streaming data to be read and processed according to the window size. An adaptive threshold that automatically adjusts itself to overcome the limitations of the miss rate on local anomalies is shown to further enhance the model predictions. The proposed algorithm is empirically validated on four study sites in Manhattan, New York City. The experimental results demonstrate that the proposed unsupervised algorithm can effectively detect different types of traffic anomalies including accidents, work zones/road closures, traffic jams, double parking events and police activities. On all sites, the average detection rate is 81.1% for traffic jams and 89.6% for police activities, respectively. For three out of the four sites, the detection rate ranges from 71.4% to 100% for accidents, work zones and double parking. An optimizer using high-pass filter is also presented to further improve off-line detection. The primary advantages of this proposed computationally-efficient method are that its only required data input is travel time information and that it does not need labeled data for training, which make it highly deployable for real-time operational applications and can be easily adopted by other cities.

The Credit Card Anti-fraud Detection Model in the Context of Dynamic Integration Selection Algorithm

Because of its close relationship with information technology, every change in the field of information technology will have an important impact on the means, time and space distribution of the final fraud of the credit card network. The rapid development and popularization of artificial intelligence (AI) software based on Deepfake technology has greatly promoted the intelligent transformation of credit card network fraud. Through the use of advanced hardware and software equipment, criminals have iterated on their own fraud tools, from the use of traditional phone calls and text messages to the use of the latest AI software. Therefore, for the problem of credit card fraud detection with missing data set labels and highly unbalanced category distribution, A credit card approval anomaly detection model DES-HBOS (Dynamic Ensemble Selection based on Histogram-based Outlier Score) is proposed. Firstly, the unsupervised anomaly detection algorithm is used to construct the false label of the training set customer. Then, the customer capability area to be measured is determined, and the classifier performance is evaluated according to Pearson correlation coefficient. Finally, a set of optimal classifiers is selected to integrate the test customers. Experiments on real credit card customer data sets show that DES-HBOS has a higher Recall and can identify more fraudulent customers than other 6 classical anomaly detection models. Comparative experiments were carried out on 4 unbalanced data sets, and the experimental results showed that DES-HBOS had stronger anomaly detection ability than HBOS.

FS-RSDD: Few-Shot Rail Surface Defect Detection with Prototype Learning.

As an important component of the railway system, the surface damage that occurs on the rails due to daily operations can pose significant safety hazards. This paper proposes a simple yet effective rail surface defect detection model, FS-RSDD, for rail surface condition monitoring, which also aims to address the issue of insufficient defect samples faced by previous detection models. The model utilizes a pre-trained model to extract deep features of both normal rail samples and defect samples. Subsequently, an unsupervised learning method is employed to learn feature distributions and obtain a feature prototype memory bank. Using prototype learning techniques, FS-RSDD estimates the probability of a test sample belonging to a defect at each pixel based on the prototype memory bank. This approach overcomes the limitations of deep learning algorithms based on supervised learning techniques, which often suffer from insufficient training samples and low credibility in validation. FS-RSDD achieves high accuracy in defect detection and localization with only a small number of defect samples used for training. Surpassing benchmarked few-shot industrial defect detection algorithms, FS-RSDD achieves an ROC of 95.2% and 99.1% on RSDDS Type-I and Type-II rail defect data, respectively, and is on par with state-of-the-art unsupervised anomaly detection algorithms.

Anomaly detection for streaming data based on grid-clustering and Gaussian distribution

A massive amount of real-time and evolving streaming data are produced from various devices and applications. Anomaly detection is one of the main tasks of streaming data mining with many practical applications. However, without prior knowledge, it is difficult to detect the anomaly accurately and quickly. In this paper, we propose an unsupervised anomaly detection algorithm (GC-ADS), which is based on the idea of grid clustering and Gaussian distribution. Specifically, the data space is first segmented using the grid structure, then the data points are mapped to grids and finally grids are clustered. The anomaly can be preliminarily judged according to the cluster density. To solve the problem that clustering cannot distinguish between noise and anomaly, based on the idea of data similarity and Gaussian distribution, a noise recognition model is designed. In addition, a data filtering model based on grid and sliding window is designed to save memory and retain valid information. The proposed method is compared with the state-of-the-art methods on the Numenta anomaly benchmark. Experimental results indicate that GC-ADS detects anomalies more accurately than other methods with less time cost.

AIDA: Analytic isolation and distance-based anomaly detection algorithm

Many unsupervised anomaly detection algorithms rely on the concept of nearest neighbours to compute the anomaly scores. Such algorithms are popular because there are no assumptions about the data, making them a robust choice for unstructured datasets. However, the number (k) of nearest neighbours, which critically affects the model performance, cannot be tuned in an unsupervised setting. Hence, we propose the new and parameter-free Analytic Isolation and Distance-based Anomaly (AIDA) detection algorithm, that combines the metrics of distance with isolation. Based on AIDA, we also introduce the Tempered Isolation-based eXplanation (TIX) algorithm, which identifies the most relevant features characterizing an outlier, even in large multi-dimensional datasets, improving the overall explainability of the detection mechanism. Both AIDA and TIX are thoroughly tested and compared with state-of-the-art alternatives, proving to be useful additions to the existing set of tools in anomaly detection.

Extension of LoRa Coverage and Integration of an Unsupervised Anomaly Detection Algorithm in an IoT Water Quality Monitoring System

High cost, long-range communication, and anomaly detection issues are associated with IoT systems in water quality monitoring. Therefore, this work proposes a prototype for a water quality monitoring system (IoT-WQMS) based on IoT technologies, which include in the system architecture a LoRa repeater and an anomaly detection algorithm. The system performs the data collection, data storage, anomaly detection, and alarm sending remotely and in real-time for the information to be captured by the multisensor node. The LoRa repeater allowed the spatial coverage of the LoRa communication to extend, making it possible to reach a place where originally there was no coverage with a single LoRa transmitter due to topography and line of sight. The prototype performed well in terms of packet loss rate, transmission time, and sensitivity, extending the long-range wireless communication distance. Indoor multinode testing validation for 29 days of the mean absolute error for average relative errors of water temperature, pH, turbidity, and total dissolved solids (TDS) were 0.65%, 0.30%, and 14.33%, respectively. The anomaly detector identified all erroneous data events due to node sensor recalibration and water recirculation pump failures. The IoT-WQMS increased the reliability of monitoring through the timely identification of any sensor malfunctions and extended the LoRa signal range, which are relevant features in the scope of in situ and real-time water quality monitoring.

Constructing a Meta-Learner for Unsupervised Anomaly Detection

Unsupervised anomaly detection (AD) is critical for a wide range of practical applications, from network security to health and medical tools. Due to the diversity of problems, no single algorithm has been found to be superior for all AD tasks. Choosing an algorithm, otherwise known as the Algorithm Selection Problem (ASP), has been extensively examined in supervised classification problems, through the use of meta-learning and AutoML, however, it has received little attention in unsupervised AD tasks. This research proposes a new meta-learning approach that identifies an appropriate unsupervised AD algorithm given a set of meta-features generated from the unlabelled input dataset. The performance of the proposed meta-learner is superior to the current state of the art solution. In addition, a mixed model statistical analysis has been conducted to examine the impact of the meta-learner components: the meta-model, meta-features, and the base set of AD algorithms, on the overall performance of the meta-learner. The analysis was conducted using more than 10,000 datasets, which is significantly larger than previous studies. Results indicate that a relatively small number of meta-features can be used to identify an appropriate AD algorithm, but the choice of a meta-model in the meta-learner has a considerable impact.

A Survey on Unsupervised Anomaly Detection Algorithms for Industrial Images

In line with the development of Industry 4.0, surface defect detection/anomaly detection becomes a topical subject in the industry field. Improving efficiency as well as saving labor costs has steadily become a matter of great concern in practice, where deep learning-based algorithms perform better than traditional vision inspection methods in recent years. While existing deep learning-based algorithms are biased towards supervised learning, which not only necessitates a huge amount of labeled data and human labor, but also brings about inefficiency and limitations. In contrast, recent research shows that unsupervised learning has great potential in tackling the above disadvantages for visual industrial anomaly detection. In this survey, we summarize current challenges and provide a thorough overview of recently proposed unsupervised algorithms for visual industrial anomaly detection covering five categories, whose innovation points and frameworks are described in detail. Meanwhile, publicly available datasets for industrial anomaly detection are introduced. By comparing different classes of methods, the advantages and disadvantages of anomaly detection algorithms are summarized. Based on the current research framework, we point out the core issue that remains to be resolved and provide further improvement directions. Meanwhile, based on the latest technological trends, we offer insights into future research directions. It is expected to assist both the research community and industry in developing a broader and cross-domain perspective.

Adversarial training of LSTM-ED based anomaly detection for complex time-series in cyber-physical-social systems

• Solving the time-series anomaly detection problem in CPSSs by deep learning approach. • Combining the advantages of LSTM and reconstruction models for time- series anomaly detection problems. • Proposing the adversarial training method to improve the robustness of Encoder-Decoder model. • Avoiding the heavy workload of traditional methods for a priori knowledge and quality labeling. • Conducting the extensive experiments to prove the validity of the method in this paper. With the development and maturity of smart cities, more and more Cyber-Physical-Social Systems (CPSSs) need to monitor a variety of time-series data from sensors and network transmissions to ensure the quality and reliability of the Cyber-Physical-Social Services. Time-series anomaly detection is a common search problem in the field of pattern recognition. Existing approaches and models of anomaly detection have solved the problem of simple smooth time-series and perform ideal recognition performance. However, in real scenarios, complex time-series with non-Gaussian noise and complex data distributions are prevalent. Compared to smooth and simple time-series, complex time-series occur more frequently in real-world settings and are difficult to model and label. To address these challenges, this paper proposes an unsupervised anomaly detection algorithm based on Long Short-Term Memory Encoder-Decoder (LSTM-ED) via an adversarial training method for complex time-series in cyber-physical-social systems with high performance. This is a novel method that incorporates adversarial learning to improve the robustness of encoder-decoder architecture, enabling it to obtain good anomaly detection results for complex time-series. In addition, LSTM is employed as the network unit of encoder-decoder architecture, which is also able to extract temporal correlation in time-series to a greater extent. We have conducted extensive experiments on four datasets from real scenarios and the results show that the accuracy of the proposed adversarial training of LSTM-ED is significantly better than that of the state-of-the-art methods, including other unsupervised methods and traditional supervised methods.

Interpretable Anomaly Detection in Event Sequences via Sequence Matching and Visual Comparison.

Anomaly detection is a common analytical task that aims to identify rare cases that differ from the typical cases that make up the majority of a dataset. When analyzing event sequence data, the task of anomaly detection can be complex because the sequential and temporal nature of such data results in diverse definitions and flexible forms of anomalies. This, in turn, increases the difficulty in interpreting detected anomalies. In this article, we propose a visual analytic approach for detecting anomalous sequences in an event sequence dataset via an unsupervised anomaly detection algorithm based on Variational AutoEncoders. We further compare the anomalous sequences with their reconstructions and with the normal sequences through a sequence matching algorithm to identify event anomalies. A visual analytics system is developed to support interactive exploration and interpretations of anomalies through novel visualization designs that facilitate the comparison between anomalous sequences and normal sequences. Finally, we quantitatively evaluate the performance of our anomaly detection algorithm, demonstrate the effectiveness of our system through case studies, and report feedback collected from study participants.

Using artificial intelligence to detect human errors in nuclear power plants: A case in operation and maintenance

Human error (HE) is an important concern in safety-critical systems such as nuclear power plants (NPPs). HE has played a role in many accidents and outage incidents in NPPs. Despite the increased automation in NPPs, HE remains unavoidable. Hence, the need for HE detection is as important as HE prevention efforts. In NPPs, HE is rather rare. Hence, anomaly detection, a widely used machine learning technique for detecting rare anomalous instances, can be repurposed to detect potential HE. In this study, we develop an unsupervised anomaly detection technique based on generative adversarial networks (GANs) to detect anomalies in manually collected surveillance data in NPPs. More specifically, our GAN is trained to detect mismatches between automatically recorded sensor data and manually collected surveillance data, and hence, identify anomalous instances that can be attributed to HE. We test our GAN on both a real-world dataset and an external dataset obtained from a testbed, and we benchmark our results against state-of-the-art unsupervised anomaly detection algorithms, including one-class support vector machine and isolation forest. Our results show that the proposed GAN provides improved anomaly detection performance. Our study is promising for the future development of artificial intelligence based HE detection systems.

Proposal of a new method for learning of diesel generator sounds and detecting abnormal sounds using an unsupervised deep learning algorithm

This study is to find a method to learn engine sound after the start-up of a diesel generator installed in nuclear power plant with an unsupervised deep learning algorithm (CNN autoencoder) and a new method to predict the failure of a diesel generator using it. In order to learn the sound of a diesel generator with a deep learning algorithm, sound data recorded before and after the start-up of two diesel generators was used. The sound data of 20 min and 2 h were cut into 7 s, and the split sound was converted into a spectrogram image. 1200 and 7200 spectrogram images were created from sound data of 20 min and 2 h, respectively. Using two different deep learning algorithms (CNN autoencoder and binary classification), it was investigated whether the diesel generator post-start sounds were learned as normal. It was possible to accurately determine the post-start sounds as normal and the pre-start sounds as abnormal. It was also confirmed that the deep learning algorithm could detect the virtual abnormal sounds created by mixing the unusual sounds with the post-start sounds. This study showed that the unsupervised anomaly detection algorithm has a good accuracy increased about 3% with comparing to the binary classification algorithm.