Anomaly Detection Techniques Research Articles

An Exhaustive Review on State-of-the-art Techniques for Anomaly Detection on Attributed Networks

Social Network sites are one of the most prominent websites used in almost every facet of life. A social network reflects relationships among social entities, including acquaintances, co-workers, or co-authors. With the extreme success of Social networks, the misuse has also been escalated and unlocked the way for various illegal behavior and security threats. Social Network anomaly detection has become a critical topic to be explored by researchers. The nature of the input data is a significant aspect of an anomaly detection technique. In the field of anomaly detection in the social network, input networks can be classified as static or dynamic, Attributed or unattributed. The number of nodes and the connections between the nodes in static networks would not change with time. Attribute networks are pervasive in various domains and constitute a vital element of modern technological architecture, where node attributes support the topological structure in data exploration. While social networks build up over time, evaluating them as if they had been static is very beneficial. Numerous studies have been done for Anomaly detection, but to the best of our knowledge, research in static attributed anomaly detection has been very limited. This review tries to portray earlier research on detecting anomalies for social static attributed networks and thoroughly discusses state-of-the-art embedding approaches

Anomaly Detection in Resource Constrained Environments With Streaming Data

Isolation Forest (or <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">iForest</i> ) is a well-known technique for anomaly detection. It is, however, a bulky approach that assumes the luxury of large storage space and is also ineffective with dynamic streaming data so common nowadays in varied application domains. In this work, we present the Preprocessed Isolation Forest ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PiForest</i> ) approach for anomaly detection that works well in resource constrained environments and is also effective on streaming data. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PiForest</i> is largely based on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">iForest</i> algorithm and to effectively handle the streaming data includes a pre-processing stage. In the pre-processing stage, Principal Component Analysis (PCA) is first harnessed to significantly reduce the dimension and bulk of the data. Subsequently, the streaming characteristic of the data is handled through a sliding window mechanism that creates sequential blocks of data for systematic processing. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PiForest</i> is able to identify anomalies as effectively as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">iForest</i> and other state-of-the-art anomaly detection techniques but has substantially low storage and prediction complexity. We conduct empirical evaluation of the proposed approach with standard data sets and show that it performs comparably with standard techniques in terms of Area Under the Receiver Operating Characteristic Curve (AUC-ROC) and is able to work with high-dimensional, streaming data. Subsequently, we do a real-world hardware implementation of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PiForest</i> and demonstrate that the approach is realistic and practicable in resource-constrained environments.

Robust feature matching using guided local outlier factor

Matching local features on two or more images is fundamental for many applications in the field of computer vision and pattern recognition. Identifying and rejecting mismatches is an important part in the framework of feature matching, due to the putative correspondences always contaminated by mismatches with the error-prone local feature detectors. In this paper, we introduce a novel method, namely Guided Local Outlier Factor (GLOF) for feature matching with gross mismatches under multi-granularity neighborhood structure-preserving. We first construct a tentative correspondence set by matching multi-features. Then, we identify and remove mismatches. Inspired by the anomaly detection technique, putative correspondences are assigned to a particular score, so abnormal instances, i.e., mismatches can be classified by a user-defined threshold. More specially, the neighborhood preserving guides the local searching procedure. Moreover, to eliminate the fluctuation of the matching results with different sizes of local neighbors, we use the multi-granularity algorithm to average out the deviation. Experimental results demonstrate that the introduced approach is superior to several state-of-the-art methods in terms of mismatch rejection on publicly available datasets.

Damage Detection Using a Graph-based Adaptive Threshold for Modal Strain Energy and Improved Water Strider Algorithm

Damage detection through an inverse optimization problem has been investigated by many researchers. Recently, Modal Strain Energy (MSE) has been utilized as an index (MSEBI) for damage localization that serves to guide the optimization. This guided approach considerably reduces the computational cost and increases the accuracy of optimization. Although this index mostly exhibits an acceptable performance, it fails to find some damaged elements' locations in some cases. The aim of this paper is twofold. Firstly, a Graph-based Adaptive Threshold (GAT) is proposed to identify some of those elements that are not detected by basic MSEBI. GAT relies on the concepts from graph theory and MSE working as a simple anomaly detection technique. Secondly, an Improved version of the Water Strider Algorithm (IWSA) is introduced, applied to the damage detection problems with incomplete modal data and noise-contaminated inputs. Several optimization algorithms, including the newly-established Water Strider Algorithm (WSA), are utilized to test the proposed method. The investigations on several damage detection problems demonstrate the GAT and IWSA's satisfactory performance compared to the previous methods.

Aircraft Fleet Health Monitoring with Anomaly Detection Techniques

Predictive maintenance has received considerable attention in the aviation industry where costs, system availability and reliability are major concerns. In spite of recent advances, effective health monitoring and prognostics for the scheduling of condition-based maintenance operations is still very challenging. The increasing availability of maintenance and operational data along with recent progress made in machine learning has boosted the development of data-driven prognostics and health management (PHM) models. In this paper, we describe the data workflow in place at an airline for the maintenance of an aircraft system and highlight the difficulties related to a proper labelling of the health status of such systems, resulting in a poor suitability of supervised learning techniques. We focus on investigating the feasibility and the potential of semi-supervised anomaly detection methods for the health monitoring of a real aircraft system. Proposed methods are evaluated on large volumes of real sensor data from a cooling unit system on a modern wide body aircraft from a major European airline. For the sake of confidentiality, data has been anonymized and only few technical and operational details about the system had been made available. We trained several deep neural network autoencoder architectures on nominal data and used the anomaly scores to calculate a health indicator. Results suggest that high anomaly scores are correlated with identified failures in the maintenance logs. Also, some situations see an increase in the anomaly score for several flights prior to the system’s failure, which paves a natural way for early fault identification.

A fast MPPT-based anomaly detection and accurate fault diagnosis technique for PV arrays

Catastrophic faults in photovoltaic (PV) arrays, if not detected timely, will significantly reduce the output power and even cause fire risks. Thus, fast and accurate fault detection and diagnosis techniques are of importance to enhance the efficiency, reliability, and safety of PV systems. However, conventional methods fail to detect faults at low mismatch levels and low irradiance levels. In this paper, a sensorless detection technique is thus proposed based on the instantaneous current reduction between two maximum power point tracking (MPPT) sampling instants. Simulations have been conducted to validate its availability to identify anomalies under varying environments, irrespective of the mismatch and irradiance levels. The results clearly demonstrate that the proposed method is less time-consuming but more accurate than the existing techniques. However, due to the presence of blocking diodes, the similarity between faults and certain partial shading (PS) conditions will make them more difficult to be classified. Hence, a unique point on the current–voltage (I-V) curve, namely inflection point, is exploited to accurately distinguish PS conditions, and further determine the mismatch level. More simulation tests under different conditions reveal that the proposed method is effective to quantitatively evaluate the faults. Accordingly, different voltages can be set to differentiate the currents in the faulty string and normal ones as much as possible for efficient fault location.

Indirectly-Supervised Anomaly Detection of Clinically-Meaningful Health Events from Smart Home Data.

Anomaly detection techniques can extract a wealth of information about unusual events. Unfortunately, these methods yield an abundance of findings that are not of interest, obscuring relevant anomalies. In this work, we improve upon traditional anomaly detection methods by introducing Isudra, an Indirectly-Supervised Detector of Relevant Anomalies from time series data. Isudra employs Bayesian optimization to select time scales, features, base detector algorithms, and algorithm hyperparameters that increase true positive and decrease false positive detection. This optimization is driven by a small amount of example anomalies, driving an indirectly-supervised approach to anomaly detection. Additionally, we enhance the approach by introducing a warm start method that reduces optimization time between similar problems. We validate the feasibility of Isudra to detect clinically-relevant behavior anomalies from over 2 million sensor readings collected in 5 smart homes, reflecting 26 health events. Results indicate that indirectly-supervised anomaly detection outperforms both supervised and unsupervised algorithms at detecting instances of health-related anomalies such as falls, nocturia, depression, and weakness.

Application of Unsupervised Anomaly Detection Techniques to Moisture Content Data from Wood Constructions

Wood is considered one of the most important construction materials, as well as a natural material prone to degradation, with fungi being the main reason for wood failure in a temperate climate. Visual inspection of wood or other approaches for monitoring are time-consuming, and the incipient stages of decay are not always visible. Thus, visual decay detection and such manual monitoring could be replaced by automated real-time monitoring systems. The capabilities of such systems can range from simple monitoring, periodically reporting data, to the automatic detection of anomalous measurements that may happen due to various environmental or technical reasons. In this paper, we explore the application of Unsupervised Anomaly Detection (UAD) techniques to wood Moisture Content (MC) data. Specifically, data were obtained from a wood construction that was monitored for four years using sensors at different positions. Our experimental results prove the validity of these techniques to detect both artificial and real anomalies in MC signals, encouraging further research to enable their deployment in real use cases.

A statistical approach for the automatic identification of the start of the chain of events leading to the disruptions at JET

This paper reports an algorithm to automatically identify the chain of events leading to a disruption, evaluating the so-called reference warning time. This time separates the plasma current flat-top of each disrupted discharge into two parts: a non-disrupted part and a pre-disrupted one. The algorithm can be framed into the anomaly detection techniques as it aims to detect the off-normal behavior of the plasma. It is based on a statistical analysis of a set of dimensionless plasma parameters computed for a selection of discharges from the JET experimental campaigns. In every data-driven model, such as the generative topographic mapping (GTM) predictor proposed in this paper, it is indeed necessary to label the samples needed for training the model itself. The samples describing the disruption-free behavior are extracted from the plasma current flat-top phase of the regularly terminated discharges. The disrupted space is described by all the samples belonging to the pre-disruptive phase of each disruptive discharge in the training set. Note that a proper selection of the pre-disruptive phase plays a key role in the prediction performance of the model. Moreover, these models, which are highly dependent on the training input space, may be particularly prone to degradation as the operational space of any experimental machine is continuously evolving. Hence, a regular schedule of model review and retrain must be planned. The proposed algorithm avoids the cumbersome and time-consuming manual identification of the warning times, helping to implement a continuous learning system that could be automated, despite being offline. In this paper, the automatically evaluated warning times are compared with those obtained with a manual analysis in terms of the impact on the mapping of the JET input parameter space using the GTM methodology. Moreover, the algorithm has been used to build the GTM of recent experimental campaigns, with promising results.

A Combination Techniques of Intrusion Prevention and Detection for Cloud Computing

Cloud computing provides scalable, on-demand, and highly available computing services over the Internet to both the public and organizations on a pay-per-use basis. It provides a variety of services such as networking, storage space, and applications. The key issue for cloud computing is ensuring the confidentiality and privacy of cloud resources and data. Enticing the user to purchase cloud services requires their trust which cannot be guaranteed unless the infrastructure is effectively protected because attacks at this level will threaten the whole system. To this end, we propose the Integrated Intrusion Prevention and Detection System (IIPDS) to prevent and detect different types of attacks to the infrastructure level of the cloud system. The proposed design uses Trusted Third Party (TTP) services and SSL/TLS protocols as intrusion prevention methods to secure the communication between the cloud provider and the user. It also uses multiple Intrusion Detection Systems (IDS) distributed over multiple cloud regions. The IDS system is capable of detecting known and unknown attacks using anomaly and rule-based (signature) intrusion detection techniques. The simulation results proved the efficiency of the system in detecting a wide range of attacks with low false positive alerts and low computational overhead.

Automatic Detection of Anomalies in Video Surveillance using Artificial Intelligence

The significance of security in day to day life is increasing, and hence the use of a video surveillance system is becoming commonly accepted in almost every public places. Even though by placing surveillance system in public places will help to trace out the culprits of the anomalous situation, it is hard to trace it out. The knowledge of the incidence time is needed to identify the event, and also, it requires an enormous data handling task. To identify the anomalous situation in real-time will help the authorities to decrease the consequences and loss during the anomalous event. The paper proposes an in-depth study of various automatic anomaly detection techniques which helps to reduce the loss occurred of the anomalous situation. The advancements in Artificial Intelligence help in the quick and automatic identification of nominal and anomalous events. The sequential and incremental learning approach in feature extraction will help to generate a model that will provide much accurate classifications and predictions of anomalies.

Applications of Generative Adversarial Networks in Anomaly Detection: A Systematic Literature Review

Anomaly detection has become an indispensable tool for modern society, applied in a wide range of applications, from detecting fraudulent transactions to malignant brain tumors. Over time, many anomaly detection techniques have been introduced. However, in general, they all suffer from the same problem: lack of data that represents anomalous behaviour. As anomalous behaviour is usually costly (or dangerous) for a system, it is difficult to gather enough data that represents such behaviour. This, in turn, makes it difficult to develop and evaluate anomaly detection techniques. Recently, generative adversarial networks (GANs) have attracted much attention in anomaly detection research, due to their unique ability to generate new data. In this paper, we present a systematic review of the literature in this area, covering 128 papers. The goal of this review paper is to analyze the relation between anomaly detection techniques and types of GANs, to identify the most common application domains for GAN-assisted and GAN-based anomaly detection, and to assemble information on datasets and performance metrics used to assess them. Our study helps researchers and practitioners to find the most suitable GAN-assisted anomaly detection technique for their application. In addition, we present a research roadmap for future studies in this area. In summary, GANs are used in anomaly detection to address the problem of insufficient amount of data for the anomalous behaviour, either through data augmentation or representation learning. The most commonly used GAN architectures are DCGANs, standard GANs, and cGANs. The primary application domains include medicine, surveillance and intrusion detection.

A Novel, Efficient, and Secure Anomaly Detection Technique Using DWU‐ODBN for IoT‐Enabled Multimedia Communication Systems

The Internet of Things (IoT) is enhancing our lives in a variety of structures, which consists of smarter cities, agribusiness, and e‐healthcare, among others. Even though the Internet of Things has many features with the consumer Internet of Things, the open nature of smart devices and their worldwide connection make IoT networks vulnerable to a variety of assaults. Several approaches focused on attack detection in Internet of Things devices, which has the longest calculation times and the lowest accuracy issues. It is proposed in this paper that an attack detection framework for Internet of Things devices, based on the DWU‐ODBN method, be developed to alleviate the existing problems. At the end of the process, the proposed method is used to identify the source of the assault. It comprises steps such as preprocessing, feature extraction, feature selection, and classification to identify the source of the attack. A random oversampler is used to preprocess the input data by dealing with NaN values, categorical features, missing values, and unbalanced datasets before being used to deal with the imbalanced dataset. When the data has been preprocessed, it is then sent to the MAD Median‐KS test method, which is used to extract features from the dataset. To categorize the data into attack and nonattack categories, the features are classified using the dual weight updation‐based optimal deep belief network (DWU‐ODBN) classification technique, which is explained in more detail below. According to the results of the experimental assessment, the proposed approach outperforms existing methods in terms of detecting intrusions and assaults. The proposed work achieves 77 seconds to achieve the attack detection with an accuracy rate of 98.1%.

Anomaly Detection in Road Traffic Using Visual Surveillance

Computer vision has evolved in the last decade as a key technology for numerous applications replacing human supervision. Timely detection of traffic violations and abnormal behavior of pedestrians at public places through computer vision and visual surveillance can be highly effective for maintaining traffic order in cities. However, despite a handful of computer vision–based techniques proposed in recent times to understand the traffic violations or other types of on-road anomalies, no methodological survey is available that provides a detailed insight into the classification techniques, learning methods, datasets, and application contexts. Thus, this study aims to investigate the recent visual surveillance–related research on anomaly detection in public places, particularly on road. The study analyzes various vision-guided anomaly detection techniques using a generic framework such that the key technical components can be easily understood. Our survey includes definitions of related terminologies and concepts, judicious classifications of the vision-guided anomaly detection approaches, detailed analysis of anomaly detection methods including deep learning–based methods, descriptions of the relevant datasets with environmental conditions, and types of anomalies. The study also reveals vital gaps in the available datasets and anomaly detection capability in various contexts, and thus gives future directions to the computer vision–guided anomaly detection research. As anomaly detection is an important step in automatic road traffic surveillance, this survey can be a useful resource for interested researchers working on solving various issues of Intelligent Transportation Systems (ITS).

Comparison of New Anomaly Detection Technique for Wind Turbine Condition Monitoring Using Gearbox SCADA Data

Anomaly detection for wind turbine condition monitoring is an active area of research within the wind energy operations and maintenance (O &amp; M) community. In this paper three models were compared for multi-megawatt operational wind turbine SCADA data. The models used for comparison were One-Class Support Vector Machine (OCSVM), Isolation Forest (IF), and Elliptical Envelope (EE). Each of these were compared for the same fault, and tested under various different data configurations. IF and EE have not previously been used for fault detection for wind turbines, and OCSVM has not been used for SCADA data. This paper presents a novel method of condition monitoring that only requires two months of data per turbine. These months were separated by a year, the first being healthy and the second unhealthy. The number of anomalies is compared, with a greater number in the unhealthy month being considered correct. It was found that for accuracy IF and OCSVM had similar performances in both training regimes presented. OCSVM performed better for generic training, and IF performed better for specific training. Overall, IF and OCSVM had an average accuracy of 82% for all configurations considered, compared to 77% for EE.

The trace kernel bandwidth criterion for support vector data description

Support vector data description (SVDD) is a popular anomaly detection technique. The computation of the SVDD classifier requires a kernel function, for which the Gaussian kernel is a common choice. The Gaussian kernel has a bandwidth parameter, and it is important to set the value of this parameter correctly to ensure good results. A small bandwidth leads to overfitting, and the resulting SVDD classifier overestimates the number of anomalies, whereas a large bandwidth leads to underfitting and an inability to detect many anomalies. In this paper, we present a new, unsupervised method for selecting the Gaussian kernel bandwidth. Our method exploits a low-rank representation of the kernel matrix to suggest a kernel bandwidth value. Our new technique is competitive with the current state of the art for low-dimensional data and performs extremely well for many classes of high-dimensional data. This method is also applicable to one-class support vector machines (OCSVM).

Performance of signal processing techniques for anomaly detection using a temperature-based measurement interpretation approach

This study investigates the effectiveness of four signal processing techniques in supporting a data-driven strategy for anomaly detection that relies on correlations between measurements of bridge response and temperature distributions. The strategy builds upon the regression-based thermal response prediction methodology which was developed by the authors to accurately predict thermal response from distributed temperature measurements. The four techniques that are investigated as part of the strategy are moving fast Fourier transform, moving principal component analysis, signal subtraction method and cointegration method. The techniques are compared on measurement time histories from a laboratory structure and a footbridge at the National Physical Laboratory. Results demonstrate that anomaly events can be detected successfully depending on the magnitude and duration of the event and the choice of an appropriate anomaly detection technique.

Authorisation, attack detection and avoidance framework for IoT devices

Internet of Things (IoT) involve large volumes of data generated from the interactions between devices and people, and security is a main alarm in IoT. Most of the anomaly detection techniques in IoT use supervised machine learning technique which involve huge overhead and high false positives. It is observed that severity of attack response was not considered. In this study, the authors propose to develop an authorisation, attack detection and avoidance framework for IoT devices. Initially, traffic collection agent continuously gathers packet level and flow level information for a given time interval. Then detection agent (DA) first checks the collected information with the attack rules table. If any matching attack pattern is found, it informs the attack type to response agent (RA). On the other hand, if no matching pattern is found, then the classification agent applies multi-class support vector machine algorithm. Once the RA obtains the attack type from DA, then it estimates the severity of attack by computing the attack frequency over different time windows and appropriate action will be performed. Experimental results show that the proposed framework reduces 13% of unauthorised access and 19% false positive rate thereby increasing the detection accuracy by 0.6% and throughput.

A Hybrid Deep Learning Framework for Unsupervised Anomaly Detection in Multivariate Spatio-Temporal Data

Multivariate time-series data with a contextual spatial attribute have extensive use for finding anomalous patterns in a wide variety of application domains such as earth science, hurricane tracking, fraud, and disease outbreak detection. In most settings, spatial context is often expressed in terms of ZIP code or region coordinates such as latitude and longitude. However, traditional anomaly detection techniques cannot handle more than one contextual attribute in a unified way. In this paper, a new hybrid approach based on deep learning is proposed to solve the anomaly detection problem in multivariate spatio-temporal dataset. It works under the assumption that no prior knowledge about the dataset and anomalies are available. The architecture of the proposed hybrid framework is based on an autoencoder scheme, and it is more efficient in extracting features from the spatio-temporal multivariate datasets compared to the traditional spatio-temporal anomaly detection techniques. We conducted extensive experiments using buoy data of 2005 from National Data Buoy Center and Hurricane Katrina as ground truth. Experiments demonstrate that the proposed model achieves more than 10% improvement in accuracy over the methods used in the comparison where our model jointly processes the spatial and temporal dimensions of the contextual data to extract features for anomaly detection.

Robustness of Short-Term Wind Power Forecasting against False Data Injection Attacks

The accuracy of wind power forecasting depends a great deal on the data quality, which is so susceptible to cybersecurity attacks. In this paper, we study the cybersecurity issue of short-term wind power forecasting. We present one class of data attacks, called false data injection attacks, against wind power deterministic and probabilistic forecasting. We show that any malicious data can be injected to historical data without being discovered by one of the commonly-used anomaly detection techniques. Moreover, we testify that attackers can launch such data attacks even with limited resources. To study the impact of data attacks on the forecasting accuracy, we establish the framework of simulating false data injection attacks using the Monte Carlo method. Then, the robustness of six representative wind power forecasting models is tested. Numerical results on real-world data demonstrate that the support vector machine and k-nearest neighbors combined with kernel density estimator are the most robust deterministic and probabilistic forecasting ones among six representative models, respectively. Nevertheless, none of them can issue accurate forecasts under very strong false data attacks. This presents a serious challenge to the community of wind power forecasting. The challenge is to study robust wind power forecasting models dealing with false data attacks.