Anomaly Detection Research Articles

An innovative framework to securely transfer data through the Internet of Things using advanced generative adversarial networks

The Internet of Things (IoT) is developing so quickly that cloud-centric computing finds it challenging to keep up with the demands of usability and low latency. Edge computing unifies computing, networking, storage and applications into a distributed open system. At the edge of the IoT, it provides intelligent services. The edge network is made up of several wired and wireless networks, and edge nodes have constrained amounts of memory and processing power. The edge network is vulnerable to several types of cyberattacks because of these factors. Large-scale network data gathering and detection for IoT security is also challenging for an IoT edge node to provide. Data analytics for intrusion detection guarantees high accuracy of intrusion detection systems (IDSs), but the implementation of such algorithms on IoT might be a challenge due to the limited resources on edge nodes. Inspired in part by these challenges, we suggest a sophisticated IDS based on a generative adversarial network (GAN). This article suggests a novel method for detecting intrusions in the IoTs networks that make use of Colony Predator Algorithm (CPA) for optimization of the detection process. Through the use of GANs to create real-looking data samples, the suggested technique helps to have reliable anomaly and intrusion detection. The integration improves the training process, making it more efficient and enabling the CPA system to better differentiate between benign and malicious activities, which exponentially raises the system's efficiency.

Remaining Useful Life Prediction Combining Advanced Anomaly Detection and Graph Isomorphic Network

Remaining Useful Life Prediction Combining Advanced Anomaly Detection and Graph Isomorphic Network

SGFM: Conditional Flow Matching for Time Series Anomaly Detection With State Space Models

SGFM: Conditional Flow Matching for Time Series Anomaly Detection With State Space Models

EAD: effortless anomalies detection, a deep learning based approach for detecting outliers in English textual data

Anomalies are the existential abnormalities in data, the identification of which is known as anomaly detection. The absence of timely detection of anomalies may affect the key processes of decision-making, fraud detection, and automated classification. Most of the existing models of anomaly detection utilize the traditional way of tokenizing and are computationally costlier, mainly if the outliers are to be extracted from a large script. This research work intends to propose an unsupervised, all-MiniLM-L6-v2-based system for the detection of outliers. The method makes use of centroid embeddings to extract outliers in high-variety, large-volume data. To avoid mistakenly treating novelty as an outlier, the Minimum Covariance Determinant (MCD) based approach is followed to count the novelty of the input script. The proposed method is implemented in a Python project, App. for Anomalies Detection (AAD). The system is evaluated by two non-related datasets-the 20 newsgroups text dataset and the SMS spam collection dataset. The robust accuracy (94%) and F1 score (0.95) revealed that the proposed method could effectively trace anomalies in a comparatively large script. The process is applicable in extracting meanings from textual data, particularly in the domains of human resource management and security.

Fractals as Pre-Training Datasets for Anomaly Detection and Localization

Anomaly detection is crucial in large-scale industrial manufacturing as it helps to detect and localize defective parts. Pre-training feature extractors on large-scale datasets is a popular approach for this task. Stringent data security, privacy regulations, high costs, and long acquisition time hinder the development of large-scale datasets for training and benchmarking. Despite recent work focusing primarily on the development of new anomaly detection methods based on such extractors, not much attention has been paid to the importance of the data used for pre-training. This study compares representative models pre-trained with fractal images against those pre-trained with ImageNet, without subsequent task-specific fine-tuning. We evaluated the performance of eleven state-of-the-art methods on MVTecAD, MVTec LOCO AD, and VisA, well-known benchmark datasets inspired by real-world industrial inspection scenarios. Further, we propose a novel method to create a dataset by combining the dynamically generated fractal images creating a “Multi-Formula” dataset. Even though pre-training with ImageNet leads to better results, fractals can achieve close performance to ImageNet under proper parametrization. This opens up the possibility for a new research direction where feature extractors could be trained on synthetically generated abstract datasets mitigating the ever-increasing demand for data in machine learning while circumventing privacy and security concerns.

Anomaly Detection and Analysis in Nuclear Power Plants

Industries are increasingly adopting digital systems to improve control and accessibility by providing real-time monitoring and early alerts for potential issues. While digital transformation fuels exponential growth, it exposes these industries to cyberattacks. For critical sectors such as nuclear power plants, a cyberattack not only risks damaging the facility but also endangers human lives. In today’s digital world, enormous amounts of data are generated, and the analysis of these data can help ensure effectiveness, including security. In this study, we analyzed the data using a deep learning model for early detection of abnormal behavior. We first examined the Asherah Nuclear Power Plant simulator by initiating three different cyberattacks, each targeting a different system, thereby collecting and analyzing data from the simulator. Second, a Bi-LSTM model was used to detect anomalies in the simulator, which detected it before the plant’s protection system was activated in response to a threat. Finally, we applied explainable AI (XAI) to acquire insight into how distinctive features contribute to the detection of anomalies. XAI provides valuable explanations of model behavior by revealing how specific features influence anomaly detection during attacks. This research proposes an effective anomaly detection technique and interpretability to better understand counter-cyber threats in critical industries, such as nuclear plants.

Deep Learning‐Based Anomaly Diagnosis Method for Capacitive Voltage Transformer Measurement Error in PIoT

ABSTRACTIn the era of power Internet of Things (PIoT), the accuracy of capacitive voltage transformers (CVTs) is crucial for maintaining the reliability of voltage measurement and protection systems in smart grids, thereby contributing to overall grid stability and efficiency. Accurate and timely detection of anomalies in CVT measurement errors is essential for preventing equipment failures, reducing maintenance costs, and improving overall system reliability. However, existing anomaly diagnosis methods often rely on statistical analysis and rule‐based approaches, which have limitations in capturing complex patterns and adapting to evolving anomaly types. This paper proposes a novel deep learning‐based anomaly diagnosis method for CVT measurement error in PIoT, called LSTM‐CVT. The proposed method leverages a long short‐term memory (LSTM) neural network architecture with three key strategies: bidirectional temporal dependency capture, hierarchical feature learning, and joint anomaly diagnosis and error estimation. The experimental results demonstrate the superior performance of LSTM‐CVT compared to state‐of‐the‐art baseline methods.

Leveraging Large Language Models for Efficient Alert Aggregation in AIOPs

Alerts are an essential tool for the detection of anomalies and ensuring the smooth operation of online service systems by promptly notifying engineers of potential issues. However, the increasing scale and complexity of IT infrastructure often result in “alert storms” during system failures, overwhelming engineers with a deluge of often correlated alerts. Therefore, effective alert aggregation is crucial in isolating root causes and accelerating failure resolution. Existing approaches typically rely on either semantic similarity or statistical methods, both of which have significant limitations, such as ignoring causal relationships or struggling to handle infrequent alerts. To overcome these drawbacks, we propose a novel two-phase alert aggregation approach. We employ temporal–spatial clustering to group alerts based on their temporal proximity and spatial attributes. In the second phase, we utilize large language models to trace the cascading effects of service failures and aggregate alerts that share the same root cause. Experimental evaluations on datasets from real-world cloud platforms demonstrate the effectiveness of our method, achieving superior performance compared to traditional aggregation techniques.

Towards accurate anomaly detection for cloud system via graph-enhanced contrastive learning

Towards accurate anomaly detection for cloud system via graph-enhanced contrastive learning

Assessment of Prenatal Transportation Stress and Sex on Gene Expression Within the Amygdala of Brahman Calves

As the amygdala is associated with fear and anxiety, it is important to determine the potential effects of gestational stressors on behavior and stress responses in offspring. The objective of this study was to investigate the effects of prenatal transportation stress on amygdala gene expression in 25-day-old Brahman calves, focusing on sex-specific differences. Amygdala tissue samples from prenatally stressed (PNS) and control bull and heifer calves were analyzed using RNA sequencing. A thorough outlier detection process, utilizing visual inspection of multidimensional scaling plots, robust principal component analysis, and PCAGrid methods, led to the exclusion of 5 of 32 samples from subsequent analyses. Differential expression analysis revealed no significant treatment differences between the control and PNS groups within either sex. However, sex-specific differences in gene expression were identified in both the control and PNS groups. The control group showed seven differentially expressed genes between sexes, while ten were identified between PNS males and females, with seven located on the X chromosome. Among these was the ubiquitin-specific peptidase 9 X-linked gene, which plays a role in neurodevelopmental pathways. When comparing males to females, regardless of treatment, a total of 58 genes were differentially expressed, with 45 showing increased expression in females. Gene enrichment analysis indicated that many differentially expressed genes are associated with infectious disease-related pathways. Future research should explore amygdala size and functional responses to various postnatal stimuli.

Pseudo Replay-based Class Continual Learning for Online New Category Anomaly Detection in Advanced Manufacturing

The incorporation of advanced sensors and machine learning techniques has enabled modern manufacturing enterprises to perform data-driven classification-based anomaly detection based on the sensor data collected in manufacturing processes. However, one critical challenge is that newly presented defect category may manifest as the manufacturing process continues, resulting in monitoring performance deterioration of previously trained machine learning models. Hence, there is an increasing need for empowering machine learning models to learn continually. Among all continual learning methods, memory-based continual learning has the best performance but faces the constraints of data storage capacity. To address this issue, this paper develops a novel pseudo replay-based continual learning framework by integrating class incremental learning and oversampling-based data generation. Without storing all the data, the developed framework could generate high-quality data representing previous classes to train machine learning model incrementally when new category anomaly occurs. In addition, it could even enhance the monitoring performance since it also effectively improves the data quality. The effectiveness of the proposed framework is validated in three cases studies, which leverages supervised classification problem for anomaly detection. The experimental results show that the developed method is very promising in detecting novel anomaly while maintaining a good performance on the previous task and brings up more flexibility in model architecture.

Prediction of cardiovascular events after carotid endarterectomy using pathological images and clinical data.

Carotid endarterectomy (CEA) is a surgical treatment for carotid artery stenosis. After CEA, some patients experience cardiovascular events (myocardial infarction, stroke, etc.); however, the prognostic factor has yet to be revealed. Therefore, this study explores the predictive factors in pathological images and predicts cardiovascular events within one year after CEA using pathological images of carotid plaques and patients' clinical data. This paper proposes a two-step method to predict the prognosis of CEA patients. The proposed method first computes the pathological risk score using an anomaly detection model trained using pathological images of patients without cardiovascular events. By concatenating the obtained image-based risk score with a patient's clinical data, a statistical machine learning-based classifier predicts the patient's prognosis. We evaluate the proposed method on a dataset containing 120 patients without cardiovascular events and 21 patients with events. The combination of autoencoder as the anomaly detection model and XGBoost as the classification model obtained the best results: area under the receiver operating characteristic curve, accuracy, sensitivity, specificity, and F1-score were 81.9%, 84.1%, 79.1%, 86.3%, and 76.6%, respectively. These values were superior to those obtained using pathological images or clinical data alone. We showed the feasibility of predicting CEA patient's long-term prognosis using pathological images and clinical data. Our results revealed some histopathological features related to cardiovascular events: plaque hemorrhage (thrombus), lymphocytic infiltration, and hemosiderin deposition, which will contribute to developing preventive treatment methods for plaque development and progression.

Fast and explainable outlier detection in geotechnical database

ABSTRACT Outlier detection is an important task in data-driven geotechnics. It is noted that many existing methods for outlier detection (e.g. Bayesian learning, neural networks) may pose computational challenge for conventional geotechnical practitioners. Towards this aspect, this study proposes a simple, fast, and explainable method for detecting outliers in geotechnical database. The principle of the method is that for a data point to be labelled as a potential outlier, the probability of observing a data point as extreme as that value should be low. To account for outliers in left-tail and right-tail, the skewness of the dataset is incorporated, and an indicator referred to as outlier score ( > 0 ) is assigned to each data point. The method also provides another indicator (referred to as dimensional outlier score) to identify which dimensions/soil properties contribute to the outlierness; hence, it is explainable. The method doesn't require any time-consuming learning or sampling procedures; hence, it is quite fast and practitioner-friendly. Multiple numerical examples are utilised to demonstrate the capability first. Finally, four publicly available geotechnical databases are utilised to demonstrate the outlier detection task. The results suggest that the outliers identified using the proposed method can be meaningful from a geotechnical point of view.

Machine Learning-Based Financial Big Data Analysis and Forecasting: From Preprocessing to Deep Learning Models

Abstract: Machine learning technology has revolutionized the financial sector by allowing faster and more accurate analysis and forecasting of large-scale financial data. This paper focuses on how machine learning ( ML ), especially deep learning models, can help to deal with high dimensional, noisy, and non-stationary financial data. Essential methodologies such as data preprocessing, feature engineering, and dimensionality reduction are imperative for preparing the raw financial data to ML algorithms. Methods such as outlier detection, normalisation for preprocessing, and feature (variable) selection for dimensionality reduction improve the models accuracy and efficiency. The paper also examines how deep learning models, such as Recurrent Neural Networks ( RNN ) and Long Short-Term Memory ( LSTM ) networks, can overcome the issues of autoregressive integrated moving average ( ARIMA ) models for financial time series prediction. An in-depth comparison of the machine learning models, ranging from supervised to unsupervised methods, is also provided to discuss their pros and cons in the financial domain, including popular applications such as credit scoring, fraud detection, and market risk prediction. The study finally concludes by discussing how optimisation methods such as hyperparameter tuning and cross-validation are imperative for ML models in complex financial scenarios to ensure their generalisation capability and avoid overfitting.

ML-based Anomaly Detection for CAN Bus Network in Agriculture Machinery

The adoption of advanced automation and next-generation technologies like the Internet of Things (IoT) and modern communication networks has revolutionized the food and agriculture sector, boosting the efficiency and precision of farm machinery. However, this increased inter-connectivity has also exposed significant vulnerabilities, particularly in Controller Area Network (CAN) protocols, widely used in advanced agricultural machinery and equipment. Due to its lack of inherent security features, CAN is susceptible to various cyber-attacks, potentially leading to severe consequences if these attacks remain undetected and unmitigated. This paper introduces a supervised machine learning (ML)-based anomaly detection system (CAN-ADS) designed to detect various cyber-attacks on CAN-based agricultural machinery. The system leverages network traffic augmentation and data balancing techniques to train ML algorithms on CAN-specific datasets. Experimental results show that CAN-ADS achieves high accuracy (approx 98%) and true-positive rates with low false-negative rates (approx 1%).

Increase of Reliability of Anomalies Detection on Images at Formation of Their Feature Vectors in Wavelet Bases

The method of detection of life rafts and lifeboats in the water area of seas and oceans after shipwrecks based on the recognition of anomalies in the processed images, which increases the probability of recognition of monitoring objects, is proposed. The approach to solving such a problem is substantiated. The formulation of the problem of object recognition from the perspective of binary classification in the detection of anomalies is presented. The analytical expression for the decision-making algorithm is obtained. The possibility of formalization of image matrices in the form of histograms of color (brightness) intensity distributions is considered. The contrast of the feature space on their basis is estimated. It is suggested that the contrast of feature spaces be increased due to the secondary processing of histograms of distributions in the basis of multiple-scale wavelet decomposition. The possibility of realization of wavelet transformations on the basis of Haar functions and Gauss wavelets of the 1st and 2nd orders is considered. The mechanism of formation of secondary feature vectors from three-dimensional wavelet transforms by averaging their coefficients along the time shift axis is substantiated. It is shown that at the same dimensionality of histograms of brightness distribution with newly formed feature vectors, the latter provide higher contrast of feature spaces. It is recommended to use a Gaussian wavelet of the 2nd order for the formalization of images in jpeg format, which provides, other things being equal, a greater magnitude of differences for images containing anomalies. An approach to probabilistic evaluation of the algorithm for automatic image recognition is developed. The analytical expression is obtained and its constituent elements are justified. Graphical dependences of the probability of correct detection (recognition) of anomalies, depending on the size in relation to the total area of the frame and the dispersion of the underlying background are given. The results of the experiment on image recognition with a lifeboat in the ocean water area are presented. The directions of further research are defined.

Revisiting streaming anomaly detection: benchmark and evaluation

Anomaly detection in streaming data is an important task for many real-world applications, such as network security, fraud detection, and system monitoring. However, streaming data often exhibit concept drift, which means that the data distribution changes over time. This poses a significant challenge for many anomaly detection algorithms, as they need to adapt to the evolving data to maintain high detection accuracy. Existing streaming anomaly detection algorithms lack a unified evaluation framework that validly assesses their performance and robustness under different types of concept drifts and anomalies. In this paper, we conduct a systematic technical review of the state-of-the-art methods for anomaly detection in streaming data. We propose a new data generator, called SCAR (Streaming data generator with Customizable Anomalies and concept dRifts), that can synthesize streaming data based on synthetic and real-world datasets from different domains. Furthermore, we adapt four static anomaly detection models to the streaming setting using a generic reconstruction strategy as baselines, and then compare them systematically with 9 existing streaming anomaly detection algorithms on 76 synthesized datasets that have various types of anomalies and concept drifts. The challenges and future research directions for anomaly detection in streaming data are also presented.

Overview of Anomaly Detection Techniques across Different Domains: A Systematic Review

An anomaly, defined as something that deviates from what is normal, expected, or usual. It signifies abnormality or an irregularity that stands out from typical behaviours or patterns. Detecting anomalies is significant among numerous sectors due to the reasons of signal potential difficulties or opportunities. For an instance, in retail, detecting anomalies in sales data might prompt for further analysis into operational issues or customer behaviour to reduce losses and capitalize on its trends. Hence, different techniques are used for Anomaly Detection. However, anomaly detection using manual method are measured for time consuming, prone to error and can be tedious process. Therefore, different approaches have been considered for anomaly detection as AI (Artificial Intelligence) methods are efficient, faster, provides high level accuracy by effectively detecting the abnormalities. Owing to these aspects, this paper focuses on compiling different techniques and emphasizes on reviewing all anomaly detection using numerous techniques like ML (Machine Learning) and DL (Deep Learning) classifiers, statistical methods, one-class classification, clustering and density-based models which helps with identifying and comprehending the diversity of detection techniques that are applied in various domains like finance, retail, healthcare and cyber security. Various existing researches on anomaly detection are reviewed in the study. In addition to an overview, certain studies also deals with applications of detection models and future trends are reviewed in precise. Finally, the challenges are identified through the analysis of existing researchers and future recommendations are provided for overcoming the gaps that are intended to create promising work in this area.

Research of anomaly detection based on dynamic anomaly detection enhancement framework

Abstract Anomaly detection plays a crucial role in various fields, from industrial defect inspection to geological detection. However, traditional approaches often struggle with insufficient discriminability and an inability to generalize to unseen anomalies. These limitations stem from the practical difficulty in gathering a comprehensive set of anomalies and the tendency to overlook anomalous instances in favor of normal samples. To address these challenges, we propose a novel Dynamic Anomaly Detection Enhancement Framework, integrating three key innovations: (1) SaliencyAug: An adaptive saliency-guided augmentation method that generates realistic pseudo-samples to enhance learning of rare anomalies, improving model generalization. (2) DynAB: A dynamic attention block that achieves effective multi-level feature fusion while minimizing redundant information, enhancing detection accuracy. (3) DualOM: A dual-head optimization module which employs separate heads for normal and anomalous sample learning, creating more explicit and discriminative decision boundaries. Extensive experiments across multiple real-world datasets demonstrate our framework's superior performance in detecting a wide range of anomalies, demonstrating 2.4% improvement over state-of-the-art methods.

Advanced Machine Learning Algorithms for Predictive Maintenance in Industrial Manufacturing Systems

In industrial manufacturing systems, predictive maintenance is the process of increasing the rate of productivity and minimizing the time that equipment takes to be out of order through early identification of the equipment that is likely to fail. The main focus of this research is to analyze the possibility of using modern approaches in machine learning to enhance the methods of predictive maintenance. We compare multiple current approaches of deep learning, ensemble methods, and anomaly detection to determine their effectiveness in predicting the maintenance requirements utilizing the sensor and operational data. With the help of a large amount of data, we consider the results of the work of each algorithm for the assessment of the predictive accuracy, the ranking of features, and the detection of anomalies. The findings highlight disparities in the effectiveness of the algorithms in terms of accuracy, precision, and recall, and the deep learning models’ ability to grasp intricate and anomalous patterns. The performance of the maintenance predictions is depicted by the use of visualizations of the performance metrics and feature importance. It also describes the drawbacks of the existing models, such as the problem of data quality and generalization. The study draws attention to the possibility of applying sophisticated machine-learning methods to improve the effectiveness of PM in industrial environments. Possible directions of future research are to enhance the generalization ability of the developed models and to expand the usage of modern trends in the machine learning field to enhance maintenance strategies.