Anomaly Detection Research Articles

Libby-Novick Beta-Liouville Distribution for Enhanced Anomaly Detection in Proportional Data

We consider the problem of anomaly detection in proportional data by investigating the Libby-Novick Beta-Liouville distribution, a novel distribution merging the salient characteristics of Liouville and Libby-Novick Beta distributions. Its main benefit, compared to the typical distributions dedicated to proportional data such as Dirichlet and Beta-Liouville, is its adaptability and explanatory power when dealing with this kind of data. Our goal is to exploit this appropriateness for modeling proportional data to achieve great performance in the anomaly detection task. First, we develop generative models, namely finite mixture models of Libby-Novick Beta-Liouville distributions. Then, we propose two discriminative techniques: Normality scores based on selecting the given distribution to approximate the softmax output vector of a deep classifier and an improved version of Support Vector Machine (SVM) by suggesting a feature mapping approach. We demonstrate the benefits of the presented approaches through a variety of experiments on both image and non-image datasets. The results demonstrate that the proposed anomaly detectors based on the Libby-Novick Beta-Liouville distribution outperform the classical distributions as well as the baseline techniques.

IoT Leak Detection System for Onshore Oil Pipeline Based on Thermography.

The vast expanses of remote onshore areas in oil-producing countries are home to a network of flow and collection pipelines that are susceptible to leaks. Most of these areas lack the infrastructure to enable the use of remote monitoring systems equipped with sensors and real-time data analysis to provide early detection of anomalies. This paper proposes a proof of concept for a monitoring system based on the Internet of Things (IoT) for real-time detection of pipeline leaks in onshore oil production fields. The proposed system, based on a thermal imaging leak detection method, informs the operator of the system's operating status via a web page. The leak detection system communicates via a Zigbee network between the IoT devices and a 4G mobile network. The results of the tests carried out show that a visual and automatic IoT-based leak detection system is possible and plausible. The proposed leak detection system enables supervisors at remote stations and field workers to monitor the operating status of pipelines via computers, tablets, or smartphones, regardless of where they are.

Edge Artificial Intelligence for Electrical Anomaly Detection Based on Process-In-Memory Chip

Neural-networks (NNs) for the current feature analysis bring novel electrical safety functions in smart circuit breakers (CBs), especially for preventing the fire hazard from electric vehicle/bike battery charging. In this work, the edge artificial intelligence (AI) solutions for the electrical anomaly detection were designed and demonstrated based on the process-in-memory (PIM) AI chip. The ultra-low power and high-performance character of PIM AI chips enable the edge solution to embed in the limited space inside the circuit breaker and to detect improper battery charging at millisecond latency.

DA-Net: A classification-guided network for dental anomaly detection from dental and maxillofacial images

Dental abnormalities (DA) are frequent signs of disorders of the mouth that cause discomfort, infection, and loss of teeth. Early and reasonably priced treatment may be possible if defective teeth in the oral cavity are automatically detected. Several research works have endeavored to create a potent deep learning model capable of identifying DA from pictures. However, because of the following problems, aberrant teeth from the oral cavity are difficult to detect: 1) Normal teeth and crowded dentition frequently overlap; 2) The lesion area on the tooth surface is tiny. This paper proposes a professional dental anomaly detection network (DA-Net) to address such issues. First, a multi-scale dense connection module (MSDC) is designed to distinguish crowded teeth from normal teeth by learning multi-scale spatial information of dentition. Then, a pixel differential convolution (PDC) module is designed to perform pathological tooth recognition by extracting small lesion features. Finally, a multi-stage convolutional attention module (MSCA) is developed to integrate spatial information and channel information to obtain abnormal teeth in small areas. Experiments on benchmarks show that DA-Net performs well in dental anomaly detection and can further assist doctors in making treatment plans. Specifically, the DA-Net method performs best on multiple detection evaluation metrics: IoU, PRE, REC, and mAP. In terms of REC and mAP indicators, the proposed DA-Net method is 1.1% and 1.3% higher than the second-ranked YOLOv7 method.

Comprehensive Review of One-Class Classification Approaches for Anomaly Detection

Comprehensive Review of One-Class Classification Approaches for Anomaly Detection

AIoT-Based Visual Anomaly Detection in Photovoltaic Sequence Data via Sequence Learning

Anomaly detection is a common analytical task aimed at identifying rare cases that differ from the majority of typical cases in a dataset. In the management of photovoltaic (PV) power generation systems, it is essential for electric power companies to effectively detect anomalies in PV sequence data, as this helps operators and experts understand and interpret anomalies within PV arrays when making response decisions. However, traditional methods that rely on manual labor and regular data collection are difficult to monitor in real time, resulting in delays in fault detection and localization. Traditional machine learning algorithms are slow and cumbersome in processing data, which affects the operational safety of PV plants. In this paper, we propose a visual analytic approach for detecting and exploring anomalous sequences in a PV sequence dataset via sequence learning. We first compare the sequences with their reconstructions through an unsupervised anomaly detection algorithm (Long Short-Term Memory) based on AutoEncoders to identify anomalies. To further enhance the accuracy of anomaly detection, we integrate the artificial intelligence of things (AIoT) technology with a strict time synchronization data collection and real-time processing algorithm. This integration ensures that data from multiple sensors are synchronized and processed in real time. Then, we analyze the characteristics of the anomalies based on the visual comparison of different PV sequences and explore the potential correlation factors to analyze the possible causes of the anomalies. Case studies based on authentic enterprise datasets demonstrate the effectiveness of our method in the anomaly detection and exploration of PV sequence data.

XAI-Based Accurate Anomaly Detector That Is Robust Against Black-Box Evasion Attacks for the Smart Grid

In the realm of smart grids, machine learning (ML) detectors—both binary (or supervised) and anomaly (or unsupervised)—have proven effective in detecting electricity theft (ET). However, binary detectors are designed for specific attacks, making their performance unpredictable against new attacks. Anomaly detectors, conversely, are trained on benign data and identify deviations from benign patterns as anomalies, but their performance is highly sensitive to the selected threshold values. Additionally, ML detectors are vulnerable to evasion attacks, where attackers make minimal changes to malicious samples to evade detection. To address these limitations, we introduce a hybrid anomaly detector that combines a Deep Auto-Encoder (DAE) with a One-Class Support Vector Machine (OCSVM). This detector not only enhances classification performance but also mitigates the threshold sensitivity of the DAE. Furthermore, we evaluate the vulnerability of this detector to benchmark evasion attacks. Lastly, we propose an accurate and robust cluster-based DAE+OCSVM ET anomaly detector, trained using Explainable Artificial Intelligence (XAI) explanations generated by the Shapley Additive Explanations (SHAP) method on consumption readings. Our experimental results demonstrate that the proposed XAI-based detector achieves superior classification performance and exhibits enhanced robustness against various evasion attacks, including gradient-based and optimization-based methods, under a black-box threat model.

Advancing athlete safety through real-time ECG monitoring for enhanced cardiovascular health in sports performance

This research paper explores the implementation and efficacy of real-time electrocardiogram (ECG) monitoring systems for athletes, emphasizing their potential to significantly enhance safety and performance in sports settings. By utilizing advanced ECG technology, the study investigates how continuous, real-time heart rate and rhythm monitoring can aid in the immediate detection of cardiovascular anomalies during high-intensity activities. The research methodology incorporates the deployment of portable ECG devices in a controlled experimental setup, analyzing data from athletes during training sessions and competitive events. Results from the study highlight the system's ability to provide swift and accurate cardiac assessments, thereby enabling timely medical interventions. Moreover, the paper discusses the technical challenges associated with real-time ECG monitoring, such as signal interference and data accuracy, and addresses privacy and ethical considerations concerning the continuous collection of health data. The discussion extends to the implications of integrating such technology within sports medicine, suggesting that while the systems offer substantial benefits in monitoring and preventing cardiac issues, they also necessitate rigorous standards for data security and ethical oversight. The conclusion advocates for a balanced approach to the adoption of these technologies, proposing future research directions that focus on enhancing system reliability and integrating artificial intelligence to predict potential health risks proactively. This study contributes to the ongoing discourse in sports health technology by providing a comprehensive analysis of real-time ECG monitoring as a transformative tool for athlete healthcare management.

Credit Anomaly Detection Method based on Bayesian Networks

Due to the virtual nature of online transactions and the underdeveloped personal credit risk assessment system of Internet finance lending platforms in China, these platforms often struggle to verify the creditworthiness of applicants prone to fraud and default, leading to loan losses. As a result, risk control has become the core of Internet lending platforms. Modern risk control relies on statistical and data mining techniques to analyze and model data, uncover patterns that indicate loan defaults, and identify anomalies. This paper proposes a Bayesian network-based credit anomaly detection method, which detects anomalies by calculating and ranking the joint anomaly probabilities of sample instances. The technique operates under unsupervised learning and can effectively handle missing and imbalanced data. This method analyzes a loan credit dataset with 32,581 samples and 12 features of the customers. The result of the analysis demonstrates the feasibility and effectiveness of this method in detecting anomalies in online loans.

Monitoring event data extracted from online news for outbreak detection

Abstract Introduction Digital disease surveillance (DDS) detects public health events from internet-based data e.g., online news. Event features depicting epidemiological and social characteristics of health events can be extracted from news using the natural language process techniques. However, few studies have leveraged the event features to support anomaly detection in DDS. We aimed to understand the distribution of the event features and explore anomaly detection using the frequency of these features. Methods We collected event data from COVID-19-related news collected from October 1 to December 31, 2021, sourced from BioCaster, an infectious-disease-focused DDS system. The predefined event features in BioCaster include disease, pathogen, location and 14 binary features, such as if an event was caused by an unclassified virus. We described the distribution of the features and detected changes in the frequency of event features using a Bayesian online change point detection. We compared the change points with the number of new cases and of genomic samples collected. Results We included 170,168 news articles reporting COVID-19 in 155 countries. The event feature indicating that an event was caused by an unclassified virus was identified as positive among 3831 (2.25%) news and 12.91% of news had positive value for the feature indicating cases who had travelled across international borders. The change points detected from these two features were temporally correlated to the emergence of the Omicron variant in corresponding countries, which was more significant in countries with at least 300 news articles. Conversely, event features irrelevant to this case study, e.g., if the cases were military workers, were identified as negative in all news and no change points were detected. Conclusions Our study highlights the potential of monitoring the frequency of event features extracted from online news for anomaly detection in DDS, which relies on sufficient news coverage. Key messages • Monitoring the event features extracted from online news provide is useful approach for automatic anomaly detection in digital disease surveillance. • Increasing media coverage is fundamental for improving the early detection in a digital disease surveillance system.

Optimisation of artefact detection in photoplethysmography heart rate data: influence of different classifiers in machine learning models

Abstract Background Heart rate (HR) tracking by wrist-worn devices using photoplethysmography (PPG) could assist in continuously following up physical activity. However, the accuracy can be impacted by (motion) artefacts. Machine learning models could help to recognise artefacts in PPG-based HR data. The choice of classifier in these machine learning models is a determing factor for task performance of the model. Purpose This study evaluates and determines the optimal classifier for a new machine learning-based approach to enhance the reliability of artefact detection in PPG-based HR data. Methods A total of 62 participants (27 cardiac rehabilitation patients, 35 healthy athletes) wore both a test device and a reference device measuring HR continuously for 24 hours. A training dataset was prepared, assigning two independent labels (i.e. anomaly and activity) to each HR episode based on the reference device data. Fitbit data were processed using our in-house designed artefact removal procedure, which involves the application of two classification models: one for anomaly detection and another for activity detection. Four distinct classifiers were employed for both models: Balanced Bagging, Balanced Bagging with Random Forest, Balanced Random Forest, and Logistic Regression. Each classifier was evaluated using area under the receiver operating characteristic curve (ROC-AUC), accuracy, sensitivity and specificity. Results Of the 1,647,328 HR data points collected, 103,095 (6.26%) were identified as artefacts. Figure 1 and Figure 2 summarise the performance of the distinct classifiers for the anomaly model and the activity model, respectively. Balanced Bagging and Balanced Bagging with Random Forest consistently demonstrate the highest AUC values and accuracies across both anomaly and activity detection models (anomaly detection: AUC = 0.95, accuracy = 89-85%; activity detection: AUC = 0.98, accuracy = 95%). Comparing these two, Balanced Bagging with Random Forest emerges as the preferred option, given the highest sensitivity in both anomaly detection (93%>86%) and activity detection models (99%>96%). In contrast, Balanced Random Forest and Logistic Regression exhibit inferior performance. In the anomaly detection model, Balanced Random Forest exhibits a lower sensitivity of 75%, while Logistic Regression performs even worse with a sensitivity of 25%. Similarly, in the activity detection model, both Balanced Random Forest and Logistic Regression demonstrate diminished performance. Conclusions Balanced Bagging with Random Forest emerges as the optimal classifier to detect anomalies and activities in continuous PPG-based HR data, thus contributing to the optimisation of our in-house designed procedure for removing artefacts. This processing aims to provide a reliable and automatic way for continuous HR monitoring, which can help monitor and guide physical activities.

Cross-validatory Z-Residual for Diagnosing Shared Frailty Models

Accurate model performance assessment in survival analysis is imperative for robust predictions and informed decision-making. Traditional residual diagnostic tools like martingale and deviance residuals lack a well-characterized reference distribution for censored regression, making numerical statistical tests based on these residuals challenging. Recently, the introduction of Z-residuals for diagnosing survival models addresses this limitation. However, concerns arise from conventional methods that utilize the entire dataset for both model parameter estimation and residual assessment, which may cause optimistic biases. This paper introduces cross-validatory Z-residuals as an innovative approach to address these limitations. Employing a cross-validation (CV) framework, the method systematically partitions the dataset into training and testing sets to reduce the optimistic bias. Our simulation studies demonstrate that, for goodness-of-fit tests and outlier detection, cross-validatory Z-residuals are significantly more powerful (e.g. power increased from 0.2 to 0.6). and more discriminative (e.g. AUC increased from 0.58 to 0.85) than Z-residuals without CV. We also compare the performance of Z-residuals with and without CV in identifying outliers in a real application that models the recurrence time of kidney infection patients. Our findings suggest that cross-validatory Z-residuals can identify outliers, which Z-residuals without CV fail to identify. The CV Z-residual is a more powerful tool than the No-CV Z-residual for checking survival models, particularly in goodness-of-fit tests and outlier detection. We have published a generic function, which is collected in an R package called Zresidual, for computing CV Z-residual for the output of the widely used survival R package.

Advancing Algorithmic Adaptability in Hyperspectral Anomaly Detection with Stacking-Based Ensemble Learning

Anomaly detection in hyperspectral imaging is crucial for remote sensing, driving the development of numerous algorithms. However, systematic studies reveal a dichotomy where algorithms generally excel at either detecting anomalies in specific datasets or generalizing across heterogeneous datasets (i.e., lack adaptability). A key source of this dichotomy may center on the singular and like biases frequently employed by existing algorithms. Current research lacks experimentation into how integrating insights from diverse biases might counteract problems in singularly biased approaches. Addressing this gap, we propose stacking-based ensemble learning for hyperspectral anomaly detection (SELHAD). SELHAD introduces the integration of hyperspectral anomaly detection algorithms with diverse biases (e.g., Gaussian, density, partition) into a singular ensemble learning model and learns the factor to which each bias should contribute so anomaly detection performance is optimized. Additionally, it introduces bootstrapping strategies into hyperspectral anomaly detection algorithms to further increase robustness. We focused on five representative algorithms embodying common biases in hyperspectral anomaly detection and demonstrated how they result in the previously highlighted dichotomy. Subsequently, we demonstrated how SELHAD learns the interplay between these biases, enabling their collaborative utilization. In doing so, SELHAD transcends the limitations inherent in individual biases, thereby alleviating the dichotomy and advancing toward more adaptable solutions.

Chemometric approaches for discriminating manufacturers of Korean handmade paper using infrared spectroscopy

The objective of this study was to identify the manufacturer of Hanji, Korean handmade paper widely used in conservation science. To achieve this, machine learning models utilizing attenuated total reflectance–infrared spectroscopy (ATR–IR) were developed to assess the robustness and effectiveness of the computed models. Principal component analysis (PCA), partial least squares–discriminant analysis (PLS–DA), decision tree (DT), and k-NN models were constructed using IR spectral data, with the spectral region between 1800 and 1500 cm⁻1 identified as the critical input variable through Variable Importance in Projection (VIP) scores. The transformation of the obtained spectra into second derivative spectra proved beneficial in this key spectral region, leading to significant improvements in model performance. Additionally, the application of DBSCAN for outlier detection was effective in refining the dataset, further enhancing the performance of the models. Specifically, the k-NN model, when applied to the selected variables and preprocessed with the second derivative transformation, achieved an F1 score of 0.92. These findings underscore the importance of focusing on the 1800–1500 cm⁻1 spectral range and applying outlier detection techniques, such as DBSCAN, to enhance the robustness and accuracy of the Hanji classification models by eliminating the influence of atypical data points.

Privacy-preserving MTS anomaly detection for network devices through federated learning

In the context of Maintenance-as-a-Service (MaaS), it is important for device vendors to develop multivariate time series (MTS) anomaly detection models that can accurately identify anomalies without compromising the privacy of customer enterprises' data. In this paper, we investigate the relationship between MTS data patterns and the parameters of unsupervised autoencoder (AE) models and show that they are highly consistent. Building on this insight, we propose a novel unsupervised federated learning (FL)-based framework called OmniFed, which cannot only address the heterogeneity of non-independent identically (non-iid) distributed data on different devices, but also achieve high-precision detection of device MTS anomalies while ensuring privacy. Specifically, OmniFed is initialized with an AE model and then trains local AE models on individual devices via federated learning. Finally, OmniFed clusters devices based on the parameters of the AE models and trains a cluster-specific MTS anomaly detection model using FL. Our experiments on two real-world datasets demonstrate that OmniFed achieves an F1-Score of 0.921, significantly higher than the best baseline method.

D2-SPDM: Faster R-CNN-Based Defect Detection and Surface Pixel Defect Mapping with Label Enhancement in Steel Manufacturing Processes

The steel manufacturing process is inherently continuous, meaning that if defects are not effectively detected in the initial stages, they may propagate through subsequent stages, resulting in high costs for corrections in the final product. Therefore, detecting surface defects and obtaining segmentation information is critical in the steel manufacturing industry to ensure product quality and enhance production efficiency. Specifically, segmentation information is essential for accurately understanding the shape and extent of defects, providing the necessary details for subsequent processes to address these defects. However, the time-consuming and costly process of generating segmentation annotations poses a significant barrier to practical industrial applications. This paper proposes a cost-efficient segmentation labeling framework that combines deep learning-based anomaly detection and label enhancement to address these challenges in the steel manufacturing process. Using ResNet-50, defects are classified, and faster region convolutional neural networks (faster R-CNNs) are employed to identify defect types and generate bounding boxes indicating the defect locations. Subsequently, recursive learning is performed using the GrabCut algorithm and the DeepLabv3+ model based on the generated bounding boxes, significantly reducing annotation costs by generating segmentation labels. The proposed framework effectively detects defects and accurately defines them, even in randomly collected images from the steel manufacturing process, contributing to both quality control and cost reduction. This study presents a novel approach for improving the quality of the steel manufacturing process and is expected to enhance overall efficiency in the steel manufacturing industry.

Enhancing IoT Security Using GA-HDLAD: A Hybrid Deep Learning Approach for Anomaly Detection

The adoption and use of the Internet of Things (IoT) have increased rapidly over recent years, and cyber threats in IoT devices have also become more common. Thus, the development of a system that can effectively identify malicious attacks and reduce security threats in IoT devices has become a topic of great importance. One of the most serious threats comes from botnets, which commonly attack IoT devices by interrupting the networks required for the devices to run. There are a number of methods that can be used to improve security by identifying unknown patterns in IoT networks, including deep learning and machine learning approaches. In this study, an algorithm named the genetic algorithm with hybrid deep learning-based anomaly detection (GA-HDLAD) is developed, with the aim of improving security by identifying botnets within the IoT environment. The GA-HDLAD technique addresses the problem of high dimensionality by using a genetic algorithm during feature selection. Hybrid deep learning is used to detect botnets; the approach is a combination of recurrent neural networks (RNNs), feature extraction techniques (FETs), and attention concepts. Botnet attacks commonly involve complex patterns that the hybrid deep learning (HDL) method can detect. Moreover, the use of FETs in the model ensures that features can be effectively extracted from spatial data, while temporal dependencies are captured by RNNs. Simulated annealing (SA) is utilized to select the hyperparameters necessary for the HDL approach. In this study, the GA-HDLAD system is experimentally assessed using a benchmark botnet dataset, and the findings reveal that the system provides superior results in comparison to existing detection methods.

Nonlinear Reinforcement Learning-Based Dynamic Test Case Prioritization with Anomaly Detection for Continuous Integration

This research article introduces DynamicR-TCP, a nonlinear approach to dynamic test case prioritization (TCP) leveraging reinforcement learning and anomaly detection. Designed to enhance the efficiency of software testing in continuous integration (CI) environments, the proposed model dynamically adapts to changing conditions by learning complex patterns from historical test data. A reinforcement learning agent, employing policy and value networks, guides nonlinear prioritization by optimizing the sequence of test case executions. The model integrates a sliding window strategy for adaptive focus on smaller test subsets and uses an experience replay buffer for iterative learning. An anomaly detection layer monitors test outcomes for deviations, triggering retraining when needed to maintain reliability. Experimental evaluation using the Defects4J dataset demonstrates the model’s superior fault detection rates, faster time to first fault, and higher computational efficiency compared to contemporary methods. An ablation study highlights the nonlinear synergy among the key components—reinforcement learning, sliding windows, and anomaly detection—showing significant performance gains when combined. This comprehensive framework advances software testing by providing an adaptive, efficient, and scalable solution, ensuring robust performance in dynamic CI environments.

Outlier detection in classification based on feature-selection-based regression

Outlier detection in classification based on feature-selection-based regression

Taxonomy and Survey of Collaborative Intrusion Detection System using Federated Learning

This review paper looks at recent research on Federated Learning (FL) for Collaborative Intrusion Detection Systems (CIDS) to establish a taxonomy and survey. The motivation behind this review comes from the difficulty of detecting coordinated cyberattacks in large-scale distributed networks. Collaborative anomalies are one of the network anomalies that need to be detected through robust collaborative learning methods. FL is promising collaborative learning method in recent research. This review aims to offer insights and lesson learn for creating a taxonomy of collaborative anomaly detection in CIDS using FL as a collaborative learning method. Our findings suggest that a taxonomy is required to map the discussion area, including an algorithm for training the learning model, the dataset, global aggregation model, system architecture, security, and privacy. Our results indicate that FL is a promising approach for collaborative anomaly detection in CIDS, and the proposed taxonomy could be useful for future research in this area. Overall, this review contributes to the growing knowledge of FL for CIDS, providing insights and lessons for researchers and practitioners. This research also concludes significant challenges, opportunities, and future directions in CIDS based on collaborative anomaly detection using FL.