Dynamic Anomaly Detection Research Articles

Dynamic Anomaly Detection in the Chinese Energy Market During Financial Turbulence Using Ratio Mutual Information and Crude Oil Price Movements

Dynamic Anomaly Detection in the Chinese Energy Market During Financial Turbulence Using Ratio Mutual Information and Crude Oil Price Movements

Research of anomaly detection based on dynamic anomaly detection enhancement framework

Abstract Anomaly detection plays a crucial role in various fields, from industrial defect inspection to geological detection. However, traditional approaches often struggle with insufficient discriminability and an inability to generalize to unseen anomalies. These limitations stem from the practical difficulty in gathering a comprehensive set of anomalies and the tendency to overlook anomalous instances in favor of normal samples. To address these challenges, we propose a novel Dynamic Anomaly Detection Enhancement Framework, integrating three key innovations: (1) SaliencyAug: An adaptive saliency-guided augmentation method that generates realistic pseudo-samples to enhance learning of rare anomalies, improving model generalization. (2) DynAB: A dynamic attention block that achieves effective multi-level feature fusion while minimizing redundant information, enhancing detection accuracy. (3) DualOM: A dual-head optimization module which employs separate heads for normal and anomalous sample learning, creating more explicit and discriminative decision boundaries. Extensive experiments across multiple real-world datasets demonstrate our framework's superior performance in detecting a wide range of anomalies, demonstrating 2.4% improvement over state-of-the-art methods.

DPDGAD: A Dual-Process Dynamic Graph-based Anomaly Detection for multivariate time series analysis in cyber-physical systems

With the rapid development of cyber–physical systems, an increasing amount of data is stored in the form of multivariate time series. Detecting anomalies within multivariate time series has become a crucial means to ensure the proper functioning of cyber–physical systems. Traditional methods for anomaly detection in multivariate time series often categorize features into temporal and spatial features based on their representative dimensions. However, these approaches tend to overlook the unique characteristics of cyber–physical systems reflected in the multivariate time series. This hinders their ability to perform well in cases with complex multivariate time series data. To overcome this challenge, we introduce a novel approach that classifies the spatiotemporal features of multivariate time series data into steady features (features that remain constant under external variations) and dynamic features (features that respond to external changes), aligning with the nature of cyber–physical systems. Addressing this challenge involves two main aspects: (1) determining how to classify steady and dynamic features in multivariate time series data and (2) leveraging these features for anomaly detection.To address these challenges, we propose a Dual-Process Dynamic Graph-based time series Anomaly Detection (DPDGAD) method. DPDGAD enhances the generalizability of anomaly detection in complex spatiotemporal environments and improves the interpretability of the detected anomalies. We validate the performance of DPDGAD through six real-world datasets generated by cyber–physical systems, achieving F1 scores of 0.7529 and 0.9009 on WADI and SWaT, respectively.

A novel LDVP-based anomaly detection method for data streams

Recently, some progress has been made in anomaly detection for data streams. However, these methods still exhibit deficiencies in effectively balancing computational efficiency and detection accuracy. In this paper, we propose a novel anomaly detection method called Dynamic Window Anomaly Detection based on Local Density of Vector dot Product (DWAD-LDVP) for data streams. DWAD-LDVP uses an adaptive dynamic sliding window mechanism with a verification model to balance the relationship between computational efficiency and detection accuracy. By converting local density values measured by the vector dot product into outlier scores, DWAD-LDVP can improve the accuracy of anomaly detection compared to SOTA methods. Furthermore, an incremental computation module is adopted to process data streams efficiently, thereby reducing unnecessary computational overhead. Simulation experiments on synthetic data and real-world datasets validate the robustness and superior performance of the proposed DWAD-LDVP method. Specifically, the experimental results indicate that the proposed DWAD-LDVP achieves competitive performance measured by accuracy, precision, recall, F1 score, and ROC AUC, confirming its effectiveness in anomaly detection tasks.

Dynamic Anomaly Detection in Gantry Transactions Using Graph Convolutional Network-Gate Recurrent Unit with Adaptive Attention

With the wide application of Electronic Toll Collection (ETC) systems, the effectiveness of the operation and maintenance of gantry equipment still need to be improved. This paper proposes a dynamic anomaly detection method for gantry transactions, utilizing the contextual attention mechanism and Graph Convolutional Network-Gate Recurrent Unit (GCN-GRU) dynamic anomaly detection method for gantry transactions. In this paper, four different classes of gantry anomalies are defined and modeled, representing gantries as nodes and the connectivity between gantries as edges. First, the spatial distribution of highway ETC gantries is modeled using the GCN model to extract gantry node features. Then, the contextual attention mechanism is utilized to capture the recent patterns of the dynamic transaction graph of the gantries, and the GRU model is used to extract the time-series characteristics of the gantry nodes to dynamically update the gantry leakage. Our model is evaluated on several experimental datasets and compared with other commonly used anomaly detection methods. The experimental results show that our model outperforms other anomaly detection models in terms of accuracy, precision, and other evaluation values of 99%, proving its effectiveness and robustness. This model has a wide application potential in real gantry detection and management.

MAD-EN: Microarchitectural Attack Detection Through System-Wide Energy Consumption

Microarchitectural attacks have become increasingly threatening the society with diverse set of attacks such as Spectre and Meltdown. Vendor patches cannot keep up with the pace of the new threats, which makes the dynamic anomaly detection tools more evident than before. Unfortunately, hardware performance counters (HPCs) utilized in previous works can detect a few microarchitectural attacks due to the small number of counters that can be profiled concurrently while introducing high performance overhead. These challenges consequently yield to inefficient detection tools in real-world security-critical systems. In this study, we introduce MAD-EN dynamic detection tool that leverages system-wide energy consumption traces collected from a generic Intel RAPL tool to detect ongoing anomalies in two different microarchitectures, namely Intel Comet Lake and Intel Tiger Lake. In the first phase of MAD-EN, we can distinguish 16 variants from 11 different micro-architectural attacks from benign applications by utilizing a binary-class CNN-based model with an F1 score of 0.998, which makes our tool the most generic attack detection tool so far. In the second phase, MAD-EN can recognize the respective attack types with a 95% accuracy by utilizing a multi-class CNN-based classification technique after the anomaly is detected. We demonstrate that MAD-EN introduces 69.3% less performance overhead compared to performance counter-based detection mechanisms, allowing more feasible real-time detection tool for generic purpose systems.

A dynamic anomaly detection method of building energy consumption based on data mining technology

Due to the equipment failure and inappropriate operation strategy, it is often difficult to achieve energy-efficient building. Anomaly detection of building energy consumption is one of the important approaches to improve building energy-saving. The great amounts of energy consumption data collected by building energy monitoring platforms (BEMS) provides potentials in using data mining technology for anomaly detection. This study proposes a dynamic anomaly detection algorithm for building energy consumption data, which realizes the dynamic detection of point anomalies and collective anomalies. The algorithm integrates unsupervised clustering algorithm with supervised algorithm to establish a semi-supervised matching mechanism, which avoids the influence of error label and improves the efficiency of anomaly detection. A particle swarm optimization (PSO) is used to optimize the unsupervised clustering algorithm. This investigation tests the effectiveness of the proposed algorithm and evaluates the performance of the energy consumption clustering algorithm by using the annual electricity consumption data of an experimental building in a university. The results show that the clustering accuracy of the algorithm can reach more than 80%, and it can effectively detect the building energy consumption data of two different forms of outliers. It can provide reliable data support for adjusting building management strategies.

Dynamic Graph-Based Anomaly Detection in the Electrical Grid

Given sensor readings over time from a power grid, how can we accurately detect when an anomaly occurs? A key part of achieving this goal is to use the network of power grid sensors to quickly detect, in real-time, when any unusual events, whether natural faults or malicious, occur on the power grid. Existing bad-data detectors in the industry lack the sophistication to robustly detect broad types of anomalies, especially those due to emerging cyber-attacks, since they operate on a single measurement snapshot of the grid at a time. New ML methods are more widely applicable, but generally do not consider the impact of topology change on sensor measurements and thus cannot accommodate regular topology adjustments in historical data. Hence, we propose <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DynWatch</small> , a domain knowledge based and topology-aware algorithm for anomaly detection using sensors placed on a dynamic grid. Our approach is accurate, outperforming existing approaches by 20 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> or more (F-measure) in experiments; and fast, averaging less than 1.7 ms per time tick per sensor on a 60K+ branch case using a laptop computer, and scaling linearly with the size of the graph.

Automated food safety early warning system in the dairy supply chain using machine learning

Traditionally, early warning systems for food safety are based on monitoring targeted food safety hazards. Optimal early warning systems, however, should identify signals that precede the development of a food safety risk. Moreover, such signals could be identified in factors from domains adjacent to the food supply chain, so-called drivers of change and other indicators. In this study, we show for the first time that such drivers and indicators may indeed represent signals that precede the detection of a food safety risk. The dairy supply chain in Europe was used as an application case. Using dynamic unsupervised anomaly detection models, anomalies were detected in indicator data expected by domain experts to impact the development of food safety risks in milk. Additionally, a Bayesian network was used to identify the chemical food safety hazards in milk associated with an anomaly for the Netherlands. The results showed that the frequency of anomalies varied per country and indicator. However, all countries showed in the period investigated (2008–2019), anomalies in the indicators “raw milk price” and “barely milk price” and no anomalies in the indicator” income of dairy farms”. A cross-correlation analysis of the number of Rapid Alert for Food and Feed (RASFF) notifications and anomalies in indicators revealed significant correlations of many indicators but difference between countries was observed. Interesting, for all countries the cross corelation with indicator “milk price” was significant, albeit the lag time varied from 5 months (United Kingdom) to 22 months (Italy). This finding suggests that severe changes in domains adjacent to the food supply chain may trigger the development of food safety problems that become visible many months later. Awareness of such relationships will provide the opportunity for food producers or inspectors to take timely measures to prevent food safety problems. • Anomalies in prices, average mean temperature and precipitation may be used as early warning factors for food safety. • Anomalies in milk prices precedes food safety issues in milk. • Bayesian network was developed to identify the food safety hazards. • An early warning system for milk supply chain in Europe was developed. • Real-time anomaly detection system in supply chains for food authorities and inspectors.

Bayesian Filtering for Dynamic Anomaly Detection and Tracking

This article presents a Bayesian approach for sequential detection of anomalies in the motion of a target and joint tracking. The anomaly is modeled as a binary (on/off) switching unknown control input that goes into action (begins to exist, or “switches on”) thereby modifying the object dynamics; and by ceasing its activity (becoming nonexistent, or “switching off”) returns the dynamics to nominal. The developed Bayesian framework brings together random finite set (RFS) theory to represent the switching nature of the anomaly, and optimal joint input and state estimation to sequentially update a hybrid state that incorporates a random vector for the kinematic state and a Bernoulli RFS for the unknown control input. In addition, a closed-form solution, the Gaussian-mixture hybrid Bernoulli filter (GM-HBF), has been developed to provide a customized solution for dynamic anomaly detection in the maritime domain characterized by linear Gaussian target dynamics. Based on the Ornstein–Uhlenbeck dynamic model for vessels, where the evolution of the object velocity is governed by a piecewise mean-reverting stochastic process, the anomaly can be represented by a change in the long-run mean velocity (i.e., the unknown control input) from the nominal condition that forces the vessel to deviate from its standard route. We demonstrate the effectiveness of the proposed GM-HBF in both simulated and real-world maritime surveillance applications and test its performance in the face of false measurements, detection uncertainty, and sensor data gaps.

Dynamic Anomaly Detection With High-Fidelity Simulators: A Convex Optimization Approach

The main objective of this article is to develop scalable dynamic anomaly detectors with high-fidelity simulators of power systems. On the one hand, models in high-fidelity simulators are typically “intractable” if one opts to describe them in a mathematical formulation in order to apply existing model-based approaches from the anomaly detection literature. On the other hand, pure data-driven methods developed primarily in the machine learning literature neglect our knowledge about the underlying dynamics of power systems. In this study, we combine tools from these two mainstream approaches to develop a data-assisted model-based diagnosis filter utilizing both the knowledge from a picked abstract model and also the data of simulation results from high-fidelity simulators. The proposed diagnosis filter aims to achieve two desired features: (i) performance robustness with respect to model mismatch; (ii) high scalability. To this end, we propose a tractable (convex) optimization-based reformulation in which decisions are the filter parameters, the model-based information introduces feasible sets, and the data from the simulator forms the objective function to-be-minimized regarding the effect of model mismatch on the filter performance. To validate the theoretical results, we implement the developed diagnosis filter in DIgSILENT PowerFactory to detect false data injection attacks on the Automatic Generation Control measurements in the three-area IEEE 39-bus system.

DyD2: Dynamic Double anomaly Detection Application to on-board space radiation faults

Anomaly detection is a crucial aspect of embedded applications. However, limited computational power, evolving environments, and lack of training data are difficulties that can limit anomaly detection algorithms. One class classification algorithms are often used for this task to circumvent the need of anomalous data in the training set. This paper presents a new machine learning algorithm for anomaly detection called Dynamic Double anomaly Detection DyD2that is suited to evolving environments and on-board requirements. The contributions made by DyD2 are thoroughly presented and an experimental evaluation is set up to compare DyD2 to state-of-the-art algorithms.

A Dynamic Anomaly Detection Approach Based on Permutation Entropy for Predicting Aging-Related Failures.

Software aging is a phenomenon referring to the performance degradation of a long-running software system. This phenomenon is an accumulative process during execution, which will gradually lead the system from a normal state to a failure-prone state. It is a crucial challenge for system reliability to predict the Aging-Related Failures (ARFs) accurately. In this paper, permutation entropy (PE) is modified to Multidimensional Multi-scale Permutation Entropy (MMPE) as a novel aging indicator to detect performance anomalies, since MMPE is sensitive to dynamic state changes. An experiment is set on the distributed database system Voldemort, and MMPE is calculated based on the collected performance metrics during execution. Finally, based on MMPE, a failure prediction model using the machine learning method to reveal the anomalies is presented, which can predict failures with high accuracy.

An anomaly detection framework for time-evolving attributed networks

Real-world information systems are naturally dynamic, and they are usually represented as a time-evolving attributed network (i.e., a sequence of static attributed networks). Recently, there is a surge of research interests in finding anomalous nodes upon attributed networks due to its significant implications in many high-impact applications, such as financial fraud detection, network intrusion detection, and opinion spam detection, to name a few. Despite the importance of anomaly detection in time-evolving attributed network, a vast majority of existing methods fail to capture the evolution of the underlying networks properly, as they regard the whole system as static and neglect the evolution process. Meanwhile, they treat all the attributes and the instances equally, ignoring the existence of noisy which may lead to the adverse effects to the detection results. To tackle these problems, in this paper, we propose a novel dynamic anomaly detection framework on time-evolving attributed networks on the basis of residual analysis, namely AMAD. Under the assumption of temporal smoothness property, AMAD leverages the small smooth disturbance between two consecutive time stamps to characterize the evolution of networks for incrementally update. Experiments conducted on both synthetic and real-world time-evolving attributed networks show the superiority of our proposed method in detecting anomalies. Moreover, our method is competitive in terms of efficiency compared to the existing work.

A Framework for Anomaly Detection in Time-Driven and Event-Driven Processes using Kernel Traces

Model-checking and verification using Kripke structures and computational tree logic* (CTL*) use abstractions from the model/process/application to create the state-transition graphs that verify the model behavior. This scheme of profiling the performance of a process imports that the depth of the process operation correlates with the level abstraction. However, because of state explosion problems, these abstractions tend to restrict the scope to create manageable execution states. Therefore, for context modeling, this procedure does not generate a fine-grained behavioral model as generated states limit the ability of the abstraction to capture the execution time interactions amongst the processes, the hardware, and the kernel. Hence, in this paper, we present an end-to-end framework that comprises auto-encoders and probabilistic models to understand the behavior of system processes and detect deviant behaviors. We test this framework with a publicly available dataset generated from an autonomous aerial vehicle (UAV) application and the results show that by creating a fine-grained model that exploits previously unharnessed properties of the system calls, we can create a dynamic anomaly detection framework that evolves as the threats change.

Detecting anomalies in financial statements using machine learning algorithm

Purpose The purpose of this paper is to evaluate the possibility of rating the credit worthiness of a firm’s quarterly financial report using a dynamic anomaly detection method. Design/methodology/approach The study uses a data set containing financial statements from Quarter 1 – 2001 to Quarter 4 – 2016 of 937 Vietnamese listed firms. In sum, 24 fundamental financial indices are chosen as control variables. The study employs the Mahalanobis distance to measure the proximity of each data point from the centroid of the distribution to point out the extent of the anomaly. Findings The finding shows that the model is capable of ranking quarterly financial reports in terms of credit worthiness. The execution of the model on all observations also revealed that most financial statements of Vietnamese listed firms are trustworthy, while almost a quarter of them are highly anomalous and questionable. Research limitations/implications The study faces several limitations, including the availability of genuine accounting data from stock exchanges, the strong assumptions of a simple statistical distribution, the restricted timeframe of financial data and the sensitivity of the thresholds for anomaly levels. Practical implications The study opens an avenue for ordinary users of financial information to process the data and question the validity of the numbers presented by listed firms. Furthermore, if fraud information is available, similar research can be conducted to examine the tendency for companies with anomalous financial reports to commit fraud. Originality/value This is the first paper of its kind that attempts to build an anomaly detection model for Vietnamese listed companies.

From Static to Dynamic Anomaly Detection With Application to Power System Cyber Security

Developing advanced diagnosis tools to detect cyber attacks is the key to security of power systems. It has been shown that multivariate data injection attacks can bypass bad data detection schemes typically built on static behavior of the systems, which misleads operators to disruptive decisions. In this article, we depart from the existing static viewpoint to develop a diagnosis filter that captures the dynamics signatures of such a multivariate intrusion. To this end, we introduce a dynamic residual generator approach formulated as robust optimization programs in order to detect a class of disruptive multivariate attacks that potentially remain stealthy in view of a static bad data detector. We investigate two possible desired features: (i) a non-zero transient and (ii) a non-zero steady-state behavior of the residual generator in the presence of an attack. In case (i), the problem is reformulated as a finite, but possibly non-convex, optimization program. We further develop a linear programming relaxation that improves the scalability, and as such practicality, of the diagnosis filter design. In case (ii), it turns out that the resulting robust program admits an exact convex reformulation, yielding a Nash equilibrium between the attacker and the residual generator. This assertion has an interesting implication: the proposed approach is not conservative in the sense that the additional knowledge of the worst-case attack does not improve the diagnosis performance. To illustrate our theoretical results, we implement the proposed diagnosis filter to detect multivariate attacks on the system measurements deployed to generate the so-called Automatic Generation Control signals in a three-area IEEE 39-bus system.

Dynamic video anomaly detection and localization using sparse denoising autoencoders

The emergence of novel techniques for automatic anomaly detection in surveillance videos has significantly reduced the burden of manual processing of large, continuous video streams. However, existing anomaly detection systems suffer from a high false-positive rate and also, are not real-time, which makes them practically redundant. Furthermore, their predefined feature selection techniques limit their application to specific cases. To overcome these shortcomings, a dynamic anomaly detection and localization system is proposed, which uses deep learning to automatically learn relevant features. In this technique, each video is represented as a group of cubic patches for identifying local and global anomalies. A unique sparse denoising autoencoder architecture is used, that significantly reduced the computation time and the number of false positives in frame-level anomaly detection by more than 2.5%. Experimental analysis on two benchmark data sets - UMN dataset and UCSD Pedestrian dataset, show that our algorithm outperforms the state-of-the-art models in terms of false positive rate, while also showing a significant reduction in computation time.

Dynamic anomaly detection using cross layer security in MANET

Mobile ad hoc networks (MANETs) are one of the emerging technologies of wireless communication, in which each node in the MANET acts as a router. In an ad hoc network, any node can communicate with any other node in the network. However, this infrastructure of mobile nodes makes this system more vulnerable to various types of attacks. A black hole attack is one such attack in which the packet is dropped maliciously. In this paper, we propose a security solution known as the honeypot-based dynamic anomaly detection using cross-layer security (HBDADCS). The proposed technique detects and isolates black hole attacks from the ad hoc network. In this work, various novel algorithms are used which involve honeypot methodology. These technologies are proposed in order to detect and isolate black hole attacks from the network. Simulation-based results prove that the proposed technique has better packet delivery and end to end to delay with a decreased network load solely due to control packets compared to the other existing techniques.

Online Adaptive Anomaly Detection for Augmented Network Flows

Traditional network anomaly detection involves developing models that rely on packet inspection. However, increasing network speeds and use of encrypted protocols make per-packet inspection unsuited for today’s networks. One method of overcoming this obstacle is aggregating packet header information and performing flow-based analysis where data flow patterns are examined rather than deep packet inspection. Many existing approaches are special purpose limited to detecting specific behavior. Also, the data reduction inherent in identifying anomalous flows hinders alert correlation. In this article, we propose and develop a dynamic anomaly detection approach for augmented network flows. We sketch network state during flow creation, enabling general-purpose threat detection. We describe an efficient flow augmentation approach based on the count-min sketch that provides per-flow-, per-node-, and per-network-level statistics parallel to flow record generation. We design and develop a support vector machine-based adaptive anomaly detection and correlation mechanism, which is capable of aggregating alerts without a priori alert classification and evolving models online. We further develop a lightweight evolving alert aggregation method and combine it with a confidence forwarding mechanism identifying a small percentage predictions for additional processing. We show effectiveness of our methods on both enterprise and backbone traces. Experimental results demonstrate its ability to maintain high accuracy without the need for offline training.