Dynamic Anomaly Research Articles

Research of anomaly detection based on dynamic anomaly detection enhancement framework

Abstract Anomaly detection plays a crucial role in various fields, from industrial defect inspection to geological detection. However, traditional approaches often struggle with insufficient discriminability and an inability to generalize to unseen anomalies. These limitations stem from the practical difficulty in gathering a comprehensive set of anomalies and the tendency to overlook anomalous instances in favor of normal samples. To address these challenges, we propose a novel Dynamic Anomaly Detection Enhancement Framework, integrating three key innovations: (1) SaliencyAug: An adaptive saliency-guided augmentation method that generates realistic pseudo-samples to enhance learning of rare anomalies, improving model generalization. (2) DynAB: A dynamic attention block that achieves effective multi-level feature fusion while minimizing redundant information, enhancing detection accuracy. (3) DualOM: A dual-head optimization module which employs separate heads for normal and anomalous sample learning, creating more explicit and discriminative decision boundaries. Extensive experiments across multiple real-world datasets demonstrate our framework's superior performance in detecting a wide range of anomalies, demonstrating 2.4% improvement over state-of-the-art methods.

Prospects of early Jurassic deposits within the Khapchagay megaswell of the Vilyuy syneclise

The study of terrigenous Jurassic deposits in the Vilyuy syneclise has become increasingly important. This is because the fields contains shallow gas reserves and were first discovered in anticline traps during the 1960s. Renewed geological exploration in this region is now providing valuable new geological and geophysical data. The aim of research is to select exploration criteria for locating hydrocarbon deposits in the territory. Seismic exploration is a leading method of research. Based on this, an analysis is conducted on the characteristics of hydrocarbon gas deposit formation and distribution within the early Jurassic stratigraphic complex of the Vilyuy oil and gas Vilyuy oil and gas bearing region. In particular, the Khapchagay megaswell has been analyzed. The study has identified the Nedzhelinskoye local uplift as an inversion structure. A dynamic anomaly of the "bright spot" type is observed at the roof of the productive lower Jurassic deposits of the Kyzylsyr suite in the central block. Well tests confirmed the difference in hydrocarbon fluid saturation between the western and eastern parts of the Nedzhelinskaya structure. The presence of dynamic anomalies of inversion type can be seen as an exploration indicator for hydrocarbon accumulation zones in the terrigenous deposits of the Vilyuy syneclise. This is a practical significance of the work.

DPDGAD: A Dual-Process Dynamic Graph-based Anomaly Detection for multivariate time series analysis in cyber-physical systems

With the rapid development of cyber–physical systems, an increasing amount of data is stored in the form of multivariate time series. Detecting anomalies within multivariate time series has become a crucial means to ensure the proper functioning of cyber–physical systems. Traditional methods for anomaly detection in multivariate time series often categorize features into temporal and spatial features based on their representative dimensions. However, these approaches tend to overlook the unique characteristics of cyber–physical systems reflected in the multivariate time series. This hinders their ability to perform well in cases with complex multivariate time series data. To overcome this challenge, we introduce a novel approach that classifies the spatiotemporal features of multivariate time series data into steady features (features that remain constant under external variations) and dynamic features (features that respond to external changes), aligning with the nature of cyber–physical systems. Addressing this challenge involves two main aspects: (1) determining how to classify steady and dynamic features in multivariate time series data and (2) leveraging these features for anomaly detection.To address these challenges, we propose a Dual-Process Dynamic Graph-based time series Anomaly Detection (DPDGAD) method. DPDGAD enhances the generalizability of anomaly detection in complex spatiotemporal environments and improves the interpretability of the detected anomalies. We validate the performance of DPDGAD through six real-world datasets generated by cyber–physical systems, achieving F1 scores of 0.7529 and 0.9009 on WADI and SWaT, respectively.

A novel LDVP-based anomaly detection method for data streams

Recently, some progress has been made in anomaly detection for data streams. However, these methods still exhibit deficiencies in effectively balancing computational efficiency and detection accuracy. In this paper, we propose a novel anomaly detection method called Dynamic Window Anomaly Detection based on Local Density of Vector dot Product (DWAD-LDVP) for data streams. DWAD-LDVP uses an adaptive dynamic sliding window mechanism with a verification model to balance the relationship between computational efficiency and detection accuracy. By converting local density values measured by the vector dot product into outlier scores, DWAD-LDVP can improve the accuracy of anomaly detection compared to SOTA methods. Furthermore, an incremental computation module is adopted to process data streams efficiently, thereby reducing unnecessary computational overhead. Simulation experiments on synthetic data and real-world datasets validate the robustness and superior performance of the proposed DWAD-LDVP method. Specifically, the experimental results indicate that the proposed DWAD-LDVP achieves competitive performance measured by accuracy, precision, recall, F1 score, and ROC AUC, confirming its effectiveness in anomaly detection tasks.

Dynamic Anomaly Detection in Gantry Transactions Using Graph Convolutional Network-Gate Recurrent Unit with Adaptive Attention

With the wide application of Electronic Toll Collection (ETC) systems, the effectiveness of the operation and maintenance of gantry equipment still need to be improved. This paper proposes a dynamic anomaly detection method for gantry transactions, utilizing the contextual attention mechanism and Graph Convolutional Network-Gate Recurrent Unit (GCN-GRU) dynamic anomaly detection method for gantry transactions. In this paper, four different classes of gantry anomalies are defined and modeled, representing gantries as nodes and the connectivity between gantries as edges. First, the spatial distribution of highway ETC gantries is modeled using the GCN model to extract gantry node features. Then, the contextual attention mechanism is utilized to capture the recent patterns of the dynamic transaction graph of the gantries, and the GRU model is used to extract the time-series characteristics of the gantry nodes to dynamically update the gantry leakage. Our model is evaluated on several experimental datasets and compared with other commonly used anomaly detection methods. The experimental results show that our model outperforms other anomaly detection models in terms of accuracy, precision, and other evaluation values of 99%, proving its effectiveness and robustness. This model has a wide application potential in real gantry detection and management.

Adaptive Sampling and Quick Anomaly Detection in Large Networks

The monitoring of data streams with a network structure have drawn increasing attention due to its wide applications in modern process control. In these applications, high-dimensional sensor nodes are interconnected with an underlying network topology. In such a case, abnormalities occurring to any node may propagate dynamically across the network and cause changes of other nodes over time. Furthermore, high dimensionality of such data significantly increased the cost of resources for data transmission and computation, such that only partial observations can be transmitted or processed in practice. Overall, how to quickly detect abnormalities in such large networks with resource constraints remains a challenge, especially due to the sampling uncertainty under the dynamic anomaly occurrences and network-based patterns. In this paper, we incorporate network structure information into the monitoring and adaptive sampling methodologies for quick anomaly detection in large networks where only partial observations are available. We develop a general monitoring and adaptive sampling method and further extend it to the case with memory constraints, both of which exploit network distance and centrality information for better process monitoring and identification of abnormalities. Theoretical investigations of the proposed methods demonstrate their sampling efficiency on balancing between exploration and exploitation, as well as the detection performance guarantee. Numerical simulations and a case study on power network have demonstrated the superiority of the proposed methods in detecting various types of shifts. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Continuous monitoring of networks for anomalous events is critical for a large number of applications involving power networks, computer networks, epidemiological surveillance, social networks, etc. This paper aims at addressing the challenges in monitoring large networks in cases where monitoring resources are limited such that only a subset of nodes in the network is observable. Specifically, we integrate network structure information of nodes for constructing sequential detection methods via effective data augmentation, and for designing adaptive sampling algorithms to observe suspicious nodes that are likely to be abnormal. Then, the method is further generalized to the case that the memory of the computation is also constrained due to the network size. The developed method is greatly beneficial and effective for various anomaly patterns, especially when the initial anomaly randomly occurs to nodes in the network. The proposed methods are demonstrated to be capable of quickly detecting changes in the network and dynamically changes the sampling priority based on online observations in various cases, as shown in the theoretical investigation, simulations and case studies.

RegraphGAN: A graph generative adversarial network model for dynamic network anomaly detection

Due to the wide application of dynamic graph anomaly detection in cybersecurity, social networks, e-commerce, etc., research in this area has received increasing attention. Graph generative adversarial networks can be used in dynamic graph anomaly detection due to their ability to model complex data, but the original graph generative adversarial networks do not have a method to learn reverse mapping and require an expensive process in recovering the potential representation of a given input. Therefore, this paper proposes a novel graph generative adversarial network by adding encoders to map real data to latent space to improve the training efficiency and stability of graph generative adversarial network models, which is named RegraphGAN in this paper. And this paper proposes a dynamic network anomaly edge detection method by combining RegraphGAN with spatiotemporal coding to solve the complex dynamic graph data and the problem of attribute-free node information coding challenges. Meanwhile, anomaly detection experiments are conducted on six real dynamic network datasets, and the results show that the dynamic network anomaly detection method proposed in this paper outperforms other existing methods.

Personalized anomaly detection using deep active learning

Abstract Anomaly detection algorithms are typically applied to static, unchanging, data features hand-crafted by the user. But how does a user systematically craft good features for anomalies that have never been seen? Here we couple deep learning with active learning – in which an Oracle iteratively labels small amounts of data selected algorithmically over a series of rounds – to automatically and dynamically improve the data features for efficient outlier detection. This approach, Anomaly Hunt (ahunt), shows excellent performance on MNIST, CIFAR10, and Galaxy-DECaLS data, significantly outperforming both standard anomaly detection and active learning algorithms with static feature spaces. Beyond improved performance, ahunt also allows the number of anomaly classes to grow organically in response to the Oracle’s evaluations. Extensive ablation studies explore the impact of Oracle question selection strategy and loss function on performance. We illustrate how the dynamic anomaly class taxonomy represents another step towards fully personalized rankings of different anomaly classes that reflect a user’s interests, allowing the algorithm to learn to ignore statistically significant but uninteresting outliers (e.g. noise). This should prove useful in the era of massive astronomical data sets serving diverse sets of users who can only review a tiny subset of the incoming data.

MAD-EN: Microarchitectural Attack Detection Through System-Wide Energy Consumption

Microarchitectural attacks have become increasingly threatening the society with diverse set of attacks such as Spectre and Meltdown. Vendor patches cannot keep up with the pace of the new threats, which makes the dynamic anomaly detection tools more evident than before. Unfortunately, hardware performance counters (HPCs) utilized in previous works can detect a few microarchitectural attacks due to the small number of counters that can be profiled concurrently while introducing high performance overhead. These challenges consequently yield to inefficient detection tools in real-world security-critical systems. In this study, we introduce MAD-EN dynamic detection tool that leverages system-wide energy consumption traces collected from a generic Intel RAPL tool to detect ongoing anomalies in two different microarchitectures, namely Intel Comet Lake and Intel Tiger Lake. In the first phase of MAD-EN, we can distinguish 16 variants from 11 different micro-architectural attacks from benign applications by utilizing a binary-class CNN-based model with an F1 score of 0.998, which makes our tool the most generic attack detection tool so far. In the second phase, MAD-EN can recognize the respective attack types with a 95% accuracy by utilizing a multi-class CNN-based classification technique after the anomaly is detected. We demonstrate that MAD-EN introduces 69.3% less performance overhead compared to performance counter-based detection mechanisms, allowing more feasible real-time detection tool for generic purpose systems.

A Novel Architecture for Network Intrusion Detection and Prevention

This paper affords an investigation, regarding experiments, which suggests that modern-day network intrusion, detection, and prevention systems (NIDPSs) have numerous shortcomings in detecting or stopping growing undesirable traffic and feature numerous threats in high-pace environments. It suggests that the NIDPS overall performance may be vulnerable withinside the face of high-pace and high-load malicious traffic in phrases of packet drops, high-quality packets without analysis, and failing to detect/save you undesirable traffic. A novel exceptional of service (QoS) structure has been designed to growth the intrusion detection and prevention overall performance. Our studies have proposed and evaluated an answer the usage of a unique QoS configuration in a multi-layer transfer to arrange packets/ traffic and parallel strategies to growth the packet processing pace. The new structure turned into examined below extraordinary traffic speeds, types, and tasks. The experimental consequences display that the structure improves the community and protection overall performance that is can cowl up to eight Gb/s with zero packets dropped. This paper additionally suggests that this number (8Gb/s) may be improved, however it relies upon at the gadget ability that is constantly limited. Keywords: Security, intrusion, prevention, Dynamic Anomaly, Network Attacks, Computer security, computer networks,

A Novel Spatiotemporal Process Feature Learning Method Based On the Pseudo-Siamese Network for Complex Chemical Process Concurrent Condition Monitoring.

The deep learning-based process monitoring method has attracted great attention due to its ability to deal with nonlinear correlation. However, the further modeling of learned deep features from process data to better depict typical process features to obtain more precise monitoring results remains a challenge. In this paper, a novel nonlinear spatiotemporal process feature learning method is proposed to extract high-value slow-varying spatiotemporal process features, with an explicit temporal relationship model for the concurrent monitoring of the static deviation and the dynamic anomaly of complex chemical processes. Different from directly mixed spatiotemporal information methods, the pseudo-Siamese autoencoder network is designed with two different subencoders to separately describe the nonlinear spatial and temporal relationships of the nonlinear dynamic input data. Correspondingly, a cost function including three losses and one orthogonal constraint is proposed to make sure that the extracted spatiotemporal process features change as slowly as possible and contain the most nonlinear dynamic information on the input data. With the explicit spatial and temporal relationship submodel, predictions are utilized to shrink the variability of the nonlinear temporal correlated data and focus on the unpredictable variabilities to improve process monitoring performance. Meanwhile, the linear dynamic information is further extracted in the reconstructed residual space by the general slow feature analysis (SFA) method to provide a more detailed analysis of the process characteristics and improve the monitoring results. The case study monitoring results demonstrate the effectiveness and superiority of the proposed method over other compared methods for concurrent process monitoring.

A dynamic anomaly detection method of building energy consumption based on data mining technology

Due to the equipment failure and inappropriate operation strategy, it is often difficult to achieve energy-efficient building. Anomaly detection of building energy consumption is one of the important approaches to improve building energy-saving. The great amounts of energy consumption data collected by building energy monitoring platforms (BEMS) provides potentials in using data mining technology for anomaly detection. This study proposes a dynamic anomaly detection algorithm for building energy consumption data, which realizes the dynamic detection of point anomalies and collective anomalies. The algorithm integrates unsupervised clustering algorithm with supervised algorithm to establish a semi-supervised matching mechanism, which avoids the influence of error label and improves the efficiency of anomaly detection. A particle swarm optimization (PSO) is used to optimize the unsupervised clustering algorithm. This investigation tests the effectiveness of the proposed algorithm and evaluates the performance of the energy consumption clustering algorithm by using the annual electricity consumption data of an experimental building in a university. The results show that the clustering accuracy of the algorithm can reach more than 80%, and it can effectively detect the building energy consumption data of two different forms of outliers. It can provide reliable data support for adjusting building management strategies.

Dynamic Graph-Based Anomaly Detection in the Electrical Grid

Given sensor readings over time from a power grid, how can we accurately detect when an anomaly occurs? A key part of achieving this goal is to use the network of power grid sensors to quickly detect, in real-time, when any unusual events, whether natural faults or malicious, occur on the power grid. Existing bad-data detectors in the industry lack the sophistication to robustly detect broad types of anomalies, especially those due to emerging cyber-attacks, since they operate on a single measurement snapshot of the grid at a time. New ML methods are more widely applicable, but generally do not consider the impact of topology change on sensor measurements and thus cannot accommodate regular topology adjustments in historical data. Hence, we propose <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DynWatch</small> , a domain knowledge based and topology-aware algorithm for anomaly detection using sensors placed on a dynamic grid. Our approach is accurate, outperforming existing approaches by 20 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> or more (F-measure) in experiments; and fast, averaging less than 1.7 ms per time tick per sensor on a 60K+ branch case using a laptop computer, and scaling linearly with the size of the graph.

A novel approach for anomaly detection in automatic meter intelligence system using machine learning and pattern recognition

Anomaly detection for sensor systems is one of the most researched topics for the Internet of Thing systems. Researchers have been attracted to machine learning classification problems that are considered the most effective techniques. The novel model is proposed by combining anomaly pattern Symbolic Aggregate Approximation (SAX), processing imbalance data and machine learning techniques for sensor anomaly detection. The advantage of anomaly patterns and machine learning leads to the the proposed model to have better performance. The proposed model consists of three phases: finding anomaly pattern features, processing imbalanced data, exploring data by machine learning model. In this paper, the main contributions with respect to previous works can be listed as follows: (i) Successful modeling the new method of SAX for time series data for finding complex and dynamic anomaly patterns. (ii) Archiving applied anomaly pattern feature into machine learning model Random Forest and hyperparameters optimisation of these model. (iii) Fitfully proposed a model combining SAX, imbalance technique, and random forest to anomaly detection. (iv) Achieving applied proposal model in automatic meter intelligence system in Vietnam. The experiential results of the proposed model have described the robustness and better performance for detecting anomalies of power meter sensors.

Automated food safety early warning system in the dairy supply chain using machine learning

Traditionally, early warning systems for food safety are based on monitoring targeted food safety hazards. Optimal early warning systems, however, should identify signals that precede the development of a food safety risk. Moreover, such signals could be identified in factors from domains adjacent to the food supply chain, so-called drivers of change and other indicators. In this study, we show for the first time that such drivers and indicators may indeed represent signals that precede the detection of a food safety risk. The dairy supply chain in Europe was used as an application case. Using dynamic unsupervised anomaly detection models, anomalies were detected in indicator data expected by domain experts to impact the development of food safety risks in milk. Additionally, a Bayesian network was used to identify the chemical food safety hazards in milk associated with an anomaly for the Netherlands. The results showed that the frequency of anomalies varied per country and indicator. However, all countries showed in the period investigated (2008–2019), anomalies in the indicators “raw milk price” and “barely milk price” and no anomalies in the indicator” income of dairy farms”. A cross-correlation analysis of the number of Rapid Alert for Food and Feed (RASFF) notifications and anomalies in indicators revealed significant correlations of many indicators but difference between countries was observed. Interesting, for all countries the cross corelation with indicator “milk price” was significant, albeit the lag time varied from 5 months (United Kingdom) to 22 months (Italy). This finding suggests that severe changes in domains adjacent to the food supply chain may trigger the development of food safety problems that become visible many months later. Awareness of such relationships will provide the opportunity for food producers or inspectors to take timely measures to prevent food safety problems. • Anomalies in prices, average mean temperature and precipitation may be used as early warning factors for food safety. • Anomalies in milk prices precedes food safety issues in milk. • Bayesian network was developed to identify the food safety hazards. • An early warning system for milk supply chain in Europe was developed. • Real-time anomaly detection system in supply chains for food authorities and inspectors.

Bayesian Filtering for Dynamic Anomaly Detection and Tracking

This article presents a Bayesian approach for sequential detection of anomalies in the motion of a target and joint tracking. The anomaly is modeled as a binary (on/off) switching unknown control input that goes into action (begins to exist, or “switches on”) thereby modifying the object dynamics; and by ceasing its activity (becoming nonexistent, or “switching off”) returns the dynamics to nominal. The developed Bayesian framework brings together random finite set (RFS) theory to represent the switching nature of the anomaly, and optimal joint input and state estimation to sequentially update a hybrid state that incorporates a random vector for the kinematic state and a Bernoulli RFS for the unknown control input. In addition, a closed-form solution, the Gaussian-mixture hybrid Bernoulli filter (GM-HBF), has been developed to provide a customized solution for dynamic anomaly detection in the maritime domain characterized by linear Gaussian target dynamics. Based on the Ornstein–Uhlenbeck dynamic model for vessels, where the evolution of the object velocity is governed by a piecewise mean-reverting stochastic process, the anomaly can be represented by a change in the long-run mean velocity (i.e., the unknown control input) from the nominal condition that forces the vessel to deviate from its standard route. We demonstrate the effectiveness of the proposed GM-HBF in both simulated and real-world maritime surveillance applications and test its performance in the face of false measurements, detection uncertainty, and sensor data gaps.

Dynamic Anomaly Detection With High-Fidelity Simulators: A Convex Optimization Approach

The main objective of this article is to develop scalable dynamic anomaly detectors with high-fidelity simulators of power systems. On the one hand, models in high-fidelity simulators are typically “intractable” if one opts to describe them in a mathematical formulation in order to apply existing model-based approaches from the anomaly detection literature. On the other hand, pure data-driven methods developed primarily in the machine learning literature neglect our knowledge about the underlying dynamics of power systems. In this study, we combine tools from these two mainstream approaches to develop a data-assisted model-based diagnosis filter utilizing both the knowledge from a picked abstract model and also the data of simulation results from high-fidelity simulators. The proposed diagnosis filter aims to achieve two desired features: (i) performance robustness with respect to model mismatch; (ii) high scalability. To this end, we propose a tractable (convex) optimization-based reformulation in which decisions are the filter parameters, the model-based information introduces feasible sets, and the data from the simulator forms the objective function to-be-minimized regarding the effect of model mismatch on the filter performance. To validate the theoretical results, we implement the developed diagnosis filter in DIgSILENT PowerFactory to detect false data injection attacks on the Automatic Generation Control measurements in the three-area IEEE 39-bus system.

DyD2: Dynamic Double anomaly Detection Application to on-board space radiation faults

Anomaly detection is a crucial aspect of embedded applications. However, limited computational power, evolving environments, and lack of training data are difficulties that can limit anomaly detection algorithms. One class classification algorithms are often used for this task to circumvent the need of anomalous data in the training set. This paper presents a new machine learning algorithm for anomaly detection called Dynamic Double anomaly Detection DyD2that is suited to evolving environments and on-board requirements. The contributions made by DyD2 are thoroughly presented and an experimental evaluation is set up to compare DyD2 to state-of-the-art algorithms.

Water dynamics in C–S–H and M-S-H cement pastes: A revised jump-diffusion and rotation-diffusion model

The Quasi-Elastic Neutron Scattering (QENS) spectra from four cement pastes are re-analyzed by a new revised jump-diffusion and rotation-diffusion model (rJRM). From the QENS fit, it can be seen that the rJRM is an improved model to fit QENS spectra within the whole detected neutron energy transfer and scattering vector. By the rJRM fitting, the structure parameters extracted from QENS spectra show that both the additives aluminum-silicate nanotubes (ASN) and carboxyl group functionalized ASN (ASN-COOH) can improve magnesium-silicate-hydrate (M-S-H) toward calcium-silicate-hydrate (C–S–H) direction in mechanical properties, but the improvement is weakened with decreasing temperatures. The extracted dynamical parameters show that there is a dynamic anomaly near 230 K in not only translational but also rotational diffusion of water confined in all the investigated samples. The anomaly in rotational diffusion is new compared with those results obtained by the QENS fit using other models.

XFinder: Detecting Unknown Anomalies in Distributed Machine Learning Scenario

In recent years, the emergence of distributed machine learning has enabled deep learning models to ensure data security and privacy while training efficiently. Anomaly detection for network traffic in distributed machine learning scenarios is of great significance for network security. Although deep neural networks have made remarkable achievements in anomaly detection for network traffic, they mainly focus on closed sets, that is, assuming that all anomalies are known. However, in a real network environment, unknown abnormalities are fatal risks faced by the system because they have no labels and occur before the known anomalies. In this study, we design and implement XFinder, a dynamic unknown traffic anomaly detection framework in distributed machine learning. XFinder adopts an online mode to detect unknown anomalies in real-time. XFinder detects unknown anomalies by the unknowns detector, transfers the unknown anomalies to the prior knowledge base by the network updater, and adopts the online mode to report new anomalies in real-time. The experimental results show that the average accuracy of the unknown anomaly detection of our model is increased by 27% and the average F1-Score is improved by 20%. Compared with the offline mode, XFinder’s detection time is reduced by an average of approximately 33% on three datasets, and can better meet the network requirement.