Detection Of Anomalous Patterns Research Articles

Threat Detection and Response Using AI and NLP in Cybersecurity

Introduction: In an age of rapid technical innovation and a growing digital world, protecting sensitive data from cyberattacks is crucial. The dynamic and complicated nature of these attacks requires novel cybersecurity solutions. Methods: This study analyses how Artificial Intelligence (AI) and Natural Language Processing (NLP) strengthen cybersecurity. The qualitative research approach is followed to gather data through a literature review of relevant scholarly articles and conduct interviews with cybersecurity specialists. Results: Recent AI advances have greatly enhanced the detection of anomalous patterns and behaviors in huge datasets, a key threat identification tool. NLP has also excelled at detecting malevolent intent in textual data, such as phishing efforts. AI and NLP enable adaptive security policies, enabling agile responses to evolving security issues. Expert interviews confirm that AI and NLP reduce false positives, improve threat intelligence, streamline network security setups, and improve compliance checks. These technologies enable responsive security policies, which give a strategic edge against developing security threats. AI and NLP's predictive skills could revolutionize cybersecurity by preventing threats. Conclusion: This study shows that AI and NLP have improved cybersecurity threat detection, automated incident response, and adaptive security policies. Overcoming threat detection, aggressive attacks and data privacy issues is essential to properly leveraging these advances and strengthening cyber resilience in a changing digital landscape.

Rule-Based Detection of Anomalous Patterns in Device Behavior for Explainable IoT Security

Rule-Based Detection of Anomalous Patterns in Device Behavior for Explainable IoT Security

MFFA: Multi-level feature fusion and anomaly map compensation for anomaly detection

Embedding similarity-based methods obtained good results in unsupervised anomaly detection (AD). This kind of method usually used feature vectors from a model pre-trained by ImageNet to calculate scores by measuring the similarity between test samples and normal samples. Ultimately, anomalous regions are localized based on the scores obtained. However, this strategy may lead to a lack of sufficient adaptability of the extracted features to the detection of anomalous patterns for industrial anomaly detection tasks. To alleviate this problem, we design a novel anomaly detection framework, MFFA, which includes a pseudo sample generation (PSG) block, a local-global feature fusion perception (LGFFP) block and an anomaly map compensation (AMC) block. The PSG block can make the pre-trained model more suitable for real-world anomaly detection tasks by combining the CutPaste augmentation. The LGFFP block aggregates shallow and deep features on different patches and inputs them to CaiT (Class-attention in image Transformers) to guide self-attention, effectively interacting local and global information between different patches, and the AMC block can compensate each other for the two anomaly maps generated by the nearest neighbor search and multivariate Gaussian fitting, improving the accuracy of anomaly detection and localization. In experiments, MVTec AD dataset is used to verify the generalization ability of the proposed method in various real-world applications. It achieves over 99.1% AUROCs in detection and 98.4% AUROCs in localization, respectively.

Process-Oriented heterogeneous graph learning in GNN-Based ICS anomalous pattern recognition

Over the past few years, massive penetrations targeting an Industrial Control System (ICS) network intend to compromise its core industrial processes. So far, numerous advanced methods have been proposed to detect anomalous patterns in the numeric data streams with respect to the heterogeneous field devices involved in the industrial processes. These methods, despite reporting decent results, usually conduct system-wise detection instead of fine-grained anomalous pattern recognition at the device level. Furthermore, lacking explicit consideration of the exclusive process-related features with respect to each differentiated device, the fitness of their application in specified industrial processes is undermined. To tackle these issues, a GNN-based Attributed Heterogeneous Graph Analyzer (the AHGA) is designed to perform device-wise anomalous pattern detection via in-depth process-oriented associativity learning. The AHGA’s framework is constructed with four building blocks: a graph processor, a feature analyzer, a link inference decoder, and an anomaly detector. Its performance is assessed and compared against multiple link inference and anomaly detection baselines over 2 popular ICS datasets (SWaT and WADI). Comparative results demonstrate the AHGA’s reliability in capturing sophisticated process-oriented relations among heterogeneous devices as well as its effectiveness in boosting the performance of anomalous pattern recognition at device-level granularity.

Reliability Improvement of Magnetic Corrosion Monitor for Long-Term Applications.

Electromagnetic techniques are widely employed for corrosion detection, and their performance for inspection of corrosion is well established. However, limited work is carried out on the development and reliability of smart corrosion monitoring devices for tracking internal or buried thickness loss due to corrosion remotely. A novel smart magnetic corrosion transducer is developed for long-term monitoring of thickness loss due to corrosion at critical locations. The reliability of the transducer is enhanced by using a dissimilar active redundancy approach. The improved corrosion monitor has been tested in the ambient environment for seven months to evaluate the stability against environmental factors and degradation. The monitor is found to show great sensitivity to detect defects due to corrosion. Detection of anomalous patterns in the time series data received from the monitors is accomplished by using Pearson's correlation coefficient. The critical component of the monitor is identified at the end of the test. Research findings reveal that, compared to the existing corrosion monitoring techniques in the industry, the detection and isolation of faulty sensor features introduced in this study can contribute to reliable monitoring of thickness loss due to corrosion in ferromagnetic structures over an extended period of time.

Calibrated Nonparametric Scan Statistics for Anomalous Pattern Detection in Graphs

We propose a new approach, the calibrated nonparametric scan statistic (CNSS), for more accurate detection of anomalous patterns in large-scale, real-world graphs. Scan statistics identify connected subgraphs that are interesting or unexpected through maximization of a likelihood ratio statistic; in particular, nonparametric scan statistics (NPSSs) identify subgraphs with a higher than expected proportion of individually significant nodes. However, we show that recently proposed NPSS methods are miscalibrated, failing to account for the maximization of the statistic over the multiplicity of subgraphs. This results in both reduced detection power for subtle signals, and low precision of the detected subgraph even for stronger signals. Thus we develop a new statistical approach to recalibrate NPSSs, correctly adjusting for multiple hypothesis testing and taking the underlying graph structure into account. While the recalibration, based on randomization testing, is computationally expensive, we propose both an efficient (approximate) algorithm and new, closed-form lower bounds (on the expected maximum proportion of significant nodes for subgraphs of a given size, under the null hypothesis of no anomalous patterns). These advances, along with the integration of recent core-tree decomposition methods, enable CNSS to scale to large real-world graphs, with substantial improvement in the accuracy of detected subgraphs. Extensive experiments on both semi-synthetic and real-world datasets are demonstrated to validate the effectiveness of our proposed methods, in comparison with state-of-the-art counterparts.

Nonparametric Subset Scanning for Detection of Heteroscedasticity

We propose heteroscedastic subset scan (HSS), a novel method for identifying covariates that are responsible for violations of the homoscedasticity assumption in regression settings. Viewing the problem as one of anomalous pattern detection, we use subset scanning techniques to efficiently identify the subset of covariates that are most “heteroscedastically relevant.” Through simulations and a real data example, we demonstrate that HSS is capable of detecting heteroscedasticity in a wide range of settings, including in cases where existing global tests lack power. Furthermore, the global power of our method compares favorably to methods such as the Breusch–Pagan test. Supplementary materials for this article are available online.

Image based statistical process monitoring via partial first order stochastic dominance

The continuously evolving digitalized manufacturing industry is pushing quality engineers to face new and complex challenges. Quality data formats are evolving from simple univariate or multivariate characteristics to big data streams consisting of sequences of images and videos in the visible or infrared range; manufacturing processes are moving from series production to more and more customized applications. In this framework, novel methods are needed to monitor and keep under statistical control the process. This study presents two novel process monitoring techniques that rely on the partial first order stochastic dominance (PFOSD) concept, applicable to in-line analysis of video image data aiming at signaling out-of-control process states. Being non-parametric, they allow dealing with complex underlying dynamics and wildly varying distributions that represent the natural process conditions. A motivating case study in metal additive manufacturing is presented, where the proposed methodology enables the in-line and in-situ detection of anomalous patterns in thermal videos captured during the production of zinc samples. Performances are investigated and compared in the presence of both simulated and real data.

Deep learning detection of anomalous patterns from bus trajectories for traffic insight analysis

Existing data-driven methods for traffic anomaly detection are modeled on taxi trajectory datasets. The concern is that the data may contain much inaccuracy about the actual traffic situations, because taxi drivers often choose optimal routes to evade from the congestions caused by traffic anomalies. We use bus trajectory data in this work. Bus trajectories can capture real traffic conditions in the road networks without drivers’ preference, which are more objective and appropriate for accurately detecting anomalous patterns for a broad range of insight analyses on traffics. We proposed a deep learning-based feature visualization method to map 3-dimensional features into a red–green–blue (RGB) color space. A color trajectory (CT) is then derived by encoding a trajectory with the RGB colors. With the spatial and temporal properties extracted from the CT, spatio-temporal outliers are detected by a novel offline detection method. We then conduct GIS map fusion to obtain insights for better understanding the traffic anomaly locations, and more importantly the influences on the road affected by the corresponding anomalies. Extended from the offline detection, an online detection method is developed for real-time detection of anomalous patterns. Our proposed methods were tested on 3 real-world bus trajectory datasets to demonstrate the performance of high accuracies, high detection rates and relatively low false alarm rates.

Detecting Pattern Anomalies in Hydrological Time Series with Weighted Probabilistic Suffix Trees

Anomalous patterns are common phenomena in time series datasets. The presence of anomalous patterns in hydrological data may represent some anomalous hydrometeorological events that are significantly different from others and induce a bias in the decision-making process related to design, operation and management of water resources. Hence, it is necessary to extract those “anomalous” knowledge that can provide valuable and useful information for future hydrological analysis and forecasting from hydrological data. This paper focuses on the problem of detecting anomalous patterns from hydrological time series data, and proposes an effective and accurate anomalous pattern detection approach, TFSAX_wPST, which combines the advantages of the Trend Feature Symbolic Aggregate approximation (TFSAX) and weighted Probabilistic Suffix Tree (wPST). Experiments with different hydrological real-world time series are reported, and the results indicate that the proposed methods are fast and can correctly detect anomalous patterns for hydrological time series analysis, and thus promote the deep analysis and continuous utilization of hydrological time series data.

Outlier and anomaly pattern detection on data streams

A data stream is a sequence of data generated continuously over time. A data stream is too big to be saved in memory, and its underlying data distribution may change over time. Outlier detection aims to find data instances which significantly deviate from the underlying data distribution. While most of outlier detection methods work in batch mode where all the data samples are available at once, the necessity for efficient outlier and anomaly pattern detection methods in a data stream has increased. Outlier detection is performed at an individual instance level, and anomalous pattern detection involves detecting a point in time where the behavior of the data becomes unusual and differs from normal behavior. Alternatively, concept drift detection methods find a concept-changing point in the streaming data and try to adapt the model to the new emerging pattern. In this paper, we provide a review of outlier detection, anomaly pattern detection, and concept drift detection for streaming data.

Efficient Discovery of Heterogeneous Treatment Effects in Randomized Experiments via Anomalous Pattern Detection

In the recent literature on estimating heterogeneous treatment effects, each proposed method makes its own set of restrictive assumptions about the intervention’s effects and which subpopulations to explicitly estimate. Moreover, the majority of the literature provides no mechanism to identify which subpopulations are the most affected–beyond manual inspection–and provides little guarantee on the correctness of the identified subpopulations. Therefore, we propose Treatment Effect Subset Scan (TESS), a new method for discovering which subpopulation in a randomized experiment is most significantly affected by a treatment. We frame this challenge as a pattern detection problem where we efficiently maximize a nonparametric scan statistic over subpopulations. Furthermore, we identify the subpopulation which experiences the largest distributional change as a result of the intervention, while making minimal assumptions about the intervention’s effects or the underlying data generating process. In addition to the algorithm, we demonstrate that the asymptotic Type I and II error can be controlled, and provide sufficient conditions for detection consistency–i.e., exact identification of the affected subpopulation. Finally, we validate the efficacy of the method by discovering heterogeneous treatment effects in simulations and in real-world data from a well-known program evaluation study.

Scalable Detection of Anomalous Patterns With Connectivity Constraints

We present GraphScan, a novel method for detecting arbitrarily shaped connected clusters in graph or network data. Given a graph structure, data observed at each node, and a score function defining the anomalousness of a set of nodes, GraphScan can efficiently and exactly identify the most anomalous (highest-scoring) connected subgraph. Kulldorff’s spatial scan, which searches over circles consisting of a center location and its k − 1 nearest neighbors, has been extended to include connectivity constraints by FlexScan. However, FlexScan performs an exhaustive search over connected subsets and is computationally infeasible for k > 30. Alternatively, the upper level set (ULS) scan scales well to large graphs but is not guaranteed to find the highest-scoring subset. We demonstrate that GraphScan is able to scale to graphs an order of magnitude larger than FlexScan, while guaranteeing that the highest-scoring subgraph will be identified. We evaluate GraphScan, Kulldorff’s spatial scan (searching over circles) and ULS in two different settings of public health surveillance. The first examines detection power using simulated disease outbreaks injected into real-world Emergency Department data. GraphScan improved detection power by identifying connected, irregularly shaped spatial clusters while requiring less than 4.3 sec of computation time per day of data. The second scenario uses contaminant plumes spreading through a water distribution system to evaluate the spatial accuracy of the methods. GraphScan improved spatial accuracy using data generated from noisy, binary sensors in the network while requiring less than 0.22 sec of computation time per hour of data.

Anomalous behavior detection in single-trajectory data

This paper presents an original approach to dynamic anomalous behavior detection in individual trajectory using a recursive Bayesian filter. The anomalous pattern detection is of great interest for navigation, driver assistance systems, surveillance as well as crisis management. In this work, we focus on the GPS trajectories of automobiles finding where the driver’s behavior shows anomalies. Such anomalous behaviors can happen in many cases, especially when the driver encounters orientation problems, i.e., taking a wrong turn, performing a detour, or losing the way. First, three high-level features, i.e., turns and their density, detour factor, and route repetition are extracted from the given trajectory geometry, for which a long-term perspective is required to observe data sequences of a significant length instead of individual time stamps. We therefore employ high-order Markov chains with a ‘dynamic memory’ to model the trajectory integrating these long-term features. The Markov model is processed by a proposed recursive Bayesian filter to infer an optimal probability distribution of the potential anomalous driving behaviors dynamically over time. The filter performs unsupervised detection in single trajectories based on local features only. No training process is required to characterize the anomalous behaviors. By analyzing the detection results of individual trajectories, collective behaviors can be derived indicating traffic issues such as congestions and turn restrictions. Experiments are performed on volunteered geographic information (VGI) data, self-acquired trajectories, and open trajectory datasets to demonstrate the potential of the proposed approach.

Detection of Multicast Video Flooding Attack using the Pattern of Bandwidth Provisioning Efficiency

The rapid increase of distributed denial-of-service (DDoS) attacks, which deplete a link bandwidth and service resource by sending a tremendous flooding of traffic to a victim server, has threatened fundamental network service stability. First, this letter analyzes the unique properties of multicast video delivery, which is vulnerable to malicious video flooding attacks. Second, the effectiveness of the observation of bandwidth provisioning efficiency in the detection of anomalous patterns of multicast flooding is discussed and a traffic-sharing density-based flooding detection (TDFD) mechanism is proposed. Numerical analysis and simulation results are given to validate the performance of the TDFD.

Evaluation of the streaming potential effect of piping phenomena using a finite cylinder model

Piping phenomena into the upstream face of an earth dam can be approximated by a finite cylinder model. A formula of the streaming potential (SP) effect of such a model was developed by integrating an equation of an infinite cylinder section. Numerical modelling allows the conclusion that the oblong SP field shape of a model is only for relatively long and shallow piping models. The SP field shape of a model can actually be circular below a certain ratio value between the depth and length of the model. The resulting SP field of a finite cylinder is symmetrical with respect to its middle point, which allows, in some cases, reaching conclusions about piping phenomena outside the measured area. This finite cylinder formula needs smaller values of moments of polarisation to generate SP data almost identical to the SP magnitudes obtained from the expression of an infinite cylinder section. A random component was added to the SP effect of several computed models. Relatively short and deep piping phenomena can pass unnoticed, especially for a low signal-to-noise ratio. However, detection of anomalous patterns can be improved if the measured data are pre-processed according to a “studentized” deviate concept, which is then subjected to a statistical test based on the Chebyshev inequality. A field example of an SP piping process was detected at the upstream face of an earth dam. Its existence was also confirmed by a geoelectrical survey.