Statistical Anomaly Research Articles

Statistical Anomaly Detection for Enhanced Cybersecurity Using AI-Based Wireless Networks

Statistical Anomaly Detection for Enhanced Cybersecurity Using AI-Based Wireless Networks

Dirichlet probability navigated fault detection via key-group memory auto-encoder under non-stationary working conditions

Real-time condition monitoring is the foundation of prognostics and health management (PHM) for mechanical systems. Constraint by extravagant cost and insurmountable obstacles to simulate real faults in actual working conditions, establishing fault detection models with historical normal data is the most promising way. Meanwhile, non-stationary fault detection is rarely studied in the literature but fits most in real working conditions. Thus, an innovative method named KGMem-DirAE is proposed to conduct real-time fault detection under non-stationary working conditions using normal data only. In the training stage, normal patterns with different semantics are recorded in the developed key-group memory module (KGMem) combined with auto-encoder. Considering the matching probabilities of encoded latent features and recorded memory units (normal patterns), a statistical anomaly score defined by aggregated negative log-likelihood of Dirichlet distributions is proposed to detect incipient failures. In order to deal with time-varying working conditions and capture fault degradation trends concurrently, normalized time-frequency maps containing fault-evolving information are obtained from vibration data. Both widely studied non-stationary and stationary run-to-failure data sets are deeply researched using the proposed KGMem-DirAE and the experimental results proved its superiority against other comparative anomaly detection methods.

Lower-ionosphere anomalies prior to strong earthquakes that occurred in north-central mainland Greece on March 2021 as revealed by multi-method analysis of VLF sub-ionospheric propagation data

In this work we present the multi-method analysis of very low frequency (VLF) data, acquired by the radio receiver with call name UWA, located in Athens (Greece), in the University of West Attica, focusing on two strong ( ) earthquakes (EQs) that occurred in north-central mainland Greece sequentially, on 3 and 4 March 2021, with very close epicenters. Specifically, we used the data acquired from seven VLF transmitters located in Europe, North/North-West to UWA, and their propagation paths include the specific EQs epicenters. We analyzed these data using multiple analysis methods in order to investigate for possible EQ-related anomalies, taking also into account all the other possibly ionosphere-influencing extreme events that occurred during the studied period. Especially, we applied the “nighttime fluctuation method” (NFM), as well as, the “terminator time method” (TTM) in order to reveal any statistical anomaly in the nighttime amplitude recordings of VLF sub-ionospheric propagation data within 15 days before each one examined EQs. Also, we calculated the scalogram (wavelet power spectrum over time) using Morlet mother wavelet of the same nighttime data searching for possible imprints of wave-like structures during the same time period. In terms of criticality analysis, first we applied the “natural time” (NT) analysis method to the daily-valued NFM VLF propagation quantities, and subsequently applied the “method of critical fluctuations” (MCF) to the raw nighttime amplitude VLF recordings, to check for any criticality signatures up to two weeks before the examined EQs. Taking into account all the above-mentioned analysis results, we conclude that there are multiple indications that the lower ionosphere was indeed disturbed due to the preparation processes of the above-mentioned EQs, offering different types of seismogenic indications.

A Security Scheme for Statistical Anomaly Detection and the Mitigation of Rank Attacks in RPL Networks (IoT Environment)

A Routing Protocol for Low-power-lossy (RPL) networks builds a Destination Oriented Directed Acyclic Graph (DODAG) to provide IPv6 connectivity for resource-constrained devices over a large variety of low-power-lossy link layer technologies. Each RPL node maintains a rank value, which quantizes its relative topological distance from the DODAG root and is calculated based on the rank of its preferred parents and the objective function being employed. The RPL routing process does not impose any check to monitor the action and conduct of the parent nodes. A malicious attacking node can exploit this weakness by faking its rank value to be much lower than the original to attract more traffic to traverse through it from its neighboring and underlying child nodes. An attacking node can choose to perform selective forwarding or a sinkhole attack (Rank Attack type 1 – RA1) or exacerbate network performance parameters by causing topological instability (Rank Attack type 2 - RA2). This paper presents the Statistically-based Anomaly Detection Scheme (SARPL) to detect RA1 and RA2 and attempts to mitigate their effects. The simulations and performance evaluations show that SARPL can successfully detect RA1 attacks in all scenarios whereas it has a positive detection rate of approximately 93% for RA2 type attacks. SARPL also significantly improves network performance parameters, such as packet delivery rate and end-to-end delay, while mitigating the effects of RA1 and RA2.

The effect on the five FDA soft contact lens material types of prolonged exposure to phospholipid-based eye drops.

The aim of this study was to assess the effects of regular phospholipid tear supplement application on five soft contact lens materials (FDA Types I-V). Twenty unworn lenses of each lens type were assessed and given an identification number. An initial set of measurements was made on each lens before being immersed in the phospholipid tear supplement solution for one hour. After one hour, the lens was removed, gently rinsed in saline, and the measurements were repeated. This process was repeated for each lens, of each type. Each lens parameter was assessed in the sequence: Refractive Index (% H2O), Dioptric Power (BVP), Lens Diameter (OD), and Lens Curvature (BCOR). All measurements of % H2O, BVP, OD and BCOR did not differ between baseline and after lenses being immersed in the phospholipid tear supplement solution for one hour [all soft contact lens materials (FDA Types I-V) p > 0.05]. However, in the type IV lenses, a statistically significant change in the % H2O was found. Only a 1% reduction was observed, and, in the investigator's opinion, the significance of these results is due more to a statistical anomaly produced by having very repeatable measurements showing very little variation. The phospholipid tear supplement solution did not adversely change the physical parameters of unworn soft contact lenses, of the five FDA Types, to the extent that would be considered clinically significant.

DDoS Attack Detection Using Enhanced Neural Network Algorithm in Software-Defined Networks

With the uncommon advancement of technology and networking, the world today is totally relying on it. The most obvious drawback of this technical progress resulted in several possible dangers. A Distributed Denial of Service (DDoS) attack is one in which a large number of compromised systems work together to prevent service from being provided to the targeted system. Consequently, to protect network servers, attack detection systems must be very effective. The proposed architecture has Software Defined Networks (SDN) which comprise controllers and SFlow agents. According to the article, the anomaly detection of statistical traffic, which is performed on both normal and pathological anomaly traces in the packet header, as well as traffic volume detection, is based on the suggested work provided in this study. The k-means clustering technique is used by the statistical anomaly detection system and the attack alert aggregation system. Low-level attack detection systems are used to generate cluster dissimilar warnings, which are then investigated further. Clusters generate meta-alerts based on the information they have gathered. After that, the report of meta-alerts is sent to the security specialists. The online alert aggregation technique, which is also known as the probabilistic model, is used to identify new assaults on a system. The k-means clustering method is used to improve the quality of the traffic data streams. The Enhanced Neural Network Algorithm (ENNA) is being used to advance the intelligent attack detection system, which is currently under development. It is utilized in open-day controllers to identify attacks with a 99.7% accuracy by using Mininet and the Python simulation tool (ODL). In future work, it is possible to evaluate how the suggested detection method would be used in the event of subnet attacks and its mitigation.

General-purpose Unsupervised Cyber Anomaly Detection via Non-negative Tensor Factorization

Distinguishing malicious anomalous activities from unusual but benign activities is a fundamental challenge for cyber defenders. Prior studies have shown that statistical user behavior analysis yields accurate detections by learning behavior profiles from observed user activity. These unsupervised models are able to generalize to unseen types of attacks by detecting deviations from normal behavior without knowledge of specific attack signatures. However, approaches proposed to date based on probabilistic matrix factorization are limited by the information conveyed in a two-dimensional space. Non-negative tensor factorization, however, is a powerful unsupervised machine learning method that naturally models multi-dimensional data, capturing complex and multi-faceted details of behavior profiles. Our new unsupervised statistical anomaly detection methodology matches or surpasses state-of-the-art supervised learning baselines across several challenging and diverse cyber application areas, including detection of compromised user credentials, botnets, spam e-mails, and fraudulent credit card transactions.

Some Years are Just Better than Others. The Importance of Public Consensus in Record Ranking

ABSTRACT Two periods, October 1973 through April 1976 and January 1986 through December 1989 – one-quarter and 40% of their respective decades – curiously lack strongly charting records. This is not a statistical anomaly nor an artifact of one national chart; records in those periods spent less time at their peaks, thus scoring less than records in closely matched comparator high-scoring adjacent periods. The lack of enduring peak popularity, churn at the top of the chart, and the absence of a dominant genre indicate a lack of public consensus as to “what’s good right now,” accounting for the anomalously low scores in those eras.

Statistical Tests for Comparing Machine Learning Algorithms

Abstract: The average result of machine learning models is determined using naive 10-fold cross-validation, paired Student's ttest, McNemar's test. The algorithm with the best average performance should outperform those with the worst. But what if the difference in average results is due to a statistical anomaly? A statistical hypothesis test is used to determine whether the mean results of the differences between the three algorithms are real. Using statistical hypothesis testing, this study will show how to compare machine learning algorithms. When choosing a model, the output of several machine learning algorithms or simulation channels is compared. The model that performs best based on your performance measure becomes the final model that can be used to predict new data. With classification and regression prediction models, this can be done using traditional machine learning and deep learning methods. It is difficult to determine whether the difference between the two models is accurate or not.

Neurodevelopmental Problems Not Linked to Antidepressant Exposure In Utero

Neurodevelopmental Problems Not Linked to Antidepressant Exposure In Utero

Isolating and targeting a highly active, stochastic dendritic cell subpopulation for improved immune responses.

Dendritic cell (DC) activation via pathogen-associated molecular patterns (PAMPs) is critical for antigen presentation and development of adaptive immune responses, but the stochastic distribution of DC responses to PAMP signaling, especially during the initial stages of immune activation, is poorly understood. In this study, we isolate a unique DC subpopulation via preferential phagocytosis of microparticles (MPs) and characterize this subpopulation of "first responders" (FRs). We present results that show these cells (1) can be isolated and studied via both increased accumulation of the micron-sized particles and combinations of cell surface markers, (2) show increased responses to PAMPs, (3) facilitate adaptive immune responses by providing the initial paracrine signaling, and (4) can be selectively targeted by vaccines to modulate both antibody and Tcell responses invivo. This study presents insights into a temporally controlled, distinctive cell population that influences downstream immune responses. Furthermore, it demonstrates potential for improving vaccine designs via FR targeting.

Plant health index as an anomaly detection tool for oil refinery processes

Early detection of significant abnormal changes is highly desirable for oil refinery processes, which consist of sophisticated unit operations handling hazardous and flammable inventories and operating at high temperature and pressure. Close monitoring and anomaly detection are vital for avoiding major accidents and losses and enable intervention before failure occurrence. A new big data analytics tool called Plant Health Index (PHI) is proposed in this work. PHI is a statistical anomaly detection software that trains its model using online normal plant operation, then uses statistical analytics to detect anomalies. For detecting the anomalies, a combined method of multivariate analysis of residuals and nonparametric models of the process is employed. The methodology provides a structured representation of the plant variables to ease the detection of problems along with the detection of operation changes of the system. The PHI system has been tested on a hydrotreating units in a refinery, which consists of catalytic reactors and separators. The current implementation tagged 170 process variables and proved effective in capturing the normal operational conditions of the plant. When placed online, PHI was able of detecting anomalies that are difficult to detect using the control system and before being detected by the alarm system.

Underdispersion: A statistical anomaly in reported Covid data.

Throughout the Covid‐19 pandemic, we have become used to seeing daily numbers of cases and deaths go up and down. But in some countries, the reported numbers show very little movement over days and weeks – they are “underdispersed”, says Dmitry Kobak, and this may be a sign that all is not right with the data

A Statistical Feature-Based Anomaly Detection Method for PFC Using Canonical Correlation Analysis

Power factor correction (PFC) converters are widely used in power systems. The anomalous state of a PFC converter can develop into the complete or partial loss of the electric system. In some industrial applications, such as wind power generation and photovoltaic power generation, there is a fluctuation in the input voltage of the PFC converter, which brings great obstacles in anomaly detection. In order to effectively recognize an anomalous state, especially when the input voltage is unstable, a statistical feature-based anomaly detection method using canonical correlation analysis (CCA) is proposed. Firstly, statistical features are used to enhance the difference between the normal state and anomalous state. Then, the proposed anomaly detection method focuses on the correlation between the input voltage and the output voltage, so even if there is a fluctuation in the input voltage, the anomaly states can be detected precisely. The experimental results show the effectiveness of the method.

A New Model-Data Co-Driven Method for Radar Ship Detection

Target detection played a vital role in radar image interpretation. The early studies usually relied on the model-driven strategy, such as the constant false alarm rate technique. This family of a method aimed to find the statistical anomaly points. The detection performance was, therefore, dependent on the statistical modeling of clutter. They suffered from the model mismatch in realistic scenarios. Recent years have witnessed a resurgence of deep neural networks, by which the high-level representations can be learned. They achieved amazing performance in many fields. However, large amounts of training samples were required to estimate the weights and the bias. For radar sensors, it is costly and difficult to collect much data with label information. To solve the problem, this article presents a model-data co-driven ship detection strategy. The clutter of radar images is first modeled by the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${\mathcal{ G}}$ </tex-math></inline-formula> distribution. The scale and the shape parameters are estimated by the data available. They are used to simulate the sea clutters. Second, the labeled targets available are extended to several different azimuths to construct the target set. Finally, the targets randomly chosen from the target set are embedded into the sea clutter. The synthesized images and the original ones are combined to train the deep architecture. Multiple comparative studies are performed to demonstrate the advantage of the proposed strategy.

An New Efficient Cluster Based Detection Mechanisms for Distributed Denial of Services (DDoS) Attacks

Cluster aggregation of statistical anomaly detection is a mechanism for defending against denial of service attack (dos) and distributed denial-of-service (DDoS) attacks. DDoS attacks are treated as a congestioncontrol problem; because most of the congestion is occurred in the malicious hosts not follow the normal endto- end congestion control. Upstream routers are also notified to drop such packets in order that the router’s resources are used to route legitimate traffic hence term cluster aggregation. If the victim suspects that the cluster aggregations are solved by most of the clients, it increases the complexity of the cluster aggregation. This aggregation solving technique allows the traversal of the attack traffic throughout the intermediate routers before reaching the destination. In this proposal, the aggregation solving mechanism is cluster aggregation to the core routers rather than having at the victim. The router based cluster aggregation mechanism checks the host system whether it is legitimate or not by providing a aggregation to be solved by the suspected host.

Mitigating TCP SYN flooding based EDOS attack in cloud computing environment using binomial distribution in SDN

Cloud Computing provides an auto-scaling feature for dynamic resource utilization to cope with their customers’ requirements and charge as ‘pay-per-use’. Attackers get the benefit of this auto-scaling feature by flooding DDoS attacks on the VMs or instances in the cloud environments. Such DDoS attacks give financial damages to the customers because of massive amount of resource utilization, known as the Economic Denial of Sustainability (EDOS) attack. Transmission Control Protocol (TCP) SYN flooding attack is known to be the most challenging attack resulting EDOS attack. Software Defined Network (SDN) being a cost-efficient solution for cloud service providers, uses the OpenFlow switches and flow tables to make rules for each incoming flow. In SDN, it is feasible to classify an incoming flow as an attack and block forwarding of this flow to the targeted VM. This research work proposes an SDN based fast and computationally cost-efficient statistical anomaly detection model, named EDOS-TCP SYN mitigation model (EDOS-TSM) to mitigate TCP SYN flooding attack from a single user and spoofed IPs. EDOS-TSM uses binomial probability, TTL field value of IP packet header and multi-TCP SYN requests to detect source-based and spoofing based attacks. The proposed model is implemented, and the performance is evaluated on an OpenStack production-based cloud with real-time traffic and attack generation. The attack mitigation results are compared with existing models in the literature. The results show a large number of false negatives in existing models and efficiency of EDOS-TSM is proved in both source based and spoofing attacks.

Utilizing Statistical Tests for Comparing Machine Learning Algorithms

The mean result of machine learning models is determined by utilizing k-fold cross-validation. The algorithm with the best average performance should surpass those with the poorest. But what if the difference in average outcomes is the consequence of a statistical anomaly? To conduct whether or not the mean result differences between two algorithms is genuine then statistical hypothesis test is utilized. Using statistical hypothesis testing, this study will demonstrate how to compare machine learning algorithms. The output of several machine learning algorithms or simulation pipelines is compared during model selection. The model that performs the best based on your performance measure becomes the last model, which can be utilized to make predictions on new data. With classification and regression prediction models it can be conducted by utilizing traditional machine learning and deep learning methods. The difficulty is to identify whether or not the difference between two models is accurate.

Watermarks in stream processing systems

Streaming data processing is an exercise in taming disorder: from oftentimes huge torrents of information, we hope to extract powerful and timely analyses. But when dealing with streaming data, the unbounded and temporally disordered nature of real-world streams introduces a critical challenge: how does one reason about the completeness of a stream that never ends? In this paper, we present a comprehensive definition and analysis of watermarks , a key tool for reasoning about temporal completeness in infinite streams. First, we describe what watermarks are and why they are important, highlighting how they address a suite of stream processing needs that are poorly served by eventually-consistent approaches: • Computing a single correct answer, as in notifications. • Reasoning about a lack of data, as in dip detection. • Performing non-incremental processing over temporal subsets of an infinite stream, as in statistical anomaly detection with cubic spline models. • Safely and punctually garbage collecting obsolete inputs and intermediate state. • Surfacing a reliable signal of overall pipeline health . Second, we describe, evaluate, and compare the semantically equivalent, but starkly different, watermark implementations in two modern stream processing engines: Apache Flink and Google Cloud Dataflow.

Multicast on‐route cluster propagation using to identify the network intrusion detection system in mobile ad hoc network

SummaryIntrusion detection, become a new paradigm in mobile ad hoc network which provide a sense of security to the existing computer and data network, is a retro‐fit approach. The goal of intrusion detection, unauthorized use, misuse, and by insiders and both external penetrator system is to identify the abuse of computer systems. The increase of the connection of the computer system will provide greater access to outsiders; for the intruder, the identification of avoidance becomes a challenge. We propose a multicast on‐route cluster propagation (MORCP) based network intrusion detection system to improve the security based on route analysis. The number of IDS of the prototype has been developed in a number of transmissions on miss‐behavioral transmission. Some of them also have been experimentally deployed in a production system. In addition, segregate path identification algorithm (SPIA) of host‐based and network‐based IDS is investigated, and the characteristics of the corresponding system are identified. In host‐based systems, most of the network‐based IDS, to construct a detection mechanism on the network traffic monitored, as well, while some adoption host audit trail, in order to detect the intrusion activity, input as the main source of information, to adopt an audit trail of the hot operating system. The overview of typical use statistical anomaly detection algorithm in IDS is also included.