Anomalies In Data Research Articles

Feasibility of anomalous event detection based on Mobile Spatial Statistics: A study of six cases in Japan

One of the primary goals of disaster risk management is to minimize the loss of life in the event of a disaster. To reduce the number of victims in disasters is important to know the amount of exposed population to a hazard. Similarly, population distribution in real-time is required to estimate post-disaster needs and resources for disaster relief. In this study, we explore the use of Mobile Spatial Statistics (MSS) as the basis to acquire offline and online population data and detect anomalies that may relate to a disaster. For instance, mobile spatial data could identify sudden changes in the number of people indicating an anomalous event or change in population dynamics. Here we use the Matrix Profile (MP) method in time-series analysis to test the feasibility of detecting anomalies in the MSS data. We selected mass gatherings and disaster events with rapid and slow onset and small and large scales. We discuss the applicability of this method for real-time anomaly detection of spatial population distribution within the context of disaster risk management. Our results show that the applicability of the MP method in anomaly detection is sensitive to the size of the areas used during monitoring and analysis, the subsequence length and the threshold applied.

Domain adaptation via gamma, Weibull, and lognormal distributions for fault detection in chemical and energy processes

AbstractThe burgeoning development of supervised machine learning (ML) has led to its widespread applications in chemical and energy processes, such as fault detection. However, in some scenarios, collecting labelled data can be costly, hazardous, or impossible. Moreover, data of the same process can follow varying distributions due to changes in, for example, devices and environment, causing ML models to be ineffective. These challenges pose a domain adaptation task, necessitating the refinement of existing ML models to tackle issues from related applications. This study proposes a domain adaptation approach to address label scarcity and data distribution variation. The method has three stages: data distribution modelling (knowledge discovery), adaptation of target domain samples to source domains (knowledge transformation), and classifier ensemble for fault detection (knowledge fusion). Gamma, Weibull, and lognormal distributions are applied for data modelling and domain adaptation. The effectiveness of the method is validated on synthetic datasets and then applied to identify anomalies in coal mine pressure data and detect faults in the Tennessee Eastman (TE) process.

Leveraging ML for Anomaly Detection in Healthcare Data Warehouses

The rapid emergence of digitalisation leads to unprecedented growth in the generation of the healthcare sector-particularly EHRs and medical equipment data. This extended the way for challenges for integrity in managing data and anomaly detection, including fraudulent transactions, medication errors, and many more system failures. Modern healthcare data poses a challenge to traditional methods of anomaly detection due to high and complex dimensionality. Machine learning provides a strong solution, using algorithms such as Gaussian Mixture Models, One-Class SVM and deep learning algorithms such as Autoencoders, and Recurrent Neural Networks in the detection of anomalies in healthcare data warehouse settings [1]. This study reports how ML can help advance care for patients, enable the validity of the data and reduce costs through real-time monitoring, fraud detection, and early detection of diseases. Applying anomaly detection through ML would most likely bring better operational performance, patient safety, and decision-making in health care for organizations as issues of poor data quality, lack of interpretability of models, and real-time detection would be addressed [2].

Enhanced Intrusion Detection System Using Deep Learning Algorithms : A Review

Intrusion Detection Systems (IDS) are crucial for protecting network infrastructures from advanced cyber threats. Traditional IDS, largely reliant on static signature detection, fail to effectively counter novel cyber attacks, leading to high false positive rates and missed zero-day exploits. This study investigates the integration of deep learning technologies into IDS to enhance their detection capabilities. By employing advanced deep learning frameworks, including Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) and other algorithms , the research explores their efficacy in identifying complex data patterns and anomalies. Furthermore, the use of big data analytics is assessed for its potential to significantly augment the predictive power of these systems, aiming to set new benchmarks in cybersecurity defenses tailored for contemporary threats.

Application of Artificial Intelligence Technology in Vulnerability Analysis of Intelligent Ship Network

The improvement in transportation efficiency, security, safety, and environmental effects may be possible due to the impending advent of autonomous ships. Automatic situational awareness, risk detection, and intelligent decision-making are the key features of the intelligent ship network, differentiating it from conventional ships. There is an immediate need to implement a system for marine information management and network security due to the growing importance of this field, which poses a risk to national and societal stability due to factors, such as the diversity and complexity of marine information types, the challenges associated with data collection, and other similar factors. By recognizing different vulnerabilities and through research cases of the ship systems and Artificial Intelligence (AI) technologies, this paper presents Adaptive Fuzzy Logic-assisted Vulnerability Analysis of Intelligent Ship Networks (AFL-VA-ISN) in various cyberattack scenarios for autonomous ship intrusion detection and information management. Fuzzy logic has been combined with AI, providing a framework for handling uncertainty and imprecision in intelligent ship networks and effective decision-making. This work presents a method for detecting anomalies in risk data based on the collaborative control structure of the Ship Information System. Maintaining the network security of intelligent ships is the primary focus of this research, which mainly employed multi-sensor nodes to evaluate data containing information about malicious attacks and placed self-execution protection organize generating nodes into place to intercept and protect against attacks. The experimental outcomes demonstrate that the suggested AFL-VA-ISN model increases the data transmission rate by 99.2%, attack detection rate by 98.5%, risk assessment rate by 97.5%, and access control rate of 96.3%, and reduces the network latency rate of 11.4% compared to other existing models.

A Novel Online Hydrological Data Quality Control Approach Based on Adaptive Differential Evolution

The quality of hydrological data has a significant impact on hydrological models, where stable and anomaly-free hydrological time series typically yield more valuable patterns. In this paper, we conduct data analysis and propose an online hydrological data quality control method based on an adaptive differential evolution algorithm according to the characteristics of hydrological data. Taking into account the characteristics of continuity, periodicity, and seasonality, we develop a Periodic Temporal Long Short-Term Memory (PT-LSTM) predictive control model. Building upon the real-time nature of the data, we apply the Adaptive Differential Evolution algorithm to optimize PT-LSTM, creating an Online Composite Predictive Control Model (OCPT-LSTM) that provides confidence intervals and recommended values for control and replacement. The experimental results demonstrate that the proposed data quality control method effectively manages data quality; detects data anomalies; provides suggested values; reduces reliance on manual intervention; provides a solid data foundation for hydrological data analysis work; and helps hydrological personnel in water resource scheduling, flood control, and other related tasks. Meanwhile, the proposed method can also be applied to the analysis of time series data in other industries.

A Review on Leveraging Machine Learning for Anomaly Detection in Cloud Cost Management

Cloud computing has revolutionized the way organizations manage and scale their IT infrastructure. However, with the increased reliance on cloud services, the need for effective cost management has become paramount. Anomalies in cloud cost data can indicate potential issues such as resource misconfigurations, security breaches, or inefficiencies, leading to unexpected financial burdens. This paper explores the application of machine learning (ML) techniques for anomaly detection in multi-cloud cost management. By leveraging supervised, unsupervised, and semi-supervised learning methods, this study aims to enhance the accuracy and efficiency of identifying cost anomalies. The paper also addresses key challenges such as high dimensionality, the dynamic nature of cloud environments, and scalability. Recent advancements in deep learning and hybrid models are discussed, providing insights into their potential for improving anomaly detection capabilities. Through comprehensive analysis and evaluation, this research contributes to the development of robust anomaly detection frameworks that can help organizations optimize their cloud expenditure and maintain financial control. Key Words: Cloud Cost Management, Anomaly Detection, Machine Learning, Supervised Learning, Unsupervised Learning, Semi-supervised Learning, Deep Learning

Detection of Electromagnetic Seismic Precursors from Swarm Data by Enhanced Martingale Analytics.

The detection of seismic activity precursors as part of an alarm system will provide opportunities for minimization of the social and economic impact caused by earthquakes. It has long been envisaged, and a growing body of empirical evidence suggests that the Earth's electromagnetic field could contain precursors to seismic events. The ability to capture and monitor electromagnetic field activity has increased in the past years as more sensors and methodologies emerge. Missions such as Swarm have enabled researchers to access near-continuous observations of electromagnetic activity at second intervals, allowing for more detailed studies on weather and earthquakes. In this paper, we present an approach designed to detect anomalies in electromagnetic field data from Swarm satellites. This works towards developing a continuous and effective monitoring system of seismic activities based on SWARM measurements. We develop an enhanced form of a probabilistic model based on the Martingale theories that allow for testing the null hypothesis to indicate abnormal changes in electromagnetic field activity. We evaluate this enhanced approach in two experiments. Firstly, we perform a quantitative comparison on well-understood and popular benchmark datasets alongside the conventional approach. We find that the enhanced version produces more accurate anomaly detection overall. Secondly, we use three case studies of seismic activity (namely, earthquakes in Mexico, Greece, and Croatia) to assess our approach and the results show that our method can detect anomalous phenomena in the electromagnetic data.

FDM: Effective and efficient incident detection on sparse trajectory data

Incident detection (ID), or the automatic discovery of anomalies from road traffic data (e.g., road sensor and GPS data), enables emergency actions (e.g., rescuing injured people) to be carried out in a timely fashion. Existing ID solutions based on data mining or machine learning often rely on dense traffic data; for instance, sensors installed in highways provide frequent updates of road information. In this paper, we ask the question: can ID be performed on sparse traffic data (e.g., location data obtained from GPS devices equipped on vehicles)? As these data may not be enough to describe the state of the roads involved, they can undermine the effectiveness of existing ID solutions. To tackle this challenge, we borrow an important insight from the transportation area, which uses trajectories (i.e., moving histories of vehicles) to derive incident patterns. We study how to obtain incident patterns from trajectories and devise a new solution (called Filter-Discovery-Match (FDM)) to detect anomalies in sparse traffic data. We have also developed a fast algorithm to support FDM. Experiments on a taxi dataset in Hong Kong and a simulated dataset show that FDM is more effective than state-of-the-art ID solutions on sparse traffic data, and is also efficient.

Anomaly Detection Using Machine Learning in Hydrochemical Data From Hot Springs: Implications for Earthquake Prediction

AbstractThis study explores the potential of machine learning algorithms for earthquake prediction, utilizing fluid chemical anomaly data from hot springs. Six hot springs, located within an active fault zone along the southeastern coast of China, were carefully chosen as hydrochemical monitoring sites for an extended period of two and a half years. Using this data, a prediction model integrating six algorithms was developed to forecast M ≥ 5 earthquakes in Taiwan. The model's performance was validated against recorded earthquake events, and the factors influencing its predictive capability were analyzed. Our comprehensive analysis conclusively demonstrates the superiority of machine learning algorithms over traditional statistical methods for earthquake prediction. Additionally, including sampling time in the data sets significantly improves the model's predictive performance. However, it is important to note that the model's predictive performance varies across different hot spring and indicators type, highlighting the importance of identifying optimal indicators for specific scenarios. The model parameters, including the anomaly detection rate (P) and earthquake response time threshold (M), significantly impact the model's predictive capabilities. Therefore, adjustments are needed to optimize the model's performance for practical use. Despite limitations such as the inability to differentiate pre‐earthquake anomalies from post‐earthquake anomalies and pinpoint the precise location of earthquakes, this study successfully showcases the potential of machine learning algorithms in earthquake prediction, paving the way for further research and improved prediction methods.

Recovering 3D Basin Basement Relief Using High‐Precision Magnetic Data Through Random Forest Regression Algorithm: A Case Study of Tianzhen‐Yanggao Sag in Datong Basin

AbstractInversion of magnetic basement interfaces in basins is essential for interpreting potential field data and studying geothermal resource distribution, as well as basin formation and evolution. This paper introduces a novel method for inverting magnetic basement interfaces using a random forest regression (RFR) algorithm that combines potential field processing and machine learning techniques. The method creates magnetic base interface models and corresponding magnetic anomaly data through the random midpoint displacement method and magnetic interface finite element forward simulation. These anomalies are then processed using techniques such as directional transformations, analytical continuation, spatial derivatives, and fractional transformations. Feature attributes are extracted, and Gini importance is utilized to measure the contributions of feature factors, identify effective features, and enhance algorithm efficiency. The validity and practicality of the method are demonstrated through the analysis of both idealized and noisy models. The proposed machine learning‐based approach is more intelligent, efficient, and accurately represents the relief of magnetic base interfaces. When applied to magnetic survey data in the Datong Basin, it produced a reliable basin base model that aligns with known structural information, paving the way for further research in magnetic interface inversion.

Finding origins of CMB anomalies in the inflationary quantum fluctuations

In this paper, we present compelling evidence for the parity asymmetry (a discrete symmetry separate from isotropy) in the Cosmic Microwave Background (CMB) map, measured through two-point temperature correlations. Any asymmetry associated with discrete symmetries, such as parity, challenges our understanding of quantum physics associated with primordial physics rather than LCDM (Λ Cold-Dark-Matter) itself. We commence by conducting a comprehensive analysis of the Planck CMB, focusing on the distribution of power in low-multipoles and temperature anticorrelations at parity conjugate points in position space. We find tension with the near scale-invariant power-law power spectrum of Standard Inflation (SI), with p-values of the order 𝒪(10-4 - 10-3). Alternatively, we explore the framework of direct-sum inflation (DSI), where a quantum fluctuation arises as a direct sum of two components evolving forward and backward in time at parity conjugate points in physical space. This mechanism results in a parity-asymmetric scale-dependent power spectrum, particularly prominent at low-multipoles, without any additional free model parameters. Our findings indicate that DSI is consistent with data on parity asymmetry, the absence of power at θ > 60°, and power suppression at low-even-multipoles which are major data anomalies in the SI model. Furthermore, we discover that the parameters characterizing the hemispherical power asymmetry anomaly become statistically insignificant when the large SI quadrupole amplitude is reduced to align with the data. DSI explains this low quadrupole with a p-value of 3.5%, 39 times higher than SI. Combining statistics from parameters measuring parity and low-ℓ angular power spectrum, we find that DSI is 50-650 times more probable than SI. In summary, our investigation suggests that while CMB temperature fluctuations exhibit homogeneity and isotropy, they also display parity-asymmetric behavior consistent with predictions of DSI. This observation provides a tantalizing evidence for the quantum mechanical nature of gravity.

Predictive Analysis for Big Mart Sales Using Machine Learning Algorithms

Abstract: Supermarkets and retail giants like Big Mart rely heavily on tracking individual item sales data to anticipate consumer demand and optimize inventory management. By mining data warehouses, anomalies, and trends can be identified, providing valuable insights for forecasting future sales volumes. Leveraging advanced machine learning techniques, such as Decision Tree Regression, predictive models can be developed to forecast sales accurately. Through the implementation of such a model, it has been observed that it surpasses the performance of existing forecasting models. This innovative approach not only enhances decision-making processes but also enables businesses like Big Mart to stay ahead in a competitive market landscape.

Systematic Review of Models Usedto Handle Class Imbalance in Anomaly Detection for Energy Consumption

The widespread integration of Smart technologies into energy consumption systems has brought about a transformative shift in monitoring and managing electricity usage. The imbalanced nature of anomaly data often results in suboptimal performance in detecting rare anomalies. This literature review analyzes models designed to address this challenge. The methodology involves a systematic literature review based on the five-step framework proposed by Khan, encompassing framing research questions, identifying relevant literature, assessing article quality, conducting a critical review, and interpreting results. The findings show that classical machine learning models like Support Vector Machines (SVM) and Random Forests (RF) are commonly used. In conclusion, classical machine learning models like SVM and RF struggle to recognize rare anomalies, while deep learning models, notably Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM), show promise for automatically learning elaborate representations and improving performance while dealing with class imbalance.

Multi-Attack Intrusion Detection for In-Vehicle CAN-FD Messages.

As an enhanced version of standard CAN, the Controller Area Network with Flexible Data (CAN-FD) rate is vulnerable to attacks due to its lack of information security measures. However, although anomaly detection is an effective method to prevent attacks, the accuracy of detection needs further improvement. In this paper, we propose a novel intrusion detection model for the CAN-FD bus, comprising two sub-models: Anomaly Data Detection Model (ADDM) for spotting anomalies and Anomaly Classification Detection Model (ACDM) for identifying and classifying anomaly types. ADDM employs Long Short-Term Memory (LSTM) layers to capture the long-range dependencies and temporal patterns within CAN-FD frame data, thus identifying frames that deviate from established norms. ACDM is enhanced with the attention mechanism that weights LSTM outputs, further improving the identification of sequence-based relationships and facilitating multi-attack classification. The method is evaluated on two datasets: a real-vehicle dataset including frames designed by us based on known attack patterns, and the CAN-FD Intrusion Dataset, developed by the Hacking and Countermeasure Research Lab. Our method offers broader applicability and more refined classification in anomaly detection. Compared with existing advanced LSTM-based and CNN-LSTM-based methods, our method exhibits superior performance in detection, achieving an improvement in accuracy of 1.44% and 1.01%, respectively.

Real-time Vital Signs Monitoring and Data Management Using a Low-Cost IoT-based Health Monitoring System

This study describes the creation and evaluation of a low-cost internet of things (IoT)-based health monitoring system for the continuous monitoring of vital signs such as temperature, pulse rate, oxygen saturation (SpO2) and blood pressure (BP) (both systolic and diastolic). Along with an organic light-emitting diode (OLED) display and an ESP8266 microcontroller, the system includes BP, non-contact temperature, SpO2 and electrocardiogram (ECG) sensors. Using the visual programming tool, Node-RED, the data from these sensors are gathered, processed and transmitted to the Google Cloud platform for archival and visualisation. The process involved mounting the sensors and microcontrollers on a special printed circuit board and designing the circuit with EasyEDA. The device measures systolic, diastolic and pulse rates from the BP sensor, as well as temperature, ECG and SpO2 values. The system works by using three push switches to read and display these values on demand. The gathered data are simultaneously shown on the OLED and sent to the Node-RED dashboard, where it is then sent to a Google Spreadsheet for archiving and analysis. This research article gives a thorough overview of the health monitoring system, the way it was implemented, and how it was successfully validated in a real-time setting. This study examines certain vital signs but additional health measures, such as respiration rate or glucose monitoring, could be included. Machine learning algorithms could also be used for predictive analytics. This would uncover data anomalies and trends early, improving healthcare management.

Machine-Learning and Physics-Based Tool for Anomaly Identification in Propulsion Systems

Launch anomalies occur frequently during the early phase of a program, with many of the anomalies attributed to propulsion systems. Approaches for identifying and mitigating potential propulsion failures can aid development programs and accelerate the resolution of root cause investigations. In reusable systems, anomaly detection methods can be employed to detect latent system health issues that could become problematic as the system ages. Modern launch support relies on human judgement for redline limit generation and visual family data comparison for many operational aspects, which makes it challenging to identify failure modes and to diagnose an anomaly. Additionally, family data comparison is unavailable for the first few launches of a new vehicle. Automated tools to quickly identify system failures of new and reusable systems can bridge these gaps. Physics-based modeling and machine learning (PBMML) offers methods that can improve the reliability of new or reusable launch vehicles by identifying propulsion anomalies or issues before they jeopardize future space missions. PBMML can then be used to inform corrective actions. This paper describes an anomaly data generation module which automates the process of simulating anomalous scenarios in launch vehicle and fluid networks, while a long-term short-term memory network is used to provide real-time anomaly classification on a simplified stage test case.

A Transformer and LSTM-Based Approach for Blind Well Lithology Prediction

Petrographic prediction is crucial in identifying target areas and understanding reservoir lithology in oil and gas exploration. Traditional logging methods often rely on manual interpretation and experiential judgment, which can introduce subjectivity and constraints due to data quality and geological variability. To enhance the precision and efficacy of lithology prediction, this study employed a Savitzky–Golay filter with a symmetric window for anomaly data processing, coupled with a residual temporal convolutional network (ResTCN) model tasked with completing missing logging data segments. A comparative analysis against the support vector regression and random forest regression model revealed that the ResTCN achieves the smallest MAE, at 0.030, and the highest coefficient of determination, at 0.716, which are indicative of its proximity to the ground truth. These methodologies significantly enhance the quality of the training data. Subsequently, a Transformer–long short-term memory (T-LS) model was applied to identify and classify the lithology of unexplored wells. The input layer of the Transformer model follows an embedding-like principle for data preprocessing, while the encoding block encompasses multi-head attention, Add & Norm, and feedforward components, integrating the multi-head attention mechanism. The output layer interfaces with the LSTM layer through dropout. A performance evaluation of the T-LS model against established rocky prediction techniques such as logistic regression, k-nearest neighbor, and random forest demonstrated its superior identification and classification capabilities. Specifically, the T-LS model achieved a precision of 0.88 and a recall of 0.89 across nine distinct lithology features. A Shapley analysis of the T-LS model underscored the utility of amalgamating multiple logging data sources for lithology classification predictions. This advancement partially addresses the challenges associated with imprecise predictions and limited generalization abilities inherent in traditional machine learning and deep learning models applied to lithology identification, and it also helps to optimize oil and gas exploration and development strategies and improve the efficiency of resource extraction.

A framework for detecting zero-day exploits in network flows

Zero-day attack detection solutions aim to proactively identify unknown threats targeting valuable assets within a given system. While many Intrusion Detection System (IDS) solutions leverage learning techniques to build novel attack detection systems, they often focus on enhancing accuracy for specific attack types, overlooking the potential for multiple attack scenarios. Therefore, we introduce a novel framework for detecting zero-day attacks that evade current detection systems. Our framework enhances attack identification and qualification through a hybrid learning approach, where supervised learning ensures detection of known attacks and unsupervised learning. It encompasses intrusion detection phases from data collection to new attack class detection by identifying anomalies in real-time network flow data. Unsupervised learning, which involves grouping similar data points into clusters, establishes minimum distances within these clusters. This process triggers cluster division when certain thresholds are reached. Finally, an online supervised learning process validates our approach’s effectiveness in identifying anomalies associated with zero-day attack flows.This approach significantly reduces the False Detection Rate (FDR) without solely focusing on optimizing machine learning (ML) and deep learning (DL) algorithms hyper-parameters. We evaluated our framework on two datasets: one from a real industrial context at IBM and the NSL-KDD dataset. The results demonstrate our framework’s ability to detect anomalies in previously zero-day attack targets. On average, we identified 71 anomalous flows per target, achieving an overall average online learning accuracy of 98.4% for the IBM dataset and 96.6% for the NSL-KDD dataset, thereby validating the detection of these new attack scenarios.

Detection of Emergent Anomalous Structure in Functional Data

Motivated by an example arising from digital networks, we propose a novel approach for detecting the emergence of anomalies in functional data. In contrast to classical functional data approaches, which detect anomalies in completely observed curves, the proposed approach seeks to identify anomalies sequentially as each point on the curve is received. The new method, the Functional Anomaly Sequential Test (FAST), captures the common profile of the curves using Principal Differential Analysis and uses a form of CUSUM test to monitor a new functional observation as it emerges. Various theoretical properties of the procedure are derived. The performance of FAST is then assessed on both simulated and telecommunications data.