Cause Of Anomalies Research Articles

Linkability measures to assess the data characteristics for record linkage.

Accurate record linkage (RL) enables consolidation and de-duplication of data from disparate datasets, resulting in more comprehensive and complete patient data. However, conducting RL with low quality or unfit data can waste institutional resources on poor linkage results. We aim to evaluate data linkability to enhance the effectiveness of record linkage. We describe a systematic approach using data fitness ("linkability") measures, defined as metrics that characterize the availability, discriminatory power, and distribution of potential variables for RL. We used the isolation forest algorithm to detect abnormal linkability values from 188 sites in Indiana and Colorado, and manually reviewed the data to understand the cause of anomalies. We calculated 10 linkability metrics for 11 potential linkage variables (LVs) across 188 sites for a total of 20680 linkability metrics. Potential LVs such as first name, last name, date of birth, and sex have low missing data rates, while Social Security Number vary widely in completeness among all sites. We investigated anomalous linkability values to identify the cause of many records having identical values in certain LVs, issues with placeholder values disguising data missingness, and orphan records. The fitness of a variable for RL is determined by its availability and its discriminatory power to uniquely identify individuals. These results highlight the need for awareness of placeholder values, which inform the selection of variables and methods to optimize RL performance. Evaluating linkability measures using the isolation forest algorithm to highlight anomalous findings can help identify fitness-for-use issues that must be addressed before initiating the RL process to ensure high-quality linkage outcomes.

Detecting faults in the cooling systems by monitoring temperature and energy

The cooling systems contribute to 40% of overall building energy consumption. Out of which, 40% is wasted because of faulty parts that cause anomalies in the cooling systems. We propose a three-stage, non-invasive part-level anomaly detection technique to identify anomalies in both cooling systems, a ducted-centralized and a ductless-split. We use COTS sensors to monitor temperature and energy without invading the cooling system. After identifying the anomalies, we find the cause of the anomaly. Based on the anomaly, the solution recommends a fix. If there is a technical fault, our proposed technique informs the technician regarding the faulty part, reducing the cost and time needed to repair it. In the first stage, we propose a domain-inspired time-series statistical technique to identify anomalies in cooling systems. We observe an AUC-ROC score of more than 0.93 in simulation and experimentation. In the second stage, we propose using a rule-based technique to identify the cause of the anomaly. We classify causes of anomalies into three classes. We observe an AUC-ROC score of 1. Based on the anomaly classification, we identify the faulty part of the cooling system in the third stage. We use the Nearest-Neighbour Density-Based Spatial Clustering of Applications with Noise (NN-DBSCAN) algorithm with transfer learning capabilities to train the model only once, where it learns the domain knowledge using the simulated data. The trained model is used in different environmental scenarios with both types of cooling systems. The proposed algorithm shows an accuracy score of 0.82 in simulation deployment and 0.88 in experimentation. In the simulation we used both ducted-centralized and ductless-split cooling systems and in the experimentation we evaluated the solution with ductless-split cooling systems. The overall accuracy of the three-stage technique is 0.82 and 0.86 in simulation and experimentation, respectively. We observe energy savings of up to 68% in simulation and 42% during experimentation, with a reduction of ten days in the cooling system’s downtime and up to 75% in repair cost.

ERPCA: Robust Principal Component Analysis for Exponential Family Distributions

AbstractRobust principal component analysis (RPCA) is a widely used method for recovering low‐rank structure from data matrices corrupted by significant and sparse outliers. These corruptions may arise from occlusions, malicious tampering, or other causes for anomalies, and the joint identification of such corruptions with low‐rank background is critical for process monitoring and diagnosis. However, existing RPCA methods and their extensions largely do not account for the underlying probabilistic distribution for the data matrices, which in many applications are known and can be highly non‐Gaussian. We thus propose a new method called RPCA for exponential family distributions (), which can perform the desired decomposition into low‐rank and sparse matrices when such a distribution falls within the exponential family. We present a novel alternating direction method of multiplier optimization algorithm for efficient decomposition, under either its natural or canonical parametrization. The effectiveness of is then demonstrated in two applications: the first for steel sheet defect detection and the second for crime activity monitoring in the Atlanta metropolitan area.

Testing Stock Market Anomalies in Nifty 50 Stocks: An Empirical Analysis in the Indian Context By IJISRT

Stock market anomalies are persistent patterns in stock returns that traditional financial models cannot explain. Anomalies challenge the Efficient Market Hypothesis (EMH), which states that all information is reflected in stock prices. This research aims to test the stock market anomalies in Nifty 50 stocks in India. The study will use various statistical methods to examine a range of calendar anomalies. The findings of this study will have important implications for investors and policymakers. If anomalies are found to exist, investors may be able to generate abnormal returns by exploiting them. Policymakers may also be interested in understanding the causes of anomalies and their impact on the overall market. This process involves examining a large dataset of stocks within the Nifty 50 index to identify any patterns or trends that may be considered abnormal or unexpected. By conducting this analysis, investors and financial analysts can gain deeper insights into the behaviour of the stock market in India and make more informed investment decisions. This study is significant for several reasons: it is one of the few studies to examine stock market anomalies in the Indian market; the study uses comprehensive statistical methods to test for anomalies. The study is expected to find evidence of some stock market anomalies in Nifty 50 stocks. The study's conclusions will have an impact on investors and policymakers.

Realtime Anomaly Detection in Healthcare IoT: A Machine Learning-Driven Security Framework

Healthcare IoT, a fast-growing field, could revolutionize patient monitoring and intervention. This interconnection raises new security concerns, requiring real-time anomaly detection to protect patient data and device integrity. This study presents a novel security framework that uses combination of Hidden Markov Models (HMM) and Support Vector Machine (SVM) to detect anomalies in real-time healthcare IoT environments with high accuracy. The framework prioritizes real-time strength and efficiency. Sensor data from wearables, medical devices, and other IoT devices is carefully segmented into time intervals. Features are carefully derived from each segment, including statistical summaries, patterns, and frequency domain characteristics. Feature engineering is essential for accurate anomaly detection. Integrating HMM and SVM capabilities is the framework’s core. HMM accurately represent concealed states within divided data sequences and analyze patterns of change over time. After that, an independent-trained SVM examines each state. Using proximity to a decision hyperplane in feature space, this SVM can classify data points as normal or anomalous. This method augments HMM temporal capabilities with SVM classification efficiency, increasing sensitivity to anomalous patterns and reducing false positives. The framework’s exceptional performance is shown by extensive evaluations on the PhysioNet Challenge 2017 dataset, which includes diverse ECG recordings with labeled anomalies. HMM-SVM outperforms Naive Bayes, LSTM, and Random Forest with 98.66% accuracy. The framework also has high precision, recall, and F1-score, indicating a refined ability to detect real anomalies and reduce false alarms. The framework prioritizes real-time understanding and application alongside its remarkable precision. HMM-SVMs reveal hidden data state changes, revealing context and possible causes of anomalies. Modular design and efficient algorithms enable seamless integration of real-time functionality in low-resource IoT devices, enabling quick and effective security responses. To conclude, this study introduces an HMM-SVM framework for fast Healthcare IoT abnormality detection. The framework emphasizes comprehensibility and real-time applicability while achieving high accuracy. This framework can protect patient data, improve device security, and create a more reliable healthcare IoT ecosystem.

Expressway ETC Transaction Data Anomaly Detection Based on TL-XGBoost

China’s widely adopted expressway ETC system provides a feasible foundation for realizing co-operative vehicle–infrastructure integration, and the accuracy of ETC data, which forms the basis of this scheme, will directly affect the safety of driving. Therefore, this study focuses on the abnormal data in an expressway ETC system. This study combines road network topology data and capture data to mine the abnormal patterns of ETC data, and it designs an abnormal identification model for expressway transaction data based on TL-XGBoost. This model categorizes expressway ETC abnormal data into four distinct classes: missing detections, opposite lane detection, duplicated detection and reverse trajectory detection. ETC transaction data from a southeastern Chinese province were used for experimentation. The results validate the model’s effectiveness, achieving an accuracy of 98.14%, a precision of 97.59%, a recall of 95.44%, and an F1-score of 96.49%. Furthermore, this study conducts an analysis and offers insights into the potential causes of anomalies in expressway ETC data.

ANOMALIES IN THREE FROG SPECIES FROM EASTERN AMAZONIA

We reported anomalies in three anuran species belonging to two families: Craugastoridae (Pristimantis gutturalis and P. chiastonotus), and Bufonidae (Rhinella castaneotica). All records were found in two conservation units, a well-preserved and protected area of the Amazonian rainforest in the state of Amapa, eastern Amazon. The anomalies detected were anophthalmia (characterized by the absence of one or both eyes) in P. gutturalis and P. chiastonotus, and brachydactyly (digit shortened) in R. castaneotica. Further ecotoxicological studies of anurans populations in protected areas are needed to investigate the causes of anomalies.

Visual Diagnostics of Parallel Performance in Training Large-Scale DNN Models.

Diagnosing the cluster-based performance of large-scale deep neural network (DNN) models during training is essential for improving training efficiency and reducing resource consumption. However, it remains challenging due to the incomprehensibility of the parallelization strategy and the sheer volume of complex data generated in the training processes. Prior works visually analyze performance profiles and timeline traces to identify anomalies from the perspective of individual devices in the cluster, which is not amenable for studying the root cause of anomalies. In this article, we present a visual analytics approach that empowers analysts to visually explore the parallel training process of a DNN model and interactively diagnose the root cause of a performance issue. A set of design requirements is gathered through discussions with domain experts. We propose an enhanced execution flow of model operators for illustrating parallelization strategies within the computational graph layout. We design and implement an enhanced Marey's graph representation, which introduces the concept of time-span and a banded visual metaphor to convey training dynamics and help experts identify inefficient training processes. We also propose a visual aggregation technique to improve visualization efficiency. We evaluate our approach using case studies, a user study and expert interviews on two large-scale models run in a cluster, namely, the PanGu- α 13B model (40 layers), and the Resnet model (50 layers).

ServiceAnomaly: An anomaly detection approach in microservices using distributed traces and profiling metrics

Anomaly detection is an essential activity for identifying abnormal behaviours in microservice-based systems. A common approach is to model the system behaviour during normal operation using either distributed traces or profiling metrics. The model is then used to detect anomalies during system operation. In this paper, we present a new anomaly detection approach, called ServiceAnomaly, for anomaly detection in microservice systems that combines distributed traces and six profiling metrics to build an annotated directed acyclic graph that characterizes the normal behaviour of the system. Unlike existing techniques, our approach captures the context propagation provided by distributed traces as a graph that is annotated with functions characterizing both linear and non-linear relationships between profiling metrics. The final annotated graph is used to detect abnormal executions during system operation. The results of applying our approach to two open-source benchmarks show that our approach detects anomalies with an F1-score up to 86%. We also show how developers can use the annotated graph to reason about the causes of anomalies

Toward identifying malicious encrypted traffic with a causality detection system

The main methods for protecting user privacy and addressing cybersecurity problems caused by encrypted traffic are non-decryption detection approaches. However, these methods face problems such as the small number of trainable features and imbalanced training datasets, which seriously affect the robustness of existing malicious encrypted traffic detection systems (TDSs). To address these issues, we identify malicious encrypted traffic with a causality explainable detection system (METCXDS). Our detection tasks increase the number of noncryptographic features and eliminate noise features to improve interpretability. First, we extract and recombine temporal and statistical features to increase the number of trainable features. Then, the influence of causal features on the results is enhanced by weighting causal reasoning features, and noise features with low weights are removed to improve the causal interpretability of the detection results. Furthermore, samples are generated through a Wasserstein generative adversarial network (WGAN) that learns the causal feature distribution to balance the training samples. Finally, the METCXDS performance is evaluated through the detection balance index and causal feature index. On the CICIDS2017 and DoHBrw2020 datasets, compared with previous TDSs, our system improves the overall F1 score by up to >10 %. And the causality of the detection results is verified by the causal effects, which identify the causes of anomalies in encrypted traffic and clarify the decision-making process of the detection system.

Troubleshooting in an IVF Laboratory

In the realm of assisted reproductive technology, troubleshooting within an in vitro fertilization (IVF) laboratory is of paramount importance to ensure successful outcomes. FMEA is a proactive methodology used to identify potential failure modes in laboratory processes, assess their possible effects, and prioritize interventions to prevent or mitigate them. In the context of IVF, FMEA aids in recognizing critical steps prone to errors, such as sample mishandling or equipment malfunctions. By quantifying the probability and severity of failures, FMEA allows laboratories to allocate resources judiciously and implement corrective measures that reduce the likelihood of adverse events. Root cause analysis (RCA) is a retrospective technique employed to delve into the underlying causes of anomalies or failures that have already occurred. In the IVF laboratory, RCA helps uncover the root reasons for unsuccessful procedures, like embryo implantation failure. This analysis aids in identifying both immediate triggers and underlying systemic issues, paving the way for targeted improvements to enhance overall success rates. By plotting key performance indicators over time, IVF laboratories can detect deviations from established norms. This enables swift corrective actions before deviations result in compromised outcomes. Control Charts thus offer a dynamic tool for monitoring laboratory processes, maintaining consistency, and swiftly addressing unexpected fluctuations. By systematically preventing potential failures, investigating root causes, and dynamically monitoring processes, IVF laboratories can improve their success rates, enhance patient experiences, and contribute to the advancement of assisted reproductive technologies.

USE OF LOW-FREQUENCY SEISMIC PROBING METHOD FOR DETERMINATION OF WELL LOCATION COORDINATES FOR DRILLING

A wide range of studies, including structural and tectonic, lithological and facies, geochemical, hydrogeological, and geophysical methods, are carried out during the development of fields at the regional stage of geological exploration. These studies are aimed at solving key problems, such as identifying lithological and stratigraphic complexes, determining the main stages of geotectonic development, and identifying promising oil and gas complexes.However, despite the extensive studies performed, they do not always provide a final confirmation of the presence or absence of oil deposits. In recent years, in order to more accurately determine the location of oil fields, the method of low-frequency passive seismic sensing (LPDS) has attracted attention.This article discusses two hypotheses that explain the causes of anomalies in the spectrum of microseismic signals over oil and gas reservoirs. The first hypothesis assumes that during harmonic seismic excitation, the deposit goes into a state of «turbulent chaos», leading to a change in the characteristics of the seismic signal. The second hypothesis states that a change in the velocity of propagation of seismic waves in the porous environment of the deposit also causes anomalies in the signal spectrum.Experimental data support both hypotheses, and represent anomalies in the spectrum of microseismic signals in the frequency range of 2-5 Hz. Understanding these anomalies is essential for improving oil and gas discovery and evaluation techniques, contributing to more efficient development of oil and gas resources and improving accuracy of well coordinates

Digital Twin-Based Fault Diagnosis Platform for Final Rolling Temperature in Hot Strip Production.

The final rolling temperature in hot rolling is an important process parameter for hot-rolled strips and greatly influences their mechanical properties and rolling stability. The diagnosis of final rolling temperature anomalies in hot rolling has always been difficult in industry. A data-driven risk assessment method for detecting final rolling temperature anomalies is proposed. In view of the abnormal setting value for the strip head, a random forest model is established to screen the process parameters with high feature importance, and the isolation forest algorithm is used to evaluate the risk associated with the remaining parameters. In view of the abnormal process curve of the full length of the strip, the Hausdorff distance algorithm is used to eliminate samples with large deviations, and a risk assessment of the curve is carried out using the LCSS algorithm. Aiming to understand the complex coupling relationship between the influencing factors, a method for identifying the causes of anomalies, combining a knowledge graph and a Bayesian network, is established. According to the results of the strip head and the full-length risk assessment model, the occurrence of the corresponding nodes in the Bayesian network is determined, and the root cause of the abnormality is finally output. By combining mechanistic modeling and data modeling techniques, it becomes possible to rapidly, automatically, and accurately detect and analyze final rolling temperature anomalies during the rolling process. When applying the system in the field, when compared to manual analysis by onsite personnel, the accuracy of deducing the causes of anomalies was found to reach 92%.

Similarity searching for fault diagnosis of defect patterns in wafer bin maps

Due to the growing complexity of processes in semiconductor manufacturing, high volumes of data are automatically generated, resulting in a greater challenge for fault detection and yield improvement. Wafer bin maps (WBMs), which represent the spatial distribution of defective dies on the wafer, may contain specific defect patterns that offer useful insight into the underlying causes of anomalies in the processes. Hence, the identification of these defect patterns helps the early detection and diagnosis of the faults. Nowadays, rare and mixed-type defects are more frequently observed, which increases the difficulty for the recognition of defect patterns. In this study, we propose a similarity searching approach for the identification of defect patterns and their potential causes. The comparison of similar patterns provides valuable information to trace the problems in the process history which may narrow the scope of troubleshooting. In particular, the tensor voting algorithm is applied to highlight the structural information in the patterns, and then the weighted best-buddies similarity (WBBS) is proposed to measure the degree of similarity. Experimental results verify the effectiveness of the proposed method in the context of both single and mixed-type defect patterns.

Systemic symptom detection in telemetry of ISS with explainability using FRAM and SpecTRM

Explainability is important for machine learning-based anomaly detection of safety critical systems. In this respect, we propose a new systemic symptom detection method by combining two methodologies: the Functional Resonance Analysis Method (FRAM) and the Specification Tools and Requirement Methodology-Requirement Language (SpecTRM-RL) with machine learning-based normal behavior prediction model. The method was verified with data of thermal control system of Japanese Experimental Module of the International Space Station, and the result found that the proposed method enables flight controllers and specialists to obtain additional information for identifying causes of anomaly with the method.

Multisensor Anomaly Detection and Interpretable Analysis for Linear Induction Motors

In this paper, a graph neural network anomaly detection framework is proposed to improve the safety of linear induction motors, a key component of high-speed maglev trains. In our framework, each sensor sequence is treated as a separate feature. The similarity and correlation between multi-dimensional features are learned as prior knowledge for graph structure learning. The spatial-temporal graph attention network incorporates prior knowledge to learn complex correlations between nodes. Furthermore, the framework optimises a joint model for anomaly detection, avoiding the trap of falling into either local or global optimisation and thus achieving the most stable detection. Experimental results of our method on four real-world datasets show that it is more accurate than other state-of-the-art methods in detecting anomalies and capturing inter-sensor correlations. Further analysis of graph attention weights and visualization subgraphs show that our framework is well interpretable and allowing users to locate the root cause of anomalies.

Anomaly Detection for Agile Satellite Attitude Control System Using Hybrid Deep-Learning Technique

Agile low-Earth-orbit (LEO) observation satellites need a robust attitude control and determination system. It is a critical satellite subsystem, which stabilizes the satellite to different desired orientations during its mission using different actuators. The detection of satellite misorientation is a highly challenging problem because it requires continuous monitoring of data from hundreds of satellite sensors to guarantee healthy operability. In this paper, the authors propose a data-driven deep-learning framework for detecting satellite misorientation by analyzing attitude control subsystem telemetry data. The proposed approach combines a hybrid predictive deep-learning model that consists of long short-term memory and convolutional neural networks in two parallel paths to predict telemetry data and a robust isolation forest classifier for anomaly detection purposes that can classify output residuals as normal or anomalous. The hybrid model was optimized by the particle swarm optimization algorithm to ensure faster fitness function convergence with optimal model hyperparameters. The suggested data-driven model was validated using real telemetry datasets, including real anomalous case studies. The experimental results proved the suggested approach’s superiority for identifying satellite misorientation as well as helping satellite operators monitor the system’s health and deduce the causes of anomalies to aid in decision-making.

An unsupervised root cause analysis method for satellite on-orbit anomalies based on causal discovery

For a modern satellite, there are usually thousands of telemetry parameters. How to find the root cause of the satellite anomalies using telemetry data is an important but difficult task. This paper proposes an unsupervised root cause analysis method for satellite on-orbit anomalies based on causal discovery. The method includes a component impact analysis method based on multivariate-to-one transfer entropy, which can quickly identify components that cause system anomalies; and a pruning strategy based on the change of causality are proposed to screen and identify the parameters of the components to determine the root cause of the anomalies. Simulation experiments and real case verification show that the accuracy, recall, and F1-score of the method in this paper all reach more than 80%, which is greatly improved compared with the existing methods. The case study also shows that when anomalies occur, the root cause analysis method can detect the root cause using only 4 min of anomalous information, which helps to stop the spread of anomalies more effectively.

Factors Associated With an Increased Risk of Facial Malformations.

Background Facial anomalies comprise a significant component of birth defects, with oral clefts being the second most common entity in this group. All organ systems within the body can be affected by congenital anomalies, mostly affecting the musculoskeletal system. Birth defects are among the leading causes of infant mortality and morbidity around the world. Objectives To find the factors associated with an increased risk of facial malformations so that steps for improving preventive measures can be taken. Methodology This was a cross-sectional study in which the data were collected from the files of infants admitted to the pediatric department. Data regarding the type of congenital anomaly, maternal investigations done during pregnancy, maternal history of medication, diabetes, hypertension, radiation exposure, smoking, and alcohol history, and family history of congenital anomalies was collected from the files of neonates and from the pediatrician. In the case of unanswered questions, the parent was contacted after 10 days with their consent. Results Of the sample size of 259 children (males: 132; females: 127), 68 (26%) had a cleft lip, 69 (27%) had a cleft palate, 110 (42%) had both cleft lip and palate, five (2%) had a cleft lip with nasal deformity, five (2%) had a cleft lip and palate with nasal deformity, and two (1%) had hypertelorism. Eight percent of neonates with craniofacial malformations had a family history of congenital malformations; 80.7% of neonates had a history of parental consanguinity; and 19.3% were unrelated. In regard to the mothers, 41.3% of the mothers had diabetes, 4% had hypertension, 4% had both gestational diabetes and hypertension, and 55% had neither of these diseases. Of the 55% of mothers with neither disease, 75% were married to their cousins, while 25% were not married within the family. Practical implications This study, highlighting the major factors contributing to the incidence of congenital facial malformations, will educate the community and establish awareness among the younger generation of the top causes of anomalies, therefore making a huge impact on increasing efforts to reduce the prevalence of congenital anomalies. Conclusion Defects of both the cleft lip and palate had the highest prevalence of facial malformations among study subjects (110 patients (42%)). Parental consanguinity is one of the leading factors associated with an increased risk of facial malformations.

Methods and software for anomalies searching in the telemetry data of a solar power plant based on the normalized power analysis

Objectives. In connection with the increase in the number of solar power plants, the automation of monitoring their performance becomes an urgent task. The search for anomalies in the operation of solar power plants is one of the main components of monitoring. The purpose of the study is to develop new methods and software algorithms for finding anomalies in the operation of solar panels based on the results of a digital twin created and trained according to the telemetry data of a solar power plant.Methods. The developed technique is based on statistical studies of deviations of power values at the point of maximum efficient operation of the solar panel calculated by the digital twin. In addition, a normalized value of the power in the maximum efficient operation of the solar panel was introduced for more accurate clustering and anomaly search.Results. Using the developed method of static search for half a year of observations, 18 anomalies were detected in the operation of the solar panels of the power plant. All cases are analyzed for the causes of anomalies in the operation of solar panels.Conclusion. It has been established that when using normalized power values in the analysis of deviations at the point of maximum power PN, it is possible to detect abnormal operation of individual panels. The level of deviation of the normalized values at the point of maximum power was calculated, indicating the presence of an anomaly in the operation of solar panel.