Local Outlier Factor Research Articles

Integrating Autoencoders with Local Outlier Factor and Isolation Forest for Effective Fraud Detection in Imbalanced Datasets

Integrating Autoencoders with Local Outlier Factor and Isolation Forest for Effective Fraud Detection in Imbalanced Datasets

Efficient Optimization: Unveiling the Application of Ensemble Learning Combined with the CMA-ES Algorithm in Hydraulic Fracturing Design

Optimizing fracturing parameters is crucial for enhancing production and reducing costs in oil and gas exploration and development. Effectively integrating geological and engineering parameters for the automated optimization of fracturing design continues to pose challenges. This study utilizes the cluster-based local outlier factor method for anomaly detection and removal from the dataset, significantly enhancing data quality. By integrating diverse models, including tree-based models and neural networks, an ensemble model for production prediction was developed. This approach successfully addresses the limitations of relying on a single model and achieves high-precision production forecasting. Furthermore, a Covariance Matrix Adaptation Evolution Strategy (CMA-ES)-based framework was established to comprehensively optimize the design parameters of fracturing projects. Optimization practices for two selected wells resulted in a 168.54% increase in production and identified the optimal design parameter configuration for all cases studied. The results of this study demonstrate the feasibility and effectiveness of the proposed ensemble prediction model and optimization framework in practical applications. Data-driven optimization strategies are expected to play a larger role in future oil and gas development, driving technological innovation and advancement in the field.

Automated quality control of small animal MR neuroimaging data

Abstract Magnetic resonance imaging (MRI) is a valuable tool for studying brain structure and function in animal and clinical studies. With the growth of public MRI repositories, access to data has finally become easier. However, filtering large datasets for potential poor-quality outliers can be a challenge. We present AIDAqc, a machine-learning-assisted automated Python-based command-line tool for small animal MRI quality assessment. Quality control features include signal-to-noise ratio (SNR), temporal SNR, and motion. All features are automatically calculated and no regions of interest are needed. Automated outlier detection for a given dataset combines the interquartile range and the machine-learning methods one-class support vector machine, isolation forest, local outlier factor, and elliptic envelope. To evaluate the reliability of individual quality control metrics, a simulation of noise (Gaussian, salt and pepper, speckle) and motion was performed. In outlier detection, single scans with induced artifacts were successfully identified by AIDAqc. AIDAqc was challenged in a large heterogeneous dataset collected from 19 international laboratories, including data from mice, rats, rabbits, hamsters, and gerbils, obtained with different hardware and at different field strengths. The results show that the manual inter-rater agreement (mean Fleiss Kappa score 0.17) is low when identifying poor-quality data. A direct comparison of AIDAqc results, therefore, showed only low-to-moderate concordance. In a manual post hoc validation of AIDAqc output, precision was high (>70%). The outlier data can have a significant impact on further postprocessing, as shown in representative functional and structural connectivity analysis. In summary, this pipeline optimized for small animal MRI provides researchers with a valuable tool to efficiently and effectively assess the quality of their MRI data, which is essential for improved reliability and reproducibility.

Detection of Impedance Inhomogeneity in Lithium-Ion Battery Packs Based on Local Outlier Factor

The inhomogeneity between cells is the main cause of failure and thermal runaway in Lithium-ion battery packs. Electrochemical Impedance Spectroscopy (EIS) is a non-destructive testing technique that can map the complex reaction processes inside the battery. It can detect and characterise battery anomalies and inconsistencies. This study proposes a method for detecting impedance inconsistencies in Lithium-ion batteries. The method involves conducting a battery EIS test and Distribution of Relaxation Times (DRT) analysis to extract characteristic frequency points in the full frequency band. These points are less affected by the State of Charge (SOC) and have a strong correlation with temperature, charge/discharge rate, and cycles. An anomaly detection characteristic impedance frequency of 136.2644 Hz was determined for a cell in a Lithium-ion battery pack. Single-frequency point impedance acquisition solves the problem of lengthy measurements and identification of anomalies throughout the frequency band. The experiment demonstrates a significant reduction in impedance measurement time, from 1.05 h to just 54 s. The LOF was used to identify anomalies in the EIS data at this characteristic frequency. The detection results were consistent with the actual conditions of the battery pack in the laboratory, which verifies the feasibility of this detection method. The LOF algorithm was chosen due to its superior performance in terms of FAR (False Alarm Rate), MAR (Missing Alarm Rate), and its fast anomaly identification time of only 0.1518 ms. The method does not involve complex mathematical models or parameter identification. This helps to achieve efficient anomaly identification and timely warning of single cells in the battery pack.

Identifying Tampered Radio-Frequency Transmissions in LoRa Networks Using Machine Learning.

Long-range networks, renowned for their long-range, low-power communication capabilities, form the backbone of many Internet of Things systems, enabling efficient and reliable data transmission. However, detecting tampered frequency signals poses a considerable challenge due to the vulnerability of LoRa devices to radio-frequency interference and signal manipulation, which can undermine both data integrity and security. This paper presents an innovative method for identifying tampered radio frequency transmissions by employing five sophisticated anomaly detection algorithms-Local Outlier Factor, Isolation Forest, Variational Autoencoder, traditional Autoencoder, and Principal Component Analysis within the framework of a LoRa-based Internet of Things network structure. The novelty of this work lies in applying image-based tampered frequency techniques with these algorithms, offering a new perspective on securing LoRa transmissions. We generated a dataset of over 26,000 images derived from real-world experiments with both normal and manipulated frequency signals by splitting video recordings of LoRa transmissions into frames to thoroughly assess the performance of each algorithm. Our results demonstrate that Local Outlier Factor achieved the highest accuracy of 97.78%, followed by Variational Autoencoder, traditional Autoencoder and Principal Component Analysis at 97.27%, and Isolation Forest at 84.49%. These findings highlight the effectiveness of these methods in detecting tampered frequencies, underscoring their potential for enhancing the reliability and security of LoRa networks.

Enhancing healthcare data integrity: fraud detection using unsupervised learning techniques

Data in healthcare forms the backbone of any treatment and decision-making for patients. However, the data from a healthcare institution can sometimes be prone to abnormalities, thereby putting treatment and patient safety in jeopardy. This paper points to the dire need for reliable anomaly detection systems in the healthcare industry. It employs various unsupervised learning methods, including Isolation Forest, Local Outlier Factor (LOF), K-Nearest Neighbours (KNN), and autoencoder models for detecting abnormalities in healthcare data with better accuracy. Anomaly detection capabilities also allow health providers to reduce risks and provide some assurance of the integrity of the data, as these capabilities are more likely to indicate unusual patient profiles or incorrect test results. Isolation Forest, LOF, and KNN are the preliminary methods for performing anomaly detection in this work, with Isolation Forest yielding the best results. Then, autoencoder models that learn subtle variations and complex patterns in data are employed. This paper seeks to enhance anomaly detection in terms of accuracy and reliability to ensure better healthcare data quality and patient safety.

Online Self-Learning-Based Raw Material Proportioning for Rotary Hearth Furnace and Intelligent Batching System Development

With the increasing awareness of environmental protection, the rotary hearth furnace system has emerged as a key technology that facilitates a win-win situation for both environmental protection and enterprise economic benefits. This is attributed to its high flexibility in raw material utilization, capability of directly supplying blast furnaces, low energy consumption, and high zinc removal rate. However, the complexity of the raw material proportioning process coupled with the rotary hearth furnace system’s reliance on human labor results in a time-consuming and inefficient process. This paper innovatively introduces an intelligent formula method for proportioning raw materials based on online clustering algorithms and develops an intelligent batching system for rotary hearth furnaces. Firstly, the ingredients of raw materials undergo data preprocessing, which involves using the local outlier factor (LOF) method to detect any abnormal values, using Kalman filtering to smooth the data, and performing one-hot encoding to represent the different kinds of raw materials. Afterwards, the affinity propagation (AP) clustering method is used to evaluate past data on the ingredients of raw materials and their ratios. This analysis aims to extract information based on human experience with ratios and create a library of machine learning formulas. The incremental AP clustering algorithm is utilized to learn new ratio data and continuously update the machine learning formula library. To ensure that the formula meets the actual production performance requirements of the rotary hearth furnace, the machine learning formula is fine-tuned based on expert experience. The integration of machine learning and expert experience demonstrates good flexibility and satisfactory performance in the practical application of intelligent formulas for rotary hearth furnaces. An intelligent batching system is developed and executed at a steel plant in China. It shows an excellent user interface and significantly enhances batching efficiency and product quality.

A robust federated learning algorithm for partially trusted environments

Due to the distributed nature of federated learning, it is vulnerable to poisoning attacks during the training process. The model’s resistance to poisoning attacks can be improved using robust aggregation algorithms. Current research on federated learning to resist poisoning attacks is mainly based on two settings: No trust or Byzantine robustness. However, both settings are not close enough to reality in practical scenarios. In many practical applications, some participants in federated learning are trustworthy. For example, participants who have participated in the training of this model before and performed very well, or participants with strong compliance and credibility such as governments and some national agencies participate in the training. In existing research, these trusted participants still have to accept the judgment of the aggregation node, which generates unnecessary computation, increases overhead, and does not take advantage of a trusted environment. Since there is no attack behavior on the trusted client, its training results are used to classify the trustworthiness of other untrusted clients and identify attack nodes with higher accuracy. Therefore, this paper proposes a robust federated learning algorithm for partially trusted environments. The proposed scheme uses the experimental results of trusted clients to judge the behavior of untrustworthy clients by the cosine similarity and the Local Outlier Factor and further identify and detect malicious clients. Experiments are performed on MNIST and CIFAR datasets. Comparison with other six aggregation algorithms under 30% attack scenario. And compared with the other four aggregation algorithms under 70% attack conditions. Our algorithm is more accurate than almost all of the other aggregation algorithms. The paper is the first to conduct robust research on federated learning in a partially trusted environment, and the proposed algorithm can more effectively resist poisoning attacks.

Anomaly detection and prediction of energy consumption for smart homes using machine learning

AbstractAs technology advances, smart homes are being increasingly adopted, thus generating massive data that pose new research challenges. We propose a machine learning framework for monitoring energy consumption in smart home devices. The proposed framework involves an anomaly detection module, followed by a predictive model to forecast energy consumption patterns in a typical smart home. We employ three outlier‐based techniques for anomaly detection: (1) local outlier factor, (2) connectivity‐based outlier factor, and (3) cluster‐based local outlier factor. Furthermore, we apply random forest, linear regression, decision tree, and the ensemble techniques of adaptive, gradient, and extreme gradient boosting to anomaly free data to develop baseline models that predict the energy consumption patterns of smart home devices. The framework is evaluated on three publicly available energy datasets collected from various smart homes. The experimental results reveal that the cluster‐based local outlier factor with extreme gradient boosting achieves promising results with high prediction accuracy.

A Hybrid Approach for Efficient DDoS Detection in Network Traffic Using CBLOF-Based Feature Engineering and XGBoost

This is one of the greatest challenges in computer network security and cannot be dealt with without a set of most recent detection techniques. This paper lays down a new hybrid technique that combines Clustering-Based Local Outlier Factor (CBLOF) and Extreme Gradient Boosting (XGBoost) to enhance accuracy while detecting Distributed Denial of Service (DDoS) from network traffic. The proposed hybrid model utilizes a CBLOF for outlier detection as feature engineering. Over the detected anomalies, classification is to be done using XGBoost classification to attain the objective. The proposed hybrid model was tested extensively on CICIDS 2017 and CICIDS 2018 datasets Compared with traditional ones, the proposed model outperformed the traditional ones with an accuracy rate of 99.99%, precision of 100%, and F1 score reflecting perfection. These results confirm this model's efficiency in terms of known and novel attack patterns and introduce a further reliable framework for the timely detection of DDoS attacks. Even if it is computation-heavy, optimization could be made towards real-time large-scale data.

Adaptive Weighted Multi-kernel Learning for Blast-Induced Flyrock Distance Prediction

AbstractIn the field of civil and mining engineering, blasting operations are widely and frequently used for rock excavation, However, some undesirable environmental problems induced by blasting operations cannot be ignored. Blast-induced flyrock is one important issue induced by blasting operation, which needs to be well predicted to identify the blasting zone’s safety zone. This study introduces an adaptive weighted multi-kernel learning model (AW-MKL) to provide an accurate prediction of blast-induced flyrock distance in Sungun Copper Mine site. The proposed model uses a combination of multi-kernel learning (MKL) approach and adaptive weighting strategy based on weighted Euclidean distance and modified local outlier factor (MLOF) to maximally improve the predictive ability of kernel ridge regression (KRR). To demonstrate the superiority of the proposed approach, six machine learning models were developed as comparisons, i.e., KRR, RF, GBDT, SVM, M5 Tree, MARS and AdaBoost. The outcomes of the proposed method achieved the highest accuracy in testing phase, with RMSE of 2.05, MAE of 0.98 and VAF of 99.92, which confirmed the strong predictive capability of the proposed AW-MKL in predicting blast-induced flyrock distance.

Monitoring the gas metal arc additive manufacturing process using unsupervised machine learning

The study aimed to assess the performance of several unsupervised machine learning (ML) techniques in online anomaly (The term “anomaly” is used here to indicate a departure from expected process behavior which may indicate a quality issue which requires further investigation. The term “defect detection” has often been used previously but the specific imperfection is often indirectly inferred.) detection during surface tension transfer (STT)-based wire arc additive manufacturing. Recent advancements in quality monitoring for wire arc manufacturing were reviewed, followed by a comparison of unsupervised ML techniques using welding current and welding voltage data collected during a defect-free deposition process. Both time domain and frequency domain feature extraction techniques were applied and compared. Three analysis methodologies were adopted: ML algorithms such as isolation forest, local outlier factor, and one-class support vector machine. The results highlight that incorporating frequency analysis, such as fast Fourier transform (FFT) and discrete wavelet transform (DWT), for feature extraction based on general frequency response and defined bandwidth frequency response, significantly improves performance, reflected in a 14% increase in F2 score, compared with time-domain features extraction. Additionally, a deep learning approach employing a convolutional autoencoder (CAE) demonstrated superior performance by processing time-frequency domain data stored as spectrograms obtained through short-time Fourier transform (STFT) analysis. The CAE method outperformed frequency domain analysis and traditional ML approaches, achieving an additional 5% improvement in F2-score. Notably, the F2-score (The F2 score is the weighted harmonic mean of the precision and recall (given a threshold value). Unlike the F1 score, which gives equal weight to precision and recall, the F2 score gives more weight to recall than to precision.) increased significantly from 0.78 in time domain analysis to 0.895 in time-frequency analysis. The study emphasizes the potential of utilizing low-cost sensors to develop anomaly detection modules with enhanced accuracy. These findings underscore the importance of incorporating advanced data processing techniques in wire arc additive manufacturing for improved quality control and process optimization.

Improving mobile security: A study on android malware detection using LOF

Abstract The ubiquity of smartphones in our daily lives has made them attractive targets for malicious actors seeking to compromise user data and device functionality. Android malware detection has become imperative to protect user privacy and device integrity. This paper presents a focused study on leveraging the Local Outlier Factor (LOF) method for Android malware detection using the DREBIN dataset. Our research addresses the need for accurate and efficient Android malware detection. We explore the LOF method, an anomaly-based detection technique, to assess its effectiveness in distinguishing malicious applications from benign ones within the Android ecosystem. Rigorous experiments using the extensive DREBIN dataset reveal LOF's superiority in accuracy, precision, recall, and False Positive Rate (FPR). We introduce additional metrics like Area Under the Curve (AUC), Matthews Correlation Coefficient (MCC), and True Negative Rate (TNR) to comprehensively evaluate LOF. Our findings highlight LOF's ability to balance false positives and false negatives, making it an ideal choice for Android malware detection. We emphasize the importance of representative datasets, such as DREBIN, for validation. In conclusion, this research positions LOF as a reliable tool for Android malware detection, offering robust protection against emerging threats. As mobile technology evolves, our study encourages further exploration of advanced techniques and real-world deployment scenarios.

Effective Outlier Detection for Ensuring Data Quality in Flotation Data Modelling Using Machine Learning (ML) Algorithms

Froth flotation, a widely used mineral beneficiation technique, generates substantial volumes of data, offering the opportunity to extract valuable insights from these data for production line analysis. The quality of flotation data is critical to designing accurate prediction models and process optimisation. Unfortunately, industrial flotation data are often compromised by quality issues such as outliers that can produce misleading or erroneous analytical results. A general approach is to preprocess the data by replacing or imputing outliers with data values that have no connection with the real state of the process. However, this does not resolve the effect of outliers, especially those that deviate from normal trends. Outliers often occur across multiple variables, and their values may occur in normal observation ranges, making their detection challenging. An unresolved challenge in outlier detection is determining how far an observation must be to be considered an outlier. Existing methods rely on domain experts’ knowledge, which is difficult to apply when experts encounter large volumes of data with complex relationships. In this paper, we propose an approach to conduct outlier analysis on a flotation dataset and examine the efficacy of multiple machine learning (ML) algorithms—including k-Nearest Neighbour (kNN), Local Outlier Factor (LOF), and Isolation Forest (ISF)—in relation to the statistical 2σ rule for identifying outliers. We introduce the concept of “quasi-outliers” determined by the 2σ threshold as a benchmark for assessing the ML algorithms’ performance. The study also analyses the mutual coverage between quasi-outliers and outliers from the ML algorithms to identify the most effective outlier detection algorithm. We found that the outliers by kNN cover outliers of other methods. We use the experimental results to show that outliers affect model prediction accuracy, and excluding outliers from training data can reduce the average prediction errors.

CREDIT CARD FRAUD DETECTION USING MACHINE LEARNING

The rise of e-commerce and digital payment systems has been accompanied by an increase in financial fraud, especially involving credit cards. Ensuring the detection of fraudulent activities is crucial to protecting users' financial assets and preserving trust in online transactions. This study introduces a novel method for detecting credit card fraud by integrating machine learning (ML) techniques with a genetic algorithm (GA) for feature selection. Feature selection plays a vital role in improving fraud detection models by pinpointing the most relevant features linked to fraudulent transactions. The effectiveness of the proposed method is assessed using a dataset from European cardholders, a widely used benchmark in this research area. In terms of performance, the Isolation Forest Algorithm exhibits slightly higher accuracy compared to the Local Outlier Factor (LOF) Algorithm. Consequently, based on accuracy alone, the Isolation Forest Algorithm is deemed more effective for this specific dataset. Keywords: Credit card fraud detection, Machine learning (ML), E-commerce.

Fuzzy support vector machine using local outlier factor and intuitionistic fuzzy sets for imbalanced datasets

Traditional classifiers are commonly used for solving class balance problems. However, many datasets exhibit class imbalance along with outliers and noise, which affect the classification accuracy, this paper proposes a fuzzy support vector machine algorithm based on local outlier factor and intuitionistic fuzzy sets. First, to highlight the importance of the minority class, the membership degree is set to the maximum value. Meanwhile, local outlier factor is calculated to measure the abnormality and further obtain the membership degree in the majority class. Then, a kernel density estimation method is designed to estimate the sample density accurately. Furthermore, based on the density distribution, intuitionistic fuzzy sets are used to assign membership and non-membership degrees to samples, effectively distinguishing between noise and outliers. Finally, different penalty coefficients for two classes are built to offset the impact of class imbalance on classification accuracy. Experimental results show the advantages of the proposed algorithm.

Anomaly detection using unsupervised machine learning algorithms: A simulation study

This study presents a comprehensive evaluation of five prominent unsupervised machine learning anomaly detection algorithms: One-Class Support Vector Machine (One-Class SVM), One-Class SVM with Stochastic Gradient Descent (SGD), Isolation Forest (iForest), Local Outlier Factor (LOF), and Robust Covariance (Elliptic Envelope). Through systematic analysis on a synthetically simulated dataset, the study assessed each algorithm’s predictive performance using accuracy, precision, recall, and F1 score specifically for outlier detection. The evaluation reveals that One-Class SVM, Isolation Forest, and Robust Covariance are more effective in identifying outliers in the synthetic simulated dataset, with Isolation Forest slightly outperforming the other algorithms in terms of balancing precision and recall. One-Class SVM with SGD shows promise in precision but needs adjustment to improve recall. Local Outlier Factor may require parameter tuning or may not be as suitable for this particular dataset’s characteristics. The findings reveal significant variations in performance, highlighting the strengths and limitations of each method in identifying anomalies. This research contributes to the field of machine learning by demonstrating that the selection of an anomaly detection algorithm should be a considered decision, taking into account the specific characteristics of the data and the operational context of its application. Future work should explore parameter optimization, the impact of dataset characteristics on model performance, and the application of these models to real-world datasets to validate their efficacy in practical anomaly detection scenarios.

Integrating machine learning for sustaining cybersecurity in digital banks

Cybersecurity continues to be an important concern for financial institutions given the technology's rapid development and increasing adoption of digital services. Effective safety measures must be adopted to safeguard sensitive financial data and protect clients from potential harm due to the rise in cyber threats that target digital organizations. The aim of this study is to investigates how machine learning algorithms are integrated into cyber security measures in the context of digital banking and its benefits and drawbacks. We initially provide a general overview of digital banks and the particular security concerns that differentiate them from conventional banks. Then, we explore the value of machine learning in strengthening cybersecurity defenses. We revealed that insider threats, distributed denial of service (DDoS) assaults, ransomware, phishing attacks, and social engineering are main cyberthreats that are digital banks exposed. We identify the appropriate machine learning algorithms such as support vector machines (SVM), recurrent neural networks (RNN), hidden markov models (HMM), and local outlier factor (LOF) that are used for detection and prevention cyberthreats. In addition, we provide a model that considers ethical concerns while constructing a cybersecurity framework to address potential vulnerabilities in digital banking systems. The advantages and disadvantages of incorporating machine learning into the cybersecurity strategy of digital banks are outlined using strengths, weaknesses, opportunities, threats (SWOT) analysis. This study seeks to provide a thorough knowledge of how machine learning may strengthen cybersecurity procedures, protect digital banks, and maintain customer trust in the ecosystem of digital banking.

Energy-Efficient Anomaly Detection and Chaoticity in Electric Vehicle Driving Behavior.

Detection of abnormal situations in mobile systems not only provides predictions about risky situations but also has the potential to increase energy efficiency. In this study, two real-world drives of a battery electric vehicle and unsupervised hybrid anomaly detection approaches were developed. The anomaly detection performances of hybrid models created with the combination of Long Short-Term Memory (LSTM)-Autoencoder, the Local Outlier Factor (LOF), and the Mahalanobis distance were evaluated with the silhouette score, Davies-Bouldin index, and Calinski-Harabasz index, and the potential energy recovery rates were also determined. Two driving datasets were evaluated in terms of chaotic aspects using the Lyapunov exponent, Kolmogorov-Sinai entropy, and fractal dimension metrics. The developed hybrid models are superior to the sub-methods in anomaly detection. Hybrid Model-2 had 2.92% more successful results in anomaly detection compared to Hybrid Model-1. In terms of potential energy saving, Hybrid Model-1 provided 31.26% superiority, while Hybrid Model-2 provided 31.48%. It was also observed that there is a close relationship between anomaly and chaoticity. In the literature where cyber security and visual sources dominate in anomaly detection, a strategy was developed that provides energy efficiency-based anomaly detection and chaotic analysis from data obtained without additional sensor data.

Adaptive threshold based outlier detection on IoT sensor data: A node-level perspective

The accuracy and reliability of IoT-based sensor networks depend on validating sensed data, including detecting outliers at the node level. This study proposes an online outlier detection approach using Multiple Linear Regression-based adaptive thresholds for real-time IoT/WSN sensor nodes. IoT sensors experience two outlier types: Errors, from sensor malfunctions or low battery, and Events, from sudden environmental changes. The Adaptive Threshold Based Outlier Detection (ATBOD) approach differentiates errors from events using an adaptive threshold that adjusts to real-time data patterns. Unlike existing methods that are used in literature, which lack automated model evolution and suffer from delays and high computational time, ATBOD enhances outlier detection sensitivity without increasing false alarms, which is crucial for efficient IoT sensor board operation. It also improves sensor board lifespan by discarding errors at the node level, preventing energy wastage from transmitting error data to the cloud. ATBOD outperforms existing algorithms, which are referenced for comparison, such as Enhanced Efficient Outlier Detection and Classification Approach (EEODCA), K Nearest Neighbor approximate outlier detection (KNN), and Modified Local Outlier Factor (LOF), in Error Detection Rate, Error False Positive Rate, and Energy Saving Ratio. These advancements represent a significant leap in performance, making ATBOD a superior method for real-time outlier detection in IoT sensor networks.