Local Outlier Factor Algorithm Research Articles

A Long-and Short-term Hot User Identification Method Based on Local Outlier Density

Power systems have many users; however, there are relatively few users focused on high-quality power supplies (namely, hot users), such as external query users, electricity theft, and information fraud from billions of users. Starting from power load data, this paper postulates that abnormal electricity load users are essential components of hot users. According to the characteristics of regional power loads, this paper first builds a cross-judgment method for abnormal loads to improve the identification accuracy, including spatial and temporal abnormal load detection methods. As the spatial method, the abnormal load detection based on the local outlier factor (LOF) algorithm detects different users from similar users using several constructed dimensional features. As the temporal method, dynamic abnormal load detection based on the short-term sequential (SS) algorithm is designed to improve the LOF missing alarm rate and computing efficiency with multi-dimensional features. This also accomplished abnormal load detection from historical dimensions. A long- and short-term hot user detection method is designed based on the outlier density in response to the insufficient support of single-point abnormal loads to judge hot users. Finally, experiments on the user load data in a certain region within a year test the abnormal detection method for long- and short-term hot user detection based on local outlier density (LOD). The experimental results indicate that this method can detect abnormal loads and search hot users. The long- and short-term hot user detection method based on the LOD can meet practical needs when only using user load data and has good generalization abilities.

An unsupervised machine learning approach for real-time damage detection in bridges

The bridge network is progressively aging, with an alarming proportion of bridges over 100 years. This situation engenders substantial risks to the overall reliability of transportation networks, requiring innovative methods for efficient management. Monitoring can provide a direct source of information about structural behavior generating alerts when changes occur. Real-time alerts enable effective infrastructure management and decision-making during damage or anomalous situations. However, monitoring can result in a large amount of data that is often difficult to convert into valuable information in real time. This paper presents an approach for real-time detection of abrupt damage occurrence in bridges using unsupervised anomaly detection algorithms and strain/acceleration measurements. The approach incorporates the separation of measurements into events having the same loading nature and the construction of three feature matrices based on statistical features, time-frequency features, and wavelet spectrum features. It includes the evaluation of five anomaly detection algorithms including Isolation Forest, One-Class Support Vector Machine, Robust Random Cut Forest, Local Outlier Factor, and Mahalanobis Distance. The approach is illustrated with a case study of a steel-bascule-railway bridge, that has experienced a brittle cracking event during monitoring. Results highlight the robustness of One-Class Support Vector Machine, Isolation Forest, and Local Outlier Factor algorithms in promptly detecting abrupt changes across different features. The separation of strain and acceleration data into loading-based events, coupled with the comparison of previous and new event features, provides robust feature matrices for effective damage detection. Enhanced detection and higher scores are particularly attributed to time-frequency domain features during damage occurrence. The presented approach can be used as a base on how to perform real-time anomaly detection within the context of bridge monitoring.

A Method for Identifying Gross Errors in Dam Monitoring Data

Real and effective monitoring data are crucial in assessing the structural safety of dams. Gross errors, resulting from manual mismeasurement, instrument failure, or other factors, can significantly impact the evaluation process. It is imperative to eliminate such anomalous data. However, existing methods for detecting gross errors in concrete dam deformation often focus on analyzing a single monitoring effect quantity. This can lead to sudden jumps in values of effect quantity caused by changes in environmental variables being mistakenly identified as gross error. Therefore, a method based on Fuzzy C-Means clustering algorithm (FCM) partitioning and density clustering algorithm (Ordering Points To Identify the Clustering Structure, OPTICS) combined with Local Outlier Factor (LOF) algorithm for gross error identification is proposed. Firstly, the FCM algorithm is used to achieve the division of measurement point areas. Then, the OPTICS and LOF algorithms are jointly utilized to determine the gross errors. Finally, the real gross errors are identified by comparing the time of occurrence of the gross errors at measurement points in the same area. Through the case study, the results indicate that the method can effectively identify spurious, gross errors in the monitoring effect quantity caused by environmental mutations. The accuracy of gross error detection is significantly improved, and the rate of misjudgment of gross errors is reduced.

Fault Diagnosis Method for Lithium-Ion Power Battery Incorporating Multidimensional Fault Features

Accurately identifying a specific faulty monomer in a battery pack in the early stages of battery failure is essential to preventing safety accidents and minimizing property damage. While there are existing lithium-ion power battery fault diagnosis methods used in laboratory settings, their effectiveness in real-world vehicle conditions is limited. To address this, fault diagnosis methods for real-vehicle conditions should incorporate fault characteristic parameters based on external battery fault characterization, enabling the accurate identification of different fault types. However, these methods are constrained when confronted with complex fault types. To overcome these limitations, this paper proposes a battery fault diagnosis method that combines multidimensional fault features. By merging different fault feature parameters and mapping them to a high-dimensional space, the method utilizes a local outlier factor (LOF) algorithm to detect anomalous values, enabling fault diagnosis in complex working conditions. This method improves the detection time by an average of 22 min compared to the extended RMSE method and maintains strong robustness while correctly detecting faults compared to other conventional methods.

Automated operational modal analysis for supertall buildings based on a three-stage strategy and modified hierarchical clustering

Several automated operational modal analysis (AOMA) methods have been developed to evaluate the dynamic characteristics of large-scale structures in real-time under environmental and operational variations. Typically, these methods need to manually set some thresholds to achieve automatic identification. However, there is a lack of uniform standards for setting these thresholds, which may result in the adoption of inappropriate settings in AOMA, thereby introducing errors in modal identification. To address this issue, this paper proposes a novel AOMA framework to automatically identify modal parameters, which mainly involves three steps. Firstly, the covariance-driven stochastic subspace identification (Cov-SSI) algorithm is employed to perform modal estimation based on measured ambient vibration records. Then, a three-stage strategy, including hard validation criteria, Gaussian mixture model clustering-driven soft validation criteria, and the local outlier factor algorithm, is presented to remove spurious modes and determine real physical modes. Finally, a modified hierarchical clustering based on the k-nearest neighbor information is applied to group the determined physical modes to achieve automatic modal identification. The effectiveness of the proposed AOMA framework in automatically identifying modal parameters is validated through a numerical simulation study for a five-floor frame structure. Furthermore, the proposed framework is applied to evaluate the modal properties of a 600 m high supertall building based on long-term monitoring records. The results show that the AOMA framework can effectively remove spurious modes and group physical modes, thus identifying the modal parameters with high accuracy. In short, the proposed AOMA framework is an effective tool for modal identification applied in field measurements or structural health monitoring.

CREDIT CARD FRAUD DETECTION USING MACHINE LEARNING ALGORITHMS

The Credit card frauds represent facile and amiable targets, particularly as the proliferation of e-commerce and various online platforms has led to a commensurate expansion in online payment modalities, thereby augmenting the susceptibility to online fraudulent activities. In response to the escalating rates of fraudulent incidents, researchers have undertaken the utilization of diverse machine learning methodologies to ascertain and scrutinize frauds within the realm of online transactions. The primary objective of this scholarly endeavor is to formulate and advance an innovative fraud detection modus operandi tailored for Streaming Transaction Data. The overarching goal is to meticulously scrutinize the historical transaction particulars of patrons and distill discernible behavioral patterns. This involves the clustering of cardholders into disparate cohorts predicated upon their transactional magnitudes. Subsequently, a sliding window strategy is employed to amalgamate transactions conducted by cardholders across distinct groups, facilitating the extraction of their respective behavioral patterns. Consecutively, diverse classifiers trained on these distinct groups, and the classifier exhibiting superior rating scores is earmarked as one of the preeminent methods for prognosticating fraudulent activities. This is succeeded by the implementation of a feedback mechanism aimed at mitigating the challenges posed by the phenomenon of concept drift. The empirical investigation detailed in this paper is grounded in the analysis of a European credit card fraud dataset. It is imperative for credit card companies to adeptly discern instances of fraudulent credit card transactions to preclude customers from incurring charges for items they did not legitimately acquire. The resolution to such quandaries lies in the realm of Data Science, a discipline whose significance, when coupled with Machine Learning, is of paramount importance. This undertaking endeavors to elucidate the construction of a model utilizing machine learning techniques for Credit Card Fraud Detection. The Credit Card Fraud Detection Problem entails the modeling of historical credit card transactions, incorporating data from those that transpired as fraudulent. Subsequently, this model is employed to ascertain the veracity of new transactions, distinguishing between fraudulent and non-fraudulent activities. The primary aim is the meticulous detection of 100% of fraudulent transactions, while concurrently minimizing instances of erroneous classifications of non-fraudulent transactions. Credit Card Fraud Detection serves as a quintessential exemplar of classification challenges. In the course of this endeavor, significant emphasis has been placed on the analysis and pre-processing of datasets. Furthermore, a diverse array of anomaly detection algorithms, including the Local Outlier Factor and Isolation Forest algorithm, have been deployed on Principal Component Analysis (PCA) transformed Credit Card Transaction data. KEYWORDS: Card-Not-Present frauds, Card-Present-Frauds, Concept Drift,,Credit card fraud, applications of machine learning, data science.

Non-Destructive Detection of Golden Passion Fruit Quality Based on Dielectric Characteristics

This study pioneered a non-destructive testing approach to evaluating the physicochemical properties of golden passion fruit by developing a platform to analyze the fruit’s electrical characteristics. By using dielectric properties, the method accurately predicted the soluble solids content (SSC), Acidity and pulp percentage (PP) in passion fruit. The investigation entailed measuring the relative dielectric constant (ε′) and dielectric loss factor (ε″) for 192 samples across a spectrum of 34 frequencies from 0.05 to 100 kHz. The analysis revealed that with increasing frequency and fruit maturity, both ε′ and ε″ showed a declining trend. Moreover, there was a discernible correlation between the fruit’s physicochemical indicators and dielectric properties. In refining the dataset, 12 outliers were removed using the Local Outlier Factor (LOF) algorithm. The study employed various advanced feature extraction techniques, including Recursive Feature Elimination with Cross-Validation (RFECV), Permutation Importance based on Random Forest Regression (PI-RF), Permutation Importance based on Linear Regression (PI-LR) and Genetic Algorithm (GA). All the variables and the selected variables after screening were used as inputs to build Extreme Gradient Boosting (XGBoost) and Categorical Boosting (Cat-Boost) models to predict the SSC, Acidity and PP in passion fruit. The results indicate that the PI-RF-XGBoost model demonstrated superior performance in predicting both the SSC (R2 = 0.9240, RMSE = 0.2595) and the PP (R2 = 0.9092, RMSE = 0.0014) of passion fruit. Meanwhile, the GA-CatBoost model exhibited the best performance in predicting Acidity (R2 = 0.9471, RMSE = 0.1237). In addition, for the well-performing algorithms, the selected features are mainly concentrated within the frequency range of 0.05–6 kHz, which is consistent with the frequency range highly correlated with the dielectric properties and quality indicators. It is feasible to predict the quality indicators of fruit by detecting their low-frequency dielectric properties. This research offers significant insights and a valuable reference for non-destructive testing methods in assessing the quality of golden passion fruit.

Research on Inconsistency Evaluation of Retired Battery Systems in Real-World Vehicles

Inconsistency is a key factor triggering safety problems in battery packs. The inconsistency evaluation of retired batteries is of great significance to ensure the safe and stable operation of batteries during subsequent gradual use. This paper summaries the commonly used diagnostic methods for battery inconsistency assessment. The local outlier factor (LOF) algorithm and the improved Shannon entropy (ImEn) algorithm are selected for validation based on the individual voltage data from real-world vehicles. Then, a comprehensive inconsistency evaluation strategy for retired batteries with many levels and indicators is established based on the three parameters of LOF, ImEn, and cell voltage range. Finally, the evaluation strategy is validated using two real-world vehicle samples of retired batteries. The results show that the proposed method can achieve the inconsistency evaluation of retired batteries quickly and effectively.

Non-intrusive temperature measurements for transient freezing in laminar internal flow using laser induced fluorescence

This work presents two color LIF temperature measurements for the transient freezing in a square channel under laminar flow conditions. This is the first time non-intrusive temperature measurements were performed within the thermal boundary layer during the transient growth of an ice layer in internal flow. A combination of a local outlier factor algorithm and a smoothing operation was used to remove the top to bottom striations and reduce the other measurement noise. The temperature uncertainty in our measurements was between σ=0.3∘C and σ=0.5∘C. For the largest temperature difference between the bulk and the melting point of 14.6 °C, good results were obtained. As such, the current campaign demonstrates the potential of LIF as a non-intrusive temperature measurement technique for solid–liquid phase change experiments. However, some artefacts were present within the vicinity of the ice-layer due to the scattering of the laser light, especially near the inlet of the channel where the ice-layer is curved instead of flat. LIF measurements taken within a short time span prior to the onset of ice freezing showed approximately 2 °C of subcooling, consistent with previous findings. In addition, an anomalous behavior within the thermal boundary layer was observed, with a much smaller temperature gradient within the first few mm above the cold plate and a point of inflection in the temperature profile. The anomalous temperature behavior is possibly attributed to enhanced natural convection as a result of the subcooling at the cold plate surface.

Unsupervised Bayesian change-point detection approach for reliable prognostics and health management of complex mechanical systems

In the prognostics and health management of mechanical systems, achieving high reliability requires high-quality data for analysis and model training to avoid intensive misalarms and false detections. This research introduces a novel unsupervised Bayesian change-point detection approach specifically tailored to improve the accuracy and dependability of data used in the performance evaluation, fault diagnostics, and predictive maintenance of such systems. By leveraging this method, the precision of change-point detection in operational data is significantly heightened, effectively discerning between normal conditions and potential outliers within the recorded data sequences. The proposed approach replaces the reliance on traditional outlier detection methods with an innovative utilization of prediction error metrics, enriched with statistical analysis to assess and bolster the robustness of the detection algorithm. Conducted case studies on data from the supervisory control and data acquisition (SCADA) system of 92 wind turbines highlight the advanced data cleaning capabilities of our method, which markedly outperforms previous techniques including quartile, Gaussian mixture models, isolation forests, and local outlier factor algorithms. The findings illustrate the methodology’s efficacy in obtaining rcleaned data, which are crucial for the reliable prognostics and health management frameworks of mechanic system, and enhance the confidence in feature extraction and performance prediction.

Imbalanced customer churn classification using a new multi-strategy collaborative processing method

The rapid advancement of big data and artificial intelligence heralds a dual-edged era of opportunities and challenges for the banking sector. Indeed, enhancing a model's capability to accurately classify imbalanced datasets represents a critical challenge within the field of customer churn prediction (CCP). In this paper, to address the challenges presented by the problem of imbalanced customer classification, a new multi-strategy collaborative processing method named IADASYN-FLCatBoost is proposed from dual perspectives: data and algorithm. At the data level, the traditional Adaptive Synthetic (ADASYN) sampling is improved, that is, the LOF (Local Outlier Factor) algorithm is introduced to eliminate outliers, and the classification features are specially processed to synthesize new minority class samples, thus an improved ADASYN (IADASYN) algorithm is obtained. At the algorithm level, the Focal Loss is embedded into the CatBoost ensemble learning framework to form a new Focal Loss-CatBoost (FLCatBoost) to make a focal-aware, cost-sensitive version of imbalanced customer churn prediction. Moreover, the empirical analysis is conducted in conjunction with the credit card customer dataset obtained from the Kaggle platform. The results of the staged comparison experiments show that the proposed method IADASYN-FLCatBoost in this paper shows the best prediction performance. Comparing the proposed method with 5 other imbalanced classification algorithms and 20 classifiers composed of classical sampling methods and ensemble learning algorithms, it is verified that the classification effect of the proposed method performs best, and the values of Recall, F1 score, G-mean and Area under Precision-Recall curve (AUPRC) have been significantly improved. In addition, further verification of the model also proves that the proposed method has certain generalizability and is still valid for other banks and customer churn datasets of other industries.

An Approach for Detecting Local Outliers in Grid Queries

The density local outlier factor algorithm (LOF) needs to calculate the distance matrix for k-nearest neighbor search. The algorithm has high time complexity and is not suitable for the detection of large-scale data sets. A local outlier detection algorithm is proposed based on grid query (LOGD). In the algorithm, the k other data points closest to the data point in the target grid must be in the target grid or in the nearest neighboring grid of the target grid, it is used to improve the neighborhood query operation of the LOF algorithm, the calculation amount of the LOF algorithm is reduced in the neighborhood query. Experimental results show that the proposed LODG algorithm can effectively reduce the time of outlier detection under the condition, the detection accuracy of the original LOF algorithm is basically the same.

Enhancing Insider Threat Detection in Imbalanced Cybersecurity Settings Using the Density-Based Local Outlier Factor Algorithm

Enhancing Insider Threat Detection in Imbalanced Cybersecurity Settings Using the Density-Based Local Outlier Factor Algorithm

UNSUPERVISED MACHINE LEARNING FOR SEISMIC ANOMALY DETECTION: LOCAL OUTLIER FACTOR ALGORITHM TO INDONESIAN EARTHQUAKE DATA

Indonesia's location on the "Ring of Fire" poses a high risk for seismic events. Addressing this, our study applied the Local Outlier Factor (LOF) algorithm for advanced seismic anomaly detection, crucial for geotectonic upheaval prediction. The LOF, adept at unsupervised learning in label-scarce datasets, analyzed data from the Indonesian Meteorology, Climatology, and Geophysical Agency, validated for integrity. Our approach, considering local density deviations, offered a refined alternative to conventional threshold-based detection, accommodating seismic data's intrinsic variability. The LOF algorithm successfully pinpointed anomalies, revealing unique seismic events unconstrained by geography or time. A comparative analysis underscored the LOF's superiority in recognizing local deviations and handling disparate data densities. These findings highlight the LOF's utility in strengthening seismic risk mitigation and anticipatory measures. The diverse anomalies identified, varying in magnitude and depth, reflect Indonesia's complex seismic interplay. To conclude, the LOF proves potent for anomaly detection, potentially elevating public safety and disaster preparedness. Future research will compare the LOF with other unsupervised methods, seeking to deepen seismic risk comprehension

Automatic Extraction of Surface Wave Dispersion Curves Using Unsupervised Learning and High‐Resolution Tau‐p Transform

AbstractDispersion curves for surface wave recordings are required input for many surface wave inversion methods to image subsurface shear wave velocity distribution, while the conventional extraction of dispersion curves requires significant amount of human interaction. This step impedes efficiency enhancement of surface wave analysis and its automation. In this paper, we present an unsupervised learning scheme to achieve efficient automatic picking of dispersion curves of the fundamental mode for surface wave gathers. This scheme is composed of four major steps: computing a frequency velocity(f‐v) spectrum for the surface wave gather using a high‐resolution Tau‐p transform improved by the iteratively shrinkage Thresholding algorithm (ISTA) algorithm, generating clusters points along the dispersion energy in the f‐v spectrum via a weighted Kmeans algorithm, filtering these cluster points by principal component analysis (PCA) and Local Outlier Factor (LOF) algorithms to remove the erroneous clusters, and fitting the remaining clusters to form the dispersion curve. Tests with synthetic and field noisy surface wave recordings demonstrated the effectiveness of this approach and its potential in automatic processing of noisy surface wave data sets.

Abnormal Data Detection Based on Adaptive Sliding Window and Weighted Multiscale Local Outlier Factor for Machinery Health Monitoring

Identifying abnormal data to improve data quality is of great importance for machinery health monitoring (MHM). Existing abnormal data detection methods generally depend on appropriate parameter settings and prior knowledge of data distribution, which result in relatively low adaptability to MHM data. To obtain more reliable MHM results, this paper proposed a novel method to detect abnormal data. First, the concept of adaptive sliding window (ASW) is defined and the advantages of ASW in avoiding data leakage and data redundancy are derived. The ASW can optimally divide the overall data into several segments. Next, statistical factors in time- and frequency-domain are extracted from these segments to generate corresponding research objects. Then, an improved weighted multiscale local outlier factor (WMLOF) algorithm is proposed herein to evaluate the data anomaly degree of each object. The WMLOF has the ability to assess and fuse the local outlier factor characteristics at multiple scales, and eventually yields a more comprehensive WMLOF value to evaluate the anomaly degree of the MHM data. Finally, the effectiveness and superiority of the ASW and WMLOF are validated by a synthetic simulation of a faulty rolling bearing and two engineering projects pertaining to a railway vehicle gearbox, and bench test data. Comparative analysis shows that the proposed abnormal data detection method based on the ASW and WMLOF strategies outperforms five reported algorithms in outlier detection.

Detection and recovery of anomalous vibration signal of rotating machinery based on LOF-MSAMP

The collected vibration signals of rotating machinery contain pulses, missing, and other low-quality anomalous data due to environmental noise interference, unstable data transmission, and data acquisition instrument failure. These low-quality data obstruct the analysis of the healthy operation condition of rotating machinery. This paper proposes a method for anomalous vibration signal detection and recovery based on the local outlier factor algorithm and the modified sparsity adaptive matching pursuit algorithm. The method combines the local outlier factor algorithm and compressive sensing theory to realize anomalous vibration signal detection and recovery. This paper evaluates the recovery performance both qualitatively and quantitatively and discusses how the proposed method’s hyperparameter selection affects the recovery results. A set of simulated signal and measured hob base signal are used to verify the proposed method. The results indicate that, when compared to the other seven reconstruction algorithms, the proposed method’s recovered signal has the lower error level and the higher waveform similarity which reaches more than 98% to the original signal, effectively improving data quality.

A Hybrid Unsupervised Density-based Approach with Mutual Information for Text Outlier Detection

The detection of outliers in text documents is a highly challenging task, primarily due to the unstructured nature of documents and the curse of dimensionality. Text document outliers refer to text data that deviates from the text found in other documents belonging to the same category. Mining text document outliers has wide applications in various domains, including spam email identification, digital libraries, medical archives, enhancing the performance of web search engines, and cleaning corpora used in document classification. To address the issue of dimensionality, it is crucial to employ feature selection techniques that reduce the large number of features without compromising their representativeness of the domain. In this paper, we propose a hybrid density-based approach that incorporates mutual information for text document outlier detection. The proposed approach utilizes normalized mutual information to identify the most distinct features that characterize the target domain. Subsequently, we customize the well-known density-based local outlier factor algorithm to suit text document datasets. To evaluate the effectiveness of the proposed approach, we conduct experiments on synthetic and real datasets comprising twelve high-dimensional datasets. The results demonstrate that the proposed approach consistently outperforms conventional methods, achieving an average improvement of 5.73% in terms of the AUC metric. These findings highlight the remarkable enhancements achieved by leveraging normalized mutual information in conjunction with a density-based algorithm, particularly in high-dimensional datasets.

LOFReg: An outlier-based regulariser for deep metric learning

Deep metric learning aims to create a feature space where the projected samples have maximised inter-class and minimised intra-class distance. Most approaches employ only distance-based metrics to achieve this objective, but neglect other properties of the projections in the embedding space, such as their density, sparsity and presence of outliers. In this paper, we propose a novel density-based regulariser, LOFReg, designed to be used as a complement to previously proposed distance-based metric learning loss functions for the re-identification (ReID) and few-shot classification (FSC) tasks. Our method is based on the well-known, in anomaly detection literature, local outlier factor (LOF) algorithm, which estimates the local density deviation of a data point with respect to its neighbours. These measurements are used in our regularisation methodology to achieve an embedding space of evenly distributed samples and to increase the generalisation ability of the model compared to solely distance-based learning. Comprehensive experiments on four publicly available datasets for ReID and FSC, demonstrate consistent improvement against previously proposed metric learning loss functions. Particularly, our experiments show up to 5% improvement in ReID settings, up to 13% in FSC settings and up to 6% improvement against other state-of-art regularisers.

VOYAGE SPEED OPTIMIZATION USING GENETIC ALGORITHM

Decreasing the fuel consumption and thus greenhouse gas emissions of vessels have emerged as a critical topic for both ship operators and policymakers in recent years. The speed of vessels has long been recognized to have the highest impact on fuel consumption. The aim of this study is to develop a speed optimization model using a time-constrained genetic algorithm (GA). Subsequent to this, this paper also presents the application of machine learning regression methods in constructing a model to predict the fuel consumption of vessels. The local outlier factor algorithm is used to eliminate outliers in prediction features. The overfitting problem is observed after hyperparameter tuning in boosting and tree-based regression prediction methods. The early stopping technique is applied for overfitted models. In this study, speed is found to be the most significant feature for fuel consumption prediction. On the other hand, GA evaluation results showed that random modifications in the default speed profile could increase GA performance and thus fuel savings more than constant speed limits during voyages.