Local Outlier Research Articles

Real-Time Early Warning Method of Distribution Transformer Load Considering Meteorological Factor Data

The traditional real-time load warning method for distribution transformers has problems such as low recall rate, low warning accuracy, and long warning time, which may lead to potential equipment failures or overload situations not being detected and dealt with in a timely manner, increasing the safety risk of transformer operation and potentially causing safety issues such as equipment damage, fire, or power outage. Therefore, a real-time early warning method of distribution transformer load considering meteorological factor data is designed. The meteorological factor data are collected by the light sensor, humidity sensor, temperature sensor and rainfall sensor, and the load data collection architecture is built by the load monitor, central master station and maintenance station to realize the load data collection of the distribution transformer. The K-nearest neighbor (KNN) method is used to process the missing values of the data, and the LOF algorithm is used to determine the local outliers and eliminate the outliers in the data set to achieve data cleaning. Considering the load loss, hot spot temperature and meteorological factors of the distribution transformer, an early warning model is built, and the cleaned data are input into the model to realize Real-time early warning of the distribution transformer load. The experimental results show that the recall rate of this method varies from 95% to 97%, the accuracy rate of early warning is always above 94%, and the maximum value of early warning time is 0.63s. Having good early warning ability.

That’s BAD: blind anomaly detection by implicit local feature clustering

Recent studies on visual anomaly detection (AD) of industrial objects/textures have achieved quite good performance. They consider an unsupervised setting, specifically the one-class setting, in which we assume the availability of a set of normal (i.e., anomaly-free) images for training. In this paper, we consider a more challenging scenario of unsupervised AD, in which we detect anomalies in a given set of images that might contain both normal and anomalous samples. The setting does not assume the availability of known normal data and thus is completely free from human annotation, which differs from the standard AD considered in recent studies. For clarity, we call the setting blind anomaly detection (BAD). We show that BAD can be converted into a local outlier detection problem and propose a novel method named PatchCluster that can accurately detect image- and pixel-level anomalies. Experimental results show that PatchCluster shows a promising performance without the knowledge of normal data, even comparable to the SOTA methods applied in the one-class setting needing it.

Research on Inconsistency Evaluation of Retired Battery Systems in Real-World Vehicles

Inconsistency is a key factor triggering safety problems in battery packs. The inconsistency evaluation of retired batteries is of great significance to ensure the safe and stable operation of batteries during subsequent gradual use. This paper summaries the commonly used diagnostic methods for battery inconsistency assessment. The local outlier factor (LOF) algorithm and the improved Shannon entropy (ImEn) algorithm are selected for validation based on the individual voltage data from real-world vehicles. Then, a comprehensive inconsistency evaluation strategy for retired batteries with many levels and indicators is established based on the three parameters of LOF, ImEn, and cell voltage range. Finally, the evaluation strategy is validated using two real-world vehicle samples of retired batteries. The results show that the proposed method can achieve the inconsistency evaluation of retired batteries quickly and effectively.

Unsupervised Bayesian change-point detection approach for reliable prognostics and health management of complex mechanical systems

In the prognostics and health management of mechanical systems, achieving high reliability requires high-quality data for analysis and model training to avoid intensive misalarms and false detections. This research introduces a novel unsupervised Bayesian change-point detection approach specifically tailored to improve the accuracy and dependability of data used in the performance evaluation, fault diagnostics, and predictive maintenance of such systems. By leveraging this method, the precision of change-point detection in operational data is significantly heightened, effectively discerning between normal conditions and potential outliers within the recorded data sequences. The proposed approach replaces the reliance on traditional outlier detection methods with an innovative utilization of prediction error metrics, enriched with statistical analysis to assess and bolster the robustness of the detection algorithm. Conducted case studies on data from the supervisory control and data acquisition (SCADA) system of 92 wind turbines highlight the advanced data cleaning capabilities of our method, which markedly outperforms previous techniques including quartile, Gaussian mixture models, isolation forests, and local outlier factor algorithms. The findings illustrate the methodology’s efficacy in obtaining rcleaned data, which are crucial for the reliable prognostics and health management frameworks of mechanic system, and enhance the confidence in feature extraction and performance prediction.

Modal identification of non‐classically damped structures using generalized sparse component analysis

SummaryModal identification method based on blind source separation (BSS) technique has gained extensive attentions for civil structures. Developing the complex modes estimation method is important in practical applications because the assumption of proportional damping is not always satisfied. Sparse component analysis (SCA) performs well in underdetermined BSS problems. However, SCA is confined to the situation of proportional damping. In this study, a generalized SCA method is proposed to extend the original SCA method to both real and complex modes identification. First, the general formulation of complex modes is extended by the analytic form to eliminate the complex conjugate part in the BSS model. A new single‐source‐point detection method that is available to handle real and complex modes is proposed. Local outlier factor method is adopted to remove the outliers in single source points. Subsequently, complex‐valued modal matrix is calculated by the clustering technique. Then, modal responses are recovered using the complex version of smoothed zero norm method, where modal frequencies and damping ratios can be extracted. Finally, the effectiveness of the proposed method is demonstrated for identification of real and complex modes, close modes, and underdetermined problem. The application to a benchmark structure demonstrates the effectiveness for practical applications.

Construction of ellipsoid convex model of bounded uncertainties with outlier detection for application in non-probabilistic topology optimization

The ellipsoid convex model can be suitably used for structural analysis and design optimization under uncertain-but-bounded parameters and loads. Such a model can be constructed using measured samples of the uncertainties. However, the presence of outliers is often unavoidable due to system fluctuations in the measurements. Thus, it is necessary to detect any outliers among the samples before constructing the uncertainty model to prevent over-conservativeness. To this end, the present paper proposes a rational approach for constructing ellipsoid convex models with outlier detection. The concept of the local outlier factor (LOF) is utilized, in conjunction with the k-nearest natural neighbor to adaptively determine the neighborhood range. Then the outliers are detected using the scaled median absolute deviation method, based on the LOF values obtained. Finally, the minimum-volume ellipsoid convex model is constructed with a mathematically strict and efficient semi-definite programming formulation using the normal samples. The validity of the proposed approach is demonstrated with numerical examples. The application of the constructed uncertainty model in non-probabilistic robust topology optimization is demonstrated, and the results show the effectiveness of the proposed approach.

An empirical study on the stochastic long-term travel demands of a large-scale metro network

The widely-existed uncertainty of origin–destination (OD) demand in transportation networks has attracted extensive attention. Most characterizations or modeling of stochastic OD demands in networks assume a homogenous probability distribution, but it lacks empirical studies of large-scale networks to justify this assumption. Given that the long-term continuous automatic fare collection data of metro networks can provide complete OD passenger demand information, this study takes the Shanghai Metro network as an example to empirically examine the stochasticity characteristics of OD passenger demands of metro network. Based on the morning peak OD demand data for 250 weekdays, a local outlier factor method is used to identify and remove outliers in the data. A clustering method is used to cluster the OD pairs, study the fluctuation and distribution characteristics of the OD passenger demands, and select the optimal distribution type through goodness-of-fit indices. The results show that 1) the coefficients of variation of morning peak OD demands in the network are mainly distributed in the range of 0.2–0.6, and different OD pairs have different fluctuations, and the degree of demand fluctuation decreases as the mean increases; 2) the probability distribution types of OD demands based on statistical characteristics are heterogeneous; and 3) the optimal distribution type of OD demands is Poisson, Lognormal/Gamma, and Normal for OD pairs characterized by a small mean and right-skewness, a small mean and skewness close to 0, and a large mean, respectively. In contrast to the simple average-based data processing of OD passenger demand in metro networks, this paper presents a new perspective of mining long-term continuous data to understand the inherent stochasticity of OD passenger demands. The results can provide more realistic and practical inputs and assumptions for theoretical research on stochastic OD demands in metro networks.

Imbalanced customer churn classification using a new multi-strategy collaborative processing method

The rapid advancement of big data and artificial intelligence heralds a dual-edged era of opportunities and challenges for the banking sector. Indeed, enhancing a model's capability to accurately classify imbalanced datasets represents a critical challenge within the field of customer churn prediction (CCP). In this paper, to address the challenges presented by the problem of imbalanced customer classification, a new multi-strategy collaborative processing method named IADASYN-FLCatBoost is proposed from dual perspectives: data and algorithm. At the data level, the traditional Adaptive Synthetic (ADASYN) sampling is improved, that is, the LOF (Local Outlier Factor) algorithm is introduced to eliminate outliers, and the classification features are specially processed to synthesize new minority class samples, thus an improved ADASYN (IADASYN) algorithm is obtained. At the algorithm level, the Focal Loss is embedded into the CatBoost ensemble learning framework to form a new Focal Loss-CatBoost (FLCatBoost) to make a focal-aware, cost-sensitive version of imbalanced customer churn prediction. Moreover, the empirical analysis is conducted in conjunction with the credit card customer dataset obtained from the Kaggle platform. The results of the staged comparison experiments show that the proposed method IADASYN-FLCatBoost in this paper shows the best prediction performance. Comparing the proposed method with 5 other imbalanced classification algorithms and 20 classifiers composed of classical sampling methods and ensemble learning algorithms, it is verified that the classification effect of the proposed method performs best, and the values of Recall, F1 score, G-mean and Area under Precision-Recall curve (AUPRC) have been significantly improved. In addition, further verification of the model also proves that the proposed method has certain generalizability and is still valid for other banks and customer churn datasets of other industries.

Machine Learning Approaches to Advanced Outlier Detection in Psychological Datasets

The core aim of this study is to determine the most effective outlier detection methodologies for multivariate psychological datasets, particularly those derived from Omani students. Due to their complex nature, such datasets demand robust analytical methods. To this end, we employed three sophisticated algorithms: local outlier factor (LOF), one-class support vector machine (OCSVM), and isolation forest (IF). Our initial findings showed 155 outliers by both LOF and IF and 147 by OCSVM. A deeper analysis revealed that LOF detected 55 unique outliers based on differences in local density, OCSVM isolated 44 unique outliers utilizing its transformed feature space, and IF identified 76 unique outliers leveraging its tree-based mechanics. Despite these varying results, all methods had a consensus for just 44 outliers. Employing ensemble techniques, both averaging and voting methods identified 155 outliers, whereas the weighted method highlighted 151, with a consensus of 150 outliers across the board. In conclusion, while individual algorithms provide distinct perspectives, ensemble techniques enhance the accuracy and consistency of outlier detection. This underscores the necessity of using multiple algorithms with ensemble techniques in analyzing psychological datasets, facilitating a richer comprehension of inherent data structures.

An Approach for Detecting Local Outliers in Grid Queries

The density local outlier factor algorithm (LOF) needs to calculate the distance matrix for k-nearest neighbor search. The algorithm has high time complexity and is not suitable for the detection of large-scale data sets. A local outlier detection algorithm is proposed based on grid query (LOGD). In the algorithm, the k other data points closest to the data point in the target grid must be in the target grid or in the nearest neighboring grid of the target grid, it is used to improve the neighborhood query operation of the LOF algorithm, the calculation amount of the LOF algorithm is reduced in the neighborhood query. Experimental results show that the proposed LODG algorithm can effectively reduce the time of outlier detection under the condition, the detection accuracy of the original LOF algorithm is basically the same.

Comparison of prediction models for soy protein isolate hydrolysates bitterness built using sensory, spectrofluorometric and chromatographic data from varying enzymes and degree of hydrolysis

The bitterness of soy protein isolate hydrolysates prepared using five proteases at varying degree of hydrolysis (DH) and its relation to physicochemical properties, i.e., surface hydrophobicity (H0), relative hydrophobicity (RH), and molecular weight (MW), were studied and developed for predictive modelling using machine learning. Bitter scores were collected from sensory analysis and assigned as the target, while the physicochemical properties were assigned as the features. The modelling involved data pre-processing with local outlier factor; model development with support vector machine, linear regression, adaptive boosting, and K-nearest neighbors algorithms; and performance evaluation by 10-fold stratified cross-validation. The results indicated that alcalase hydrolysates were the most bitter, followed by protamex, flavorzyme, papain, and bromelain. Distinctive correlation results were found among the physicochemical properties, influenced by the disparity of each protease. Among the features, the combination of RH-MW fitted various classification models and resulted in the best prediction performance.

A performance study of local outlier detection methods for mineral exploration with geochemical compositional data

In exploration geochemistry, mineral deposits are typically characterised by an enrichment of the targeted elements, and thus their element composition differs from that of samples in a local neighbourhood. Local outlier detection methods aim at identifying local changes. In contrast to conventional outlier detection procedures, local outlier detection methods are multivariate methods for outlier identification that incorporate the spatial neighbourhood of the samples. It is essential that geochemical data are treated as compositional data, and the requirements for their treatment depend on the specific local outlier detection method. We demonstrate how prominent local outlier detection methods can be used for mineral exploration with geochemical data that vary in scale, in the sampling density, and in data quality. The methods are compared based on known mineralisations, and recommendations for their usefulness are provided.

Anomaly Detection in Binary Time Series Data: An unsupervised Machine Learning Approach for Condition Monitoring

A key element of smart manufacturing is condition monitoring and heath controlling of production machines. In today's rapidly evolving landscape of industrial machinery and equipment, optimizing the operation of production lines is critical to ensure high productivity and product quality. Timely detection and prevention of faults in the production process plays a crucial role in minimizing downtime, reducing costs, and ensuring optimal performance. The scientific challenge here is that the increasing number of sensors and actuators with digital input and output signals in the production machines creates different patterns, which are difficult to evaluate using conventional statistical methods. Another difficulty is identifying the cause of the failure to be able to intervene rapidly and in a focused manner in the event of irregularities. For this reason, this research study presented a comprehensive analysis of anomaly detection in binary time series data using various machine learning models. The study included preprocessing of the dataset, normalizing the data, and evaluation of the anomaly detection performance of the different models. The accuracy, detection rate, and F1-score are used as evaluation measures. The execution time of each model is also analyzed. In addition, the identification of sensors that cause anomalies is investigated and the impact of false detections is discussed. Experimental results show the strengths and weaknesses of each model and provide valuable insights for selecting the appropriate anomaly detection approach. The Isolation Forest, Local Outlier Factor, DBSCAN, and kMeans models show high precision and detection, while the Autoencoder and Variational Autoencoder models show high precision but lower detection. The one-class Support Vector Machine model achieves balanced performance. AutoML shows excellent results in recognition rate but is not real-time capable. The results highlight the trade-offs between performance and computational efficiency and point to further research potential in real-time implementation.

Heterogeneous density-based clustering with a dual-functional memristive array.

In the big data era, the requirement for data clustering methods that can handle massive and heterogeneous datasets with varying distributions increases. This study proposes a clustering algorithm for data sets with heterogeneous density using a dual-mode memristor crossbar array for data clustering. The array consists of a Ta/HfO2/RuO2 memristor operating in analog or digital modes, controlled by the reset voltage. The digital mode shows low dispersion and a high resistance ratio, and the analog mode enables precise conductance tuning. The local outlier factor is introduced to handle a heterogeneous density, and the required Euclidean and K-distances within the given dataset are calculated in the analog mode in parallel. In the digital mode, clustering is performed based on the connectivity among data points after excluding the detected outliers. The proposed algorithm boasts linear time complexity for the entire process. Extensive evaluations of synthetic datasets demonstrate significant improvement over representative density-based algorithms, and the datasets with heterogeneous density are clustered feasibly. Finally, the proposed algorithm is used to cluster the single-molecule localization microscopy data, demonstrating the feasibility of the suggested method for real-world problems.

Hybrid computing framework security in dynamic offloading for IoT-enabled smart home system.

In the distributed computing era, cloud computing has completely changed organizational operations by facilitating simple access to resources. However, the rapid development of the IoT has led to collaborative computing, which raises scalability and security challenges. To fully realize the potential of the Internet of Things (IoT) in smart home technologies, there is still a need for strong data security solutions, which are essential in dynamic offloading in conjunction with edge, fog, and cloud computing. This research on smart home challenges covers in-depth examinations of data security, privacy, processing speed, storage capacity restrictions, and analytics inside networked IoT devices. We introduce the Trusted IoT Big Data Analytics (TIBDA) framework as a comprehensive solution to reshape smart living. Our primary focus is mitigating pervasive data security and privacy issues. TIBDA incorporates robust trust mechanisms, prioritizing data privacy and reliability for secure processing and user information confidentiality within the smart home environment. We achieve this by employing a hybrid cryptosystem that combines Elliptic Curve Cryptography (ECC), Post Quantum Cryptography (PQC), and Blockchain technology (BCT) to protect user privacy and confidentiality. Additionally, we comprehensively compared four prominent Artificial Intelligence anomaly detection algorithms (Isolation Forest, Local Outlier Factor, One-Class SVM, and Elliptic Envelope). We utilized machine learning classification algorithms (random forest, k-nearest neighbors, support vector machines, linear discriminant analysis, and quadratic discriminant analysis) for detecting malicious and non-malicious activities in smart home systems. Furthermore, the main part of the research is with the help of an artificial neural network (ANN) dynamic algorithm; the TIBDA framework designs a hybrid computing system that integrates edge, fog, and cloud architecture and efficiently supports numerous users while processing data from IoT devices in real-time. The analysis shows that TIBDA outperforms these systems significantly across various metrics. In terms of response time, TIBDA demonstrated a reduction of 10-20% compared to the other systems under varying user loads, device counts, and transaction volumes. Regarding security, TIBDA's AUC values were consistently higher by 5-15%, indicating superior protection against threats. Additionally, TIBDA exhibited the highest trustworthiness with an uptime percentage 10-12% greater than its competitors. TIBDA's Isolation Forest algorithm achieved an accuracy of 99.30%, and the random forest algorithm achieved an accuracy of 94.70%, outperforming other methods by 8-11%. Furthermore, our ANN-based offloading decision-making model achieved a validation accuracy of 99% and reduced loss to 0.11, demonstrating significant improvements in resource utilization and system performance.

Comparative Analysis of ML-Based Outlier Detection Techniques for IoT-Based Smart Energy Management Systems

With the development and advancement of ICST, data-driven technology such as the Internet of Things (IoT) and Smart Technology including Smart Energy Management Systems (SEMS) has become a trend in many regions and around the globe. There is no doubt that data quality and data quality problems are among the most vital topics to be addressed for a successful application of IoT-based SEMS. Poor data in such major yet delicate systems will affect the quality of life (QoL) of millions, and even cause destruction and disruption to a country. This paper aims to tackle this problem by searching for suitable outlier detection techniques from the many developed ML-based outlier detection methods. Three methods are chosen and analyzed for their performances, namely the K-Nearest Neighbour (KNN)+ Mahalanobis Distance (MD), Minimum Covariance Determinant (MCD), and Local Outlier Factor (LOF) models. Three sensor-collected datasets that are related to SEMS and with different data types are used in this research, they are pre-processed and split into training and testing datasets with manually injected outliers. The training datasets are then used for searching the patterns of the datasets through training of the models, and the trained models are then tested with the testing datasets, using the found patterns to identify and label the outliers in the datasets. All the models can accurately identify the outliers, with their average accuracies scoring over 95%. However, the average execution time used for each model varies, where the KNN+MD model has the longest average execution time at 12.99 seconds, MCD achieving 3.98 seconds for execution time, and the LOF model at 0.60 seconds, the shortest among the three.

Enhancing Insider Threat Detection in Imbalanced Cybersecurity Settings Using the Density-Based Local Outlier Factor Algorithm

Enhancing Insider Threat Detection in Imbalanced Cybersecurity Settings Using the Density-Based Local Outlier Factor Algorithm

Anomaly identification of English online learning data based on local outlier factor

Anomaly identification of English online learning data based on local outlier factor

Anomaly detection in IoT environment using machine learning

AbstractThis research paper delves into the security concerns within Internet of Things (IoT) networks, emphasizing the need to safeguard the extensive data generated by interconnected physical devices. The presence of anomalies and faults in the sensors and devices deployed within IoT networks can significantly impact the functionality and outcomes of IoT systems. The primary focus of this study is the identification of anomalies in IoT devices arising sensor tampering, with an emphasis on the application of machine learning techniques. While supervised methods like one‐class SVM, Gaussian Naive Bayes, and XG Boost have proven effective in anomaly detection, there has been a noticeable scarcity of research employing unsupervised methods. This scarcity is mainly attributed to the absence of well‐defined ground truths for model training. This research takes an innovative approach by investigating the utility of unsupervised algorithms, including Isolation Forest and Local Outlier Factor, alongside supervised techniques to enhance the precision of anomaly detection.

Identifying Sensors Data Integrity Threats of Smart Agriculture: A Collaborative Filtering Approach

HighlightsData integrity threat is one of the challenges faced by smart agriculture, where humans make complex decisions based on the results and recommendations of automated farm/livestock monitoring systems.Smart or digital farming heavily relies on data, which can face various integrity threats including data modification, data destruction, etc.The use of collaborative filtering as a lightweight method to detect abnormal data has proven effective from our experimental study. It can detect suspicious data with high accuracy and precision enhancing digital agricultural security.Abstract. Smart agriculture leverages human intelligence and artificial intelligence in digital farming, which heavily relies on data collection, analysis, and decision support delivery. Farmers need reliable data and a decision support system to make accurate decisions and take appropriate actions. The Internet of Things (IoT) not only creates the backbone for farm data collection for intelligent farming but also brings some security concerns to the agriculture field. Though many studies exist on security in agriculture, particularly device identification and tracking, cryptography, blockchain, and other security approaches, the concerns about agricultural data integrity have not been fully explored in the current literature. Data integrity threats could cause huge losses to farmers, threatening food security, especially in developing countries. Given the early stage of development in the domain and the rapid adoption of IoT, ensuring the trustworthiness of data from various devices poses a significant challenge. This challenge is exacerbated in wireless sensor network scenarios, particularly in harsh environments where potential avenues for physical attacks exist. This article evaluates a data integrity threat detection technique in an IoT wireless network. It proposes an approach to identify potential data integrity failures or threats. The effectiveness of this approach is demonstrated through an empirical use-case study focusing on agriculture applications. Through experimental and trace-based simulations, we illustrate that threats can potentially be identified with a 91% accuracy rate and approximately 98% precision and recall. The proposed solution could be deployed in distributed and centralized digital agricultural systems to identify and predict real-time data integrity issues or threats. Keywords: Collaborative filtering, Data integrity threats, Local outlier factor, Smart agriculture.