Outlier Detection Techniques Research Articles

A robust distance-based approach for detecting multidimensional outliers

ABSTRACT Identifying outliers in data analysis is a critical task, as outliers can significantly influence the results and conclusions drawn from a dataset. This study explores the use of the Mahalanobis distance metric for detecting outliers in multivariate data, focusing on a novel approach inspired by the work of M. Falk, [On mad and comedians, Ann. Inst. Stat. Math. 49 (1997), pp. 615–644]. The proposed method is rigorously tested through extensive simulation analysis, where it demonstrates high True Positive Rates (TPR) and low False Positive Rates (FPR) when compared to other existing outlier detection techniques. Through extensive simulation analysis, we empirically evaluate the affine equivariance and breakdown properties of our proposed distance measure and it is evident from the outputs that our robust distance measure demonstrates effective results with respect to the measures FPR and TPR. The proposed method was applied to seven different datasets, showing promising true positive rates (TPR) and false positive rates (FPR), and it outperformed several well-known outlier identification approaches. We can effectively use our proposed distance measure in fields demanding outlier detection.

Chemometric approaches for discriminating manufacturers of Korean handmade paper using infrared spectroscopy

The objective of this study was to identify the manufacturer of Hanji, Korean handmade paper widely used in conservation science. To achieve this, machine learning models utilizing attenuated total reflectance–infrared spectroscopy (ATR–IR) were developed to assess the robustness and effectiveness of the computed models. Principal component analysis (PCA), partial least squares–discriminant analysis (PLS–DA), decision tree (DT), and k-NN models were constructed using IR spectral data, with the spectral region between 1800 and 1500 cm⁻1 identified as the critical input variable through Variable Importance in Projection (VIP) scores. The transformation of the obtained spectra into second derivative spectra proved beneficial in this key spectral region, leading to significant improvements in model performance. Additionally, the application of DBSCAN for outlier detection was effective in refining the dataset, further enhancing the performance of the models. Specifically, the k-NN model, when applied to the selected variables and preprocessed with the second derivative transformation, achieved an F1 score of 0.92. These findings underscore the importance of focusing on the 1800–1500 cm⁻1 spectral range and applying outlier detection techniques, such as DBSCAN, to enhance the robustness and accuracy of the Hanji classification models by eliminating the influence of atypical data points.

Z-K-R: A Novel Framework in Intrusion Detection system through enhanced techniques

Abstract Intrusion detection systems (IDS) are an important tool for securing computer networks from various types of cyberattacks. The increasing complexity of network attacks demands more sophisticated approaches to intrusion detection. This paper presents an innovative method for IDS that involves combining Z-Score outlier detection, KMeans clustering, and Random Forest classification techniques. We tested our methodology using the CICIDS2017 dataset, which is a standardization dataset for intrusion detection that is frequently utilized. Our proposed approach first uses Z-Score outlier detection to identify abnormal traffic flows in the network. Next, KMeans clustering is used to group the traffic flows into different clusters based on their similarity. Finally, Random Forest classification is used to classify each traffic flow into normal or abnormal categories. Based on our experimental results, our approach for intrusion detection shows superior performance compared to several other state-of-the-art methods in terms of accuracy and precision. Our proposed method achieved an accuracy rate of 95.75% and a precision of 95.76%, surpassing the performance of KNN, SVM, and decision trees approaches. In conclusion, the proposed Z-K-R approach offers a promising solution for IDS by leveraging the strengths of Z-Score outlier detection, KMeans clustering, and Random Forest classification techniques. This strategy has the potential to increase the efficiency of IDS and boost network security in applications that take place in the real world.

Counterfeit banknote identification based on outliers detection methods

Counterfeit banknotes in circulation in the economy are a long-standing and recurring problem. Technological improvements are applied to produce increasingly convincing counterfeit banknotes. In contrast, sophisticated strategies to identify counterfeit notes remain an up-to-date and fascinating research topic of great interest. The literature presents several techniques for detecting counterfeit banknotes. In this study, outlier detection methodologies are presented as adapted mechanisms to assist in the fight against counterfeit banknotes. This study addresses and tests multivariate outlier detection techniques' efficiency in identifying counterfeit banknotes. A comprehensive computational experiment proves the great adaptability of the methods to multivariate outliers in the identification of counterfeit banknotes. In particular, the Data-driven Cluster Analysis Method (DDCAM) technique presented promising and optimistic results through several quality and performance measurement metrics, suggesting a hopeful future for detecting counterfeit banknotes.

Design of a precise ensemble expert system for crop yield prediction using machine learning analytics

AbstractAgriculture is facing significant challenges in the development of crop yield forecasts, which are important aspects of decision‐making at the international, regional, and local levels. The area of agriculture is attracting growing attention because of increasing the demand for food supplies. To ensure future food supplies, crop yield prediction (CYP) provides the best decision‐making to assist farmers in agricultural yield forecasting efficiently. Nevertheless, CYP is a difficult endeavor because of the intricacy of the underlying mechanisms and the effect of numerous factors, including weather patterns, soil characteristics, and crop management techniques. In today's era, ensemble learning (EL) approaches have recently demonstrated significant promise for enhancing the reliability and accuracy of CYP. The success of the EL techniques depends on several facts, including how the base learner models are trained and how these are combined. This study provides important insights into the EL techniques for CYP. This paper proposes an expert system model named precise ensemble expert system for crop yield prediction (PEESCYP) to predict the best crop for agricultural land. The proposed PEESCYP model employs multiple imputation by chained equation (MICE) data imputation technique to treat the missing values of the collected dataset, the isolation forest (IF) technique for outlier detection, the ant colony optimization (ACO) technique to perform feature selection, robust scaling (RS) technique to perform data normalization, and the extra tree (ET) is used for classification to overcome the variance and overfitting problem of the single classifiers. The measurements of the proposed PEESCYP model have been collected by means of accuracy, precision, recall, and F‐1 score using a prepared dataset, which is collected from International Crops Research Institute for the Semi‐Arid Tropics (ICRISAT), and the proposed model is compared with different single‐classifier based ML models, EL models, and various existing models available in the literature. The results of this experiment underline that the proposed PEESCYP model outperforms the others.

Outlier detection and selection of representative fluid samples using machine learning: a case study of Iranian oil fields

During the development of a field, many fluid samples are taken from wells. Selecting a robust fluid sample as the reservoir representative helps to have a better field characterization, reliable reservoir simulation, valid production forecast, efficient well placement and finally achieving optimized ultimate recovery. First, this paper aims to detect and separate the samples that have been collected under poor conditions or analyzed in a non-standard way. Moreover, it introduces a novel ranking method to score the samples based on the amount of coordination with other fluid samples in the region. The dataset includes 136 fluid samples from five reservoirs in Iranian fields, each of them consisting of 21 key parameters. Five acknowledged machine learning based anomaly detection techniques are implemented to compare fluid samples and detect those whose results deviate from others, indicating non-standard samples. To ensure the proper detection of outlier data, the results are compared with the traditional validation method of gas-oil ratio estimation. All five outlier detection methods demonstrate acceptable performance with average accuracy of 79% compared to traditional validation. Furthermore, the fluid samples with the highest scores in scoring-based algorithms are introduced as the best reservoir’s representative fluid. Finally, fuzzy logic is used to obtain a final score for each sample, taking the results of the six methods as input and ranking the samples based on their output score. The study confirms the robustness of the novel approach for fluid validation using outlier detection techniques and the value of machine learning and fuzzy logic for sample ranking, excelling in considering all critical fluid parameters simultaneously over traditional methods.

Corporate financial distress prediction using the risk-related information content of annual reports

This study presents a financial distress prediction model focusing on the linguistic analysis of risk-related sections of corporate annual reports. Here, we introduce a novel methodology that leverages BERT-based contextualized embedding models for nuanced extraction of financial sentiment and topic coherence. This stands in contrast to existing research, which predominantly relies on dictionary-based or non-contextual word embeddings and addresses their limitations in context sensitivity. Furthermore, we apply an innovative financial distress prediction model that combines the robust XGBoost algorithm with unsupervised outlier detection techniques. This hybrid model is specifically designed to tackle the issue of class imbalance, a persistent challenge in financial distress prediction. The efficacy of the proposed model is empirically validated using a comprehensive dataset of 2545 companies listed on major global stock exchanges. Our findings indicate that the introduced model not only significantly outperforms most existing state-of-the-art financial distress prediction models in terms of predictive accuracy, but also significantly outperforms the Loughran & McDonald dictionary-based approach and the Word2Vec model, underlining its potential as a superior analytical tool for financial distress prediction.

Outlier detection for keystroke biometric user authentication.

User authentication is a fundamental aspect of information security, requiring robust measures against identity fraud and data breaches. In the domain of keystroke dynamics research, a significant challenge lies in the reliance on imposter datasets, particularly evident in real-world scenarios where obtaining authentic imposter data is exceedingly difficult. This article presents a novel approach to keystroke dynamics-based authentication, utilizing unsupervised outlier detection techniques, notably exemplified by the histogram-based outlier score (HBOS), eliminating the necessity for imposter samples. A comprehensive evaluation, comparing HBOS with 15 alternative outlier detection methods, highlights its superior performance. This departure from traditional dependence on imposter datasets signifies a substantial advancement in keystroke dynamics research. Key innovations include the introduction of an alternative outlier detection paradigm with HBOS, increased practical applicability by reducing reliance on extensive imposter data, resolution of real-world challenges in simulating fraudulent keystrokes, and addressing critical gaps in existing authentication methodologies. Rigorous testing on Carnegie Mellon University's (CMU) keystroke biometrics dataset validates the effectiveness of the proposed approach, yielding an impressive equal error rate (EER) of 5.97%, a notable area under the ROC curve of 97.79%, and a robust accuracy (ACC) of 89.23%. This article represents a significant advancement in keystroke dynamics-based authentication, offering a reliable and efficient solution characterized by substantial improvements in accuracy and practical applicability.

Enhanced Thermal Comfort Prediction Model by Addressing Outliers and Data Imbalance

The relationship between individuals and their thermal environment is pivotal not only for comfort but also for health and productivity. Thermal comfort, as defined by ASHRAE, reflects an individual's satisfaction with their ambient thermal conditions and can be gauged using the ASHRAE scale. In the past, traditional thermal comfort prediction models such as the Predicted Mean Vote (PMV) were used to evaluate thermal comfort. Nevertheless, the emergence of machine learning provides a more dynamic approach to predict thermal comfort of occupants. However, the subjective nature of thermal comfort introduces data ambiguities challenge which lead to the existence of outliers. Moreover, data imbalances within the dataset can cause the machine learning models to not learn the minority class effectively, resulting in the deterioration of the model. This research has developed an enhanced thermal comfort prediction model to predict the occupant’s thermal comfort by leveraging the outlier detection technique and synthetic data generator, particularly the Isolation Forest and SMOTE. The experiment showed that the proposed model is able to achieve an accuracy of 74.94%. This exhibited a slight improvement compared to the findings in prior research of using Random Forest prediction model.

Correlation-based outlier detection for ships’ in-service datasets

With the advent of big data, it has become increasingly difficult to obtain high-quality data. Solutions are required to remove undesired outlier samples from massively large datasets. Ship operators rely on high-frequency in-service datasets recorded onboard the ships for monitoring the performance of their fleet. The large in-service datasets are known to be highly unbalanced, making it difficult to adopt ordinary outlier detection techniques, as they would also result in the removal of rare but quite valuable data samples. Thus, the current work proposes to establish a correlation-based outlier detection scheme for ships’ in-service datasets using two well-known dimensionality reduction methods, namely, Principal Component Analysis (PCA) and Autoencoders. The correlation-based approach detects samples which do not fit the prominent correlations present in the dataset and avoids misidentifying the rare but correlation-following samples in the sparse regions of data domain. The study also attempts to provide the physical meaning of the latent variables obtained using PCA. The effectiveness of the proposed methodology is proven using an actual dataset recorded onboard a ship.

Comparison between Statistical Approaches and Data Mining Algorithms for Outlier Detection

Outliers are observation values that are very different from most observations. The presence of outliers in data can have a negative impact on research but can contain important information for other research. So, identifying outliers before conducting data analysis is a crucial thing to do. Outlier detection methods/techniques were first pioneered by researchers in statistics. However, due to rapid technological advances which have an impact on the ease of collecting extensive data, the development of outlier detection techniques is now handled mainly by researchers in the field of computer science (data mining) using computing facilities. This research aims to examine the results of simulation studies by comparing methods for identifying several outliers using statistical approaches and data mining algorithm approaches in various predetermined data scenarios. Based on the scenario carried out, the outlier detection method using a statistical approach is generally better than the outlier detection method using a data mining-based approach. Suggestions for further research are to improve the data mining method by focusing more on statistical analysis apart from focusing on data processing computing time so that the expected results of outlier detection are faster and more precise.

Time-aware outlier detection in health physique monitoring in edge-aided sport education decision-makings

The increasing popularity of various intelligent sensor and mobile communication technologies has enabled quick health physique sensing, monitoring, collection and analyses of students, which significantly promoted the development of sport education. Through collecting the students’ physiological signals and transmitted them to edge servers, we can precisely analyze and judge whether a student is in poor health (e.g., an outlier). However, with time elapsing, the accumulated physiological signals of students become massive, which places a heavy burden on the quick storage and in-time processing of physiological data of students. In this situation, it is becoming a necessity to develop a time-aware outlier detection technique for health physique evaluation of students in a time-efficient way. Considering this challenge, we propose a novel time-aware outlier detection method named TOD based on Locality-Sensitive Hashing. TOD condenses extensive physiological student data into a concise set of health indices. Leveraging these indices, we can efficiently identify potential student outliers from a large pool of candidates with precision and speed. Finally, we have designed a group of simulated experiments based on WS-DREAM dataset. Experiment results prove the feasibility and superiority of the TOD method compared with other existing methods.

DeepFore: A Deep Reinforcement Learning Approach for Power Forecasting in Renewable Energy Systems

An open network known as the “energy internet” links every component of the whole energy supply chains, from the generations. Due to their ability to mimic regional flow dynamics that have an impact on wind farm production, regional meteorological models are increasingly being used as a general tool for wind resource forecasting. In this study, higher vertical and horizontal resolutions WRF (weather research and forecasting) paradigm simulation are used to anticipate and validate production for a genuine onshore wind farm. This paper proposed a DeepFore which is a power forecasting system for hybrid renewable energy systems. Initially, the dataset is generated by the hybrid system. This data is preprocessed to improve the quality of the data by incorporating, filtering and outlier detection techniques. Then, this enriched data is fed into K++ means clustering algorithm to separate the normal data from faulty data. With the normal and original data, Teaching-Learning based optimization algorithm attempts to realize the optimal features which are important for forecasting. Finally, Deep SARSA which is deep reinforcement learning algorithm is incorporated to determine the power generated by the hybrid system. Better winds energy prediction estimations enable more efficient utilization of the produced electricity, according to computational models.

Multi-Step Internet Traffic Forecasting Models with Variable Forecast Horizons for Proactive Network Management.

The ISP (Internet Service Provider) industry relies heavily on internet traffic forecasting (ITF) for long-term business strategy planning and proactive network management. Effective ITF frameworks are necessary to manage these networks and prevent network congestion and over-provisioning. This study introduces an ITF model designed for proactive network management. It innovatively combines outlier detection and mitigation techniques with advanced gradient descent and boosting algorithms, including Gradient Boosting Regressor (GBR), Extreme Gradient Boosting (XGB), Light Gradient Boosting Machine (LGB), CatBoost Regressor (CBR), and Stochastic Gradient Descent (SGD). In contrast to traditional methods that rely on synthetic datasets, our model addresses the problems caused by real aberrant ISP traffic data. We evaluated our model across varying forecast horizons-six, nine, and twelve steps-demonstrating its adaptability and superior predictive accuracy compared to traditional forecasting models. The integration of the outlier detection and mitigation module significantly enhances the model's performance, ensuring robust and accurate predictions even in the presence of data volatility and anomalies. To guarantee that our suggested model works in real-world situations, our research is based on an extensive experimental setup that uses real internet traffic monitoring from high-speed ISP networks.

An optimized outlier detection function for multibeam echo-sounder data

Multibeam sonar data are a valuable tool for seafloor mapping and geological studies. However, the presence of outliers in multibeam data can distort the results of analyses and reduce the accuracy of seafloor maps. In this paper, we define a weighting function based on the performance of various outlier detection techniques (ODTs) for detecting outliers in multibeam data, which calculates an outlier probability score for each sounding. Our results show that each ODT has its own strengths and weaknesses, and that a combination of outlier detection techniques is promising to improve reproducibility, explainability and the accuracy of the detection process. To address the challenge of detecting outliers in multibeam data, we propose a weighted outlier detection function that outperforms individual outlier detection techniques in terms of precision, recall and F1 scores by considering their strengths and combining them in a way that accounts for variations in the data. The function detects various types of outliers with high precision and recall values, resulting in valuable improvements in outlier detection performance for multibeam data. Overall, our proposed workflow has the potential to significantly improve the way multibeam data cleaning is performed, with the weighted outlier detection function being applied first, detecting most of the outlier automatically, followed by a domain-expert review of a small group of soundings whose automatic outlier labeling is not unequivocal.

Advancing Fault Detection in Building Automation Systems through Deep Learning

This study proposes a deep learning model utilizing the BACnet (Building Automation and Control Network) protocol for the real-time detection of mechanical faults and security vulnerabilities in building automation systems. Integrating various machine learning algorithms and outlier detection techniques, this model is capable of monitoring and learning anomaly patterns in real-time. The primary aim of this paper is to enhance the reliability and efficiency of buildings and industrial facilities, offering solutions applicable across diverse industries such as manufacturing, energy management, and smart grids. Our findings reveal that the developed algorithm detects mechanical faults and security vulnerabilities with an accuracy of 96%, indicating its potential to significantly improve the safety and efficiency of building automation systems. However, the full validation of the algorithm’s performance in various conditions and environments remains a challenge, and future research will explore methodologies to address these issues and further enhance performance. This research is expected to play a vital role in numerous fields, including productivity improvement, data security, and the prevention of human casualties.

Machine learning based technique for outlier detection and result prediction in combustion diagnostics

With the utilization of data correlation, this work proposed a novel machine learning-based technique for outlier detection and result prediction to cover the deficiency and economize the cost of current combustion diagnostic technology. For outlier detection, the measurement results were performed through cluster analysis. Results demonstrated the measurement outlier could be successfully detected and eliminated by DBSCAN and the proposed GBCN algorithm. For result prediction, an artificial neural network (ANN) was established based on the correlation of the measured data, and its performance was investigated by the statistical method. Results indicated the ANN could well “learn” the distribution characteristic of flame temperature with a high value of R and a low value of MAPE by selecting Tanh and Sigmoid as activation functions. When there were 30 % and 50 % continuous unmeasured temperature data inside the flame, the ANN could still predict their values approximately based on the correlation between the remaining data. With the increase of prediction amount to 60 %, the prediction performance decreased significantly. However, even if there were 70 % random unmeasured temperature data inside the flame, the whole temperature field could still be obtained by the prediction of ANN with high accuracy based on the correlation between a small amount of measured data.

Contrast threshold recovery after bleaching during cone‐mediated mesopic adaptation: The effect of test conditions

Aims/Purpose: To determine the optimal luminance conditions and the staircase procedures for evaluating the time course of macular contrast sensitivity recovery after photobleaching in order to decide the best protocol to assess cone‐mediated mesopic adaptation.Methods: The dynamics of contrast threshold (CT) recovery following photopigment bleach were measured psychophysically for five minutes in twenty‐four healthy adults (23.7 ± 3.7 years old). The stimuli used were sine‐wave gratings of low‐spatial frequency (1 cycle‐per‐degree). In each subject, six tests were performed, combining three luminance levels (0.1, 1.0, and 10 cd/m2) and two modified staircase procedures (3‐down/1‐up and 2‐down/1‐up) to estimate CTs. The test order was randomized, conducted in two visits, with a 15‐min washout period between tests. Outlier detection techniques were applied to verify the integrity of the sample. Non‐linear exponential‐based optimization was adopted to fit CT recovery functions to estimate the time constant (τ, seconds) of cone sensitivity recovery and the final CT (CTf, log units). Classical and heteroscedastic ANOVA analyses were performed to assess the effect of test conditions.Results: The 2‐down/1‐up staircase allowed for denser point sampling of CTs measurements than the 3‐down/1‐up. Under the lowest luminance test condition, the CT recovery was modelled by bumpier functions, for both staircases procedures, than the other luminance levels. Mean cone τ increased (21.7, 42.5, and 72.0 seconds (p < 0.001)) and CTf worsened (−1.99, −1.85, and − 1.47 log units (p < 0.001)) as luminance level decreased. The best fit (median R2 = 0.87) was achieved under 1 cd/m2 mesopic luminance conditions. There were no statistically significant differences in mean cone τ and CTf between the two staircases procedures.Conclusions: An intermediate mesopic light level and a 2‐down/1‐up staircase provided the best protocol for assessing cone‐mediated mesopic adaptation.

Modified Keypoint-Based Copy Move Area Detection

Detecting and identifying manipulated portions within images poses a formidable challenge in research. Manipulated images, often created using image editing software such as Picasa or Photoshop, serve to obscure information and intentionally mislead viewers. Consequently, ensuring the authenticity of images becomes imperative prior to extracting meaningful data. One prevalent form of tampering is copy-move forgery, where objects are intentionally duplicated within an image using region of the same image. This study introduces a method for detecting copy-move forgery areas based on locating Scale-Invariant Feature Transform (SIFT) keypoints in images. The SIFT technique is employed for feature extraction, while feature descriptors are matched using brute force matching. Subsequently, a clustering algorithm is applied to group spatially proximate keypoints, enabling the detection of cloned regions. Specifically, Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is utilized to identify forged areas within the image. To mitigate erroneous forgery detection and reduce false positives, outlier detection techniques are employed. Experimental evaluations are conducted on the MICC-F220 and MICC-F600 datasets, with comparisons drawn against previously established methodologies.

One-Class SVM Probabilistic Outputs.

One-class support vector machine (SVM) is an extension of SVM to handle unlabeled data. As a mature technique for outlier detection, one-class SVM has been widely used in many applications. However, similar to standard two-class SVM, the design of one-class SVM does not give probabilistic outputs. For two-class SVM, some methods have been proposed to effectively obtain probabilistic outputs, but due to the difficulty of no-label information, less attention has been paid to one-class SVM. Our aim in this work is to propose some practically viable techniques to generate probabilistic outputs for one-class SVM. We investigate existing methods for two-class SVM and explain why they may not be suitable for one-class SVM. Due to the lack of label information, we think a feasible setting is to have probabilities mimic to the decision values of training data. Based on this principle, we propose several new methods. Detailed experiments on both artificial and real-world data demonstrate the effectiveness of the proposed methods.