Distance-based Outlier Detection Method Research Articles

Human Activity Recognition with Smartwatch Data by using Mahalanobis Distance-Based Outlier Detection and Ensemble Learning Methods

Recognition of human activities is part of smart healthcare applications. In this context, the detection of human actions with high accuracy has been a field that has been working for many years. With the increase in the usage of smart devices, smartphones and smartwatches have become the constant equipment of these studies thanks to their internal sensors. Sometimes abnormal data are included in data sets due to the way the data were collected and for reasons arising from the sensors. For this reason, it becomes important to detect outlier data. In this study, step counter and heart rate sensors were used in addition to an accelerometer and gyroscope in order to detect human activities. Afterward, the outliers were detected and cleared with a Mahalanobis distance-based approach. With the aim of achieving a better classification performance, machine learning methods were used by strengthening them with ensemble learning methods. The obtained results showed that step counter, heart rate sensors, and ensemble learning methods positively affect the success of the classification. In addition, it was found that the Mahalanobis distance-based outlier detection method increased the classification accuracy significantly.

A Probabilistic Transformation of Distance-Based Outliers

The scores of distance-based outlier detection methods are difficult to interpret, and it is challenging to determine a suitable cut-off threshold between normal and outlier data points without additional context. We describe a generic transformation of distance-based outlier scores into interpretable, probabilistic estimates. The transformation is ranking-stable and increases the contrast between normal and outlier data points. Determining distance relationships between data points is necessary to identify the nearest-neighbor relationships in the data, yet most of the computed distances are typically discarded. We show that the distances to other data points can be used to model distance probability distributions and, subsequently, use the distributions to turn distance-based outlier scores into outlier probabilities. Over a variety of tabular and image benchmark datasets, we show that the probabilistic transformation does not impact outlier ranking (ROC AUC) or detection performance (AP, F1), and increases the contrast between normal and outlier score distributions (statistical distance). The experimental findings indicate that it is possible to transform distance-based outlier scores into interpretable probabilities with increased contrast between normal and outlier samples. Our work generalizes to a wide range of distance-based outlier detection methods, and, because existing distance computations are used, it adds no significant computational overhead.

Rapid Detection of Soil Texture Attribute based on Mid-Infrared Spectral Library In Salt Affected Soils of Hungary

Quantifying among others the soil's physical properties is essential for the assessment of the diverse soil environmental functions including water balance of soils and pore structure, water erosion and various soil hydraulic properties. The mid-infrared (MIR) spectroscopy is a useful technique to predict soil attributes with high accuracy, efficiency and low cost. In this study, we examined the ability of our MIR soil spectral library in predicting the clay, silt, sand content of salt affected Hungarian soils. This research is part of a project to establish a MIR spectral library in the frame of the Hungarian Soil Information and Mentoring System (SIMS) survey. Salt affected soils type data was extracted from the spectral library then transformation of spectral reflectance values to absorbance values were performed. Moving average filtering method was applied to absorbance spectra before performing principal components analysis. To determine outlier samples and to select the proper samples for model calibration, Mahalanobis distance-based outlier detection method and Kennard-Stone Sampling selection method were applied on the principal component scores. Spectral and reference soil data were combined and split into training and testing datasets. MIR prediction models were built for sand, clay, and silt content using Partial Least Square Regression (PLSR) method. Coefficient determination, root mean square error and ratio performance to deviation were used to assess the models performance. The prediction accuracies of calibration sets for soil physical texture were excellent while the validation results were slightly lower but still with a good level of prediction.

Little data is often enough for distance-based outlier detection

Many real-world use cases benefit from fast training and prediction times, and much research went into speeding up distance-based outlier detection methods to millions of data points. Contrary to popular belief, our findings suggest that little data is often enough for distance-based outlier detection models. We show that using only a tiny fraction of the data to train distance-based outlier detection models often leads to no significant reduction in predictive performance and detection variance over a wide range of tabular datasets. Furthermore, we compare a data reduction based on random subsampling and clustering-based prototypes and show that both approaches yield similar outlier detection results. Simple random subsampling, thus, proves to be a useful benchmark and baseline for future research on speeding up distance-based outlier detection.

Efficient Detection Using Soft Computing Approach of Modified Fuzzy C-Means Based Outlier Detection in Electronics Patient Records Systems

Outliers are the data objects or data elements which are deviated from the observation or population in the dataset. Normally outliers are considered as noise data but in recent days outliers are taken vital role in the applications like healthcare systems to energetically check for any malignant, irregular, or abnormal behaviour. Accuracy of the outlier detection is purely based upon the efficiency of outlier detection methods and application where the outlier detection is involved. In this paper we design hybrid three phase modified fuzzy c-means and diverse distance based outlier detection method for distributed dataset to detect the unusual usage of data and illegitimate approach in large-scale integral networks, especially in healthcare sectors (EPR systems). The proposed algorithm combining the features of modified C-Means Fuzzification, Z-score and Manhattan distance in outlier identification. The proposed algorithm provides efficiency in outlier detection on univariate EPRdata. This paper also mainly focuses the application intrusion detection in healthcare data. The algorithm is tested on real world dataset of machine learning database repository (UCIML).

An efficient outlier detection approach on weighted data stream based on minimal rare pattern mining

The distance-based outlier detection method detects the implied outliers by calculating the distance of the points in the dataset, but the computational complexity is particularly high when processing multidimensional datasets. In addition, the traditional outlier detection method does not consider the frequency of subsets occurrence, thus, the detected outliers do not fit the definition of outliers (i.e., rarely appearing). The pattern mining-based outlier detection approaches have solved this problem, but the importance of each pattern is not taken into account in outlier detection process, so the detected outliers cannot truly reflect some actual situation. Aimed at these problems, a two-phase minimal weighted rare pattern mining-based outlier detection approach, called MWRPM-Outlier, is proposed to effectively detect outliers on the weight data stream. In particular, a method called MWRPM is proposed in the pattern mining phase to fast mine the minimal weighted rare patterns, and then two deviation factors are defined in outlier detection phase to measure the abnormal degree of each transaction on the weight data stream. Experimental results show that the proposed MWRPM-Outlier approach has excellent performance in outlier detection and MWRPM approach outperforms in weighted rare pattern mining.

A New Mining Method to Detect Real Time Substance Use Events from Wearable Biosensor Data Stream.

Detecting real time substance use is a critical step for optimizing behavioral interventions to prevent drug abuse. Traditional methods based on self-reporting or urine screening are inefficient or intrusive for drug use detection, and inappropriate for timely interventions. For example, self-report suffers from distortion or recall bias; while urine screening often detects drug use that occurred only within the previous 72 hours. Methods for real-time substance use detection are severely underdeveloped, partly due to the novelty of wearable biosensor technique and the lack of substantive clinical data for evaluation. We propose a new real-time drug use event detection method using data obtained from wearable biosensors. Specifically, this method is built upon the slide window technique to process the data stream, and a distance-based outlier detection method to identify substance use events. This novel method is designed to examine how to detect and set up the thresholds of parameters in real-time drug use event detection for wearable biosensor data streams. Our numerical analyses empirically identified the thresholds of parameters used to detect the cocaine use and showed that this proposed method could be adapted to detect other substance use events.

Developing a Modified Logistic Regression Model for Diabetes Mellitus and Identifying the Important Factors of Type II Dm

Background/Objectives: Different methods can be applied to create predictive models for the clinical data with binary outcome variable. This research aims to explore the process of constructing the modified predictive model of Logistic Regression (LR). Method/Statistical Analysis: To improve the accuracy of prediction, the Distance based Outlier Detection (DBOD)is used for pre-processing and Bipolar Sigmoid Function calculated using Neuro based Weight Activation Function is used in Logistic Regression instead of Sigmoid Function. Datasets were collected from clinical laboratory of AR Hospital in Madurai for the three years 2012, 2013 and 2014 are used for analysis. Data pre-processing is done to avoid the existence of insignificant data in the dataset. The detected outliers, using DBOD method are treated using a method closest to the normal range. A comparative study among different distance measures likes Euclidean and Manhattan etc. are done for DBOD method. The pre-processed data finally is fed as input to the Logistic Regression model. Maximum likelihood estimation is used to fit the model. Logistic Model is built from the Sigmoid Function using the Regression Coefficients. The accuracy of the model is evaluated by 10 fold cross validation. Findings: Logistic Model is built from the Sigmoid Function using the Regression Coefficients, produces the accuracy of 79%. The Sigmoid Function calculated using Random Weight Function provides the prediction accuracy of 84.2% and the Bipolar Sigmoid Function calculated using Neuro based Weight Activation function provides the prediction accuracy of 90.4%. On comparison, Bipolar Sigmoid Function calculated using Neuro weight activation function outperforms well than the Sigmoid Function calculated using regression coefficients. Improvements/Applications: The accuracy of Logistic Regression is improved from 79% to 90.4%. The most important factors: Erythrocyte Sedimentation Rate (ESR) and Estimation of Mean blood Glucose are identified from positive subjects of Diabetes Mellitus. The analysis is done for the 31 Diabetes Disease attributes of three years dataset. Keywords: Bipolar Sigmoid Neuro-Weight Activation Function, Distance based Outlier Detection Method, Logistic Regression, Random Weight Function, Sigmoid Activation Function, Type 2 Diabetes Risk Factors

Coding SNPs as Intrinsic Markers for Sample Tracking in Large-Scale Transcriptome Studies

Large-scale transcriptome profiling in clinical studies often involves assaying multiple samples of a patient to monitor disease progression, treatment effect, and host response in multiple tissues. Such profiling is prone to human error, which often results in mislabeled samples. Here, we present a method to detect mislabeled sample outliers using coding single nucleotide polymorphisms (cSNPs) specifically designed on the microarray and demonstrate that the mislabeled samples can be efficiently identified by either simple clustering of allele-specific expression scores or Mahalanobis distance-based outlier detection method. Based on our results, we recommend the incorporation of cSNPs into future transcriptome array designs as intrinsic markers for sample tracking.

Distance-based detection and prediction of outliers

A distance-based outlier detection method that finds the top outliers in an unlabeled data set and provides a subset of it, called outlier detection solving set, that can be used to predict the outlierness of new unseen objects, is proposed. The solving set includes a sufficient number of points that permits the detection of the top outliers by considering only a subset of all the pairwise distances from the data set. The properties of the solving set are investigated, and algorithms for computing it, with subquadratic time requirements, are proposed. Experiments on synthetic and real data sets to evaluate the effectiveness of the approach are presented. A scaling analysis of the solving set size is performed, and the false positive rate, that is, the fraction of new objects misclassified as outliers using the solving set instead of the overall data set, is shown to be negligible. Finally, to investigate the accuracy in separating outliers from inliers, ROC analysis of the method is accomplished. Results obtained show that using the solving set instead of the data set guarantees a comparable quality of the prediction, but at a lower computational cost.