Outliers In Categorical Data Research Articles

An outlier removal method based on PCA-DBSCAN for blood-SERS data analysis.

Surface-enhanced Raman spectroscopy (SERS) has shown promising potential in cancer screening. In practical applications, Raman spectra are often affected by deviations from the spectrometer, changes in measurement environments, and anomalies in spectrum characteristic peak intensities due to improper sample storage. Previous research has overlooked the presence of outliers in categorical data, leading to significant impacts on model learning outcomes. In this study, we propose a novel method, called Principal Component Analysis and Density Based Spatial Clustering of Applications with Noise (PCA-DBSCAN) to effectively remove outliers. This method employs dimensionality reduction and spectral data clustering to identify and remove outliers. The PCA-DBSCAN method introduces adjustable parameters (Eps and MinPts) to control the clustering effect. The effectiveness of the proposed PCA-DBSCAN method is verified through modeling on outlier-removed datasets. Further refinement of the machine learning model and PCA-DBSCAN parameters resulted in the best cancer screening model, achieving 97.41% macro-average recall and 97.74% macro-average F1-score. This paper introduces a new outlier removal method that significantly improves the performance of the SERS cancer screening model. Moreover, the proposed method serves as inspiration for outlier detection in other fields, such as biomedical research, environmental monitoring, manufacturing, quality control, and hazard prediction.

Homophily outlier detection in non-IID categorical data

Most of existing outlier detection methods assume that the outlier factors (i.e., outlierness scoring measures) of data entities (e.g., feature values and data objects) are Independent and Identically Distributed (IID). This assumption does not hold in real-world applications where the outlierness of different entities is dependent on each other and/or taken from different probability distributions (non-IID). This may lead to the failure of detecting important outliers that are too subtle to be identified without considering the non-IID nature. The issue is even intensified in more challenging contexts, e.g., high-dimensional data with many noisy features. This work introduces a novel outlier detection framework and its two instances to identify outliers in categorical data by capturing non-IID outlier factors. Our approach first defines and incorporates distribution-sensitive outlier factors and their interdependence into a value-value graph-based representation. It then models an outlierness propagation process in the value graph to learn the outlierness of feature values. The learned value outlierness allows for either direct outlier detection or outlying feature selection. The graph representation and mining approach is employed here to well capture the rich non-IID characteristics. Our empirical results on 15 real-world data sets with different levels of data complexities show that (i) the proposed outlier detection methods significantly outperform five state-of-the-art methods at the 95%/99% confidence level, achieving 10–28% AUC improvement on the 10 most complex data sets; and (ii) the proposed feature selection methods significantly outperform three competing methods in enabling subsequent outlier detection of two different existing detectors.

A Novel Ensemble Method for Detecting Outliers in Categorical Data

A Novel Ensemble Method for Detecting Outliers in Categorical Data

Correlation and Probability Based Similarity Measure for Detecting Outliers in Categorical Data

Determining the similarity or distance among data objects is an important part in many research fields such as statistics, data mining, machine learning etc. There are many measures available in the literature to define the distance between two numerical data objects. It is difficult to define such a metric to measure the similarity between two categorical data objects since categorical data objects are not ordered. Only a few distance measures are available in the literature to find the similarities among categorical data objects. This paper presents a comparative evaluation of various similarity measures for categorical data and also introduces a novel similarity measure for categorical data based on occurrence frequency and correlation. We evaluated the performance of these similarity measures in the context of outlier detection task in data mining using real world data sets. Experimental results show that the proposed similarity measure outperform the existing similarity measures to detect outliers in categorical datasets. The performances are evaluated in the context of outlier detection task in data mining.

A hybrid algorithm for mining local outliers in categorical data

A hybrid algorithm for mining local outliers in categorical data

A hybrid algorithm for mining local outliers in categorical data

Outlier detection is an important task in data mining. Many approaches have been developed to detect outliers. However, most researches focus on global outlier detection. In many situations, the local outlier detection is more valuable than the global outlier detection. In this paper, the existing methods for outlier detection are discussed firstly, and then the definition of local outlier and some formulas are given. Also a hybrid algorithm for mining local outlier is proposed which is based on clustering algorithm and standard deviation in statistics. By calculating the standard deviation of a cluster and local outlier factor of an object in the cluster, we can identify that the clusters with higher standard deviation may have outliers, and the objects with higher local outlier factor can be recognised as outliers. Experimental results on real datasets show that the proposed algorithm is correct and effective for mining local outliers.

A Semisupervised Approach to the Detection and Characterization of Outliers in Categorical Data.

In this paper, we introduce a new approach of semisupervised anomaly detection that deals with categorical data. Given a training set of instances (all belonging to the normal class), we analyze the relationship among features for the extraction of a discriminative characterization of the anomalous instances. Our key idea is to build a model that characterizes the features of the normal instances and then use a set of distance-based techniques for the discrimination between the normal and the anomalous instances. We compare our approach with the state-of-the-art methods for semisupervised anomaly detection. We empirically show that a specifically designed technique for the management of the categorical data outperforms the general-purpose approaches. We also show that, in contrast with other approaches that are opaque because their decision cannot be easily understood, our proposed approach produces a discriminative model that can be easily interpreted and used for the exploration of the data.

Detecting outliers in categorical data through rough clustering

Outlier detection is an important data mining task with many contemporary applications. Clustering based methods for outlier detection try to identify the data objects that deviate from the normal data. However, the uncertainty regarding the cluster membership of an outlier object has to be handled appropriately during the clustering process. Additionally, carrying out the clustering process on data described using categorical attributes is challenging, due to the difficulty in defining requisite methods and measures dealing with such data. Addressing these issues, a novel algorithm for clustering categorical data aimed at outlier detection is proposed here by modifying the standard $$k$$k-modes algorithm. The uncertainty regarding the clustering process is addressed by considering a soft computing approach based on rough sets. Accordingly, the modified clustering algorithm incorporates the lower and upper approximation properties of rough sets. The efficacy of the proposed rough $$k$$k-modes clustering algorithm for outlier detection is demonstrated using various benchmark categorical data sets.

A relative patterns discovery for enhancing outlier detection in categorical data

Outlier (also known as anomaly) detection technology is widely applied to many areas, such as diagnosing diseases, evaluating credit, and investigating cybercrime. Recently, several studies, based on frequent itemset mining (FIM), have been proposed to detect outliers in categorical data. For efficiency, these FIM-based studies pruned (ignored) the majority of data by either imposing a threshold or restricting the length of the pattern or both, and they further adopted the limited information to evaluate observations. In spite of high efficiency, such a pruning approach encounters the problem of distortion, i.e., the accuracy decreases to a low level of discernment or even causes the contrary judgment in certain cases. In this paper, we introduce the concept relative patterns discovery from a new perspective on association analysis. To efficiently explore the relative patterns, we devise a hash-index-based intersecting approach (called the HA). Based on the knowledge of relative patterns, we propose an unsupervised approach (called the UA) to evaluate which observations are anomalous. Instead of using the limited information, our method can differentiate the features of observations without the problem of distortion. The results of the empirical investigation, conducted with eight real-world datasets on the UCI Machine Learning Repository, demonstrate that our method generally outperforms the previous studies not only in accuracy but also in efficiency. We also demonstrate that the execution complexity of our method is significantly efficient, especially in high-dimensional data. Furthermore, our method can represent a natural panorama of data, which is appropriate in controlled experiments for discovering more decisive factors in outlier detection.

A ranking-based algorithm for detection of outliers in categorical data

Outlier detection being an important data mining problem has attracted a lot of research interest in the recent past. As a result, various methods for outlier detection have been developed particularly for dealing with numerical data, whereas categorical data needs some attention. Addressing this requirement, we propose a two-phase algorithm for detecting outliers in categorical data based on a novel definition of outliers. In the first phase, this algorithm explores a clustering of the given data, followed by the ranking phase for determining the set of most likely outliers. The proposed algorithm is expected to perform better as it can identify different types of outliers, employing two independent ranking schemes based on the attribute value frequencies and the inherent clustering structure in the given data. Unlike some existing methods, the computational complexity of this algorithm is not affected by the number of outliers to be detected. The efficacy of this algorithm is demonstrated through experiments on various public domain categorical data sets.

A Model for Improving Classifier Accuracy using Outlier Analysis

Anomalies are those records, which have different behavior and do not comply with the remaining records in the dataset. Outlier analysis is the concept to find anomalies in Datasets.  Detecting outliers efficiently is an important issue in many fields of science, medicine and technology. Many methods are available to detect anomalies in numerical datasets but a limited number of methods available for categorical datasets. In this work, a novel method to detect outliers in categorical data based on entropy is proposed. This algorithm finds anomalies based on each record score and has great intuitive appeal. These scores called BAD scores. This algorithm utilizes the frequency of each value in the dataset. Greedy method needs k- scans of dataset to find ‘k’ outliers where as the proposed method needs only one scan of dataset and it calculates BAD score of each record directly. It avoids the problem of giving ‘k’ as an input and can find any number of outliers based on our data set directly.AVF method has less time complexity when compared with the other methods like Greedy, FPOF and FDOD. Greedy has good accuracy when compared with other methods like AVF and FPOF, FDOD (which are based on frequency patterns of all combinations of values in each record). Our algorithm shows better results in accuracy than AVF algorithm and Greedy. But this method has reached nearest to AVF in time complexity. This algorithm has been applied on Nursery dataset and Bank dataset taken from “UCI Machine Learning Repository”. In this work, it is proposed to extend Normal distribution [11], and Fuzzy concept [12] to BAD score [13] that is NAVF combined with Fuzzy AVF is applied to BAD Score.  Numerical attributes are excluded from Datasets for our analysis. The experimental results show that it is efficient for outlier detection in categorical dataset.

Outlier Analysis of Categorical Data using NAVF

Outlier mining is an important task to discover the data records which have an exceptional behavior comparing with other records in the remaining dataset. Outliers do not follow with other data objects in the dataset. There are many effective approaches to detect outliers in numerical data. But for categorical dataset there are limited approaches. We propose an algorithm NAVF (Normally distributed attribute value frequency) to detect outliers in categorical data. This algorithm utilizes the frequent pattern data mining method. It avoids problem of giving k-outliers to get optimal accuracy in any classification models in previous work like Greedy, AVF, FPOF, and FDOD while finding outliers. The algorithm is applied on UCI ML Repository datasets like Nursery, Breast cancer mushroom and bank dataset by excluding numerical attributes. The experimental results show that it is efficient for outlier detection in categorical dataset.Keywords: Outliers, Categorical, AVF, NAVF, FPOF, FDOD(ProQuest: ... denotes formulae omitted.)1 IntroductionOutlier analysis is an important research field in many applications like credit card fraud, intrusion detection in networks, medi-cal field .This analysis concentrate on detect-ing infrequent data records in dataset.Most of the existing systems are concentrated on numerical attributes or ordinal attributes .Sometimes categorical attribute values can be converted into numerical values. This process is not always preferable. In this paper we discuss a simple method for categorical data is presented.AVF method is one of the efficient methods to detect outliers in categorical data. The mechanism in this method is that, it calcu-lates frequency of each value in each data at-tribute and finds their probability, and then it finds the attribute value frequency for each record by averaging probabilities and selects top k- outliers based on the least AVF score. The parameter used in this method is only k, the no. of outliers. FPOF is based on frequent patterns which are adopted from Apriority algorithm [1]. This calculates fre-quent patterns item sets from each object. From these frequencies it calculates FPOF score and finds the least k- outliers as the least FPOF scores. This method takes more time to detect outliers comparing with AVF. The parameters used in it are σ, a threshold value to decide frequent sub sets in each data object. The next method is based on Entropy score. Greedy [2] is another method to detect outliers from categorical data. The previous approaches used to detect outliers were2 Existing ApproachesStatistical basedThis method adopted a parametric model that describes the distribution of the data and the data was mostly unvaried [3, 4]. The main drawbacks of this method are difficulty of finding a correct model for different datasets and their efficiency decreases as the no. of dimensions increases [4]. To rectify this problem the Principle component method can be used. Another method to handle high di-mensional datasets is to convert the data re-cords in layers however; these ideas are not practical for more than or equal to three di-mensions.Distance-BasedDistance based methods do not make any as-sumptions about the distribution of the data records because they must compute the dis-tances between records. But these make a high complexity. So these methods are not useful for large datasets. There are some im-provements exist in the distance-based algo-rithms, such as Knorr's et al. [5], they have explained that apart of dataset records belong to each outlier must be less than some threshold value. Still it is an exponential on the number of nearest neighbours.Density BasedThese methods are based on finding the den-sity of the data and identifying outliers as those lying in regions with low density. Bre-unig et al. have calculated a local outlier fac-tor (LOF) to identify whether an object con-tains sufficient neighbour around it or not[6]. …

Hyper Graph Deviation Comparison & Cluster Relation In Categorical Data

Outlier detection is one of the most important issues in recent years. Outlier detection is the process of detecting errors in data. The recent methods are mostly based on Numerical data, but these methods are not suitable for real time data such as web pages, business transactions etc., which are known as Categorical data. It is difficult to find outliers in categorical data. In this paper, we propose an approach to find outliers those are Comparison of Deviations. In Comparison of deviation method, we use hyper graph to calculate the deviations of each object in the database, and we measure the similarity between attributes in database.