Outlier Probability Research Articles

Application of the Variable Precision Rough Sets Model to Estimate the Outlier Probability of Each Element

In a data mining process, outlier detection aims to use the high marginality of these elements to identify them by measuring their degree of deviation from representative patterns, thereby yielding relevant knowledge. Whereas rough sets (RS) theory has been applied to the field of knowledge discovery in databases (KDD) since its formulation in the 1980s; in recent years, outlier detection has been increasingly regarded as a KDD process with its own usefulness. The application of RS theory as a basis to characterise and detect outliers is a novel approach with great theoretical relevance and practical applicability. However, algorithms whose spatial and temporal complexity allows their application to realistic scenarios involving vast amounts of data and requiring very fast responses are difficult to develop. This study presents a theoretical framework based on a generalisation of RS theory, termed the variable precision rough sets model (VPRS), which allows the establishment of a stochastic approach to solving the problem of assessing whether a given element is an outlier within a specific universe of data. An algorithm derived from quasi-linearisation is developed based on this theoretical framework, thus enabling its application to large volumes of data. The experiments conducted demonstrate the feasibility of the proposed algorithm, whose usefulness is contextualised by comparison to different algorithms analysed in the literature.

Testing the Presence of Outliers to Assess Misspecification in Regression Models

Algorithms used to detect outliers in regression models have a positive probability of finding outliers even when the data generation process has no outliers. We propose two sets of tests for the overall presence of outliers based on the false-discovery rate of outliers. First, `simple' tests on whether the proportion (or number) of detected outliers deviates from its expected value. Second, `scaling' tests on whether the proportion (or number) of detected outliers decreases proportionally with the level of the cut-off used to detect outliers. The proposed tests can be uniformly applied to regressions regardless of whether the regressors are stationary, deterministically trending, unit root, or explosive processes. We show the versatility of the tests in a classic cross-sectional model of economic growth as well as a panel difference-in-differences model of CO2 emissions in response to the introduction of North America's first major carbon tax. Our tests show the presence of significant outliers in emissions in the un-taxed control group which results in an over-estimation of the emissions reductions in response to the carbon tax.

Large-deviation probabilities for correlated Gaussian processes and intermittent dynamical systems.

In its classical version, the theory of large deviations makes quantitative statements about the probability of outliers when estimating time averages, if time series data are identically independently distributed. We study large-deviation probabilities (LDPs) for time averages in short- and long-range correlated Gaussian processes and show that long-range correlations lead to subexponential decay of LDPs. A particular deterministic intermittent map can, depending on a control parameter, also generate long-range correlated time series. We illustrate numerically, in agreement with the mathematical literature, that this type of intermittency leads to a power law decay of LDPs. The power law decay holds irrespective of whether the correlation time is finite or infinite, and hence irrespective of whether the central limit theorem applies or not.

Robust AOA based acoustic source localization method with unreliable measurements

Abstract The performance degradation problem of the angle of arrival (AOA)-based acoustic localization methods in the presence of unreliable bearing measurements (outliers) is addressed in this paper. Two typical M-estimators based on Tukey and Huber functions are applied to tackle the problem. Both functions are solved by the iterative reweighted nonlinear least squares (IRNLS) method. Considering the Huber function is convex in nature, it is specifically utilized to mitigate the influence of large residuals on pseudolinear estimator (PLE) by convex optimization. To make the IRNLS method more feasible to use, an approximate relationship between the outlier probability and the bound parameter is provided. The robustness and effectiveness of the proposed methods are clearly demonstrated through a series of simulation results in the presence of various unreliable measurements.

Improving Lithium-Ion Battery Pack Diagnostics by Optimizing the Internal Allocation of Demand Current for Parameter Identifiability

This article optimizes the allocation of external current demand among parallel strings of cells in a lithium-ion battery pack to improve Fisher identifiability for these strings. The article is motivated by the fact that better battery parameter identifiability can enable the more accurate detection of unhealthy outlier cells. This is critical for pack diagnostics. The literature shows that it is possible to optimize the cycling of a single battery cell for identifiability, thereby improving the speed and accuracy with which its health-related parameters can be estimated. However, the applicability of this idea to online pack management is limited by the fact that overall pack current is typically dictated by the user, and difficult to optimize. We circumvent this challenge by optimizing the internal allocation of total pack current for identifiability. We perform this optimization for two pack designs: one that exploits switching control to allocate current passively among parallel strings of cells, and one that incorporates bidirectional DC–DC conversion for active charge shuttling among the strings. A novel evolutionary algorithm optimizes identifiability for each pack design, and a local outlier probability (LoOP) algorithm is then used for diagnostics. Simulation studies show significant improvements in diagnostic accuracy for an automotive protocol.

Terrestrial Mollusks as Chronological Records in Brazilian Shellmounds

AbstractAmong other zooarchaeological remains, terrestrial snails’ shells from theThaumastusandMegalobulimusgenera are found in some Brazilian shellmounds, presenting a potential substitute for charcoal in radiocarbon dating analyses, as reliable representatives of the atmospheric carbon isotopic ratio. In this paper, we present statistically similar results of both charcoal and land snails samples from the same archaeological contexts in three settlements on the coast of Rio de Janeiro. The Manitiba I shellmound results range from 4.2 to 3.7 ka cal BP (95.4%), contemporary with the Saquarema shellmound, occupied during the period from 4.3 to 3.6 ka cal BP (95.4%). For the Usiminas shellmound, two groups of samples revealed different periods of time for two occupational layers from 2.3 to 2.1 ka cal BP (95.4%) and from 1.6 and 1.3 ka cal BP (95.4%). A model constraining each group of samples to within a single phase has a general agreement of 97% with only two outliers out of 22 dates, yielding minimum individual agreement of 74% and 7% posterior outlier probability for Saquarema shells. These are good examples of sites in which the occupation chronology can be successfully obtained by the radiocarbon dating of land snails.

External analysis and moving window LoOP based monitoring for a class of multi-mode timevarying process

Moving window local outlier probability U+0028 MWLoOP U+0029 is an outlier detecting method which was proposed for monitoring time-varying industrial processes; however, for the practical industrial processes, besides the timevarying characters caused by deactivation of catalyst, measuring instrument drifting and so on, the operation mode is often switched as the adjusting of the feedstock, changes in market demands and so on. Although the MWLoOP algorithm can deal with the time-varying process data, the multi-mode process data will lead to a mass of false alarm. To solve this problem, an external analysis moving window local outlier probability U+0028 EA-MWLoOP U+0029 algorithm is proposed in this study. The external analysis is employed to eliminate the influence of operation mode change on the process data, then the MWLoOP method can deal with complex distribution time-varying data, and give an outlier probability. Finally, the corresponding statistic and control limit are constructed to detect the process fault. In addition, while the monitoring model is being updated, the control limit is not necessary to update. The performance of this method is evaluated through a case study of a non-isothermal continuous stirred tank reactor U+0028 CSTR U+0029.

An incremental and approximate local outlier probability algorithm for intrusion detection and its evaluation

ABSTRACTThis paper proposes a novel incremental modification to the Local Outlier Probabilities algorithm, which is commonly used for anomaly detection, to allow it to detect outliers nearly instantly in data streams. The proposed incremental algorithm’s strength is based on denying the insertion of incremental points into the data set. This precludes the anomaly scores of other points having to update (saving valuable computational time) while resulting in a small amount of error, as compared to an exact approach. This work aims to allow low-resource machines, such as small or older satellites, to perform incremental anomaly detection on large static data sets quickly, trading accuracy impairment for speed of detection.

Outlier detection and robust regression for correlated data

Outlier detection has attracted considerable interest in various areas. Existing outlier detection methods usually assume independence of the modeling errors among the data points but this assumption does not hold in a number of applications. In this paper we propose a probabilistic method for outlier detection and robust updating of linear regression problems involving correlated data. First, suspicious data points will be identified using the minimum volume ellipsoid method and the maximum trimmed likelihood method. Then, the outlierness of each suspicious data point will be determined according to the proposed outlier probability in consideration of possible correlation among the data points. The proposed method is assessed and validated through simulated and real data.

FAST PROBABILISTIC FUSION OF 3D POINT CLOUDS VIA OCCUPANCY GRIDS FOR SCENE CLASSIFICATION

High resolution consumer cameras on Unmanned Aerial Vehicles (UAVs) allow for cheap acquisition of highly detailed images, e.g., of urban regions. Via image registration by means of Structure from Motion (SfM) and Multi View Stereo (MVS) the automatic generation of huge amounts of 3D points with a relative accuracy in the centimeter range is possible. Applications such as semantic classification have a need for accurate 3D point clouds, but do not benefit from an extremely high resolution/density. In this paper, we, therefore, propose a fast fusion of high resolution 3D point clouds based on occupancy grids. The result is used for semantic classification. In contrast to state-of-the-art classification methods, we accept a certain percentage of outliers, arguing that they can be considered in the classification process when a per point belief is determined in the fusion process. To this end, we employ an octree-based fusion which allows for the derivation of outlier probabilities. The probabilities give a belief for every 3D point, which is essential for the semantic classification to consider measurement noise. For an example point cloud with half a billion 3D points (cf. Figure 1), we show that our method can reduce runtime as well as improve classification accuracy and offers high scalability for large datasets.

FAST PROBABILISTIC FUSION OF 3D POINT CLOUDS VIA OCCUPANCY GRIDS FOR SCENE CLASSIFICATION

Abstract. High resolution consumer cameras on Unmanned Aerial Vehicles (UAVs) allow for cheap acquisition of highly detailed images, e.g., of urban regions. Via image registration by means of Structure from Motion (SfM) and Multi View Stereo (MVS) the automatic generation of huge amounts of 3D points with a relative accuracy in the centimeter range is possible. Applications such as semantic classification have a need for accurate 3D point clouds, but do not benefit from an extremely high resolution/density. In this paper, we, therefore, propose a fast fusion of high resolution 3D point clouds based on occupancy grids. The result is used for semantic classification. In contrast to state-of-the-art classification methods, we accept a certain percentage of outliers, arguing that they can be considered in the classification process when a per point belief is determined in the fusion process. To this end, we employ an octree-based fusion which allows for the derivation of outlier probabilities. The probabilities give a belief for every 3D point, which is essential for the semantic classification to consider measurement noise. For an example point cloud with half a billion 3D points (cf. Figure 1), we show that our method can reduce runtime as well as improve classification accuracy and offers high scalability for large datasets.

VFA: A Variable-Factor Authentication Framework for Mobile Users

Multi-factor authentication (MFA) has been widely used in various scenarios. By combining multiple forms of authentication, MFA effectively provides security assurance. Due to the rapid developments of mobile devices, especially smart phones, more and more sensitive information is now stored or accessible on smart phones. How to protect smart phones' security is now more important than ever. Unfortunately, because of the special features of smart phones such as computational limitations and input constraints, existing MFA schemes could not be directly used on smart phones. In this paper, we propose a new concept of Variable-Factor Authentication (VFA) for smart phones. VFA dynamically adjusts the number of authentication factors based on whether a user is suspicious or not. We implement a prototype to exam the performance. The experiment results show that, compared to MFA, VFA provides significant convenience to legitimate users whereas maintain the security protection to suspicious users. Index Terms—Multi-factor authentication, variable authentication factors, local outlier probabilities

Neighborhood based global coordination for multimode process monitoring

A novel framework named neighborhood based global coordination (NBGC) for model alignment and multimode process monitoring is proposed in this paper. To identify the different patterns in the training database, a new clustering method is derived by utilizing the serial correlations between adjacent samples. With local outlier probability (LoOP) which can exhibit the novelty of the augmented samples, the fracture parts between multiple modes can be located. Then, an arrangement approach is conducted to piece together the similar but disconnecting segments of samples. Next, conventional principal component analysis (PCA) is applied for each separated cluster. Different from the traditional approaches where process monitoring will be performed individually and results from all local models will be summarized, the proposed method aims at involving the inter-mode correlations by aligning the local models together into a global model. A new objective function is proposed to ensure that both the local and nonlocal information can be included. Finally the utility and feasibility of NBGC are demonstrated through a numerical example and TE benchmark process.

Adaptive Local Outlier Probability for Dynamic Process Monitoring

Complex industrial processes often have multiple operating modes and present time-varying behavior. The data in one mode may follow specific Gaussian or non-Gaussian distributions. In this paper, a numerically efficient moving window local outlier probability algorithm is proposed. Its key feature is the capability to handle complex data distributions and incursive operating condition changes including slow dynamic variations and instant mode shifts. First, a two-step adaption approach is introduced and some designed updating rules are applied to keep the monitoring model up-to-date. Then, a semi-supervised monitoring strategy is developed with an updating switch rule to deal with mode changes. Based on local probability models, the algorithm has a superior ability in detecting faulty conditions and fast adapting to slow variations and new operating modes. Finally, the utility of the proposed method is demonstrated with a numerical example and a non-isothermal continuous stirred tank reactor.

Novel Outlier-Resistant Extended Kalman Filter for Robust Online Structural Identification

AbstractStructural health monitoring (SHM) using dynamic response measurement has received tremendous attention over the last decades. In practical circumstances, outliers may exist in the measurements that lead to undesirable identification results. Therefore, detection and special treatment of outliers are important. Unfortunately, this issue has rarely been taken into systematic consideration in SHM. In this paper, a novel outlier-resistant extended Kalman filter (OR-EKF) is proposed for outlier detection and robust online structural parametric identification using dynamic response data possibly contaminated with outliers. Instead of definite judgment on the outlierness of a data point, the proposed OR-EKF provides the probability of outlier for the measurement at each time step. By excluding the identified outliers, the OR-EKF ensures the stability and reliability of the estimation. In the illustrative examples, the OR-EKF is applied to parametric identification for structural systems with time-varyin...

Continuous Outlier Monitoring on Uncertain Data Streams

Outlier detection on data streams is an important task in data mining. The challenges become even larger when considering uncertain data. This paper studies the problem of outlier detection on uncertain data streams. We propose Continuous Uncertain Outlier Detection (CUOD), which can quickly determine the nature of the uncertain elements by pruning to improve the efficiency. Furthermore, we propose a pruning approach — Probability Pruning for Continuous Uncertain Outlier Detection (PCUOD) to reduce the detection cost. It is an estimated outlier probability method which can effectively reduce the amount of calculations. The cost of PCUOD incremental algorithm can satisfy the demand of uncertain data streams. Finally, a new method for parameter variable queries to CUOD is proposed, enabling the concurrent execution of different queries. To the best of our knowledge, this paper is the first work to perform outlier detection on uncertain data streams which can handle parameter variable queries simultaneously. Our methods are verified using both real data and synthetic data. The results show that they are able to reduce the required storage and running time.

ON THE CONSTRUCTION OF A NEW STELLAR CLASSIFICATION TEMPLATE LIBRARY FOR THE LAMOST SPECTRAL ANALYSIS PIPELINE

The LAMOST spectral analysis pipeline, called the 1D pipeline, aims to classify and measure the spectra observed in the LAMOST survey. Through this pipeline, the observed stellar spectra are classified into different subclasses by matching with template spectra. Consequently, the performance of the stellar classification greatly depends on the quality of the template spectra. In this paper, we construct a new LAMOST stellar spectral classification template library, which is supposed to improve the precision and credibility of the present LAMOST stellar classification. About one million spectra are selected from LAMOST Data Release One to construct the new stellar templates, and they are gathered in 233 groups by two criteria: (1) pseudo g – r colors obtained by convolving the LAMOST spectra with the Sloan Digital Sky Survey ugriz filter response curve, and (2) the stellar subclass given by the LAMOST pipeline. In each group, the template spectra are constructed using three steps. (1) Outliers are excluded using the Local Outlier Probabilities algorithm, and then the principal component analysis method is applied to the remaining spectra of each group. About 5% of the one million spectra are ruled out as outliers. (2) All remaining spectra are reconstructed using the first principal components of each group. (3) The weighted average spectrum is used as the template spectrum in each group. Using the previous 3 steps, we initially obtain 216 stellar template spectra. We visually inspect all template spectra, and 29 spectra are abandoned due to low spectral quality. Furthermore, the MK classification for the remaining 187 template spectra is manually determined by comparing with 3 template libraries. Meanwhile, 10 template spectra whose subclass is difficult to determine are abandoned. Finally, we obtain a new template library containing 183 LAMOST template spectra with 61 different MK classes by combining it with the current library.

A generalised approach to outlier identification in pavement condition data

Pavement condition data often provide the quantitative measure by which funding for maintenance is prioritised. It is vital that these data are of the best quality possible. Measurement data are prone to inaccuracies and will exhibit outliers and deviations from the inferred expected condition. If a systematic trend in outliers is identified, for example a range of outliers adjacent to each other, then there may be a systematic bias in the measurements which can be reduced or removed from future data measurements. This paper introduces a new approach of visualising the probability of measurement outliers by representing the network as a distance/time matrix, with colours associated with different outlier probabilities. To assist in identifying significant outlier trends, a combined method of supervised data mining is proposed, combining expert knowledge and a new minimum message length criterion to select significant trends of systematic outliers.

Cancer Outlier Analysis Based on Mixture Modeling of Gene Expression Data

Molecular heterogeneity of cancer, partially caused by various chromosomal aberrations or gene mutations, can yield substantial heterogeneity in gene expression profile in cancer samples. To detect cancer-related genes which are active only in a subset of cancer samples or cancer outliers, several methods have been proposed in the context of multiple testing. Such cancer outlier analyses will generally suffer from a serious lack of power, compared with the standard multiple testing setting where common activation of genes across all cancer samples is supposed. In this paper, we consider information sharing across genes and cancer samples, via a parametric normal mixture modeling of gene expression levels of cancer samples across genes after a standardization using the reference, normal sample data. A gene-based statistic for gene selection is developed on the basis of a posterior probability of cancer outlier for each cancer sample. Some efficiency improvement by using our method was demonstrated, even under settings with misspecified, heavy-tailed t-distributions. An application to a real dataset from hematologic malignancies is provided.

Statistical Analysis of Biomarkers for Personalized Medicine

Recently there has been enhanced and advanced biomedical technology such as high-throughput microarrays and molecular imaging to monitor SNPs, gene and protein expressions, and so forth, to provide exhaustive situations for individuals. In principle we could get much more information to know the biological and medical status from such data sets, which are viewed as biomarkers in a wide sense to help to do identification, association, and prediction studies for phenotypes such as cancer subtypes, prognosis, treatment responsiveness, and adverse reactions for personalized medicine. However, it is frequently difficult to extract only an informative part in the hull of data sets which include a garbage heap of uninformative observations. In particular, we cannot confirm statistical evidence because of the small sample size relative to the dimension of the data when we achieve the practical use of such statistical applications. For example, if we implement machine learning methods for the prediction of the treatment effects, then we typically face difficulties to confirm the reproducibility and robustness for the performance. In the meanwhile, effective study design is crucial for developing and validating biomarkers. In effect we present eight papers challenging this difficult situation of statistical analysis for biomarkers including statistical estimation, prediction, and testing hypothesis. The paper, “The number of candidate variants in exome sequencing for Mendelian Disease under no genetic heterogeneity,” by J. Nishino and S. Mano studies the expected number of candidate single nucleotide variants (SNVs) in exome data for autosomal dominant or recessive Mendelian disorders so that the filtering method is useful with an assumption of “no genetic heterogeneity.” The paper, “Radial basis function-sparse partial least squares for application to brain imaging data,” by H. Yoshida et al. proposes a novel statistical method, called radial basis function sparse partial least squares (RBF-sPLS), for investigating the relationship between clinical characteristics and brain morphology based on three-dimensional MRI data. The paper, “Power analysis of C-TDT for small sample size genome-wide association studies by the joint use of case-parent trios and pairs,” by F. Rajabli et al. addresses family-based genetic association studies with missing genotype information for one of the parents. The paper provides a simulation study for the disequilibrium test statistic across different sample sizes for a family-based genome-wide association study. The paper, “Genomic biomarkers for personalized medicine: development and validation in clinical studies” by S. Matsui addresses the two critical statistical analyses of high-dimensional genomic data, gene screening and prediction, in the framework of development and validation of genomic biomarkers or signatures with biomarker-based clinical trial designs to assess clinical utility of a biomarker or a new treatment with a companion biomarker. The paper, “Multiple suboptimal solutions for prediction rules in gene expression data,” by O. Komori et al. reanalyzes the gene expression data for pattern recognition of phenotypes to extract informative features embedded in the data. The paper concludes that it is not possible to extract only informative genes with high performance in the all observed genes and points out that the mutual coherence among genes is too high to find the solution of informative genes. The paper, “An empirical Bayes optimal discovery procedure based on semiparametric hierarchical mixture models,” by H. Noma and S. Matsui proposes an empirical Bayes optimal discovery procedure with the smoothing-by-roughening approach in semiparametric hierarchical mixture models. This approach provides analytically a tractable posterior distribution with the flexible prior section in genome-wide studies. The paper, “Cancer outlier analysis based on mixture modeling of gene expression data,” by K. Mori et al. proposes a gene-based statistic for gene selection on the basis of a posterior probability of cancer outlier for each cancer sample with an application to real data from hematologic malignancies. The paper, “A robust rerank approach for feature selection and its application to pooling-based GWA studies,” by J. Liu et al. proposes a method based on the concept of rank-over-variable which is shown to be insensitive to the selection of tuning parameters compared with t-statistics, AUC statistics, and SAM with simulation studies and real data analysis of pooling-based genome-wide association (GWA) study to demonstrate the usefulness of their method. Shinto Eguchi Shigeyuki Matsui Su-Yun Huang Chuhsing Kate Hsiao