Framework For Outlier Detection Research Articles

Enhancing mass spectrometry data analysis: A novel framework for calibration, outlier detection, and classification

Mass spectrometry (MS) is a powerful analytical technique in metabolomics, enabling the identification and quantification of metabolites. However, analyzing MS data poses challenges such as batch effects, outliers, and high-dimensional data. In this paper, we propose a comprehensive framework for MS data analysis. The framework integrates data calibration, outlier detection, and automatic classification modules. Data calibration is performed using a deep autoencoder to remove batch effects. Outlier detection combines multiple algorithms through ensemble learning to identify and remove outliers. Automatic classification utilizes a transformer model to handle high-dimensional data and capture global feature relationships. Experimental results on myocardial infarction (MI) and coronary heart disease (CHD) datasets demonstrate the effectiveness of the framework. It outperforms traditional classification models and achieves higher accuracy. The proposed framework provides a robust solution for MS data analysis, facilitating more accurate classification and enabling reliable biological insights in metabolomics research.

A geometric framework for outlier detection in high‐dimensional data

AbstractOutlier or anomaly detection is an important task in data analysis. We discuss the problem from a geometrical perspective and provide a framework which exploits the metric structure of a data set. Our approach rests on the manifold assumption, that is, that the observed, nominally high‐dimensional data lie on a much lower dimensional manifold and that this intrinsic structure can be inferred with manifold learning methods. We show that exploiting this structure significantly improves the detection of outlying observations in high dimensional data. We also suggest a novel, mathematically precise and widely applicable distinction between distributional and structural outliers based on the geometry and topology of the data manifold that clarifies conceptual ambiguities prevalent throughout the literature. Our experiments focus on functional data as one class of structured high‐dimensional data, but the framework we propose is completely general and we include image and graph data applications. Our results show that the outlier structure of high‐dimensional and non‐tabular data can be detected and visualized using manifold learning methods and quantified using standard outlier scoring methods applied to the manifold embedding vectors.This article is categorized under: Technologies > Structure Discovery and Clustering Fundamental Concepts of Data and Knowledge > Data Concepts Technologies > Visualization

E 3Outlier: a Self-Supervised Framework for Unsupervised Deep Outlier Detection.

Existing unsupervised outlier detection (OD) solutions face a grave challenge with surging visual data like images. Although deep neural networks (DNNs) prove successful for visual data, deep OD remains difficult due to OD's unsupervised nature. This paper proposes a novel framework named E 3Outlier that can perform effective and end-to-end deep outlier removal. Its core idea is to introduce self-supervision into deep OD. Specifically, our major solution is to adopt a discriminative learning paradigm that creates multiple pseudo classes from given unlabeled data by various data operations, which enables us to apply prevalent discriminative DNNs (e.g., ResNet) to the unsupervised OD problem. Then, with theoretical and empirical demonstration, we argue that inlier priority, a property that encourages DNN to prioritize inliers during self-supervised learning, makes it possible to perform end-to-end OD. Meanwhile, unlike frequently-used outlierness measures (e.g., density, proximity) in previous OD methods, we explore network uncertainty and validate it as a highly effective outlierness measure, while two practical score refinement strategies are also designed to improve OD performance. Finally, in addition to the discriminative learning paradigm above, we also explore the solutions that exploit other learning paradigms (i.e., generative learning and contrastive learning) to introduce self-supervision for E 3Outlier. Such extendibility not only brings further performance gain on relatively difficult datasets, but also enables E 3Outlier to be applied to other OD applications like video abnormal event detection. Extensive experiments demonstrate that E 3Outlier can considerably outperform state-of-the-art counterparts by 10%-30% AUROC. Demo codes are available at https://github.com/demonzyj56/E3Outlier.

TTDD: a time series Tester, Transformer, and Decomposer framework for outlier Detection

TTDD: a time series Tester, Transformer, and Decomposer framework for outlier Detection

An efficient ensemble framework for outlier detection using bio-inspired algorithm

Outliers are unexpected observations present in data that has to be identified systematically to prevent from catastrophic effects. The detection of outliers plays a crucial role in many application domains, as they provide exciting and critical information about abnormal events. So, in this research work, the bio-inspired ensemble-based outlier detection (BIEOD) is employed to determine the outliers. This ensemble framework combines the sequential and parallel ensembles, thereby creating a hybrid model that utilises the advantages of both feature bagging and XGBoost. The chicken swarm optimisation (CSO) is used to increase the deviation between the outliers and inliers according to the chicken competition in optimising the outlier scores. Finally, the XGBoost predicts the outliers based on the optimised feature set. The obtained results on the benchmark datasets reveal that this framework offers significant advancement over the base learners and other existing ensemble methods of outlier detection.

An Effective Ensemble-based Framework for Outlier Detection in Evolving Data Streams

In the last few years, data streams have drawn lots of researchers’ attention due to their various applications, such as healthcare monitoring systems, fraud and intrusion detection, the internet of things (IoT), and financial market applications. A data stream is an unbounded sequence of data continually generated over time and is prone to evolution. Outliers in streaming data are the elements that significantly deviate from the majority of elements and then have to be detected as they may be error values or events of interest. Detection of outliers is a challenging issue in streaming data and is one of the most crucial tasks in data stream mining. Existing outlier detection methods for static data are unsuitable for use in data stream settings due to the unique characteristics of streaming data such as unpredictability, uncertainty, high-dimensionality, and changes in data distribution. Thus, in this paper, a novel ensemble learning framework called Ensemble-based Streaming Outlier Detection (ESOD) is presented to perfectly detect outliers over streaming data using a sliding window technique that is updated in response to the incoming events from the data streaming environment to overcome the concept evolution nature of streaming data. The proposed framework has three phases, namely the training phase, testing/offline phase, and outlier detection/online phase. A detection weighted vote technique is used to determine the final decisions for potential outliers. In the extensive experimental study, which was conducted on 11 real-world benchmark datasets, the proposed framework was assessed using many accuracy metrics. The experiment results showed that the proposed framework beats many other state-of-the-art methods.

FairLOF: Fairness in Outlier Detection

An outlier detection method may be considered fair over specified sensitive attributes if the results of outlier detection are not skewed toward particular groups defined on such sensitive attributes. In this paper, we consider the task of fair outlier detection. Our focus is on the task of fair outlier detection over multiple multi-valued sensitive attributes (e.g., gender, race, religion, nationality and marital status, among others), one that has broad applications across modern data scenarios. We propose a fair outlier detection method, FairLOF, that is inspired by the popular LOF formulation for neighborhood-based outlier detection. We outline ways in which unfairness could be induced within LOF and develop three heuristic principles to enhance fairness, which form the basis of the FairLOF method. Being a novel task, we develop an evaluation framework for fair outlier detection, and use that to benchmark FairLOF on quality and fairness of results. Through an extensive empirical evaluation over real-world datasets, we illustrate that FairLOF is able to achieve significant improvements in fairness at sometimes marginal degradations on result quality as measured against the fairness-agnostic LOF method. We also show that a generalization of our method, named FairLOF-Flex, is able to open possibilities of further deepening fairness in outlier detection beyond what is offered by FairLOF.

A highly efficient framework for outlier detection in urban traffic flow

The outliers in traffic flow represent the anomalies or emergencies in the road. The detection and research of outliers will help to reveal the mechanism of such events. Aiming at the problem of outlier detection in urban traffic flow, this paper innovatively proposes a highly efficient traffic outlier detection framework based on the study of road traffic flow patterns. The main research works are as follows: (1) data pre-processing, the road traffic flow matrix of the roads is calculated based on the collected GPS data, the non-negative matrix factorisation algorithm is chosen to reduce the dimension of the matrix. (2) Road traffic flow pattern extraction, the fuzzy C-means clustering algorithm with the Optimal k-cluster centre (K-FCM) is adopted to cluster the roads with the same road traffic flow pattern. (3) Outlier detection model training and evaluation, kernel density estimation is introduced to fit the probability density of roads traffic flow matrices which are used to train the back propagation neural network based on particle swarm optimisation to obtain the outlier detection and evaluation model, and a threshold is introduced to optimise the precision and recall of the model. The experimental results show that: the average precision and recall of the proposed method in this paper are 95.38% and 96.23%, respectively, and the average detection time is 28.4 seconds. The method has high accuracy, high efficiency and good practical significance.

An efficient ensemble framework for outlier detection using bio-inspired algorithm

An efficient ensemble framework for outlier detection using bio-inspired algorithm

EPCO-23. COMPARATIVE TRANSCRIPTOMICS TO IDENTIFY TARGETED THERAPY CANDIDATES IN HIGH GRADE GLIOMA

Abstract Genomic characterization is often used for the identification of therapeutic targets in tumors. Recently, comparative transcriptomics has begun to be utilized for this purpose. In this pilot, we compare the transcriptome of a patient with recurrent high grade glioma (HGG) to our cohort to identify potential therapies. We reviewed transcriptomic profiles from patients who had resection of HGG at our institution over the past year as well as the UCSC cancer compendium. Briefly, tumor RNA was extracted from embedded tumor tissue sections with tumor cellularity higher than 20%. RNA libraries were sequenced to obtain approximately 65 million reads on an Illumina HiSeq 4000 System utilizing patterned flow cell technology. The RNA profile of a 24 male with Li-Fraumeni syndrome and recurrent HGG with leptomeningeal spread underwent comparative transcriptomics to identify targets. A Bayesian statistical framework for gene expression outlier detection was used. These comparisons allowed for the identification of genes and pathways that are significantly overexpressed. Our internal HGG cohort consisted of 44 adult patients and was evenly distributed among the 4 HGG Verhaak subtypes. Our patient of interest had druggable outlier expression in HDAC1, STAT1 and STAT2 in comparison to our internal cohort indicating vorinostat and ruxolitinib as potential therapies, respectively. We then compared our patient of interest to 12,747 patients in the cancer compendium and STAT2 expression was high but not an outlier. In comparison to 738 glioma samples, STAT1 and STAT2 were outliers but not HDAC1 again indicating ruxolitinib as a potential targeted therapy. The patient did not have outlier expression in notch transcriptional targets or immune checkpoint biomarkers when compared to all cohorts. In conclusion, comparative Transcriptomics can identify therapeutic targets in a patient with recurrent HGG even in small cohorts. In our pilot, we identified ruxolitinib as a potential candidate to treat leptomeningeal recurrence.

Bayesian Framework for Detecting Gene Expression Outliers in Individual Samples.

Many antineoplastics are designed to target upregulated genes, but quantifying upregulation in a single patient sample requires an appropriate set of samples for comparison. In cancer, the most natural comparison set is unaffected samples from the matching tissue, but there are often too few available unaffected samples to overcome high intersample variance. Moreover, some cancer samples have misidentified tissues of origin or even composite-tissue phenotypes. Even if an appropriate comparison set can be identified, most differential expression tools are not designed to accommodate comparisons to a single patient sample. We propose a Bayesian statistical framework for gene expression outlier detection in single samples. Our method uses all available data to produce a consensus background distribution for each gene of interest without requiring the researcher to manually select a comparison set. The consensus distribution can then be used to quantify over- and underexpression. We demonstrate this method on both simulated and real gene expression data. We show that it can robustly quantify overexpression, even when the set of comparison samples lacks ideally matched tissue samples. Furthermore, our results show that the method can identify appropriate comparison sets from samples of mixed lineage and rediscover numerous known gene-cancer expression patterns. This exploratory method is suitable for identifying expression outliers from comparative RNA sequencing (RNA-seq) analysis for individual samples, and Treehouse, a pediatric precision medicine group that leverages RNA-seq to identify potential therapeutic leads for patients, plans to explore this method for processing its pediatric cohort.

HYBRID MACHINE LEARNING BASED FRAMEWORK FOR OUTLIER DETECTION IN HIGH DIMENSIONAL DATA

High dimensional data throws challenges like curse of dimensionality and data imbalance. This often leads to suboptimal performance of algorithms used for outlier detection. The existing methods based on unsupervised and supervised learning have their strengths. For instance, supervised learning can predict outliers based on quality of training while the unsupervised learning methods are good at learning complex patterns. Combining both supervised and unsupervised approaches gain the synergic effect of the both methods. In this paper, we proposed a framework known as Hybrid Machine Learning based Framework for Outlier Detection (HMLF-OD). It has an underlying algorithm named Hybrid Outlier Prediction (HOP). The algorithm combines the strengths of unsupervised outlier and supervised outlier methods. It results in an improved feature space that causes performance improvement. The algorithm is evaluated with seven real-world outlier detection datasets namely Cardio, Letter, Arrhythmia, Mnist, Satellite, Speech and Mammography. TensorFlow is used as backend. The experimental results revealed that the proposed framework outperforms the state of the art.

A Bayesian Framework for Robust Quantitative Trait Locus Mapping and Outlier Detection.

We introduce a Bayesian framework for simultaneous feature selection and outlier detection in sparse high-dimensional regression models, with a focus on quantitative trait locus (QTL) mapping in experimental crosses. More specifically, we incorporate the robust mean shift outlier handling mechanism into the multiple QTL mapping regression model and apply LASSO regularization concurrently to the genetic effects and the mean-shift terms through the flexible extended Bayesian LASSO (EBL) prior structure, thereby combining QTL mapping and outlier detection into a single sparse model representation problem. The EBL priors on the mean-shift terms prevent outlying phenotypic values from distorting the genotype-phenotype association and allow their detection as cases with outstanding mean shift values following the LASSO shrinkage. Simulation results demonstrate the effectiveness of our new methodology at mapping QTLs in the presence of outlying phenotypic values and simultaneously identifying the potential outliers, while maintaining a comparable performance to the standard EBL on outlier-free data.

A Semantic Approach for Outlier Detection in Big Data Streams

In recent years, the world faced a big revolution in data generation and collection technologies. The volume, velocity and veracity of data have changed drastically and led to new types of challenges related to data analysis, modeling and prediction. One of the key challenges is related to the semantic analysis of textual data especially in big data streams settings. The existing solutions focus on either topic analysis or the sentiment analysis. Moreover, the semantic outlier detection over data streams as one of the key problems in data mining and data analysis fields has less focus. In this paper, we introduce a new concept of semantic outlier through which the topic of the textual data is considered as the primary content of the data stream while the sentiment is considered as the context in which the data has been generated and affected. Also, we propose a framework for semantic outlier detection in big data streams which incorporates the contextual detection concepts. The advantage of the proposed concept is that it incorporates both topic and sentiment analysis into one single process; while at the same time the framework enables the implementation of different algorithms and approaches for semantic analysis.

Framework for DNA Quantification and Outlier Detection Using Multidimensional Standard Curves.

Real-time PCR is a highly sensitive and powerful technology for the quantification of DNA and has become the method of choice in microbiology, bioengineering, and molecular biology. Currently, the analysis of real-time PCR data is hampered by only considering a single feature of the amplification profile to generate a standard curve. The current “gold standard” is the cycle-threshold (Ct) method which is known to provide poor quantification under inconsistent reaction efficiencies. Multiple single-feature methods have been developed to overcome the limitations of the Ct method; however, there is an unexplored area of combining multiple features in order to benefit from their joint information. Here, we propose a novel framework that combines existing standard curve methods into a multidimensional standard curve. This is achieved by considering multiple features together such that each amplification curve is viewed as a point in a multidimensional space. Contrary to only considering a single-feature, in the multidimensional space, data points do not fall exactly on the standard curve, which enables a similarity measure between amplification curves based on distances between data points. We show that this framework expands the capabilities of standard curves in order to optimize quantification performance, provide a measure of how suitable an amplification curve is for a standard, and thus automatically detect outliers and increase the reliability of quantification. Our aim is to provide an affordable solution to enhance existing diagnostic settings through maximizing the amount of information extracted from conventional instruments.

A Novel Framework for Context-aware Outlier Detection in Big Data Streams

A Novel Framework for Context-aware Outlier Detection in Big Data Streams

Multi-View Low-Rank Analysis with Applications to Outlier Detection

Detecting outliers or anomalies is a fundamental problem in various machine learning and data mining applications. Conventional outlier detection algorithms are mainly designed for single-view data. Nowadays, data can be easily collected from multiple views, and many learning tasks such as clustering and classification have benefited from multi-view data. However, outlier detection from multi-view data is still a very challenging problem, as the data in multiple views usually have more complicated distributions and exhibit inconsistent behaviors. To address this problem, we propose a multi-view low-rank analysis (MLRA) framework for outlier detection in this article. MLRA pursuits outliers from a new perspective, robust data representation. It contains two major components. First, the cross-view low-rank coding is performed to reveal the intrinsic structures of data. In particular, we formulate a regularized rank-minimization problem, which is solved by an efficient optimization algorithm. Second, the outliers are identified through an outlier score estimation procedure. Different from the existing multi-view outlier detection methods, MLRA is able to detect two different types of outliers from multiple views simultaneously. To this end, we design a criterion to estimate the outlier scores by analyzing the obtained representation coefficients. Moreover, we extend MLRA to tackle the multi-view group outlier detection problem. Extensive evaluations on seven UCI datasets, the MovieLens, the USPS-MNIST, and the WebKB datasets demon strate that our approach outperforms several state-of-the-art outlier detection methods.

Mining multidimensional contextual outliers from categorical relational data

A wide range of methods have been proposed for detecting different types of outliers in both the full attribute space and its subspaces. However, the interpretability of outliers, that is, explaining in what ways and to what extent an object is an outlier, remains a critical issue. In this paper, we focus on improving the interpretability of outliers. Particularly, we develop a notion of multidimensional contextual outliers to model the context of an outlier, and propose a framework for contextual outlier detection. Intuitively, a contextual outlier is a small group of objects that share strong similarity with a significantly larger reference group of objects on some attributes, but deviate dramatically on some other attributes. In contextual outlier detection, we identify not only the outliers, but also their associated contextual information including (1) comparing to what reference group of objects the detected object(s) is/are an outlier; (2) the attributes defining the unusual behavior of the outlier(s) compared against the reference group; (3) the population of similar outliers sharing the same context; and (4) the outlier degree, which measures the population ratio between the reference group and the outlier group. We present an algorithm and conduct extensive experiments to evaluate our approach.

Hierarchical Density Estimates for Data Clustering, Visualization, and Outlier Detection

An integrated framework for density-based cluster analysis, outlier detection, and data visualization is introduced in this article. The main module consists of an algorithm to compute hierarchical estimates of the level sets of a density, following Hartigan’s classic model of density-contour clusters and trees. Such an algorithm generalizes and improves existing density-based clustering techniques with respect to different aspects. It provides as a result a complete clustering hierarchy composed of all possible density-based clusters following the nonparametric model adopted, for an infinite range of density thresholds. The resulting hierarchy can be easily processed so as to provide multiple ways for data visualization and exploration. It can also be further postprocessed so that: (i) a normalized score of “outlierness” can be assigned to each data object, which unifies both the global and local perspectives of outliers into a single definition; and (ii) a “flat” (i.e., nonhierarchical) clustering solution composed of clusters extracted from local cuts through the cluster tree (possibly corresponding to different density thresholds) can be obtained, either in an unsupervised or in a semisupervised way. In the unsupervised scenario, the algorithm corresponding to this postprocessing module provides a global, optimal solution to the formal problem of maximizing the overall stability of the extracted clusters. If partially labeled objects or instance-level constraints are provided by the user, the algorithm can solve the problem by considering both constraints violations/satisfactions and cluster stability criteria. An asymptotic complexity analysis, both in terms of running time and memory space, is described. Experiments are reported that involve a variety of synthetic and real datasets, including comparisons with state-of-the-art, density-based clustering and (global and local) outlier detection methods.

A Framework for Outlier Detection in Geographic Spatial Data

Outlier detection is very interesting, useful and challenging problem in the field of data mining. Because of sparse data clustering algorithm which are based on distance will not work to find outliers in spatial data. Problem of finding irregular feature in spatial data need to be explore. Many existing approaches have been proposed to overcome the problem of outlier detection in spatial Geographic data. In this paper an efficient clustering and density based outlier detection framework has been proposed. The process of outlier detection has been categorized into two steps in the first step data has been clustered together based on any density based DBSCAN algorithm and in the second stage outlier detection is performed using LOF. The purpose is to perform clustering and outlier mining simultaneously to improve feasibility of framework. To verify the efficiency and robustness of proposed method, comparative study of proposed approach and several existing approaches are presented in detail, various simulation results demonstrate the effectiveness of the proposed approach.