Outlier Cluster Research Articles

Simplicial and Minimal-Variance Distances in Multivariate Data Analysis

In this paper, we study the behaviour of the so-called k-simplicial distances and k-minimal-variance distances between a point and a sample. The family of k-simplicial distances includes the Euclidean distance, the Mahalanobis distance, Oja’s simplex distance and many others. We give recommendations about the choice of parameters used to calculate the distances, including the size of the sub-sample of simplices used to improve computation time, if needed. We introduce a new family of distances which we call k-minimal-variance distances. Each of these distances is constructed using polynomials in the sample covariance matrix, with the aim of providing an alternative to the inverse covariance matrix, that is applicable when data is degenerate. We explore some applications of the considered distances, including outlier detection and clustering, and compare how the behaviour of the distances is affected for different parameter choices.

Consensus-Clustering-Based Automatic Distribution Matching for Cross-Domain Image Steganalysis

Image steganalysis is a technique to detect whether an image contains hidden information. Although the existing cross-domain steganalysis methods have been presented to narrow the distribution gap between different domains, it is still challenging to effectively capture the transferable steganalysis representations under the condition of severe distribution shifts. To address this issue, we propose a novel consensus-clustering-based automatic distribution matching scheme, called CADM, which can automatically and accurately match inconsistent distributions in cross-domain steganalysis scenarios. First, the original steganalysis features are clustered by the spatially constrained fuzzy <i>c</i>-means (SCFCM) algorithm with controllable parameters to fully perceive and mine inherent structural relationships. Subsequently, the cluster consensus knowledge is derived from the perspective of intra-domain and inter-domain to facilitate the clustering and the matching. In this way, the representations of weak stego signals can be augmented by identifying cluster centers that can be combined across domains. Ultimately, the cycle-consistent optimization and adaptation is achieved by gradually adjusting the learning strength of well-aligned and poorly-aligned samples to promote the positive transfer of overlapped clusters and prevent the negative transfer of outlier clusters. Furthermore, extensive experiments on various benchmark databases for cross-domain steganalysis demonstrate the superiority of CADM over the current state-of-the-art methods.

Operational Modes Detection in Industrial Gas Turbines Using an Ensemble of Clustering Methods.

Operational modes of a process are described by a number of relevant features that are indicative of the state of the process. Hundreds of sensors continuously collect data in industrial systems, which shows how the relationship between different variables changes over time and identifies different modes of operation. Gas turbines’ operational modes are usually defined regarding their expected energy production, and most research works either are focused a priori on obtaining these modes solely based on one variable, the active load, or assume a fixed number of states and build up predictive models to classify new situations as belonging to the predefined operational modes. However, in this work, we take into account all available parameters based on sensors’ data because other factors can influence the system status, leading to the identification of a priori unknown operational modes. Furthermore, for gas turbine management, a key issue is to detect these modes using a real-time monitoring system. Our approach is based on using unsupervised machine learning techniques, specifically an ensemble of clusters to discover consistent clusters, which group data into similar groups, and to generate in an automatic way their description. This description, upon interpretation by experts, becomes identified and characterized as operational modes of an industrial process without any kind of a priori bias of what should be the operational modes obtained. Our proposed methodology can discover and identify unknown operational modes through data-driven models. The methodology was tested in our case study with Siemens gas turbine data. From available sensors’ data, clusters descriptions were obtained in an automatic way from aggregated clusters. They improved the quality of partitions tuning one consistency parameter and excluding outlier clusters by defining filtering thresholds. Finally, operational modes and/or sub-operational modes were identified with the interpretation of the clusters description by process experts, who evaluated the results very positively.

Oral Cancer Discrimination and Novel Oral Epithelial Dysplasia Stratification Using FTIR Imaging and Machine Learning.

The Fourier transform infrared (FTIR) imaging technique was used in a transmission model for the evaluation of twelve oral hyperkeratosis (HK), eleven oral epithelial dysplasia (OED), and eleven oral squamous cell carcinoma (OSCC) biopsy samples in the fingerprint region of 1800–950 cm−1. A series of 100 µm × 100 µm FTIR imaging areas were defined in each sample section in reference to the hematoxylin and eosin staining image of an adjacent section of the same sample. After outlier removal, signal preprocessing, and cluster analysis, a representative spectrum was generated for only the epithelial tissue in each area. Two representative spectra were selected from each sample to reflect intra-sample heterogeneity, which resulted in a total of 68 representative spectra from 34 samples for further analysis. Exploratory analyses using Principal component analysis and hierarchical cluster analysis showed good separation between the HK and OSCC spectra and overlaps of OED spectra with either HK or OSCC spectra. Three machine learning discriminant models based on partial least squares discriminant analysis (PLSDA), support vector machines discriminant analysis (SVMDA), and extreme gradient boosting discriminant analysis (XGBDA) were trained using 46 representative spectra from 12 HK and 11 OSCC samples. The PLSDA model achieved 100% sensitivity and 100% specificity, while both SVM and XGBDA models generated 95% sensitivity and 96% specificity, respectively. The PLSDA discriminant model was further used to classify the 11 OED samples into HK-grade (6), OSCC-grade (4), or borderline case (1) based on their FTIR spectral similarity to either HK or OSCC cases, providing a potential risk stratification strategy for the precancerous OED samples. The results of the current study support the application of the FTIR-machine learning technique in early oral cancer detection.

Distance Based Pattern Driven Mining for Outlier Detection in High Dimensional Big Dataset

Detection of outliers or anomalies is one of the vital issues in pattern-driven data mining. Outlier detection detects the inconsistent behavior of individual objects. It is an important sector in the data mining field with several different applications such as detecting credit card fraud, hacking discovery and discovering criminal activities. It is necessary to develop tools used to uncover the critical information established in the extensive data. This paper investigated a novel method for detecting cluster outliers in a multidimensional dataset, capable of identifying the clusters and outliers for datasets containing noise. The proposed method can detect the groups and outliers left by the clustering process, like instant irregular sets of clusters (C) and outliers (O), to boost the results. The results obtained after applying the algorithm to the dataset improved in terms of several parameters. For the comparative analysis, the accurate average value and the recall value parameters are computed. The accurate average value is 74.05% of the existing COID algorithm, and our proposed algorithm has 77.21%. The average recall value is 81.19% and 89.51% of the existing and proposed algorithm, which shows that the proposed work efficiency is better than the existing COID algorithm.

GIS-based analysis of spatial\u2013temporal correlations of urban traffic accidents

BackgroundUnderstanding the spatial–temporal distribution characteristics of urban road traffic accidents is important for urban road traffic safety management. Based on the road traffic data of Wales in 2017, the spatial–temporal distribution of accidents is formed.MethodsThe density analysis method is used to identify the areas with high accident incidence and the areas with high accident severity. Then, two types of spatial clustering analysis models, outlier analysis and hot spot analysis are used to further identify the regions with high accident severity.ResultsThe results of density analysis and cluster analysis are compared. The results of density analysis show that, in terms of accident frequency and accident severity, Swansea, Neath Port Talbot, Bridgend, Merthyr Tydfil, Cardiff, Caerphilly, Newport, Denbighshire, Vale of Glamorgan, Rhondda Cynon Taff, Flintshire and Wrexham have high accident frequency and accident severity per unit area. Cluster analysis results are similar to the density analysis. Finally, the temporal distribution characteristics of traffic accidents are analyzed according to month, week, day and hour. Accidents are concentrated in July and August, frequently in the morning rush hour and at dusk, with the most accidents occurring on Saturday.ConclusionsBy comparing the two methods, it can be concluded that the density analysis is simple and easy to understand, which is conducive to understanding the spatial distribution characteristics of urban traffic accidents directly. Cluster analysis can be accurate to the accident point and obtain the clustering characteristics of road accidents.

Spatio-Temporal association rule based deep annotation-free clustering (STAR-DAC) for unsupervised person re-identification

Multi-camera video surveillance environment has a variety of emerging research problems among, which person re-identification is the premier one. Unsupervised person re-identification has been explored less in literature than the supervised approach. Images acquired from the video surveillance systems are unlabeled, which denotes that it is naturally an unsupervised learning problem. The state-of-the-art unsupervised methods seek external annotations support such as incorporating transfer learning techniques, partial labeling of train images, etc., which makes them not purely unsupervised and unsuitable for practical real-world surveillance settings. Identity mismatch happens due to the similar costumes and complex environmental factors. To resolve this issue, we introduce a new framework named Spatio-Temporal Association Rule based Deep Annotation-free Clustering (STAR-DAC) which incrementally clusters the unlabeled person re-identification images based on visual features and performs cluster fine-tuning through the mined spatio-temporal association rules. STAR formulations leveraged upto 75% of images for reliable sample selection through cluster fine-tuning. STAR based fine-tune algorithm aims to attain ground-truth labels of an unlabeled dataset and eliminate cluster outliers to stabilize the evaluation. Experiments are performed on image and video-based benchmark person re-identification datasets such as DukeMTMC re-ID, Market1501, MSMT17, CUHK03, GRID and Dukevideo re-ID, iLIDSVid, ViPer respectively. Experimental results clearly show that the proposed STAR-DAC framework outperforms the state-of-the-art methods in case of large scale datasets with multiple cameras.

Studying and Clustering Cities Based on Their Non-Emergency Service Requests

This study offers a new perspective in analyzing 311 service requests (SRs) across the country by representing cities based on the types of their SRs. This not only uncovers temporal patterns of SRs in each city over the years but also detects cities with the most or least similarity to other cities based on their SR types. The first challenge is to gather 311 SRs for different cities and standardize their types since they differ in various cities. Implementing our analyses on close to 42 million SR records in 20 cities from 2006 to 2019 is the second challenge. Representing clusters of cities and outliers effectively, and providing justifications for them, is the last challenge. Our attempt resulted in 79 standardized SR types. We applied the principal component analysis to depict cities on a two-dimensional canvas based on their standardized SR types. Among our main findings are the following: many cities are observing a fall in requests regarding the condition of roads and sidewalks but a rise in requests concerning transportation and traffic; requests regarding garbage, cleaning, rodents, and complaints have also been rising in some cities; new types of requests have emerged and soared in recent years, such as requests for information and regarding shared mobility devices; requests about parking meters, information, sidewalks, curbs, graffities, and missed garbage pick up have the highest variance in their rates across different cities, i.e., they have a large rate in some cities while a low rate in others; the most consistent outliers, in terms of SR types, are Washington DC, Baltimore, Las Vegas, Philadelphia, Chicago, and Baton Rouge.

A New Procedure of Clustering Based on Multivariate Outlier Detection

Clustering is an extremely important task in a wide variety of ap plication domains especially in management and social science research. In this paper, an iterative procedure of clustering method based on multivariate outlier detection was proposed by using the famous Mahalanobis distance. At first, Mahalanobis distance should be calculated for the entire sample, then using T 2 -statistic fix a UCL. Above the UCL are treated as outliers which are grouped as outlier cluster and repeat the same procedure for the remaining inliers, until the variance-covariance matrix for the variables in the last cluster achieved singularity. At each iteration, multivariate test of mean used to check the discrimination between the outlier clusters and the inliers. Moreover, multivariate control charts also used to graphically visual izes the iterations and outlier clustering process. Finally multivariate test of means helps to firmly establish the cluster discrimination and validity. This paper employed this procedure for clustering 275 customers of a famous two wheeler in India based on 19 different attributes of the two wheeler and its company. The result of the proposed technique confirms there exist 5 and 7 outlier clusters of customers in the entire sample at 5% and 1% significance level respectively.

Geographic disparities in violent crime during the COVID-19 lockdown in Miami-Dade County, Florida, 2018-2020.

ObjectivesThis study uses two cluster detection techniques to identify clusters of violent crime during the 3 months of the 2020 COVID-19 lockdown in Miami-Dade County compared to that during an equivalent period in 2018 and 2019.MethodsViolent crime data from the Miami-Dade Central Records Bureau were analyzed. The Local Indicators of Spatial Association statistics and a space‐time permutation statistic were used to identify clusters of violent crimes and outliers, and Global Moran’s I tool was used to assess spatial patterning in violent crime. Neighborhood disadvantage data were obtained from the American Community Survey 5-year estimates linked with arrest locations.ResultsViolent crime arrests fell by 7.1% in 2020. Arrests were concentrated in predominantly Black disadvantaged neighborhoods in the northern part, and similar results were produced for core clusters by the two cluster techniques with positive global Moran’s I for all study years. Although accounting for only 17% of the county population, nearly half of violent crime arrests were for Black or African American. Males comprised most violent crime arrests.ConclusionsCrime prevention and intervention efforts should be focused on both high-risk places and offenders.

Сравнительный анализ качества робастных модификаций метода главных компонент при сжатии коррелированных данных

Principal component analysis is one of the methods traditionally used to solve the problem of reducing the dimensionality of a multidimensional vector with correlated components. We constructed the principal components using a special representation of the covariance or correlation matrix of the indicators observed. The classical principal component analysis uses Pearson sample correlation coefficients as estimates of the correlation matrix elements. These estimates are extremely sensitive to sample contamination and anomalous observations. To robustify the principal component analysis, we propose to replace the sample estimates of correlation matrices with well-known robust analogues, which include Spearman's rank correlation coefficient, Minimum Covariance Determinant estimates, orthogonalized Gnanadesikan --- Kettenring estimates, and Olive --- Hawkins estimates. The study aims to carry out a comparative numerical analysis of the classical principal component analysis and its robust modifications. For this purpose, we simulated nine-dimensional vectors with known correlation matrix structures and introduced a special metric that allows us to evaluate the quality of data compression. Our extensive numerical experiment has shown that the classical principal component analysis boasts the best compression quality for a Gaussian distribution of observations. When observations are characterised by a Student's t-distribution with three degrees of freedom, as well as when a cluster of outliers, individual anomalous observations, or symmetric contaminations described by the Tukey distribution are present in the data, it is the Gnanadesikan --- Kettenring and Olive --- Hawkins estimates modifying the principal component analysis that show the best compression quality. The quality of the classical principal component analysis and Spearman’s rank modification decreases in these cases

ODED: Outlier Detection in Educational Data

Clustering data streams is one of the prominent tasks of discovering hidden patterns in data streams. It refers to the process of clustering newly arrived data into continuously and dynamically changing segmentation patterns. The current data stream clustering algorithms are lacking general clear steps for analysing new incoming data chunks. However, the majority of existing data stream solutions are adapting the clustering methods of static data to work with data stream setting. The main issue of concern is to propose a solution can improve the performance of existing approaches and present correct clusters and outliers. Data arriving in streams often contain outliers, which may have equal importance as clusters. Thus, it is desirable for data stream clustering algorithms to be able to detect the outliers as well as the clusters. The data stream clustering algorithms should be able to minimise the effects of noise and outliers data in a given dataset. This article presents a stream mining algorithm to cluster the data stream and monitor its evolution. Even though outlier detection is expected to be present in data streams, explicit outlier detection is rarely done in stream clustering algorithms. The proposed method is capable of explicit outlier detection and cluster evolution analysis. Relationship between outlier detection and the occurrence of physical events has been studied by applying the algorithm on the education data stream. Experiments led to the conclusion that the outlier detection accompanied by a change in the number of clusters indicates a significant education event. This kind of online monitoring and its results can be utilized in education systems in various ways. Viber education data streams produced by Viber groups are used to conduct this study.

Threshold-Based Hierarchical Clustering for Person Re-Identification.

Unsupervised domain adaptation is a challenging task in person re-identification (re-ID). Recently, cluster-based methods achieve good performance; clustering and training are two important phases in these methods. For clustering, one major issue of existing methods is that they do not fully exploit the information in outliers by either discarding outliers in clusters or simply merging outliers. For training, existing methods only use source features for pretraining and target features for fine-tuning and do not make full use of all valuable information in source datasets and target datasets. To solve these problems, we propose a Threshold-based Hierarchical clustering method with Contrastive loss (THC). There are two features of THC: (1) it regards outliers as single-sample clusters to participate in training. It well preserves the information in outliers without setting cluster number and combines advantages of existing clustering methods; (2) it uses contrastive loss to make full use of all valuable information, including source-class centroids, target-cluster centroids and single-sample clusters, thus achieving better performance. We conduct extensive experiments on Market-1501, DukeMTMC-reID and MSMT17. Results show our method achieves state of the art.

Cross domain-based ontology construction via Jaccard Semantic Similarity with hybrid optimization model

Semantic web technology seems to be in the infant stage as only little efforts have been taken on ontology construction with cross-domain application. This paper intends to take an effort on a new workspace, in which the ontology construction model under cross-domain application is performed. The core concern of this work is on two decision-making process namely data filtering and data annotation. Certain process is followed in this work: (i) Preprocessing (ii) Proposed Jaccard Similarity Evaluation (iii) Data filtering and Outlier Detection (iv) Semantic annotation and clustering. More particularly, data filtering is performed based on the evaluated similarity function. The outliers are identified and grouped separately. The data annotation is performed based on the semantics and thereby the clustering process takes place to form the ontology precisely. This clustering process obviously relies to the optimization crisis as the optimal centroid selection becomes the greatest issue. In order to solve this, this paper extends with the introduction of a hybrid algorithm named Circling Insisted-Rider Optimization Algorithm (CI-ROA), which hybrids the concept of Whale Optimization Algorithm (WOA) and Rider Optimization Algorithm (ROA), respectively. Finally, the performance of proposed work is compared and proved over other state-of-the-art models.

Improved approaches for density-based outlier detection in wireless sensor networks

Density-based algorithms are important data clustering techniques used to find arbitrary shaped clusters and outliers. Recently, outlier detectors through density-based clustering are applied to supervise data streams including wireless sensor networks (WSN’s). In this article, we compare two density-based methods, DBSCAN and OPTICS, using proposed configuration and specific classifier to identify outlier and normal clusters. For simulation, in MATLAB, we use real data of WSN’s from Intel Berkeley lab in that we introduce white Gaussian noise for different signal-to-noise ratio per data vector. We evaluate the two algorithms under different input parameters using several performance metrics as detection rate, false alarm rate. Results indicate that the DBSCAN scheme is more accurate and comprehensive compared with existing approaches for WSN’s. At the same time, OPTICS remains an interesting solution for a hierarchical study of datasets with an identification of anomalies.

A Fast Audit Doubt Finding Model

In order to improve the audit efficiency and accuracy under the condition of big data, an audit doubt discovery model (CLOWF) based on an adaptive clustering outlier detection algorithm is proposed. The model first uses the slope ratio method to obtain the K value of the K-means++ algorithm, thereby effectively ensuring the clustering effect; At the same time, for the problem of slow pruning speed, a parallel calculation method for the data radius and centroid of each cluster is proposed and designed; Combined with the weighted local outlier factor detection algorithm to calculate the outlier factor, the data greater than the epsilon threshold of outlier is proposed as an indicator to determine the audit doubt. Through auditing application cases, comparing the local outlier factor detection algorithm and the cluster-based outlier detection algorithm, the results show that the execution time of the CLOWF model is the shortest and the accuracy rate is 97.3%.

Stereo Matching with Spatiotemporal Disparity Refinement Using Simple Linear Iterative Clustering Segmentation

Stereo matching is a challenging problem, especially for computer vision, e.g., three-dimensional television (3DTV) or 3D visualization. The disparity maps from the video streams must be estimated. However, the estimated disparity sequences may cause undesirable flickering errors. These errors result in poor visual quality for the synthesized video and reduce the video coding information. In order to solve this problem, we here propose a spatiotemporal disparity refinement method for local stereo matching using the simple linear iterative clustering (SLIC) segmentation strategy, outlier detection, and refinements of the temporal and spatial domains. In the outlier detection, the segmented region in the initial disparity is used to distinguish errors in the binocular disparity. Based on the color similarity and disparity difference, we recalculate the aggregated cost to determine adaptive disparities to recover the disparity errors in disparity sequences. The flickering errors are also effectively removed, and the object boundaries are well preserved. Experiments using public datasets demonstrated that our proposed method creates high-quality disparity maps and obtains a high peak signal-to-noise ratio compared to state-of-the-art methods.

Robust Incremental Outlier Detection Approach Based on a New Metric in Data Streams

Detecting outliers in real time from multivariate streaming data is a vital and challenging research topic in many areas. Recently introduced the incremental Local Outlier Factor (iLOF) approach and its variants have received considerable attention as they achieve high detection performance in data streams with varying distributions. However, these iLOF-based approaches still have some major limitations: i) Poor detection in high-dimensional data; ii) The difficulty of determining the proper nearest neighbor number k; iii) Instead of labeling the outlier, assigning a score to each sample that indicates the probability to be an outlier; iv) Inability to detect a long sequence (small cluster) of outliers. This article proposes a new robust outlier detection method (RiLOF) based on iLOF that can effectively overcome these limitations. In the RiLOF method, a novel metric called Median of Nearest Neighborhood Absolute Deviation (MoNNAD) has been developed that uses the median of the local absolute deviation of the samples LOF values. Unlike the previously reported LOF-based approaches, RiLOF is capable of achieving outlier detection in different data stream applications using the same hyperparameters. Extensive experiments performed on 15 different real-world data sets demonstrate that RiLOF remarkably outperforms 12 different state-of-the-art competitors.

Predict high-frequency trading marker via manifold learning

High-frequency trading (HFT) has continuously evolved and dominated trading in the past decades. However, HFT trading marker prediction is a rarely investigated problem in Fintech literature. In this study, we first propose a novel manifold learning based HFT trading marker prediction model: M-SCAN to handle this challenge. Our study takes advantage of manifold embeddings of HFT data and seeks potential markers among the outliers in density-based clustering. We further propose HFT trading marker evaluation algorithms to validate the prediction effectiveness besides unveiling trading marker discovery via visualization. Our results demonstrate locally linear embedding (LLE) outperforms its peers in capturing trading markers in terms of accuracy and complexity for its local data structure keeping mechanism in embedding though different stocks may demonstrate their model preference under M-SCAN. Our studies also propose novel entropy and variance concentration ratios (VCR) to quantify HFT data and show that a high-entropy dataset is more likely to have better trading marker prediction than a low-entropy one. To the best of our knowledge, it is the first study in HFT trading marker prediction and will inspire coming studies in this area.

Comparing Genomic Signatures of Selection Between the Abbassa Strain and Eight Wild Populations of Nile Tilapia (Oreochromis niloticus) in Egypt.

Domestication to captive rearing conditions, along with targeted selective breeding have genetic consequences that vary from those in wild environments. Nile tilapia (Oreochromis niloticus) is one of the most translocated and farmed aquaculture species globally, farmed throughout Asia, North and South America, and its African native range. In Egypt, a breeding program established the Abbassa Strain of Nile tilapia (AS) in 2002 based on local broodstock sourced from the Nile River. The AS has been intensively selected for growth and has gone through genetic bottlenecks which have likely shifted levels and composition of genetic diversity within the strain. Consequently, there are questions on the possible genetic impact AS escapees may have on endemic populations of Nile tilapia. However, to date there have been no genetic studies comparing genetic changes in the domesticated AS to local wild populations. This study used 9,827 genome-wide SNPs to investigate population genetic structure and signatures of selection in the AS (generations 9–11) and eight wild Nile tilapia populations from Egypt. SNP analyses identified two major genetic clusters (captive and wild populations), with wild populations showing evidence of isolation-by-distance among the Nile Delta and upstream riverine populations. Between genetic clusters, approximately 6.9% of SNPs were identified as outliers with outliers identified on all 22 O. niloticus chromosomes. A lack of localized outlier clustering on the genome suggests that no genes of major effect were presently detected. The AS has retained high levels of genetic diversity (Ho_All = 0.21 ± 0.01; He_All = 0.23 ± 0.01) when compared to wild populations (Ho_All = 0.18 ± 0.01; He_All = 0.17 ± 0.01) after 11 years of domestication and selective breeding. Additionally, 565 SNPs were unique within the AS line. While these private SNPs may be due to domestication signals or founder effects, it is suspected that introgression with blue tilapia (Oreochromis aureus) has occurred. This study highlights the importance of understanding the effects of domestication in addition to wild population structure to inform future management and dissemination decisions. Furthermore, by conducting a baseline genetic study of wild populations prior to the dissemination of a domestic line, the effects of aquaculture on these populations can be monitored over time.