Outlier Cluster Research Articles

Numerical Considerations and a new implementation for invariant coordinate selection

Invariant coordinate selection (ICS) is a multivariate data transformation and a dimension reduction method that can be useful in many different contexts. It can be used for outlier detection or cluster identification, and can be seen as an independent component or a non-Gaussian component analysis method. The usual implementation of ICS is based on a joint diagonalization of two scatter matrices, and may be numerically unstable in some ill-conditioned situations. We focus on one-step M-scatter matrices and propose a new implementation of ICS based on a pivoted QR factorization of the centered data set. This factorization avoids the direct computation of the scatter matrices and their inverse and brings numerical stability to the algorithm. Furthermore, the row and column pivoting leads to a rank revealing procedure that allows computation of ICS when the scatter matrices are not full rank. Several artificial and real data sets illustrate the interest of using the new implementation compared to the original one.

A New Outlier Detection Algorithm Based on Fast Density Peak Clustering Outlier Factor

Outlier detection is an important field in data mining, which can be used in fraud detection, fault detection, and other fields. This article focuses on the problem where the density peak clustering algorithm needs a manual parameter setting and time complexity is high; the first is to use the k nearest neighbors clustering algorithm to replace the density peak of the density estimate, which adopts the KD-Tree index data structure calculation of data objects k close neighbors. Then it adopts the method of the product of density and distance automatic selection of clustering centers. In addition, the central relative distance and fast density peak clustering outliers were defined to characterize the degree of outliers of data objects. Then, based on fast density peak clustering outliers, an outlier detection algorithm was devised. Experiments on artificial and real data sets are performed to validate the algorithm, and the validity and time efficiency of the proposed algorithm are validated when compared to several conventional and innovative algorithms.

Experimental Analysis of Fuzzy Clustering Techniques for Outlier Detection

Fuzzy clustering is an effective clustering approach that divides the dataset into fuzzy segments. Fuzzy clustering and outlier detection are two interconnected processes. Outlier detection is essential because it discloses hidden patterns and crucial information about a dataset and is beneficial in a wide range of applications like fraud detection, military surveillance, identifying computer network intrusions, image processing, insurance or health care, etc. Many fuzzy clustering techniques are robust to anomalies since they limit the outlier's impact on the cluster's centroid.In this paper, we have compared four fuzzy clustering techniques namely Fuzzy C-means (FCM), Noise Clustering (NC), Credibilistic Fuzzy C-means (CFCM), and Density Oriented Fuzzy C - Means (DOFCM) on the basis of some crucial properties that are essential for efficient outlier detection. To better evaluate the feasibility of all these techniques, we experimentally evaluated these techniques on six datasets (two Real and four Synthetic) in the presence of noise and outliers. The results of the comparative analysis show that DOFCM outperforms all other techniques in terms of outlier detection and cluster formation. This research will surely benefit any researcher willing to detect outliers in their study using fuzzy clustering or either implicitly or explicitly working on fuzzy clusters.

Study on the Spatial and Temporal Characteristics of Weibo Users’ Online Shopping Festival Concerns: A Case Study of “Double 11 Shopping Carnival”

Online shopping festivals have become powerful ways for major e-commerce platforms to conduct promotions. To explore the rule and hidden information of the change of users’ attention to online shopping festivals, it can provide scientific and effective marketing reference, and promote the sustainable development of online shopping festivals. This paper takes the “Double 11” online shopping festival as the research object. Based on the LDA+LSA+NMF integrated theme extraction model, kernel density analysis, grouping analysis and cluster outlier analysis in ArcGIS technology are adopted. We explore the evolution pattern of microblog users’ attention to the online shopping festival in the spatial and temporal dimensions. In the temporal dimension, the number of releases showed a stepwise increase, and concerns showed annual topic evolution and daily three-stage differential evolution. In the spatial dimension, the release volume showed a gradient decreasing trend from coastal to inland and from east to west, and concerns showed certain regional differences. The main contribution of this paper is to propose a funnel theme extraction model with multi-model integration, and provide a new perspective for the spatial research of online shopping festival based on ArcGIS spatial analysis technology.

Unsupervised outlier detection applied to SARS-CoV-2 nucleotide sequences can identify sequences of common variants and other variants of interest

As of June 2022, the GISAID database contains more than 11 million SARS-CoV-2 genomes, including several thousand nucleotide sequences for the most common variants such as delta or omicron. These SARS-CoV-2 strains have been collected from patients around the world since the beginning of the pandemic. We start by assessing the similarity of all pairs of nucleotide sequences using the Jaccard index and principal component analysis. As shown previously in the literature, an unsupervised cluster analysis applied to the SARS-CoV-2 genomes results in clusters of sequences according to certain characteristics such as their strain or their clade. Importantly, we observe that nucleotide sequences of common variants are often outliers in clusters of sequences stemming from variants identified earlier on during the pandemic. Motivated by this finding, we are interested in applying outlier detection to nucleotide sequences. We demonstrate that nucleotide sequences of common variants (such as alpha, delta, or omicron) can be identified solely based on a statistical outlier criterion. We argue that outlier detection might be a useful surveillance tool to identify emerging variants in real time as the pandemic progresses.

Spatial epidemiology of bacterial meningitis in the Upper West Region of Ghana: Analysis of disease surveillance data 2018-2020.

The use of a Geographic Information System in identifying meningitis hotspots in the Upper West Region (UWR) remains underutilized, making spatial targeting of meningitis hotspots difficult. We therefore utilized surveillance data enabled with GIS technology to target meningitis outbreaks in the UWR. Secondary data analysis was conducted in the study. The dynamics of bacterial meningitis in space and time were studied using epidemiological data from 2018 to 2020. Spot map and choropleths were used to depict the distribution of cases in the region. Moran's I statistics were used to assess spatial autocorrelation. Getis-Ord Gi*(d) and Anselin Local Moran's statistics were used to identify hotspots and spatial outliers within the study area. A Geographic Weighted Regression model was also used to examine how socio bio-climatic conditions influence the spread of meningitis. There were 1176 cases of bacterial meningitis, 118 deaths, and 1058 survivors between 2018 and 2020. Nandom municipality had the highest Attack Rate (AR) at 492/100,000 persons, followed by Nadowli-Kaleo district at 314/100,000 persons. Jirapa had the highest case fatality rate (CFR) at 17%. The spatio-temporal analysis showed spatial diffusion of meningitis prevalence from the western half of the UWR to the east with a significant number of hotpots and cluster outliers. Bacterial meningitis does not occur at random. Populations (10.9%) under sub-districts identified as hotspots are exceptionally at higher risk of outbreaks. Targeted interventions should be directed towards clustered hotspots, focusing on zones with low prevalence fenced off by high prevalence zones.

An Anomaly Detection Framework for Twitter Data

An anomaly indicates something unusual, related to detecting a sudden behavior change, and is also helpful in detecting irregular and malicious behavior. Anomaly detection identifies unusual events, suspicious objects, or observations that differ significantly from normal behavior or patterns. Discrepancies in data can be observed in different ways, such as outliers, standard deviation, and noise. Anomaly detection helps us understand the emergence of specific diseases based on health-related tweets. This paper aims to analyze tweets to detect the unusual emergence of healthcare-related tweets, especially pre-COVID-19 and during COVID-19. After pre-processing, this work collected more than 44 thousand tweets and performed topic modeling. Non-negative matrix factorization (NMF) and latent Dirichlet allocation (LDA) were deployed for topic modeling, and a query set was designed based on resultant topics. This query set was used for anomaly detection using a sentence transformer. K-means was also employed for clustering outlier tweets from the cleaned tweets based on similarity. Finally, an unusual cluster was selected to identify pandemic-like healthcare emergencies. Experimental results show that the proposed framework can detect a sudden rise of unusual tweets unrelated to regular tweets. The new framework was employed in two case studies for anomaly detection and performed with 78.57% and 70.19% accuracy.

An ensemble-based outlier detection method for clustered and local outliers with differential potential spread loss

Most unsupervised outlier detection methods from the global perspective cannot effectively detect local outliers when normal samples present clusters with different densities. Moreover, existing local outlier detectors are not sensitive to clustered outliers. This paper proposes an unsupervised outlier detection method with differential potential spread loss (DPSL) to simultaneously detect global, local, and clustered outliers. DPSL first subsamples the original dataset several times. For each subset, the samples are sequentially linked in reverse to construct the first-level potential chains according to their nearest neighbor relationship, whose starting points are called first-level potential peak points. Secondly, the lower-level potential peak points are extracted to build higher-level potential chains, and this process is performed iteratively until the global potential peak point is obtained. The potential spread loss of a sample, which measures its local anomaly degree, is defined based on the distance and the ratio of isolation radius between the sample and its potential peak point on the same potential chain. The potential circles are established according to the levels of potential peak points and determine the global anomaly degree of the samples. Finally, DPSL calculates the outlier scores of all samples by combining the potential spread losses and potential circle levels of the subset samples and integrates the results obtained by every subsampling. DPSL outperforms 11 typical unsupervised outlier detection methods on 12 artificial datasets with four types of clustered outlier distributions. Meanwhile, it performs well on other 12 artificial datasets with various local outliers and 35 real-world datasets.

Data-Driven Approach toward Long-Term Equipment Condition Assessment in Sterile Drug Product Manufacturing.

A two-stage data-driven methodology for long-term equipment condition assessment in drug product manufacturing is presented with a case study for a commercially operating aseptic filling line. The methodology leverages process monitoring data. Sensor measurements are partitioned using process information and maintenance schedules that are available on different databases. Data is processed to tackle heterogeneity in sources and formats. The data is cleaned to remove the effects of short-term variabilities and to enhance underlying long-term trends. Two approaches are presented for data analysis: first, anomaly detection using independent component analysis (ICA), where clusters of outliers are identified. The frequency and timing of such outliers yield important insights regarding maintenance schedules and actions. The second approach enables condition monitoring using principal component analysis (PCA). Long-term operational baselines are identified and shifts therein are linked with different process and equipment faults. This approach highlights the impact of equipment deterioration on shifting operational data baselines and shows the potential for the combined application of ICA and PCA for equipment condition monitoring. It can be applied within predictive maintenance applications where the installation of new specialized sensors is difficult, like in the pharmaceutical industry.

Indoor Wireless Multipaths Outlier Detection and Clustering

Wireless communication systems have grown and developed significantly in recent years to fulfill the growing demand for high data rates across a wireless medium. Channel models have been used to develop various sturdy wireless systems for indoor and outdoor applications, and these are simulated in the form of datasets. The presence of outliers in clusters has been a concern in datasets, as it affects the standard deviation and mean of the dataset which reduces the data accuracy. In this study, the outliers in the Cooperation in Science and Technology (COST) 2100 MIMO channel model dataset were shifted to the means of the clusters using the Mean Shift Outlier Detection method. Afterward, the data is clustered using simultaneous clustering and model selection matrix affinity (SCAMSMA). The Mean Shift Outlier Detection method identified 52 and 46 multipaths as outliers and improved the clustering accuracy of the indoor scenarios by 3.5% and 0.93%, respectively. It also increased the precision of the clustering based on the decrease in standard deviation of the Jaccard indices from 0.2435 to 0.1807 and 0.3038 to 0.2075.

Geospatial Analysis of Cervical Cancer Distribution in South Sulawesi Province

Background: Cervical cancer, which is classified as a non-communicable disease, is a health problem that is of global concern at this time.1 Indonesia ranks second in the highest number of cervical cancer cases in the world with 32,469 cases per year. 1 For this reason, optimization efforts are carried out to prevent the increase in the prevalence of cervical cancer patients in the Province of South Sulawesi.
 Objective: The purpose of this study was to make a geospatial analysis of the distribution of cervical cancer patients.
 Methods: Geospatial analysis using Global Moran's I and Local Moran's I.
 Result: The results of the geospatial analysis of the prevalence of cervical cancer in South Sulawesi Province show that in 2016 there were two spatial hotspot clusters (H-H), one coldspot spatial cluster (L-L), two spatial outlier clusters (H-L), and one spatial outlier cluster (L-H). In 2019, there were only two spatial hotspot clusters. Geospatial analysis of the prevalence of cervical cancer shows an increase in efforts to prevent cervical cancer from 2016 to 2019. However, there are still spatial hotspot clusters in 2019, especially in rural areas..
 Conclusion: The efforts to prevent cervical cancer need to be optimized, especially in rural areas, in the future.

Ponder: Enabling Balloon-Borne Based Solar Unmanned Aerial Vehicle's Take Off Diagnosis under Little Data.

Balloon-borne based solar unmanned aerial vehicle (short for BS-UAV) has been researched prevalently due to the promising application area of near-space (i.e., 20–100 km above the ground) and the advantages of taking off. However, BS-UAV encounters serious fault in its taking off phase. The fault in taking off hinders the development of BS-UAV and causes great loss to human property. Thus, timely diagnosing the running state of BS-UAV in taking off phase is of great importance. Unfortunately, due to lack of fault data in the taking off phase, timely diagnosing the running state becomes a key challenge. In this paper, we propose Ponder to diagnose the running state of BS-UAV in the taking off phase. The key idea of Ponder is to take full advantage of existing data and complement fault data first and then diagnose current states. First, we compress existing data into a low-dimensional space. Then, we cluster the low-dimensional data into normal and outlier clusters. Third, we generate fault data with different aggression at different clusters. Finally, we diagnose fault state for each sampling at the taking off phase. With three datasets collected on real-world flying at different times, we show that Ponder outperforms existing diagnosing methods. In addition, we demonstrate Ponder’s effectiveness over time. We also show the comparable overhead.

An Inconsistency Fault Diagnosis Method for Lithium-Ion Cells in the Battery Pack Based on Piecewise Dimensionality Reduction and Outlier Identification

Abstract The inconsistency of cells in the battery pack is one of the main causes of battery failure. In practical applications, the terminal voltage is an important parameter that is easy to obtain and can characterize the inconsistency of cells. In this paper, a fault diagnosis method based on piecewise dimensionality reduction and outlier identification is proposed according to the voltage inconsistency of cells in the battery pack. This method uses a piecewise aggregate approximation (PAA) algorithm with a shift factor to reduce the dimension of the cell voltage time series, after which a deletion mechanism is designed based on the clustering algorithm and outlier identification to calculate the clustering quality after deleting each cell, reflecting the deviate degree of each cell. In addition, a safety management strategy is designed based on the Z-score method, and an abnormality coefficient is set to evaluate the inconsistency of cells. The effectiveness of the proposed diagnosis method is verified by monitoring the voltage data of two real-world electric vehicles. The verification results show that the method can not only detect the inconsistency before the failure of the faulty cell in the battery pack in advance, but also reduce the risk of computational explosion caused by the voltage time series and accurately locate the faulty cell.

Patterns and Influencing Factors of Express Outlets in China

China’s express delivery industry has developed rapidly in the past decade, and the spatial distribution of express delivery outlets can reflect variations in regional development to a certain extent. Previous studies lacked point of interest (POI) data as the research object to analyze the limitations of express delivery outlets, and also lack a focus on the patterns that specifically occur in China. This study implemented spatial analysis methods such as nearest neighbor index, kernel density, and exploratory spatial data analysis to explore the spatial distribution patterns and characteristics of China’s express outlets, and used geographic detector factor detection to objectively analyze the factors that affect their distribution. The following are the main conclusions of this study: (1) in China, express delivery outlets are more abundant in the east and sparser in the west; (2) the local agglomeration of express outlets presents a concentric structure, and outlier clusters appeared in the northeast and central regions; (3) the distribution of the express outlets resulted from the combined action of multiple factors (e.g., population factors, etc.). Our study not only explores and analyzes the overall situation in China but also broadens the scope of express outlet research.

Spatial clustering of heroin-related overdose incidents: a case study in Cincinnati, Ohio

BackgroundDrug overdose is one of the top leading causes of accidental death in the U.S., largely due to the opioid epidemic. Although the opioid epidemic is a nationwide issue, it has not affected the nation uniformly.MethodsWe combined multiple data sources, including emergency medical service response, American Community Survey data, and health facilities datasets to analyze distributions of heroin-related overdose incidents in Cincinnati, Ohio at the census block group level. The Ripley’s K function and the local Moran’s I statistics were performed to examine geographic variation patterns in heroin-related overdose incidents within the study area. Then, conditional cluster maps were plotted to examine a relationship between heroin-related incident rates and sociodemographic characteristics of areas as well as the resources for opioid use disorder treatment.ResultsThe global spatial analysis indicated that there was a clustered pattern of heroin-related overdose incident rates at every distance across the study area. The univariate local spatial analysis identified 7 hot spot clusters, 27 cold spot clusters, and 1 outlier cluster. Conditional cluster maps showed characteristics of neighborhoods with high heroin overdose rates, such as a higher crime rate, a high percentage of the male, a high poverty level, a lower education level, and a lower income level. The hot spots in the Southwest areas of Cincinnati had longer distances to opioid treatment programs and buprenorphine prescribing physicians than the median, while the hot spots in the South-Central areas of the city had shorter distances to those health resources.ConclusionsOur study showed that the opioid epidemic disproportionately affected Cincinnati. Multi-phased spatial clustering models based on various data sources can be useful to identify areas that require more policy attention and targeted interventions to alleviate high heroin-related overdose rates.

Fast Algorithm for Contactless Partial Discharge Detection on Remote Gateway Device

Detection of the high impedance fault caused by vegetation is one of the biggest challenges brought by the usage of covered conductors in medium voltage overhead power lines. One of the accompanying events of long-term contact of vegetation with the XLPE insulation is partial discharge which damages the insulation. Current systems for the detection of partial discharge have two major problems: the price and difficult installation. In this paper, an approach for the detection of the partial discharge from the data collected by the antenna is described. This approach is focused on the small computational demand and low false positive rate. Thanks to the small computational requirements, it can be run on the remote gateway devices which are collecting the data from the antenna. It is composed of four steps: outlier detection, outlier clustering, feature extraction, and classification. It is shown that this approach greatly improves the detection rate and lowers false positives compared to the previous algorithm used for partial discharge detection based on the data from an antenna, making it fit to use in the production environment.

Robust unsupervised image categorization based on variational autoencoder with disentangled latent representations

Recently, deep generative models have been successfully applied to unsupervised clustering analyses, due to the model capabilities for learning good representations of the input data from a lower-dimensional latent space. In this work, we propose a robust deep generative clustering method based on a variational autoencoder (VAE) for unsupervised image categorization. The merits of our method can be summarized as follows. First, each latent representation generated by the encoder is disentangled into the cluster representation and generation representation, where the cluster representation is responsible for preserving the clustering information, while the generation representation is responsible for conserving the generation information. Thus, by only utilizing the cluster representation, we can improve the performance and efficiency of clustering tasks without interference from generating tasks. Second, a Student’s-t mixture model is adopted as the prior over the cluster representation to enhance the robustness of our method against clustering outliers. Third, we propose a biaugmentation module to promote the training stability for our model. In contrast with most of the existing deep generative clustering methods that require a pretraining step to stabilize the training process, our model is able to provide a stable training process through feature disentanglement and data augmentation. We validate the proposed robust deep generative clustering method through extensive experiments by comparing it with state-of-the-art methods on unsupervised image categorization.

An iterative approach to unsupervised outlier detection using ensemble method and distance-based data filtering

Outlier or anomaly detection is the process through which datum/data with different properties from the rest of the data is/are identified. Their importance lies in their use in various domains such as fraud detection, network intrusion detection, and spam filtering. In this paper, we introduce a new outlier detection algorithm based on an ensemble method and distance-based data filtering with an iterative approach to detect outliers in unlabeled data. The ensemble method is used to cluster the unlabeled data and to filter out potential isolated outliers from the same by iteratively using a cluster membership threshold until the Dunn index score for clustering is maximized. The distance-based data filtering, on the other hand, removes the potential outlier clusters from the post-clustered data based on a distance threshold using the Euclidean distance measure of each data point from the majority cluster as the filtering factor. The performance of our algorithm is evaluated by applying it to 10 real-world machine learning datasets. Finally, we compare the results of our algorithm to various supervised and unsupervised outlier detection algorithms using Precision@n and F-score evaluation metrics.

Detection of local and clustered outliers based on the density–distance decision graph

Outlier detection tasks refer to identifying the objects that have different characteristics from the normal observations. Most existing approaches detect outliers from the global perspective, which can effectively detect global outliers and most clustered outliers but cannot detect local outliers when the normal samples form clusters with different densities. The methods based on local outlier factors can effectively detect local outliers, but when the number of outliers increases, the more occurrences of clustered outliers will lead to the degeneration of the detection performance. We proposed an outlier detection method based on density–distance decision graph to detect local, global and clustered outliers simultaneously. Firstly, kernel density estimation and local reachable distance are combined to calculate the local density. The density ratio of the neighbors of an instance to itself is calculated as the degree of local outliers. Then, we propose a metric named density lifting distance as the degree of global outliers, which is calculated by the distance between k nearest neighbors with higher density of the instance and itself. The density ratio and density lift distance are combined to draw the density–distance decision graph, and the product of two metrics is calculated as the final outlier score. Comprehensive experiments were conducted on 8 synthetic datasets and 16 real-world datasets compared with 12 state-of-the-art methods. The results show that the proposed method works well when the samples form clusters with different densities as well as the percentage of outliers varies, and outperforms the state-of-the-art methods tested in terms of AUC.

The Relationship between the Mechanism of Sarcopenia and Exercise Based on Data Mining.

Due to the increasing prosperity of human life science and technology, many huge research results have been obtained, and the scientific research of molecular biology is developing rapidly. Therefore, the output of biological genome data has increased exponentially, which constitutes a huge amount of data analysis. The seemingly chaotic and massive amount of data information actually contains a large amount of data and information of great key scientific significance and value. Therefore, this kind of genomic data information not only contains the information content that describes the characteristics of human life but also contains the information content that can express the essence of the biological organism. It includes macroeconomic information that can reflect the basic structure and capabilities of living organisms and microinformation in related fields of molecular biology. This massive amount of genetic data is usually closely related to each other, can influence each other, and does not exist alone. In the article, the causes of uncertain data and the classification of uncertain data are introduced, and the basic concepts and related algorithms of data mining are explained. Focusing on the research and analysis of abnormal point detection and clustering algorithms in uncertain data mining technology, this paper solves the problem of how to obtain more diverse and accurate outlier detection and cluster analysis results in uncertain data. The results showed that whether it was related to obesity or not, the Lp(a) level of the sarcopenia group was significantly higher than that of the nonsarcopenia group. At the same time, the correlation analysis showed that ASM/height was negatively correlated with Lp(a). ASM/height is one of the criteria for diagnosing sarcoidosis, and it is also the core of the analysis. Among the 1956 tumor patients collected in this study, 432 had sarcopenia, accounting for 22.08%, and the incidence of sarcopenia in patients with gastrointestinal tumors increased.