Inherent Cluster Structure Research Articles

Joint Projected Fuzzy Neighborhood Preserving C-means Clustering with Local Adaptive Learning

Graph-based dimensionality reduction approaches are extensively applied for preserving the intrinsic structures of high-dimensional data. To capture discriminative information from data clusters that lack a clear manifold structure, previous studies have combined clustering algorithms with local manifold structure preservation. These methods assume that two neighbor data points share the same cluster label. However, the assumption may not hold for Fuzzy C-Means which may assign two nearby data points to different clusters depending on the distance between data points and cluster centroids. This inconsistency degrades the performance in clustering and dimensionality reduction. To tackle this issue, we propose a novel approach for dimensionality reduction, termed joint Projected Fuzzy Neighborhood Preserving C-means Clustering with Local Adaptive Learning (PFNPCM). PFNPCM learns cluster centroids, sparse membership grades, embedded subspaces, and structured graphs simultaneously and extracts both global and local structures of data. Through clustering data that preserves the local neighborhood structure in a low-dimensional space instead of the original one, PFNPCM can effectively capture both global and local information from the inherent manifold structure and cluster structure. The employment of ℓ2,1-norm as the distance metric in the clustering and the preservation of neighborhood structure further ensure the robustness of clustering results against noise or outliers. To achieve the consistency of the captured global and local discriminative information, PFNPCM adopts a novel fuzzy local similarity measure based on the manifold structure, and the manifold assumption thus holds when partitioning unlabeled samples into subgroups. Massive experiments on diverse well-known datasets demonstrate that PFNPCM is superior to some state-of-the-art methods.

Synchronization-based semi-supervised data streams classification with label evolution and extreme verification delay

The critical need for classifying streaming data arises from its widespread use in real-world industries, where analyzing continuous, dynamic, and evolving data streams accurately and promptly is essential for informed decision-making and gaining predictive insights. Existing research mainly focuses on abundant supervision, overlooking the scarcity and delayed availability of labels, which can vary in timing. Addressing this, the article introduces a new learning method that employs a synchronization-based core support extraction technique. This technique is designed to manage changing concepts and delayed partial labeling by extracting key data points that act as pseudo-labels. Thanks to the concept of synchronization, these extracted key data points accurately represent the inherent local cluster structure in an intuitive manner and better maintain the class structure. Consequently, these pseudo-labels are utilized for classifying future incoming data batches. Furthermore, the method incorporates a knowledge base to summarize and represent all incoming streaming data. Building upon this knowledge base, an ensemble model for classification and an efficient new class detector are proposed. Both operate in a local fashion to ensure robust learning, even in complex class distributions. Evaluations on benchmark datasets reveal a consistent performance lead, surpassing established algorithms by up to 10%, achieving state-of-the-art results.

M3W: Multistep Three-Way Clustering.

Three-way clustering has been an active research topic in the field of cluster analysis in recent years. Some efforts are focused on the technique due to its feasibility and rationality. We observe, however, that the existing three-way clustering algorithms struggle to obtain more information and limit the fault tolerance excessively. Moreover, although the one-step three-way allocation based on a pair of fixed, global thresholds is the most straightforward way to generate the three-way cluster representations, the clusters derived from a pair of global thresholds cannot exactly reveal the inherent clustering structure of the dataset, and the threshold values are often difficult to determine beforehand. Inspired by sequential three-way decisions, we propose an algorithm, called multistep three-way clustering (M3W), to address these issues. Specifically, we first use a progressive erosion strategy to construct a multilevel structure of data, so that lower levels (or external layers) can gather more available information from higher levels (or internal layers). Then, we further propose a multistep three-way allocation strategy, which sufficiently considers the neighborhood information of every eroded instance. We use the allocation strategy in combination with the multilevel structure to ensure that more information is gradually obtained to increase the probability of being assigned correctly, capturing adaptively the inherent clustering structure of the dataset. The proposed algorithm is compared with eight competitors using 18 benchmark datasets. Experimental results show that M3W achieves superior performance, verifying its advantages and effectiveness.

Multi-View Kernel Learning for Identification of Disease Genes.

Gene expression data sets and protein-protein interaction (PPI) networks are two heterogeneous data sources that have been extensively studied, due to their ability to capture the co-expression patterns among genes and their topological connections. Although they depict different traits of the data, both of them tend to group co-functional genes together. This phenomenon agrees with the basic assumption of multi-view kernel learning, according to which different views of the data contain a similar inherent cluster structure. Based on this inference, a new multi-view kernel learning based disease gene identification algorithm, termed as DiGId, is put forward. A novel multi-view kernel learning approach is proposed that aims to learn a consensus kernel, which efficiently captures the heterogeneous information of individual views as well as depicts the underlying inherent cluster structure. Some low-rank constraints are imposed on the learned multi-view kernel, so that it can effectively be partitioned into k or fewer clusters. The learned joint cluster structure is used to curate a set of potential disease genes. Moreover, a novel approach is put forward to quantify the importance of each view. In order to demonstrate the effectiveness of the proposed approach in capturing the relevant information depicted by individual views, an extensive analysis is performed on four different cancer-related gene expression data sets and PPI network, considering different similarity measures.

A new interpoint distance-based clustering algorithm using kernel density estimation

A novel nonparametric clustering algorithm is proposed using the interpoint distances between the members of the data to reveal the inherent clustering structure existing in the given set of data, where we apply the classical nonparametric univariate kernel density estimation method to the interpoint distances to estimate the density around a data member. Our clustering algorithm is simple in its formation and easy to apply resulting in well-defined clusters. The algorithm starts with objective selection of the initial cluster representative and always converges independently of this choice. The method finds the number of clusters itself and can be used irrespective of the nature of underlying data by using an appropriate interpoint distance measure. The cluster analysis can be carried out in any dimensional space with viability to high-dimensional use. The distributions of the data or their interpoint distances are not required to be known due to the design of our procedure, except the assumption that the interpoint distances possess a density function. Data study shows its effectiveness and superiority over the widely used clustering algorithms.

Responses of Salmonella biofilms to oxidizing biocides: Evidence of spatial clustering.

The spatial organization of biofilm bacterial communities can be influenced by several factors, including growth conditions and challenge with antimicrobials. Differential survival of clusters of cells within biofilms has been observed. In this work, we present a variety of methods to identify, quantify and statistically analyse clusters of live cells from images of two Salmonella strains with differential biofilm forming capacity exposed to three oxidizing biocides. With a support vector machine approach, we showed spatial separation between the two strains, and, using statistical testing and high-performance computing (HPC), we determined conditions which possess an inherent cluster structure. Our results indicate that there is a relationship between biocide potency and inherent biofilm formation capacity with the tendency to select for spatial clusters of survivors. There was no relationship between positions of clusters of live or dead cells within stressed biofilms. This work identifies an approach to robustly quantify clusters of physiologically distinct cells within biofilms and suggests work to understand how clusters form and survive is needed. SIGNIFICANCE STATEMENT: Control of biofilm growth remains a major challenge and there is considerable uncertainty about how bacteria respond to disinfection within a biofilm and how clustering of cells impacts survival. We have developed a methodological approach to identify and statistically analyse clusters of surviving cells in biofilms after biocide challenge. This approach can be used to understand bacterial behaviour within biofilms under stress and is widely applicable.

Structured subspace learning-induced symmetric nonnegative matrix factorization

Symmetric NMF (SNMF) is able to determine the inherent cluster structure with the constructed graph. However, the mapping between the empirically constructed similarity representation and the desired one may contain complex structural and hierarchical information, which is not easy to capture with single learning approaches. Then, we propose a novel Structured Subspace Learning-induced Symmetric Nonnegative Matrix Factorization (SSLSNMF) in this paper. Based on the similarity space to be learned, SSLSNMF further learns a latent subspace, which considers the global and local structure of the data. Since SSLSNMF is formulated in a semi-supervised way, the supervisory information with constraints of cannot-link and must-link is also utilized. To guarantee that the latent similarity subspace to be discriminative, the global and local structures of the data as well as structural consistencies for limited labels in a semi-supervised manner are learned simultaneously in the proposed framework. An effective alternating iterative algorithm is proposed with proved convergence. Experiments conducted on six benchmark datasets show that better results can be obtained compared with state-of-the-art methods.

Application of SKATER and Ward’s methods in grouping Indonesian provinces based on monthly expenditure per capita of food commodity groups

Clustering is a fundamental data mining instrument that intends to find inherent cluster structure in data. Spatial clustering methods are usually used to assess the demographic data characterization. This study aims to classify provinces in Indonesia based on monthly expenditure per capita according to food commodity groups by using Ward’s and Spatial ‘K’luster analysis by tree edge removal (SKATER) methods and to identify a better classification between the two methods. The variables of this research constitute percentages of expenditure per capita for 14 groups of food commodities of 34 provinces in Indonesia during March 2018. The results of the first analysis (excluding outliers) revealed that SKATER method produced standard deviation rasio of 0.236, better than Ward’s method that produced standard deviation rasio of 0.370. However, from the second analysis (including outliers), the outcomes showed that the Ward’s method generated standard deviation rasio of 0.170, better than SKATER method that delivered standard deviation rasio of 0.199. Moreover, it can be concluded that the second analysis is better than the first analysis because it produced smaller standard deviation ratios based on the Ward’s and SKATER methods contrasted with the first one.

Robust Generalized Fuzzy Systems Training From High-Dimensional Time-Series Data Using Local Structure Preserving PLS

Establishing fuzzy models from time-series data with predictive capabilities for numerical targets typically requires dimension reduction techniques to overcome the severe curse of dimensionality effects. Linear projection methods are promising candidates in this context as they—unlike nonlinear dimension reduction techniques—preserve interpretability of the resulting models. However, linear projections do not reveal the inherent (nonlinear, local) cluster structure of the data and are thus not ideally suited for the identification of fuzzy rule bases. To overcome this limitation, in this article, we present a new fuzzy modeling approach that combines generalized fuzzy systems modeling with a local structure preserving variant of partial least squares (PLS). In contrast to ordinary PLS, our approach maps a weighted (adjacency) graph on the directions associated with high covariance with the response in order to emphasize local data structures when constructing the latent variable (LV) space. This operates into the direction from which the (training of the) fuzzy model benefits, as therein local regions are represented by submodels in form of generalized Takagi-Sugeno (TS) fuzzy rules. The local structure preserving LV space is obtained by solving a new penalized objective function, which assures global optimality of the solutions by virtue of the specific properties of the Laplacian matrix. Local regions are characterized in two ways: through nearest neighbor points (assuming fixed local region sizes) and through density regions identified by clustering (achieving variable local region sizes). To establish a robust time-series-based forecast model, the training of a generalized TS fuzzy model is conducted in the LV space with reduced dimensionality. It is realized by an iterative robust version of Gen-Smart-EFS , allowing multiple passes over the complete datasets until convergence of the antecedent space. Consequent parameters are estimated by a fuzzily weighted elastic net approach, embedding a convex combination of ridge regression and Lasso to achieve robust solutions also in case of ill-posed problems and meeting the (more stable and interpretable) local learning spirit. The new approach is termed as LS-PLS-Fuzzy , short for local structure preserving PLS fuzzy regression . It was successfully evaluated on three real-world application scenarios including a total of 11 time-series-based prediction modeling problems with different proportions between the number of training samples and original input dimensionality. Our results show significantly improved model accuracy compared to related State-of-the-Art (SoA) modeling approaches, an alternative dimension reduction technique, and when using conventional PLS and/or nongeneralized fuzzy rules.

Cluster-preserving sampling from fully-dynamic streaming graphs

Current sampling techniques on graphs (i.e., network-structured data) mainly study static graphs and suppose that the whole graph is available at all times. However, a surge of graphs are becoming too large-scale and/or fully-dynamic (i.e., nodes and edges are added or deleted arbitrarily) to be handled with small memory footprint. Moreover, the intrinsic property (i.e., clustering structure) has been ignored and is not preserved well when the sampling performs. To solve these issues, we propose a Cluster-preserving Partially Induced Edge Sampling (CPIES) algorithm that dynamically keep up-to-date samples in an online fashion and preserve the inherent clustering structure in streaming graphs. The experiments on various synthetic and real-world graphs demonstrated that the proposed CPIES algorithm is capable of preserving the inherent clustering structure while sampling from the fully-dynamic streaming graphs, and performs better than other competing algorithms in cluster-related properties.

To cluster, or not to cluster: An analysis of clusterability methods

Clustering is an essential data mining tool that aims to discover inherent cluster structure in data. For most applications, applying clustering is only appropriate when cluster structure is present. As such, the study of clusterability, which evaluates whether data possesses such structure, is an integral part of cluster analysis. However, methods for evaluating clusterability vary radically, making it challenging to select a suitable measure. In this paper, we perform an extensive comparison of measures of clusterability and provide guidelines that clustering users can reference to select suitable measures for their applications.

Optimizing cluster structures with inner product induced norm based dissimilarity measures: Theoretical development and convergence analysis

Dissimilarity measures play a key role in exploring the inherent cluster structure of the data for any partitional clustering algorithm. Commonly used dissimilarity functions for clustering purpose are so far confined to the Euclidean, exponential and Mahalanobish distances. In this article we develop generalized algorithms to solve the partitional clustering problems formulated with a general class of Inner Product Induced Norm (IPIN) based dissimilarity measures. We provide an in-depth mathematical analysis of the underlying optimization framework and analytically address the issue of existence of a solution and its uniqueness. In absence of a closed form solution, we develop a fast stochastic gradient descent algorithm and the Minimization by Incremental Surrogate Optimization (MISO) algorithm (in case of constrained optimization) with exponential convergence rate to obtain the solution. We carry out a convergence analysis of the fuzzy and k-means clustering algorithms with the IPIN based dissimilarity measures and also establish how these algorithms guarantee convergence to a stationary point. In addition, we investigate the nature of the stationary point. Novelty of the paper lies in the introduction of a generalized class of divergence measures, development of fuzzy and k-means clustering algorithms with the general class of divergence measures and a thorough convergence analysis of the developed algorithms.

Exploring Data Sets for Clusters and Validating Single Clusters

Cluster analysis is often used to find clusters and algorithms are designed and tuned to find the “right” clusters. Instead of searching for the “best” clustering algorithm, we argue that a clear concept of what the aim of a cluster analysis is and a better understanding of the data – especially based on visualisations – can be more crucial than the search for the right algorithm. In this paper, we revisit a method called dynamic data assigning assessment clustering that was intended both to asses the inherent cluster structure in a data set as well as to find the clusters. Here we extend this algorithm for better visualisation of possible cluster structures and also to validate single clusters that were found by other algorithms. Although this new approach can help to identify clusters, it is supporting tool and not used as a clustering algorithm itself.

A ranking-based algorithm for detection of outliers in categorical data

Outlier detection being an important data mining problem has attracted a lot of research interest in the recent past. As a result, various methods for outlier detection have been developed particularly for dealing with numerical data, whereas categorical data needs some attention. Addressing this requirement, we propose a two-phase algorithm for detecting outliers in categorical data based on a novel definition of outliers. In the first phase, this algorithm explores a clustering of the given data, followed by the ranking phase for determining the set of most likely outliers. The proposed algorithm is expected to perform better as it can identify different types of outliers, employing two independent ranking schemes based on the attribute value frequencies and the inherent clustering structure in the given data. Unlike some existing methods, the computational complexity of this algorithm is not affected by the number of outliers to be detected. The efficacy of this algorithm is demonstrated through experiments on various public domain categorical data sets.

An Interactive Approach to Multiobjective Clustering of Gene Expression Patterns

Some recent studies have posed the problem of data clustering as a multiobjective optimization problem, where several cluster validity indices are simultaneously optimized to obtain tradeoff clustering solutions. A number of cluster validity index measures are available in the literature. However, none of the measures can perform equally well in all kinds of datasets. Depending on the dataset properties and its inherent clustering structure, different cluster validity measures perform differently. Therefore, it is important to find the best set of validity indices that should be optimized simultaneously to obtain good clustering results. In this paper, a novel interactive genetic algorithm-based multiobjective approach is proposed that simultaneously finds the clustering solution as well as evolves the set of validity measures that are to be optimized simultaneously. The proposed method interactively takes the input from the human decision maker (DM) during execution and adaptively learns from that input to obtain the final set of validity measures along with the final clustering result. The algorithm is applied for clustering real-life benchmark gene expression datasets and its performance is compared with that of several other existing clustering algorithms to demonstrate its effectiveness. The results indicate that the proposed method outperforms the other existing algorithms for all the datasets considered here.

Network statistical analysis in peer-to-peer application

The rich statistical characteristics in peer-to-peer (p2p) network have recently attracted much research interest. This paper reveals the internal network statistical characteristics in the user network and resource network, both of which are abstracted from the real application downloading logs. The two-segment degree and weight distribution of user nodes indicate the dynamic of p2p users, and the similar power-law distribution of resource nodes shows the popularity diversity. Furthermore, we found that these two networks have the inherent cluster structure, only minority of clusters contain a large number of nodes, and the majority have fewer nodes in it. In user network, users in the same cluster have similar file-sharing interest, in contrast to the different user interest between clusters; meanwhile, there are obvious correlations between different resource categories in resource clusters.

A Repetitive Branch-and-Bound Procedure for Minimum Within-Cluster Sums of Squares Partitioning.

Minimization of the within-cluster sums of squares (WCSS) is one of the most important optimization criteria in cluster analysis. Although cluster analysis modules in commercial software packages typically use heuristic methods for this criterion, optimal approaches can be computationally feasible for problems of modest size. This paper presents a new branch-and-bound algorithm for minimizing WCSS. Algorithmic enhancements include an effective reordering of objects and a repetitive solution approach that precludes the need for splitting the data set, while maintaining strong bounds throughout the solution process. The new algorithm provided optimal solutions for problems with up to 240 objects and eight well-separated clusters. Poorly separated problems with no inherent cluster structure were optimally solved for up to 60 objects and six clusters. The repetitive branch-and-bound algorithm was also successfully applied to three empirical data sets from the classification literature.

A Hierarchical Clustering Approach for Large Compound Libraries

A modified version of the k-means clustering algorithm was developed that is able to analyze large compound libraries. A distance threshold determined by plotting the sum of radii of leaf clusters was used as a termination criterion for the clustering process. Hierarchical trees were constructed that can be used to obtain an overview of the data distribution and inherent cluster structure. The approach is also applicable to ligand-based virtual screening with the aim to generate preferred screening collections or focused compound libraries. Retrospective analysis of two activity classes was performed: inhibitors of caspase 1 [interleukin 1 (IL1) cleaving enzyme, ICE] and glucocorticoid receptor ligands. The MDL Drug Data Report (MDDR) and Collection of Bioactive Reference Analogues (COBRA) databases served as the compound pool, for which binary trees were produced. Molecules were encoded by all Molecular Operating Environment 2D descriptors and topological pharmacophore atom types. Individual clusters were assessed for their purity and enrichment of actives belonging to the two ligand classes. Significant enrichment was observed in individual branches of the cluster tree. After clustering a combined database of MDDR, COBRA, and the SPECS catalog, it was possible to retrieve MDDR ICE inhibitors with new scaffolds using COBRA ICE inhibitors as seeds. A Java implementation of the clustering method is available via the Internet (http://www.modlab.de).

Detecting stable clusters using principal component analysis.

Clustering is one of the most commonly used tools in the analysis of gene expression data (1, 2) . The usage in grouping genes is based on the premise that co-expression is a result of co-regulation. It is thus a preliminary step in extracting gene networks and inference of gene function (3, 4) . Clustering of experiments can be used to discover novel phenotypic aspects of cells and tissues (3, 5, 6) , including sensitivity to drugs (7) , and can also detect artifacts of experimental conditions (8) . Clustering and its applications in biology are presented in greater detail in the chapter by Zhao and Karypis (see also (9) ). While we focus on gene expression data in this chapter, the methodology presented here is applicable for other types of data as well. Clustering is a form of unsupervised learning, i.e. no information on the class variable is assumed, and the objective is to find the “natural” groups in the data. However, most clustering algorithms generate a clustering even if the data has no inherent cluster structure, so external validation tools are required. Given a set of partitions of the data into an increasing number of clusters (e.g. by a hierarchical clustering algorithm, or k-means), such a validation tool will tell the user the number of clusters in the data (if any). Many methods have been proposed in the literature to address this problem (10–15) . Recent studies have shown the advantages of sampling-based methods (12, 14) . These methods are based on the idea that when a partition has captured the structure in the data, this partition should be stable with respect to perturbation of the data. Bittner et al. (16) used a similar approach to validate clusters representing gene expression of melanoma patients. The emergence of cluster structure depends on several choices: data representation and normalization, the choice of a similarity measure and clustering algorithm. In this chapter we extend the stability-based validation of cluster structure, and propose stability as a figure of merit that is useful for comparing clustering solutions, thus helping in making these choices. We use this framework to demonstrate the ability of Principal Component Analysis (PCA) to extract features relevant to the cluster structure. We use stability as a tool for simultaneously choosing the number of principal components and the number of clusters; we compare the performance of different similarity measures and normalization schemes. The approach is demonstrated through a case study of yeast gene expression data from Eisen et al. (1) . For yeast, a functional classification of a large number of genes is known, and we use this classification for validating the results produced by clustering. A method for comparing clustering solutions specifically applicable to gene expression data was introduced in (17) . However, it cannot be used to choose the number of clusters, and is not directly applicable in choosing the number of principal components. The results of clustering are easily corrupted by the addition of noise: even a few