Spectral Clustering Ensemble Research Articles

Auto-weighted Graph Reconstruction for efficient ensemble clustering

Ensemble clustering (EC) is an important field in data clustering, which aims to combine multiple base clustering results of a given dataset into a single consensus result. Previous studies have shown that one of the effective methods is to convert the base clustering results into a graph partitioning problem. Existing spectral ensemble clustering methods typically obtain a spectral embedding matrix for the graph and then perform k-means, separating the graph construction and clustering tasks into two distinct stages. To address these challenges, we propose a novel and efficient ensemble clustering method named Auto-weighted Graph Reconstruction for efficient ensemble clustering. This method integrates weighted base clusters, consensus distance graph construction, and clustering into a unified framework. A consensus distance graph is constructed, from which a stable and discrete label matrix is derived using spectral clustering. An effective optimization algorithm is then employed to address the resulting problem. Finally, the experimental results on eight real-world datasets verify the effectiveness and superiority of the proposed method.

Spectral ensemble clustering with doubly stochastic co-association matrix

Although co-association (CA) matrix-based spectral ensemble clustering (SEC) has achieved significant success, it faces two persistent challenges: (1) CA matrix-based SEC may result in imbalanced cluster outcomes, and (2) SEC encounters difficulties due to the high time and space complexity resulting from the quadratic scaling of the CA matrix with the number of data points. To tackle these issues, we propose a doubly stochastic co-association (DSCA) matrix construction method. We have theoretically demonstrated that DSCA matrix-based SEC can achieve a more balanced cluster outcome compared to CA matrix-based SEC. Leveraging the DSCA matrix, we introduce two novel methods: Fast SEC (FSEC) and Approximate SEC (ASEC). FSEC efficiently solves the normalized cut on the DSCA matrix as it is symmetric and doubly stochastic. In ASEC, we employ random samples rather than the entire big dataset for ensemble clustering. After obtaining the ensemble clustering result of benchmark random sample, the approximate ensemble clustering result for all data points is derived using our newly proposed probability nearest neighbors (PNN) algorithm. Experimental results on both real-world and synthetic datasets are evaluated in terms of accuracy (ACC), normalized mutual information (NMI), purity, F1-score, and adjusted rand index (ARI) and confirm the scalability and effectiveness of the proposed FSEC and ASEC.

Anchor-based fast spectral ensemble clustering

Ensemble clustering can obtain better and more robust results by fusing multiple base clusterings, which has received extensive attention. Although many representative algorithms have emerged in recent years, this field still has two tricky problems. First, spectral clustering can identify clusters of arbitrary shapes, but the high time and space complexity limit its application in generating base clusterings. Most existing algorithms utilize k-means to generate base clusterings, and the clustering effect on nonlinearly separable datasets needs further improvement. Second, ensemble clustering algorithms should generate multiple base clusterings. Even if low-complexity algorithms are applied, the running time is also long, which seriously affects the application of ensemble clustering algorithms on large-scale datasets. To tackle these problems, we propose a fast K-nearest neighbors approximation method, construct an anchor graph to approximate the similarity matrix, and use singular value decomposition (SVD) instead of eigenvalue decomposition (EVD) to reduce the time and space complexity of conventional spectral clustering. At the same time, we obtain multiple base clusterings by running spectral embedding once. Finally, we convert these base clusterings into a bipartite graph and use transfer cut to get the final clustering results. The proposed algorithms significantly reduce the running time of ensemble clustering. Experimental results on large-scale datasets fully prove the efficiency and superiority of our proposed algorithm.

LiteWSEC: A Lightweight Framework for Web-Scale Spectral Ensemble Clustering

Spectral Clustering (SC) is an effective clustering method for its excellent performance in partitioning non-linearly distributed data. On the other hand, Ensemble Clustering (EC), a different clustering technology, can promote cluster quality by ensembling the results of base clusterings. In this work, we concentrate on an EC framework that utilizes SC as the base method. Nevertheless, SC suffers from scalability due to its high computational complexity in constructing the Laplacian graph and computing the corresponding eigendecomposition. In the past decades, many efforts have been made to it. However, SC suffers from the scalability issue in processing extensive data, especially in web-scale scenarios. Additionally, EC requires multiple clustering results as the ensemble bases, which further aggravates resource consumption. To address this issue, LiteWSEC, a simple yet efficient Lightweight Framework for Web-scale Spectral Ensemble Clustering, is proposed to cluster web-scale data with limited resource requirements. It adopts the Web-scale Spectral Clustering (WSC) as the base method, which has minimal space overhead without computing overall embedding explicitly. LiteWSEC is highly flexible in the memory requirement, which is adaptive to the available resource. It can partition web-scale data (e.g., <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n = 8,000~k$</tex-math></inline-formula> ) in an resource-limited host (e.g., memory is restricted to 1 GB). Experiments on real-world, large-scale, and web-scale datasets demonstrate both the efficiency and effectiveness of LiteWSEC over state-of-the-art SC and EC methods.

Fuzzy-Rough induced spectral ensemble clustering

Ensemble clustering helps achieve fast clustering under abundant computing resources by constructing multiple base clusterings. Compared with the standard single clustering algorithm, ensemble clustering integrates the advantages of multiple clustering algorithms and has stronger robustness and applicability. Nevertheless, most ensemble clustering algorithms treat each base clustering result equally and ignore the difference of clusters. If a cluster in a base clustering is reliable/unreliable, it should play a critical/uncritical role in the ensemble process. Fuzzy-rough sets offer a high degree of flexibility in enabling the vagueness and imprecision present in real-valued data. In this paper, a novel fuzzy-rough induced spectral ensemble approach is proposed to improve the performance of clustering. Specifically, the significance of clusters is differentiated, and the unacceptable degree and reliability of clusters formed in base clustering are induced based on fuzzy-rough lower approximation. Based on defined cluster reliability, a new co-association matrix is generated to enhance the effect of diverse base clusterings. Finally, a novel consensus spectral function is defined by the constructed adjacency matrix, which can lead to significantly better results. Experimental results confirm that the proposed approach works effectively and outperforms many state-of-the-art ensemble clustering algorithms and base clustering, which illustrates the superiority of the novel algorithm.

LSEC: Large-scale spectral ensemble clustering

A fundamental problem in machine learning is ensemble clustering, that is, combining multiple base clusterings to obtain improved clustering result. However, most of the existing methods are unsuitable for large-scale ensemble clustering tasks owing to efficiency bottlenecks. In this paper, we propose a large-scale spectral ensemble clustering (LSEC) method to balance efficiency and effectiveness. In LSEC, a large-scale spectral clustering-based efficient ensemble generation framework is designed to generate various base clusterings with low computational complexity. Thereafter, all the base clusterings are combined using a bipartite graph partition-based consensus function to obtain improved consensus clustering results. The LSEC method achieves a lower computational complexity than most existing ensemble clustering methods. Experiments conducted on ten large-scale datasets demonstrate the efficiency and effectiveness of the LSEC method. The MATLAB code of the proposed method and experimental datasets are available at https://github.com/Li-Hongmin/MyPaperWithCode.

Scalable spectral ensemble clustering via building representative co-association matrix

Ensemble clustering via building co-association matrix and combining multiple basic partitions from the same dataset into the consensus one has been widely used in spectral clustering and subspace clustering. However, with the ever-increasing cost of calculating the co-association matrix, the conventional ensemble clustering algorithm is no longer fit for dealing with the large-scale datasets due to its less scalability and time-consuming. In this paper, we propose a scalable spectral ensemble clustering method via building a representative co-association matrix to improve the ensemble clustering problem. Our method mainly includes constructing a sparse matrix to select the representative points and building the co-association matrix, and a robust and denoising representation for the co-association matrix can be learned through a low-rank constraint in a unified optimization framework. The experiments verify the high efficiency and scalability but less time cost of our method compared with state-of-art clustering methods in the six real-world datasets, especially in the large-scale datasets.

Constraint projections for semi‐supervised spectral clustering ensemble

SummaryCluster ensemble combines multiple base clustering results in a suitable way to improve the accuracy of the clustering result. In the conventional cluster ensemble frameworks, pairwise constraints and constraint projections have not been used together, and spectral clustering algorithm is rarely adopted to serve as the consensus function. In this paper, we design a constraint projections for semi‐supervised spectral clustering ensemble (CPSSSCE) model. It takes advantages of spectral clustering algorithm and executes semi‐supervised learning twice. Compared to traditional cluster ensemble approaches, CPSSSCE is characterized by several properties. First, the original data are transformed to lower‐dimensional representations by constraint projection before base clustering. Second, a similarity matrix is constructed using the base clustering results and modified using pairwise constraints. Third, the spectral clustering algorithm is applied to process the similarity matrix to obtain a consensus cluster result. Extensive experiments on standard University of California Irvine Machine Learning Repository (UCI) and Microsoft datasets demonstrated that the CPSSSCE is superior to other cluster ensemble algorithms including a semi‐supervised spectral clustering ensemble.

Incorporating URL embedding into ensemble clustering to detect web anomalies

Web anomaly detection aims to find deviations from normal behaviour that happened in our system at most of the time. With the development of the Internet, it is vital for the security of the Internet to detect web-based anomalies. Clustering based on feature extraction by manually has been verified as a significant way to detect new anomalies. But the presentations of these features cannot express semantic information of the URLs. In addition, few studies try to cluster the anomalies into specific types like SQL-injection. In order to solve these two problems, we provide a weighted deep learning enabled subspace spectral ensemble clustering approach for web anomaly detection called WDL-SSEC. This approach has three steps. Firstly, an ensemble clustering model is applied to separate anomalies from normal samples. Then we use word2vec to get the semantical presentations of tokens and concatenate weighted tokens to get vectors of the URLs. Finally, another ensemble clustering based on subspace and locally adaptive clustering (LAC) multi-cluster anomalies into specific types. Our approach is run on a real-life data set. The results achieves better performance than existing approaches, , which demonstrates that our model has the ability to cluster anomalies into appropriate types.

Robust Spectral Ensemble Clustering via Rank Minimization

Ensemble Clustering (EC) is an important topic for data cluster analysis. It targets to integrate multiple Basic Partitions (BPs) of a particular dataset into a consensus partition. Among previous works, one promising and effective way is to transform EC as a graph partitioning problem on the co-association matrix, which is a pair-wise similarity matrix summarized by all the BPs in essence. However, most existing EC methods directly utilize the co-association matrix, yet without considering various noises (e.g., the disagreement between different BPs and the outliers) that may exist in it. These noises can impair the cluster structure of a co-association matrix, and thus mislead the final graph partitioning process. To address this challenge, we propose a novel Robust Spectral Ensemble Clustering (RSEC) algorithm in this article. Specifically, we learn low-rank representation (LRR) for the co-association matrix to uncover its cluster structure and handle the noises, and meanwhile, we perform spectral clustering with the learned representation to seek for a consensus partition. These two steps are jointly proceeded within a unified optimization framework. In particular, during the optimizing process, we leverage consensus partition to iteratively enhance the block-diagonal structure of LRR, in order to assist the graph partitioning. To solve RSEC, we first formulate it by using nuclear norm as a convex proxy to the rank function. Then, motivated by the recent advances in non-convex rank minimization, we further develop a non-convex model for RSEC and provide it a solution by the majorization--minimization Augmented Lagrange Multiplier algorithm. Experiments on 18 real-world datasets demonstrate the effectiveness of our algorithm compared with state-of-the-art methods. Moreover, several impact factors on the clustering performance of our approach are also explored extensively.

Spectral clustering ensemble model based on distance decision

Spectral clustering (SC) is a kind of clustering method which shows superior performance in recent years; however, the method is sensitive to the initial scale parameter. In order to improve the accuracy and stability of clustering, this Letter proposes a novel approach of SC ensemble model based on distance decision. First of all, the model performs several times clustering for data samples by SC, and corresponding joint Laplacian matrix is obtained. Then, a new method of distance decision is proposed, the joint Laplacian matrix is transformed into a cumulative distance matrix using the fuzzy theory. Finally, the distance matrix is introduced into the density peaks (DP) algorithm, and the final results of clustering are obtained by the improved DP algorithm for clustering ensemble. The experimental results show that the clustering ensemble model proposed in this Letter is more effective than other classical clustering ensemble model on the 12 data sets in University of California Irvine machine learning database.

Spectral Clustering with Eigenvalue Similarity Metric Method for POL-SAR Image Segmentation of Land Cover

A simple and fast approach based on eigenvalue similarity metric for Polarimetric SAR image segmentation of Land Cover is proposed in this paper. The approach uses eigenvalues of the coherency matrix as to construct similarity metric of clustering algorithm to segment SAR image. The Mahalanobis distance is used to metric pairwise similarity between pixels to avoid the manual scale parameter tuning in previous spectral clustering method. Furthermore, the spatial coherence constraints and spectral clustering ensemble are employed to stabilize and improve the segmentation performance. All experiments are carried out on three sets of Polarimetric SAR data. The experimental results show that the proposed method is superior to other comparison methods.

Spectral Ensemble Clustering via Weighted K-Means: Theoretical and Practical Evidence

As a promising way for heterogeneous data analytics, consensus clustering has attracted increasing attention in recent decades. Among various excellent solutions, the co-association matrix based methods form a landmark, which redefines consensus clustering as a graph partition problem. Nevertheless, the relatively high time and space complexities preclude it from wide real-life applications. We, therefore, propose Spectral Ensemble Clustering (SEC) to leverage the advantages of co-association matrix in information integration but run more efficiently. We disclose the theoretical equivalence between SEC and weighted K-means clustering, which dramatically reduces the algorithmic complexity. We also derive the latent consensus function of SEC, which to our best knowledge is the first to bridge co-association matrix based methods to the methods with explicit global objective functions. Further, we prove in theory that SEC holds the robustness, generalizability, and convergence properties. We finally extend SEC to meet the challenge arising from incomplete basic partitions, based on which a row-segmentation scheme for big data clustering is proposed. Experiments on various real-world data sets in both ensemble and multi-view clustering scenarios demonstrate the superiority of SEC to some state-of-the-art methods. In particular, SEC seems to be a promising candidate for big data clustering.

Fast Constrained Spectral Clustering and Cluster Ensemble with Random Projection.

Constrained spectral clustering (CSC) method can greatly improve the clustering accuracy with the incorporation of constraint information into spectral clustering and thus has been paid academic attention widely. In this paper, we propose a fast CSC algorithm via encoding landmark-based graph construction into a new CSC model and applying random sampling to decrease the data size after spectral embedding. Compared with the original model, the new algorithm has the similar results with the increase of its model size asymptotically; compared with the most efficient CSC algorithm known, the new algorithm runs faster and has a wider range of suitable data sets. Meanwhile, a scalable semisupervised cluster ensemble algorithm is also proposed via the combination of our fast CSC algorithm and dimensionality reduction with random projection in the process of spectral ensemble clustering. We demonstrate by presenting theoretical analysis and empirical results that the new cluster ensemble algorithm has advantages in terms of efficiency and effectiveness. Furthermore, the approximate preservation of random projection in clustering accuracy proved in the stage of consensus clustering is also suitable for the weighted k-means clustering and thus gives the theoretical guarantee to this special kind of k-means clustering where each point has its corresponding weight.

A Hybrid Spectral Clustering and Deep Neural Network Ensemble Algorithm for Intrusion Detection in Sensor Networks.

The development of intrusion detection systems (IDS) that are adapted to allow routers and network defence systems to detect malicious network traffic disguised as network protocols or normal access is a critical challenge. This paper proposes a novel approach called SCDNN, which combines spectral clustering (SC) and deep neural network (DNN) algorithms. First, the dataset is divided into k subsets based on sample similarity using cluster centres, as in SC. Next, the distance between data points in a testing set and the training set is measured based on similarity features and is fed into the deep neural network algorithm for intrusion detection. Six KDD-Cup99 and NSL-KDD datasets and a sensor network dataset were employed to test the performance of the model. These experimental results indicate that the SCDNN classifier not only performs better than backpropagation neural network (BPNN), support vector machine (SVM), random forest (RF) and Bayes tree models in detection accuracy and the types of abnormal attacks found. It also provides an effective tool of study and analysis of intrusion detection in large networks.

An Approximate Spectral Clustering Ensemble for High Spatial Resolution Remote-Sensing Images

Unsupervised clustering of high spatial resolution remote-sensing images plays a significant role in detailed land-cover identification, especially for agricultural and environmental monitoring. A recently promising method is approximate spectral clustering (SC) which enables spectral partitioning for large datasets to extract clusters with distinct characteristics without a parametric model. It also facilitates the use of various information types via advanced similarity criteria. However, it requires an empirical selection of a similarity criterion optimal for the corresponding application. To address this challenge, we propose an approximate SC ensemble (ASCE2) which fuses partitionings obtained by different similarity representations. Contrary to existing spectral ensembles for remote-sensing applications, the proposed ASCE2 employs neural gas quantization instead of random sampling, advanced similarity criteria instead of traditional distance-based Gaussian kernel with different decay parameters, and a two-level ensemble. We evaluate the proposed ASCE2 with three measures (accuracy, adjusted Rand index, and normalized mutual information) using five remote-sensing images, two of which are commonly available. We apply the ASCE2 in two applications for agricultural monitoring: 1) land-cover identification to determine orchard fields using a WorldView-2 image (0.5-m spatial resolution) and 2) finding lands in good agricultural condition using multitemporal RapidEye images (5-m spatial resolution). Experimental results indicate a significant betterment of the resulting partitionings obtained by the proposed ensemble, with respect to the evaluation measures in these applications.

An efficient automatic SAR image segmentation framework in AIS using kernel clustering index and histogram statistics

Artificial immune system (AIS) has been proven effective for pattern classification by its characteristics of learning and adaptability from the vertebrate immune system. However, little focus has been placed on the synthetic aperture radar (SAR) image segmentation by AIS. In this paper, we present an efficient automatic framework in AIS for SAR image segmentation. It aims at simultaneously solving the following three different crucial issues: (1) the automatic ability of searching true number of land-cover in SAR images; (2) the objective functions in guiding the segmentation of the images with complicated multiplicative noises; and (3) reduction of the computational complexity of segmenting SAR images with large sizes. By the proposed framework here, it can mitigate the above difficulties to a certain degree. Furthermore, a reasonable spatial filtering and watershed transformation are employed in the initial stage of the framework, and then a novel clustering index in Gaussian kernel is designed to lead the searching process, which is beneficial to find the partitions for the clustering problem with highly overlapping and contaminating samples. Besides, we also propose an efficient computing paradigm in AIS with variable length of chromosomes to search the optimal partitions of SAR images, which can find the optimal numbers of clusters automatically. Finally, in order to speed up the segmentation process of SAR image, we employ the histogram statistics to implement the pixels partition; therefore, the segmenting time is dependent on the small number of gray-levels, not the great amounts of whole image pixels. To test the segmentation performances of the proposed algorithm, a detailed experimental analysis was conducted on two simulated SAR images and four complicated real ones. Other four state-of-the-art image segmentation methods are employed for comparison, which are genetic clustering by variable string length encoding (VGA), fast generalized fuzzy C-means clustering (FGFCM), fuzzy local information C-means clustering algorithm (FLICM) and graph partitioning method: spectral clustering ensemble (SCE).

A robust adaptive clustering analysis method for automatic identification of clusters

Identifying the optimal cluster number and generating reliable clustering results are necessary but challenging tasks in cluster analysis. The effectiveness of clustering analysis relies not only on the assumption of cluster number but also on the clustering algorithm employed. This paper proposes a new clustering analysis method that identifies the desired cluster number and produces, at the same time, reliable clustering solutions. It first obtains many clustering results from a specific algorithm, such as Fuzzy C-Means (FCM), and then integrates these different results as a judgement matrix. An iterative graph-partitioning process is implemented to identify the desired cluster number and the final result. The proposed method is a robust approach as it is demonstrated its effectiveness in clustering 2D data sets and multi-dimensional real-world data sets of different shapes. The method is compared with cluster validity analysis and other methods such as spectral clustering and cluster ensemble methods. The method is also shown efficient in mesh segmentation applications. The proposed method is also adaptive because it not only works with the FCM algorithm but also other clustering methods like the k-means algorithm.

Adaptive spectral clustering ensemble selection via resampling and population-based incremental learning algorithm

In this paper, we explore a novel ensemble method for spectral clustering. In contrast to the traditional clustering ensemble methods that combine all the obtained clustering results, we propose the adaptive spectral clustering ensemble method to achieve a better clustering solution. This method can adaptively assess the number of the component members, which is not owned by many other algorithms. The component clusterings of the ensemble system are generated by spectral clustering (SC) which bears some good characteristics to engender the diverse committees. The selection process works by evaluating the generated component spectral clustering through resampling technique and population-based incremental learning algorithm (PBIL). Experimental results on UCI datasets demonstrate that the proposed algorithm can achieve better results compared with traditional clustering ensemble methods, especially when the number of component clusterings is large.

Bagging-based spectral clustering ensemble selection

Traditional clustering ensemble methods combine all obtained clustering results at hand. However, we can often achieve a better clustering solution if only parts of the clustering results available are combined. In this paper, we generalize the selective clustering ensemble algorithm proposed by Azimi and Fern and a novel clustering ensemble method, SELective Spectral Clustering Ensemble (SELSCE), is proposed. The component clusterings of the ensemble system are generated by spectral clustering (SC) capable of engendering diverse committees. The random scaling parameter, Nyström approximation are used to perturb SC for producing the components of the ensemble system. After the generation of component clusterings, the bagging technique, usually applied in supervised learning, is used to assess the component clustering. We randomly pick part of the available clusterings to get a consensus result and then compute normalized mutual information (NMI) or adjusted rand index (ARI) between the consensus result and the component clusterings. Finally, the components are ranked by aggregating multiple NMI or ARI values. The experimental results on UCI dataset and images demonstrate that the proposed algorithm can achieve a better result than the traditional clustering ensemble methods.