Approximate Spectral Clustering Research Articles

Approximate spectral clustering using both reference vectors and topology of the network generated by growing neural gas.

Spectral clustering (SC) is one of the most popular clustering methods and often outperforms traditional clustering methods. SC uses the eigenvectors of a Laplacian matrix calculated from a similarity matrix of a dataset. SC has serious drawbacks: the significant increases in the time complexity derived from the computation of eigenvectors and the memory space complexity to store the similarity matrix. To address the issues, I develop a new approximate spectral clustering using the network generated by growing neural gas (GNG), called ASC with GNG in this study. ASC with GNG uses not only reference vectors for vector quantization but also the topology of the network for extraction of the topological relationship between data points in a dataset. ASC with GNG calculates the similarity matrix from both the reference vectors and the topology of the network generated by GNG. Using the network generated from a dataset by GNG, ASC with GNG achieves to reduce the computational and space complexities and improve clustering quality. In this study, I demonstrate that ASC with GNG effectively reduces the computational time. Moreover, this study shows that ASC with GNG provides equal to or better clustering performance than SC.

Refining a [formula omitted]-nearest neighbor graph for a computationally efficient spectral clustering

Spectral clustering became a popular choice for data clustering for its ability of uncovering clusters of different shapes. However, it is not always preferable over other clustering methods due to its computational demands. One of the effective ways to bypass these computational demands is to perform spectral clustering on a subset of points (data representatives) then generalize the clustering outcome, this is known as approximate spectral clustering (ASC). ASC uses sampling or quantization to select data representatives. This makes it vulnerable to 1) performance inconsistency (since these methods have a random step either in initialization or training), 2) local statistics loss (because the pairwise similarities are extracted from data representatives instead of data points). We proposed a refined version of k-nearest neighbor graph, in which we keep data points and aggressively reduce number of edges for computational efficiency. Local statistics were exploited to keep the edges that do not violate the intra-cluster distances and nullify all other edges in the k-nearest neighbor graph. We also introduced an optional step to automatically select the number of clusters C. The proposed method was tested on synthetic and real datasets. Compared to ASC methods, the proposed method delivered a consistent performance despite significant reduction of edges.

A Novel Approximate Spectral Clustering Algorithm With Dense Cores and Density Peaks

Spectral clustering is becoming more and more popular because it has good performance in discovering clusters with varying characteristics. However, it suffers from high computational cost, unstable clustering results and noises. This work presents a novel approximate spectral clustering based on dense cores and density peaks, called DCDP-ASC. It first finds a reduced data set by introducing the concept of dense cores; then defines a new distance based on the common neighborhood of dense cores and calculates geodesic distances between dense cores according to the new defined distance; after that constructs a decision graph with a parameter-free local density and geodesic distance for obtaining initial centers; finally calculates the similarity between dense cores with their new defined geodesic distance, employs normalized spectral clustering method to divide dense cores, and expands the result on dense cores to the whole data set by assigning each point to its representative. The results on some challenging data sets and the comparison of our algorithm with some other excellent methods demonstrate that the proposed method DCDP-ASC is more advantageous in identifying complex structured clusters containing a lot of noises.

Optimization of Mining–Mineral Processing Integration Using Unsupervised Machine Learning Algorithms

Traditional ore-waste discrimination schemes often do not take into consideration the impact of fluctuations of the head grade, which have on the performance of mineral processing facilities. This research introduces the use of target grades for processing destinations as an alternative to cut off grade-based methods and models, in each processing destination, the losses due to deviation from targets via the Taguchi loss function. Three unsupervised learning algorithms, k-means clustering, CLARA and k-mean-based approximate spectral clustering, are presented to group mine planning blocks into clusters of similar grades with different processing destinations. In addition, a technique considering uncertainties associated with block grades is proposed to generate new sequences that reduce variation in processing capacities across the life of mine (LoM). The case study in this paper involves the treatment of a realistically large mining dataset. The results showed that clustering methods outperform cutoff grade-based method when divergence from target grades is penalized and that reclassification of blocks based on data from geostatistical simulations could achieve smoother capacities for processing streams across the LoM.

Approximate spectral clustering density-based similarity for noisy datasets

Approximate spectral clustering (ASC) was developed to overcome heavy computational demands of spectral clustering (SC). It maintains SC ability in predicting non-convex clusters. Since it involves a preprocessing step, ASC defines new similarity measures to assign weights on graph edges. Connectivity matrix (CONN) is an efficient similarity measure to construct graphs for ASC. It defines the weight between two vertices as the number of points assigned to them during vector quantization training. However, this relationship is undirected, where it is not clear which of the vertices is contributing more to that edge. Also, CONN could be tricked by noisy density between clusters. We defined a directed version of CONN, named DCONN, to get insights on vertices contributions to edges. Also, we provided filtering schemes to ensure CONN edges are highlighting potential clusters. Experiments reveal that the proposed filtering was highly efficient when noise cannot be tolerated by CONN.

A general framework for scalable spectral clustering based on document models

We propose a new, general framework for fast, approximate spectral clustering on large data sets. We first consider the special setting of cosine similarity for clustering sparse data (e.g., documents under the bag-of-words model) or data of at most a few hundred dimensions (e.g., small images). We show that in those cases various versions of spectral clustering, such as the Ng-Jordan-Weiss algorithm (NIPS 2001), Normalized Cut (Shi and Malik, 2000), and Diffusion Maps (Coifman and Lafon, 2006), can be implemented solely based on three kinds of efficient operations on the data matrix – elementwise manipulation, matrix-vector multiplication and low-rank SVD, thus eliminating the need to compute the weight matrix. For general similarity and any kind of data, we present a landmark-based technique that first converts the given data (or a landmark set selected from them) to a collection of “documents” and then applies to them the scalable implementation of spectral clustering with cosine similarity. Our algorithm is simple to implement and fast to run, with additional benefits such as a naturally embedded outliers removal step. We conduct extensive experiments to compare our algorithm with a few existing methods to demonstrate its superior performance.

Approximate spectral clustering with eigenvector selection and self-tuned k

The recently emerged spectral clustering surpasses conventional clustering methods by detecting clusters of any shape without the convexity assumption. Unfortunately, with a computational complexity of O(n3), it was infeasible for multiple real applications, where n could be large. This stimulates researchers to propose the approximate spectral clustering (ASC). However, most of ASC methods assumed that the number of clusters k was known. In practice, manual setting of k could be subjective or time consuming. The proposed algorithm has two relevance metrics for estimating k in two vital steps of ASC. One for selecting the eigenvectors spanning the embedding space, and the other to discover the number of clusters in that space. The algorithm used a growing neural gas (GNG) approximation, GNG is superior in preserving input data topology. The experimental setup demonstrates the efficiency of the proposed algorithm and its ability to compete with similar methods where k was set manually.

Approximate sparse spectral clustering based on local information maintenance for hyperspectral image classification.

Sparse spectral clustering (SSC) has become one of the most popular clustering approaches in recent years. However, its high computational complexity prevents its application to large-scale datasets such as hyperspectral images (HSIs). In this paper, we propose two efficient approximate sparse spectral clustering methods for HSIs clustering in which clustering performance is improved by utilizing local information among the data. Firstly, we construct a smaller representative dataset on which sparse spectral clustering is performed. Then the labels of ground object are extending to whole dataset based on the local information according to two extending strategies. The first one is that the local interpolation is utilized to improve the extension of the clustering result. The other one is that the label extension is turned to a problem of subspace embedding, and is fulfilled by locally linear embedding (LLE). Several experiments on HSIs demonstrated that the proposed algorithms are effective for HSIs clustering.

A semi-supervised approximate spectral clustering algorithm based on HMRF model

Before clustering, we usually have some background knowledge about the data structure. Pairwise constraints are commonly used background knowledge. For graph partition problems, pairwise constraints can be naturally added to the graph edge. This paper integrates pairwise constraints into the objective function of graph cuts and derive the semi-supervised approximate spectral clustering based on Hidden Markov Random Fields (HMRF). This algorithm utilize the mathematical connection between HMRF semi-supervised clustering and approximate weighted kernel k-means. The approximate weighted kernel k-means is used to calculate the optimal clustering results of HMRF spectral clustering. The effectiveness of the proposed algorithm is verified on several benchmark data sets. Experiments show that adding more pairwise constraints will help improve the clustering performance. Our method has advantages for the challenging clustering tasks of large-scale nonlinear data because of the high efficiency and less memory consumption.

An Approximate Spectral Clustering Ensemble for High Spatial Resolution Remote-Sensing Images

Unsupervised clustering of high spatial resolution remote-sensing images plays a significant role in detailed land-cover identification, especially for agricultural and environmental monitoring. A recently promising method is approximate spectral clustering (SC) which enables spectral partitioning for large datasets to extract clusters with distinct characteristics without a parametric model. It also facilitates the use of various information types via advanced similarity criteria. However, it requires an empirical selection of a similarity criterion optimal for the corresponding application. To address this challenge, we propose an approximate SC ensemble (ASCE2) which fuses partitionings obtained by different similarity representations. Contrary to existing spectral ensembles for remote-sensing applications, the proposed ASCE2 employs neural gas quantization instead of random sampling, advanced similarity criteria instead of traditional distance-based Gaussian kernel with different decay parameters, and a two-level ensemble. We evaluate the proposed ASCE2 with three measures (accuracy, adjusted Rand index, and normalized mutual information) using five remote-sensing images, two of which are commonly available. We apply the ASCE2 in two applications for agricultural monitoring: 1) land-cover identification to determine orchard fields using a WorldView-2 image (0.5-m spatial resolution) and 2) finding lands in good agricultural condition using multitemporal RapidEye images (5-m spatial resolution). Experimental results indicate a significant betterment of the resulting partitionings obtained by the proposed ensemble, with respect to the evaluation measures in these applications.

Approximate spectral clustering with utilized similarity information using geodesic based hybrid distance measures

Spectral clustering has been popular thanks to its ability to extract clusters of varying characteristics without using a parametric model in expense of high computational cost required for eigendecomposition of pairwise similarities. In order to utilize its advantages in large datasets where it is infeasible due to its computational burden, approximate spectral clustering (ASC) methods apply spectral clustering on a reduced set of points (data representatives) selected by sampling or quantization. This two-step approach (i.e. finding the representatives and then clustering them) brings new opportunities for precise similarity definition such as manifold based topological relations, data distribution within the Voronoi polyhedra of the representatives, and their geodesic distance information, which are often ignored in similarity definition for ASC. In this study, we propose geodesic based hybrid similarity criteria which enable the use of different types of information for accurate similarity representation in ASC. Despite the fact that geodesic concept has been widely used in clustering, our contribution is the unique way of representing data topology to form geodesic relations and jointly harnessing various information types including topology, distance and density. The proposed criteria are tested using both sampling (selective sampling) and quantization (neural gas and k-means++) approaches. Experiments on artificial datasets, well-known small/medium-size real datasets, and four large datasets (four remote-sensing images), with different types of clusters, show that the proposed geodesic based hybrid similarity criteria outperform traditional similarity criteria in terms of clustering accuracies and several cluster validity indices.

Efficient eigen-updating for spectral graph clustering

Partitioning a graph into groups of vertices such that those within each group are more densely connected than vertices assigned to different groups, known as graph clustering, is often used to gain insight into the organisation of large scale networks and for visualisation purposes. Whereas a large number of dedicated techniques have been recently proposed for static graphs, the design of on-line graph clustering methods tailored for evolving networks is a challenging problem, and much less documented in the literature. Motivated by the broad variety of applications concerned, ranging from the study of biological networks to the analysis of networks of scientific references through the exploration of communications networks such as the World Wide Web, it is the main purpose of this paper to introduce a novel, computationally efficient, approach to graph clustering in the evolutionary context. Namely, the method promoted in this article can be viewed as an incremental eigenvalue solution for the spectral clustering method described by Ng et al. (2001) [25]. The incremental eigenvalue solution is a general technique for finding the approximate eigenvectors of a symmetric matrix given a change. As well as outlining the approach in detail, we present a theoretical bound on the quality of the approximate eigenvectors using perturbation theory. We then derive a novel spectral clustering algorithm called Incremental Approximate Spectral Clustering (IASC). The IASC algorithm is simple to implement and its efficacy is demonstrated on both synthetic and real datasets modelling the evolution of a HIV epidemic, a citation network and the purchase history graph of an e-commerce website.

Local information-based fast approximate spectral clustering

Spectral clustering has become one of the most popular clustering approaches in recent years. However, its high computational complexity prevents its application to large-scale datasets. To address this complexity, approximate spectral clustering methods have been proposed. In these methods, computational costs are reduced by using approximation techniques, such as the Nyström method, or by constructing a smaller representative dataset on which spectral clustering is performed. However, the computational efficiency of these approximation methods is achieved at the cost of performance degradation. In this paper, we propose an efficient approximate spectral clustering method in which clustering performance is improved by utilizing local information among the data, while the scalability to the large-scale datasets is retained. Specifically, we improve the approximate spectral clustering method in two aspects. First, a sparse affinity graph is adopted to improve the performance of spectral clustering on the small representative dataset. Second, local interpolation is utilized to improve the extension of the clustering result. Experiments are conducted on several real-world datasets, showing that the proposed method is efficient and outperforms the state-of-the-art approximate spectral clustering algorithms.

A scalable approach to spectral clustering with SDD solvers

The promise of spectral clustering is that it can help detect complex shapes and intrinsic manifold structure in large and high dimensional spaces. The price for this promise is the expensive computational cost for computing the eigen-decomposition of the graph Laplacian matrix--so far a necessary subroutine for spectral clustering. In this paper we bypass the eigen-decomposition of the original Laplacian matrix by leveraging the recently introduced near-linear time solver for symmetric diagonally dominant (SDD) linear systems and random projection. Experiments on several synthetic and real datasets show that the proposed approach has better clustering quality and is faster than the state-of-the-art approximate spectral clustering methods.

An Approximate Spectral Clustering for Community Detection Based on Coarsening Networks

An Approximate Spectral Clustering for Community Detection Based on Coarsening Networks

Vector quantization based approximate spectral clustering of large datasets

Spectral partitioning, recently popular for unsupervised clustering, is infeasible for large datasets due to its computational complexity and memory requirement. Therefore, approximate spectral clustering of data representatives (selected by various sampling methods) was used. Alternatively, we propose to use neural networks (self-organizing maps and neural gas), which are shown successful in quantization with small distortion, as preliminary sampling for approximate spectral clustering (ASC). We show that they usually outperform k-means sampling (which was shown superior to various sampling methods), in terms of clustering accuracy obtained by ASC. More importantly, for quantization based ASC, we introduce a local density-based similarity measure – constructed without any user-set parameter – which achieves accuracies superior to the accuracies of commonly used distance based similarity.

Spectral Clustering: An empirical study of Approximation Algorithms and its Application to the Attrition Problem

Clustering is the problem of separating a set of objects into groups (called clusters) so that objects within the same cluster are more similar to each other than to those in different clusters. Spectral clustering is a now well-known method for clustering which utilizes the spectrum of the data similarity matrix to perform this separation. Since the method relies on solving an eigenvector problem, it is computationally expensive for large datasets. To overcome this constraint, approximation methods have been developed which aim to reduce running time while maintaining accurate classification. In this article, we summarize and experimentally evaluate several approximation methods for spectral clustering. From an applications standpoint, we employ spectral clustering to solve the so-called attrition problem, where one aims to identify from a set of employees those who are likely to voluntarily leave the company from those who are not. Our study sheds light on the empirical performance of existing approximate spectral clustering methods and shows the applicability of these methods in an important business optimization related problem.