Cluster Indicator Matrix Research Articles

A novel data representation framework based on nonnegative manifold regularisation

ABSTRACT Representation learning techniques have been frequently applied in multimedia content analysis and retrieval. In this study, an efficient multimedia data clustering method is presented, which consists of two independent algorithms. First, we propose a new representation framework by incorporating sparse coding and manifold regularisation in an optimisation objective function, the cluster indicator matrix is estimated by introducing sparsity norm coarsely. Second, we refine the estimated cluster indicator matrix by performing spectral rotation such that an optimal assignment for clustering can be learned. Compared with existing methods, we have the following merits: our method takes into account the global matrix reconstruction information and locality manifold information simultaneously. Therefore, global and locality information both are respected. Additionally, theoretical justification about the novel representation method is presented in this study. Comprehensive experiments demonstrate the effectiveness and efficiency of our method in comparison with the state-of-the-art clustering methods on six real-world image datasets.

Multi-View Clustering in Latent Embedding Space

Previous multi-view clustering algorithms mostly partition the multi-view data in their original feature space, the efficacy of which heavily and implicitly relies on the quality of the original feature presentation. In light of this, this paper proposes a novel approach termed Multi-view Clustering in Latent Embedding Space (MCLES), which is able to cluster the multi-view data in a learned latent embedding space while simultaneously learning the global structure and the cluster indicator matrix in a unified optimization framework. Specifically, in our framework, a latent embedding representation is firstly discovered which can effectively exploit the complementary information from different views. The global structure learning is then performed based on the learned latent embedding representation. Further, the cluster indicator matrix can be acquired directly with the learned global structure. An alternating optimization scheme is introduced to solve the optimization problem. Extensive experiments conducted on several real-world multi-view datasets have demonstrated the superiority of our approach.

Enhanced Balanced Min Cut

Spectral clustering is a hot topic and many spectral clustering algorithms have been proposed. These algorithms usually solve the discrete cluster indicator matrix by relaxing the original problems, obtaining the continuous solution and finally obtaining a discrete solution that is close to the continuous solution. However, such methods often result in a non-optimal solution to the original problem since the different steps solve different problems. In this paper, we propose a novel spectral clustering method, named as Enhanced Balanced Min Cut (EBMC). In the new method, a new normalized cut model is proposed, in which a set of balance parameters are learned to capture the differences among different clusters. An iterative method with proved convergence is used to effectively solve the new model without eigendecomposition. Theoretical analysis reveals the connection between EBMC and the classical normalized cut. Extensive experimental results show the effectiveness and efficiency of our approach in comparison with the state-of-the-art methods.

Multi-View Data Fusion Oriented Clustering via Nuclear Norm Minimization.

Image clustering remains challenging when handling image data from heterogeneous sources. Fusing the independent and complementary information existing in heterogeneous sources together facilitates to improve the image clustering performance. To this end, we propose a joint learning framework of multi-view image data fusion and clustering based on nuclear norm minimization. Specifically, we first formulate the problem as matrix factorization to a shared clustering indicator matrix and a representative coefficient matrix. The former is constrained with orthogonality and nonnegativity, which ensures the validation of clustering assignments. The latter is imposed with nuclear norm minimization to achieve compression of principal components for performance improvement. Then, an alternating minimization strategy is employed to efficiently decompose the multi-variable optimization problem into several small solvable sub-problems with closed-form solutions. Extensive experimental results on real-world image and video datasets demonstrate the superiority of proposed method over other state-of-the-art methods.

A robust dimensionality reduction and matrix factorization framework for data clustering

Most existing Non-negative Matrix Factorization (NMF) related data clustering techniques directly decompose the original feature space while have not well considered the fact that the low dimensional feature space is always embedded in the high dimensional feature space and can better reveal the spatial distribution of data. In this letter, we propose a new matrix factorization model, which unites the objectives of clustering and dimensionality reduction simultaneously. In the proposed framework, the clustering based on matrix factorization is actually executed on the embedded subspace which may provide more accurate and reasonable solutions. Furthermore, we use the l2,1-norm instead of the conventional l2-norm to enhance the clustering results and make the clustering framework more robust to the noises and outliers. Meanwhile, in order to preserve as much as possible local similarity of the data, we have also employed an affinity matrix with special learning to introduce the manifold learning into the cluster indicator matrix. An optimization procedure based on Augmented Lagrangian Method (ALM) is devised to effectively solve the proposed problem and explicitly show the clustering results. Experimental results on the benchmark datasets with different proprieties exhibit the superior performance of the proposed method.

Auto-weighted multi-view clustering via kernelized graph learning

Datasets are often collected from different resources or comprised of multiple representations (i.e., views). Multi-view clustering aims to analyze the multi-view data in an unsupervised way. Owing to the efficiency of uncovering the hidden structures of data, graph-based approaches have been investigated widely for various multi-view learning tasks. However, similarity measurement in these methods is challenging since the construction of similarity graph is impacted by several factors such as the scale of data, neighborhood size, choice of similarity metric, noise and outliers. Moreover, nonlinear relationships usually exist in real-world datasets, which have not been considered by most existing methods. In order to address these challenges, a novel model which simultaneously performs multi-view clustering task and learns similarity relationships in kernel spaces is proposed in this paper. The target optimal graph can be directly partitioned into exact c connected components if there are c clusters. Furthermore, our model can assign ideal weight for each view automatically without additional parameters as previous methods do. Since the performance is often sensitive to the input kernel matrix, the proposed model is further extended with multiple kernel learning ability. With the proposed joint model, three subtasks including construct the most accurate similarity graph, automatically allocate optimal weight for each view and find the cluster indicator matrix can be simultaneously accomplished. By this joint learning, each subtask can be mutually enhanced. Experimental results on benchmark datasets demonstrate that our model outperforms other state-of-the-art multi-view clustering algorithms.

Spectral Clustering by Joint Spectral Embedding and Spectral Rotation.

Spectral clustering is an important clustering method widely used for pattern recognition and image segmentation. Classical spectral clustering algorithms consist of two separate stages: 1) solving a relaxed continuous optimization problem to obtain a real matrix followed by 2) applying K -means or spectral rotation to round the real matrix (i.e., continuous clustering result) into a binary matrix called the cluster indicator matrix. Such a separate scheme is not guaranteed to achieve jointly optimal result because of the loss of useful information. To obtain a better clustering result, in this paper, we propose a joint model to simultaneously compute the optimal real matrix and binary matrix. The existing joint model adopts an orthonormal real matrix to approximate the orthogonal but nonorthonormal cluster indicator matrix. It is noted that only in a very special case (i.e., all clusters have the same number of samples), the cluster indicator matrix is an orthonormal matrix multiplied by a real number. The error of approximating a nonorthonormal matrix is inevitably large. To overcome the drawback, we propose replacing the nonorthonormal cluster indicator matrix with a scaled cluster indicator matrix which is an orthonormal matrix. Our method is capable of obtaining better performance because it is easy to minimize the difference between two orthonormal matrices. Experimental results on benchmark datasets demonstrate the effectiveness of the proposed method (called JSESR).

Robust Dimension Reduction for Clustering With Local Adaptive Learning.

In pattern recognition and data mining, clustering is a classical technique to group matters of interest and has been widely employed to numerous applications. Among various clustering algorithms, K-means (KM) clustering is most popular for its simplicity and efficiency. However, with the rapid development of the social network, high-dimensional data are frequently generated, which poses a considerable challenge to the traditional KM clustering as the curse of dimensionality. In such scenarios, it is difficult to directly cluster such high-dimensional data that always contain redundant features and noises. Although the existing approaches try to solve this problem using joint subspace learning and KM clustering, there are still the following limitations: 1) the discriminative information in low-dimensional subspace is not well captured; 2) the intrinsic geometric information is seldom considered; and 3) the optimizing procedure of a discrete cluster indicator matrix is vulnerable to noises. In this paper, we propose a novel clustering model to cope with the above-mentioned challenges. Within the proposed model, discriminative information is adaptively explored by unifying local adaptive subspace learning and KM clustering. We extend the proposed model using a robust l2,1 -norm loss function, where the robust cluster centroid can be calculated in a weighted iterative procedure. We also explore and discuss the relationships between the proposed algorithm and several related studies. Extensive experiments on kinds of benchmark data sets demonstrate the advantage of the proposed model compared with the state-of-the-art clustering approaches.

Consensus learning guided multi-view unsupervised feature selection

Multi-view unsupervised feature selection has been proven to be an effective approach to reduce the dimensionality of multi-view data. One of its key issues is how to exploit the underlying common structures across different views. In this paper, we propose a consensus learning guided multi-view unsupervised feature selection method, which embeds multi-view feature selection into a non-negative matrix factorization based clustering with sparse constrain. The proposed method learns latent feature matrices from all the views, and optimizes a consensus matrix such that the difference between the cluster indicator matrix of each view and the consensus matrix is minimized. The parameters for balancing the weights of different views are automatically adjusted, and a sparse constraint is imposed on the latent feature matrices to perform feature selection. After that, we design an effective iterative algorithm to solve the resultant optimization problem. Extensive experiments have been conducted on six publicly multi-view datasets, and the results demonstrate that the proposed algorithm outperforms several other state-of-the-art single view and multi-view unsupervised feature selection methods in terms of clustering tasks, validating the effectiveness of the proposed multi-view unsupervised feature selection method. The source code of our algorithm will be available on our on-line page: http://tangchang.net/.

Co-regularized unsupervised feature selection

Abstract Unsupervised feature selection (UFS) is very challenging due to the lack of label information. Most UFS methods generate pseudo labels by spectral clustering, matrix factorization or dictionary learning, and convert UFS into a supervised feature selection problem. Generally, the features that can preserve the data distribution (i.e., cluster indicator matrix) should be selected. Under the matrix factorization framework, when the cluster indicator matrix is orthogonal, the cluster base (center) matrix can be used to select features that well reconstruct the data. However, almost all the UFS algorithms only select features that can either well preserve the cluster structure or reconstruct the data. In this paper, we propose a novel co-regularized unsupervised feature selection (CUFS) algorithm. Joint l2,1-norm co-regularization is imposed on multiple feature selection matrices to ensure that the selected features can both preserve data distribution and data reconstruction. Extensive experiments on benchmark datasets show that CUFS is superior to the state-of-the-art UFS algorithms.

Twin Learning for Similarity and Clustering: A Unified Kernel Approach

Many similarity-based clustering methods work in two separate steps including similarity matrix computation and subsequent spectral clustering. However similarity measurement is challenging because it is usually impacted by many factors, e.g., the choice of similarity metric, neighborhood size, scale of data, noise and outliers. Thus the learned similarity matrix is often not suitable, let alone optimal, for the subsequent clustering. In addition, nonlinear similarity often exists in many real world data which, however, has not been effectively considered by most existing methods. To tackle these two challenges, we propose a model to simultaneously learn cluster indicator matrix and similarity information in kernel spaces in a principled way. We show theoretical relationships to kernel k-means, k-means, and spectral clustering methods. Then, to address the practical issue of how to select the most suitable kernel for a particular clustering task, we further extend our model with a multiple kernel learning ability. With this joint model, we can automatically accomplish three subtasks of finding the best cluster indicator matrix, the most accurate similarity relations and the optimal combination of multiple kernels. By leveraging the interactions between these three subtasks in a joint framework, each subtask can be iteratively boosted by using the results of the others towards an overall optimal solution. Extensive experiments are performed to demonstrate the effectiveness of our method.

Robust Manifold Matrix Factorization for Joint Clustering and Feature Extraction

Low-rank matrix approximation has been widely used for data subspace clustering and feature representation in many computer vision and pattern recognition applications. However, in order to enhance the discriminability, most of the matrix approximation based feature extraction algorithms usually generate the cluster labels by certain clustering algorithm (e.g., the kmeans) and then perform the matrix approximation guided by such label information. In addition, the noises and outliers in the dataset with large reconstruction errors will easily dominate the objective function by the conventional ℓ2-norm based squared residue minimization. In this paper, we propose a novel clustering and feature extraction algorithm based on an unified low-rank matrix factorization framework, which suggests that the observed data matrix can be approximated by the production of projection matrix and low dimensional representation, among which the low-dimensional representation can be approximated by the cluster indicator and latent feature matrix simultaneously. Furthermore, we have proposed using the ℓ2,1-norm and integrating the manifold regularization to further promote the proposed model. A novel Augmented Lagrangian Method (ALM) based procedure is designed to effectively and efficiently seek the optimal solution of the problem. The experimental results in both clustering and feature extraction perspectives demonstrate the superior performance of the proposed method.

Re-weighted Discriminatively Embedded K-Means for Multi-view Clustering.

Recent years, more and more multi-view data are widely used in many real world applications. This kind of data (such as image data) are high dimensional and obtained from different feature extractors, which represents distinct perspectives of the data. How to cluster such data efficiently is a challenge. In this paper, we propose a novel multi-view clustering framework, called Re-weighted Discriminatively Embedded KMeans (RDEKM), for this task. The proposed method is a multiview least-absolute residual model which induces robustness to efficiently mitigates the influence of outliers and realizes dimension reduction during multi-view clustering. Specifically, the proposed model is an unsupervised optimization scheme which utilizes Iterative Re-weighted Least Squares to solve leastabsolute residual and adaptively controls the distribution of multiple weights in a re-weighted manner only based on its own low-dimensional subspaces and a common clustering indicator matrix. Furthermore, theoretical analysis (including optimality and convergence analysis) and the optimization algorithm are also presented. Compared to several state-of-the-art multi-view clustering methods, the proposed method substantially improves the accuracy of the clustering results on widely used benchmark datasets, which demonstrates the superiority of the proposed work.

Discriminative Nonnegative Spectral Clustering with Out-of-Sample Extension

Data clustering is one of the fundamental research problems in data mining and machine learning. Most of the existing clustering methods, for example, normalized cut and (k)-means, have been suffering from the fact that their optimization processes normally lead to an NP-hard problem due to the discretization of the elements in the cluster indicator matrix. A practical way to cope with this problem is to relax this constraint to allow the elements to be continuous values. The eigenvalue decomposition can be applied to generate a continuous solution, which has to be further discretized. However, the continuous solution is probably mixing-signed. This result may cause it deviate severely from the true solution, which should be naturally nonnegative. In this paper, we propose a novel clustering algorithm, i.e., discriminative nonnegative spectral clustering, to explicitly impose an additional nonnegative constraint on the cluster indicator matrix to seek for a more interpretable solution. Moreover, we show an effective regularization term which is able to not only provide more useful discriminative information but also learn a mapping function to predict cluster labels for the out-of-sample test data. Extensive experiments on various data sets illustrate the superiority of our proposal compared to the state-of-the-art clustering algorithms.

Nonnegative Spectral Clustering with Discriminative Regularization

Clustering is a fundamental research topic in the field of data mining. Optimizing the objective functions of clustering algorithms, e.g. normalized cut and k-means, is an NP-hard optimization problem. Existing algorithms usually relax the elements of cluster indicator matrix from discrete values to continuous ones. Eigenvalue decomposition is then performed to obtain a relaxed continuous solution, which must be discretized. The main problem is that the signs of the relaxed continuous solution are mixed. Such results may deviate severely from the true solution, making it a nontrivial task to get the cluster labels. To address the problem, we impose an explicit nonnegative constraint for a more accurate solution during the relaxation. Besides, we additionally introduce a discriminative regularization into the objective to avoid overfitting. A new iterative approach is proposed to optimize the objective. We show that the algorithm is a general one which naturally leads to other extensions. Experiments demonstrate the effectiveness of our algorithm.

Image Clustering Using Local Discriminant Models and Global Integration

In this paper, we propose a new image clustering algorithm, referred to as clustering using local discriminant models and global integration (LDMGI). To deal with the data points sampled from a nonlinear manifold, for each data point, we construct a local clique comprising this data point and its neighboring data points. Inspired by the Fisher criterion, we use a local discriminant model for each local clique to evaluate the clustering performance of samples within the local clique. To obtain the clustering result, we further propose a unified objective function to globally integrate the local models of all the local cliques. With the unified objective function, spectral relaxation and spectral rotation are used to obtain the binary cluster indicator matrix for all the samples. We show that LDMGI shares a similar objective function with the spectral clustering (SC) algorithms, e.g., normalized cut (NCut). In contrast to NCut in which the Laplacian matrix is directly calculated based upon a Gaussian function, a new Laplacian matrix is learnt in LDMGI by exploiting both manifold structure and local discriminant information. We also prove that K-means and discriminative K-means (DisKmeans) are both special cases of LDMGI. Extensive experiments on several benchmark image datasets demonstrate the effectiveness of LDMGI. We observe in the experiments that LDMGI is more robust to algorithmic parameter, when compared with NCut. Thus, LDMGI is more appealing for the real image clustering applications in which the ground truth is generally not available for tuning algorithmic parameters.