Co-clustering Algorithms Research Articles

Penalized nonnegative matrix tri-factorization for co-clustering

Nonnegative matrix factorization has been widely used in co-clustering tasks which group data points and features simultaneously. In recent years, several proposed co-clustering algorithms have shown their superiorities over traditional one-side clustering, especially in text clustering and gene expression. Due to the NP-completeness of the co-clustering problems, most existing methods relaxed the orthogonality constraint as nonnegativity, which often deteriorates performance and robustness as a result. In this paper, penalized nonnegative matrix tri-factorization is proposed for co-clustering problems, where three penalty terms are introduced to guarantee the near orthogonality of the clustering indicator matrices. An iterative updating algorithm is proposed and its convergence is proved. Furthermore, the high-order nonnegative matrix tri-factorization technique is provided for symmetric co-clustering tasks and a corresponding algorithm with proved convergence is also developed. Finally, extensive experiments in six real-world datasets demonstrate that the proposed algorithms outperform the compared state-of-the-art co-clustering methods.

Graph modularity maximization as an effective method for co-clustering text data

In this paper we show how the modularity measure can serve as a useful criterion for co-clustering document-term matrices. We present and investigate the performance of CoClus, a novel, effective block-diagonal co-clustering algorithm which directly maximizes this modularity measure. The maximization is performed using an iterative alternating optimization procedure, in contrast to algorithms that use spectral relaxations of the discrete optimization problems. Extensive comparative experiments performed on various document-term datasets demonstrate that this approach is very effective, stable, and outperforms other block-diagonal co-clustering algorithms devoted to the same task. Another important advantage of using modularity in the co-clustering context is that it provides a novel, simple way of determining the appropriate number of co-clusters.Availability: an implementation of CoClus is available as part of the recently released coclust Python package which is available at: https://pypi.python.org/pypi/coclust

Hard and fuzzy diagonal co-clustering for document-term partitioning

We propose a hard and a fuzzy diagonal co-clustering algorithms built upon the double K-means to address the problem of document-term co-clustering. At each iteration, the proposed algorithms seek a diagonal block structure of the data by minimizing a criterion based on both the variance within the class and the centroid effect. In addition to be easy-to-interpret and effective on sparse binary and continuous data, the proposed algorithms, Hard Diagonal Double K-means (DDKM) and Fuzzy Diagonal Double K-means (F-DDKM), are also faster than other state-of-the-art clustering algorithms. We evaluate our contribution using synthetic data sets, and real data sets commonly used in document clustering.

Co-ClusterD: A Distributed Framework for Data Co-Clustering with Sequential Updates

Co-clustering has emerged to be a powerful data mining tool for two-dimensional co-occurrence and dyadic data. However, co-clustering algorithms often require significant computational resources and have been dismissed as impractical for large data sets. Existing studies have provided strong empirical evidence that expectation-maximization (EM) algorithms (e.g., k-means algorithm) with sequential updates can significantly reduce the computational cost without degrading the resulting solution. Motivated by this observation, we introduce sequential updates for alternate minimization co-clustering (AMCC) algorithms which are variants of EM algorithms, and also show that AMCC algorithms with sequential updates converge. We then propose two approaches to parallelize AMCC algorithms with sequential updates in a distributed environment. Both approaches are proved to maintain the convergence properties of AMCC algorithms. Based on these two approaches, we present a new distributed framework, Co-ClusterD, which supports efficient implementations of AMCC algorithms with sequential updates. We design and implement Co-ClusterD, and show its efficiency through two AMCC algorithms: fast nonnegative matrix tri-factorization (FNMTF) and information theoretic co-clustering (ITCC). We evaluate our framework on both a local cluster of machines and the Amazon EC2 cloud. Empirical results show that AMCC algorithms implemented in Co-ClusterD can achieve a much faster convergence and often obtain better results than their traditional concurrent counterparts.

Bezdek-Type Fuzzified Co-Clustering Algorithm

In this study, two co-clustering algorithms based on Bezdek-type fuzzification of fuzzy clustering are proposed for categorical multivariate data. The two proposed algorithms are motivated by the fact that there are only two fuzzy co-clustering methods currently available – entropy regularization and quadratic regularization – whereas there are three fuzzy clustering methods for vectorial data: entropy regularization, quadratic regularization, and Bezdek-type fuzzification. The first proposed algorithm forms the basis of the second algorithm. The first algorithm is a variant of a spherical clustering method, with the kernelization of a maximizing model of Bezdek-type fuzzy clustering with multi-medoids. By interpreting the first algorithm in this way, the second algorithm, a spectral clustering approach, is obtained. Numerical examples demonstrate that the proposed algorithms can produce satisfactory results when suitable parameter values are selected.

Constraint Co-Projections for Semi-Supervised Co-Clustering.

Co-clustering aims to simultaneously cluster the objects and features to explore intercorrelated patterns. However, it is usually difficult to obtain good co-clustering results by just analyzing the object-feature correlation data due to the sparsity of the data and the noise. Meanwhile, most co-clustering algorithms cannot take the prior information into consideration and may produce unmeaningful results. Semi-supervised co-clustering aims to incorporate the known prior knowledge into the co-clustering algorithm. In this paper, a new technique named constraint co-projections for semi-supervised co-clustering (CPSSCC) is presented. Constraint co-projections can not only make use of two popular techniques including pairwise constraints and constraint projections, but also simultaneously perform the object constraint projections and feature constraint projections. The two popular techniques are illustrated for semi-supervised co-clustering when some objects and features are believed to be in the same cluster a priori. Furthermore, we also prove that the co-clustering problem can be formulated as a typical eigen-problem and can be efficiently solved with the selected eigenvectors. To the best of our knowledge, constraint co-projections is first stated in this paper and this is the first work on using CPSSCC. Extensive experiments on benchmark data sets demonstrate the effectiveness of the proposed method. This paper also shows that CPSSCC has some favorable features compared with previous related co-clustering algorithms.

Self-representation based dual-graph regularized feature selection clustering

Feature selection algorithms eliminate irrelevant and redundant features, even the noise, while preserving the most representative features. They can reduce the dimension of the dataset, extract essential features in high dimensional data and improve learning quality. Existing feature selection algorithms are all carried out in data space. However, the information of feature space cannot be fully exploited. To compensate for this drawback, this paper proposes a novel feature selection algorithm for clustering, named self-representation based dual-graph regularized feature selection clustering (DFSC). It adopts the self-representation property that data can be represented by itself. Meanwhile, the local geometrical information of both data space and feature space are preserved simultaneously. By imposing the l2,1-norm constraint on the self-representation coefficients matrix in data space, DFSC can effectively select the most representative features for clustering. We give the objective function, develop iterative updating rules and provide the convergence proof. Two kinds of extensive experiments on some datasets demonstrate the effectiveness of DFSC. Extensive comparisons over several state-of-the-art feature selection algorithms illustrate that additionally considering the information of feature space based on self-representation property improves clustering quality. Meanwhile, because the additional feature selection process can select the most important features to preserve the intrinsic structure of dataset, the proposed algorithm achieves better clustering results compared with some co-clustering algorithms.

Spectral co-clustering ensemble

The goal of co-clustering is to simultaneously cluster the rows and columns of an input data matrix. It overcomes several limitations associated with traditional clustering methods by allowing automatic discovery of similarity based on a subset of attributes. However, different co-clustering models usually produce very distinct results since each algorithm has its own bias due to the optimization of different criteria. The idea of combining different co-clustering results emerged as an alternative approach for improving the performance of co-clustering algorithms. Similar to clustering ensembles, co-clustering ensembles provide a framework for combining multiple base co-clusterings of a dataset to generate a stable and robust consensus co-clustering result. In this paper, a novel co-clustering ensemble algorithm named spectral co-clustering ensemble (SCCE) is presented. SCCE performs ensemble tasks on base row clusters and column clusters of a dataset simultaneously, and obtains an optimization co-clustering result. Meanwhile, SCCE is a matrix decomposition based approach which can be formulated as a bipartite graph partition problem and solve it efficiently with the selected eigenvectors. To the best of our knowledge, this is the first work on using spectral algorithm for co-clustering ensemble. Extensive experiments on benchmark datasets demonstrate the effectiveness of the proposed method. Our study also shows that SCCE has some favorable merits compared with many state of the art methods.

HICC: an entropy splitting-based framework for hierarchical co-clustering

Two-dimensional contingency tables or co-occurrence matrices arise frequently in various important applications such as text analysis and web-log mining. As a fundamental research topic, co-clustering aims to generate a meaningful partition of the contingency table to reveal hidden relationships between rows and columns. Traditional co-clustering algorithms usually produce a predefined number of flat partition of both rows and columns, which do not reveal relationship among clusters. To address this limitation, hierarchical co-clustering algorithms have attracted a lot of research interests recently. Although successful in various applications, the existing hierarchical co-clustering algorithms are usually based on certain heuristics and do not have solid theoretical background. In this paper, we present a new co-clustering algorithm, HICC, with solid theoretical background. It simultaneously constructs a hierarchical structure of both row and column clusters, which retains sufficient mutual information between rows and columns of the contingency table. An efficient and effective greedy algorithm is developed, which grows a co-cluster hierarchy by successively performing row-wise or column-wise splits that lead to the maximal mutual information gain. Extensive experiments on both synthetic and real datasets demonstrate that our algorithm can reveal essential relationships of row (and column) clusters and has better clustering precision than existing algorithms. Moreover, the experiments on real dataset show that HICC can effectively reveal hidden relationships between rows and columns in the contingency table.

Fuzzy Co-Clustering Algorithms Based on Fuzzy Relational Clustering and TIBA Imputation

In this paper, two types of fuzzy co-clustering algorithms are proposed. First, it is shown that the base of the objective function for the conventional fuzzy co-clustering method is very similar to the base for entropy-regularized fuzzy nonmetric model. Next, it is shown that the non-sense clustering problem in the conventional fuzzy co-clustering algorithms is identical to that in fuzzy nonmetric model algorithms, in the case that all dissimilarities among rows and columns are zero. Based on this discussion, a method is proposed applying entropy-regularized fuzzy nonmetric model after all dissimilarities among rows and columns are set to some values using a TIBA imputation technique. Furthermore, since relational fuzzy cmeans is similar to fuzzy nonmetricmodel, in the sense that both methods are designed for homogeneous relational data, a method is proposed applying entropyregularized relational fuzzyc-means after imputing all dissimilarities among rows and columns with TIBA. Some numerical examples are presented for the proposed methods.

Dual Exclusive Partition in Fuzzy CoDoK and SCAD-based Fuzzy Co-clustering

Fuzzy co-clustering is a basic technique for revealing intrinsic co-cluster structures from cooccurrence information among objects and items. In most of fuzzy co-clustering algorithms, objects and items are partitioned based on different constraints. Objects are forced to be exclusively partitioned like as Fuzzy c-Means (FCM), while item memberships often represent just the relative significance of items in each cluster, i.e., items can be shared by multiple clusters. In a previous work, exclusive partition of items were achieved by introducing a penalty term in Fuzzy Clustering for Categorical Multivariate data (FCCM), which is an FCM-type co-clustering with entropy regularization mechanism. In this paper, the applicability of dual exclusive partition of objects and items are discussed in the frameworks of Fuzzy CoDoK and SCAD-based fuzzy co-clustering.

Hierarchical co-clustering: off-line and incremental approaches

Clustering data is challenging especially for two reasons. The dimensionality of the data is often very high which makes the cluster interpretation hard. Moreover, with high-dimensional data the classic metrics fail in identifying the real similarities between objects. The second challenge is the evolving nature of the observed phenomena which makes the datasets accumulating over time. In this paper we show how we propose to solve these problems. To tackle the high-dimensionality problem, we propose to apply a co-clustering approach on the dataset that stores the occurrence of features in the observed objects. Co-clustering computes a partition of objects and a partition of features simultaneously. The novelty of our co-clustering solution is that it arranges the clusters in a hierarchical fashion, and it consists of two hierarchies: one on the objects and one on the features. The two hierarchies are coupled because the clusters at a certain level in one hierarchy are coupled with the clusters at the same level of the other hierarchy and form the co-clusters. Each cluster of one of the two hierarchies thus provides insights on the clusters of the other hierarchy. Another novelty of the proposed solution is that the number of clusters is possibly unlimited. Nevertheless, the produced hierarchies are still compact and therefore more readable because our method allows multiple splits of a cluster at the lower level. As regards the second challenge, the accumulating nature of the data makes the datasets intractably huge over time. In this case, an incremental solution relieves the issue because it partitions the problem. In this paper we introduce an incremental version of our algorithm of hierarchical co-clustering. It starts from an intermediate solution computed on the previous version of the data and it updates the co-clustering results considering only the added block of data. This solution has the merit of speeding up the computation with respect to the original approach that would recompute the result on the overall dataset. In addition, the incremental algorithm guarantees approximately the same answer than the original version, but it saves much computational load. We validate the incremental approach on several high-dimensional datasets and perform an accurate comparison with both the original version of our algorithm and with the state of the art competitors as well. The obtained results open the way to a novel usage of the co-clustering algorithms in which it is advantageous to partition the data into several blocks and process them incrementally thus incorporating data gradually into an on-going co-clustering solution.

Locally Discriminative Coclustering

Different from traditional one-sided clustering techniques, coclustering makes use of the duality between samples and features to partition them simultaneously. Most of the existing co-clustering algorithms focus on modeling the relationship between samples and features, whereas the intersample and interfeature relationships are ignored. In this paper, we propose a novel coclustering algorithm named Locally Discriminative Coclustering (LDCC) to explore the relationship between samples and features as well as the intersample and interfeature relationships. Specifically, the sample-feature relationship is modeled by a bipartite graph between samples and features. And we apply local linear regression to discovering the intrinsic discriminative structures of both sample space and feature space. For each local patch in the sample and feature spaces, a local linear function is estimated to predict the labels of the points in this patch. The intersample and interfeature relationships are thus captured by minimizing the fitting errors of all the local linear functions. In this way, LDCC groups strongly associated samples and features together, while respecting the local structures of both sample and feature spaces. Our experimental results on several benchmark data sets have demonstrated the effectiveness of the proposed method.

Constrained co-clustering with non-negative matrix factorisation

Co-clustering refers to the problem of deriving sub-matrices of the data matrix by simultaneously clustering the rows (data instances) and columns (features) of the matrix. While very effective in discovering useful knowledge, many of the co-clustering algorithms adopt a completely unsupervised approach. Integration of domain knowledge can guide the co-clustering process and greatly enhance the overall performance. We propose a semi-supervised Non-negative Matrix-factorisation (SS-NMF) based framework to integrate domain knowledge in the form of must-link and cannot-link constraints. Specifically, we augment the data matrix by integrating the constraints using metric learning and then perform NMF to obtain co-clustering. Under the proposed framework, we present two approaches to integrate domain knowledge, viz. a distance metric learning approach and an information theoretic metric learning approach. Through experiments performed on real-world web service data and publicly available text datasets, we demonstrate the performance of the proposed SS-NMF based approach for data co-clustering.

A General Framework for Fast Co-clustering on Large Datasets Using Matrix Decomposition.

Simultaneously clustering columns and rows (co-clustering) of large data matrix is an important problem with wide applications, such as document mining, microarray analysis, and recommendation systems. Several co-clustering algorithms have been shown effective in discovering hidden clustering structures in the data matrix. For a data matrix of m rows and n columns, the time complexity of these methods is usually in the order of m × n (if not higher). This limits their applicability to data matrices involving a large number of columns and rows. Moreover, an implicit assumption made by existing co-clustering methods is that the whole data matrix needs to be held in the main memory. In this paper, we propose a general framework, CRD, for co-clustering large datasets utilizing recently developed sampling-based matrix decomposition methods. The time complexity of our approach is linear in m and n. And it does not require the whole data matrix be in the main memory. Extensive experimental results on synthetic and several well-known real-life datasets show that CRD achieves competitive accuracy to existing co-clustering methods but with much less computational cost.

Multimodal News Story Clustering With Pairwise Visual Near-Duplicate Constraint

Story clustering is a critical step for news retrieval, topic mining, and summarization. Nonetheless, the task remains highly challenging owing to the fact that news topics exhibit clusters of varying densities, shapes, and sizes. Traditional algorithms are found to be ineffective in mining these types of clusters. This paper offers a new perspective by exploring the pairwise visual cues deriving from near-duplicate keyframes (NDK) for constraint-based clustering. We propose a constraint-driven co-clustering algorithm (CCC), which utilizes the near-duplicate constraints built on top of text, to mine topic-related stories and the outliers. With CCC, the duality between stories and their underlying multimodal features is exploited to transform features in low-dimensional space with normalized cut. The visual constraints are added directly to this new space, while the traditional DBSCAN is revisited to capitalize on the availability of constraints and the reduced dimensional space. We modify DBSCAN with two new characteristics for story clustering: 1) constraint-based centroid selection and 2) adaptive radius. Experiments on TRECVID-2004 corpus demonstrate that CCC with visual constraints is more capable of mining news topics of varying densities, shapes and sizes, compared with traditional <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -means, DBSCAN, and spectral co-clustering algorithms.

A heuristic-based fuzzy co-clustering algorithm for categorization of high-dimensional data

Fuzzy co-clustering is a technique that performs simultaneous fuzzy clustering of objects and features. It is known to be suitable for categorizing high-dimensional data, due to its dynamic dimensionality reduction mechanism achieved through simultaneous feature clustering. We introduce a new fuzzy co-clustering algorithm called Heuristic Fuzzy Co-clustering with the Ruspini's condition (HFCR), which addresses several issues in some prominent existing fuzzy co-clustering algorithms. Among these issues are the performance on data sets with overlapping feature clusters and the unnatural representation of feature clusters. The key idea behind HFCR is the formulation of the dual-partitioning approach for fuzzy co-clustering, replacing the existing partitioning-ranking approach. HFCR adopts an efficient and practical heuristic method that can be shown to be more robust than our earlier effort for the dual-partitioning approach. We explain the proposed algorithm in details and provide an analytical study on its advantages. Experimental results on 10 large benchmark document data sets confirm the effectiveness of the new algorithm.

A partitioning based algorithm to fuzzy co-cluster documents and words

In this paper, a new algorithm fuzzy co-clustering with Ruspini’s condition (FCR) is proposed for co-clustering documents and words. Compared to most existing fuzzy co-clustering algorithms, FCR is able to generate fuzzy word clusters that capture the natural distribution of words, which may be beneficial for information retrieval. We discuss the principle behind the algorithm through some theoretical discussions and illustrations. These, together with experiments on two standard datasets show that FCR can discover the naturally existing document-word co-clusters.