Multiple Kernel K-means Research Articles

Fast Approximated Multiple Kernel K-Means

Fast Approximated Multiple Kernel K-Means

On the Consistency and Large-Scale Extension of Multiple Kernel Clustering.

Existing multiple kernel clustering (MKC) algorithms have two ubiquitous problems. From the theoretical perspective, most MKC algorithms lack sufficient theoretical analysis, especially the consistency of learned parameters, such as the kernel weights. From the practical perspective, the high complexity makes MKC unable to handle large-scale datasets. This paper tries to address the above two issues. We first make a consistency analysis of an influential MKC method named Simple Multiple Kernel k-Means (SimpleMKKM). Specifically, suppose that ∧γn are the kernel weights learned by SimpleMKKM from the training samples. We also define the expected version of SimpleMKKM and denote its solution as γ*. We establish an upper bound of ||∧γn-γ*||∞ in the order of ~O(1/√n), where n is the sample number. Based on this result, we also derive its excess clustering risk calculated by a standard clustering loss function. For the large-scale extension, we replace the eigen decomposition of SimpleMKKM with singular value decomposition (SVD). Consequently, the complexity can be decreased to O(n) such that SimpleMKKM can be implemented on large-scale datasets. We then deduce several theoretical results to verify the approximation ability of the proposed SVD-based method. The results of comprehensive experiments demonstrate the superiority of the proposed method.

COPS: A novel platform for multi-omic disease subtype discovery via robust multi-objective evaluation of clustering algorithms.

Recent research on multi-view clustering algorithms for complex disease subtyping often overlooks aspects like clustering stability and critical assessment of prognostic relevance. Furthermore, current frameworks do not allow for a comparison between data-driven and pathway-driven clustering, highlighting a significant gap in the methodology. We present the COPS R-package, tailored for robust evaluation of single and multi-omics clustering results. COPS features advanced methods, including similarity networks, kernel-based approaches, dimensionality reduction, and pathway knowledge integration. Some of these methods are not accessible through R, and some correspond to new approaches proposed with COPS. Our framework was rigorously applied to multi-omics data across seven cancer types, including breast, prostate, and lung, utilizing mRNA, CNV, miRNA, and DNA methylation data. Unlike previous studies, our approach contrasts data- and knowledge-driven multi-view clustering methods and incorporates cross-fold validation for robustness. Clustering outcomes were assessed using the ARI score, survival analysis via Cox regression models including relevant covariates, and the stability of the results. While survival analysis and gold-standard agreement are standard metrics, they vary considerably across methods and datasets. Therefore, it is essential to assess multi-view clustering methods using multiple criteria, from cluster stability to prognostic relevance, and to provide ways of comparing these metrics simultaneously to select the optimal approach for disease subtype discovery in novel datasets. Emphasizing multi-objective evaluation, we applied the Pareto efficiency concept to gauge the equilibrium of evaluation metrics in each cancer case-study. Affinity Network Fusion, Integrative Non-negative Matrix Factorization, and Multiple Kernel K-Means with linear or Pathway Induced Kernels were the most stable and effective in discerning groups with significantly different survival outcomes in several case studies.

Ensemble 3D CNN and U-Net-based brain tumour classification with MKKMC segmentation

Advanced brain cancer is the deadliest type with just a few months survival rate. Existing technologies hinder the objective of forecasting cancer. This work aims to fulfil the pressing requirement for timely and precise identification of advanced-stage brain tumours, which are notorious for their markedly reduced life expectancy. It presents an innovative hybrid approach for predicting brain tumours and improves diagnostic capabilities. The Multiple Kernel K-Means Cluster Algorithm (MKKCA) is used to segment brain MRI images effectively, differentiating healthy and tumorous tissues. After segmentation, a hybrid approach with 3D-Convolutional Neural Network (CNN) and U-Net has been utilized for classification. The objective is to effectively and accurately distinguish normal and pathological brain images. To enhance the efficiency, we include the Improved Whale Optimization Algorithm (IWOA), which guarantees accurate and dependable performance via location updates. The methodology demonstrates outstanding precision with 98.5% accuracy rate, 98.56% specificity, 91% sensitivity, 87.45% precision and a recall rate of 96% with the F-Measure at 96.02%. These findings, obtained using MATLAB, demonstrate a substantial performance improvement compared to current approaches. This development not only represents a significant addition to diagnostic imaging but also a crucial role in the prediction and treatment of brain cancers.

Multiple kernel k-means clustering with block diagonal property

Multiple kernel k-means clustering with block diagonal property

Simple multiple kernel k-means with kernel weight regularization

Multiple kernel clustering (MKC) aims to determine the optimal kernel from several pre-computed basic kernels. Most of MKC algorithms follow a common assumption that the optimal kernel is linearly combined by basic kernels. Based on a min–max framework, a newly proposed MKC method termed simple multiple kernel k-means can acquire a high-quality unified kernel. Although it has achieved promising clustering performance, we have observed that it cannot benefit from any regularization or prior knowledge, thus the learned kernel weight coefficients may be seriously sparse or over-selected. To tackle this issue, we have developed a new algorithm termed simple multiple kernel k-means with kernel weight regularization (SMKKM-KWR), where we introduce average coefficients to avoid too sparse kernel weights. Specially, we add the average kernel coefficients as a regularization term to prevent the learned weight coefficients being far from the average values. After that, an efficient optimization strategy is proposed to solve the new resultant problem. By this way, the unified partition can learn clustering structure from fusion information of all the kernel views, with the goal to reach better clustering performance. Extensive experiments on nine benchmark datasets and four large-scale datasets have demonstrated the effectiveness and efficiency of the proposed algorithm.

Localized Incomplete Multiple Kernel k-Means With Matrix-Induced Regularization.

Localized incomplete multiple kernel k -means (LI-MKKM) is recently put forward to boost the clustering accuracy via optimally utilizing a quantity of prespecified incomplete base kernel matrices. Despite achieving significant achievement in a variety of applications, we find out that LI-MKKM does not sufficiently consider the diversity and the complementary of the base kernels. This could make the imputation of incomplete kernels less effective, and vice versa degrades on the subsequent clustering. To tackle these problems, an improved LI-MKKM, called LI-MKKM with matrix-induced regularization (LI-MKKM-MR), is proposed by incorporating a matrix-induced regularization term to handle the correlation among base kernels. The incorporated regularization term is beneficial to decrease the probability of simultaneously selecting two similar kernels and increase the probability of selecting two kernels with moderate differences. After that, we establish a three-step iterative algorithm to solve the corresponding optimization objective and analyze its convergence. Moreover, we theoretically show that the local kernel alignment is a special case of its global one with normalizing each base kernel matrices. Based on the above observation, the generalization error bound of the proposed algorithm is derived to theoretically justify its effectiveness. Finally, extensive experiments on several public datasets have been conducted to evaluate the clustering performance of the LI-MKKM-MR. As indicated, the experimental results have demonstrated that our algorithm consistently outperforms the state-of-the-art ones, verifying the superior performance of the proposed algorithm.

SimpleMKKM: Simple Multiple Kernel K-Means.

We propose a simple yet effective multiple kernel clustering algorithm, termed simple multiple kernel k-means (SimpleMKKM). It extends the widely used supervised kernel alignment criterion to multi-kernel clustering. Our criterion is given by an intractable minimization-maximization problem in the kernel coefficient and clustering partition matrix. To optimize it, we equivalently rewrite the minimization-maximization formulation as a minimization of an optimal value function, prove its differenentiablity, and design a reduced gradient descent algorithm to decrease it. Furthermore, we prove that the resultant solution of SimpleMKKM is the global optimum. We theoretically analyze the performance of SimpleMKKM in terms of its clustering generalization error. After that, we develop extensive experiments to investigate the proposed SimpleMKKM from the perspectives of clustering accuracy, advantage on the formulation and optimization, variation of the learned consensus clustering matrix with iterations, clustering performance with varied number of samples and base kernels, analysis of the learned kernel weight, the running time and the global convergence. The experimental study demonstrates the effectiveness of the proposed SimpleMKKM by considerably and consistently outperforming state of the art multiple kernel clustering alternatives. In addition, the ablation study shows that the improved clustering performance is contributed by both the novel formulation and new optimization. Our work provides a more effective approach to integrate multi-view data for clustering, and this could trigger novel research on multiple kernel clustering. The source code and data for SimpleMKKM are available at https://github.com/xinwangliu/SimpleMKKMcodes/.

Hyperparameter-Free Localized Simple Multiple Kernel K-Means With Global Optimum.

The newly proposed localized simple multiple kernel k-means (SimpleMKKM) provides an elegant clustering framework which sufficiently considers the potential variation among samples. Although achieving superior clustering performance in some applications, we observe that it is required to pre-specify an extra hyperparameter, which determines the size of the localization. This greatly limits its availability in practical applications since there is a little guideline to set a suitable hyperparameter in clustering tasks. To overcome this issue, we firstly parameterize a neighborhood mask matrix as a quadratic combination of a set of pre-computed base neighborhood mask matrices, which corresponds to a group of hyperparameters. We then propose to jointly learn the optimal coefficient of these neighborhood mask matrices together with the clustering tasks. By this way, we obtain the proposed hyperparameter-free localized SimpleMKKM, which corresponds to a more intractable minimization-minimization-maximization optimization problem. We rewrite the resultant optimization as a minimization of an optimal value function, prove its differentiability, and develop a gradient based algorithm to solve it. Furthermore, we theoretically prove that the obtained optimum is the global one. Comprehensive experimental study on several benchmark datasets verifies its effectiveness, comparing with several state-of-the-art counterparts in the recent literature. The source code for hyperparameter-free localized SimpleMKKM is available at https://github.com/xinwangliu/SimpleMKKMcodes/.

Localized Simple Multiple Kernel K-Means Clustering with Matrix-Induced Regularization.

Multikernel clustering achieves clustering of linearly inseparable data by applying a kernel method to samples in multiple views. A localized SimpleMKKM (LI-SimpleMKKM) algorithm has recently been proposed to perform min-max optimization in multikernel clustering where each instance is only required to be aligned with a certain proportion of the relatively close samples. The method has improved the reliability of clustering by focusing on the more closely paired samples and dropping the more distant ones. Although LI-SimpleMKKM achieves remarkable success in a wide range of applications, the method keeps the sum of the kernel weights unchanged. Thus, it restricts kernel weights and does not consider the correlation between the kernel matrices, especially between paired instances. To overcome such limitations, we propose adding a matrix-induced regularization to localized SimpleMKKM (LI-SimpleMKKM-MR). Our approach addresses the kernel weight restrictions with the regularization term and enhances the complementarity between base kernels. Thus, it does not limit kernel weights and fully considers the correlation between paired instances. Extensive experiments on several publicly available multikernel datasets show that our method performs better than its counterparts.

Fusion Multiple Kernel K-means

Multiple kernel clustering aims to seek an appropriate combination of base kernels to mine inherent non-linear information for optimal clustering. Late fusion algorithms generate base partitions independently and integrate them in the following clustering procedure, improving the overall efficiency. However, the separate base partition generation leads to inadequate negotiation with the clustering procedure and a great loss of beneficial information in corresponding kernel matrices, which negatively affects the clustering performance. To address this issue, we propose a novel algorithm, termed as Fusion Multiple Kernel k-means (FMKKM), which unifies base partition learning and late fusion clustering into one single objective function, and adopts early fusion technique to capture more sufficient information in kernel matrices. Specifically, the early fusion helps base partitions keep more beneficial kernel details, and the base partitions learning further guides the generation of consensus partition in the late fusion stage, while the late fusion provides positive feedback on two former procedures. The close collaboration of three procedures results in a promising performance improvement. Subsequently, an alternate optimization method with promising convergence is developed to solve the resultant optimization problem. Comprehensive experimental results demonstrate that our proposed algorithm achieves state-of-the-art performance on multiple public datasets, validating its effectiveness. The code of this work is publicly available at https://github.com/ethan-yizhang/Fusion-Multiple-Kernel-K-means.

Clustering via multiple kernel k-means coupled graph and enhanced tensor learning

Clustering via multiple kernel k-means coupled graph and enhanced tensor learning

Discrete and Parameter-Free Multiple Kernel k-Means.

The multiple kernel k -means (MKKM) and its variants utilize complementary information from different sources, achieving better performance than kernel k -means (KKM). However, the optimization procedures of most previous works comprise two stages, learning the continuous relaxation matrix and obtaining the discrete one by extra discretization procedures. Such a two-stage strategy gives rise to a mismatched problem and severe information loss. Even worse, most existing MKKM methods overlook the correlation among prespecified kernels, which leads to the fusion of mutually redundant kernels and bad effects on the diversity of information sources, finally resulting in unsatisfying results. To address these issues, we elaborate a novel Discrete and Parameter-free Multiple Kernel k -means (DPMKKM) model solved by an alternative optimization method, which can directly obtain the cluster assignment results without subsequent discretization procedure. Moreover, DPMKKM can measure the correlation among kernels by implicitly introducing a regularization term, which is able to enhance kernel fusion by reducing redundancy and improving diversity. Noteworthily, the time complexity of optimization algorithm is successfully reduced, through masterly utilizing of coordinate descent technique, which contributes to higher algorithm efficiency and broader applications. What's more, our proposed model is parameter-free avoiding intractable hyperparameter tuning, which makes it feasible in practical applications. Lastly, extensive experiments conducted on a number of real-world datasets illustrated the effectiveness and superiority of the proposed DPMKKM model.

Spare simple MKKM with semi‐infinite linear program optimization

Multiple kernel clustering (MKC) optimally combines a group of predefined kernel matrices to improve clustering performance. Although demonstrating promising performance in various applications, most of existing approaches adopt the min–min formulation, which could be sensitive to perturbation with adversarial samples. Moreover, existing MKC algorithms often involve several hypermeters preventing them into further real applications. To address these issues, we propose a parameter-free effective sparse simple multiple kernel k-means algorithm with max–min optimization formulation in this paper. To be specific, we propose to optimize the widely used unsupervised kernel alignment criterion by minimizing the kernel coefficient and maximizing the clustering partition matrix. Unlike traditional min–min formulation, the max–min kernel alignment is robust to adversarial sample perturbation and free of hyper-parameters. An optimization method based on semi-infinite linear program is designed to solve the complicated optimization problem. Extensive experiments on six multiple kernel benchmark data sets demonstrate the effectiveness of the proposed method.

Multiple kernel clustering with late fusion consensus local graph preserving

Multiple kernel clustering (MKC) methods aim at integrating an optimal partition from a set of precalculated kernel matrices. Though achieving success in various applications, we observe that existing MKC methods: (i) lack of representation flexibility; and (ii) do not considerably preserve the locality structure in partition space. These issues may adversely affect the learning procedure of MKC, leading to unsatisfying clustering performance. In this paper, we propose a late fusion MKC method with local graph refinement to address the aforementioned issues. Different from existing MKC mechanisms, our method unifies the traditional weighted multiple kernel k-means, kernel partition, and graph construction into a single optimization procedure. The local graph is utilized to preserve the locality information in partition space and therefore all of the counterparts can be boosted for mutual clustering improvements. By this way, our approach enhances the local graph structure in partition space and enjoys more flexible kernel representations, leading to significant clustering improvements. Moreover, a three-step alternate algorithm is developed to solve the resultant optimization problem with proved convergence. Extensive experiments are conducted on several multiple kernel benchmark datasets to compare the proposed algorithm with the state-of-the-art ones, and the results well demonstrate its effectiveness and superiority.

Trade-off Between Efficiency and Effectiveness: A Late Fusion Multi-view Clustering Algorithm

Late fusion multi-view clustering (LFMVC) algorithms aim to integrate the base partition of each single view into a consensus partition. Base partitions can be obtained by performing kernel k-means clustering on all views. This type of method is not only computationally efficient, but also more accurate than multiple kernel k-means, and is thus widely used in the multi-view clustering context. LFMVC improves computational efficiency to the extent that the computational complexity of each iteration is reduced from O(n3) to O(n) (where n is the number of samples). However, LFMVC also limits the search space of the optimal solution, meaning that the clustering results obtained are not ideal. Accordingly, in order to obtain more information from each base partition and thus improve the clustering performance, we propose a new late fusion multi-view clustering algorithm with a computational complexity of O(n2). Experiments on several commonly used datasets demonstrate that the proposed algorithm can reach quickly convergence. Moreover, compared with other late fusion algorithms with computational complexity of O(n), the actual time consumption of the proposed algorithm does not significantly increase. At the same time, comparisons with several other state-of-the-art algorithms reveal that the proposed algorithm also obtains the best clustering performance.

Group-based local adaptive deep multiple kernel learning with lp norm.

The deep multiple kernel Learning (DMKL) method has attracted wide attention due to its better classification performance than shallow multiple kernel learning. However, the existing DMKL methods are hard to find suitable global model parameters to improve classification accuracy in numerous datasets and do not take into account inter-class correlation and intra-class diversity. In this paper, we present a group-based local adaptive deep multiple kernel learning (GLDMKL) method with lp norm. Our GLDMKL method can divide samples into multiple groups according to the multiple kernel k-means clustering algorithm. The learning process in each well-grouped local space is exactly adaptive deep multiple kernel learning. And our structure is adaptive, so there is no fixed number of layers. The learning model in each group is trained independently, so the number of layers of the learning model maybe different. In each local space, adapting the model by optimizing the SVM model parameter α and the local kernel weight β in turn and changing the proportion of the base kernel of the combined kernel in each layer by the local kernel weight, and the local kernel weight is constrained by the lp norm to avoid the sparsity of basic kernel. The hyperparameters of the kernel are optimized by the grid search method. Experiments on UCI and Caltech 256 datasets demonstrate that the proposed method is more accurate in classification accuracy than other deep multiple kernel learning methods, especially for datasets with relatively complex data.

Challenges of Clustering Multimodal Clinical Data: Review of Applications in Asthma Subtyping

BackgroundIn the current era of personalized medicine, there is increasing interest in understanding the heterogeneity in disease populations. Cluster analysis is a method commonly used to identify subtypes in heterogeneous disease populations. The clinical data used in such applications are typically multimodal, which can make the application of traditional cluster analysis methods challenging.ObjectiveThis study aimed to review the research literature on the application of clustering multimodal clinical data to identify asthma subtypes. We assessed common problems and shortcomings in the application of cluster analysis methods in determining asthma subtypes, such that they can be brought to the attention of the research community and avoided in future studies.MethodsWe searched PubMed and Scopus bibliographic databases with terms related to cluster analysis and asthma to identify studies that applied dissimilarity-based cluster analysis methods. We recorded the analytic methods used in each study at each step of the cluster analysis process.ResultsOur literature search identified 63 studies that applied cluster analysis to multimodal clinical data to identify asthma subtypes. The features fed into the cluster algorithms were of a mixed type in 47 (75%) studies and continuous in 12 (19%), and the feature type was unclear in the remaining 4 (6%) studies. A total of 23 (37%) studies used hierarchical clustering with Ward linkage, and 22 (35%) studies used k-means clustering. Of these 45 studies, 39 had mixed-type features, but only 5 specified dissimilarity measures that could handle mixed-type features. A further 9 (14%) studies used a preclustering step to create small clusters to feed on a hierarchical method. The original sample sizes in these 9 studies ranged from 84 to 349. The remaining studies used hierarchical clustering with other linkages (n=3), medoid-based methods (n=3), spectral clustering (n=1), and multiple kernel k-means clustering (n=1), and in 1 study, the methods were unclear. Of 63 studies, 54 (86%) explained the methods used to determine the number of clusters, 24 (38%) studies tested the quality of their cluster solution, and 11 (17%) studies tested the stability of their solution. Reporting of the cluster analysis was generally poor in terms of the methods employed and their justification.ConclusionsThis review highlights common issues in the application of cluster analysis to multimodal clinical data to identify asthma subtypes. Some of these issues were related to the multimodal nature of the data, but many were more general issues in the application of cluster analysis. Although cluster analysis may be a useful tool for investigating disease subtypes, we recommend that future studies carefully consider the implications of clustering multimodal data, the cluster analysis process itself, and the reporting of methods to facilitate replication and interpretation of findings.

Composite Kernel and Hybrid Discriminative Random Field Model Based on Feature Fusion for PolSAR Image Classification

To effectively fuse the high-dimensional features, in this letter, we propose the composite kernel and hybrid discriminative random field model, abbreviated as CK-hybrid discriminative random field (HDRF), for polarimetric synthetic aperture radar (PolSAR) image classification. In the CK-HDRF model, given high-dimensional features with different characteristics, the unary potential is constructed by relating multiple kernel k-means (MKKM) clustering to the traditional HDRF model. In this way, the high-dimensional decomposition and texture features can be well fused, thus making their deserved contributions to the inference of the attributive class and further increasing the discrimination capacity of CK-HDRF. The pairwise potential is constructed by the generalized Ising model with an additional edge penalty function, and thus, it can well capture the underlying spatial relationship and maintain the edge locations in classification. Moreover, the statistics of PolSAR data are modeled by the Wishart-generalized gamma (WG Γ) distribution. Experiments on real PolSAR images demonstrate the effectiveness of CK-HDRF in classification.

Single-Cell RNA-Sequencing Data Clustering via Locality Preserving Kernel Matrix Alignment

Single-cell RNA-sequencing (scRNA-seq) data provide opportunities to reveal new insights into many biological problems such as elucidating cell types. An effective approach to elucidate cell types in complex tissues is to partition the cells into several separated subgroups via clustering techniques, where the cells in a specific cluster belong to the same cell type based on gene expression patterns. In this work, we present a novel multiple kernel clustering framework for scRNA-seq data clustering via locality preserving kernel alignment. Specifically, we first generate a series of similarity kernel matrices by using different kernel functions. Then we transfer the clustering task to a multiple kernel k-means problem with the kernels aligned in a local manner, i.e., the similarity of a sample to its k-nearest neighbours are aligned with the ideal similarity matrix. In our method, the clustering process focuses on closer sample pairs that shall stay together, and avoids involving unreliable similarity evaluation for farther sample pairs. In addition, we construct a local Laplacian matrix for each sample to constrain that closer samples should be allocated similar labels. In such a manner, the local structure of the data can be well preserved and utilized to produce better alignment for clustering. An alternate updating algorithm with theoretical analysis is developed to solve the proposed problem. We evaluate the performance of the proposed method on various real scRNA-seq data, and the results show that our method can obtain superior results when compared with other state-of-the-art approaches.