Consensus Kernel Research Articles

Joint consensus kernel learning and adaptive hypergraph regularization for graph-based clustering

Recent advancements in multiple kernel learning-based graph clustering methods have demonstrated significant promise in effectively learning a consensus kernel matrix from various candidate kernel matrices. This approach enables the creation of a low-dimensional representation in the kernel space of high-dimensional data through self-expressiveness. However, a key challenge remains in capturing the latent geometric properties embedded within different kernel matrices to enhance data representation. In this paper, we propose a novel method called joint consensus kernel learning and adaptive hypergraph regularization for graph-based clustering (JKHR). Our approach integrates an innovative adaptive hypergraph Laplacian regularizer, which is characterized by the fusion of multiple nearest neighbor kernel graphs, into the multiple kernel learning-based graph clustering framework. JKHR jointly and adaptively optimizes both the consensus kernel matrix and the hypergraph Laplacian regularizer to achieve a low-dimensional representation that effectively preserves the intrinsic geometry of the data. Experimental results on both synthetic and real benchmark datasets demonstrate that JKHR outperforms state-of-the-art self-expressiveness-based graph clustering methods as well as traditional clustering techniques.

Consensus local graph for multiple kernel clustering

Due to the ability to represent the relationship among data, graph has received extensive attention in clustering field. As illustrated by existing studies, learning graph in kernel space is an effective way to capture the structure of data. Among kernel-based methods, multiple kernel learning (MKL) is increasingly popular since it can automatically utilize the complementary information contained in base kernels. Although many existing MKL-based graph learning algorithms have promising learning abilities, we observe that they (1) put too much attention on learning the consensus kernel which probably contains lots of redundant information and loses the diversity of base kernels; (2) ignore the local structure of the representations in multiple kernel spaces. To eliminate the bad effect of these issues, we propose a novel graph learning method, termed consensus local graph based on MKL (CLGMKL), for clustering. In CLGMKL, a low-rank kernel matrix is used to extract the discriminative information of each base kernel and a local graph is constructed to capture the local structure contained in each new kernel. Then CLGMKL combines these local graphs in a self-weighed way. Since the above processes are associated with each other, we jointly optimize them to obtain the overall optimality. An effective learning scheme with proved convergence is developed to optimize the combined objective function. Finally, extensive experiments on some popular datasets are conducted to test the effectiveness of the presented method. As illustrated, CLGMKL is more competitive than the state-of-the-art algorithms.

Pure kernel graph fusion tensor subspace clustering under non-negative matrix factorization framework

Currently, graph-based multi-kernel subspace clustering methods have achieved rich research results in dealing with nonlinear data structures. However, most of the methods still have the following two limitations: (1) the model optimization goal is finding the optimal consensus kernel rather than the optimal affinity graph, which leads to kernel noise always interfering with the learning of the target affinity graph in the optimization process; (2) in the process of finding the optimal affinity graph, only the correlation between different kernel views is exploited, ignoring the higher-order correlation between multiple kernel views. In light of this, we propose a pure kernel graph fusion tensor (PKGT) subspace clustering under non-negative matrix factorization (NMF) framework. Specifically, we first use NMF to construct local affinity feature graphs with standard block diagonal structure for each base kernel matrix, and then achieve adaptive weight assignment by optimizing the inconsistencies among multiple local affinity feature graphs. Finally, we use tensor to capture the higher-order correlation between multiple local affinity feature graphs to provide more feature information for the learning of optimal target affinity graph, which enhances the model clustering performance. Extensive experiments on nine real datasets demonstrate that PKGT has superior clustering performance. The code is available at https://github.com/ZS0124/PKGT.

Multi-View Kernel Learning for Identification of Disease Genes.

Gene expression data sets and protein-protein interaction (PPI) networks are two heterogeneous data sources that have been extensively studied, due to their ability to capture the co-expression patterns among genes and their topological connections. Although they depict different traits of the data, both of them tend to group co-functional genes together. This phenomenon agrees with the basic assumption of multi-view kernel learning, according to which different views of the data contain a similar inherent cluster structure. Based on this inference, a new multi-view kernel learning based disease gene identification algorithm, termed as DiGId, is put forward. A novel multi-view kernel learning approach is proposed that aims to learn a consensus kernel, which efficiently captures the heterogeneous information of individual views as well as depicts the underlying inherent cluster structure. Some low-rank constraints are imposed on the learned multi-view kernel, so that it can effectively be partitioned into k or fewer clusters. The learned joint cluster structure is used to curate a set of potential disease genes. Moreover, a novel approach is put forward to quantify the importance of each view. In order to demonstrate the effectiveness of the proposed approach in capturing the relevant information depicted by individual views, an extensive analysis is performed on four different cancer-related gene expression data sets and PPI network, considering different similarity measures.

Consensus kernel subspace clustering based on coefficient discriminant information

At present, multiple kernel subspace clustering technology has been extensively used in various fields of machine learning and computer vision. However, the traditional multiple kernel subspace clustering method still has two problems. On the one hand, the combination of kernel functions is uncertain. Choosing inappropriate kernel functions from different sources or data with different characteristics will make the results inaccurate. On the other hand, data sources are extensive and subspace division is not accurate. In this paper, we unify coefficient discriminant information and multiple kernel subspace clustering into one process. Firstly, we combine the consensus Hilbert space to propose a consensus kernel method. This method reconstructs a set of different kernel matrices into kernel matrices more in line with the data characteristics, which can effectively solve the uncertainty of the combination of kernel functions. Secondly, the diagonal structure of blocks is constructed by subspace clustering, which can effectively solve the problems of different data sources. In the process, we propose a coefficient discriminant information which can constrain the data in the coefficient space to group as many of the same types of data as possible, and separate different types of data as much as possible. Finally, we propose a new iterative optimization method based on the Alternating Direction Multiplier Method (ADMM) to solve the final optimization goal. Our experiment is conducted on twenty different data sets. According to the final experimental results, our proposed algorithm has higher ACC, NMI and ARI than the latest clustering methods.

Enhanced Soft 3D Reconstruction Method with an Iterative Matching Cost Update Using Object Surface Consensus.

In this paper, we propose a multi-view stereo matching method, EnSoft3D (Enhanced Soft 3D Reconstruction) to obtain dense and high-quality depth images. Multi-view stereo is one of the high-interest research areas and has wide applications. Motivated by the Soft3D reconstruction method, we introduce a new multi-view stereo matching scheme. The original Soft3D method is introduced for novel view synthesis, while occlusion-aware depth is also reconstructed by integrating the matching costs of the Plane Sweep Stereo (PSS) and soft visibility volumes. However, the Soft3D method has an inherent limitation because the erroneous PSS matching costs are not updated. To overcome this limitation, the proposed scheme introduces an update process of the PSS matching costs. From the object surface consensus volume, an inverse consensus kernel is derived, and the PSS matching costs are iteratively updated using the kernel. The proposed EnSoft3D method reconstructs a highly accurate 3D depth image because both the multi-view matching cost and soft visibility are updated simultaneously. The performance of the proposed method is evaluated by using structured and unstructured benchmark datasets. Disparity error is measured to verify 3D reconstruction accuracy, and both PSNR and SSIM are measured to verify the simultaneous enhancement of view synthesis.

Robust multiple kernel subspace clustering with block diagonal representation and low-rank consensus kernel

Multiple kernel learning (MKL) is widely used to deal with nonlinear clustering problem, because MKL can obtain the optimal consensus kernel by designing kernel weighting strategy, which avoids the limitation of single-kernel learning restricted by the selected kernel function. However, existing MKL subspace clustering algorithms ignore noise and the low-rank structure of the data in the feature space. We propose a new robust multiple kernel subspace clustering algorithm (LRMKSC) with block diagonal representation (BDR) and low-rank consensus kernel (LRCK), which is more systematic and comprehensive for data processing. In particular, (1) to learn the optimal consensus kernel, we design an automatic weighting strategy using Mixture Correntropy Induced Metric (MCIM), which not only sets the optimal weight for each kernel, but also improves the robustness of LRMKSC by suppressing noise; (2) to explore low-rank structure of input data in feature space, we propose the Weighted Truncated Schatten p-Norm (WTSN) and apply it to the low-rank constraint of the optimal consensus kernel; (3) considering the block diagonal property of the affinity matrix, we apply block diagonal constraint to the coefficient matrix. LRMKSC combines MKL, LRCK and BDR to solve these problems simultaneously. Through the interaction of three technologies, the results of other technologies are used in the overall optimal solution to iteratively improve the efficiency of each technology, and finally form an overall optimal algorithm for processing nonlinear structural data. Compared with the most advanced MKL subspace clustering algorithms, extensive experiments on image and text datasets verify the competitiveness of LRMKSC.

Hierarchical Multiple Kernel Clustering

Current multiple kernel clustering algorithms compute a partition with the consensus kernel or graph learned from the pre-specified ones, while the emerging late fusion methods firstly construct multiple partitions from each kernel separately, and then obtain a consensus one with them. However, both of them directly distill the clustering information from kernels or graphs to partition matrices, where the sudden dimension drop would result in loss of advantageous details for clustering. In this paper, we provide a brief insight of the aforementioned issue and propose a hierarchical approach to perform clustering while preserving advantageous details maximumly. Specifically, we gradually group samples into fewer clusters, together with generating a sequence of intermediary matrices of descending sizes. The consensus partition with is simultaneously learned and conversely guides the construction of intermediary matrices. Nevertheless, this cyclic process is modeled into an unified objective and an alternative algorithm is designed to solve it. In addition, the proposed method is validated and compared with other representative multiple kernel clustering algorithms on benchmark datasets, demonstrating state-of-the-art performance by a large margin.

Multiple Kernel Clustering with Kernel k-Means Coupled Graph Tensor Learning

Kernel k-means (KKM) and spectral clustering (SC) are two basic methods used for multiple kernel clustering (MKC), which have both been widely used to identify clusters that are non-linearly separable. However, both of them have their own shortcomings: 1) the KKM-based methods usually focus on learning a discrete clustering indicator matrix via a combined consensus kernel, but cannot exploit the high-order affinities of all pre-defined base kernels; and 2) the SC-based methods require a robust and meaningful affinity graph in kernel space as input in order to form clusters with desired clustering structure. In this paper, a novel method, kernel k-means coupled graph tensor (KCGT), is proposed to graciously couple KKM and SC for seizing their merits and evading their demerits simultaneously. In specific, we innovatively develop a new graph learning paradigm by leveraging an explicit theoretical connection between clustering indicator matrix and affinity graph, such that the affinity graph propagated from KKM enjoys the valuable block diagonal and sparse property. Then, by using this graph learning paradigm, base kernels can produce multiple candidate affinity graphs, which are stacked into a low-rank graph tensor for capturing the high-order affinity of all these graphs. After that, by averaging all the frontal slices of the tensor, a high-quality affinity graph is obtained. Extensive experiments have shown the superiority of KCGT compared with the state-of-the-art MKC methods.

Consensus Affinity Graph Learning for Multiple Kernel Clustering.

Significant attention to multiple kernel graph-based clustering (MKGC) has emerged in recent years, primarily due to the superiority of multiple kernel learning (MKL) and the outstanding performance of graph-based clustering. However, many existing MKGC methods design a fat model that poses challenges for computational cost and clustering performance, as they learn both an affinity graph and an extra consensus kernel cumbersomely. To tackle this challenging problem, this article proposes a new MKGC method to learn a consensus affinity graph directly. By using the self-expressiveness graph learning and an adaptive local structure learning term, the local manifold structure of the data in kernel space is preserved for learning multiple candidate affinity graphs from a kernel pool first. After that, these candidate affinity graphs are synthesized to learn a consensus affinity graph via a thin autoweighted fusion model, in which a self-tuned Laplacian rank constraint and a top- k neighbors sparse strategy are introduced to improve the quality of the consensus affinity graph for accurate clustering purposes. The experimental results on ten benchmark datasets and two synthetic datasets show that the proposed method consistently and significantly outperforms the state-of-the-art methods.

Simultaneous Global and Local Graph Structure Preserving for Multiple Kernel Clustering.

Multiple kernel learning (MKL) is generally recognized to perform better than single kernel learning (SKL) in handling nonlinear clustering problem, largely thanks to MKL avoids selecting and tuning predefined kernel. By integrating the self-expression learning framework, the graph-based MKL subspace clustering has recently attracted considerable attention. However, the graph structure of data in kernel space is largely ignored by previous MKL methods, which is a key concept of affinity graph construction for spectral clustering purposes. In order to address this problem, a novel MKL method is proposed in this article, namely, structure-preserving multiple kernel clustering (SPMKC). Specifically, SPMKC proposes a new kernel affine weight strategy to learn an optimal consensus kernel from a predefined kernel pool, which can assign a suitable weight for each base kernel automatically. Furthermore, SPMKC proposes a kernel group self-expressiveness term and a kernel adaptive local structure learning term to preserve the global and local structure of the input data in kernel space, respectively, rather than the original space. In addition, an efficient algorithm is proposed to solve the resulting unified objective function, which iteratively updates the consensus kernel and the affinity graph so that collaboratively promoting each of them to reach the optimum condition. Experiments on both image and text clustering demonstrate that SPMKC outperforms the state-of-the-art MKL clustering methods in terms of clustering performance and computational cost.

Multikernel Clustering via Non-Negative Matrix Factorization Tailored Graph Tensor Over Distributed Networks

Next-generation wireless networks are witnessing an increasing number of clustering applications, and produce a large amount of non-linear and unlabeled data. In some degree, single kernel methods face the challenging problem of kernel choice. To overcome this problem for non-linear data clustering, multiple kernel graph-based clustering (MKGC) has attracted intense attention in recent years. However, existing MKGC methods suffer from two common problems: (1) they mainly aim to learn a consensus kernel from multiple candidate kernels, slight affinity graph learning, such that cannot fully exploit the underlying graph structure of non-linear data; (2) they disregard the high-order correlations between all base kernels, which cannot fully capture the consistent and complementary information of all kernels. In this paper, we propose a novel non-negative matrix factorization (NMF) tailored graph tensor MKGC method for non-linear data clustering, namely TMKGC. Specifically, TMKGC integrates NMF and graph learning together in kernel space so as to learn multiple candidate affinity graphs. Afterwards, the high-order structure information of all candidate graphs is captured in a 3-order tensor kernel space by introducing tensor singular value decomposition based tensor nuclear norm, such that an optimal affinity graph can be obtained subsequently. Based on the alternating direction method of multipliers, the effective local and distributed solvers are elaborated to solve the proposed objective function. Extensive experiments have demonstrated the superiority of TMKGC compared to the state-of-the-art MKGC methods.

Multiple Kernel k-Means With Low-Rank Neighborhood Kernel

Multiple kernel clustering algorithms achieve promising performances by exploring the complementary information from kernel matrices corresponding to each data view. Most of the existing methods aim to construct a consensus kernel for the afterward clustering. However, they ignore that the desired kernel is supposed to reveal the cluster structure among samples and thus to be low rank. As a consequence, the corresponding clustering performance could decrease. To address this issue, we propose a low-rank kernel learning approach for multiple kernel clustering. Specifically, instead of regularizing the consensus kernel with low-rank constraints, we use a re-parameterize scheme for the kernel matrix. Meanwhile, the consensus kernel is located in the neighborhood area of the linear combination of base kernels. An alternate optimization strategy is designed to solve the resulting optimization problem. We evaluate the proposed method on 13 benchmark datasets with 9 state-of-the-art algorithms. As is demonstrated by experimental results, our proposed algorithm achieves superior clustering scores against the compared algorithms on the reported popular multiple kernel datasets.

Multiple kernel clustering with pure graph learning scheme

Existing graph-based multiple kernel clustering (GMKC) methods usually adopt a hybridization scheme, termed HGMKC for short, which binds multiple kernel learning (MKL) and graph-based clustering (GBC) together, the former aims to construct a consensus kernel and the latter is used to learn an affinity graph based on the consensus kernel. Obviously, HGMKC performs graph learning and spectral clustering separately. Additionally, the performance of this hybridization is largely determined by MKL, which is empirically contrary to the fact that graph learning is the key of graph-based methods, rather than kernel learning. In this paper, we propose a pure GMKC method, dubbed PGMKC, which contains two parts, including candidate kernel graphs learning (CKGL) and kernel graph fusion (KGF). Specially, CKGL concentrates on constructing multiple candidate kernel graphs in Hilbert kernel space relying on kernel self-expressiveness; KGF leverages a flexibly auto-weighted graph fusion strategy and a connectivity regularizer to yield a consensus kernel graph directly. Compared to HGMKC, PGMKC avoids paying too much attention to consensus kernel construction, thus can focus more on affinity graph learning. Moreover, an efficient and effective optimization algorithm is developed to solve the proposed model. Experimental results on ten benchmark datasets demonstrate that the proposed PGMKC performs better than the state-of-the-art HGMKC competitors.

Simultaneous learning coefficient matrix and affinity graph for multiple kernel clustering

Due to the effectiveness of handling non-linear data and avoiding kernel customization, multiple kernel clustering (MKC) has been widely investigated and achieved promising results for challenging non-linear clustering tasks. Generally, the existing MKC methods mainly consist of first learning a coefficient matrix by leveraging multiple kernel learning (MKL) and sample self-expressiveness property, and then constructing an affinity graph relying on this coefficient matrix to accomplish spectral clustering. However, the quality of the affinity matrix (graph) is largely determined by the learned coefficient matrix, thus the two independent steps are not conducive to learn an optimal affinity graph. To tackle this problem, in this paper, we propose a new MKC method that uses one-step learning scheme rather than two-step learning scheme to learn an affinity graph, termed SLMKC. Specifically, SLMKC bridges the relationship between the affinity matrix and coefficient matrix by an adaptive local structure learning strategy, so it can simultaneously learn both in a mutual promotion manner. Furthermore, a self-weighted MKL strategy is introduced to learn an optimal consensus kernel, which can avoid selecting a specific kernel function and tuning its associated parameters. Extensive experiments validate that our SLMKC outperforms the state-of-the-art MKC competitors significantly.

Multiple Kernel Driven Clustering With Locally Consistent and Selfish Graph in Industrial IoT

In the cognitive computing of intelligent industrial Internet of Things, clustering is a fundamental machine learning problem to exploit the latent data relationships. To overcome the challenge of kernel choice for nonlinear clustering tasks, multiple kernel clustering (MKC) has attracted intensive attention. However, existing graph-based MKC methods mainly aim to learn a consensus kernel as well as an affinity graph from multiple candidate kernels, which cannot fully exploit the latent graph information. In this article, we propose a novel pure graph-based MKC method. Specifically, a new graph model is proposed to preserve the local manifold structure of the data in kernel space so as to learn multiple candidate graphs. Afterward, the latent consistency and selfishness of these candidate graphs are fully considered. Furthermore, a graph connectivity constraint is introduced to avoid requiring any postprocessing clustering step. Comprehensive experimental results demonstrate the superiority of our method.

Robust Multi-View Subspace Clustering Via Weighted Multi-Kernel Learning and Co-Regularization

Using multi-kernel learning to deal with the non-linear relationship of data has become a new research topic in the field of multi-view subspace clustering. However, the existing methods have the following three defects: 1) the simple consensus kernel weighting strategy cannot give full play to the advantages of multiple kernels; 2) they are sensitive to non-Gaussian noise and their learning affinity matrices cannot meet the block diagonal properties required by clustering, resulting in low clustering performance; 3) the complementary feature information between the data of each view cannot be fully mined. In this paper, a novel robust multi-view subspace clustering method is proposed based on weighted multi-kernel learning and co-regularization (WMKMSC). Based on the self-expression learning framework, block diagonal regularizer (BDR), multi-kernel learning strategy and co-regularization are integrated into the proposed model. Especially, as a robust learning method, the mixture correntropy is used to construct a robust multi-kernel weighting strategy, which is helpful to learn the best consensus kernel. Our method is more effective and robust than several of the most advanced methods on five commonly used datasets.

Multiple kernel subspace clustering with local structural graph and low-rank consensus kernel learning

Multiple kernel learning (MKL) methods are generally believed to perform better than single kernel learning (SKL) methods in handling nonlinear subspace clustering problem, largely thanks to MKL avoids selecting and tuning a pre-defined kernel. However, previous MKL methods mainly focused on how to define a kernel weighting strategy, but ignored the structural characteristics of the input data in both the original space and the kernel space. In this paper, we first propose a novel graph-based MKL method for subspace clustering, namely, Local Structural Graph and Low-Rank Consensus Multiple Kernel Learning (LLMKL). It jointly learns an optimal affinity graph and a suitable consensus kernel for clustering purpose by elegantly integrating the MKL technology, the global structure in the kernel space, the local structure in the original space, and the Hilbert space self-expressiveness property in a unified optimization model. In particular, to capture the data global structure, we employ a substitute of the desired consensus kernel, and then introduce a low-rank constraint on the substitute to encourage that the structure of linear subspaces is present in the feature space. Moreover, the data local structure is explored by building a complete graph, where each sample is treated as a node, and an edge codes the pairwise affinity between two samples. By such, the consensus kernel learning and the affinity graph learning can promote each other such that the data in resulting Hilbert space are both self-expressive and low-rank. Experiments on both image and text clustering well demonstrate that LLMKL outperforms the state-of-the-art methods.

Joint correntropy metric weighting and block diagonal regularizer for robust multiple kernel subspace clustering

Nonlinear kernel-based subspace clustering methods that can reveal the multi-cluster nonlinear structure of samples are an emerging research topic. However, the existing kernel subspace clustering methods have the following three flaws: 1) their clustering performance is largely determined by the chosen kernel function; 2) they may lack robustness in the presence of non-Gaussian noise and impulsive noise; and 3) their learned affinity matrix can not hold the desired block diagonal property for clustering purpose, which possibly leads to incorrect clustering when using spectral clustering. In this paper, we propose a Joint Robust Multiple Kernel Subspace Clustering (JMKSC) method for data clustering, which has two primary innovations. First, our multiple kernel weighting strategy introduces the correntropy metric weighting instead of a fixed, or inappropriately assigned weighting, which is more robust to the non-Gaussian noise and contributes to learning the optimal consensus kernel. Second, our method encourages acquiring an affinity matrix with the optimal block diagonal property based on the block diagonal regularizer (BDR) and the self-expressiveness property. Experiments on several different types of datasets confirm that the proposed JMKSC significantly outperforms several state-of-the-art single kernel and multiple kernel subspace clustering methods in terms of accuracy, NMI and purity.

Unsupervised Robust Multiple Kernel Learning via Extracting Local and Global Noises

Kernel-based clustering methods can capture the non-linear structure and identify arbitrarily shaped clusters, so they have been widely used in machine learning tasks. Since the performance of kernel methods critically depends on the choices of kernels, multiple kernel learning methods are proposed to alleviate the effort for kernel designing. The conventional multiple kernel learning methods learn a consensus kernel by linearly combining all candidate kernels, whereas ignoring the influence of the noises. To improve the robustness of multiple kernel learning methods, in this paper, we analyze the local and global noises and design regularized terms to characterize them respectively. Then, we propose a novel local and global de-noising multiple kernel learning method which can explicitly extract the local and global noises to recover the clean kernels for multiple kernel learning. After that, a block coordinate descent algorithm is presented to solve the optimization problem. Finally, the extensive experiments on benchmark data sets will demonstrate the effectiveness of the proposed algorithm.