Kernel Clustering Research Articles

커널을 이용한 전역 클러스터링의 비선형화

Fuzzy c-means(FCM)는 퍼지 집합을 응용한 간단하지만 효율적인 클러스터링 방법 중 하나이다. FCM은 여러 응용 분야에서 성공적으로 활용되어 왔지만, 초기화와 잡음에 민감하고 볼록한 형태의 클러스터들만 다룰 수 있는 문제점이 있다. 이 논문에서는 이러한 FCM의 문제점을 해결하기 위해 전역 클러스터링(global clustering) 기법과 커널 클러스터링(kernel clustering) 기법을 결합하여 새로운 비선형 클러스터링 기법인 커널 전역 FCM(kernel global fuzzy c-means, KG-FCM)을 제안한다. 전역 클러스터링은 클러스터링의 초기화를 위한 방법 중 하나로, 순차적으로 클러스터를 하나씩 추가함으로써 초기화에 민감한 FCM의 한계를 극복할 수 있도록 해준다. FCM의 잡음 민감성과 볼록한 클러스터들만 다룰 수 있는 한계를 극복하기 위한 방법은 여러 가지가 있으며 커널 클러스터링이 그 중 하나이다. 커널 클러스터링은 사용하는 커널을 바꿈으로써 쉽게 확장이 가능하므로 이 논문에서는 커널 클러스터링을 사용하였다. 두 방법을 결합함으로써 제안한 방법은 위에서 언급한 문제점들을 해결할 수 있으며, 이는 가상 및 실제 데이터를 이용한 실험 결과를 통해 확인할 수 있다. Fuzzy c-means (FCM) is a simple but efficient clustering algorithm using the concept of a fuzzy set that has been proved to be useful in many areas. There are, however, several well known problems with FCM, such as sensitivity to initialization, sensitivity to outliers, and limitation to convex clusters. In this paper, global fuzzy c-means (G-FCM) and kernel fuzzy c-means (K-FCM) are combined to form a non-linear variant of G-FCM, called kernel global fuzzy c-means (KG-FCM). G-FCM is a variant of FCM that uses an incremental seed selection method and is effective in alleviating sensitivity to initialization. There are several approaches to reduce the influence of noise and accommodate non-convex clusters, and K-FCM is one of them. K-FCM is used in this paper because it can easily be extended with different kernels. By combining G-FCM and K-FCM, KG-FCM can resolve the shortcomings mentioned above. The usefulness of the proposed method is demonstrated by experiments using artificial and real world data sets.

A weighted cluster kernel PCA prediction model for multi-subject brain imaging data

Brain imaging data have shown great promise as a useful predictor for psychiatric conditions, cognitive functions and many other neural-related outcomes. Development of prediction models based on imaging data is challenging due to the high dimensionality of the data, noisy measurements, complex correlation structures among voxels, small sample sizes, and between-subject heterogeneity. Most existing prediction approaches apply a dimension reduction method such as PCA on whole brain images as a preprocessing step. These approaches usually do not take into account of the cluster structure among voxels and between-subject differences. We propose a weighted cluster kernel PCA predictive model that addresses the challenges in brain imaging data. We first divide voxels into clusters based on neuroanatomic parcellation or data-driven methods, then extract cluster-specific principal features using kernel PCA and define the prediction model based on the principal features. Finally, we propose a weighted estimation method for the prediction model where each subject is weighted according to the percent of variance explained by the principal features. The proposed method allows assessment of relative importance of various brain regions in prediction; captures nonlinearity in feature space; and helps guard against overfitting for outlying subjects in predictive model building. We evaluate the performance of our method through simulation studies. A real fMRI data example is also used to illustrate the method.

Approximate Kernel Clustering

In the kernel clustering problem we are given a large $n\times n$ positive semi-definite matrix $A=(a_{ij})$ with $\sum_{i,j=1}^na_{ij}=0$ and a small $k\times k$ positive semi-definite matrix $B=(b_{ij})$. The goal is to find a partition $S_1,...,S_k$ of $\{1,... n\}$ which maximizes the quantity $$ \sum_{i,j=1}^k (\sum_{(i,j)\in S_i\times S_j}a_{ij})b_{ij}. $$ We study the computational complexity of this generic clustering problem which originates in the theory of machine learning. We design a constant factor polynomial time approximation algorithm for this problem, answering a question posed by Song, Smola, Gretton and Borgwardt. In some cases we manage to compute the sharp approximation threshold for this problem assuming the Unique Games Conjecture (UGC). In particular, when $B$ is the $3\times 3$ identity matrix the UGC hardness threshold of this problem is exactly $\frac{16\pi}{27}$. We present and study a geometric conjecture of independent interest which we show would imply that the UGC threshold when $B$ is the $k\times k$ identity matrix is $\frac{8\pi}{9}(1-\frac{1}{k})$ for every $k\ge 3$.

Semi-supervised clustering with metric learning: An adaptive kernel method

Most existing representative works in semi-supervised clustering do not sufficiently solve the violation problem of pairwise constraints. On the other hand, traditional kernel methods for semi-supervised clustering not only face the problem of manually tuning the kernel parameters due to the fact that no sufficient supervision is provided, but also lack a measure that achieves better effectiveness of clustering. In this paper, we propose an adaptive Semi-supervised Clustering Kernel Method based on Metric learning (SCKMM) to mitigate the above problems. Specifically, we first construct an objective function from pairwise constraints to automatically estimate the parameter of the Gaussian kernel. Then, we use pairwise constraint-based K-means approach to solve the violation issue of constraints and to cluster the data. Furthermore, we introduce metric learning into nonlinear semi-supervised clustering to improve separability of the data for clustering. Finally, we perform clustering and metric learning simultaneously. Experimental results on a number of real-world data sets validate the effectiveness of the proposed method.

Bootstrap confidence intervals for reservoir model selection techniques

Stochastic spatial simulation allows generation of multiple realizations of spatial variables. Due to the computational time required for evaluating the transfer function, uncertainty quantification of these multiple realizations often requires a selection of a small subset of realization. However, by selecting only a few realizations, one may risk biasing the P10, P50, and P90 estimates as compared to the original multiple realizations. The objective of this study is to develop a methodology to quantify confidence intervals for the estimated P10, P50, and P90 quantiles when only a few models are retained for response evaluation. We use the parametric bootstrap technique, which evaluates the variability of the statistics obtained from uncertainty quantification and constructs confidence intervals. Using this technique, we compare the confidence intervals when using two selection methods: the traditional ranking technique and the distance-based kernel clustering technique (DKM). The DKM has been recently developed and has been shown to be effective in quantifying uncertainty. The methodology is demonstrated using two examples. The first example is a synthetic example, which uses bi-normal variables and serves to demonstrate the technique. The second example is from an oil field in West Africa where the uncertain variable is the cumulative oil production coming from 20 wells. The results show that, for the same number of transfer function evaluations, the DKM method has equal or smaller error and confidence interval compared to ranking.

Uncertainty Quantification in Reservoir Performance Using Distances and Kernel Methods—Application to a West Africa Deepwater Turbidite Reservoir

SummaryStochastic simulation allows generating multiple reservoir models that can be used to characterize reservoir uncertainty. In many practical situations, the large computation time needed for flow simulation does not allow an evaluation of flow on all reservoir models. In this paper, we propose a method to select a subset of reservoir models reflecting the same uncertainty in flow response as the full set. Using the concept of distance, we map the reservoir models to a low-dimensional space where kernel clustering is applied to identify a subset of representative reservoir models of the entire set. Flow simulation and subsequently uncertainty quantification are performed on this subset. A case study is presented of an architecturally complex deepwater turbidite offshore reservoir with large uncertainty in the type of depositional system present.

Semi-supervised Kernel-based Fuzzy Clustering for Gear Outlier Detection

Kernel clustering is investigated in mechanical fault detection,and a semi-supervised kernel-based fuzzy clustering method is presented for gear fault early detection. The difficulty in early detection of mechanical incipient fault is to extract the weak fault information in noises. The semi-supervised kernel clustering method utilizes a few of known samples,combined with a larger amount of unknown samples to perform semi-supervised learning,and obtains good efficiency. The experiments are conducted on a gearbox,where a surface defect of tooth pitting is introduced.The result of semi-supervised kernel clustering is compared with that of unsupervised kernel clustering,which demonstrates the superiority of the semi-supervised method for gear failure detection.

クラスタリング技法とカーネル関数

クラスタリング技法とカーネル関数

Graph nodes clustering with the sigmoid commute-time kernel: A comparative study

This work addresses the problem of detecting clusters in a weighted, undirected, graph by using kernel-based clustering methods, directly partitioning the graph according to a well-defined similarity measure between the nodes (a kernel on a graph). The proposed algorithms are based on a two-step procedure. First, a kernel or similarity matrix, providing a meaningful similarity measure between any couple of nodes, is computed from the adjacency matrix of the graph. Then, the nodes of the graph are clustered by performing a kernel clustering on this similarity matrix. Besides the introduction of a prototype-based kernel version of the gaussian mixtures model and Ward’s hierarchical clustering, in addition to the already known kernel k-means and fuzzy k-means, a new kernel, called the sigmoid commute-time kernel ( K CT S ) is presented. The joint use of the K CT S kernel matrix and kernel clustering appears to be quite effective. Indeed, this methodology provides the best results on a systematic comparison with a selection of graph clustering and communities detection algorithms on three real-world databases. Finally, some links between the proposed hierarchical kernel clustering and spectral clustering are examined.

Kernel-based spectral color image segmentation

In this work, we propose a new algorithm for spectral color image segmentation based on the use of a kernel matrix. A cost function for spectral kernel clustering is introduced to measure the correlation between clusters. An efficient multiscale method is presented for accelerating spectral color image segmentation. The multiscale strategy uses the lattice geometry of images to construct an image pyramid whose hierarchy provides a framework for rapidly estimating eigenvectors of normalized kernel matrices. To prevent the boundaries from deteriorating, the image size on the top level of the pyramid is generally required to be around 75 x 75, where the eigenvectors of normalized kernel matrices would be approximately solved by the Nyström method. Within this hierarchical structure, the coarse solution is increasingly propagated to finer levels and is refined using subspace iteration. In addition, to make full use of the abundant color information contained in spectral color images, we propose using spectrum extension to incorporate the geometric features of spectra into similarity measures. Experimental results have shown that the proposed method can perform significantly well in spectral color image segmentation as well as speed up the approximation of the eigenvectors of normalized kernel matrices.

A new information theoretic analysis of sum-of-squared-error kernel clustering

The contribution of this paper is to provide a new input space analysis of the properties of sum-of-squared-error K-means clustering performed in a Mercer kernel feature space. Such an analysis has been missing until now, even though kernel K-means has been popular in the clustering literature. Our derivation extends the theory of traditional K-means from properties of mean vectors to information theoretic properties of Parzen window estimated probability density functions (pdfs). In particular, Euclidean distance-based kernel K-means is shown to maximize an integrated squared error divergence measure between cluster pdfs and the overall pdf of the data, while a cosine similarity-based approach maximizes a Cauchy–Schwarz divergence measure. Furthermore, the iterative rules which assign data points to clusters in order to maximize these criteria are shown to depend on the cluster pdfs evaluated at the data points, in addition to the Renyi entropies of the clusters. The Bayes rule is shown to be a special case.

Representing Spatial Uncertainty Using Distances and Kernels

Assessing uncertainty of a spatial phenomenon requires the analysis of a large number of parameters which must be processed by a transfer function. To capture the possibly of a wide range of uncertainty in the transfer function response, a large set of geostatistical model realizations needs to be processed. Stochastic spatial simulation can rapidly provide multiple, equally probable realizations. However, since the transfer function is often computationally demanding, only a small number of models can be evaluated in practice, and are usually selected through a ranking procedure. Traditional ranking techniques for selection of probabilistic ranges of response (P10, P50 and P90) are highly dependent on the static property used. In this paper, we propose to parameterize the spatial uncertainty represented by a large set of geostatistical realizations through a distance function measuring “dissimilarity” between any two geostatistical realizations. The distance function allows a mapping of the space of uncertainty. The distance can be tailored to the particular problem. The multi-dimensional space of uncertainty can be modeled using kernel techniques, such as kernel principal component analysis (KPCA) or kernel clustering. These tools allow for the selection of a subset of representative realizations containing similar properties to the larger set. Without losing accuracy, decisions and strategies can then be performed applying a transfer function on the subset without the need to exhaustively evaluate each realization. This method is applied to a synthetic oil reservoir, where spatial uncertainty of channel facies is modeled through multiple realizations generated using a multi-point geostatistical algorithm and several training images.

Nonlinear clustering-based support vector machine for large data sets

This paper presents a kernel clustering-based support vector machine (KCB-SVM) that generalizes the linear clustering-based support vector machine (CB-SVM) to solve nonlinear classification problems in a novel way. It can not only handles large data sets, but can also have nonlinear discriminant power. By introducing kernel clustering, KCB-SVM unifies the metrics in the clustering and training stages. Elaborately designed clustering features summarize all the information required for further clustering and training, which allows only one scan of the total data sets. Experiments on both artificial and real large data sets show that the KCB-SVM not only achieves better classification accuracy than random sampling, active learning and CB-SVM, but also retains the ability to handle large data sets.

Cluster Kernels: Resource-Aware Kernel Density Estimators over Streaming Data

A variety of real-world applications heavily relies on an adequate analysis of transient data streams. Due to the rigid processing requirements of data streams, common analysis techniques as known from data mining are not directly applicable. A fundamental building block of many data mining and analysis approaches is density estimation. It provides a well-defined estimation of a continuous data distribution, a fact, which makes its adaptation to data streams desirable. A convenient method for density estimation utilizes kernels. The computational complexity of kernel density estimation, however, renders its application to data streams impossible. In this paper, we tackle this problem and propose our Cluster Kernel approach which provides continuously computed kernel density estimators over streaming data. Not only do Cluster Kernels meet the rigid processing requirements of data streams, they also allocate only a constant amount of memory, even with the opportunity to adapt it dynamically to changing system resources. For this purpose, we develop an intelligent merge scheme for Cluster Kernels and utilize continuously collected local statistics to resample already processed data. We focus on Cluster Kernels for one-dimensional data streams, but also address the multi-dimensional case. We validate the efficacy of Cluster Kernels for a variety of real-world data streams in an extensive experimental study.

Fuzzy Kernel Clustering of RNA Secondary Structure Ensemble Using a Novel Similarity Metric

Measuring the (dis)similarity between RNA secondary structures is critical for the study of RNA secondary structures and has implications to RNA functional characterization. Although a number of methods have been developed for comparing RNA structural similarities, their applications have been limited by the complexity of the required computation. In this paper, we present a novel method for comparing the similarity of RNA secondary structures generated from the same RNA sequence, i.e., a secondary structure ensemble, using a matrix representation of the RNA structures. Relevant features of the RNA secondary structures can be easily extracted through singular value decomposition (SVD) of the representing matrices. We have mapped the feature vectors of the singular values to a kernel space, where (dis)similarities among the mapped feature vectors become more evident, making clustering of RNA secondary structures easier to handle. The pair-wise comparison of RNA structures is achieved through computing the distance between the singular value vectors in the kernel space. We have applied a fuzzy kernel clustering method, using this similarity metric, to cluster the RNA secondary structure ensembles. Our application results suggest that our fuzzy kernel clustering method is highly promising for classifications of RNA structure ensembles, because of its low computational complexity and high clustering accuracy.

Selecting marker genes for cancer classification using supervised weighted kernel clustering and the support vector machine

Due to recent interest in the analysis of DNA microarray data, new methods have been considered and developed in the area of statistical classification. In particular, according to the gene expression profile of existing data, the goal is to classify the sample into a relevant diagnostic category. However, when classifying outcomes into certain cancer types, it is often the case that some genes are not important, while some genes are more important than others. A novel algorithm is presented for selecting such relevant genes referred to as marker genes for cancer classification. This algorithm is based on the Support Vector Machine (SVM) and Supervised Weighted Kernel Clustering (SWKC). To investigate the performance of this algorithm, the methods were applied to a simulated data set and some real data sets. For comparison, some other well-known methods such as Prediction Analysis of Microarrays (PAM), Support Vector Machine-Recursive Feature Elimination (SVM-RFE), and a Structured Polychotomous Machine (SPM) were considered. The experimental results indicate that the proposed SWKC/SVM algorithm is conceptually much simpler and performs more efficiently than other existing methods used in identifying marker genes for cancer classification. Furthermore, the SWKC/SVM algorithm has the advantage that it requires much less computing time compared with the other existing methods.

MRI brain image segmentation and bias field correction based on fast spatially constrained kernel clustering approach

A fast spatially constrained kernel clustering algorithm is proposed for segmenting medical magnetic resonance imaging (MRI) brain images and correcting intensity inhomogeneities known as bias field in MRI data. The algorithm using kernel technique implicitly maps image data to a higher dimensional kernel space in order to improve the separability of data and provide more potential for effectively segmenting MRI data. Based on the technique, a speed-up scheme for kernel clustering and an approach for correcting spurious intensity variation of MRI images have been implemented. The fast kernel clustering and bias field correcting benefit each other in an iterative matter and have dramatically reduced the time complexity of kernel clustering. The experiments on simulated brain phantoms and real clinical MRI data have shown that the proposed algorithm generally outperforms the corresponding traditional algorithms when segmenting MRI data corrupted by high noise and gray bias field.

Artificial immune kernel clustering network for unsupervised image segmentation

An immune kernel clustering network (IKCN) is proposed based on the combination of the artificial immune network and the support vector domain description (SVDD) for the unsupervised image segmentation. In the network, a new antibody neighborhood and an adaptive learning coefficient, which is inspired by the long-term memory in cerebral cortices are presented. Starting from IKCN algorithm, we divide the image feature sets into subsets by the antibodies, and then map each subset into a high dimensional feature space by a mercer kernel, where each antibody neighborhood is represented as a support vector hypersphere. The clustering results of the local support vector hyperspheres are combined to yield a global clustering solution by the minimal spanning tree (MST), where a predefined number of clustering is not needed. We compare the proposed methods with two common clustering algorithms for the artificial synthetic data set and several image data sets, including the synthetic texture images and the SAR images, and encouraging experimental results are obtained.

Scaling up Kernel Grower Clustering Method for Large Data Sets via Core-sets

Kernel grower is a novel kernel clustering method proposed recently by Camastra and Verri. It shows good performance for various data sets and compares favorably with respect to popular clustering algorithms. However, the main drawback of the method is the weak scaling ability in dealing with large data sets, which restricts its application greatly. In this paper, we propose a scaled-up kernel grower method using core-sets, which is significantly faster than the original method for large data clustering. Meanwhile, it can deal with very large data sets. Numerical experiments on benchmark data sets as well as synthetic data sets show the efficiency of the proposed method. The method is also applied to real image segmentation to illustrate its performance.

Constructing Tumor Progression Pathways and Biomarker Discovery with Fuzzy KernelKmeansand DNA Methylation Data

Constructing pathways of tumor progression and discovering the biomarkers associated with cancer is critical for understanding the molecular basis of the disease and for the establishment of novel chemotherapeutic approaches and in turn improving the clinical efficiency of the drugs. It has recently received a lot of attention from bioinformatics researchers. However, relatively few methods are available for constructing pathways. This article develops a novel entropy kernel based kernel clustering and fuzzy kernel clustering algorithms to construct the tumor progression pathways using CpG island methylation data. The methylation data which come from tumor tissues diagnosed at different stages can be used to distinguish epigenotype and phenotypes the describe the molecular events of different phases. Using kernel and fuzzy kernel kmeans, we built tumor progression trees to describe the pathways of tumor progression and find the possible biomarkers associated with cancer. Our results indicate that the proposed algorithms together with methylation profiles can predict the tumor progression stages and discover the biomarkers efficiently. Software is available upon request.