Kernel Eigenvector Research Articles

Lamb waves in cubic crystals: crossing fundamental branches

The current research is concerned with the study of Lamb waves propagating in a traction-free plate with cubic anisotropy along the crystallographic axes. Splitting all cubic crystals into two non-intersecting classes revealed either the appearance or disappearance of the crossing points of the fundamental branches. The spectral analysis of the resolvent matrix in the vicinity of the crossing points showed existence of a two-fold degeneracy with two distinct kernel eigenvectors and absence of the non-semisimple degeneracy, meaning stability of the flexural and symmetric fundamental branches passing through the crossing point.

A nonlinear calibration transfer method based on joint kernel subspace

A nonlinear strategy is proposed to develop calibration transfer method with the available joint spectral data composed of the master and slave spectra of standard samples. Its core idea is to minimize the instrument-induced spectral variation in the reproducing kernel Hilbert space (RKHS) where the joint spectral data is implicitly mapped by proper kernel function. The nonlinear feature representation of master or slave spectra are reconstructed by singular value decomposition (SVD) in the RKHS. Then the transferred feature of slave spectra approaching the reconstructed feature of master spectra in the RKHS can be acquired by the procedure that (1) calculates the transfer matrix to match the maser and slave kernel features in the joint kernel subspace and (2) utilizes the transferred slave kernel feature to reconstruct the nonlinear feature representation of slave spectra. As a better feature representation suited for multivariate calibration, both the reconstructed feature of master calibration spectra and the transferred feature of slave test spectra are derived in the RKHS. A kernel partial least squares (KPLS) model built on the former is applied to the latter. The KPLS master calibration model is equivalent to a partial least squares (PLS) model built with the corresponding reconstructed master kernel feature in the joint kernel subspace. By exploiting the kernel eigenvector representations and kernel trick, a series of computationally less demanding formulas with linear operation is derived to realize the nonlinear calibration model, termed as joint kernel subspace based calibration transfer (JKSCT). JKSCT is compared with boxcar signal transfer (BST), piecewise direct standardization (PDS), multi-level simultaneous component analysis (MSCA), canonical correlation analysis based calibration transfer (CCACT), generalized least squares (GLS), slope and bias correction (SBC), spectral space transformation (SST), external parameter orthogonalization (EPO) and double competitive adaptive reweighted sampling (DCARS) on three spectral datasets. Experimental results show that JKSCT performs at least comparable with the DCARS or SST, and frequently better than the other methods.

Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm

How many training data are needed to learn a supervised task? It is often observed that the generalization error decreases as n −β where n is the number of training examples and β is an exponent that depends on both data and algorithm. In this work we measure β when applying kernel methods to real datasets. For MNIST we find β ≈ 0.4 and for CIFAR10 β ≈ 0.1, for both regression and classification tasks, and for Gaussian or Laplace kernels. To rationalize the existence of non-trivial exponents that can be independent of the specific kernel used, we study the teacher–student framework for kernels. In this scheme, a teacher generates data according to a Gaussian random field, and a student learns them via kernel regression. With a simplifying assumption—namely that the data are sampled from a regular lattice—we derive analytically β for translation invariant kernels, using previous results from the kriging literature. Provided that the student is not too sensitive to high frequencies, β depends only on the smoothness and dimension of the training data. We confirm numerically that these predictions hold when the training points are sampled at random on a hypersphere. Overall, the test error is found to be controlled by the magnitude of the projection of the true function on the kernel eigenvectors whose rank is larger than n. Using this idea we predict the exponent β from real data by performing kernel PCA, leading to β ≈ 0.36 for MNIST and β ≈ 0.07 for CIFAR10, in good agreement with observations. We argue that these rather large exponents are possible due to the small effective dimension of the data.

Spectrally Sparse Nonparametric Regression via Elastic Net Regularized Smoothers

Nonparametric regression frameworks, such as generalized additive models (GAMs) and smoothing spline analysis of variance (SSANOVA) models, extend the generalized linear model (GLM) by allowing for unknown functional relationships between an exponential family response variable and a collection of predictor variables. The unknown functional relationships are typically estimated using penalized likelihood estimation, which adds a roughness penalty to the (negative) log-likelihood function. In this article, I propose a spectral parameterization of a smoothing spline, which allows for an efficient application of Elastic Net regression to smooth and select eigenvectors of a kernel matrix. The classic (ridge regression) solution for a smoothing spline is a special case of the proposed kernel eigenvector smoothing and selection operator. Extensions for tensor product smoothers are developed for both the GAM and SSANOVA frameworks. Using simulated and real data examples, I demonstrate that the proposed approach offers practical and computational gains over typical approaches for fitting GAMs, SSANOVA models, and Elastic Net penalized GLMs. Supplementary materials for this article are available online.

On the mathematical foundations of stable RKHSs

Reproducing kernel Hilbert spaces (RKHSs) are key spaces for machine learning that are becoming popular also for linear system identification. In particular, the so-called stable RKHSs can be used to model absolutely summable impulse responses. In combination e.g. with regularized least squares they can then be used to reconstruct dynamic systems from input–output data. In this paper we provide new structural properties of stable RKHSs. The relation between stable kernels and other fundamental classes, like those containing absolutely summable or finite-trace kernels, is elucidated. These insights are then brought into the feature space context. First, it is proved that any stable kernel admits feature maps induced by a basis of orthogonal eigenvectors in ℓ2. The exact connection with classical system identification approaches that exploit such kind of functions to model impulse responses is also provided. Then, the necessary and sufficient stability condition for RKHSs designed by formulating kernel eigenvectors and eigenvalues is obtained. Overall, our new results provide novel mathematical foundations of stable RKHSs with impact on stability tests, impulse responses modeling and computational efficiency of regularized schemes for linear system identification.

Optimized Kernel Entropy Components

This brief addresses two main issues of the standard kernel entropy component analysis (KECA) algorithm: the optimization of the kernel decomposition and the optimization of the Gaussian kernel parameter. KECA roughly reduces to a sorting of the importance of kernel eigenvectors by entropy instead of variance, as in the kernel principal components analysis. In this brief, we propose an extension of the KECA method, named optimized KECA (OKECA), that directly extracts the optimal features retaining most of the data entropy by means of compacting the information in very few features (often in just one or two). The proposed method produces features which have higher expressive power. In particular, it is based on the independent component analysis framework, and introduces an extra rotation to the eigen decomposition, which is optimized via gradient-ascent search. This maximum entropy preservation suggests that OKECA features are more efficient than KECA features for density estimation. In addition, a critical issue in both the methods is the selection of the kernel parameter, since it critically affects the resulting performance. Here, we analyze the most common kernel length-scale selection criteria. The results of both the methods are illustrated in different synthetic and real problems. Results show that OKECA returns projections with more expressive power than KECA, the most successful rule for estimating the kernel parameter is based on maximum likelihood, and OKECA is more robust to the selection of the length-scale parameter in kernel density estimation.

Enhancing semi-supervised learning through label-aware base kernels

Currently, a large family of kernel design methods for semi-supervised learning (SSL) problems builds the kernel by weighted averaging of predefined base kernels (i.e., those spanned by kernel eigenvectors). Optimization of the base kernel weights has been studied extensively in the literature. However, little attention was devoted to designing high-quality base kernels. The eigenvectors of the kernel matrix, which are computed irrespective of class labels, may not always reveal useful structures of the target. As a result, the generalization performance can be poor however hard the base kernel weighting is tuned. On the other hand, there are many SSL algorithms whose focus are not on kernel design but on the estimation of the class labels directly. Motivated by the label propagation approach, in this paper we propose to construct novel kernel eigenvectors by injecting the class label information under the framework of eigenfunction extrapolation. A set of “label-aware” base kernels can be obtained with greatly improved quality, which leads to higher target alignment and henceforth better performance. Our approach is computationally efficient, and demonstrates encouraging performance in semi-supervised classification and regression tasks.

Improving Semi-Supervised Target Alignment via Label-Aware Base Kernels

Semi-supervised kernel design is an essential step for obtaining good predictive performance in semi-supervised learning tasks. In the current literatures, a large family of algorithms builds the new kernel by using the weighted average of predefined base kernels. While optimal weighting schemes have been studied extensively, the choice of base kernels received much less attention. Many methods simply adopt the empirical kernel matrices or its eigenvectors. Such base kernels are computed irrespective of class labels and may not always reflect useful structures in the data. As a result, in case of poor base kernels, the generalization performance can be degraded however hard their weights are tuned. In this paper, we propose to construct high-quality base kernels with the help of label information to globally improve the final target alignment. In particular, we devise label-aware kernel eigenvectors under the framework of semi-supervised eigenfunction extrapolation, which span base kernels that are more useful for learning. Such base kernels are individually better aligned to the learning target, so their mixture will more likely generate a good classifier. Our approach is computationally efficient, and demonstrates encouraging performance in semi-supervised classification and regression.

Minimal configuration bicyclic graphs

The nullity η(G) of a graph G is the multiplicity of zero as an eigenvalue of the adjacency matrix of G. If η(G) = 1, then the core of G is the subgraph induced by the vertices associated with the nonzero entries of the kernel eigenvector. The set of vertices which are not in the core is the periphery of G. A graph G with nullity one is minimal configuration if no two vertices in the periphery are adjacent and deletion of any vertex in the periphery increases the nullity. An ∞-graph ∞(p, l, q) is a graph obtained by joining two vertex-disjoint cycles C p and C q by a path of length l ≥ 0. Let ℬ* be the class of bicyclic graphs with an ∞-graph as an induced subgraph. In this article, we characterize the graphs in ℬ* which are minimal configurations.

Minimal configuration unicyclic graphs

A graph G is singular with nullity η( G ), if zero is an eigenvalue of its adjacency matrix with multiplicity η( G ). If η(G) = 1, then the core of G is the subgraph induced by the vertices associated with the non-zero entries of the kernel eigenvector. The set of vertices which are not in the core is called the periphery of G. A graph G with nullity one is called a minimal configuration if no two vertices in the periphery are adjacent and deletion of any vertex in the periphery increases the nullity. In this article, we describe the structure of a singular unicyclic graph and single out the class of unicyclic graphs which are minimal configurations.

Coalesced and embedded nut graphs in singular graphs

A nut graph has a non-invertible (singular) 0-1 adjacency matrix with non-zero entries in every kernel eigenvector. We investigate how the concept of nut graphs emerges as an underlying theme in the theory of singular graphs. It is known that minimal configurations (MCs) are necessarily found as subgraphs of singular graphs. We construct MCs having nut graphs as subgraphs. Nut graphs can be coalesced with singular graphs at particular vertices or grown into a family of core graphs of larger nullity by adding a vertex at a time. Moreover, we propose a construction of nut line graph of trees by coalescence and a local enlargement of nut fullerenes and tetravalent nut graphs.

Nonbonding Orbitals in Fullerenes: Nuts and Cores in Singular Polyhedral Graphs

A zero eigenvalue in the spectrum of the adjacency matrix of the graph representing an unsaturated carbon framework indicates the presence of a nonbonding pi orbital (NBO). A graph with at least one zero in the spectrum is singular; nonzero entries in the corresponding zero-eigenvalue eigenvector(s) (kernel eigenvectors) identify the core vertices. A nut graph has a single zero in its adjacency spectrum with a corresponding eigenvector for which all vertices lie in the core. Balanced and uniform trivalent (cubic) nut graphs are defined in terms of (-2, +1, +1) patterns of eigenvector entries around all vertices. In balanced nut graphs all vertices have such a pattern up to a scale factor; uniform nut graphs are balanced with scale factor one for every vertex. Nut graphs are rare among small fullerenes (41 of the 10 190 782 fullerene isomers on up to 120 vertices) but common among the small trivalent polyhedra (62 043 of the 398 383 nonbipartite polyhedra on up to 24 vertices). Two constructions are described, one that is conjectured to yield an infinite series of uniform nut fullerenes, and another that is conjectured to yield an infinite series of cubic polyhedral nut graphs. All hypothetical nut fullerenes found so far have some pentagon adjacencies: it is proved that all uniform nut fullerenes must have such adjacencies and that the NBO is totally symmetric in all balanced nut fullerenes. A single electron placed in the NBO of a uniform nut fullerene gives a spin density distribution with the smallest possible (4:1) ratio between most and least populated sites for an NBO. It is observed that, in all nut-fullerene graphs found so far, occupation of the NBO would require the fullerene to carry at least 3 negative charges, whereas in most carbon cages based on small nut cubic polyhedra, the NBO would be the highest occupied molecular orbital (HOMO) for the uncharged system.

Kernel matched subspace detectors for hyperspectral target detection

In this paper, we present a kernel realization of a matched subspace detector (MSD) that is based on a subspace mixture model defined in a high-dimensional feature space associated with a kernel function. The linear subspace mixture model for the MSD is first reformulated in a high-dimensional feature space and then the corresponding expression for the generalized likelihood ratio test (GLRT) is obtained for this model. The subspace mixture model in the feature space and its corresponding GLRT expression are equivalent to a nonlinear subspace mixture model with a corresponding nonlinear GLRT expression in the original input space. In order to address the intractability of the GLRT in the feature space, we kernelize the GLRT expression using the kernel eigenvector representations as well as the kernel trick where dot products in the feature space are implicitly computed by kernels. The proposed kernel-based nonlinear detector, so-called kernel matched subspace detector (KMSD), is applied to several hyperspectral images to detect targets of interest. KMSD showed superior detection performance over the conventional MSD when tested on several synthetic data and real hyperspectral imagery.