Covariance Kernel Research Articles

Average Case (s, t)-Weak Tractability of L2-Approximation with Weighted Covariance Kernels

Average Case (s, t)-Weak Tractability of L2-Approximation with Weighted Covariance Kernels

Flexible Bayesian modeling for longitudinal binary and ordinal responses

Longitudinal studies with binary or ordinal responses are widely encountered in various disciplines, where the primary focus is on the temporal evolution of the probability of each response category. Traditional approaches build from the generalized mixed effects modeling framework. Even amplified with nonparametric priors placed on the fixed or random effects, such models are restrictive due to the implied assumptions on the marginal expectation and covariance structure of the responses. We tackle the problem from a functional data analysis perspective, treating the observations for each subject as realizations from subject-specific stochastic processes at the measured times. We develop the methodology focusing initially on binary responses, for which we assume the stochastic processes have Binomial marginal distributions. Leveraging the logits representation, we model the discrete space processes through continuous space processes. We utilize a hierarchical framework to model the mean and covariance kernel of the continuous space processes nonparametrically and simultaneously through a Gaussian process prior and an Inverse-Wishart process prior, respectively. The prior structure results in flexible inference for the evolution and correlation of binary responses, while allowing for borrowing of strength across all subjects. The modeling approach can be naturally extended to ordinal responses. Here, the continuation-ratio logits factorization of the multinomial distribution is key for efficient modeling and inference, including a practical way of dealing with unbalanced longitudinal data. The methodology is illustrated with synthetic data examples and an analysis of college students’ mental health status data.

Conformally invariant random fields, Liouville quantum gravity measures, and random Paneitz operators on Riemannian manifolds of even dimension

AbstractFor large classes of even‐dimensional Riemannian manifolds , we construct and analyze conformally invariant random fields. These centered Gaussian fields , called co‐polyharmonic Gaussian fields, are characterized by their covariance kernels k which exhibit a precise logarithmic divergence: . They share a fundamental quasi‐invariance property under conformal transformations. In terms of the co‐polyharmonic Gaussian field , we define the Liouville Quantum Gravity measure, a random measure on , heuristically given as and rigorously obtained as almost sure weak limit of the right‐hand side with replaced by suitable regular approximations . In terms on the Liouville Quantum Gravity measure, we define the Liouville Brownian motion on and the random GJMS operators. Finally, we present an approach to a conformal field theory in arbitrary even dimension with an ansatz based on Branson's ‐curvature: we give a rigorous meaning to the Polyakov–Liouville measure and we derive the corresponding conformal anomaly. The set of admissible manifolds is conformally invariant. It includes all compact 2‐dimensional Riemannian manifolds, all compact non‐negatively curved Einstein manifolds of even dimension, and large classes of compact hyperbolic manifolds of even dimension. However, not every compact even‐dimensional Riemannian manifold is admissible. Our results concerning the logarithmic divergence of the kernel rely on new sharp estimates for heat kernels and higher order Green kernels on arbitrary closed manifolds.

Online kernel-based clustering

A novel online joint kernel learning and clustering (OKC) framework is derived which is capable of determining time-varying clustering configurations without the need for training data. To facilitate clustering via sparse kernel factorization, a novel time-dependent metric is devised to quantify closeness of candidate kernel similarity matrices to a block diagonal structure. The task is later performed online as newly acquired data are processed making it appropriate for non-stationary settings. The necessary minimization to carry out kernel learning and clustering is performed via an effective interplay among block coordinate descent, difference of convex functions minimization and projected subgradient descent which results in an online iterative algorithm that updates online the kernel covariance matrix, as well as the associated clustering configuration. The resulting recursive kernel updates are proved to converge to a bounded limit. Detailed numerical examples utilizing both synthetic and real data show the superior clustering accuracy achieved by the novel approach over existing alternatives while requiring considerably less running time.

Continuously indexed graphical models

Abstract Let X={Xu}u∈U be a real-valued Gaussian process indexed by a set U. We show that X can be viewed as a graphical model with an uncountably infinite graph, where each Xu is a vertex. This graph is characterized by the reproducing property of X’s covariance kernel, without restricting U to be finite or countable, allowing the modelling of stochastic processes in continuous time/space. Unlike traditional methods, this characterization is not based on zero entries of an inverse covariance, posing challenges for structure estimation. We propose a plug-in methodology that targets graph recovery up to a finite resolution and shows consistency for graphs which are sufficiently regular and that can be applied to virtually any measurement regime. Furthermore, we derive convergence rates and finite-sample guarantees for the method, and demonstrate its performance through a simulation study and two data analyses.

Cryo-EM Heterogeneity Analysis using Regularized Covariance Estimation and Kernel Regression.

Proteins and the complexes they form are central to nearly all cellular processes. Their flexibility, expressed through a continuum of states, provides a window into their biological functions. Cryogenic electron microscopy (cryo-EM) is an ideal tool to study these dynamic states as it captures specimens in non-crystalline conditions and enables high-resolution reconstructions. However, analyzing the heterogeneous distributions of conformations from cryo-EM data is challenging. We present RECOVAR, a method for analyzing these distributions based on principal component analysis (PCA) computed using a REgularized COVARiance estimator. RECOVAR is fast, robust, interpretable, expressive, and competitive with the state-of-art neural network methods on heterogeneous cryo-EM datasets. The regularized covariance method efficiently computes a large number of high-resolution principal components that can encode rich heterogeneous distributions of conformations and does so robustly thanks to an automatic regularization scheme. The novel reconstruction method based on adaptive kernel regression resolves conformational states to a higher resolution than all other tested methods on extensive independent benchmarks while remaining highly interpretable. Additionally, we exploit favorable properties of the PCA embedding to estimate the conformational density accurately. This density allows for better interpretability of the latent space by identifying stable states and low free-energy motions. Finally, we present a scheme to navigate the high-dimensional latent space by automatically identifying these low free-energy trajectories. We make the code freely available at https://github.com/ma-gilles/recovar .

Testing covariance separability for continuous functional data

Analyzing the covariance structure of data is a fundamental task of statistics. While this task is simple for low‐dimensional observations, it becomes challenging for more intricate objects, such as multi‐variate functions. Here, the covariance can be so complex that just saving a non‐parametric estimate is impractical and structural assumptions are necessary to tame the model. One popular assumption for space‐time data is separability of the covariance into purely spatial and temporal factors. In this article, we present a new test for separability in the context of dependent functional time series. While most of the related work studies functional data in a Hilbert space of square integrable functions, we model the observations as objects in the space of continuous functions equipped with the supremum norm. We argue that this (mathematically challenging) setup enhances interpretability for users and is more in line with practical preprocessing. Our test statistic measures the maximal deviation between the estimated covariance kernel and a separable approximation. Critical values are obtained by a non‐standard multiplier bootstrap for dependent data. We prove the statistical validity of our approach and demonstrate its practicability in a simulation study and a data example.

Revised LOFAR upper limits on the 21-cm signal power spectrum at z ≈ 9.1 using machine learning and gaussian process regression

ABSTRACT The use of Gaussian Process Regression (GPR) for foregrounds mitigation in data collected by the LOw-Frequency ARray (LOFAR) to measure the high-redshift 21-cm signal power spectrum has been shown to have issues of signal loss when the 21-cm signal covariance is misestimated. To address this problem, we have recently introduced covariance kernels obtained by using a Machine Learning based Variational Auto-Encoder (VAE) algorithm in combination with simulations of the 21-cm signal. In this work, we apply this framework to 141 h (${\approx} 10$ nights) of LOFAR data at $z \approx 9.1$, and report revised upper limits of the 21-cm signal power spectrum. Overall, we agree with past results reporting a 2-$\sigma$ upper limit of $\Delta ^2_{21} \ \lt\ (80)^2~\rm mK^2$ at $k = 0.075~h~\rm Mpc^{-1}$. Further, the VAE-based kernel has a smaller correlation with the systematic excess noise, and the overall GPR-based approach is shown to be a good model for the data. Assuming an accurate bias correction for the excess noise, we report a 2-$\sigma$ upper limit of $\Delta ^2_{21} \ \lt\ (25)^2~\rm mK^2$ at $k = 0.075~h~\rm Mpc^{-1}$. However, we still caution to take the more conservative approach to jointly report the upper limits of the excess noise and the 21-cm signal components.

On the properties of multivariate isotropic Random fields on the Ball

We consider multivariate isotropic random fields on the ball Bd. We first study their regularity properties in terms of Sobolev spaces. We further derive conditions guaranteeing the Hölder continuity of their covariance kernels and we prove the existence of sample Hölder continuous modifications for Gaussian random fields. Furthermore, we measure the error of truncated approximations of the corresponding series' representations. Moreover our developments are supported by numerical experiments. The majority of our results are new for multivariate random fields indexed over other domains, too. We express some of them for the case of the sphere.

Sparse Cholesky factorization for solving nonlinear PDEs via Gaussian processes

In recent years, there has been widespread adoption of machine learning-based approaches to automate the solving of partial differential equations (PDEs). Among these approaches, Gaussian processes (GPs) and kernel methods have garnered considerable interest due to their flexibility, robust theoretical guarantees, and close ties to traditional methods. They can transform the solving of general nonlinear PDEs into solving quadratic optimization problems with nonlinear, PDE-induced constraints. However, the complexity bottleneck lies in computing with dense kernel matrices obtained from pointwise evaluations of the covariance kernel, and its partial derivatives, a result of the PDE constraint and for which fast algorithms are scarce. The primary goal of this paper is to provide a near-linear complexity algorithm for working with such kernel matrices. We present a sparse Cholesky factorization algorithm for these matrices based on the near-sparsity of the Cholesky factor under a novel ordering of pointwise and derivative measurements. The near-sparsity is rigorously justified by directly connecting the factor to GP regression and exponential decay of basis functions in numerical homogenization. We then employ the Vecchia approximation of GPs, which is optimal in the Kullback-Leibler divergence, to compute the approximate factor. This enables us to compute ϵ \epsilon -approximate inverse Cholesky factors of the kernel matrices with complexity O ( N log d ⁡ ( N / ϵ ) ) O(N\log ^d(N/\epsilon )) in space and O ( N log 2 d ⁡ ( N / ϵ ) ) O(N\log ^{2d}(N/\epsilon )) in time. We integrate sparse Cholesky factorizations into optimization algorithms to obtain fast solvers of the nonlinear PDE. We numerically illustrate our algorithm’s near-linear space/time complexity for a broad class of nonlinear PDEs such as the nonlinear elliptic, Burgers, and Monge-Ampère equations. In summary, we provide a fast, scalable, and accurate method for solving general PDEs with GPs and kernel methods.

Tractability of Multivariate Approximation Problem on Euler and Wiener Integrated Processes

This paper examines the tractability of multivariate approximation problems under the normalized error criterion for a zero-mean Gaussian measure in an average-case setting. The Gaussian measure is associated with a covariance kernel, which is represented by the tensor product of one-dimensional kernels corresponding to Euler and Wiener integrated processes with non-negative and nondecreasing smoothness parameters {rd}d∈N. We give matching sufficient and necessary conditions for various concepts of tractability in terms of the asymptotic properties of the regularity parameters, except for (s, 0)-WT.

Smallest gaps between zeros of stationary Gaussian processes

In this paper, we study the smallest gaps between successive zeros of nondegenerate smooth stationary centered Gaussian processes on the real line with the assumption that the covariance kernel κ(x) and its derivatives decay to 0 as |x|→∞. We prove that, after rescaling, the smallest gaps converge to a Poisson point process with a specific rate. Moreover, the positions where these smallest gaps occur tend to a uniform distribution. Consequently, we can derive the limiting density for the k-th smallest gap.

Uniform approximation of common Gaussian process kernels using equispaced Fourier grids

The high efficiency of a recently proposed method for computing with Gaussian processes relies on expanding a (translationally invariant) covariance kernel into complex exponentials, with frequencies lying on a Cartesian equispaced grid. Here we provide rigorous error bounds for this approximation for two popular kernels—Matérn and squared exponential—in terms of the grid spacing and size. The kernel error bounds are uniform over a hypercube centered at the origin. Our tools include a split into aliasing and truncation errors, and bounds on sums of Gaussians or modified Bessel functions over various lattices. For the Matérn case, motivated by numerical study, we conjecture a stronger Frobenius-norm bound on the covariance matrix error for randomly-distributed data points. Lastly, we prove bounds on, and study numerically, the ill-conditioning of the linear systems arising in such regression problems.

Images of Gaussian and other stochastic processes under closed, densely-defined, unbounded linear operators

Gaussian Processes (GPs) are widely-used tools in spatial statistics and machine learning and the formulae for the mean function and covariance kernel of a GP [Formula: see text] that is the image of another GP [Formula: see text] under a linear transformation [Formula: see text] acting on the sample paths of [Formula: see text] are well known, almost to the point of being folklore. However, these formulae are often used without rigorous attention to technical details, particularly when [Formula: see text] is an unbounded operator such as a differential operator, which is common in many modern applications. This note provides a self-contained proof of the claimed formulae for the case of a closed, densely-defined operator [Formula: see text] acting on the sample paths of a square-integrable (not necessarily Gaussian) stochastic process. Our proof technique relies upon Hille’s theorem for the Bochner integral of a Banach-valued random variable.

Critical point asymptotics for Gaussian random waves with densities of any Sobolev regularity

We consider Gaussian random monochromatic waves u on the plane depending on a real parameter s that is directly related to the regularity of its Fourier transform. Specifically, the Fourier transform of u is fdσ, where dσ is the Hausdorff measure on the unit circle and the density f is a function on the circle that, roughly speaking, has exactly s−12 derivatives in L2 almost surely. When s=0, one recovers the classical setting for random waves with a translation-invariant covariance kernel. The main thrust of this paper is to explore the connection between the regularity parameter s and the asymptotic behavior of the number N(∇u,R) of critical points that are contained in the disk of radius R≫1. More precisely, we show that the expectation EN(∇u,R) grows like the area of the disk when the regularity is low enough (s<32) and like the diameter when the regularity is high enough (s>52), and that the corresponding exponent changes according to a linear interpolation law in the intermediate regime. The transitions occurring at the endpoint cases involve the square root of the logarithm of the radius. Interestingly, the highest asymptotic growth rate occurs only in the classical translation-invariant setting, s=0. A key step of the proof of this result is the obtention of precise asymptotic expansions for certain Neumann series of Bessel functions. When the regularity parameter is s>5, we show that in fact N(∇u,R) grows like the diameter with probability 1, albeit the ratio is not a universal constant but a random variable.

Non-Separable Covariance Kernels for Spatiotemporal Gaussian Processes Based on a Hybrid Spectral Method and the Harmonic Oscillator

Non-Separable Covariance Kernels for Spatiotemporal Gaussian Processes Based on a Hybrid Spectral Method and the Harmonic Oscillator

Covariant quantum kernels for data with group structure

Covariant quantum kernels for data with group structure

Efficient learning methods for large-scale optimal inversion design

In this work, we investigate various approaches that use learning from training data to solve inverse problems, following a bi-level learning approach. We consider a general framework for optimal inversion design, where training data can be used to learn optimal regularization parameters, data fidelity terms, and regularizers, thereby resulting in superior variational regularization methods. In particular, we describe methods to learn optimal $ p $ and $ q $ norms for $ {\rm L}^p-{\rm L}^q $ regularization and methods to learn optimal parameters for regularization matrices defined by covariance kernels. We exploit efficient algorithms based on Krylov projection methods for solving the regularized problems, both at training and validation stages, making these methods well-suited for large-scale problems. Our experiments show that the learned regularization methods perform well even when there is some inexactness in the forward operator, resulting in a mixture of model and measurement error.

Constructing training set using distance between learnt graphical models of time series data on patient physiology, to predict disease scores.

Interventional endeavours in medicine include prediction of a score that parametrises a new subject's susceptibility to a given disease, at the pre-onset stage. Here, for the first time, we provide reliable learning of such a score in the context of the potentially-terminal disease VOD, that often arises after bone marrow transplants. Indeed, the probability of surviving VOD, is correlated with early intervention. In our work, the VOD-score of each patient in a retrospective cohort, is defined as the distance between the (posterior) probability of a random graph variable-given the inter-variable partial correlation matrix of the time series data on variables that represent different aspects of patient physiology-and that given such time series data of an arbitrarily-selected reference patient. Such time series data is recorded from a pre-transplant to a post-transplant time, for each patient in this cohort, though the data available for distinct patients bear differential temporal coverage, owing to differential patient longevities. Each graph is a Soft Random Geometric Graph drawn in a probabilistic metric space, and the computed inter-graph distance is oblivious to the length of the time series data. The VOD-score learnt in this way, and the corresponding pre-transplant parameter vector of each patient in this retrospective cohort, then results in the training data, using which we learn the function that takes VOD-score as its input, and outputs the vector of pre-transplant parameters. We model this function with a vector-variate Gaussian Process, the covariance structure of which is kernel parametrised. Such modelling is easier than if the score variable were the output. Then for any prospective patient, whose pre-transplant variables are known, we learn the VOD-score (and the hyperparameters of the covariance kernel), using Markov Chain Monte Carlo based inference.

Conditional Karhunen–Loève regression model with Basis Adaptation for high-dimensional problems: Uncertainty quantification and inverse modeling

We propose a methodology for improving the accuracy of surrogate models of the observable response of physical systems as a function of the systems’ spatially heterogeneous parameter fields, with applications to uncertainty quantification and parameter estimation in high-dimensional problems. Practitioners often formulate finite-dimensional representations of spatially heterogeneous parameter fields using truncated unconditional Karhunen–Loève expansions (KLEs) for a certain choice of unconditional covariance kernel and construct surrogate models of the observable response with respect to the KLE coefficients. When direct measurements of the parameter fields are available, we propose improving the accuracy of these surrogate models by representing the parameter fields via conditional Karhunen-Loève expansions (CKLEs). CKLEs are constructed by conditioning the covariance kernel of the unconditional expansion on the direct measurements of the parameter field via Gaussian process regression, and then truncating the corresponding KLE.We apply the proposed methodology to constructing surrogate models via the Basis Adaptation (BA) method of the stationary hydraulic head response, measured at spatially discrete observation locations, of a groundwater flow model of the Hanford Site, as a function of the 1000-dimensional representation of the model’s log-transmissivity field. We find that BA surrogate models of the hydraulic head based on CKLEs are more accurate than BA surrogate models based on unconditional expansions for forward uncertainty quantification tasks. Furthermore, we find that inverse estimates of the hydraulic transmissivity field computed using CKLE-based BA surrogate models are more accurate than those computed using unconditional BA surrogate models.