Product Kernel Research Articles

Operator learning with Gaussian processes

Operator learning focuses on approximating mappings G†:U→V between infinite-dimensional spaces of functions, such as u:Ωu→R and v:Ωv→R. This makes it particularly suitable for solving parametric nonlinear partial differential equations (PDEs). Recent advancements in machine learning (ML) have brought operator learning to the forefront of research. While most progress in this area has been driven by variants of deep neural networks (NNs), recent studies have demonstrated that Gaussian process (GP)/kernel-based methods can also be competitive. These methods offer advantages in terms of interpretability and provide theoretical and computational guarantees. In this article, we introduce a hybrid GP/NN-based framework for operator learning, leveraging the strengths of both deep neural networks and kernel methods. Instead of directly approximating the function-valued operator G†, we use a GP to approximate its associated real-valued bilinear form G˜†:U×V∗→R. This bilinear form is defined by the dual pairing G˜†(u,φ)≔[φ,G†(u)], which allows us to recover the operator G† through G†(u)(y)=G˜†(u,δy). The mean function of the GP can be set to zero or parameterized by a neural operator and for each setting we develop a robust and scalable training mechanism based on maximum likelihood estimation (MLE) that can optionally leverage the physics involved. Numerical benchmarks demonstrate our method’s scope, scalability, efficiency, and robustness; showing that (1) it enhances the performance of a base neural operator by using it as the mean function of a GP, and (2) it enables the construction of zero-shot data-driven models that can make accurate predictions without any prior training. Additionally, our framework (a) naturally extends to cases where G†:U→∏s=1SVs maps into a vector of functions, and (b) benefits from computational speed-ups achieved through product kernel structures and Kronecker product matrix representations of the underlying kernel matrices.11GitHub repository: https://github.com/Bostanabad-Research-Group/GP-for-Operator-Learning.

The phase diagram of kernel interpolation in large dimensions

Abstract The generalization ability of kernel interpolation in large dimensions, i.e., 𝑛 ≍ 𝑑𝛾 for some 𝛾 > 0, might be one of the most interesting problems in the recent renaissance of kernel regression, since it may help us understand the ‘benign overfitting phenomenon’ reported in the neural networks literature. Focusing on the inner product kernel on the unit sphere, we fully characterize the exact order of both the variance and bias of large-dimensional kernel interpolation under various source conditions 𝑠 ≥ 0. Consequently, we obtain the (𝑠, 𝛾)-phase diagram of large-dimensional kernel interpolation, i.e., we determine the regions in the (𝑠, 𝛾)-plane where the kernel interpolation is minimax optimal, sub-optimal and inconsistent.

Exact Calculation of the Probabilities of Rare Events in Cluster-Cluster Aggregation.

We develop an action formalism to calculate probabilities of rare events in cluster-cluster aggregation for arbitrary collision kernels and establish a pathwise large deviation principle with total mass being the rate. As an application, the rate function for the number of surviving particles as well as the optimal evolution trajectory are calculated exactly for the constant, sum, and product kernels. For the product kernel, we argue that the second derivative of the rate function has a discontinuity. The theoretical results agree with simulations tailored to the calculation of rare events.

Effect of dimensionality on convergence rates of kernel ridge regression estimator

Despite the curse of dimensionality, kernel ridge regression often exhibits good performance in practical applications, even when the dimension is moderately large. However, it has been shown that kernel ridge regression cannot be free from the curse of dimensionality. Until now, the literature on kernel ridge regression has suggested that the gap between theory and practice in relation to dimensionality has not narrowed. In this study, we first investigate when the influence of dimensionality does not significantly affect the convergence rate of the kernel ridge regression. Specifically, we study the convergence rate of L2 and L∞ risks for the kernel ridge estimator, with a focus on reproducing kernel Hilbert space (RKHS) generated by a product kernel. We show that the univariate optimal convergence rate up to a logarithmic factor in L2 and L∞ risks can be achieved by controlling the size of the RKHS. The result of a numerical study confirms our theoretical findings.

Singular integrals with product kernels associated with mixed homogeneities and Hardy spaces

Singular integrals with product kernels associated with mixed homogeneities and Hardy spaces

Tractability of Multivariate Approximation Problem on Euler and Wiener Integrated Processes

This paper examines the tractability of multivariate approximation problems under the normalized error criterion for a zero-mean Gaussian measure in an average-case setting. The Gaussian measure is associated with a covariance kernel, which is represented by the tensor product of one-dimensional kernels corresponding to Euler and Wiener integrated processes with non-negative and nondecreasing smoothness parameters {rd}d∈N. We give matching sufficient and necessary conditions for various concepts of tractability in terms of the asymptotic properties of the regularity parameters, except for (s, 0)-WT.

Predicting pairwise interaction affinities with ℓ0-penalized least squares–a nonsmooth bi-objective optimization based approach*

In this paper, we introduce a novel nonsmooth optimization-based method LMBM-Kron ℓ 0 LS for solving large-scale pairwise interaction affinity prediction problems. The aim of LMBM-Kron ℓ 0 LS is to produce accurate predictions using as sparse a model as possible. We apply the least squares approach with Kronecker product kernels for a loss function and a continuous formulation of ℓ 0 pseudonorm for regularization. Thus, we end up solving a nonsmooth optimization problem. In addition, we apply a specific bi-objective criterion to strike a balance between the prediction accuracy of the learned model and the sparsity of the obtained solution. We compare LMBM-Kron ℓ 0 LS with some state-of-the-art methods using three benchmark and two simulated data sets under four distinct experimental settings, including zero-shot learning. Moreover, both binary and continuous interaction affinity labels are considered with LMBM-Kron ℓ 0 LS. The results show that LMBM-Kron ℓ 0 LS finds sparse solutions without sacrificing too much in the prediction performance.

Relations between product and flag Triebel-Lizorkin spaces

Nagel, Ricci and Stein proved that product kernels are finite sums of flag kernels in the Euclidean space. We show that the product Triebel-Lizorkin space is the intersection of two flag Triebel-Lizorkin spaces. This extends a main result in [Chang D-C, Han Y, Wu X. Relations between product and flag Hardy spaces. J Geom Anal. 2021;31(7):6601–6623]. As an application, we provide a new proof of the boundedness of product singular integral operators on product Triebel-Lizorkin spaces.

Event-Triggered Boundary Control of Semilinear Hyperbolic Systems

We present an event-triggered boundary control scheme for hyperbolic systems. The trigger condition is based on predictions of the state on determinate sets, and the control input is updated only when the predictions deviate from the reference by a given margin. Nominal closed-loop stability, the absence of Zeno behaviour, and robustness to uncertainty and disturbances, are all established analytically. For the special case of linear systems, the trigger condition can be expressed in closed-form as an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L_{2}$</tex-math></inline-formula> -scalar product of kernels with the distributed state. The presented controller can also be combined with existing observers to solve the event-triggered output-feedback control problem. A numerical simulation demonstrates the effectiveness of the proposed approach.

Homotopy perturbation method and its convergence analysis for nonlinear collisional fragmentation equations

The exploration of collisional fragmentation pheno-mena remains largely unexplored, yet it holds considerable importance in numerous engineering and physical processes. Given the nonlinear nature of the governing equation, only a limited number of analytical solutions for the number density function corresponding to empirical kernels are available in the literature. This article introduces a semi-analytical approach using the homotopy perturbation method to obtain series solutions for the nonlinear collisional fragmentation equation. The method presented here can be readily adapted to solve both linear and nonlinear integral equations, eliminating the need for domain discretization. To gain deeper insights intothe accuracy of the proposed method, a convergence analysis is conducted. This analysis employs the concept of contractive mapping within the Banach space, a well-established technique universally acknowledged for ensuring convergence. Various collisional kernels (product and polymerization kernels), breakage distribution functions (binary and multiple breakage) and various initial particle distributions are considered to obtain the new series solutions. The obtained results are successfully compared against finite volume method [26] solutions in terms of number density functions and their moments. The error between the exact and obtained series solutions is shown in plots and tables to confirm the applicability and accuracy of the proposed method.

A New Multivariate Product Kernel Functions of the Beta Polynomial Family

A New Multivariate Product Kernel Functions of the Beta Polynomial Family

Sparse matrix‐vector and matrix‐multivector products for the truncated SVD on graphics processors

SummaryMany practical algorithms for numerical rank computations implement an iterative procedure that involves repeated multiplications of a vector, or a collection of vectors, with both a sparse matrix and its transpose. Unfortunately, the realization of these sparse products on current high performance libraries often deliver much lower arithmetic throughput when the matrix involved in the product is transposed. In this work, we propose a hybrid sparse matrix layout, named CSRC, that combines the flexibility of some well‐known sparse formats to offer a number of appealing properties: (1) CSRC can be obtained at low cost from the popular CSR (compressed sparse row) format; (2) CSRC has similar storage requirements as CSR; and especially, (3) the implementation of the sparse product kernels delivers high performance for both the direct product and its transposed variant on modern graphics accelerators thanks to a significant reduction of atomic operations compared to a conventional implementation based on CSR. This solution thus renders considerably higher performance when integrated into an iterative algorithm for the truncated singular value decomposition (SVD), such as the randomized SVD or, as demonstrated in the experimental results, the block Golub–Kahan–Lanczos algorithm.

Temporal Network Kernel Density Estimation

Kernel density estimation (KDE) is a widely used method in geography to study concentration of point pattern data. Geographical networks are 1.5 dimensional spaces with specific characteristics, analyzing events occurring on networks (accidents on roads, leakages of pipes, species along rivers, etc.). In the last decade, they required the extension of spatial KDE. Several versions of Network KDE (NKDE) have been proposed, each with their particular advantages and disadvantages, and are now used on a regular basis. However, scant attention has been given to the temporal extension of NKDE (TNKDE). In practice, when the studied events happen at specific time points and are constrained on a network, the methodologies used by geographers tend to overlook either the network or the temporal dimension. Here we propose a TNKDE based on the recent development of NKDE and the product of kernels. We also adapt classical methods of KDE (Diggle's correction, Abramson's adaptive bandwidth and bandwidth selection by leave‐one‐out maximum likelihood). We also illustrate the method with Montreal road crashes involving a pedestrian between 2016 and 2019.

Performance Portable Batched Sparse Linear Solvers

Solving large number of small linear systems is increasingly becoming a bottleneck in computational science applications. While dense linear solvers for such systems have been studied before, batched sparse linear solvers are just starting to emerge. In this paper, we discuss algorithms for solving batched sparse linear systems and their implementation in the Kokkos Kernels library. The new algorithms are performance portable and map well to the hierarchical parallelism available in modern accelerator architectures. The sparse matrix vector product (SPMV) kernel is the main performance bottleneck of the Krylov solvers we implement in this work. The implementation of the batched SPMV and its performance are therefore discussed thoroughly in this paper. The implemented kernels are tested on different Central Processing Unit (CPU) and Graphic Processing Unit (GPU) architectures. We also develop batched Conjugate Gradient (CG) and batched Generalized Minimum Residual (GMRES) solvers using the batched SPMV. Our proposed solver was able to solve 20,000 sparse linear systems on V100 GPUs with a mean speedup of 76x and 924x compared to using a parallel sparse solver with a block diagonal system with all the small linear systems, and compared to solving the small systems one at a time, respectively. We see mean speedup of 0.51 compared to dense batched solver of cuSOLVER on V100, while using lot less memory. Thorough performance evaluation on three different architectures and analysis of the performance are presented.

An improved Representative Particle Monte Carlo method for the simulation of particle growth

Context. A rocky planet is formed out of the agglomeration of around 1040 cosmic dust particles. As dust aggregates grow by coagulation, their number decreases. But until they have grown to hundreds of kilometres, their number still remains well above the number of particles a computer model can handle directly. The growth from micrometres to planetesimal-sized objects therefore has to be modelled using statistical methods, often using size distribution functions or Monte Carlo methods. However, when the particles reach planetary masses, they must be treated individually. This can be done by defining two classes of objects: a class of many small bodies or dust particles treated in a statistical way, and a class of individual bodies such as one or more planets. This introduces a separation between small and big objects, but it leaves open how to transition from small to big objects, and how to treat objects of intermediate sizes. Aims. We aim to improve the Representative Particle Monte Carlo (RPMC) method, which is often used for the study of dust coagulation, to be able to smoothly transition from the many-particle limit into the single-particle limit. Results. Our new version of the RPMC method allows for variable swarm masses, making it possible to refine the mass resolution where needed. It allows swarms to consist of few numbers of particles, and it includes a treatment of the transition from swarm to individual particles. The correctness of the method for a simplified two-component test case is validated with an analytical argument. The method is found to retain statistical balance and to accurately describe runaway growth, as is confirmed with the standard constant kernel, linear kernel, and product kernel tests as well as by comparison with a fiducial non-representative Monte Carlo simulation.

Information Theory With Kernel Methods

We consider the analysis of probability distributions through their associated covariance operators from reproducing kernel Hilbert spaces. We show that the von Neumann entropy and relative entropy of these operators are intimately related to the usual notions of Shannon entropy and relative entropy, and share many of their properties. They come together with efficient estimation algorithms from various oracles on the probability distributions. We also consider product spaces and show that for tensor product kernels, we can define notions of mutual information and joint entropies, which can then characterize independence perfectly, but only partially conditional independence. We finally show how these new notions of relative entropy lead to new upper-bounds on log partition functions, that can be used together with convex optimization within variational inference methods, providing a new family of probabilistic inference methods.

Efficient cross-validation traversals in feature subset selection

Sparse and robust classification models have the potential for revealing common predictive patterns that not only allow for categorizing objects into classes but also for generating mechanistic hypotheses. Identifying a small and informative subset of features is their main ingredient. However, the exponential search space of feature subsets and the heuristic nature of selection algorithms limit the coverage of these analyses, even for low-dimensional datasets. We present methods for reducing the computational complexity of feature selection criteria allowing for higher efficiency and coverage of screenings. We achieve this by reducing the preparation costs of high-dimensional subsets {mathscr {O}}({n}m^2) to those of one-dimensional ones {mathscr {O}}(m^2). Our methods are based on a tight interaction between a parallelizable cross-validation traversal strategy and distance-based classification algorithms and can be used with any product distance or kernel. We evaluate the traversal strategy exemplarily in exhaustive feature subset selection experiments (perfect coverage). Its runtime, fitness landscape, and predictive performance are analyzed on publicly available datasets. Even in low-dimensional settings, we achieve approximately a 15-fold increase in exhaustively generating distance matrices for feature combinations bringing a new level of evaluations into reach.

On equilibrium solution to a singular coagulation equation with source and efflux

In this study, we analyze the existence of a steady-state solution to a coagulation equation with source and efflux for the following singular coagulation kernel: K(x,y)=1+xλ+yλ(xy)σ,where 0≤σ≤12 and 0≤λ−σ≤1. The uniqueness of the steady-state solution is found in the space of functions which are continuous in (0,∞). Also, we find an explicit form of the equilibrium solution to the problem with sources and effluxes for the constant and product kernels; further, for a linear kernel, we provide a closed-form of the equilibrium solution. We provide a numerical example to support the proposed study.

An efficient kernel-based method for solving nonlinear generalized Benjamin-Bona-Mahony-Burgers equation in irregular domains

An iterative kernel-based method is proposed for solving the nonlinear generalized Benjamin-Bona-Mahony-Burgers equation in regular and irregular domains. The method is based on the positive definite kernels pseudospectral method. We have used an iterative linearization scheme to overcome the nonlinearity of the problem. The convergence of the linearization scheme is proved. The space-time kernels are used for the full discretization of the problems in the temporal and spatial domains. Here, we construct the multidimensional kernels using the product of one-dimensional kernels. Numerical results and comparisons are presented for the several two and three-dimensional problems with the various domains. The obtained results confirm the efficiency and accuracy of the proposed method.

A BLIS-like matrix multiplication for machine learning in the RISC-V ISA-based GAP8 processor

We address the efficient realization of matrix multiplication (gemm), with application in the convolution operator for machine learning, for the RISC-V core present in the GreenWaves GAP8 processor. Our approach leverages BLIS (Basic Linear Algebra Instantiation Software) to develop an implementation that (1) re-organizes the gemm algorithm adapting its micro-kernel to exploit the hardware-supported dot product kernel in the GAP8; (2) explicitly orchestrates the data transfers across the hierarchy of scratchpad memories via DMA (direct memory access); and (3) operates with integer arithmetic.