Kernel Approximation Research Articles

P-Edge/vertex-connected vertex cover: Parameterized and approximation algorithms

We introduce and study two natural generalizations of the Connected Vertex Cover (VC) problem: the p-Edge-Connected and p-Vertex-Connected VC problem (where p≥2 is a fixed integer). We obtain an 2O(pk)nO(1)-time algorithm for p-Edge-Connected VC and an 2O(k2)nO(1)-time algorithm for p-Vertex-Connected VC. Thus, like Connected VC, both constrained VC problems are FPT. Furthermore, like Connected VC, neither problem admits a polynomial kernel unless NP ⊆ coNP/poly, which is highly unlikely. We prove however that both problems admit time efficient polynomial sized approximate kernelization schemes. Finally, we describe a 2(p+1)-approximation algorithm for the p-Edge-Connected VC. The proofs for the new VC problems require more sophisticated arguments than for Connected VC. In particular, for the approximation algorithm we use Gomory-Hu trees and for the approximate kernels a result on small-size spanning p-vertex/edge-connected subgraphs of a p-vertex/edge-connected graph by Nishizeki and Poljak (1994) [30] and Nagamochi and Ibaraki (1992) [27].

GPU-accelerated approximate kernel method for quantum machine learning.

We introduce Quantum Machine Learning (QML)-Lightning, a PyTorch package containing graphics processing unit (GPU)-accelerated approximate kernel models, which can yield trained models within seconds. QML-Lightning includes a cost-efficient GPU implementation of FCHL19, which together can provide energy and force predictions with competitive accuracy on a microsecond per atom timescale. Using modern GPU hardware, we report learning curves of energies and forces as well as timings as numerical evidence for select legacy benchmarks from atomistic simulation including QM9, MD-17, and 3BPA.

Stabilized Lagrange Interpolation Collocation Method: A meshfree method incorporating the advantages of finite element method

Since most approximation functions in meshfree methods are rational functions which do not possess the Kronecker delta property, how to achieve exact integration and accurately impose the essential boundary conditions are two typical difficulties for meshfree methods. In this paper, a new stabilized Lagrange interpolation collocation method (SLICM) is proposed in which the Lagrange interpolation (LI) is employed for the approximation in a meshfree method. This method can satisfy the high order integration constraints which can conserves the high order consistency conditions in the integration form. This property leads to the exact integration in the subdomains and optimal convergence for the proposed method. Meanwhile, performing the integration in subdomains can also reduce the condition number of discrete matrix, which improves the stability of the algorithm. Since the Lagrange interpolation approximation has Kronecker delta property, the essential boundary conditions can be simply and exactly imposed like the finite element method, which further improves the accuracy of this method. Convergence studies present that the same convergence rate can be attained for utilizing the odd and even order LI shape functions, while the convergence rate is reduced if the odd order basis function is employed in the reproducing kernel (RK) approximation. Numerical examples validate the high accuracy and convergence as well as good stability of the presented method, which can outperform the direct collocation method and the stabilized collocation method based on RK approximation.

Deep Dynamic Scene Deblurring From Optical Flow

Deblurring can not only provide visually more pleasant pictures and make photography more convenient, but also can improve the performance of objection detection as well as tracking. However, removing dynamic scene blur from images is a non-trivial task as it is difficult to model the non-uniform blur mathematically. Several methods first use single or multiple images to estimate optical flow (which is treated as an approximation of blur kernels) and then adopt non-blind deblurring algorithms to reconstruct the sharp images. However, these methods cannot be trained in an end-to-end manner and are usually computationally expensive. In this paper, we explore optical flow to remove dynamic scene blur by using the multi-scale spatially variant recurrent neural network (RNN). We utilize FlowNets to estimate optical flow from two consecutive images in different scales. The estimated optical flow provides the RNN weights in different scales so that the weights can better help RNNs to remove blur in the feature spaces. Finally, we develop a convolutional neural network (CNN) to restore the sharp images from the deblurred features. Both quantitatively and qualitatively evaluations on the benchmark datasets demonstrate that the proposed method performs favorably against state-of-the-art algorithms in terms of accuracy, speed, and model size.

Stabilized and variationally consistent integrated meshfree formulation for advection-dominated problems

Meshfree methods such as reproducing kernel (RK) approximation are suitable for modeling fluid flow problems because of their flexibility in controlling local smoothness and order of basis, as well as straightforward construction of higher-order gradients. However, Eulerian-described partial differential equations often suffer from numerical instability in the solution of Bubnov–Galerkin​ methods because of the non-self-adjoint advection terms. This is true for both mesh-based and meshfree methods. Although stabilized Petrov–Galerkin formulation, such as the variational multiscale method, has addressed this issue, a careless selection of the numerical quadrature can still result in variational inconsistency in the Galerkin weak form, which leads to a suboptimal convergence. Strong advection could also amplify the unstable modes from a reduced quadrature. This study provides a variationally consistent (VC) approach to correct the loss of Galerkin exactness in nodally integrated meshfree modeling for the advection diffusion equation. A gradient stabilization method is proposed to enhance the coercivity of the system. Several numerical examples are provided to verify the effectiveness and efficiency of the proposed approaches in modeling advection dominated problems.

Towards a Unified Quadrature Framework for Large-Scale Kernel Machines.

In this paper, we develop a quadrature framework for large-scale kernel machines via a numerical integration representation. Considering that the integration domain and measure of typical kernels, e.g., Gaussian kernels, arc-cosine kernels, are fully symmetric, we leverage a numerical integration technique, deterministic fully symmetric interpolatory rules, to efficiently compute quadrature nodes and associated weights for kernel approximation. Thanks to the full symmetric property, the applied interpolatory rules are able to reduce the number of needed nodes while retaining a high approximation accuracy. Further, we randomize the above deterministic rules by the classical Monte-Carlo sampling and control variates techniques with two merits: 1) The proposed stochastic rules make the dimension of the feature mapping flexibly varying, such that we can control the discrepancy between the original and approximate kernels by tuning the dimnension. 2) Our stochastic rules have nice statistical properties of unbiasedness and variance reduction. In addition, we elucidate the relationship between our deterministic/stochastic interpolatory rules and current typical quadrature based rules for kernel approximation, thereby unifying these methods under our framework. Experimental results on several benchmark datasets show that our methods compare favorably with other representative kernel approximation based methods.

Tutorial on PCA and approximate PCA and approximate kernel PCA

Principal Component Analysis (PCA) is one of the most widely used data analysis methods in machine learning and AI. This manuscript focuses on the mathematical foundation of classical PCA and its application to a small-sample-size scenario and a large dataset in a high-dimensional space scenario. In particular, we discuss a simple method that can be used to approximate PCA in the latter case. This method can also help approximate kernel PCA or kernel PCA (KPCA) for a large-scale dataset. We hope this manuscript will give readers a solid foundation on PCA, approximate PCA, and approximate KPCA.

Predicting the opening state of a group of windows in an open-plan office by using machine learning models

Window operation is among one of the most influencing factors on the indoor air quality (IAQ). The opening state of the windows can modify the air exchange rate and as such the pollutant transfer between indoor and outdoor environments. In this paper, we focus on the modeling of the windows opening state in a real open-plan office with five windows. For this purpose, three machine learning-based models were implemented: (i) Decision Tree, (ii) k-Nearest Neighbors and (iii) Kernel Approximation. IAQ, climatic parameters and the opening state of the windows have been monitored during an entire period of 18 months. The information about: (i) the environmental factors from the previous 24th hour and (ii) the current time (month, day of the week, hour of the day) was used to predict the current state of the windows. The predictor importance estimation and the calculated autocorrelation functions showed that the three most relevant factors were: the previous 24th hour of the windows status, the current time and the previous 24th hour of the prevailing mean outdoor air temperature. The three models perform well with the testing sets according to the different evaluation indicators. The developed methods can be helpful for understanding occupant behavior and also for controlling indoor air pollutants levels in buildings, either as a standalone model or a part of a real-time IAQ monitoring system.

Efficient randomized tensor-based algorithms for function approximation and low-rank kernel interactions

In this paper, we introduce a method for multivariate function approximation using function evaluations, Chebyshev polynomials, and tensor-based compression techniques via the Tucker format. We develop novel randomized techniques to accomplish the tensor compression, provide a detailed analysis of the computational costs, provide insight into the error of the resulting approximations, and discuss the benefits of the proposed approaches. We also apply the tensor-based function approximation to develop low-rank matrix approximations to kernel matrices that describe pairwise interactions between two sets of points; the resulting low-rank approximations are efficient to compute and store (the complexity is linear in the number of points). We present an adaptive version of the function and kernel approximation that determines an approximation that satisfies a user-specified relative error over a set of random points. We extend our approach to the case where the kernel requires repeated evaluations for many values of (hyper)parameters that govern the kernel. We give detailed numerical experiments on example problems involving multivariate function approximation, low-rank matrix approximations of kernel matrices involving well-separated clusters of sources and target points, and a global low-rank approximation of kernel matrices with an application to Gaussian processes. We observe speedups up to 18X over standard matrix-based approaches.

Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond.

The class of random features is one of the most popular techniques to speed up kernel methods in large-scale problems. Related works have been recognized by the NeurIPS Test-of-Time award in 2017 and the ICML Best Paper Finalist in 2019. The body of work on random features has grown rapidly, and hence it is desirable to have a comprehensive overview on this topic explaining the connections among various algorithms and theoretical results. In this survey, we systematically review the work on random features from the past ten years. First, the motivations, characteristics and contributions of representative random features based algorithms are summarized according to their sampling schemes, learning procedures, variance reduction properties and how they exploit training data. Second, we review theoretical results that center around the following key question: how many random features are needed to ensure a high approximation quality or no loss in the empirical/expected risks of the learned estimator. Third, we provide a comprehensive evaluation of popular random features based algorithms on several large-scale benchmark datasets and discuss their approximation quality and prediction performance for classification. Last, we discuss the relationship between random features and modern over-parameterized deep neural networks (DNNs), including the use of high dimensional random features in the analysis of DNNs as well as the gaps between current theoretical and empirical results. This survey may serve as a gentle introduction to this topic, and as a users' guide for practitioners interested in applying the representative algorithms and understanding theoretical results under various technical assumptions. We hope that this survey will facilitate discussion on the open problems in this topic, and more importantly, shed light on future research directions. Due to the page limit, we suggest the readers refer to the full version of this survey https://arxiv.org/abs/2004.11154.

Approximate kernel PCA: Computational versus statistical trade-off

Kernel methods are powerful learning methodologies that allow to perform nonlinear data analysis. Despite their popularity, they suffer from poor scalability in big data scenarios. Various approximation methods, including random feature approximation, have been proposed to alleviate the problem. However, the statistical consistency of most of these approximate kernel methods is not well understood except for kernel ridge regression wherein it has been shown that the random feature approximation is not only computationally efficient but also statistically consistent with a minimax optimal rate of convergence. In this paper, we investigate the efficacy of random feature approximation in the context of kernel principal component analysis (KPCA) by studying the trade-off between computational and statistical behaviors of approximate KPCA. We show that the approximate KPCA is both computationally and statistically efficient compared to KPCA in terms of the error associated with reconstructing a kernel function based on its projection onto the corresponding eigenspaces. The analysis hinges on Bernstein-type inequalities for the operator and Hilbert–Schmidt norms of a self-adjoint Hilbert–Schmidt operator-valued U-statistics, which are of independent interest.

A deformation-dependent coupled Lagrangian/semi-Lagrangian meshfree hydromechanical formulation for landslide modeling

The numerical modelling of natural disasters such as landslides presents several challenges for conventional mesh-based methods such as the finite element method (FEM) due to the presence of numerically challenging phenomena such as severe material deformation and fragmentation. In contrast, meshfree methods such as the reproducing kernel particle method (RKPM) possess unique features conducive to modelling extreme events such as the absence of a structured mesh and the ease of adaptive refinement, among others. While the semi-Lagrangian reproducing kernel (SL-RK) shape functions of RKPM defined in the current configuration have proven to be effective in extreme event modelling, the computational cost for the re-evaluation of the shape functions at every time step is costly. In this work, a deformation-dependent coupling of the Lagrangian reproducing kernel (L-RK) and SL-RK approximations is proposed for the solution of a hydro-mechanical formulation for effective simulations of landslides. The ramp function is constructed based on an equivalent plastic strain as a deformation-dependent transition from L-RK shape functions to SL-RK ones as the deformation progresses. The particular focus of the paper will be on modelling seepage-induced landslides with a mixed u–p formulation to couple the solid and fluid phases. Examples are presented to examine the effectiveness of this coupled Lagrangian/semi-Lagrangian reproducing kernel (L–SL RK) formulation and to highlight its performance in landslide modelling.

A reduced-order fast reproducing kernel collocation method for nonlocal models with inhomogeneous volume constraints

This paper is concerned with the implementations of the meshfree-based reduced-order model (ROM) to time-dependent nonlocal models with inhomogeneous volume constraints. Generally, when using ROM for nonlocal models, the projection of nonlocal volume constraints needs to be computed in every time step to handle the nonlocal boundary conditions. Up to now, only finite element methods (FEM) can work well in constructing ROM for nonlocal models, since the interpolation property of the FEM basis functions makes it easy to obtain such a projection. But if one tries to develop ROM based on existing meshfree methods for nonlocal models, the projection in every time step will lead to a full-order discrete system and is highly time-consuming, since the basis functions of these methods do not meet interpolation property. To overcome the above difficulties, we introduce a mixed reproducing kernel (RK) approximation with nodal interpolation property to develop a meshfree collocation method for nonlocal models and use it to construct ROM. Thanks to the nodal interpolation property, the projection of nonlocal boundary conditions can be obtained explicitly. This ROM is developed using numerical results as snapshots by a full-order model in a small time interval [ 0 , t 1 ] . The surrogate model, which is constructed by POD (proper orthogonal decomposition)-Galerkin approach, leads to a discrete system with far fewer degrees of freedom than the original meshfree method. Numerical experiments for nonlocal problems including nonlocal diffusion and peridynamics are presented to show that our method meets almost the same accuracy with a very small computational cost compared with the full-order meshfree approach.

Efficient kernel fuzzy clustering via random Fourier superpixel and graph prior for color image segmentation

The kernel fuzzy clustering algorithms can explore the non-linear relations of pixels in an image. However, most of kernel-based methods are computationally expensive for color image segmentation and neglect the inherent locality information in images. To alleviate these limitations, this paper proposes a novel kernel fuzzy clustering framework for fast color image segmentation. More specifically, we first design a new superpixel generation method that uses random Fourier maps to approximate Gaussian kernels and explicitly represent high-dimensional features of pixels. Clustering superpixels instead of large-sized pixels speeds up the segmentation of a color image significantly. More importantly, the features of superpixels used by fuzzy clustering are also calculated in the approximated kernel space and the local relationships between superpixels are depicted as a graph prior and appended into the objective function of fuzzy clustering as a Kullback–Leibler divergence term. This results in a new fuzzy clustering model that can further improve the accuracy of the image segmentation. Experiments on synthetic and real-world color image datasets verify the superiority and high efficiency of the proposed approach.

A shift model based on particle collisions – preserving kinetic energy and potential energy in a constant force field – to avoid particle clustering in SPH

In Smoothed Particle Hydrodynamics (SPH), the motion of particles is based on symmetric inter-particle forces, such that the conservation of momentum is guaranteed. Inter-particle forces, however, can not prevent particle clustering. Clustering may occur for several reasons. A fundamental issue is the tensile instability, which is caused by the properties of the kernel gradient. Clustering may also be caused by discontinuities in the pressure (e.g. due to surface tension) and the pressure gradient (e.g. due to gravity), which may lead to instabilities around the interface between two fluids (Kruisbrink et al., 2018 [1]). Wall penetration is also a form of particle clustering.To suppress particle clustering, the use of kinematic conditions (motion) rather than dynamic conditions (forces) was previously explored in the concept of particle collisions by Kruisbrink et al. (2018) [1]. In this concept, the conservation of momentum is always satisfied, whilst the conservation of energy is only satisfied for fully elastic collisions. They conclude that inelastic collisions are more stable than elastic collisions, but that energy is dissipated. In the present paper, the particle collision model is further explored and extended to a particle collision shift model, which in itself is non-dissipative, i.e. it preserves kinetic energy as well as potential energy in a constant force field, like gravitation. This is achieved by changing the particle positions due to inter-particle collisions, but not their velocities. Other popular and commonly used particle shift methods, such as the Fickian shift method of Lind et al. (2012) [2], are based on the particle concentration. They are non-conservative and as such dissipative to some extent.In this paper, the new particle collision shift model is introduced and explored in several case studies. This model is compared with another stabilization method, the Fickian Shift model of Lind et al. (2012) [2]. The particle collision shift model performs slightly better in terms of particle stability and reduction of energy dissipation in most of the case studies. However, it is simple (one algorithm only with few coefficients, no special treatment at the free surface) and efficient (no kernel approximation, less CPU time). This work is based on incompressible SPH (ISPH). The shift model may however also be applied to weakly compressible SPH.

PDE-Based Group Equivariant Convolutional Neural Networks

We present a PDE-based framework that generalizes Group equivariant Convolutional Neural Networks (G-CNNs). In this framework, a network layer is seen as a set of PDE-solvers where geometrically meaningful PDE-coefficients become the layer’s trainable weights. Formulating our PDEs on homogeneous spaces allows these networks to be designed with built-in symmetries such as rotation in addition to the standard translation equivariance of CNNs. Having all the desired symmetries included in the design obviates the need to include them by means of costly techniques such as data augmentation. We will discuss our PDE-based G-CNNs (PDE-G-CNNs) in a general homogeneous space setting while also going into the specifics of our primary case of interest: roto-translation equivariance. We solve the PDE of interest by a combination of linear group convolutions and nonlinear morphological group convolutions with analytic kernel approximations that we underpin with formal theorems. Our kernel approximations allow for fast GPU-implementation of the PDE-solvers; we release our implementation with this article in the form of the LieTorch extension to PyTorch, available at https://gitlab.com/bsmetsjr/lietorch. Just like for linear convolution, a morphological convolution is specified by a kernel that we train in our PDE-G-CNNs. In PDE-G-CNNs, we do not use non-linearities such as max/min-pooling and ReLUs as they are already subsumed by morphological convolutions. We present a set of experiments to demonstrate the strength of the proposed PDE-G-CNNs in increasing the performance of deep learning-based imaging applications with far fewer parameters than traditional CNNs.

Predicting Hepatitis B to be acute or chronic in an infected person using machine learning algorithm

Hepatitis B is a viral infection which causes liver damage. It can lead to death. This hepatitis B along with Hepatitis C can cause hepatocellular carcinoma and liver cirrhosis. In this paper it is discussed about Hepatitis B found positive in a person's blood test is acute or chronic. This research work plans to code an endurance forecast model for the dataset which contains the boundaries or data of Hepatitis-B patients. At first the information will be pre-prepared, to improve fit for additional handling and for being in satisfactory configuration for the calculations. At that point, several calculations to indicate the forecast and draw out the precision of the model. What's more, further contrast those calculations with indicate the calculation with most adequacy. The precision is determined by contrasting the anticipated result and ongoing result of the patient. In light of thinking about different boundaries, the model will anticipate the danger of a patient of his endurance rate of acute or chronic infected person accuracy. In this paper we use Stochastic Gradient algorithm to find the Co-connection between boundaries of the date set, kernel approximation to finalise the resulting accuracy of the acute or choric prediction of patients and SVM method we use to clustering the kernel approximation calculation and connection analysis.

Advanced general particle dynamics with nonlocal foundation for fracture analysis

AbstractThe general particle dynamics (GPD), which is based on the kernel approximation method and Navier–Stokes equation, was developed from smoothed‐particle hydrodynamics to simulate the fracture behaviors by using the collective of damage variable to describe the damage evolution behaviors. In this paper, an advanced GPD with a nonlocal foundation is proposed for better description of the solid mechanics. The nonlocal vector calculus and microscopic constitutive model are employed to derive the governing equations in the proposed theory. The novel GPD with a nonlocal core is a generalization, which permits the inclusion of the microscopic constitutive model and the multi‐scale modeling compared with the previous GPD. To accomplish this generalization, the relationship between GPD and the previous kernel‐based theory is established. The novel GPD is superior in the fracture problems and multi‐scale modeling. Some numerical experiments are launched to verify the ability of the novel GPD.

A highly efficient and accurate Lagrangian–Eulerian stabilized collocation method (LESCM) for the fluid–rigid body interaction problems with free surface flow

A Lagrangian–Eulerian stabilized collocation method (LESCM) for the fluid–structure interaction problems involving free surface flow is presented in this work, in which the structure is modeled by a rigid body. This method is an evolution of the material point method and particle-in-cell methods which are based on the hybrid Lagrangian–Eulerian description. The problem domain of the fluid and structures is discretized into the Lagrangian particles which carry the information, and the problem domain covering the entire movement space is discretized into the uniformly distributed Eulerian background nodes. The coupling governing equations of the fluid, structures and interfaces are solved by the meshfree stabilized collocation method (SCM) employing the reproducing kernel (RK) approximation on the Eulerian nodes. The solution is very efficient since the Eulerian nodes are set to be the initial positions in each time step and it is no need to reconstruct the shape function. The information mappings between the Lagrangian particles and the Eulerian nodes are also conducted by the RK approximation which can keep the mass and momentum conservation of the solution. The cell-cut algorithm is introduced to couple the fluid and the structures which can solve the fluid pressure and the fluid–structure interactional force simultaneously and avoid the complicated iterations of the traditional interaction algorithms. Several numerical examples including the collapse of water column with a rigid barrier, water entry of a half-buoyant circular cylinder and a rigid box rotating and sinking in water are simulated, which demonstrate the high accuracy, high efficiency and good stability of the proposed method. This method can be extensively applied to the engineering applications of fluid–rigid body interactions.

TRF: Learning Kernels with Tuned Random Features

Random Fourier features (RFF) are a popular set of tools for constructing low-dimensional approximations of translation-invariant kernels, allowing kernel methods to be scaled to big data. Apart from their computational advantages, by working in the spectral domain random Fourier features expose the translation invariant kernel as a density function that may, in principle, be manipulated directly to tune the kernel. In this paper we propose selecting the density function from a reproducing kernel Hilbert space to allow us to search the space of all translation-invariant kernels. Our approach, which we call tuned random features (TRF), achieves this by approximating the density function as the RKHS-norm regularised least-squares best fit to an unknown ``true'' optimal density function, resulting in a RFF formulation where kernel selection is reduced to regularised risk minimisation with a novel regulariser. We derive bounds on the Rademacher complexity for our method showing that our random features approximation method converges to optimal kernel selection in the large N,D limit. Finally, we prove experimental results for a variety of real-world learning problems, demonstrating the performance of our approach compared to comparable methods.