Gaussian Process Kernel Research Articles

Kernel Selection for Gaussian Process in Cosmology: With Approximate Bayesian Computation Rejection and Nested Sampling

The Gaussian process (GP) has gained much attention in cosmology due to its ability to reconstruct cosmological data in a model-independent manner. In this study, we compare two methods for GP kernel selection: approximate Bayesian computation (ABC) rejection and nested sampling. We analyze three types of data: cosmic chronometer data, type Ia supernovae data, and gamma-ray burst data, using five kernel functions. To evaluate the differences between kernel functions, we assess the strength of evidence using Bayes factors. Our results show that, for ABC rejection, the Matérn kernel with ν = 5/2 (M52 kernel) outperformes the commonly used radial basis function (RBF) kernel in approximating all three data sets. Bayes factors indicate that the M52 kernel typically supports the observed data better than the RBF kernel but with no clear advantage over other alternatives. However, nested sampling gives different results, with the M52 kernel losing its advantage. Nevertheless, Bayes factors indicate no significant dependence of the data on each kernel.

Probabilistic Forecasting of Electricity Demand Incorporating Mobility Data

Due to extreme weather conditions and anomalous events such as the COVID-19 pandemic, utilities and grid operators worldwide face unprecedented challenges. These unanticipated changes in trends introduce new uncertainties in conventional short-term electricity demand forecasting (EDF) since its result depends on recent usage as an input variable. In order to quantify the uncertainty of EDF effectively, this paper proposes a comprehensive probabilistic EFD method based on Gaussian process regression (GPR) and kernel density estimation (KDE). GPR is a non-parametric method based on Bayesian theory, which can handle the uncertainties in EDF using limited data. Mobility data is incorporated to manage uncertainty and pattern changes and increase forecasting model scalability. This study first performs a correlation study for feature selection that comprises weather, renewable and non-renewable energy, and mobility data. Then, different kernel functions of GPR are compared, and the optimal function is recommended for real applications. Finally, real data are used to validate the effectiveness of the proposed model and are elaborated with three scenarios. Comparison results with other conventional adopted methods show that the proposed method can achieve high forecasting accuracy with a minimum quantity of data while addressing forecasting uncertainty, thus improving decision-making.

Exciton diffusion in amorphous organic semiconductors: Reducing simulation overheads with machine learning.

Simulations of exciton and charge hopping in amorphous organic materials involve numerous physical parameters. Each of these parameters must be computed from costly abinitio calculations before the simulation can commence, resulting in a significant computational overhead for studying exciton diffusion, especially in large and complex material datasets. While the idea of using machine learning to quickly predict these parameters has been explored previously, typical machine learning models require long training times, which ultimately contribute to simulation overheads. In this paper, we present a new machine learning architecture for building predictive models for intermolecular exciton coupling parameters. Our architecture is designed in such a way that the total training time is reduced compared to ordinary Gaussian process regression or kernel ridge regression models. Based on this architecture, we build a predictive model and use it to estimate the coupling parameters which enter into an exciton hopping simulation in amorphous pentacene. We show that this hopping simulation is able to achieve excellent predictions for exciton diffusion tensor elements and other properties as compared to a simulation using coupling parameters computed entirely from density functional theory. This result, along with the short training times afforded by our architecture, shows how machine learning can be used to reduce the high computational overheads associated with exciton and charge diffusion simulations in amorphous organic materials.

Systematics in asteroseismic modelling: application of a correlated noise model for oscillation frequencies

ABSTRACT The detailed modelling of stellar oscillations is a powerful approach to characterizing stars. However, poor treatment of systematics in theoretical models leads to misinterpretations of stars. Here, we propose a more principled statistical treatment for the systematics to be applied to fitting individual mode frequencies with a typical stellar model grid. We introduce a correlated noise model based on a Gaussian process (GP) kernel to describe the systematics given that mode frequency systematics are expected to be highly correlated. We show that tuning the GP kernel can reproduce general features of frequency variations for changing model input physics and fundamental parameters. Fits with the correlated noise model better recover stellar parameters than traditional methods that either ignore the systematics or treat them as uncorrelated noise.

Model selection and signal extraction using Gaussian Process regression

We present a novel computational approach for extracting localized signals from smooth background distributions. We focus on datasets that can be naturally presented as binned integer counts, demonstrating our procedure on the CERN open dataset with the Higgs boson signature, from the ATLAS collaboration at the Large Hadron Collider. Our approach is based on Gaussian Process (GP) regression — a powerful and flexible machine learning technique which has allowed us to model the background without specifying its functional form explicitly and separately measure the background and signal contributions in a robust and reproducible manner. Unlike functional fits, our GP-regression-based approach does not need to be constantly updated as more data becomes available. We discuss how to select the GP kernel type, considering trade-offs between kernel complexity and its ability to capture the features of the background distribution. We show that our GP framework can be used to detect the Higgs boson resonance in the data with more statistical significance than a polynomial fit specifically tailored to the dataset. Finally, we use Markov Chain Monte Carlo (MCMC) sampling to confirm the statistical significance of the extracted Higgs signature.

Gaussian processes for glitch-robust gravitational-wave astronomy

ABSTRACT Interferometric gravitational-wave observatories have opened a new era in astronomy. The rich data produced by an international network enable detailed analysis of the curved space-time around black holes. With nearly 100 signals observed so far and thousands expected in the next decade, their population properties enable insights into stellar evolution and the expansion of our Universe. However, the detectors are afflicted by transient noise artefacts known as ‘glitches’ which contaminate the signals and bias inferences. Of the 90 signals detected to date, 18 were contaminated by glitches. This feasibility study explores a new approach to transient gravitational-wave data analysis using Gaussian processes, which model the underlying physics of the glitch-generating mechanism rather than the explicit realization of the glitch itself. We demonstrate that if the Gaussian process kernel function can adequately model the glitch morphology, we can recover the parameters of simulated signals. Moreover, we find that the Gaussian processes kernels used in this work are well suited to modelling long-duration glitches which are most challenging for existing glitch-mitigation approaches. Finally, we show how the time-domain nature of our approach enables a new class of time-domain tests of General Relativity, performing a re-analysis of the inspiral-merger-ringdown test on the first observed binary black hole merger. Our investigation demonstrates the feasibility of the Gaussian processes as an alternative to the traditional framework but does not yet establish them as a replacement. Therefore, we conclude with an outlook on the steps needed to realize the full potential of the Gaussian process approach.

Non-parametric Gaussian process kernel DMD and LS-SVM predictors revisited — A unifying approach

In this work, the prediction of a time series is formulated as a gaussian process regression, for different levels of noise. The gaussian regressor is translated into lower rank Dynamic Mode Decomposition methods that use kernels (K-DMD) — Kernel regression and Least Squares Support Vector Machines. The presented unified approach delivers an algorithm where the optimisation of the marginal likelihood function can be used to find the parameters of the kernel regression. The viability of the procedure is demonstrated on a chaotic series, with quite good adjustment results being obtained.

Bayesian Kernelized Matrix Factorization for Spatiotemporal Traffic Data Imputation and Kriging

Missingness and corruption are common problems for real-world traffic data. How to accurately perform imputation and prediction based on incomplete or even sparse traffic data becomes a critical research question in intelligent transportation systems. Low-rank matrix factorization (MF) is a common solution for the general missing value imputation problem. To better characterize and encode the strong spatial and temporal consistency in traffic data, existing work has introduced flexible spatial/temporal Gaussian process (GP) priors to model the latent factors in MF framework, which also allows us to perform kriging for unseen locations and virtual sensors. However, learning the hyperparameters in GP kernels remains a challenging task. In this paper, we present a Bayesian kernelized matrix factorization (BKMF) model with an efficient Markov chain Monte Carlo (MCMC) sampling algorithm for model inference. By learning the kernel hyperparameters from their marginal posteriors through a slice sampling treatment and updating the latent factors alternatively with Gibbs sampling, we achieve a fully Bayesian model for the spatiotemporally kernelized (i.e., GP prior regularized) MF framework. We apply BKMF on both imputation and kriging tasks, and our results demonstrate the superiority of BKMF compared with state-of-the-art spatiotemporal models. In addition, we also explore the effects of different GP kernels in characterizing networked spatiotemporal traffic state data.

Probabilistic learning for pulsar classification

In this work, we explore the possibility of using probabilistic learning to identify pulsar candidates. We make use of Deep Gaussian Process (DGP) and Deep Kernel Learning (DKL). Trained on a balanced training set in order to avoid the effect of class imbalance, the performance of the models, achieving relatively high probability of differentiating the positive class from the negative one (roc-auc ∼ 0.98), is very promising overall. We estimate the predictive entropy of each model predictions and find that DKL is more confident than DGP in its predictions and provides better uncertainty calibration. Upon investigating the effect of training with imbalanced dataset on the models, results show that each model performance decreases with an increasing number of the majority class in the training set. Interestingly, with a number of negative class 10× that of positive class, the models still provide reasonably well calibrated uncertainty, i.e. an expected Uncertainty Calibration Error (UCE) less than 6%. We also show in this study how, in the case of relatively small amount of training dataset, a convolutional neural network based classifier trained via Bayesian Active Learning by Disagreement (BALD) performs. We find that, with an optimized number of training examples, the model — being the most confident in its predictions — generalizes relatively well and produces the best uncertainty calibration which corresponds to UCE = 3.118%.

A Generative Model for Quasar Spectra

We build a multi-output generative model for quasar spectra and the properties of their black hole engines, based on a Gaussian process latent-variable model. This model treats every quasar as a vector of latent properties such that the spectrum and all physical properties of the quasar are associated with non-linear functions of those latent parameters; the Gaussian process kernel functions define priors on the function space. Our generative model is trained with a justifiable likelihood function that allows us to treat heteroscedastic noise and missing data correctly, which is crucial for all astrophysical applications. It can simultaneously predict unobserved spectral regions and the physical properties of quasars in held-out test data. We apply the model to rest-frame ultraviolet and optical quasar spectra for which precise black hole masses (based on reverberation-mapping measurements) are available. Unlike reverberation-mapping studies that require multi-epoch data, our model predicts black hole masses from single-epoch spectra—even with limited spectral coverage. We demonstrate the capabilities of the model by predicting black hole masses and unobserved spectral regions. We find that we predict black hole masses at close to the best possible accuracy.

Nonparametric inference of stochastic differential equations based on the relative entropy rate

The information detection of complex systems from data is currently undergoing a revolution, driven by the emergence of big data and machine learning methodology. Discovering governing equations and quantifying the dynamical properties of complex systems are among the central challenges. In this work, we devised a nonparametric approach to learning the relative entropy rate from observations of stochastic differential equations with different drift functions. The estimator corresponding to the relative entropy rate is then presented via the Gaussian process kernel theory. Meanwhile, this approach enables us to extract the governing equations. We illustrate our approach with several examples. Numerical experiments show the proposed approach performs well for rational drift functions, not only polynomial drift functions.

Estimating the stiffness of kiwifruit based on the fusion of instantaneous tactile sensor data and machine learning schemes

Measuring the ripeness of fruits is one of the critical factors in achieving real-time quality control and sorting of fruit by growers and postharvest managers. However, recent tactile sensing approaches for fruit ripeness detection have suffered setbacks due to: (1) the nonlinear relationship between the sensor output and the true stiffness of fruits; and (2) the angle of contact, referred to as the inclination angle, between the sensor and the outer surface of the fruit. In this paper, we propose a non-destructive tactile sensing approach for estimating the stiffness of fruits, using kiwifruit as a case study. Our sensor configuration is based on a three-probe piezoresistive cantilever beam, allowing us to obtain relatively stable sensor outputs that are independent of the inclination angle of the fruit surface. Our stiffness estimation approach is based on the combination of instantaneous sensor outputs with 63 regression-based machine learning models comprising of neural networks, Gaussian process, support vector machines, and decision trees. For experiments, we used several kiwifruit samples at diverse ripeness levels. The extracted sensor data was used to train the learning models over a 10-fold cross-validation technique, allowing us to find the nonlinear relationships between the instantaneous sensor outputs and the ground truth stiffness of the fruit. Our pairwise statistical comparison by the Wilcoxon test at 5% significance revealed the competitive performance frontiers of our approach for stiffness prediction; the Gaussian process kernel functions and the binary trees outperformed other models at a mean squared error (MSE) of 1.0 and 2×10−23, respectively. Most neural network models achieved competitive learning performance at MSE less than 10−5 and the utmost performance being a pyramidal class of feed-forward neural architectures. The results portray the potential of achieving accurate ripeness estimation of fruit using intelligent tactile sensors with fast machine learning schemes across the supply chain.

A Bayesian Optimization Approach to the Simultaneous Extraction of Intrinsic Physical Parameters fromT1 andT2 Relaxation Responses

SummaryNuclear magnetic resonance (NMR) relaxation responses in porous media provide a sensitive probe of the microstructure and yet are influenced by a number of factors which are not easily detangled. Low-field T2 transverse relaxation measurements can be carried out quickly and are frequently used as pore size distributions, while adding T1 longitudinal relaxation measurements provides additional insights into surface properties and fluid content. Here we present an inverse solution workflow extracting related intrinsic physical parameters of the system by fitting experiment and numerical simulation(s). An efficient NMR forward solver for the simultaneous calculation of T1 and T2 responses is introduced, which honors existing inequality relationships between T1 and T2 parameters. We propose a Bayesian optimization approach that jointly identifies T1- and T2-related properties satisfying physical constraints by simultaneously fitting T1 and T2 experiments to simulations. This dual-task inverse solution workflow (DT-ISW) identifies the solution by minimizing the sum of the L2 norm of the fitting residuals of both T1 and T2 distributions into a single objective and jointly models the two highly correlated objectives with high efficiency using the vector-valued Gaussian process (GP) kernel for transfer learning. A multimodal search strategy is used to identify nonunique solution sets of the problem. The workflow is demonstrated on Bentheimer sandstone, identifying five intrinsic physical parameters. The performance of the joint DT-ISW (DT-ISW-J) is compared to a sequential DT-ISW (DT-ISW-S) approach as well as an independent single-task ISW (ST-ISW) of the T1 and T2 responses. Both dual-task versions converge more than two times faster than the single-task version. DT-ISW-J equally minimizes the L2 norm of T1 and T2 fitting residuals whereas DT-ISW-S only preferentially minimizes the objective assigned higher importance. A Pareto optimal solution (POS) is provided to allow operators to subjectively balance the preference of T1 and T2 data fits for the slightly conflicting objectives. The ability to extract five intrinsic physical parameters simultaneously provides new techniques for tracking wettability alteration and assessing the influence of clay amount and distribution on petrophysical property estimates.

0082 External Validation of an Enhanced Machine Learning Algorithm: Polysomnography-Based Narcolepsy-Like Feature Assessment and Clinician Notification in Routine Sleep Medicine Clinics

Abstract Introduction Polysomnography (PSG) contains quantitative information that, using machine learning (ML) algorithms, may aid the identification of type 1 narcolepsy. This study aimed to evaluate the utility of a previously developed quantitative tool (ML evaluation of PSG data) for detecting narcolepsy (types 1 and 2) in a “real-world” sleep clinic population. Methods Nocturnal PSG studies from a random sample of sleep clinic patients (narcolepsy, n=302; controls, n=21,535) were randomly split (1:1) into a training set and a validation set for algorithm testing. A separate, external PSG dataset was used for additional testing of the final model. Sleep stage probability graphs (hypnodensities) were estimated from PSGs on 15-second epochs using a previously developed convolutional neural network. Feature engineering was applied to hypnodensities to create a feature vector that was used to train a Gaussian process (GP) model to identify patients with a high probability of having narcolepsy. Features could be scaled to the 85th percentile, zero-mean and unit-variance, or unscaled. A recursive feature-elimination scheme was compared with training the GP kernel’s length scale for determining the subset of features that best discriminate narcolepsy and controls. A synthetic minority oversampling technique was applied in combination with random undersampling to balance the distribution of cases and controls in the training set. Several kernels and relevant hyperparameters were evaluated. Model performance (specificity and sensitivity) was examined using receiver operating characteristics with the goal of achieving an area under the curve (AUC) ≥0.80. Results The final GP model used a Matérn 5/2 covariance kernel with the length scale hyperparameter trained to determine the feature subset selection. Input features were normalized to zero-mean and unit-variance. The model had AUC=0.9960 and AUC=0.8014 for classifying narcolepsy in the training and validation sets, respectively. Sensitivity ranged from 73% to 65% when specificity was between 75% and 80%. Final performance evaluation of the model for classifying narcolepsy in an external PSG dataset is ongoing; results will be presented at the congress. Conclusion An ML-based algorithm can offer an objective, sensitive, and specific tool for alerting sleep clinicians about patients at risk for narcolepsy, using nocturnal PSG in general sleep medicine clinics. Support (If Any) Jazz Pharmaceuticals.

Structural optimization of metamaterials based on periodic surface modeling

Metamaterials are engineered microstructures or composites which exhibit novel properties that are not found in nature. Structural optimization methods for shape and topology optimization have been developed in designing metamaterials. The computational cost of the traditional topology optimization methods is high due to the high-dimensional design space. Shape optimization and topology optimization are also performed separately. To improve the search efficiency, a new structural optimization method based on periodic surface modeling is proposed to simultaneously optimize the shape and topology of metamaterials. A wide variety of topology can be represented with a small number of parameters in the periodic surface models. Therefore, the search space is significantly reduced. A mixed-integer Bayesian optimization method is also developed with a new Gaussian process kernel, which incorporates both discrete and continuous design parameters in the periodic surface models. Both the mixed-integer Bayesian optimization method and conventional population-based genetic algorithms are used to solve the structural optimization problem. The design of mechanical metamaterials with high strength–weight ratio and negative Poisson’s ratio is demonstrated.

On Neural Network Kernels and the Storage Capacity Problem

In this short note, we reify the connection between work on the storage capacity problem in wide two-layer treelike neural networks and the rapidly growing body of literature on kernel limits of wide neural networks. Concretely, we observe that the "effective order parameter" studied in the statistical mechanics literature is exactly equivalent to the infinite-width neural network gaussian process kernel. This correspondence connects the expressivity and trainability of wide two-layer neural networks.

Bayesian Calibration of Traffic Flow Fundamental Diagrams Using Gaussian Processes

Modeling the relationship between vehicle speed and density on the road is a fundamental problem in traffic flow theory. Recent research found that using the least-squares (LS) method to calibrate single-regime speed-density models is biased because of the uneven distribution of samples. This paper explains the issue of the LS method from a statistical perspective: the biased calibration is caused by the correlations/dependencies in regression residuals. Based on this explanation, we propose a new calibration method for single-regime speed-density models by modeling the covariance of residuals via a zero-mean Gaussian Process (GP). Our approach can be viewed as a generalized least-squares (GLS) method with a specific covariance structure (i.e., kernel function) and is a generalization of the existing LS and the weighted least-squares (WLS) methods. Next, we use a sparse approximation to address the scalability issue of GPs and apply a Markov chain Monte Carlo (MCMC) sampling scheme to obtain the posterior distributions of the parameters for speed-density models and the hyperparameters (i.e., length scale and variance) of the GP kernel. Finally, we calibrate six well-known single-regime speed-density models with the proposed method. Results show that the proposed GP-based methods (1) significantly reduce the biases in the LS calibration, (2) achieve a similar effect as the WLS method, (3) can be used as a non-parametric speed-density model, and (4) provide a Bayesian solution to estimate posterior distributions of parameters and speed-density functions.

Conditional Deep Gaussian Processes: Empirical Bayes Hyperdata Learning.

It is desirable to combine the expressive power of deep learning with Gaussian Process (GP) in one expressive Bayesian learning model. Deep kernel learning showed success as a deep network used for feature extraction. Then, a GP was used as the function model. Recently, it was suggested that, albeit training with marginal likelihood, the deterministic nature of a feature extractor might lead to overfitting, and replacement with a Bayesian network seemed to cure it. Here, we propose the conditional deep Gaussian process (DGP) in which the intermediate GPs in hierarchical composition are supported by the hyperdata and the exposed GP remains zero mean. Motivated by the inducing points in sparse GP, the hyperdata also play the role of function supports, but are hyperparameters rather than random variables. It follows our previous moment matching approach to approximate the marginal prior for conditional DGP with a GP carrying an effective kernel. Thus, as in empirical Bayes, the hyperdata are learned by optimizing the approximate marginal likelihood which implicitly depends on the hyperdata via the kernel. We show the equivalence with the deep kernel learning in the limit of dense hyperdata in latent space. However, the conditional DGP and the corresponding approximate inference enjoy the benefit of being more Bayesian than deep kernel learning. Preliminary extrapolation results demonstrate expressive power from the depth of hierarchy by exploiting the exact covariance and hyperdata learning, in comparison with GP kernel composition, DGP variational inference and deep kernel learning. We also address the non-Gaussian aspect of our model as well as way of upgrading to a full Bayes inference.

Toward the accurate estimation of elliptical side orifice discharge coefficient applying two rigorous kernel-based data-intelligence paradigms

In the present study, two kernel-based data-intelligence paradigms, namely, Gaussian Process Regression (GPR) and Kernel Extreme Learning Machine (KELM) along with Generalized Regression Neural Network (GRNN) and Response Surface Methodology (RSM), as the validated schemes, employed to precisely estimate the elliptical side orifice discharge coefficient in rectangular channels. A total of 588 laboratory data in various geometric and hydraulic conditions were used to develop the models. The discharge coefficient was considered as a function of five dimensionless hydraulically and geometrical variables. The results showed that the machine learning models used in this study had shown good performance compared to the regression-based relationships. Comparison between machine learning models showed that GPR (RMSE = 0.0081, R = 0.958, MAPE = 1.3242) and KELM (RMSE = 0.0082, R = 0.9564, MAPE = 1.3499) models provide higher accuracy. Base on the RSM model, a new practical equation was developed to predict the discharge coefficient. Also, the sensitivity analysis of the input parameters showed that the main channel width to orifice height ratio (B/b) has the most significant effect on determining the discharge coefficient. The leveraged approach was applied to identify outlier data and applicability domain.

Evolving Gaussian process kernels from elementary mathematical expressions for time series extrapolation

Choosing the best kernel is crucial in many Machine Learning applications. Gaussian Processes are a state-of-the-art technique for regression and classification that heavily relies on a kernel function. However, in the Gaussian Processes literature, kernels have usually been either ad hoc designed, selected from a predefined set, or searched for in a space of compositions of kernels which have been defined a priori. In this paper, we propose a Genetic Programming algorithm that represents a kernel function as a tree of elementary mathematical expressions. By means of this representation, a wider set of kernels can be modeled, where potentially better solutions can be found, although new challenges also arise. The proposed algorithm is able to overcome these difficulties and find kernels that accurately model the characteristics of the data. This method has been tested in several real-world time series extrapolation problems, improving the state-of-the-art results while reducing the complexity of the kernels.