Multi-task Gaussian Processes Research Articles

Correcting missingness in passively-generated mobile data with Multi-Task Gaussian Processes

The prevalence of mobile devices and the ubiquity of network connectivity have generated a massive amount of temporally- and spatially-stamped data. A key characteristic of this mobile data is the prevalence of sparsity—for every recorded user, there is a significant amount of missing data. Sparsity leads to bias in the inferred mobility patterns and thus, correcting bias by imputing the missing data potentially creates opportunities for directly using the corrected mobile trajectory data for large-scale simulations in various applications. We propose a multi-task Gaussian process regression model to correct missingness in mobile data. Gaussian processes (GPs) allow for flexible modeling of diverse (i.e., non-linear, locally periodic) data patterns and quantify prediction uncertainty in an interpretable manner. We develop a methodological framework for applying GPs to mobile data. In doing so, we consider the correlations between users’ coordinates (latitudes and longitudes) through multi-task learning and adjust for individual-level differences in data characteristics through parameter initialization and optimization. We introduce and demonstrate the use of smooth (i.e., rational quadratic) and periodic kernels in modeling human mobility data. Relatedly, we analyze our model’s parameters and imputation accuracy with respect to different types of trips (e.g. slower vs faster trips). We also demonstrate our model’s performance in two experiments with real app-based data, in which it outperforms alternative imputation methods. Our implementation is open-source at https://github.com/ekinugurel/GPSImpute.

Heterogeneous multi-task Gaussian Cox processes

Heterogeneous multi-task Gaussian Cox processes

Multi-objective Bayesian alloy design using multi-task Gaussian processes

In design applications, correlations among material properties (such as the tendency for stronger materials to be less ductile) are often neglected. This approach is echoed in multi-objective optimization techniques which treat each performance characteristic as an independent objective, aiming to optimize scalar functions and find optimal Pareto fronts. However, this overlooks the statistical relationships between performance characteristics inherent in a material system. To address this, we propose the use of Bayesian optimization, a highly efficient black-box optimization algorithm known for constructing Gaussian processes (GPs) – uncorrelated surrogates - to model objective functions. Rather than evaluating multiple GPs for each objective function separately, we argue for a shift towards jointly modeling these objective functions, considering their statistical correlations. This integrated approach utilizes naturally occurring relationships among material properties, providing additional information to enhance the performance of the design framework. This requires the replacement of multiple independent GPs with a single multi-task GP, employing a correlation matrix to construct a multi-task kernel function, wherein each task corresponds to a single objective function. We anticipate this refined methodology will better leverage material correlations, improving design optimization results.

Estimating the effects of a California gun control program with multitask Gaussian processes

Estimating the effects of a California gun control program with multitask Gaussian processes

Advanced stationary and nonstationary kernel designs for domain-aware Gaussian processes

Gaussian process regression is a widely-applied method for function approximation and uncertainty quantification. The technique has gained popularity recently in the machine learning community due to its robustness and interpretability. The mathematical methods we discuss in this paper are an extension of the Gaussian-process framework. We are proposing advanced kernel designs that only allow for functions with certain desirable characteristics to be elements of the reproducing kernel Hilbert space (RKHS) that underlies all kernel methods and serves as the sample space for Gaussian process regression. These desirable characteristics reflect the underlying physics; two obvious examples are symmetry and periodicity constraints. In addition, non-stationary kernel designs can be defined in the same framework to yield flexible multi-task Gaussian processes. We will show the impact of advanced kernel designs on Gaussian processes using several synthetic and two scientific data sets. The results show that including domain knowledge, communicated through advanced kernel designs, has a significant impact on the accuracy and relevance of the function approximation.

An improved virtual metrology method in chemical vapor deposition systems via multitask gaussian processes and adaptive active learning

An improved virtual metrology method in chemical vapor deposition systems via multitask gaussian processes and adaptive active learning

A conditional one-output likelihood formulation for multitask Gaussian processes

Multitask Gaussian processes (MTGP) are the Gaussian process (GP) framework’s solution for multioutput regression problems in which the T elements of the regressors cannot be considered conditionally independent given the observations. Standard MTGP models assume that there exist both a multitask covariance matrix as a function of an intertask matrix, and a noise covariance matrix. These matrices need to be approximated by a low rank simplification of order P in order to reduce the number of parameters to be learnt from T2 to TP. Here we introduce a novel approach that simplifies the multitask learning process by reducing it to a set of conditioned univariate GPs without the need for any low rank approximations, therefore completely eliminating the need to select an adequate value for hyperparameter P. At the same time, by extending this approach with both a hierarchical and an approximate model, the proposed extensions are capable of recovering the multitask covariance and noise matrices after learning only 2T parameters, avoiding the validation of any model hyperparameter and reducing the overall complexity of the model as well as the risk of overfitting. Experimental results over synthetic and real problems confirm the advantages of this inference approach in its ability to accurately recover the original noise and signal matrices, as well as the achieved performance improvement in comparison to other state of art MTGP approaches. We have also integrated the model with standard GP toolboxes, showing that it is computationally competitive with state of the art options.

MAGMA: inference and prediction using multi-task Gaussian processes with common mean

A novel multi-task Gaussian process (GP) framework is proposed, by using a common mean process for sharing information across tasks. In particular, we investigate the problem of time series forecasting, with the objective to improve multiple-step-ahead predictions. The common mean process is defined as a GP for which the hyper-posterior distribution is tractable. Therefore an EM algorithm is derived for handling both hyper-parameters optimisation and hyper-posterior computation. Unlike previous approaches in the literature, the model fully accounts for uncertainty and can handle irregular grids of observations while maintaining explicit formulations, by modelling the mean process in a unified GP framework. Predictive analytical equations are provided, integrating information shared across tasks through a relevant prior mean. This approach greatly improves the predictive performances, even far from observations, and may reduce significantly the computational complexity compared to traditional multi-task GP models. Our overall algorithm is called Magma (standing for Multi tAsk GPs with common MeAn). The quality of the mean process estimation, predictive performances, and comparisons to alternatives are assessed in various simulated scenarios and on real datasets.

Scalable multi-task Gaussian processes with neural embedding of coregionalization

Multi-task regression attempts to exploit the task similarity in order to achieve knowledge transfer across related tasks for improving the quality of prediction and alleviating the demand of big data. The application of Gaussian process (GP) in this scenario yields the non-parametric yet informative Bayesian multi-task regression paradigm. Multi-task GP (MTGP) provides not only the prediction mean but also the associated prediction variance to quantify uncertainty, thus gaining popularity in various scenarios. The linear model of coregionalization (LMC) is a well-known MTGP paradigm which exploits the dependency of tasks through linear combination of several independent and diverse GPs. The LMC however suffers from high model complexity and limited model capability when handling complicated multi-task cases. To this end, we develop the neural embedding of coregionalization that transforms the latent GPs into a high-dimensional latent space to induce rich yet diverse behaviors. Furthermore, we use advanced variational inference as well as sparse approximation to devise a tight and compact evidence lower bound (ELBO) for higher quality of scalable model inference. Extensive numerical experiments have been conducted to verify the higher prediction quality and better generalization of our model, named NSVLMC, on various real-world multi-task datasets and the cross-fluid modeling of unsteady fluidized bed.

A Conditional One-Output Likelihood Formulation for Multitask Gaussian Processes

Multitask Gaussian processes (MTGP) are the Gaussian process (GP) framework's solution for multioutput regression problems in which the $T$ elements of the regressors cannot be considered conditionally independent given the observations. Standard MTGP models assume that there exist both a multitask covariance matrix as a function of an intertask matrix, and a noise covariance matrix. These matrices need to be approximated by a low rank simplification of order $P$ in order to reduce the number of parameters to be learnt from $T^2$ to $TP$. Here we introduce a novel approach that simplifies the multitask learning by reducing it to a set of conditioned univariate GPs without the need for any low rank approximations, therefore completely eliminating the requirement to select an adequate value for hyperparameter $P$. At the same time, by extending this approach with both a hierarchical and an approximate model, the proposed extensions are capable of recovering the multitask covariance and noise matrices after learning only $2T$ parameters, avoiding the validation of any model hyperparameter and reducing the overall complexity of the model as well as the risk of overfitting. Experimental results over synthetic and real problems confirm the advantages of this inference approach in its ability to accurately recover the original noise and signal matrices, as well as the achieved performance improvement in comparison to other state of art MTGP approaches. We have also integrated the model with standard GP toolboxes, showing that it is computationally competitive with state of the art options.

Generalized Convolution Spectral Mixture for Multitask Gaussian Processes.

Multitask Gaussian processes (MTGPs) are a powerful approach for modeling dependencies between multiple related tasks or functions for joint regression. Current kernels for MTGPs cannot fully model nonlinear task correlations and other types of dependencies. In this article, we address this limitation. We focus on spectral mixture (SM) kernels and propose an enhancement of this type of kernels, called multitask generalized convolution SM (MT-GCSM) kernel. The MT-GCSM kernel can model nonlinear task correlations and dependence between components, including time and phase delay dependence. Each task in MT-GCSM has its GCSM kernel with its number of convolution structures, and dependencies between all components from different tasks are considered. Another constraint of current kernels for MTGPs is that components from different tasks are aligned. Here, we lift this constraint by using inner and outer full cross convolution between a base component and the reversed complex conjugate of another base component. Extensive experiments on two synthetic and three real-life data sets illustrate the difference between MT-GCSM and previous SM kernels as well as the practical effectiveness of MT-GCSM.

Predicting Adsorption Energies Using Multifidelity Data.

In this paper, we show that the binding energy of small adsorbates on transition-metal surfaces can be modeled to a high level of fidelity with data from multiple sources using multitask Gaussian processes (MT-GPs). This allows us to take advantage of the relatively abundant "low fidelity" data (such as from density functional theory computations) and small amounts of "high fidelity" computational (e.g., using the random phase approximation) or experimental data. We report two case studies here-one using purely computational datasets and the other using a combination of experimental and computational datasets-to explore the performance of MT-GPs. In both cases, the performance of MT-GPs is significantly better than single-task models built on a single data source. We posit that this method can be used to learn improved models from fused datasets, thereby maximizing model accuracy under tight computational and experimental budget.

Gaussian Processes for Vegetation Parameter Estimation from Hyperspectral Data with Limited Ground Truth

An important application of airborne- and satellite-based hyperspectral imaging is the mapping of the spatial distribution of vegetation biophysical and biochemical parameters in an environment. Statistical models, such as Gaussian processes, have been very successful for modeling vegetation parameters from captured spectra, however their performance is highly dependent on the amount of available ground truth. This is a problem because it is generally expensive to obtain ground truth information due to difficulties and costs associated with sample collection and analysis. In this paper, we present two Gaussian processes based approaches for improving the accuracy of vegetation parameter retrieval when ground truth is limited. The first is the adoption of covariance functions based on well-established metrics, such as, spectral angle and spectral correlation, which are known to be better measures of similarity for spectral data owing to their resilience to spectral variabilities. The second is the joint modeling of related vegetation parameters by multitask Gaussian processes so that the prediction accuracy of the vegetation parameter of interest can be improved with the aid of related vegetation parameters for which a larger set of ground truth is available. We experimentally demonstrate the efficacy of the proposed methods against existing approaches on three real-world hyperspectral datasets and one synthetic dataset.

Estimation of blood pressure from non-invasive data.

Blood pressure (BP) is one of the most important physiological parameter that can provide crucial information for health care. The widely used cuff based technology is not very convenient or comfortable as it occludes the blood flow in the arteries during the time of measurement. In past, Phonocardiogram (PCG), Electrocardiogram (ECG) and Photoplethysmogram (PPG) signals have been used to predict the BP values. In this paper, we propose to estimate the blood pressure from PPG using Multi Task Gaussian Processes (MTGPs) and compare with Artificial Neural networks (ANNs). Both MTGPs and ANNs are evaluated on the clinical data obtained from MIMIC Database. The performance of the proposed method is found to be comparable or better than the existing methods of computing BP from non-invasive data.

Multi-task learning for subthalamic nucleus identification in deep brain stimulation

Deep brain stimulation (DBS) of Subthalamic nucleus (STN) is the most successful treatment for advanced Parkinson’s disease. Localization of the STN through Microelectrode recordings (MER) is a key step during the surgery. However, it is a complex task even for a skilled neurosurgeon. Different researchers have developed methodologies for processing and classification of MER signals to locate the STN. Previous works employ the classical paradigm of supervised classification, assuming independence between patients. The aim of this paper is to introduce a patient-dependent learning scenario, where the predictive ability for STN identification at the level of a particular patient, can be used to improve the accuracy for STN identification in other patients. Our inspiration is the multi-task learning framework, that has been receiving increasing interest within the machine learning community in the last few years. To this end, we employ the multi-task Gaussian processes framework that exhibits state of the art performance in multi-task learning problems. In our context, we assume that each patient undergoing DBS is a different task, and we refer to the method as multi-patient learning. We show that the multi-patient learning framework improves the accuracy in the identification of STN in a range from 4.1 to 7.7%, compared to the usual patient-independent setup, for two different datasets. Given that MER are non stationary and noisy signals. Traditional approaches in machine learning fail to recognize accurately the STN during DBS. By contrast in our proposed method, we properly exploit correlations between patients with similar diseases, obtaining an additional information. This information allows to improve the accuracy not only for locating STN for DBS but also for other biomedical signal classification problems.

Automated Multi-class Classification of Remotely Sensed Hyperspectral Imagery Via Gaussian Processes with a Non-stationary Covariance Function

The ability to automatically classify hyperspectral imagery is of fundamental economic importance to the mining industry. A method of automated multi-class classification based on multi-task Gaussian processes (MTGPs) is proposed for classification of remotely sensed hyperspectral imagery. It is proved that because of the illumination invariance of the hyperspectral curves, the covariance function of the Gaussian process (GPs) has to be non-stationary. To enable multi-class classification of the hyperspectral imagery, a non-stationary multi-task observation angle-dependent covariance function is derived. In order to test MTGP, it was applied to data acquired in the laboratory and also in field. First, the MTGP was applied to hyperspectral imagery acquired under artificial light from samples of rock of known mineral composition. Data from a high-resolution field spectrometer are used to train the GPs. Second, the MTGP was applied to imagery of a vertical rock wall acquired under natural illumination. Spectra from hyperspectral imagery acquired in the laboratory are used to train the GPs. Results were compared with those obtained using the spectral angle mapper (SAM). In laboratory imagery, MTGP outperformed SAM across several metrics, including overall accuracy (MTGP: 0.96–0.98; SAM: 0.91–0.93) and the kappa coefficient of agreement (MTGP: 0.95–0.97; SAM: 0.88–0.91). MTGP applied to hyperspectral imagery of the rock wall gave broadly similar results to those from SAM; however, there were important differences. Some rock types were confused by SAM, but not by MTGP. Comparison of classified imagery with ground truth maps showed that MTGP outperformed SAM.

Multitask Gaussian processes for multivariate physiological time-series analysis.

Gaussian process (GP) models are a flexible means of performing nonparametric Bayesian regression. However, GP models in healthcare are often only used to model a single univariate output time series, denoted as single-task GPs (STGP). Due to an increasing prevalence of sensors in healthcare settings, there is an urgent need for robust multivariate time-series tools. Here, we propose a method using multitask GPs (MTGPs) which can model multiple correlated multivariate physiological time series simultaneously. The flexible MTGP framework can learn the correlation between multiple signals even though they might be sampled at different frequencies and have training sets available for different intervals. Furthermore, prior knowledge of any relationship between the time series such as delays and temporal behavior can be easily integrated. A novel normalization is proposed to allow interpretation of the various hyperparameters used in the MTGP. We investigate MTGPs for physiological monitoring with synthetic data sets and two real-world problems from the field of patient monitoring and radiotherapy. The results are compared with standard Gaussian processes and other existing methods in the respective biomedical application areas. In both cases, we show that our framework learned the correlation between physiological time series efficiently, outperforming the existing state of the art.