Hierarchical Gaussian Process Research Articles

An application of hierarchical Gaussian processes to the detection of anomalies in star light curves

This study is concerned with astronomical time-series called light-curves that represent the brightness of celestial objects over a period of time. We consider the task of finding anomalous light-curves of periodic variable stars. We employ a Hierarchical Gaussian Process to create a general and stable model of time-series for anomaly detection, and apply this approach to the light-curve problem. Hierarchical Gaussian Processes require only a few additional parameters compared to conventional Gaussian Processes and incur negligible additional computational complexity. Moreover, since the additional parameters are objectively optimised in a principled probabilistic framework one does not need to resort to grid searches for parameter selection. Experimentally, we demonstrate that our approach outperforms several baselines on both synthetic and light-curve data. Of particular interest is that the proposed method generalises very well from small subsets of the data, achieving near perfect precision of outlier detection even with as few as seven instances.

Patient-Specific Physiological Monitoring and Prediction Using Structured Gaussian Processes

The management of patient well-being can be performed by monitoring continuous time-series vital-sign data via low-cost wearable devices. Automated algorithms may then be used with the resulting data to provide early warning of deterioration of the health of an individual. Such algorithms are typically trained for a large population without considering the time-variability and inter-subject variability of the data being collected. In the case where limited numbers of subjects are available, it is difficult to create a generalized population model from a small sample size. Furthermore, some “normal” patients may exhibit different physiological patterns when compared to other “normal” patients, forming multiple “normal” clusters/subgroups. This also makes inferring a population model difficult. It is, therefore, preferable to develop patient/subgroup-specific time-series models to overcome these challenges. We propose using Bayesian hierarchical Gaussian processes to infer the hidden latent structure of the vital sign's trajectory for each individual patient or group of patients who share similar patterns. We further demonstrate the feasibility of such a model in novelty detection, using the symmetric Kullback-Leibler divergence. This allows us to identify any patterns that correspond to “normal” or “abnormal” physiology, and further classifying “abnormal” patterns from a model of “normal” latent trajectories. We tested our approach using two real datasets for different monitoring scenarios. Our model was compared to the performance of the state-of-the-art unsupervised clustering algorithms, demonstrating at least 10% improvement in accuracy. We further benchmarked against two one-class classifiers and showed at least 5% accuracy improvement when using the proposed metrics in identifying abnormal physiological episodes.

Hierarchical Facial Age Estimation Using Gaussian Process Regression

Automatic age estimation from facial images has attracted increasing attention due to its promising potential in real-life computer vision applications. However, due to uncontrollable environments, insufficient and incomplete training data, strong person-specific and large within- age span variations, age estimation has become a challenging problem. Among published age estimation, hierarchical age estimation methods have achieved comparable performance improvement than single level approaches. Most of the published hierarchical approaches have mainly used support vector machines to classify age groups followed by support vector regression for withina- age group age estimation. In this paper, we present a novel hierarchical Gaussian process framework for automatic age estimation. It consists of multi-class Gaussian process classifier to classify the input images into different age groups followed by a warped Gaussian process regression to model group specific aging patterns. In this paper, we separately tune the hyper-parameters for each age group at the regression stage. Compared with existing single level Gaussian process approaches for age estimation, our approach is computationally efficient at both the levels of hierarchy. Partitioning data into different age groups and learning group-wise hyper-parameters is computationally more efficient than learning complete training data. Misclassifications at the group boundaries are compensated at the regression stage by overlapping the neighboring age ranges. Finally, through extensive experiments on two popular aging datasets, the FG-NET and the Morph-II, we demonstrate the effectiveness of our algorithm in improving age estimation performance.

Spatio-Temporal Structured Sparse Regression With Hierarchical Gaussian Process Priors

This paper introduces a new sparse spatio-temporal structured Gaussian process regression framework for online and offline Bayesian inference. This is the first framework that gives a time-evolving representation of the interdependencies between the components of the sparse signal of interest. A hierarchical Gaussian process describes such structure and the interdependencies are represented via the covariance matrices of the prior distributions. The inference is based on the expectation propagation method and the theoretical derivation of the posterior distribution is provided in the paper. The inference framework is thoroughly evaluated over synthetic, real video and electroencephalography (EEG) data where the spatio-temporal evolving patterns need to be reconstructed with high accuracy. It is shown that it achieves 15% improvement of the F-measure compared with the alternating direction method of multipliers, spatio-temporal sparse Bayesian learning method and one-level Gaussian process model. Additionally, the required memory for the proposed algorithm is less than in the one-level Gaussian process model. This structured sparse regression framework is of broad applicability to source localisation and object detection problems with sparse signals.

Best Linear Unbiased Prediction for Multifidelity Computer Experiments

Recently it becomes a growing trend to study complex systems which contain multiple computer codes with different levels of accuracy, and a number of hierarchical Gaussian process models are proposed to handle such multiple-fidelity codes. This paper derives the best linear unbiased prediction for three popular classes of multiple-level Gaussian process models. The predictors all have explicit expressions at each untried point. Empirical best linear unbiased predictors are also provided by plug-in methods with generalized maximum likelihood estimators of unknown parameters.

Predicting local ice loads on ship bow as a function of ice and operational conditions in the Southern Sea

ABSTRACTTransportation in ice-prone waters is a timely topic because of the pursuit of natural resources and sea routes. S.A. Agulhas II voyaged in Antarctic waters during the austral summer of 2013–14. This paper studies how the local ice-induced loads on the ship bow are affected by different ice and operational conditions. A hierarchical Gaussian process model was used to study how the load distribution parameters change in different conditions. The operational conditions are described by ship speed, whereas the ice conditions varied from thin, first-year ice, to thick, multiyear ice. Because of this variation, the ice conditions are not represented by ice thickness alone but also by latitude, temperature and voyage time which represent the unmeasured ice strength. The predicted loads from the model match well with the measured loads. According to the model, loads increase in higher ship speed and ice thickness combination, further south, later in the summer and in higher temperatures.

Joint hierarchical Gaussian process model with application to personalized prediction in medical monitoring.

A two-level Gaussian process (GP) joint model is proposed to improve personalized prediction of medical monitoring data. The proposed model is applied to jointly analyze multiple longitudinal biomedical outcomes, including continuous measurements and binary outcomes, to achieve better prediction in disease progression. At the population level of the hierarchy, two independent GPs are used to capture the nonlinear trends in both the continuous biomedical marker and the binary outcome, respectively; at the individual level, a third GP, which is shared by the longitudinal measurement model and the longitudinal binary model, induces the correlation between these two model components and strengthens information borrowing across individuals. The proposed model is particularly advantageous in personalized prediction. It is applied to the motivating clinical data on cystic fibrosis disease progression, for which lung function measurements and onset of acute respiratory events are monitored jointly throughout each patient's clinical course. The results from both the simulation studies and the cystic fibrosis data application suggest that the inclusion of the shared individual-level GPs under the joint model framework leads to important improvements in personalized disease progression prediction.

Hierarchical Gaussian Processes model for multi-task learning

Multi-task learning (MTL) has been proved to improve performance of individual tasks by learning multiple related tasks together. Recently Nonparametric Bayesian Gaussian Process (GP) models have also been adapted to MTL and exhibit enough flexibility due to its non-parametric nature, thus can exempt from the assumption about the probability distributions of variables. To date, there have had two approaches proposed to implement GP-based MTL, i.e., cross-covariance-based and joint feature learning methods. Although successfully applied in scenarios such as face verification and collaborative filtering, these methods have their own drawbacks, for example, the cross-covariance-based method suffers from poor scalability because of the large covariance matrix involved; while the joint feature learning method can just implicitly incorporate relation between tasks, thus leading to a failure in explicitly exploiting the prior knowledge like correlation between tasks, which is crucial for further promoting MTLs. To address both issues, in this paper, we establish a two layer unified framework called Hierarchical Gaussian Process Multi-task Learning (HGPMT) method to jointly learn the latent shared features among tasks and a multi-task model. Furthermore, since the HGPMT does not need to involve the cross-covariance, its computational complexity is much lower. Finally, experimental results on both toy multi-task regression dataset and real datasets demonstrate its superiority in performance of multi-task learning to recently proposed approaches.

Gaussian process framework for temporal dependence and discrepancy functions in Ricker-type population growth models

Density dependent population growth functions are of central importance to population dynamics modelling because they describe the theoretical rate of recruitment of new individuals to a natural population. Traditionally, these functions are described with a fixed functional form with temporally constant parameters and without species interactions. The Ricker stock-recruitment model is one such function that is commonly used in fisheries stock assessment. In recent years, there has been increasing interest in semiparametric and temporally varying population growth models. The former are related to the general statistical approach of using semiparametric discrepancy functions, such as Gaussian processes (GP), to model deviations around the expected parametric function. In the latter, the reproductive rate, which is a key parameter describing the population growth rate, is assumed to vary in time. In this work, we introduce how these existing Ricker population growth models can be formulated under the same statistical approach of hierarchical GP models. We also show how the time invariant semiparametric approach can be extended and combined with the time varying reproductive rate using a GP model. Then we extend these models to the multispecies setting by incorporating cross-covariances among species with a continuous time covariance structure using the linear model of coregionalization. As a case study, we examine the productivity of three Pacific salmon populations. We compare the alternative Ricker population growth functions using model posterior probabilities and leave-one-out cross validation predictive densities. Our results show substantial temporal variation in maximum reproductive rates and reveal temporal dependence among the species, which have direct management implications. However, our results do not support inclusion of semiparametric discrepancy function and they suggest that the semiparametric discrepancy functions may lead to challenges in parameter identifiability more generally.

Functional Autoregression for Sparsely Sampled Data

We develop a hierarchical Gaussian process model for forecasting and inference of functional time series data. Unlike existing methods, our approach is especially suited for sparsely or irregularly sampled curves and for curves sampled with nonnegligible measurement error. The latent process is dynamically modeled as a functional autoregression (FAR) with Gaussian process innovations. We propose a fully nonparametric dynamic functional factor model for the dynamic innovation process, with broader applicability and improved computational efficiency over standard Gaussian process models. We prove finite-sample forecasting and interpolation optimality properties of the proposed model, which remain valid with the Gaussian assumption relaxed. An efficient Gibbs sampling algorithm is developed for estimation, inference, and forecasting, with extensions for FAR(p) models with model averaging over the lag p. Extensive simulations demonstrate substantial improvements in forecasting performance and recovery of the autoregressive surface over competing methods, especially under sparse designs. We apply the proposed methods to forecast nominal and real yield curves using daily U.S. data. Real yields are observed more sparsely than nominal yields, yet the proposed methods are highly competitive in both settings. Supplementary materials, including R code and the yield curve data, are available online.

Prediction of tool-wear in turning of medical grade cobalt chromium molybdenum alloy (ASTM F75) using non-parametric Bayesian models

We present a novel approach to estimating the effect of control parameters on tool wear rates and related changes in the three force components in turning of medical grade Co–Cr–Mo (ASTM F75) alloy. Co–Cr–Mo is known to be a difficult to cut material which, due to a combination of mechanical and physical properties, is used for the critical structural components of implantable medical prosthetics. We run a designed experiment which enables us to estimate tool wear from feed rate and cutting speed, and constrain them using a Bayesian hierarchical Gaussian process model which enables prediction of tool wear rates for untried experimental settings. The predicted tool wear rates are non-linear and, using our models, we can identify experimental settings which optimise the life of the tool. This approach has potential in the future for realtime application of data analytics to machining processes.

Combining functional data with hierarchical Gaussian process models

Gaussian process models have been used in applications ranging from machine learning to fisheries management. In the Bayesian framework, the Gaussian process is used as a prior for unknown functions, allowing the data to drive the relationship between inputs and outputs. In our research, we consider a scenario in which response and input data are available from several similar, but not necessarily identical, sources. When little information is known about one or more of the populations it may be advantageous to model all populations together. We present a hierarchical Gaussian process model with a structure that allows distinct features for each source as well as shared underlying characteristics. Key features and properties of the model are discussed and demonstrated in a number of simulation examples. The model is then applied to a data set consisting of three populations of Rotifer Brachionus calyciflorus Pallas. Specifically, we model the log growth rate of the populations using a combination of lagged population sizes. The various lag combinations are formally compared to obtain the best model inputs. We then formally compare the leading hierarchical Gaussian process model with the inferential results obtained under the independent Gaussian process model.

Predicting ice-induced load amplitudes on ship bow conditional on ice thickness and ship speed in the Baltic Sea

Transportation in ice prone waters is a timely topic due to the pursuit for arctic natural resources and sea routes. One important safety aspect in designing ships that enter ice prone waters is to determine the ice-induced loads on ships. However, ice is a particularly inconsistent material; therefore it is difficult to predict the occurring loads when the ship hull breaks the ice. We propose a novel probabilistic, Bayesian, method for modeling and predicting ice load distributions in different ice and operational conditions. We assume the ice loads to be generated from a random process whose parameters change as a function of ice thickness and ship speed. We test four alternative hierarchical Gaussian Process models. The best model shows good performance in predictive validation tests. According to the results the probability of high ice loads increases with increasing ice thickness and increasing speed. The model can be used to predict continuously ice loads in different ice thickness and speed conditions and, with further development, has potential to be utilized in determining the safe way to operate ships in different conditions.

On nearest-neighbor Gaussian process models for massive spatial data.

Gaussian Process (GP) models provide a very flexible nonparametric approach to modeling location-and-time indexed datasets. However, the storage and computational requirements for GP models are infeasible for large spatial datasets. Nearest Neighbor Gaussian Processes (Datta A, Banerjee S, Finley AO, Gelfand AE. Hierarchical nearest-neighbor gaussian process models for large geostatistical datasets. J Am Stat Assoc 2016., JASA) provide a scalable alternative by using local information from few nearest neighbors. Scalability is achieved by using the neighbor sets in a conditional specification of the model. We show how this is equivalent to sparse modeling of Cholesky factors of large covariance matrices. We also discuss a general approach to construct scalable Gaussian Processes using sparse local kriging. We present a multivariate data analysis which demonstrates how the nearest neighbor approach yields inference indistinguishable from the full rank GP despite being several times faster. Finally, we also propose a variant of the NNGP model for automating the selection of the neighbor set size.

Assessment of DPOAE test-retest difference curves via hierarchical Gaussian processes.

Distortion product otoacoustic emissions (DPOAE) testing is a promising alternative to behavioral hearing tests and auditory brainstem response testing of pediatric cancer patients. The central goal of this study is to assess whether significant changes in the DPOAE frequency/emissions curve (DP-gram) occur in pediatric patients in a test-retest scenario. This is accomplished through the construction of normal reference charts, or credible regions, that DP-gram differences lie in, as well as contour probabilities that measure how abnormal (or in a certain sense rare) a test-retest difference is. A challenge is that the data were collected over varying frequencies, at different time points from baseline, and on possibly one or both ears. A hierarchical structural equation Gaussian process model is proposed to handle the different sources of correlation in the emissions measurements, wherein both subject-specific random effects and variance components governing the smoothness and variability of each child's Gaussian process are coupled together.

Dynamic Gesture Recognition based on Image Sequence

This paper proposed an algorithm for 3D hands tracking on the learned hierarchical latent variable space, which employs a Hierarchical Gaussian Process Latent Variable Model(HGPLVM) to learn the hierarchical latent space of hands motion and the nonlinear mapping from the hierarchical latent space to the pose space simultaneously. Nonlinear mappings from the hierarchical latent space to the space of hand images are constructed using radial basis function interpolation method. With these mappings, particles can be projected into hand images and measured in the image space directly. Particle filters with fewer particles are used to track the hand on the learned hierarchical low-dimensional space. Then the Hierarchical Conditional Random Field, which can capture extrinsic class dynamics and learn the relationship between motions of hand parts and different hand gestures simultaneously, is presented to model the continuous hand gestures. Experimental results show that our proposed method can track articulated hand robustly and approving recognition performance has also been achieved on the user-defined hand gesture dataset.

Modeling supernovae light curves: An application of hierarchical smoothing splines

AbstractWith large data collection projects such as the Dark Energy Survey underway, data from distant supernovae (SNe) are becoming increasingly available. As the quantity of information increases, the ability to quickly and accurately classify SNe has become essential. An area of great interest is the development of a strictly photometric classification mechanism. The first step in the advancement of modern photometric classification is the estimation of individual Supernova (SN) light curves. We propose the use of hierarchical Gaussian processes to model light curves. Individual SN light curves are assigned a Gaussian process prior centered at a type‐specific mean curve which is also assigned a Gaussian process prior. Properties inherent in this Bayesian non‐parametric form of modeling yield flexible yet smooth curves estimates with a unique quantification of the error surrounding these curve estimates. Specifying the hierarchical structure relates individual SN light curves in such a way that borrowing strength across curves is possible. This allows for the estimation of SN light curves in entirety even when data are sparse. Additionally, it also yields a meaningful representation of SN class differences in the form of mean curves. The differences inherent in these mean curves may eventually allow for classification of SNe.

Gaussian Process Regression-Based Video Anomaly Detection and Localization With Hierarchical Feature Representation.

This paper presents a hierarchical framework for detecting local and global anomalies via hierarchical feature representation and Gaussian process regression (GPR) which is fully non-parametric and robust to the noisy training data, and supports sparse features. While most research on anomaly detection has focused more on detecting local anomalies, we are more interested in global anomalies that involve multiple normal events interacting in an unusual manner, such as car accidents. To simultaneously detect local and global anomalies, we cast the extraction of normal interactions from the training videos as a problem of finding the frequent geometric relations of the nearby sparse spatio-temporal interest points (STIPs). A codebook of interaction templates is then constructed and modeled using the GPR, based on which a novel inference method for computing the likelihood of an observed interaction is also developed. Thereafter, these local likelihood scores are integrated into globally consistent anomaly masks, from which anomalies can be succinctly identified. To the best of our knowledge, it is the first time GPR is employed to model the relationship of the nearby STIPs for anomaly detection. Simulations based on four widespread datasets show that the new method outperforms the main state-of-the-art methods with lower computational burden.

Joint inference of misaligned irregular time series with application to Greenland ice core data

Abstract. Ice cores provide insight into the past climate over many millennia. Due to ice compaction, the raw data for any single core are irregular in time. Multiple cores have different irregularities; and when considered together, they are misaligned in time. After processing, such data are made available to researchers as regular time series: a data product. Typically, these cores are independently processed. This paper considers a fast Bayesian method for the joint processing of multiple irregular series. This is shown to be more efficient than the independent alternative. Furthermore, our explicit framework permits a reliable modelling of the impact of the multiple sources of uncertainty. The methodology is illustrated with the analysis of a pair of ice cores. Our data products, in the form of posterior marginals or joint distributions on an arbitrary temporal grid, are finite Gaussian mixtures. We can also produce process histories to study non-linear functionals of interest. More generally, the concept of joint analysis via hierarchical Gaussian process models can be widely extended, as the models used can be viewed within the larger context of continuous space–time processes.

Hierarchical Bayesian Spatio Temporal Model Comparison on the Earth Trapped Particle Forecast

We compared two hierarchical Bayesian spatio temporal (HBST) results, Gaussian process (GP) and autoregressive (AR) models, on the Earth trapped particle forecast. Two models were employed on the South Atlantic Anomaly (SAA) region. Electron of >30 keV (mep0e1) from National Oceanic and Atmospheric Administration (NOAA) 15-18 satellites data was chosen as the particle modeled. We used two weeks data to perform the model fitting on a 5°x5° grid of longitude and latitude, and 31 August 2007 was set as the date of forecast. Three statistical validations were performed on the data, i.e. the root mean square error (RMSE), mean absolute percentage error (MAPE) and bias (BIAS). The statistical analysis showed that GP model performed better than AR with the average of RMSE = 0.38 and 0.63, MAPE = 11.98 and 17.30, and BIAS = 0.32 and 0.24, for GP and AR, respectively. Visual validation on both models with the NOAA map's also confirmed the superior of the GP than the AR. The variance of log flux minimum = 0.09 and 1.09, log flux maximum = 1.15 and 1.35, and in successively represents GP and AR.