Multiple Output Gaussian Process Research Articles

Multi-step and multi-task learning to predict quality-related variables in wastewater treatment processes

In wastewater treatment processes, lack of hardware sensors together with unacceptable dynamics, strong nonlinearity and large time delay often leads to a large number of key variables that are difficult to measure online accurately and timely, then frustrate safe operations of the processes. To accurately and timely capture the short-term behavior changes and trend development of critical variables, a novel neural network based soft-sensing model is proposed to take full use of multi-task learning, direct multi-step prediction strategy and evolutionary algorithm to formulate a novel multi-task multi-step evolution (MTMSE) neural network. Firstly, single-output MTMSE (SO-MTMSE) neural network is used to realize the dynamic monitoring of a single variable. Moreover, by considering the spatiotemporal interaction among the data, the model is extended to multi-output MTMSE (MO-MTMSE) neural network to simultaneously realize multi-step prediction of multiple variables, thus providing a desired reference for optimizing the wastewater treatment processes. Finally, the proposed model is applied to the benchmark simulation model 2 (BSM2) and a full-scale wastewater treatment plant in Shenzhen. And the results show that the proposed dynamic soft sensor model outperforms the standard methods, such as autoregressive moving average model (ARMA) and multiple output gaussian process regression (MGPR).

Dependence in constrained Bayesian optimization

Constrained Bayesian optimization optimizes a black-box objective function subject to black-box constraints. For simplicity, most existing works assume that multiple constraints are independent. To ask, when and how does dependence between constraints help?, we remove this assumption and implement probability of feasibility with dependence (Dep-PoF) by applying multiple output Gaussian processes (MOGPs) as surrogate models and using expectation propagation to approximate the probabilities. We compare Dep-PoF and the independent version PoF. We propose two new acquisition functions incorporating Dep-PoF and test them on synthetic and practical benchmarks. Our results are largely negative: incorporating dependence between the constraints does not help much. Empirically, incorporating dependence between constraints may be useful if: (i) the solution is on the boundary of the feasible region(s) or (ii) the feasible set is very small. When these conditions are satisfied, the predictive covariance matrix from the MOGP may be poorly approximated by a diagonal matrix and the off-diagonal matrix elements may become important. Dep-PoF may apply to settings where (i) the constraints and their dependence are totally unknown and (ii) experiments are so expensive that any slightly better Bayesian optimization procedure is preferred. But, in most cases, Dep-PoF is indistinguishable from PoF.

A machine learning method for the prediction of ship motion trajectories in real operational conditions

This paper presents a big data analytics method for the proactive mitigation of grounding risk. The model encompasses the dynamics of ship motion trajectories while accounting for kinematic uncertainties in real operational conditions. The approach combines K-means and DB-SCAN (Density-Based Spatial Clustering of Applications with Noise) big data clustering methods with Principal Component Analysis (PCA) to group environmental factors. A Multiple-Output Gaussian Process Regression (MOGPR) method is consequently used to predict selected ship motion dynamics. Ship sway is defined as the deviation between a ship and her motion trajectory centreline. Surge accelerations are used to idealise the time-varying manoeuvring of ships in various routes. Operational conditions are simulated by Automatic Identification System (AIS), General Bathymetric Chart of the Oceans (GEBCO), and nowcast hydro-meteorological data records. A Dynamic Time Warping (DTW) method is adopted to identify ship centre-line trajectories along selected paths. The machine learning algorithm is applied for ship motion predictions of Ro-Pax ships operating between two ports in the Gulf of Finland. Ship motion dynamics are visualised along the ship's route using a Gaussian Progress Regression (GPR) flow method. Results indicate that the present methodology may assist with predicting the probabilistic distribution of ship dynamics (speed, sway distance, drift angle, and surge accelerations) and grounding risk along selected ship routes.

Synergy of Sentinel-1 and Sentinel-2 Time Series for Cloud-Free Vegetation Water Content Mapping with Multi-Output Gaussian Processes

Optical Earth Observation is often limited by weather conditions such as cloudiness. Radar sensors have the potential to overcome these limitations, however, due to the complex radar-surface interaction, the retrieving of crop biophysical variables using this technology remains an open challenge. Aiming to simultaneously benefit from the optical domain background and the all-weather imagery provided by radar systems, we propose a data fusion approach focused on the cross-correlation between radar and optical data streams. To do so, we analyzed several multiple-output Gaussian processes (MOGP) models and their ability to fuse efficiently Sentinel-1 (S1) Radar Vegetation Index (RVI) and Sentinel-2 (S2) vegetation water content (VWC) time series over a dry agri-environment in southern Argentina. MOGP models not only exploit the auto-correlations of S1 and S2 data streams independently but also the inter-channel cross-correlations. The S1 RVI and S2 VWC time series at the selected study sites being the inputs of the MOGP models proved to be closely correlated. Regarding the set of assessed models, the Convolutional Gaussian model (CONV) delivered noteworthy accurate data fusion results over winter wheat croplands belonging to the 2020 and 2021 campaigns (NRMSEwheat2020 = 16.1%; NRMSEwheat2021 = 10.1%). Posteriorly, we removed S2 observations from the S1 & S2 dataset corresponding to the complete phenological cycles of winter wheat from September to the end of December to simulate the presence of clouds in the scenes and applied the CONV model at the pixel level to reconstruct spatiotemporally-latent VWC maps. After applying the fusion strategy, the phenology of winter wheat was successfully recovered in the absence of optical data. Strong correlations were obtained between S2 VWC and S1 & S2 MOGP VWC reconstructed maps for the assessment dates (R2¯wheat−2020 = 0.95, R2¯wheat−2021 = 0.96). Altogether, the fusion of S1 SAR and S2 optical EO data streams with MOGP offers a powerful innovative approach for cropland trait monitoring over cloudy high-latitude regions.

Large scale multi-output multi-class classification using Gaussian processes

Multi-output Gaussian processes (MOGPs) can help to improve predictive performance for some output variables, by leveraging the correlation with other output variables. In this paper, our main motivation is to use multiple-output Gaussian processes to exploit correlations between outputs where each output is a multi-class classification problem. MOGPs have been mostly used for multi-output regression. There are some existing works that use MOGPs for other types of outputs, e.g., multi-output binary classification. However, MOGPs for multi-class classification has been less studied. The reason is twofold: 1) when using a softmax function, it is not clear how to scale it beyond the case of a few outputs; 2) most common type of data in multi-class classification problems consists of image data, and MOGPs are not specifically designed to image data. We thus propose a new MOGPs model called Multi-output Gaussian Processes with Augment & Reduce (MOGPs-AR) that can deal with large scale classification and downsized image input data. Large scale classification is achieved by subsampling both training data sets and classes in each output whereas downsized image input data is handled by incorporating a convolutional kernel into the new model. We show empirically that our proposed model outperforms single-output Gaussian processes in terms of different performance metrics and multi-output Gaussian processes in terms of scalability, both in synthetic and in real classification problems. We include an example with the Ommiglot dataset where we showcase the properties of our model.

Correlation-concerned Bayesian optimization for multi-objective airfoil design

Airfoil design based on Bayesian optimization generally involves high-fidelity simulations, whose crux in terms of efficiency has always challenged existing optimization frameworks. Theoretically, fully mining the useful information in the evaluated samples is expected to speed up the optimization process. In this paper, a multi-objective Bayesian optimization framework explicitly modeling objective correlations is proposed for airfoil design to make the most of the expensive simulations, where a convolved multiple output Gaussian process (CMOGP) surrogate model and a correlated Pareto-frontier entropy search (cPFES) infill sampling strategy are incorporated respectively for reliable capture of the complex relationships involved and final acquisition of well-distributed Pareto airfoils. Specially, a correlation-concerned representative Pareto-frontier sampling method is also proposed to guide candidate selection in cPFES to a prospective optimization direction efficiently. Comparative results on synthetic problems and an airfoil aerodynamic-stealth design problem demonstrate that modeling objective correlations contributes to both the improvement in prediction accuracy for promising regions and the accelerated convergence to a superior and well-distributed Pareto-frontier within a limited budget. It indicates potential practicability of the proposed framework for multi-objective airfoil design as well as other engineering optimization problems to alleviate efficiency issues.

On-chain analytics for sentiment-driven statistical causality in cryptocurrencies

This paper establishes a new framework for assessing multimodal statistical causality between cryptocurrency market (cryptomarket) sentiment and cryptocurrency price processes. In order to achieve this, we present an efficient algorithm for multimodal statistical causality analysis based on Multiple-Output Gaussian Processes. Signals from different information sources (modalities) are jointly modelled as a Multiple-Output Gaussian Process, and then using a novel approach to statistical causality based on Gaussian Processes (GPs), we study linear and non-linear causal effects between the different modalities. We demonstrate the effectiveness of our approach in a machine learning application by studying the relationship between cryptocurrency spot price dynamics and sentiment time-series data specific to the crypto sector, which we conjecture influences retail investor behaviour. The investor sentiment is extracted from cryptomarket news data via methods developed in the area of statistical machine learning known as Natural Language Processing (NLP). To capture sentiment, we present a novel framework for text to time-series embedding, which we then use to construct a sentiment index from publicly available news articles. We conduct a statistical analysis of our sentiment statistical index model and compare it to alternative state-of-the-art sentiment models popular in the NLP literature. In regard to the multimodal causality, the investor sentiment is our primary modality of exploration, in addition to price and a blockchain technology-related indicator (hash rate). Analysis shows that our approach is effective in modelling causal structures of variable degree of complexity between heterogeneous data sources and illustrates the impact that certain modelling choices for the different modalities can have on detecting causality. A solid understanding of these factors is necessary to gauge cryptocurrency adoption by retail investors and provide sentiment- and technology-based insights about the cryptocurrency market dynamics.

Li-Ion Battery State of Health Estimation and Remaining Useful Life Prediction Through a Model-Data-Fusion Method

The prognostics and health management of Li-ion batteries in electric vehicles are challenging due to the time-varying and nonlinear battery degradation. This article proposes a model-data-fusion method for battery state-of-health estimation and remaining useful life prediction. First, combined with metabolic gray model and multiple-output Gaussian process regression, a dynamic and data-driven battery degradation model is established to simulate battery complicated degradation behaviors, which takes the capacity degradation as the state variable and takes the internal resistance and polarization resistance from battery Thevenin model as the input variables. Second, to suppress the measurement noises of online battery information, a particle filter is utilized to track the battery capacity degradation for state-of-health estimation and extrapolate the degradation trajectory for remaining useful life prediction. Furthermore, battery ageing experiments are conducted to verify the proposed model-data-fusion method. The verification results show that the proposed method can provide an accurate and robustness state of health estimation and remaining useful life prediction at different temperatures.

Process-Structure-Property Modeling for Severe Plastic Deformation Processes Using Orientation Imaging Microscopy and Data-Driven Techniques

Machining is a severe plastic deformation process, wherein the workpiece material is subjected to high deformation rates and temperatures. During metal machining, the dynamic recrystallization mechanism causes grain refinement into the sub-micron range. In this study, we investigate the microstructure evolution of oxygen-free high conductivity copper (OFHC Cu) subject to a machining process where the cutting speed and rake angle are controlled to manipulate the process strain, strain rate, and temperatures. Microstructures of the deformed chips are quantified using orientation imaging microscopy and novel statistical descriptors that capture the morphology and local lattice misorientations generated during the several mechanistic stages of the dynamic recrystallization process. Mechanical properties of the resulting chips are quantified using spherical nanoindentation protocols. A multiple output Gaussian process regression model is used to simultaneously model the structure-property evolution, which differs from more common approaches that establish such relationships sequentially. This modeling strategy is particularly attractive since it can flexibly provide both structure and property uncertainty estimates. In addition, the statistical modeling framework allows for the inclusion of multi-fidelity data. The statistical metrics utilized serve as efficient microstructure descriptors, which retain the physics of the observed structures without having to introduce ad hoc microstructure feature definitions.

A new hybrid aerodynamic optimization framework based on differential evolution and invasive weed optimization

Since many aerodynamic optimization problems in the area of aeronautics contain highly nonlinear objectives and multiple local optima, it is still a challenge for most of the traditional optimization methods to find the global optima. In this paper, a new hybrid optimization framework based on Differential Evolution and Invasive Weed Optimization (IWO_DE/Ring) is developed, which combines global and local search to improve the performance, where a Multiple-Output Gaussian Process (MOGP) is used as the surrogate model. We first use several test functions to verify the performance of the IWO_DE/Ring method, and then apply the optimization framework to a supercritical airfoil design problem. The convergence and the robustness of the proposed framework are compared against some other optimization methods. The IWO_DE/Ring-based approach provides much quicker and steadier convergence than the traditional methods. The results show that the stability of the dynamic optimization process is an important indication of the confidence in the obtained optimum, and the proposed optimization framework based on IWO_DE/Ring is a reliable and promising alternative for complex aeronautical optimization problems.

Scalable Joint Models for Reliable Uncertainty-Aware Event Prediction.

Missing data and noisy observations pose significant challenges for reliably predicting events from irregularly sampled multivariate time series (longitudinal) data. Imputation methods, which are typically used for completing the data prior to event prediction, lack a principled mechanism to account for the uncertainty due to missingness. Alternatively, state-of-the-art joint modeling techniques can be used for jointly modeling the longitudinal and event data and compute event probabilities conditioned on the longitudinal observations. These approaches, however, make strong parametric assumptions and do not easily scale to multivariate signals with many observations. Our proposed approach consists of several key innovations. First, we develop a flexible and scalable joint model based upon sparse multiple-output Gaussian processes. Unlike state-of-the-art joint models, the proposed model can explain highly challenging structure including non-Gaussian noise while scaling to large data. Second, we derive an optimal policy for predicting events using the distribution of the event occurrence estimated by the joint model. The derived policy trades-off the cost of a delayed detection versus incorrect assessments and abstains from making decisions when the estimated event probability does not satisfy the derived confidence criteria. Experiments on a large dataset show that the proposed framework significantly outperforms state-of-the-art techniques in event prediction.

Gaussian process regression for forecasting battery state of health

Accurately predicting the future capacity and remaining useful life of batteries is necessary to ensure reliable system operation and to minimise maintenance costs. The complex nature of battery degradation has meant that mechanistic modelling of capacity fade has thus far remained intractable; however, with the advent of cloud-connected devices, data from cells in various applications is becoming increasingly available, and the feasibility of data-driven methods for battery prognostics is increasing. Here we propose Gaussian process (GP) regression for forecasting battery state of health, and highlight various advantages of GPs over other data-driven and mechanistic approaches. GPs are a type of Bayesian non-parametric method, and hence can model complex systems whilst handling uncertainty in a principled manner. Prior information can be exploited by GPs in a variety of ways: explicit mean functions can be used if the functional form of the underlying degradation model is available, and multiple-output GPs can effectively exploit correlations between data from different cells. We demonstrate the predictive capability of GPs for short-term and long-term (remaining useful life) forecasting on a selection of capacity vs. cycle datasets from lithium-ion cells.

Bayesian non-parametric approaches to reconstructing oscillatory systems and the Nyquist limit

Reconstructing continuous signals from discrete time-points is a challenging inverse problem encountered in many scientific and engineering applications. For oscillatory signals classical results due to Nyquist set the limit below which it becomes impossible to reliably reconstruct the oscillation dynamics. Here we revisit this problem for vector-valued outputs and apply Bayesian non-parametric approaches in order to solve the function estimation problem. The main aim of the current paper is to map how we can use of correlations among different outputs to reconstruct signals at a sampling rate that lies below the Nyquist rate. We show that it is possible to use multiple-output Gaussian processes to capture dependences between outputs which facilitate reconstruction of signals in situation where conventional Gaussian processes (i.e. this aimed at describing scalar signals) fail, and we delineate the phase and frequency dependence of the reliability of this type of approach. In addition to simple toy-models we also consider the dynamics of the tumour suppressor gene p53, which exhibits oscillations under physiological conditions, and which can be reconstructed more reliably in our new framework.

Derivative processes for modelling metabolic fluxes

Motivation: One of the challenging questions in modelling biological systems is to characterize the functional forms of the processes that control and orchestrate molecular and cellular phenotypes. Recently proposed methods for the analysis of metabolic pathways, for example, dynamic flux estimation, can only provide estimates of the underlying fluxes at discrete time points but fail to capture the complete temporal behaviour. To describe the dynamic variation of the fluxes, we additionally require the assumption of specific functional forms that can capture the temporal behaviour. However, it also remains unclear how to address the noise which might be present in experimentally measured metabolite concentrations.Results: Here we propose a novel approach to modelling metabolic fluxes: derivative processes that are based on multiple-output Gaussian processes (MGPs), which are a flexible non-parametric Bayesian modelling technique. The main advantages that follow from MGPs approach include the natural non-parametric representation of the fluxes and ability to impute the missing data in between the measurements. Our derivative process approach allows us to model changes in metabolite derivative concentrations and to characterize the temporal behaviour of metabolic fluxes from time course data. Because the derivative of a Gaussian process is itself a Gaussian process, we can readily link metabolite concentrations to metabolic fluxes and vice versa. Here we discuss how this can be implemented in an MGP framework and illustrate its application to simple models, including nitrogen metabolism in Escherichia coli.Availability and implementation: R code is available from the authors upon request.Contact: j.norkunaite@imperial.ac.uk; m.stumpf@imperial.ac.ukSupplementary information: Supplementary data are available at Bioinformatics online.

Modeling Multiresponse Surfaces for Airfoil Design with Multiple-Output-Gaussian-Process Regression

Response surfaces are frequently used in airfoil design due to less resource requirement compared with direct slow simulations. In many cases, multiple responses need to be modeled to achieve multiple objectives. Considering the correlations between multiple responses in modeling the nonlinear relationship between airfoil shapes and aerodynamic performance, the authors construct multiresponse surfaces for airfoil design with multiple-output-Gaussian-process-regression model. The authors simulate computational data to evaluate the prediction accuracy and stability of the multiple-output Gaussian process in airfoil design, compared with other popular alternative approaches, kriging, and backpropagation and radial-basis-function neural networks. In the experiments, response surfaces from the airfoil shapes, parameterized by the class/shape-function-transformation method, to lift, drag, and pitching-moment coefficients are constructed. The results indicate that the multiple-output Gaussian process receives higher prediction accuracy and stability in modeling multiresponse surfaces than other popular methods when there are significant correlations between responses.

Semi-parametric learning for visual odometry

This paper addresses the visual odometry problem from a machine learning perspective. Optical flow information from a single camera is used as input for a multiple-output Gaussian process (MOGP) framework, that estimates linear and angular camera velocities. This approach has several benefits. (1) It substitutes the need for conventional camera calibration, by introducing a semi-parametric model that is able to capture nuances that a strictly parametric geometric model struggles with. (2) It is able to recover absolute scale if a range sensor (e.g. a laser scanner) is used for ground-truth, provided that training and testing data share a certain similarity. (3) It is naturally able to provide measurement uncertainties. We extend the standard MOGP framework to include the ability to infer joint estimates (full covariance matrices) for both translation and rotation, taking advantage of the fact that all estimates are correlated since they are derived from the same vehicle. We also modify the common zero mean assumption of a Gaussian process to accommodate a standard geometric model of the camera, thus providing an initial estimate that is then further refined by the non-parametric model. Both Gaussian process hyperparameters and camera parameters are trained simultaneously, so there is still no need for traditional camera calibration, although if these values are known they can be used to speed up training. This approach has been tested in a wide variety of situations, both 2D in urban and off-road environments (two degrees of freedom) and 3D with unmanned aerial vehicles (six degrees of freedom), with results that are comparable to standard state-of-the-art visual odometry algorithms and even more traditional methods, such as wheel encoders and laser-based Iterative Closest Point. We also test its limits to generalize over environment changes by varying training and testing conditions independently, and also by changing cameras between training and testing.

Computationally Efficient Convolved Multiple Output Gaussian Processes

Recently there has been an increasing interest in regression methods that deal with multiple outputs. This has been motivated partly by frameworks like multitask learning, multisensor networks or structured output data. From a Gaussian processes perspective, the problem reduces to specifying an appropriate covariance function that, whilst being positive semi-definite, captures the dependencies between all the data points and across all the outputs. One approach to account for non-trivial correlations between outputs employs convolution processes. Under a latent function interpretation of the convolution transform we establish dependencies between output variables. The main drawbacks of this approach are the associated computational and storage demands. In this paper we address these issues. We present different efficient approximations for dependent output Gaussian processes constructed through the convolution formalism. We exploit the conditional independencies present naturally in the model. This leads to a form of the covariance similar in spirit to the so called PITC and FITC approximations for a single output. We show experimental results with synthetic and real data, in particular, we show results in school exams score prediction, pollution prediction and gene expression data.