Multi-output Gaussian Process Research Articles

Choose Appropriate Subproblems for Collaborative Modeling in Expensive Multiobjective Optimization.

In dealing with the expensive multiobjective optimization problem, some algorithms convert it into a number of single-objective subproblems for optimization. At each iteration, these algorithms conduct surrogate-assisted optimization on one or multiple subproblems. However, these subproblems may be unnecessary or resolved. Operating on such subproblems can cause server inefficiencies, especially in the case of expensive optimization. To overcome this shortcoming, we propose an adaptive subproblem selection (ASS) strategy to identify the most promising subproblems for further modeling. To better leverage the cross information between the subproblems, we use the collaborative multioutput Gaussian process surrogate to model them jointly. Moreover, the commonly used acquisition functions (also known as infill criteria) are investigated in this article. Our analysis reveals that these acquisition functions may cause severe imbalances between exploitation and exploration in multiobjective optimization scenarios. Consequently, we develop a new acquisition function, namely, adaptive lower confidence bound (ALCB), to cope with it. The experimental results on three different sets of benchmark problems indicate that our proposed algorithm is competitive. Beyond that, we also quantitatively validate the effectiveness of the ASS strategy, the CoMOGP model, and the ALCB acquisition function.

On Negative Transfer and Structure of Latent Functions in Multioutput Gaussian Processes

On Negative Transfer and Structure of Latent Functions in Multioutput Gaussian Processes

A multi-fidelity machine learning approach to high throughput materials screening

The ever-increasing capability of computational methods has resulted in their general acceptance as a key part of the materials design process. Traditionally this has been achieved using a so-called computational funnel, where increasingly accurate - and expensive – methodologies are used to winnow down a large initial library to a size which can be tackled by experiment. In this paper we present an alternative approach, using a multi-output Gaussian process to fuse the information gained from both experimental and computational methods into a single, dynamically evolving design. Common challenges with computational funnels, such as mis-ordering methods, and the inclusion of non-informative steps are avoided by learning the relationships between methods on the fly. We show this approach reduces overall optimisation cost on average by around a factor of three compared to other commonly used approaches, through evaluation on three challenging materials design problems.

Control strategy of multi-stand work roll bending and shifting on the crown for UVC hot rolling mill based on MOGPR approach

In order to improve the crown control ability and high quality to meet the Profile Contour Flatness Control (PCFC) requirements of wide electrical steel in the hot strip mills (HSMs), the control strategy of multi-stand work roll bending forces and shifting stroke on the crown for the universal variable crown (UVC) mill is established based on the Multi-output Gaussian Process Regression (MOGPR) method. The UVC technology with wide crown control ability is proposed for the upstream stands, as effective and critical crown control for upstream stands, of the HSMs. The prediction model of UVC work roll shifting was developed by integrating the MOGPR method and the production process data, and it has high prediction accuracy and adaptability compared with BPNN and SVR models by using multiple evaluation indicators. The multi-stand control strategy of the UVC work roll bending forces and shifting stroke is obtained according to the shape mechanism model and the prediction model based on the MOGPR method. Compared with the conventional control strategy, it can make the most of shifting stroke and the work roll bending forces fluctuates less, of which, F1, F2, and F3 are reduced by 21.85 %, 36.15 %, and 19.67 %, respectively. The industrial application of the control strategy during rolling achieves remarkable results that the measured high precision crown hit rate of electrical steel is significantly improved in the 1450 mm HSMs.

A Framework of Improving Human Demonstration Efficiency for Goal-Directed Robot Skill Learning

Robot learning from humans allows robots to automatically adjust to stochastic and dynamic environments by learning from nontechnical end user’s demonstrations, which is best known as robot programming by demonstration, robot learning from demonstration, apprenticeship learning, and imitation learning. Although most of those methods are probabilistic, and their performances intensively depend on the demonstrated data, measuring and evaluating human demonstrations are rarely investigated. A poorly demonstrated data set with useless prior knowledge or redundant demonstrations increases the complexity and time cost of robot learning. To solve these problems, a goal-directed robot skill learning framework named GPm-MOGP is presented. It 1) decides when and where to add a new demonstration by calculating the trajectory uncertainty; 2) determines which demonstration is useless or redundant by Kullback–Leibler (KL) divergence; 3) implements robot skill learning with a minimum number of demonstrations using a multioutput Gaussian process; and 4) learns orientation uncertainty and representation by combining logarithmic and exponential maps. The proposed framework significantly reduces the demonstrated effort of nontechnical end users who lack an understanding of how and what the robot learns during the demonstrating process. To evaluate the proposed framework, a pick-and-place experiment was designed with five unseen goals to verify the effectiveness of our methods. This experiment is well illustrated with two phases: 1) demonstration efficiency and 2) skill representation and reproduction. The results indicate an improvement of 60% in human demonstration efficiency, compared to common learning from demonstrations (LfD) applications that require at least ten demonstrations, and the robot average success rate of pick-and-place task reaches 85%.

A probabilistic forecasting approach for air quality spatio-temporal data based on kernel learning method

Reliable air quality index forecasting is an important guarantee for government to take early emergency measures to protect the ecological environment and public health. Nevertheless, providing accurate and reliable forecasting is still a difficult and challenging task, owing to the high nonlinearity and noise in the AQI (air quality index). This study develops a novel framework based on feature selection, the Error Correction Model, and the Residual estimation method with new structure fast kernels, to conveniently improve the prediction performance and efficiently quantify the uncertainty of the point prediction generated by any pre-trained model. This framework takes some representative ML (machine learning) tools as examples, with the significant input variables extracted by the two-stage feature selection method for accurate AQI point prediction, and models their residuals by the multi-output Gaussian process and the multi-output Student−t process, which rely on the new multi-output I/O kernels with chronological memory to embed the relevant information of the point prediction and the residuals as well as of different output for making predictions on residual distributions, thereby enhancing prediction performance and enriching uncertainty information. Besides, an acceleration method based on the Orthogonal Random Features method is also proposed to enable the framework to handle large-scale data at an acceptable time cost. The framework is validated by the AQI Spatio-temporal data of four observation points in Shijiazhuang, China, and the result reveals that the proposed framework provides valid prediction intervals and enhances the accuracy of the ML models’ point predictions. Specifically, the framework reduces the prediction error of almost all basic models, and provides shorter prediction intervals than other common methods. Especially, the multi-output GP model or multi-output TP model obtains the best improvement in point prediction and the shortest prediction intervals.

A Fully Natural Gradient Scheme for Improving Inference of the Heterogeneous Multioutput Gaussian Process Model.

A recent novel extension of multioutput Gaussian processes (GPs) handles heterogeneous outputs, assuming that each output has its own likelihood function. It uses a vector-valued GP prior to jointly model all likelihoods' parameters as latent functions drawn from a GP with a linear model of coregionalization (LMC) covariance. By means of an inducing points' framework, the model is able to obtain tractable variational bounds amenable to stochastic variational inference (SVI). Nonetheless, the strong conditioning between the variational parameters and the hyperparameters burdens the adaptive gradient optimization methods used in the original approach. To overcome this issue, we borrow ideas from variational optimization introducing an exploratory distribution over the hyperparameters, allowing inference together with the posterior's variational parameters through a fully natural gradient (NG) optimization scheme. Furthermore, in this work, we introduce an extension of the heterogeneous multioutput model, where its latent functions are drawn from convolution processes. We show that our optimization scheme can achieve better local optima solutions with higher test performance rates than adaptive gradient methods for both the LMC and the convolution process model. We also show how to make the convolutional model scalable by means of SVI and how to optimize it through a fully NG scheme. We compare the performance of the different methods over the toy and real databases.

Expedient validation of LED reliability with anomaly detection through multi-output Gaussian process regression

The existing LED reliability assessment procedures are costly and time-consuming. It poses even greater challenges for the purchasers to validate the manufacturer test results, prior to goods acceptance. In absence of a simple and expedient validation, purchasers often bear the reliability risk in good faith. Leveraging on the 3-stage early degradation modes, as a signature, for LED under the moisture-electrical-temperature (MET) test, an expedient assessment framework is proposed to validate if the supplied LED batches conform to the contracted reliability. Multi-output Gaussian process regression (MOGPR) is performed on sparse reference data to establish an acceptable threshold for the degradation signature. When regressed against the reference data with contracted reliability, the test LED that displays signature anomaly will be deliberated for rejection. The acceptable threshold is calibrated to minimise false negative rate and false positive rate. The procedure is repeated in accordance with acceptable quality level (AQL) or existing sampling policies to accept or reject the batch. The framework achieves a reduction of the validation time from 6000 h to 100 h. While the initial motivation is to facilitate acceptance by purchasers, manufacturer may also adopt this approach for out-going quality control (QC) or rapid evaluation during design prototyping.

State of Health Trajectory Prediction Based on Multi-Output Gaussian Process Regression for Lithium-Ion Battery

Lithium-ion battery state of health (SOH) accurate prediction is of great significance to ensure the safe reliable operation of electric vehicles and energy storage systems. However, safety issues arising from the inaccurate estimation and prediction of battery SOH have caused widespread concern in academic and industrial communities. In this paper, a method is proposed to build an accurate SOH prediction model for battery packs based on multi-output Gaussian process regression (MOGPR) by employing the initial cycle data of the battery pack and the entire life cycling data of battery cells. Firstly, a battery aging experimental platform is constructed to collect battery aging data, and health indicators (HIs) that characterize battery aging are extracted. Then, the correlation between the HIs and the battery capacity is evaluated by the Pearson correlation analysis method, and the HIs that own a strong correlation to the battery capacity are screened. Finally, two MOGPR models are constructed to predict the HIs and SOH of the battery pack. Based on the first MOGPR model and the early HIs of the battery pack, the future cycle HIs can be predicted. In addition, the predicted HIs and the second MOGPR model are used to predict the SOH of the battery pack. The experimental results verify that the approach has a competitive performance; the mean and maximum values of the mean absolute error (MAE) and root mean square error (RMSE) are 1.07% and 1.42%, and 1.77% and 2.45%, respectively.

A multi-output multi-fidelity Gaussian process model for non-hierarchical low-fidelity data fusion

Multi-fidelity (MF) surrogate model has been widely used in dealing with computationally expensive problems as it can make a trade-off between computational cost and modeling accuracy. Most existing MF models are under the assumption of a hierarchical relationship between different fidelity models. In other words, the fidelity level of each low-fidelity (LF) model can be clearly identified. However, the fidelity rankings of the LF models are indistinguishable in many practical problems. Hence, the existing MF models may be inapplicable when dealing with multiple non-hierarchical LF data. To address this issue, a multi-fidelity Gaussian process model for multiple non-hierarchical LF data fusion is developed in multi-output scenarios. It focuses on approximating multiple dependent high-fidelity (HF) responses assisted by multiple non-hierarchical LF data. The multi-output Gaussian process modeling technique makes the proposed model able to catch the latent correlations between outputs. In such a structure, both the information of the multiple LF data and the correlations between outputs can be fully exploited to further enhance the prediction accuracy. The effectiveness of the proposed model is tested using two analytical examples and a metamaterial vibration isolator design problem. It is shown that the proposed model gives the best predictions among the tested models for most cases. Results show that the proposed method can be a potential multi-output multi-fidelity modeling approach for computationally expensive problems in dealing with multiple non-hierarchical LF data.

Multi-outputs Gaussian process for predicting Burkina Faso COVID-19 spread using correlations from the weather parameters.

The novel coronavirus has affected all regions of the world, but each country has experienced different rates of infection. In West Africa, in particular, infection rates remain low as compared to other parts of the world. This heterogeneity in the spread of COVID-19 raises a lot of questions that are still unanswered. However, some studies point out that people's mobility, size of gatherings, rate of testing, and weather have a great impact on the COVID-19 spread. In this work, we first evaluate the correlation between meteorological parameters and COVID-19 cases using Spearman's rank correlation. Secondly, multi-output Gaussian processes (MOGP) are used to predict the daily confirmed COVID-19 cases by exploring its relationships with meteorological parameters. The number of daily reported COVID-19 cases, as well as, weather variables collected from March 9, 2020, to October 18, 2021, were used in the analysis. The weather variables considered in the analysis are the mean temperature, relative humidity, wind direction, insolation, precipitation, and wind speed. The predicting model was constructed exploiting the correlation between the data of the daily confirmed COVID-19 cases and data of the weather variables. The results show that a significant correlation between the daily confirmed COVID-19 cases was found with humidity, wind direction, wind speed, and insolation. These parameters are used to construct the predictive model using the Multi-Output Gaussian process (MOGP). Different combinations of the data of meteorological parameters together with the data of daily reported COVID-19 cases were used to derive different models. We found that the best predictor is obtained using the combination of Humidity and insolation. This model is then used to predict the daily confirmed COVID-19 cases knowing the humidity and Insolation.

Geometrical deviation modeling and monitoring of 3D surface based on multi-output Gaussian process

Geometrical deviation is an important factor in determining the quality of a three-dimensional (3D) Surface. For 3D surfaces with complex shapes, the high-definition measurement (HDM) technology can provide detailed information on surface topography, which inspired new challenges in characterizing, modeling, and monitoring geometrical deviations. This paper proposes a spherical multi-output Gaussian process (S-MOGP) method to model and monitor 3D surfaces. Firstly, the surface in the 3D coordinate system is mapped to the spherical 2D parameter domain. Secondly, a state equation based on the multi-output Gaussian process is established to model the 3D surface. Finally, statistics are calculated and control charts are presented to monitor the geometrical deviation. The results of simulations and a case study show that the proposed method can effectively model 3D surfaces and monitor the geometrical deviations.

Knowledge-based turbomachinery design system via a deep neural network and multi-output Gaussian process

The requirements of future aeroengines challenge turbomachinery designs to be quieter, greener, and more efficient; furthermore, they must be developed at considerably reduced cycles and costs. Traditional turbomachinery design systems involve the iterative design–evaluation–improvement process, leading to high computational costs and considerable reliance on expert experience. In this paper, a knowledge-based turbomachinery aerodynamic design system consisting of a design of experiment framework and a two-stage modeling is proposed. In the first phase of the two-stage modeling, a deep neural network (DNN)-enhanced multi-output Gaussian process model is constructed. This model determines the optimal loading distribution from reference designs that are optimized for the desired performance. In the second stage, a DNN model is developed to predict the turbine profile geometry for a given loading distribution. This formulation enables the automatic determination of the optimal turbine geometry without an iterative process. Furthermore, the knowledge in the design system grows with the increasing database. The results indicate that the L2 relative errors for predicting the optimal pressure coefficient distribution and turbine profile geometry are less than 0.25% and 1.7%, respectively. A design case of a low-pressure turbine is used to validate the entire design system. The results reveal that the turbine geometry generated by the proposed design system outperforms that of the benchmark case in terms of the design and off-design aerodynamic performance. The foregoing observations are finally confirmed through experiments.

Nonlinear Online Multioutput Gaussian Process for Multistream Data Informatics

The revolution of Industry 4.0 and Internet of Things facilitate large and cheap data acquisition from complex systems. However, the online processing and analytics of such multi-stream data still suffer difficulties. The key reason is that data from different sensors (multi-stream data) contain nonlinear within-and-cross sensor correlations, which poses challenges in the modeling and analysis of such data, especially for the online scenarios. To jointly model multi-stream data, multi-output Gaussian Process (MGP) has become popular and achieved great success. However, existing methods assume different outputs are linearly correlated, which significantly limits their applications. In this paper, we propose a nonlinear online MGP framework to simultaneously realize nonlinear correlation modeling and online processing of multi-stream data. Specifically, a tailored co-variance structure is developed based on the convolution process, which can capture nonlinear self-and-cross correlations. To realize online processing, Bayesian analysis and marginalized particle filter are applied to estimate function values and model parameters. Comparing to existing methods, the proposed method features superiorities in i) the non-linear characterization of self-and-cross correlations, and ii) the efficient modeling and prediction capability. These features make the proposed method especially appropriate for industrial informatics with complex relationship and large-scale data sets. The proposed method is validated with simulation studies and a real-case study of biophysical signals.

Long-term prediction enhancement based on multi-output Gaussian process regression integrated with production plans for oxygen supply network

The variation tendency of oxygen demand plays a crucial role in planning and scheduling in the steel industry. Traditional prediction methods merely utilize historical data without considering any future production information. In steel enterprises, the production plan is a kind of easy-to-obtain future production information, which specifies the load adjustment, start-up, and shutdown of equipment in the future. In this paper, a prediction enhancement method based on multi-output Gaussian process regression (MOGPR) for the oxygen demand prediction is first developed. The proposed method models the correlations over three data sources, i.e., the plan-based oxygen prediction, the history-based oxygen prediction, and the oxygen measurement, and transfers the meaningful knowledge between them. Two long-term oxygen demand prediction methods integrated with prediction enhancement are further proposed. Finally, two industrial case studies are used to compare and demonstrate the effectiveness and performance of the proposed methods.

A marine propeller design method based on two-fidelity data levels

A Simulation Based Design Optimization method for marine propellers design using a two-fidelity levels metamodel for global design space exploration and optimization is presented. Response surfaces are built using the co-Kriging approximation, i.e. a multi-output Gaussian process that combines large low-fidelity dataset with few, costly, high-fidelity data. The method is applied for the hydrodynamic shape optimization of the E779A propeller using, as fidelity levels, two different physical models for the propeller performances prediction, namely an inviscid, potential based Boundary Element Method (low-fidelity) and a viscous, finite volume RANS solver (high-fidelity). Results demonstrate the feasibility of multi-objective, constrained, design procedures, like those involving marine propellers, using these multi-fidelity response surfaces. At the same time, the need of good correlations between low- and high-fidelity data feeding the response surfaces is highlighted as a requisite for robust and reliable predictions using these approximated methods.

Generalized multi-output Gaussian process censored regression

When modelling censored observations (i.e. data in which the value of a measurement or observation is un-observable beyond a given threshold), a typical approach in current regression methods is to use a censored-Gaussian (i.e. Tobit) model to describe the conditional output distribution. In this paper, as in the case of missing data, we argue that exploiting correlations between multiple outputs can enable models to better address the bias introduced by censored data. To do so, we introduce a heteroscedastic multi-output Gaussian process model which combines the non-parametric flexibility of GPs with the ability to leverage information from correlated outputs under input-dependent noise conditions. To address the resulting inference intractability, we further devise a variational bound to the marginal log-likelihood suitable for stochastic optimization. We empirically evaluate our model against other generative models for censored data on both synthetic and real world tasks and further show how it can be generalized to deal with arbitrary likelihood functions. Results show how the added flexibility allows our model to better estimate the underlying non-censored (i.e. true) process under potentially complex censoring dynamics.

Lithium battery state-of-health estimation and remaining useful lifetime prediction based on non-parametric aging model and particle filter algorithm

State of health estimation (SOH) and remaining useful lifetime (RUL) prediction are significant health indicators for improving the safety and reliability of battery systems. Herein, a data-fusion method is developed to establish a non-parametric degradation model and a particle filter algorithm for forecasting battery health conditions. Firstly, a dynamic battery aging state-space system is developed, in which Gaussian process regression is applied to establish state equation using historical capacity series and current capacity as input and output variables, respectively. Meanwhile, multi-output Gaussian process regression maps the relationship between capacity degradation and battery health indicators to construct an observation equation. Second, two filter methods are unitized to obtain the smooth differential thermal voltammetry curves and the significant health indicators are extracted from partial differential thermal voltammetry curves. Third, the short-term SOH estimation and long-term RUL prediction are carried out using a particle filter algorithm. Moreover, two types of five batteries with various designed cases are conducted to verify and analyze the proposed method. The results show that the estimation errors of short-term SOH are within 4% and prediction errors of long-term RUL are around 7% (relative error/EOL, 12/159), which indicate the proposed method has an excellent capability for accurate and robust forecasting battery health conditions. • A non-parametric state-space representation for battery degradation is developed. • Battery health conditions are estimated and predicted in multi-timescale perspective. • The health indicators are extracted from partial DTV curves in different dimensions. • Two types of five batteries with various designed cases are discussed and analyzed.

Modeling Neonatal EEG Using Multi-Output Gaussian Processes

Neonatal seizures are sudden events in brain activity with detrimental effects in neurological functions usually related to epileptic fits. Though neonatal seizures can be identified from electroencephalography (EEG), this is a challenging endeavour since expert visual inspection of EEG recordings is time consuming and prone to errors due the data’s nonstationarity and low signal-to-noise ratio. Towards the greater aim of automatic clinical decision making and monitoring, we propose a multi-output Gaussian process (MOGP) framework for neonatal EEG modelling. In particular, our work builds on the multi-output spectral mixture (MOSM) covariance kernel and shows that MOSM outperforms other commonly-used covariance functions in the literature when it comes to data imputation and hyperparameter-based seizure detection. To the best of our knowledge, our work is the first attempt at modelling and classifying neonatal EEG using MOGPs. Our main contributions are: i) the development of an MOGP-based framework for neonatal EEG analysis; ii) the experimental validation of the MOSM covariance kernel on real-world neonatal EEG for data imputation; and iii) the design of features for EEG based on MOSM hyperparameters and their validation for seizure detection (classification) in a patient specific approach.

Bayesian optimization of peripheral intraneural stimulation protocols to evoke distal limb movements

Objective. Motor neuroprostheses require the identification of stimulation protocols that effectively produce desired movements. Manual search for these protocols can be very time-consuming and often leads to suboptimal solutions, as several stimulation parameters must be personalized for each subject for a variety of target motor functions. Here, we present an algorithm that efficiently tunes peripheral intraneural stimulation protocols to elicit functionally relevant distal limb movements. Approach. We developed the algorithm using Bayesian optimization (BO) with multi-output Gaussian Processes (GPs) and defined objective functions based on coordinated muscle recruitment. We applied the algorithm offline to data acquired in rats for walking control and in monkeys for hand grasping control and compared different GP models for these two systems. We then performed a preliminary online test in a monkey to experimentally validate the functionality of our method. Main results. Offline, optimal intraneural stimulation protocols for various target motor functions were rapidly identified in both experimental scenarios. Using the model that performed best, the algorithm converged to stimuli that evoked functionally consistent movements with an average number of actions equal to 20% of the search space size in both the rat and monkey animal models. Online, the algorithm quickly guided the observations to stimuli that elicited functional hand gestures, although more selective motor outputs could have been achieved by refining the objective function used. Significance. These results demonstrate that BO can reliably and efficiently automate the tuning of peripheral neurostimulation protocols, establishing a translational framework to configure peripheral motor neuroprostheses in clinical applications. The proposed method can also potentially be applied to optimize motor functions using other stimulation modalities.