Performance Of Gaussian Process Regression Research Articles

Machine learning techniques for predictive modelling and uncertainty quantification of the mechanical properties of woven carbon fibre composites

Carbon fibre composites owe their exceptional mechanical properties to the characteristics of their constituents; fibres and resin. However, uncertainty in these individual constituent properties can introduce significant variations in the macroscale mechanical performance of composites. These variations are amplified in the presence of various plies in the composite and pose a challenge to quantify during manufacturing stages. In this study, uncertain constituent material properties are propagated to the macroscale composite through multiscale homogenisation to compute the varied resultant thin-shell stiffness matrices. An in-house-built automated simulation environment has been used to create a physics-conforming trainable database for model training, testing, validation and uncertainty quantification using Gaussian Process Regression (GPR), Artificial Neural Networks (ANNs), and Multiple Linear Regression (MLR). Through rigorous comparative analyses, the superior performance of GPR is demonstrated when compared to alternative techniques with R² ≥ 0.99 and NRMSE < 10⁻⁷. Further, the findings approve the validity of GPR for extrapolatory predictions. For instance, a randomly sampled [-5, 5] % variation of constituent material properties leads to extreme uncertainties in macroscale ± 0.067 %, for axial stiffness with excellent GPR predictive accuracy of R² = 0.99. This slightly increases to ± 0.14 % for a variation of [-10, 10] %. Moreover, the extreme uncertainties indicate that the direct axial and bending stiffnesses are more sensitive to variations in constituent properties than others. Additional investigations presented in the paper produce predictive models and uncertainty bounds for all stiffness entries that are of high importance for macroscale property control during manufacturing.

Nonparametric modeling of a high-speed USV at three speed regions based on Gaussian process regression with a hybrid kernel function

Due to the complexity and variability of the ocean environment, it is difficult for high-speed unmanned surface vehicles (USVs) to construct a unified mathematical model for nonlinear motion under different Froude numbers. In this paper, a nonparametric modeling method for high-speed USVs at three speed regions is presented based on Gaussian process regression with a hybrid kernel function (HKF-GPR). First, the hybrid kernel functions are designed based on the forecasting performance of Gaussian process regression with different single kernel functions (SKF-GPR) using sea trial speed data from the “Jiuhang 750” USV. Second, to better capture the dynamic performance of the USVs, the sea trial speed data are clustered into three speed regions by an improved k-means++ algorithm; these regions are the displacement, semiplanning, and planning regions. Finally, USV modeling and forecasting are performed in the three speed regions using HKF-GPR. The trained model shows good modeling accuracy and forecasting, verifying the effectiveness of the proposed method.

Operational Status Evaluation of Smart Electricity Meters Using Gaussian Process Regression With Optimized-ARD Kernel

Operational Status Evaluation (OSE) is essential for the health management of Smart Electricity Meters (SEM). This paper develops a machine learning-enabled OSE method for SEM. Specifically, the Gaussian Process Regression (GPR) is developed for modeling analysis, where an Optimized Automatic Relevance Determination (OARD) kernel is first used. Though the conventional ARD kernel structure can capture the potential mapping relationship between running time, temperature, humidity, and Measurement Error (ME), it cannot extract the influence degree of different stresses on the ME. To address this problem, an OARD kernel structure is proposed to identify the contribution of each part in ARD structure to the target data. Furthermore, the quartiles line instead of 95% confidence interval is exploited to enhance the performance of GPR for long-term prediction. Combining the two above improvements, a novel OSE model is established for the future-oriented long-term operation of the SEM. It is the first-known data-driven application that utilizes the GPR with OARD kernel to perform OSE for SEM. Actual SEM datasets collected from both dry and hot region are used for model validation and prediction. The results demonstrate that the proposed GPR model with OARD Matern52 kernel outperforms other conventional kernel approaches in the aspect of interpretability. More importantly, the operational status of the SEM in future can be assessed via the proposed OSE framework.

Gaussian Process Regression With Interpretable Sample-Wise Feature Weights.

Gaussian process regression (GPR) is a fundamental model used in machine learning (ML). Due to its accurate prediction with uncertainty and versatility in handling various data structures via kernels, GPR has been successfully used in various applications. However, in GPR, how the features of an input contribute to its prediction cannot be interpreted. Here, we propose GPR with local explanation, which reveals the feature contributions to the prediction of each sample while maintaining the predictive performance of GPR. In the proposed model, both the prediction and explanation for each sample are performed using an easy-to-interpret locally linear model. The weight vector of the locally linear model is assumed to be generated from multivariate Gaussian process priors. The hyperparameters of the proposed models are estimated by maximizing the marginal likelihood. For a new test sample, the proposed model can predict the values of its target variable and weight vector, as well as their uncertainties, in a closed form. Experimental results on various benchmark datasets verify that the proposed model can achieve predictive performance comparable to those of GPR and superior to that of existing interpretable models and can achieve higher interpretability than them, both quantitatively and qualitatively.

Freeze-thaw durability estimation for concrete through the Gaussian process regression with kernel convolution

This study aimed to improve the performance of Gaussian process regression (GPR) estimation through kernel convolution for relative dynamic modulus (RDM) of the freeze–thaw (FT) test. The FT test includes various parameters that affect its outcome. However, FT test results must be measured by humans directly, and due to the characteristics of the experiment, output data is very small. This makes it difficult to apply AI to predict the FT durability process. This study focused on the Bayesian approach of the GPR method with small data. The results showed that kernel convolution improved the estimation performance, as demonstrated by the validation of the fitted GPR with a convoluted kernel. The convolution work could be simplified as the 5/2 Matèrn kernel and periodic kernel showed perfect independence from each other. The independence checking results showed that the chi-square was 0.8579, the Fisher test was 0.98, and the Pearson correlation was 0.00129 respectively in the posterior state. The drawbacks of each kernel were reinforced by the strong points of the other when the kernels performed the convolution, resulting in a reasonable range of RDM estimation. Therefore, it was proved that Bayesian regression performance could be improved through kernel convolution.

Physics-based Penalization for Hyperparameter Estimation in Gaussian Process Regression

In Gaussian Process Regression (GPR), hyperparameters are often estimated by maximizing the marginal likelihood function. However, this data-dominant hyperparameter estimation process can lead to poor extrapolation performance and often violates known physics, especially in sparse data scenarios. In this paper, we embed physics-based knowledge through penalization of the marginal likelihood objective function and study the effect of this new objective on consistency of optimal hyperparameters and quality of GPR fit. Three case studies are presented, where physics-based knowledge is available in the form of linear Partial Differential Equations (PDEs), while initial or boundary conditions are not known so direct forward simulation of the model is challenging. The results reveal that the new hyperparameter set obtained from the augmented marginal likelihood function can improve the prediction performance of GPR, reduce the violation of the underlying physics, and mitigate overfitting problems.

Runoff prediction using hydro-meteorological variables and a new hybrid ANFIS-GPR model

Abstract Precise and credible runoff forecasting is extraordinarily vital for various activities of water resources deployment and implementation. The neoteric contribution of the current article is to develop a hybrid model (ANFIS-GPR) based on adaptive neuro-fuzzy inference system (ANFIS) and Gaussian process regression (GPR) for monthly runoff forecasting in the Beiru river of China, and the optimal input schemes of the models are discussed in detail. Firstly, variables related to runoff are selected from the precipitation, soil moisture content, and evaporation as the first set of input schemes according to correlation analysis (CA). Secondly, principal component analysis (PCA) is used to eliminate the redundant information between the original input variables for forming the second set of input schemes. Finally, the runoff is predicted based on different input schemes and different models, and the prediction performance is compared comprehensively. The results show that the input schemes jointly established by CA and PCA (CA-PCA) can greatly improve the prediction accuracy. ANFIS-GPR displays the best forecasting performance among all the peer models. In the single models, the performance of GPR is better than that of ANFIS.

Predicting physical fatigue in athletes in rope skipping training using ECG signals

Physical fatigue is a crucial factor that leads to a decrease in performance, especially in athletes. This study proposes effective physiological indicators and methods to predict physical fatigue. We designed a ”training-cognitive task” experiment using specialized electrocardiogram (ECG) acquisition devices (Shimmer3) to collect ECG data during the training process. The participants were required to complete a 2-back cognitive task before and after one-hour of rope skipping training. After the experiment, we analyzed the heart rate variability (HRV) signals of the subjects using the time domain, frequency domain, and nonlinear dynamics. Pearson’s correlation coefficient analysis and paired t-tests were applied to the measured indicators. Subsequently, selected ECG indicators with statistical significance, the particle swarm optimization-support vector regression (PSO-SVR), and extreme learning machine (ELM) were used to build regression models that predicted subjective and objective physical fatigue. Moreover, we have compared the performance of Gaussian process regression (GPR), random forest (RF), k-nearest neighbors (KNN), and adaptive boosting (AdaBoost) based machine learning model for prediction. The results indicated that PSO-SVR had a better performance in predicting subjective physical fatigue, with mean absolute percentage error (MAPE) of 11.53%, and ELM achieved competitive outcomes for objective physical fatigue prediction with MAPE of 19.77%. This study provides a reliable experimental basis for identifying effective physical fatigue.

Analysis of noisy transient signals based on Gaussian process regression

Dynamic systems such as cells or tissues generate, either spontaneously or in response to stimuli, transient signals that carry information about the system. Characterization of recorded transients is often hampered by a low signal-to-noise ratio (SNR). Reduction of the noise by filtering has limited use due to partial signal distortion. Occasionally, transients can be approximated by a mathematical function, but such a function may not hold correctly if recording conditions change. We introduce here the model-independent approximation method for general noisy transient signals based on the Gaussian process regression. The method was implemented in the software TransientAnalyzer, which detects transients in a record, finds their best approximation by the Gaussian process, constructs a surrogate spline function, and estimates specified signal parameters. The method and software were tested on a cellular model of the calcium concentration transient corrupted by various SNR levels and recorded at a low sampling frequency. Statistical analysis of the model data sets provided the error of estimation <7.5% and the coefficient of variation of estimates <17% for peak SNR = 5. The performance of Gaussian process regression on signals of diverse experimental origin was even better than fitting by a function. The software and its description are available on GitHub.

A proficient approach to forecast COVID-19 spread via optimized dynamic machine learning models

This study aims to develop an assumption-free data-driven model to accurately forecast COVID-19 spread. Towards this end, we firstly employed Bayesian optimization to tune the Gaussian process regression (GPR) hyperparameters to develop an efficient GPR-based model for forecasting the recovered and confirmed COVID-19 cases in two highly impacted countries, India and Brazil. However, machine learning models do not consider the time dependency in the COVID-19 data series. Here, dynamic information has been taken into account to alleviate this limitation by introducing lagged measurements in constructing the investigated machine learning models. Additionally, we assessed the contribution of the incorporated features to the COVID-19 prediction using the Random Forest algorithm. Results reveal that significant improvement can be obtained using the proposed dynamic machine learning models. In addition, the results highlighted the superior performance of the dynamic GPR compared to the other models (i.e., Support vector regression, Boosted trees, Bagged trees, Decision tree, Random Forest, and XGBoost) by achieving an averaged mean absolute percentage error of around 0.1%. Finally, we provided the confidence level of the predicted results based on the dynamic GPR model and showed that the predictions are within the 95% confidence interval. This study presents a promising shallow and simple approach for predicting COVID-19 spread.

Runoff forecasting model based on CEEMD and combination model: a case study in the Manasi River, China

Abstract Accurate forecasting of runoff is necessary for water resources management. However, the runoff time series consists of complex nonlinear and non-stationary characteristics, which makes forecasting difficult. To contribute towards improved forecasting accuracy, a novel combination model based on complementary ensemble empirical mode decomposition (CEEMD) for runoff forecasting is proposed and applied in this paper. Firstly, the original runoff series is decomposed into a limited number of intrinsic mode functions (IMFs) and one residual based on CEEMD, which makes the runoff time series stationary. Then, approximate entropy is introduced to judge the complexity of each IMF and residual. According to the calculation results of approximate entropy, the high complexity components are predicted by Gaussian process regression (GPR), the medium complexity components are predicted by support vector machine (SVM), and the low complexity components are predicted by autoregressive integrated moving average model (ARIMA). The advantages of each forecasting model are used to forecast the appropriate components. In order to solve the problem that the forecasting performance of GPR and SVM is affected by their parameters, an improved fireworks algorithm (IFWA) is proposed to optimize the parameters of two models. Finally, the final forecasting result is obtained by adding the forecasted values of each component. The runoff data collected from the Manasi River, China is chosen as the research object. Compared with some state-of-the-art forecasting models, the comparison result curve between the forecasted value and actual value of runoff, the forecasting error, the histogram of the forecasting error distribution, the performance indicators and related statistical indicators show that the developed forecasting model has higher prediction accuracy and is able to reflect the change laws of runoff correctly.

Comparison of Bayesian, k-Nearest Neighbor and Gaussian process regression methods for quantifying uncertainty of suspended sediment concentration prediction

Suspended sediment transport in river system is a complex process influenced by many factors that their interactions lead to nonlinear and high scatter of concentration-discharge relationships. This makes the model prediction subject to high uncertainty and providing one value as the model prediction is somehow useless and cannot provide adequate information about the model accuracy and associated uncertainty. Current study compares the efficiency of Bayesian (i.e. Bayesian segmented linear regression (BSLR) and Bayesian linear model (BLR)), Gaussian Process Regression (GPR) and k-Nearest Neighbor (k-NN) in quantifying uncertainty of the suspended sediment concentration prediction in three watersheds namely Arazkoseh, Oghan and Jajrood located in Iran. Three input combinations including, contemporary discharge, slow and quick flow components and contemporary, one and two antecedent days discharge, were used. The BSLR model was able to identify threshold value, furthermore, pre-threshold and post-threshold slopes of BSLR model indicated that for Arazkoseh watershed channel and for Oghan and Jajrood watersheds, upland area are dominate sediment sources. In all three studied cases, given prediction interval width and the percent of enclosed observed data by prediction interval, k-NN model provided more reliable prediction interval. Moreover, separation stream flow into slow and quick flow components lead to improved performance of GPR and k-NN models in the studied watersheds, and the best results for Arazkoseh and Oghan watersheds were obtained when slow and quick flow components were used as the model input.

Alfalfa yield estimation based on time series of Landsat 8 and PROBA-V images: An investigation of machine learning techniques and spectral-temporal features

Remote Sensing (RS) technology provides regular monitoring of alfalfa farms, as a major source of forage production worldwide. Phenological characteristics derived from time series of RS imagery provide a valuable information source to estimate crop yield accurately. In this study, we computed spectral vegetation indices (SVIs) from time series of Landsat 8 and PROBA-V images to extract temporal characteristics of alfalfa farms throughout the growth periods in three consecutive years in the Moghan plain, Iran. Then, several new spectral-temporal features were developed based on phenological characteristics of alfalfa during the growing season. Such features particularly describe geometry and variations of the temporal curves and are thus invaluable in describing phenological attributes. We conducted several feature selection methods due to the variety of features. Machine learning (ML) methods, including ridge, lasso, Gaussian Process Regression (GPR), Random Forest Regression (RFR), Boosted Regression Trees (BRT), and Support Vector Regression (ν-SVR) were utilized to build inversion models in order to estimate alfalfa yields, where the results showed satisfactory performance of GPR using the selected features by GS (RMSE=1114.0 kg/ha), RReliefF (RMSE=1157.7 kg/ha) and Boruta (RMSE=1210.2 kg/ha) as compared to the complete feature dataset (RMSE=1237.4 kg/ha). Overall, the developed phenological features coupled with feature selection methods resulted in the appropriate performance of the ML methods in alfalfa yield estimation.

Feature selection and Gaussian process prediction of rougher copper recovery

The emergence of advanced computational techniques provides opportunity in better modelling complex, nonlinear relationship of industrial processes. In this work, the performance of Gaussian Process Regression (GPR) models in predicting rougher copper recovery, following development with and without model-selected, relevant rougher flotation variables has been investigated. Randomly selected training and validation data sets as well as an independent testing data set (later than the original training and validation sets) were used to demonstrate the model predictive behaviour. The results showed that exponential GPR covariance function, modelled with selected relevant rougher flotation variables (throughput, feed particle size, xanthate dosage, frother dosage and froth depth), was the best model yielding correlation coefficient (r), root mean square error (RMSE), normalised root mean square error (NRMSE), variance accounted for (VAF) and performance index (PI) values of 0.99, 0.10, 0.01%, 99.99% and 1.97, respectively for training set. In the same manner, the validation set yielded 0.95, 0.40, 0.06%, 91.0%, and 1.75 whilst the testing set gave 0.96, 0.39, 0.07%, 92.10% and 1.78, confirming that exponential GPR algorithm can make good rougher copper recovery prediction given a set of useful rougher flotation variables. In all cases, the Regularised Neighbourhood Component Analysis algorithm performed best better in selecting useful flotation variables for rougher copper recovery prediction. Sobol’s global sensitivity analysis indicated feed particle size as the most sensitive input variable recording first and total sobol indices values of ~63% and ~67%, respectively. Partial dependence plots revealed the functional relationship between the top two sensitive input variables (feed particle size and froth depth of tank cell 1) and predicted rougher copper recovery, suggesting their critical operating regime.

Data-Aided Sensing for Gaussian Process Regression in IoT Systems

In this article, for efficient data collection with limited bandwidth, data-aided sensing (DAS) is applied to the Gaussian process regression (GPR) that is used to learn data sets collected from sensors in the Internet-of-Things systems. We focus on the interpolation of sensors' measurements from a small number of measurements uploaded by a fraction of sensors using GPR with DAS. Thanks to active sensor selection, it is shown that GPR with DAS can provide a good estimate of a complete data set compared to that with random selection. With multichannel ALOHA, DAS is generalized for distributed selective uploading when sensors can have feedback of predictions of their measurements so that each sensor can decide whether or not it uploads by comparing its measurement with the predicted one. Numerical results show that modified multichannel ALOHA with predictions can help improve the performance of GPR with DAS compared to the conventional multichannel ALOHA with equal uploading probability.

Challenging handheld NIR spectrometers with moisture analysis in plant matrices: Performance of PLSR vs. GPR vs. ANN modelling

The global demand for natural products grows rapidly, intensifying the request for the development of high-throughput, fast, non-invasive tools for quality control applicable on-site. Moisture content is one of the most important quality parameters of natural products. It determines their market suitability, stability and shelf life and should preferably be constantly monitored. Miniaturized near-infrared (NIR) spectroscopy is a powerful method for on-site analysis, potentially fulfilling this requirement. Here, a feasibility study for applicability and analytical performance of three miniaturized NIR spectrometers and two benchtop instruments was evaluated in that scenario. The case study involved 192 dried plant extracts composed of five different plants harvested in different countries at various times within two years. The reference analysis by Karl Fischer titration determined the water content in this sample set between 1.36% and 6.47%. For the spectroscopic analysis half of the samples were laced with a drying agent to comply with the industry standard. The performance of various calibration models for NIR analysis was evaluated on the basis of root-mean square error of prediction (RMSEP) determined for an independent test set. Partial least squares regression (PLSR), Gaussian process regression (GPR) and artificial neural network (ANN) models were constructed for the spectral sets from each instrument. GPR and ANN models performed superior for all samples measured by handheld spectrometers and for native ones analyzed by benchtop instruments. Moreover, the accuracy penalty when analyzing native samples was lower for GPR and ANN prediction as well. With GPR or ANN calibration, miniaturized spectrometers offered the prediction performance at the level of the benchtop instruments. Therefore, in this analytical application miniaturized spectrometers can be used on-site with no penalty to the performance vs. laboratory-based NIR analysis.

Applicability of machine learning in modeling of atmospheric particle pollution in Bangladesh.

Atmospheric particle pollution causes acute and chronic health effects. Predicting the concentrations of PM2.5 and PM10, therefore, is a prerequisite to avoid the consequences and mitigate the complications. This research utilized the machine learning (ML) models such as linear-support vector machine (L-SVM), medium Gaussian-support vector machine (M-SVM), Gaussian process regression (GPR), artificial neural network (ANN), random forest regression (RFR), and a time series model namely PROPHET. Atmospheric NOX, SO2, CO, and O3, along with meteorological variables from Dhaka, Chattogram, Rajshahi, and Sylhet for the period of 2013 to 2019, were utilized as exploratory variables. Results showed that the overall performance of GPR performed better particularly for Dhaka in predicting the concentration of both PM2.5 and PM10 while ANN performed best in case of Chattogram and Sylhet for predicting PM2.5. However, in terms of predicting PM10, M-SVM and RFR were selected respectively. Therefore, this study recommends utilizing “ensemble learning” models by combining several best models to advance application of ML in predicting pollutants’ concentration in Bangladesh.Electronic supplementary materialThe online version of this article (10.1007/s11869-020-00878-8) contains supplementary material, which is available to authorized users.

Compositional kernel learning using tree-based genetic programming for Gaussian process regression

Although Gaussian process regression (GPR) is a powerful Bayesian nonparametric regression model for engineering problems, its predictive performance is highly dependent on a kernel for covariance function of GPR. However, choosing a proper kernel is still challenging even for experts. To choose proper kernel automatically, this study proposes a compositional kernel (CPK) learning using tree-based genetic programming (GEP). The optimal structure of the kernel is defined as a compositional representation based on sums and products of eight base-kernels. The CPK can be encoded as a tree-structure, so that tree-based GEP is employed to discover an optimal tree-structure of the CPK. To avoid overly complex solution in GEP, the proposed method introduced a dynamic maximum tree-depth technique. The novelty of the proposed method is to utilize more flexible and efficient learning capability to learn the relationship between input and output than existing methods. To evaluate the learning capability of the proposed method, seven test functions were firstly investigated with various noise levels, and its predictive accuracy was compared with existing methods. Reliability problems in both parallel and series systems were introduced to evaluate the performance of the proposed method for efficient reliability assessment. The results show that the proposed method generally outperforms or performs similarly to the best one among existing methods. In addition, it is also shown that proper kernel function can significantly improve the performance of GPR as the training data increases. Stated differently, the proposed method can learn the function of being fitted efficiently with less training samples than existing methods. In this context, the proposed method can make powerful and automatic predictive modeling based on GPR in engineering problems.

Estimation of municipal waste generation of Turkey using socio-economic indicators by Bayesian optimization tuned Gaussian process regression.

Accurate estimation of municipal solid waste (MSW) generation has become a crucial task in decision-making processes for the MSW planning and management systems. In this study, the Gaussian process regression (GPR) model tuned by Bayesian optimization was used to forecast the MSW generation of Turkey. The Bayesian optimization method, which can efficiently optimize the hyperparameters of kernel functions in the machine learning algorithms, was applied to reduce the computation redundancy and enhance the estimation performance of the models. Four socio-economic indicators such as population, gross domestic product per capita, inflation rate, and the unemployment rate were used as input variables. The performance of the Bayesian GPR (BGPR) model was compared with the multiple linear regression (MLR) and Bayesian support vector regression (BSVR) models. Different performance measures such as mean absolute deviation (MAD), root mean square error (RMSE), and coefficient of determination (R2) values were used to evaluate the performance of the models. The exponential-GPR model tuned by Bayesian optimization showed superior performance with minimum MAD (0.0182), RMSE (0.0203), and high R2 (0.9914) values in the training phase and minimum MAD (0.0342), RMSE (0.0463), and high R2 (0.9841) values in the testing phase. The results of this study can help decision-makers to be aware of social-economic factors associated with waste management and ensure optimal usage of their resources in future planning.

Prediction of sediment transport rates in gravel-bed rivers using Gaussian process regression

AbstractEstimating sediment transport rate in rivers has high importance due to the difficulties and costs associated with its measurement, which has drawn the attention of experts in water engineering. In this study, Gaussian process regression (GPR) is applied to predict the sediment transport rate for 19 gravel-bed rivers in the United States. To compare the performance of GPR, the support vector machine (SVM) as a common type of kernel-based models was developed. Model inputs of sediment transport were prepared based on two scenarios: the first scenario considers only hydraulic characteristics and the second scenario was formed using hydraulic and sediment properties. Obtained results revealed that the GPR models present better performance compared to the SVM models and other empirical sediment transport formulas. Also, it was found that incorporating the second scenario as input led to better predictions. In addition, performing sensitivity analysis showed that the ratio of average velocity to shear flow velocity is the most effective parameter in predicting the sediment transport rate of gravel-bed rivers.