Financial Applications of Gaussian Processes and Bayesian Optimization

Physics-aware Gaussian processes in remote sensing

Machine learning with knowledge constraints for process optimization of open-air perovskite solar cell manufacturing

Algorithms for machine learning have found extensive use in numerous fields and applications. One important aspect of effectively utilizing these algorithms is tuning the hyperparameters to match the specific task at hand. The selection and configuration of hyperparameters directly impact the performance of machine learning models. Achieving optimal hyperparameter settings often requires a deep understanding of the underlying models and the appropriate optimization techniques. While there are many automatic optimization techniques available, each with its own advantages and disadvantages, this article focuses on hyperparameter optimization for well-known machine learning models. It explores cutting-edge optimization methods such as metaheuristic algorithms, deep learning-based optimization, Bayesian optimization, and quantum optimization, and our paper focused mainly on metaheuristic and Bayesian optimization techniques and provides guidance on applying them to different machine learning algorithms. The article also presents real-world applications of hyperparameter optimization by conducting tests on spatial data collections for landslide susceptibility mapping. Based on the experiment's results, both Bayesian optimization and metaheuristic algorithms showed promising performance compared to baseline algorithms. For instance, the metaheuristic algorithm boosted the random forest model's overall accuracy by 5% and 3%, respectively, from baseline optimization methods GS and RS, and by 4% and 2% from baseline optimization methods GA and PSO. Additionally, for models like KNN and SVM, Bayesian methods with Gaussian processes had good results. When compared to the baseline algorithms RS and GS, the accuracy of the KNN model was enhanced by BO-TPE by 1% and 11%, respectively, and by BO-GP by 2% and 12%, respectively. For SVM, BO-TPE outperformed GS and RS by 6% in terms of performance, while BO-GP improved results by 5%. The paper thoroughly discusses the reasons behind the efficiency of these algorithms. By successfully identifying appropriate hyperparameter configurations, this research paper aims to assist researchers, spatial data analysts, and industrial users in developing machine learning models more effectively. The findings and insights provided in this paper can contribute to enhancing the performance and applicability of machine learning algorithms in various domains.

Gaussian process regression is a well-established Bayesian machine learning method. We propose a new approach to Gaussian process regression using quantum kernels based on parameterized quantum circuits. By employing a hardware-efficient feature map and careful regularization of the Gram matrix, we demonstrate that the variance information of the resulting quantum Gaussian process can be preserved. We also show that quantum Gaussian processes can be used as a surrogate model for Bayesian optimization, a task that critically relies on the variance of the surrogate model. To demonstrate the performance of this quantum Bayesian optimization algorithm, we apply it to the hyperparameter optimization of a machine learning model which performs regression on a real-world dataset. We benchmark the quantum Bayesian optimization against its classical counterpart and show that quantum version can match its performance.

Abstarct Engineering design problems typically require optimizing a quality measure by finding the right combination of controllable input parameters. In Additive Manufacturing (AM), the output characteristics of the process can often be non-stationary functions of the process parameters. Bayesian Optimization (BO) is a methodology to optimize such “black-box” functions, i.e., the input–output relationship is unknown and expensive to compute. Optimization tasks involving “black-box” functions widely use BO with Gaussian Process (GP) regression surrogate model. Using GPs with standard kernels is insufficient for modeling non-stationary functions, while GPs with non-stationary kernels are typically over-parameterized. On the other hand, a Deep Gaussian Process (DGP) can overcome GPs’ shortcomings by considering a composition of multiple GPs. Inference in a DGP is challenging due to its structure resulting in a non-Gaussian posterior, and using DGP as a surrogate model for BO is not straightforward. Stochastic Imputation (SI)-based inference is promising in speed and accuracy for BO. This work proposes a bootstrap aggregation-based procedure to effectively utilize the SI-based inference for BO with a DGP surrogate model. The proposed BO algorithm DGP-SI-BO is faster and empirically better than the state-of-the-art BO method in optimizing non-stationary functions. Several analytical test functions and a case study in metal AM simulation demonstrate the applicability of the proposed method.

Bayesian optimization is a powerful technique for finding extrema of an objective function, a closed-form expression of which is not given but expensive evaluations at query points are available. Gaussian Process (GP) regression is often used to estimate the objective function and uncertainty estimates that guide GP-Upper Confidence Bound (GP-UCB) to determine where next to sample from the objective function, balancing exploration and exploitation. In general, it requires an auxiliary optimization to tune the hyperparameter in GP-UCB, which is sometimes not easy to carry out in practice. In this paper we present a simple practical method which improves GP-UCB, especially in cases where the objective function is not smooth with sharp peaks and valleys. We first present a geometric interpretation of GP-UCB on which we base our development of the clustering-guided method to select the next observation. Clustering is applied to two-dimensional vectors whose entries correspond to the posterior mean and standard deviation computed by GP regression, which is followed by utility maximization with GP-UCB, in order to determine where next to sample from the objective function. Experiments on various functions demonstrate our method alleviates the chance of being trapped in local extrema, making small efforts for auxiliary optimization.

The correct modeling of the interest rates term structure should definitely be considered an aspect of primary importance since the forward rates and the discount factors used in any financial and risk analysis are calculated from such structure. The turbulence of the markets in recent years, with negative interest rates followed by their recent substantial rise, the period of the COVID pandemic crisis, the political instabilities linked to the war between Ukraine and Russia have very often led to observe anomalies in the shape of the interest rate curve that are difficult to represent using traditional econometric models, to the point that researchers have to address this modeling problem using Machine Learning methodologies. The purpose of this study is to design a model selection heuristic which, starting from the traditional ones (Nelson-Siegel, Svensson and de Rezende-Ferreira) up to the Gaussian Process (GP) Regression, is able to define the best representation for a generic term structure. This approach has been tested over the past five years on term structures denominated in five different currencies: the Swiss Franc (CHF), the Euro (EUR), the British Pound (GBP), the Japanese Yen (JPY) and the U.S. Dollar (USD).

<div class="section abstract"><div class="htmlview paragraph">A common scenario in engineering design is the evaluation of expensive black-box functions: simulation codes or physical experiments that require long evaluation times and/or significant resources, which results in lengthy and costly design cycles. In the last years, Bayesian optimization has emerged as an efficient alternative to solve expensive black-box function design problems. Bayesian optimization has two main components: a probabilistic surrogate model of the black-box function and an acquisition functions that drives the design process. Successful Bayesian optimization strategies are characterized by accurate surrogate models and well-balanced acquisition functions. The Gaussian process (GP) regression model is arguably the most popular surrogate model in Bayesian optimization due to its flexibility and mathematical tractability. GP regression models are defined by two elements: the mean and covariance functions. In some modeling scenarios, the prescription of proper mean and covariance functions can be a difficult task, e.g., when modeling non-stationary functions and heteroscedastic noise. Motivated by recent advancements in the deep learning community, this study explores the implementation of deep Gaussian processes (DGPs) as surrogate models for Bayesian optimization in order to build flexible predictive models from simple mean and covariance functions. The proposed methodology employs DGPs as the surrogate models and the Euclidean-based expected improvement as the acquisition function. This approach is compared with a strategy that employs GP regression models. These methodologies solve two analytical problems and one engineering problem: the design of sandwich composite armors for blast mitigation. The analytical problems involve non-convex and segmented Pareto fronts. The engineering problem involves expensive finite element simulations, three design variables, and two expensive black-box function objectives. The results show that the architecture of the DGP model plays an important role in the performance of the optimization approach. If the DGP architecture is adequate, the implementation of DGPs produces satisfactory results; otherwise, the use of GP regression models is preferable.</div></div>

Earth observation from satellite sensory data pose challenging problems, where machine learning is currently a key player. In recent years, Gaussian Process (GP) regression and other kernel methods have excelled in biophysical parameter estimation tasks from space. GP regression is based on solid Bayesian statistics, and generally yield efficient and accurate parameter estimates. However, GPs are typically used for inverse modeling based on concurrent observations and in situ measurements only. Very often a forward model encoding the well-understood physical relations is available though. In this work, we review three GP models that respect and learn the physics of the underlying processes in the context of inverse modeling. First, we will introduce a Joint GP (JGP) model that combines in situ measurements and simulated data in a single GP model. Second, we present a latent force model (LFM) for GP modeling that encodes ordinary differential equations to blend data-driven modeling and physical models of the system. The LFM performs multi-output regression, adapts to the signal characteristics, is able to cope with missing data in the time series, and provides explicit latent functions that allow system analysis and evaluation. Finally, we present an Automatic Gaussian Process Emulator (AGAPE) that approximates the forward physical model via interpolation, reducing the number of necessary nodes. Empirical evidence of the performance of these models will be presented through illustrative examples of vegetation monitoring and atmospheric modeling.

Gaussian cycles and Bayesian optimization (BO) are effective and sound among the machine learning groups for their applications in finance and various other fields. Whether it is the asset management, market creation, options trading, or risk management, algorithms of machine learning are being used [1]. Regardless of doubt about past executions, we should now concede that machine learning is altering the financial industry. Almost every segment in the industry has massively invested in machine learning and mostly believes that this is only the start. Indeed, even regulators are intently watching this turn of events and its effect on the sector. Until now the greatest pointer is the improvement of the financial market employing. These days, quant finance applicants should pass certification in machine learning, or at least know about this innovation and have experience in the Python programming language [2].

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

Bayesian optimization, coupled with Gaussian process regression and acquisition functions, has proven to be a powerful tool in the field of experimental design.Nevertheless, it demands a profound proficiency in software programming, machine learning, and statistical concepts.This steep learning curve presents a substantial obstacle when implementing Bayesian optimization for experimental design.In order to overcome this challenge, a user-friendly graphical interface for Gaussian process regression and acquisition functions is proposed.This accessible tool can be readily accessed via web browsers, courtesy of the established CADS platform (available at https://cads.eng.hokudai.ac.jp/).Thus, the interface offers to perform Bayesian optimization without any programming or any extensive prior knowledge about Bayesian optimization and machine learning.

With the sustainable development of the construction industry, recycled aggregate (RA) has been widely used in concrete preparation to reduce the environmental impact of construction waste. Compressive strength is an essential measure of the performance of recycled aggregate concrete (RAC). In order to understand the correspondence between relevant factors and the compressive strength of recycled concrete and accurately predict the compressive strength of RAC, this paper establishes a model for predicting the compressive strength of RAC using machine learning and hyperparameter optimization techniques. RAC experimental data from published literature as the dataset, extreme gradient boosting (XGBoost), random forest (RF), K-nearest neighbour (KNN), support vector machine regression Support Vector Regression (SVR), and gradient boosted decision tree (GBDT) RAC compressive strength prediction models were developed. The models were validated and compared using correlation coefficients (R2), Root Mean Square Error (RMSE), mean absolute error (MAE), and the gap between the experimental results of the predicted outcomes. In particular, The effects of different hyperparameter optimization techniques (Grid search, Random search, Bayesian optimization-Tree-structured Parzen Estimator, Bayesian optimization- Gaussian Process Regression) on model prediction efficiency and prediction accuracy were investigated. The results show that the optimal combination of hyperparameters can be searched in the shortest time using the Bayesian optimization algorithm based on TPE (Tree-structured Parzen Estimator); the BO-TPE-GBDT RAC compressive strength prediction model has higher prediction accuracy and generalisation ability. This high-performance compressive strength prediction model provides a basis for RAC’s research and practice and a new way to predict the performance of RAC.

Optimization under uncertainty is inherent to many PSE applications ranging from process design to RTO. Reaching process true optima often involves learning from experimentation, but actual experiments involve a cost (economic, resources, time) and a budget limit usually exists. Finding the best trade-off on cumulative process performance and experimental cost over a finite budget is a Partially Observable Markov Decision Process (POMDP), known to be computationally intractable. This paper follows the nonmyopic Bayesian optimization (BO) approximation to POMDPs developed by the machine-learning community, that naturally enables the use of hybrid plant surrogate models formed by fundamental laws and Gaussian processes (GP). Although nonmyopic BO using GPs may look more tractable, evaluating multi-step decision trees to find the best first-stage candidate action to apply is still expensive with evolutionary or NLP optimizers. Hence, we propose modelling the value function of the first-stage decision also with a GP, whose data will correspond to virtual evaluations of second-stage decision trees build upon myopic rollouts. Thus, the nonmyopic initial decision can be efficiently optimized via BO and the virtually learned value function. Effectiveness of the approach is demonstrated in a wide benchmark with synthetically generated functions as well as to optimize small batch production with a chemical reactor.

Wind turbines are growing bigger to becomemore cost-efficient. This does increase the severity of the vibrations that are present in the turbine blades, both due to predictable effects like wind shear and tower shadow, and due to less predictable effects like turbulence and flutter. If wind turbines are to become bigger and more cost-efficient, these vibrations need to be reduced. This can be done by installing trailing-edge flaps to the blades. Because of the variety of circumstances which the turbine should operate in, this results in large uncertainties. As such, we need methods that can take stochastic effects into account. Preferably we develop an algorithmthat can learn from online data how the flaps affect the wind turbine and how to optimally control them. A simple prior analysis can be done using a linearized version of the system. In this case it is important to know not only the expected cost (damage) that will be incurred by the wind turbine in various situations, but also the spread of this cost. This can for instance be done by looking at the variance of the cost function. Various expressions are available to analytically calculate this variance. Alternatively, we can prescribe a degree of stability for the system. Due to the limitations of linear approximations of systems, it is more effective to apply nonlinear regression methods. A promising one is Gaussian Process (GP) regression. Given a training set (X, y) it can predict function values f (x¤) for test points x¤. It has its basis in Bayesian probability theory, which allows it to not only make this prediction, but also give information (the variance) about its accuracy. The usual way in which GP regression is applied has a few important limitations. Most importantly, it is computationally intensive, especially when applied to constantly growing data sets. In addition, it has difficulties dealing with noise present in the training input points x. There are methods to solve either of these issues, but these tricks generally do not work well together, or their combination requires many computational resources. However, by making the right approximations, like Taylor expansions and at times even linearizations, Gaussian process regression can be applied efficiently, in an online way, to data sets with noisy input points. This enables GP regression to be used for system identification problems like online non-linear black-box modeling. Another limitation is that it can be difficult to find the optimum of a Gaussian process. The reason is that the optimum of a Gaussian process is not a fixed point but a random variable. The distribution of this optimum cannot be calculated analytically, but we can use particle methods to approximate it. We can subsequently use this principle to efficiently explore an unknown nonlinear function, trying to locate its optimum. To do so, we sample a point x from the optimum distribution, measure what the function value f (x) at this point is, update the Gaussian process approximation of the function, update the optimum distribution and repeat this process until the distribution has converged. Finding the optimum of a function like this has shown to have competitive performance at keeping the cumulative regret low, compared to similar algorithms. In addition, it allows wind turbines to tune the gains of a fixed-structure controller so as to optimize a nonlinear cost function like the damage equivalent load. All these improvements are a step forward in the application of Gaussian process regression to wind turbine applications. But as is always the case with research, there are still many things left to improve further.