Experienced Optimization with Reusable Directional Model for Hyper-Parameter Search

Optimization methods play a crucial role in various fields and applications. In some optimization problems, the derivative information of the objective function is unavailable. Such black-box optimization problems need to be solved by derivative-free optimization methods. At the same time, optimization problems with ellipsoidal constraints are important and have widespread applications in various fields as well. Following the development of the late professor M. J. D. Powell’s efficient derivative-free trust-region optimization methods, this paper considers solving derivative-free optimization problems on the ellipsoid. Our new optimization solver EC-NEWUOA for problems on the ellipsoid in ℜ n is designed based on Powell’s derivative-free software NEWUOA for unconstrained optimization problems. The proposed techniques for our new method mainly include using the Courant penalty function, the augmented Lagrangian method, and the projection technique. Details about the method and theoretical analysis are included in this paper. We also compare our new method with other algorithms by solving test problems and then show the numerical advantages of our new method.

A structured version of derivative-free random pattern search optimization algorithms is introduced, which is able to exploit coordinate partially separable structure (typically associated with sparsity) often present in unconstrained and bound-constrained optimization problems. This technique improves performance by orders of magnitude and makes it possible to solve large problems that otherwise are totally intractable by other derivative-free methods. A library of interpolation-based modelling tools is also described, which can be associated with the structured or unstructured versions of the initial pattern search algorithm. The use of the library further enhances performance, especially when associated with structure. The significant gains in performance associated with these two techniques are illustrated using a new freely-available release of the Brute Force Optimizer (BFO) package firstly introduced in [Porcelli and Toint 2017 ], which incorporates them. An interesting conclusion of the numerical results presented is that providing global structural information on a problem can result in significantly less evaluations of the objective function than attempting to building local Taylor-like models.

SummaryAlthough it is possible to apply traditional optimization algorithms to determine the Pareto front of a multiobjective optimization problem, the computational cost is extremely high when the objective function evaluation requires solving a complex reservoir simulation problem and optimization cannot benefit from adjoint-based gradients. This paper proposes a novel workflow to solve bi-objective optimization problems using the distributed quasi-Newton (DQN) method, which is a well-parallelized and derivative-free optimization (DFO) method. Numerical tests confirm that the DQN method performs efficiently and robustly.The efficiency of the DQN optimizer stems from a distributed computing mechanism that effectively shares the available information discovered in prior iterations. Rather than performing multiple quasi-Newton optimization tasks in isolation, simulation results are shared among distinct DQN optimization tasks or threads. In this paper, the DQN method is applied to the optimization of a weighted average of two objectives, using different weighting factors for different optimization threads. In each iteration, the DQN optimizer generates an ensemble of search points (or simulation cases) in parallel, and a set of nondominated points is updated accordingly. Different DQN optimization threads, which use the same set of simulation results but different weighting factors in their objective functions, converge to different optima of the weighted average objective function. The nondominated points found in the last iteration form a set of Pareto-optimal solutions. Robustness as well as efficiency of the DQN optimizer originates from reliance on a large, shared set of intermediate search points. On the one hand, this set of searching points is (much) smaller than the combined sets needed if all optimizations with different weighting factors would be executed separately; on the other hand, the size of this set produces a high fault tolerance, which means even if some simulations fail at a given iteration, the DQN method’s distributed-parallel information-sharing protocol is designed and implemented such that the optimization process can still proceed to the next iteration.The proposed DQN optimization method is first validated on synthetic examples with analytical objective functions. Then, it is tested on well-location optimization (WLO) problems by maximizing the oil production and minimizing the water production. Furthermore, the proposed method is benchmarked against a bi-objective implementation of the mesh adaptive direct search (MADS) method, and the numerical results reinforce the auspicious computational attributes of DQN observed for the test problems.To the best of our knowledge, this is the first time that a well-parallelized and derivative-free DQN optimization method has been developed and tested on bi-objective optimization problems. The methodology proposed can help improve efficiency and robustness in solving complicated bi-objective optimization problems by taking advantage of model-based search algorithms with an effective information-sharing mechanism.NOTE: This paper is also published as part of the 2021 SPE Reservoir Simulation Conference Special Issue.

Although it is possible to apply traditional optimization algorithms to determine the Pareto front of a multi-objective optimization problem, the computational cost is extremely high, when the objective function evaluation requires solving a complex reservoir simulation problem and optimization cannot benefit from adjoint-based gradients. This paper proposes a novel workflow to solve bi-objective optimization problems using the distributed quasi-Newton (DQN) method, which is a well-parallelized and derivative-free optimization (DFO) method. Numerical tests confirm that the DQN method performs efficiently and robustly. The efficiency of the DQN optimizer stems from a distributed computing mechanism which effectively shares the available information discovered in prior iterations. Rather than performing multiple quasi-Newton optimization tasks in isolation, simulation results are shared among distinct DQN optimization tasks or threads. In this paper, the DQN method is applied to the optimization of a weighted average of two objectives, using different weighting factors for different optimization threads. In each iteration, the DQN optimizer generates an ensemble of search points (or simulation cases) in parallel and a set of non-dominated points is updated accordingly. Different DQN optimization threads, which use the same set of simulation results but different weighting factors in their objective functions, converge to different optima of the weighted average objective function. The non-dominated points found in the last iteration form a set of Pareto optimal solutions. Robustness as well as efficiency of the DQN optimizer originates from reliance on a large, shared set of intermediate search points. On the one hand, this set of searching points is (much) smaller than the combined sets needed if all optimizations with different weighting factors would be executed separately; on the other hand, the size of this set produces a high fault tolerance. Even if some simulations fail at a given iteration, DQN’s distributed-parallel information-sharing protocol is designed and implemented such that the optimization process can still proceed to the next iteration. The proposed DQN optimization method is first validated on synthetic examples with analytical objective functions. Then, it is tested on well location optimization problems, by maximizing the oil production and minimizing the water production. Furthermore, the proposed method is benchmarked against a bi-objective implementation of the MADS (Mesh Adaptive Direct Search) method, and the numerical results reinforce the auspicious computational attributes of DQN observed for the test problems. To the best of our knowledge, this is the first time that a well-parallelized and derivative-free DQN optimization method has been developed and tested on bi-objective optimization problems. The methodology proposed can help improve efficiency and robustness in solving complicated bi-objective optimization problems by taking advantage of model-based search optimization algorithms with an effective information-sharing mechanism.

Description: Gradient-based optimization algorithms can be very efficient in history matching problems. Since many commercial reservoir simulators do not have an adjoint formulation built in, exploring capability and applicability of derivative-free optimization (DFO) algorithms is crucial. DFO algorithms treat the simulator as a black box and generate new searching points using objective function values only. DFO algorithms usually require more function evaluations, but this obstacle can be overcome by exploiting parallel computing. Application: This paper tests three DFO algorithms, Very Fast Simulated Annealing (VFSA), Simultaneous Perturbation and Multivariate Interpolation (SPMI) and Quadratic Interpolation Model-based (QIM). Both SPMI and QIM are model-based methods. The objective function is approximated by a quadratic model interpolating perturbation points evaluated in previous iterations, and new search points are obtained by minimizing the quadratic model within a trust region. VFSA is a stochastic search method. These algorithms were tested data with two synthetic cases (IC fault model and Brugge model) and one deepwater field case. Principal Component Analysis is applied to the Brugge case to parameterize the reservoir model vector to less than 40 parameters. Conclusions: We obtained good matches with all three derivative-free methods. In terms of number of iterations used for converging and the final converged value of the objective function, SPMI outperforms the others. Since SPMI generates a large number of perturbation and search points simultaneously in one iteration, it requires more computer resources. QIM does not generate as many interpolation points as SPMI, and it converges more slowly. VFSA is a sequential method and usually requires hundreds of iterations to converge. With enough computer resources available, applying the SPMI method is the best choice. When the number of computer cluster nodes is limited, QIM is the best choice. We recommend applying VFSA when using a single computer.

<p>The goal of a learning algorithm is to receive a training data set as input and provide a hypothesis that can generalize to all possible data points from a domain set. The hypothesis is chosen from hypothesis classes with potentially different complexity orders that represent controlling parameters in the learning process, also denoted as hyperparameters. Linear regression modeling is an important category of learning algorithms. Uncertainty of target samples in practical applications affects the generalization performance of the learned model. Failing to choose a proper model or hypothesis class can lead to serious issues such as underfitting or overfitting. These issues have been addressed by alternating cost functions or by utilizing cross-validation methods. These approaches can introduce new hyperparameters with their own new challenges and uncertainties or increase the computational complexity of the learning algorithm. On the other hand, the theory of probably approximately correct (PAC) aims at defining learnability based on probabilistic settings. Despite its theoretical value, PAC does not address practical learning issues on many occasions. This thesis is inspired by the foundation of PAC and is motivated by existing regression learning issues. The proposed approach, denoted by ε-Confidence Approximately Correct (ε-CoAC), utilizes Kullback—Leibler divergence (relative entropy) and proposes a new related typical set in the set of hyperparameters to tackle the learnability issue. ε-CoAC learnability is able to validate the learning process as a function of data length and as a function of the complexity order of the hypothesis class. Moreover, it enables the learner to compare hypothesis classes of different complexity order (hyperparameters) and choose among them the optimum with the minimum ε in the ε-CoAC framework. The ε-CoAC learnability not only overcomes the issues of overfitting and underfitting, but also shows advantages and superiority over the well-known cross-validation method in terms of time consumption and in terms of accuracy. A valuable application of ε-CoAC learnability is presented for simultaneous model order and time delay selection for LTI systems. Classical methods have approached this problem from two separate angles for time-delay estimation and for order selection with different cost functions. The ε-CoAC approach solves the problem with a unified cost function. The proposed method not only outperforms existing approaches but is also shown to be more robust to variations of the signal to noise ratio (SNR). The approach is also extended for online impulse response estimation and introduces efficient stopping criteria that are extremely valuable in practical applications. For the second hyperparameter analysis in machine learning, the challenge of regularization hyperparameter selection for the Support Vector Machine (SVM) algorithm is addressed. The regularization parameter controls the model capacity and the trade-off between the training and the generalization errors. It is shown that interestingly the introduced Separability and Scatteredness (S&S) ratio plays a key role in SVM hyperparameter selection, including kernel hyperparameters. Importance of S&S ratio in this context is similar to the role of the signal-to-noise ratio in the signal processing context. The proposed method outperforms existing cross-validation approaches, especially in the sense of computational complexity. For the hyperparameter selection in unsupervised learning, the fundamentals of ε-CoAC learnability is utilized by viewing the problem of clustering from a new angle. The application of the proposed stochastic based hyperparameter selecting algorithm can be generalized in the form of a validity index. The new validity index is shown to be superior to the state-of-the-art validity indices in the sense of accuracy and robustness to the cluster shape. Finally, the proposed validation index approach is extended for application in graph node clustering. The approach shows advantages over the existing methods in the sense of conductance and graph-based normalizing cuts.</p>

<p>The goal of a learning algorithm is to receive a training data set as input and provide a hypothesis that can generalize to all possible data points from a domain set. The hypothesis is chosen from hypothesis classes with potentially different complexity orders that represent controlling parameters in the learning process, also denoted as hyperparameters. Linear regression modeling is an important category of learning algorithms. Uncertainty of target samples in practical applications affects the generalization performance of the learned model. Failing to choose a proper model or hypothesis class can lead to serious issues such as underfitting or overfitting. These issues have been addressed by alternating cost functions or by utilizing cross-validation methods. These approaches can introduce new hyperparameters with their own new challenges and uncertainties or increase the computational complexity of the learning algorithm. On the other hand, the theory of probably approximately correct (PAC) aims at defining learnability based on probabilistic settings. Despite its theoretical value, PAC does not address practical learning issues on many occasions. This thesis is inspired by the foundation of PAC and is motivated by existing regression learning issues. The proposed approach, denoted by ε-Confidence Approximately Correct (ε-CoAC), utilizes Kullback—Leibler divergence (relative entropy) and proposes a new related typical set in the set of hyperparameters to tackle the learnability issue. ε-CoAC learnability is able to validate the learning process as a function of data length and as a function of the complexity order of the hypothesis class. Moreover, it enables the learner to compare hypothesis classes of different complexity order (hyperparameters) and choose among them the optimum with the minimum ε in the ε-CoAC framework. The ε-CoAC learnability not only overcomes the issues of overfitting and underfitting, but also shows advantages and superiority over the well-known cross-validation method in terms of time consumption and in terms of accuracy. A valuable application of ε-CoAC learnability is presented for simultaneous model order and time delay selection for LTI systems. Classical methods have approached this problem from two separate angles for time-delay estimation and for order selection with different cost functions. The ε-CoAC approach solves the problem with a unified cost function. The proposed method not only outperforms existing approaches but is also shown to be more robust to variations of the signal to noise ratio (SNR). The approach is also extended for online impulse response estimation and introduces efficient stopping criteria that are extremely valuable in practical applications. For the second hyperparameter analysis in machine learning, the challenge of regularization hyperparameter selection for the Support Vector Machine (SVM) algorithm is addressed. The regularization parameter controls the model capacity and the trade-off between the training and the generalization errors. It is shown that interestingly the introduced Separability and Scatteredness (S&S) ratio plays a key role in SVM hyperparameter selection, including kernel hyperparameters. Importance of S&S ratio in this context is similar to the role of the signal-to-noise ratio in the signal processing context. The proposed method outperforms existing cross-validation approaches, especially in the sense of computational complexity. For the hyperparameter selection in unsupervised learning, the fundamentals of ε-CoAC learnability is utilized by viewing the problem of clustering from a new angle. The application of the proposed stochastic based hyperparameter selecting algorithm can be generalized in the form of a validity index. The new validity index is shown to be superior to the state-of-the-art validity indices in the sense of accuracy and robustness to the cluster shape. Finally, the proposed validation index approach is extended for application in graph node clustering. The approach shows advantages over the existing methods in the sense of conductance and graph-based normalizing cuts.</p>

Blackbox optimization refers to problems where the structure of the objective and constraint functions cannot be exploited. This is often the case when their evaluation requires the execution of a (usually time-consuming) simulation using computational models that are typically inaccessible by the user. The term Derivative-Free Optimization refers to the use of algorithms that utilize only function values because their partial derivatives are either not defined or not available; gradient approximations may sometimes be obtained, but the amount of work required to ensure they are dependable may not be worth the effort. Both blackbox and derivative-free optimization have attracted significant, and still increasing, interest from researchers over the last decade. Thus, we felt that it was time to dedicate a special issue of OPTE to this topic. Blackbox and derivative-free optimization methods are often the only realistic and practical tools available to engineers working on simulation-based design. It is obvious that if the design optimization problem at hand allows an evaluation or reliable approximation of the gradients, then efficient gradient-based methods should be used. Blackbox and derivative-free algorithms are not competitors of gradient-based methods; they are a fallback when gradient-based algorithms cannot be used. The design engineering community is increasingly becoming aware that

Production optimization using derivative free methods applied to Brugge field case

Recently, a novel distributed quasi-Newton (DQN) derivative-free optimization (DFO) method was developed for generic reservoir performance optimization problems including well-location optimization (WLO) and well-control optimization (WCO). DQN is designed to effectively locate multiple local optima of highly nonlinear optimization problems. However, its performance has neither been validated by realistic applications nor compared to other DFO methods. We have integrated DQN into a versatile field-development optimization platform designed specifically for iterative workflows enabled through distributed-parallel flow simulations. DQN is benchmarked against alternative DFO techniques, namely, the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method hybridized with Direct Pattern Search (BFGS-DPS), Mesh Adaptive Direct Search (MADS), Particle Swarm Optimization (PSO), and Genetic Algorithm (GA). DQN is a multi-thread optimization method that distributes an ensemble of optimization tasks among multiple high-performance-computing nodes. Thus, it can locate multiple optima of the objective function in parallel within a single run. Simulation results computed from one DQN optimization thread are shared with others by updating a unified set of training data points composed of responses (implicit variables) of all successful simulation jobs. The sensitivity matrix at the current best solution of each optimization thread is approximated by a linear-interpolation technique using all or a subset of training-data points. The gradient of the objective function is analytically computed using the estimated sensitivities of implicit variables with respect to explicit variables. The Hessian matrix is then updated using the quasi-Newton method. A new search point for each thread is solved from a trust-region subproblem for the next iteration. In contrast, other DFO methods rely on a single-thread optimization paradigm that can only locate a single optimum. To locate multiple optima, one must repeat the same optimization process multiple times starting from different initial guesses for such methods. Moreover, simulation results generated from a single-thread optimization task cannot be shared with other tasks. Benchmarking results are presented for synthetic yet challenging WLO and WCO problems. Finally, DQN method is field-tested on two realistic applications. DQN identifies the global optimum with the least number of simulations and the shortest run time on a synthetic problem with known solution. On other benchmarking problems without a known solution, DQN identified compatible local optima with reasonably smaller numbers of simulations compared to alternative techniques. Field-testing results reinforce the auspicious computational attributes of DQN. Overall, the results indicate that DQN is a novel and effective parallel algorithm for field-scale development optimization problems.

The article proposes the optimization of the selection of machine learning algorithms used in tomographic applications. This study used electrical impedance tomography (EIT) to illustrate the distribution of moisture inside the walls of the buildings under study. The first task is to discover the ideal settings of hyperparameters used in machine learning algorithms to increase the efficiency of obtaining reliable tomographic images. The second aim of the research is to choose the optimal method of converting measurements into images. The process of turning input observations into output photos is handled by machine learning models. This is called an ill-posed problem or an inverted problem that is difficult to solve because there are not enough arguments. Ensuring the selection of the correct model hyperparameters is an essential task of machine learning. The selection of these hyperparameters has a direct impact on the quality of the reconstruction. Using the k-nearest neighbors algorithm as an example, this article shows how hyperparameter optimization can be applied to regression and classification models. This technology was created to track and visualize the distribution of moisture inside the walls of buildings and other structures. The facts revealed during the investigation showed that the proposed techniques are effective.

Machine learning (ML) has proven to be highly effective in solving complex problems in various domains, thanks to its ability to identify specific data tasks, perform feature engineering, and learn quickly. However, designing and training ML models is a complicated task and requires optimization. The effectiveness of ML models is highly dependent on the selection of hyperparameters that determines their performance. Hyperparameter optimization (HPO) is a systematic search process to find the optimal combinations of hyperparameters to achieve robust performance. Traditional HPO methods such as grid and random search take a lot of computing time when used in large-scale applications. Recently, various automated search strategies, such as Bayesian optimization (BO) and evolutionary algorithms, have been developed to significantly reduce the computing time. In this paper, we use state-of-the-art HPO frameworks, namely BO, Optuna, HyperOpt, and Keras Tuner, for optimizing the ML and deep learning models for the classification tasks and evaluate their comparative performance using two different sets of experiments. The first one uses different ML classifiers to solve the optimal parameter selection problem with HPO. The second one attempts to optimize the convolutional neural network (CNN) architecture using HPO frameworks to improve its performance in the image classification task. We use four publicly available real-world datasets including one image dataset. The experimental results show that HyperOpt - TPE outperforms the other HPO frameworks for the ML classifiers and achieves up to 94.12% of accuracy with 30 minutes for performing the optimization. Similarly, for the CNN model, HyperOpt-TPE outperforms the other HPO frameworks by improving 34% of the classification accuracy, while taking 2 hours and 24 minutes of computing time.

The emergence of the systematic study of complexity as a science has resulted from the growing recognition that the fundamental assumptions upon which Newtonian physics is based are not satisfied throughout most of science, e.g., time is not necessarily uniformly flowing in one direction, nor is space homogeneous. Herein we discuss how the fractional calculus (FC), renormalization group (RG) theory and machine learning (ML) have each developed independently in the study of distinct phenomena in which one or more of the underlying assumptions of Newtonian formalism is violated. FC has been shown to help us better understand complex systems, improve the processing of complex signals, enhance the control of complex networks, increase optimization performance, and even extend the enabling of the potential for creativity. RG allows one to investigate the changes of a dynamical system at different scales. For example, in quantum field theory, divergent parts of a calculation can lead to nonsensical infinite results. However, by applying RG, the divergent parts can be adsorbed into fewer measurable quantities, yielding finite results. To date, ML is a fashionable research topic and will probably remain so into the foreseeable future. How a model can learn efficiently (optimally) is always essential. The key to learnability is designing efficient optimization methods. Although extensive research has been carried out on the three topics separately, few studies have investigated the association triangle between the FC, RG, and ML. To initiate the study of their interdependence, herein the authors discuss the critical connections between them (Fig. 1). In the FC and RG, scaling laws reveal the complexity of the phenomena discussed. The authors emphasize that the FC’s and RG’s critical connection is the form of inverse power laws (IPL), and the IPL index provides a measure of the level of complexity. For FC and ML, the critical connections in big data, wherein variability, optimization, and non-local models are described. The authors introduce the derivative-free and gradient-based optimization methods and explain how the FC could contribute to these study areas. In the end, the association between the RG and ML is also explained. The mutual information, feature extraction, and locality are also discussed. Many of the cross-sectional studies suggest a connection between the RG and ML. The RG has a superficial similarity to deep neural networks (DNNs) structure in which one marginalizes over hidden degrees of freedom. The authors remark in the conclusions that the association triangle between FC, RG, and ML, form a stool on which the foundation to complexity science might comfortably sit for a wide range of future research topics.

_ This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 206267, “Benchmarking and Field Testing of the Distributed Quasi-Newton Derivative-Free Optimization Method for Field Development Optimization,” by Faruk Alpak, SPE, Yixuan Wang, SPE, and Guohua Gao, SPE, Shell, et al. The paper has not been peer reviewed. _ Recently, a novel distributed quasi-Newton (DQN) derivative-free optimization (DFO) method was developed for generic reservoir-performance optimization problems, including well-location optimization (WLO) and well-control optimization (WCO). DQN is designed to locate multiple local optima of highly nonlinear optimization problems effectively. However, its performance has been neither validated by realistic applications nor compared with other DFO methods. Field-testing results reinforce the auspicious computational attributes of DQN. Background An optimization problem is posed as the minimization or maximization of an objective function by modifying the control variables (x) within a search domain. The objective function is a highly nonlinear function of x and may have multiple local optima. In this paper, the objective function is assumed to be twice differentiable. DFO methods can be classified into local search methods and global search methods. In the complete paper, the authors’ goal is to locate multiple local optima of the objective function. Their focus is on local search DFO optimization methods, which include direct search methods and model-based methods. Current DFO methods reviewed in the complete paper have a common feature: Only one best approximation to the solution is updated in each iteration, and only one optimal solution is identified in the last iteration. Therefore, they are referred to as single-thread DFO methods. They do not represent an efficient approach because simulation results obtained by one optimization task starting from one initial guess are not shared with other optimization tasks that start from different initial guesses. The distributed Gauss-Newton method and the DQN method benchmarked in the paper are referred to as multiple-thread optimization methods. The authors have integrated DQN into a versatile field-development optimization platform designed specifically for iterative work flows enabled through distributed-parallel flow simulations.

Generation and Benefit of Surrogate Models for Blackbox Chemical Flowsheet Optimization