Improvement In Expectancy Research Articles

A comprehensive framework based on Bayesian optimization and skip connections artificial neural networks to predict buildings energy performance

Artificial Neural Networks (ANNs) are a powerful modeling and prediction tool. However, in the energy and buildings fields, certain articles have been highlighted for reducing ANNs' ability to aid in building energy prediction. This result stems from the author's limited ability to regulate a large number of hyper-parameters. As a result, this paper provides a comprehensive study of ANN challenges and solutions in a single framework. The study's research aims may be stated as defining the necessary data preparation stages; selecting the appropriate ANN design for each study scenario with its hyper-parameters; and presenting the approach for training and evaluating ANNs to obtain the best performance. All of these concerns are presented in this study, compiled into a comprehensive framework, and implemented on two different types of real-life data: 1) CBECS data for US commercial buildings, and 2) ASHRAE data for different buildings around the world. The framework optimization loop mainly relies on the Bayesian optimization technique, which employs Gaussian Processes (GP) as a surrogate function; a combination of two concepts (i.e., Expected Improvement (EI) and Probability Improvement (PI)) as an acquisition function; hyper-parameters as a dimension space optimization; and adjusted R2 value as a target optimization score to be improved. Various skip connection ANN architectures are applied to demonstrate the ability to modify ANNs' limitations in the energy and building fields, assuring excellent accuracy and reliability in forecasting building energy performance, with adjusted R2 of more than 0.93 for both data sets.

Solving Highly Expensive Optimization Problems via Evolutionary Expected Improvement

Although many methods have been proposed to solve expensive optimization problems (EOPs), they often consume hundreds of function evaluations (FEs) to find the optimal solution, which is unacceptable when facing highly EOPs. To reduce the number of FEs, we incorporate the population distribution into the well-known expected improvement (EI); thus, a new infill criterion called evolutionary EI (EEI) is proposed. In EEI, the covariance matrix adaptation evolution strategy is used to provide the population distribution. Compared with the original EI, EEI focuses more on promising regions provided by the population distribution, thus, reducing the FEs wasted in unpromising regions. By employing EEI as the infill criterion of Bayesian optimization, a new algorithm called EEI-BO is designed. Moreover, we also introduce an extended version of EEI-BO, called EEI-BO <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{+}$</tex-math> </inline-formula> , to handle multitask EOPs. To verify the effectiveness of EEI-BO, it is used to solve 10 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$-$</tex-math> </inline-formula> , 20 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$-$</tex-math> </inline-formula> , and 30-D test problems by using only 40, 50, and 60 FEs, respectively. The results show that EEI-BO is able to obtain high-quality solutions by consuming limited FEs. In addition, we apply EEI-BO to deal with the lightweight and crashworthiness design of the side body of an automobile. The results demonstrate that EEI-BO performs well on solving it. Furthermore, the performance of EEI-BO <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{+}$</tex-math> </inline-formula> is investigated by nine test problems. The results show that it has the capability to solve multitask EOPs with fast convergence speed.

A Hierarchical Expected Improvement Method for Bayesian Optimization

The Expected Improvement (EI) method, proposed by Jones, Schonlau, andWelch, is a widely used Bayesian optimization method, which makes use of a fitted Gaussian process model for efficient black-box optimization. However, one key drawback of EI is that it is overly greedy in exploiting the fitted Gaussian process model for optimization, which results in suboptimal solutions even with large sample sizes. To address this, we propose a new hierarchical EI (HEI) framework, which makes use of a hierarchical Gaussian process model. HEI preserves a closed-form acquisition function, and corrects the over-greediness of EI by encouraging exploration of the optimization space. We then introduce hyperparameter estimation methods which allow HEI to mimic a fully Bayesian optimization procedure, while avoiding expensive Markov-chain Monte Carlo sampling steps. We prove the global convergence of HEI over a broad function space, and establish near-minimax convergence rates under certain prior specifications. Numerical experiments show the improvement of HEI over existing Bayesian optimization methods, for synthetic functions and a semiconductor manufacturing optimization problem. Supplementary materials for this article are available online.

Analysis of the Effectiveness of Metaheuristic Methods on Bayesian Optimization in the Classification of Visual Field Defects.

Bayesian optimization (BO) is commonly used to optimize the hyperparameters of transfer learning models to improve the model's performance significantly. In BO, the acquisition functions direct the hyperparameter space exploration during the optimization. However, the computational cost of evaluating the acquisition function and updating the surrogate model can become prohibitively expensive due to increasing dimensionality, making it more challenging to achieve the global optimum, particularly in image classification tasks. Therefore, this study investigates and analyses the effect of incorporating metaheuristic methods into BO to improve the performance of acquisition functions in transfer learning. By incorporating four different metaheuristic methods, namely Particle Swarm Optimization (PSO), Artificial Bee Colony (ABC) Optimization, Harris Hawks Optimization, and Sailfish Optimization (SFO), the performance of acquisition function, Expected Improvement (EI), was observed in the VGGNet models for visual field defect multi-class classification. Other than EI, comparative observations were also conducted using different acquisition functions, such as Probability Improvement (PI), Upper Confidence Bound (UCB), and Lower Confidence Bound (LCB). The analysis demonstrates that SFO significantly enhanced BO optimization by increasing mean accuracy by 9.6% for VGG-16 and 27.54% for VGG-19. As a result, the best validation accuracy obtained for VGG-16 and VGG-19 is 98.6% and 98.34%, respectively.

Parallel efficient global optimization by using the minimum energy criterion

In optimization problems, the expensive black-box function implies severely restricted budgets in terms of evaluation. Some Bayesian optimization methods have been proposed to solve this problem, such as expected improvement (EI) and hierarchical expected improvement (HEI). Neither EI nor HEI is parallel, which depends on a one-point-at-a-time strategy. In this work, a new parallel Bayesian framework based on the minimum energy criterion is proposed to improve these popular one-point methods. The new proposed framework can save time and costs by reducing the number of iterations and avoid the local optimization trap by encouraging the exploration of the optimization space. Additionally, a shrink-augment strategy is also introduced to correct the local surrogate model for the black-box function adaptively, which could also benefit the optimization. Some numerical and illustrative experiments are presented to demonstrate the superiority of our proposed method over some other Bayesian methods. The results show that the novel framework can balance exploitation and exploration well and has great performance in global optimization.

An active learning reliability algorithm using DMSSA‐optimized Kriging model and parallel infilling strategy

AbstractDue to the limited sample sizes and highly complicated performance functions, how to improve the reliability calculation accuracy and efficiency is an important issue for complex equipment working in harsh environments. This paper proposes an active learning reliability algorithm based on the double‐mutation slap swarm algorithm‐optimized (DMSSA) Kriging surrogate model, parallel infilling strategy and subset simulation (SS). In the method, the uniform design (UD) is employed to select the initial sampling points and their real responses of performance functions are estimated. The DMSSA integrates Cauchy and Gauss mutation to explore the optimal hyperparameter of Kriging model with high accuracy. The parallel infilling strategy combines the expected improvement (EI), minimizing prediction (MP) and mean squared error (MSE) criteria to find new sampling points, which are adaptively added into the database in each iteration to update the design of experiment (DoE). Ultimately, the SS method is employed to deal with the small failure probability problems. Five various examples are analyzed to verify the calculation accuracy and efficiency of the proposed method.

Multi-surrogate-assisted stochastic fractal search algorithm for high-dimensional expensive problems

Surrogate models have been radically used in metaheuristic algorithms owing to their capacity in solving computationally expensive problems. However, despite the promising performance of surrogate-assisted metaheuristic algorithms in coping with low-dimensional problems, they failed to tackle high-dimensional problems efficiently. Thus, a multi-surrogate-assisted stochastic fractal search algorithm (MSASFS) is proposed in this paper. Several improvements are integrated into the algorithm design: (1) By combining the original coordinate system with the eigencoordinate system, an improved surrogate-assisted differential evolution (SDE) updating mechanism is proposed to ameliorate the generalization ability of the algorithm and extend the scope of exploration. (2) A new expected improvement (EI) pre-screening strategy based on the Gaussian process (GP) model is employed to select promising candidate solutions. (3) Two different surrogate models are applied to enhance the robustness of the proposed algorithm. The effectiveness of MSASFS is further demonstrated by numerical experiments on some widely used benchmark problems with dimensions ranging from 30 to 200 and parameter estimation problem of fractional-order chaotic systems. The results reveal that, compared with state-of-the-art surrogate-assisted evolutionary algorithms (SAEAs), the proposed algorithm can effectively solve high-dimensional expensive problems. Furthermore, MSASFS shows a more significant efficiency when the dimension of problems becomes higher.

An efficient global optimization algorithm combining revised expectation improvement criteria and Kriging

The efficient global optimization (EGO) algorithm is a kind of Bayesian optimization algorithm that uses the Kriging interpolation model and expectation improvement (EI) criteria as surrogate model and acquisition function, respectively. However, the greediness of EI criteria can lead the EGO algorithm to fall into local optima. Owing to this, revised expectation improvement (REI) criteria are proposed by introducing a balance factor to adjust the exploitation and exploration of EI criteria, and the corresponding algorithm is called the revised efficient global optimization (REGO) algorithm. In order to motivate exploration, and ensure that the computational cost is acceptable, a Latin hypercube based indicator is proposed to denote a balance factor from the viewpoint of sample distribution. Several test functions and an airfoil optimization problem are applied to verify the performance of the REGO algorithm. The results show that the REGO algorithm has acceptable computational cost, a strong ability to find global optima, and good robustness.

A Fast Multipoint Expected Improvement for Parallel Expensive Optimization

The multipoint expected improvement (EI) criterion is a well-defined parallel infill criterion for expensive optimization. However, the exact calculation of the classical multipoint EI involves evaluating a significant amount of multivariate normal cumulative distribution functions, which makes the inner optimization of this infill criterion very time consuming when the number of infill samples is large. To tackle this problem, we propose a novel fast multipoint EI criterion in this work. The proposed infill criterion is calculated using only univariate normal cumulative distributions; thus, it is easier to implement and cheaper to compute than the classical multipoint EI criterion. It is shown that the computational time of the proposed fast multipoint EI is several orders lower than the classical multipoint EI on the benchmark problems. In addition, we propose to use cooperative coevolutionary algorithms (CCEAs) to solve the inner optimization problem of the proposed fast multipoint EI by decomposing the optimization problem into multiple subproblems with each subproblem corresponding to one infill sample and solving these subproblems cooperatively. Numerical experiments show that using CCEAs can improve the performance of the proposed algorithm significantly compared with using standard evolutionary algorithms. This work provides a fast and efficient approach for parallel expensive optimization.

An Adaptive Parallel EI Infilling Strategy Extended by Non-Parametric PMC Sampling Scheme for Efficient Global Optimization

This paper proposes an adaptive parallel Expected Improvement (EI) sampling scheme applied to Efficient Global Optimization, called non-parametric Population Monte Carlo sampling (NPMS) scheme. The NPMS scheme introduces the Population Monte Carlo (PMC) method and the Density-Based Spatial Clustering (DBSCAN) method to address the problem of candidate points aggregation in the parallel EI strategy. In the first stage, samples are uniformly generated from EI function and converge to high EI value sub-domains by PMC iterative succession. A non-parametric sampling method is used to aid PMC convergence and provides valuable sub-domains ranges for the scheme. In the second stage, learning from current potential information, the DBSCAN method is used to cluster the samples and converge to candidate points. During NPMS benchmarking, we verified the feasibility and analyzed the impact of parameters on the scheme. Compared to original EI strategy, NPMS improved EGO minimum results by 14.6% and reduced candidate points by 15.8%. In addition, four test groups are used to compare the NPMS with five other EI-based strategies. Results show that the NPMS scheme has advantages in finding results and saving costs. Mainly, in large input space test group, NPMS saves 34.9% optimization costs and can find better results than parallel strategies. NPMS scheme adaptive sampling provides stable optimization results in the poor prior information, which is expected to provide efficient sampling solutions in more complex problems.

Using Deep Learning with Bayesian–Gaussian Inspired Convolutional Neural Architectural Search for Cancer Recognition and Classification from Histopathological Image Frames

We propose a neural architectural search model which examines histopathological images to detect the presence of cancer in both lung and colon tissues. In recent times, deep artificial neural networks have made tremendous impacts in healthcare. However, obtaining an optimal artificial neural network model that could yield excellent performance during training, evaluation, and inferencing has been a bottleneck for researchers. Our method uses a Bayesian convolutional neural architectural search algorithm in collaboration with Gaussian processes to provide an efficient neural network architecture for efficient colon and lung cancer classification and recognition. The proposed model learns by using the Gaussian process to estimate the required optimal architectural values by choosing a set of model parameters through the exploitation of the expected improvement (EI) values, thereby minimizing the number of sampled trials and suggesting the best model architecture. Several experiments were conducted, and a landmark performance was obtained in both validation and test data through the evaluation of the proposed model on a dataset consisting of 25,000 images of five different classes with convergence and F1‐score matrices.

Volute Optimization Based on Self-Adaption Kriging Surrogate Model

Optimizing the volute performance can effectively improve the efficiency of a centrifugal fan by changing the volute geometric parameter, so the self-adaption Kriging surrogate model is used to optimize the volute geometric parameter. Firstly, volute radius Rd, the radius of tongue r, and outlet angle of the volute θ are selected as the optimization parameters of the volute, and latin hypercube sampling is used to configure the initial sample points, the corresponding three-dimensional aerodynamic model under each sample point configuration is constructed. CFD software is used to simulate the aerodynamic efficiency and total pressure of the centrifugal fan under each initial sample point configuration. Secondly, the Kriging surrogate model of initial sample point configuration parameters, aerodynamic efficiency, and total pressure of volute is constructed, and sample points are added by expectation improvement (EI) method to improve the fitting accuracy of Kriging surrogate model. Finally, the high-precision Kriging surrogate model is used as the fitness function of NSGA-II algorithm to find the Pareto optimal solution under multiobjective optimization, and the optimization target are aero dynamical efficiency and total pressure. The rationality of the above method is verified by optimizing the 9–19.4A type centrifugal fan volute. The efficiency of the optimized fan under working conditions is increased by 1%, and the total pressure under working conditions is not reduced. The optimized volute can effectively improve the overall performance of the centrifugal fan. This study is helpful to promote the application of numerical optimization design method in the volute of centrifugal fan. It provides reference for the optimization design of high-performance centrifugal fan.

Dispersion‐enhanced sequential batch sampling for adaptive contour estimation

AbstractIn computer simulation and optimal design, sequential batch sampling offers an appealing way to iteratively stipulate optimal sampling points based upon existing selections and efficiently construct surrogate modeling. Nonetheless, the issue of near duplicates poses tremendous quandary for sequential learning. It refers to the situation that selected critical points cluster together in each sampling batch, which are individually but not collectively informative towards the optimal design. Near duplicates severely diminish the computational efficiency as they barely contribute extra information towards update of the surrogate. To address this issue, we impose a dispersion criterion on concurrent selection of sampling points, which essentially forces a sparse distribution of critical points in each batch, and demonstrate the effectiveness of this approach in adaptive contour estimation. Specifically, we adopt Gaussian process surrogate to emulate the simulator, acquire variance reduction of the critical region from new sampling points as a dispersion criterion, and combine it with the modified expected improvement (EI) function for critical batch selection. The critical region here is the proximity of the contour of interest. This proposed approach is vindicated in numerical examples of a two‐dimensional four‐branch function, a four‐dimensional function with a disjoint contour of interest and a time‐delay dynamic system.

Multiple Surrogate-Model-Based Optimization Method Using the Multimodal Expected Improvement Criterion for Expensive Problems

In this article, a multiple surrogate-model-based optimization method using the multimodal expected improvement criterion (MSMEIC) is proposed. In MSMEIC, an important region is first identified and used alternately with the whole space. Then, in each iteration, three common surrogate models, kriging, radial basis function (RBF), and quadratic response surface (QRS), are constructed, and a multipoint expected improvement (EI) criterion that selects the highest peak and other peaks of EI is proposed to obtain several potential candidates. Furthermore, the optimal predictions of the three surrogate models are regarded as potential candidates. After deleting redundant candidates, the remaining points are saved as the new sampling points. Finally, several well-known benchmark functions and an engineering application are employed to assess the performance of MSMEIC. The testing results demonstrate that, compared with four recent counterparts, the proposed method can obtain more precise solutions more efficiently and with strong robustness.

A new active learning approach for adsorbate-substrate structural elucidation in silico.

Adsorbate interactions with substrates (e.g. surfaces and nanoparticles) are fundamental for several technologies, such as functional materials, supramolecular chemistry, and solvent interactions. However, modeling these kinds of systems in silico, such as finding the optimum adsorption geometry and energy, is challenging, due to the huge number of possibilities of assembling the adsorbate on the surface. In the current work, we have developed an artificial intelligence (AI) approach based on an active learning (AL) method for adsorption optimization on the surface of materials. AL uses machine learning (ML) regression algorithms and their uncertainties to make a decision (based on a policy) for the next unexplored structures to be computed, increasing, though, the probability of finding the global minimum with a small number of calculations. The methodology allows an accurate and automated structural elucidation of the adsorbate on the surface, based on the minimization of the total electronic energy. The new AL method for adsorption optimization was developed and implemented in the quantum machine learning software/agent for material design and discovery (QMLMaterial) program and was applied for C60@TiO2 anatase (101). It marks another software extension with a new feature in addition to the automatic structural elucidation of defects in materials and of nanoparticles as well. SCC-DFTB calculations were used to build the complex search surfaces with a reasonably low computational cost. An artificial neural network (NN) was employed in the AL framework evaluated together with two uncertainty quantification methods: K-fold cross-validation and non-parametric bootstrap (BS) resampling. Also, two different acquisition functions for decision-making were used: expected improvement (EI) and the lower confidence bound (LCB).

A surrogate-based parallel optimization of analog circuits using multi-acquisition functions

Surrogate-based optimization has been widely used in analog circuit sizing. Surrogate-based optimization efficiency in a parallel computing environment has become an emerging problem. In this paper, a surrogate-based parallel optimization of analog circuits using multi-acquisition functions (SPMA) is proposed. The Surrogate Model-Aware Search Mechanism (SMAS) is used as the basis. Under the SMAS framework, modeling and search are concentrated in promising areas. The probability of improvement (PI), expected improvement (EI), and lower confidence bound (LCB) combined in the multi-acquisition function each balance exploration and exploitation in its own way. It is necessary to configure enough training data points for the local model constructed in the promising area to improve the exploration ability of the multi-acquisition functions. Multiple mutation operations are used to provide more promising candidate designs and increase the probability that the surrogate model can prescreen to the optimal solutions. SPMA combines the advantages of multi-acquisition functions and multiple mutation operations, and has been verified by a two-stage amplifier and a three-stage amplifier, as well as three well-known benchmark functions. The optimization results show that SPMA is superior to state-of-the-art methods in terms of efficiency and effectiveness.

Cross-Validation--based Adaptive Sampling for Gaussian Process Models

In many real-world applications, we are interested in approximating black-box, costly functions as accurately as possible with the smallest number of function evaluations. A complex computer code is an example of such a function. In this work, a Gaussian process (GP) emulator is used to approximate the output of complex computer code. We consider the problem of extending an initial experiment (set of model runs) sequentially to improve the emulator. A sequential sampling approach based on leave-one-out (LOO) cross-validation is proposed that can be easily extended to a batch mode. This is a desirable property since it saves the user time when parallel computing is available. After fitting a GP to training data points, the expected squared LOO (ES-LOO) error is calculated at each design point. ES-LOO is used as a measure to identify important data points. More precisely, when this quantity is large at a point it means that the quality of prediction depends a great deal on that point and adding more samples nearby could improve the accuracy of the GP. As a result, it is reasonable to select the next sample where ES-LOO is maximised. However, ES-LOO is only known at the experimental design and needs to be estimated at unobserved points. To do this, a second GP is fitted to the ES-LOO errors and where the maximum of the modified expected improvement (EI) criterion occurs is chosen as the next sample. EI is a popular acquisition function in Bayesian optimisation and is used to trade-off between local/global search. However, it has a tendency towards exploitation, meaning that its maximum is close to the (current) "best" sample. To avoid clustering, a modified version of EI, called pseudo expected improvement, is employed which is more explorative than EI yet allows us to discover unexplored regions. Our results show that the proposed sampling method is promising.

Aseismic Optimization of Mega-sub Controlled Structures Based on Gaussian Process Surrogate Model

Due to the complex seismic characteristics of mega-sub controlled structures (MSCS), it is difficult to give full play to their advantages in earthquake resistance by traditional design methods. Meta-heuristic optimization algorithms can be used to improve the seismic performance, but the structural response needs to be calculated repeatedly, which results in high computation cost. To overcome these challenges, an efficient aseismic optimization design procedure for engineering application is developed. In this procedure, the model of optimization problem is established based on time history analysis (THA). Gaussian process regression (GPR) surrogate models are employed to predict the values of the objective and constraint functions. The expected improvement (EI) and constrained expected improvement (CEI) criteria are adopted to update the training sample set and obtain the optimal solution. Then, two examples are presented to validate the effectiveness and efficiency of this method in optimization problems of structures under earthquake loads. Finally, it is applied to optimizations of a MSCS and a mega frame structure (MFS), respectively. The response and cost of the optimized structures are reduced, and the MSCS shows better earthquake resistant capacities.

Batch sequential adaptive designs for global optimization

Efficient global optimization (EGO) is one of the most popular sequential adaptive design (SAD) methods for expensive black-box optimization problems. A well-recognized weakness of the original EGO in complex computer experiments is that it is serial, and hence the modern parallel computing techniques cannot be utilized to speed up the running of simulator experiments. For those multiple points EGO methods, the heavy computation and points clustering are the obstacles. In this work, a novel batch SAD method, named “Accelerated EGO”, is forwarded by using a refined sampling/importance resampling (SIR) method to search the points with large expected improvement (EI) values. The computation burden of the new method is much lighter, and the points clustering is also avoided. The efficiency of the proposed batch SAD is validated by nine classic test functions with dimension from 2 to 12. The empirical results show that the proposed algorithm indeed can parallelize original EGO, and gain much improvement compared against the other parallel EGO algorithm especially under high-dimensional case. Additionally, the new method is applied to the hyper-parameter tuning of support vector machine (SVM) and XGBoost models in machine learning. Accelerated EGO obtains comparable cross validation accuracy with other methods and the CPU time can be reduced a lot due to the parallel computation and sampling method.

Designing staggered platelet composite structure with Gaussian process regression based Bayesian optimization

The staggered platelet composite structure, one of the most well-known examples of biomimetics, is inspired by the microstructure of nacre, where stiff mineral platelets are stacked with a small fraction of soft polymer in a brick-and-mortar style. Significant efforts have been made to establish a framework for designing a staggered platelet pattern that achieves an excellent balance of toughness and stiffness. However, because no analytical formula for accurately predicting its toughness is available because of the complexity of the failure mechanism of realistic composites, existing studies have investigated either idealized composites with simplified material properties or realistic composites designed by heuristics. In the present study, we propose a Bayesian optimization framework to design a staggered platelet structure that renders high toughness. Gaussian process regression (GPR) was adopted to model statistically the complex relationship between the shape of the staggered platelet array and the resultant toughness. The Markov chain Monte Carlo algorithm was used to determine the optimal kernel hyperparameter set for the GPR. Starting with 14 initial training data collected with uniaxial tensile tests, a GPR-based Bayesian optimization using the expected improvement (EI) acquisition function was carried out. As a result, it was possible to design a staggered platelet pattern with a toughness 12% higher than that of the best sample in the initial training set, and this improvement was achieved after five iterations of our optimization cycle. As this optimization framework does not require any material theories and models, this process can be easily adapted and applied to various other material optimization problems based on a limited set of experiments or computational simulations.