Multi-objective Bayesian Optimization Research Articles

Structural optimization of serpentine channel water-cooled plate for lithium-ion battery modules based on multi-objective Bayesian optimization algorithm

Maintaining the battery within its optimal operating temperature range while preventing thermal runaway is crucial. Serpentine channel water-cooled plate (SCWCP) has been widely employed in battery pack cooling. The challenge lies in enhancing the cooling efficiency of SCWCP while minimizing energy consumption. Due to the high efficiency and robustness of the multi-objective Bayesian optimization (MOBO), it is employed to systematically optimize the SCWCP for lithium batteries. The width, depth, and turning radius of the serpentine flow channels are optimized to minimize both the maximum battery module temperature (Tmax) and the pumping power (PP) of the SCWCP. The MOBO process integrates structural parameter adjustments into Computational Fluid Dynamics (CFD) simulations through an automated iterative approach. Subsequently, a Pareto front is generated based on Tmax and PP, and the K-means clustering algorithm identifies four design solutions with different performance orientations. Compared with the initial design, Tmax of the optimal design decreases slightly, but with a reduction in PP of 71 %. Compared to other evolutionary algorithms, the MOBO algorithm exhibits superior computational efficiency in providing an optimal design solution set at a lower computational cost.

An efficient parallel multi-fidelity multi-objective Bayesian optimization method and application to 3-stage axial compressor with 144 variables

Parallel infill sampling is a promising approach to improve the efficiency of multi-fidelity multi-objective Bayesian optimization (MOBO). In existing literature, the number of infill samples per iteration is typically limited to 10. Additionally, the application of the multi-fidelity MOBO method in engineering optimization designs with over 100 variables is rare. To that end, a novel generalized expected improvement matrix (GEIM) criterion is proposed by using generalized reference values for the element in expected improvement matrix. Parallel infill sampling strategy based on GEIM is developed and incorporated into the multi-fidelity MOBO framework. The distinct feature of the proposed method is that the number of infill samples at each iteration can be significantly larger than existing methods (i.e. as much as 100), so as to further enhance the optimization efficiency. Empirical experiment results on analytic problems show that the proposed parallel multi-fidelity MOBO method is highly competitive in comparison with existing methods. To address the curse of dimensionality in optimizing multi-stage axial compressor, an efficient modeling method for the Hierarchical Kriging (HK) is incorporated. The HK predictor for the efficiency of the 3-stage compressor with 144 variables is built in 16.71 s with acceptable accuracy. Remarkable improvements over the two objectives of the 3-stage compressor with 144 variables are achieved simultaneously within 870 high-fidelity CFD simulations.

Comparative Study of Multi‐objective Bayesian Optimization and NSGA‐III based Approaches for Injection Molding Process

AbstractInjection molding is a prevalent method for producing plastic components, yet determining the ideal process parameters has predominantly relied on heuristic approaches. In this research, a data‐driven injection molding process optimization framework is developed to simultaneously minimize warpage, cycle time, and clamping force. Employing multi‐objective Bayesian optimization (MBO), the framework is applied to a fan blade model for verification. Incorporating those objectives enables the selection of an injection molding machine with the proper maximum clamping force within acceptable tolerance and production time. Furthermore, non‐moldable regimes are considered in design space, which allows less prior knowledge for setting design space. The optimized results are compared with those obtained using NSGA‐III, a well‐established genetic algorithm‐based optimization technique, as a benchmark. The optimization frameworks produce a Pareto front for the three‐dimensional outputs, revealing distinct trade‐off relationships between cycle time and warpage, as well as clamping force and warpage. The MBO framework demonstrates a superior Pareto front compared to NSGA‐III when utilizing a limited data set, underscoring its benefits in scenarios where costly simulations or experiments are necessary. These findings are anticipated to contribute to the optimization of manufacturing processes, ultimately enhancing productivity in real‐world industries.

A deep learning-aided multi-objective optimization of a downstream process for production of monoclonal antibody products

Monoclonal antibodies (mAbs) have emerged as the dominant class of therapeutic moieties due to their superior therapeutic potential and safety profile. Downstream processing plays a pivotal role in ensuring the purity, yield, and quality of these therapeutic products. The biopharmaceutical industry generates a significant amount of data, making it an attractive target for creating machine learning (ML) and deep learning (DL) applications. This paper reports a DL-based 2D-convolutional neural network (2D-CNN) to predict: (i) Protein A mAb elute concentration, (ii) Cation exchange (CEX) mAb elute concentration, (iii) % Acidic variant, (iv) Aggregate (%), and (v) % Basic variant from process data that is routinely collected during biopharma manufacturing, including Protein A elution UV, CEX elution UV, CEX elution conductivity, and post viral inactivation pH profiles. Further, interpretation of predicted outputs using the SHAPLEY technique and optimization through multi-objective Bayesian optimization methods have been attempted so as to match the biosimilar to the reference molecule with respect to the critical quality attributes. It is observed that the proposed model outperforms other published approaches in prediction. The optimum results were experimentally validated by performing three consecutive runs, and a mean percentage deviation of less than 3 % was reported. The proposed approach demonstrates the feasibility of real-time prediction and optimization in biopharmaceutical product manufacturing using CNN-aided modeling and multi-objective weighted sum optimization.

A general multi-objective Bayesian optimization framework for the design of hybrid schemes towards adaptive complex flow simulations

Achieving accuracy with underresolved simulation of complex compressible flows with multiscale flow structures is a challenge. Either the numerical dissipation or the resolution and thereby the numerical cost is impractically high. Also, in the design of numerical solvers, the application of a solver for specific flow classes is balanced by robustness allowing the study of a broad range of flows. In this study, we propose a hybrid fifth-order targeted essentially non-oscillatory (TENO5)-based scheme tailored to optimally simulate compressible flows with underresolved dilatational and vortical multiscale structures. For optimal design, three data-driven objectives are defined. A novel objective that derives from the numerical dissipation rate analysis is a key element to deal with underresolved complex flows in practical applications. The optimization process employs a multi-objective Bayesian optimization framework with an expected hypervolume improvement and three flow configurations representative for a broad range of two- and three-dimensional flows with genuine and non-genuine subgrid scales. The optimized hybrid scheme is validated by comparing with shock-capturing schemes of the weighted essentially non-oscillatory (WENO)- and TENO- families with flows of complex shock interactions, Kelvin-Helmholtz instabilities, shock-vortex interactions, vortical and turbulent flows

CFD-based multi-objective Bayesian optimization approach for compact and efficient hydrogen production in steam reforming reactor

This paper introduces a multi-objective Bayesian optimization approach for concurrently optimizing the size and productivity of a steam methane reforming reactor. Seven design variables, resulting in a total of 3,929,310 possible combinations, were explored, leading to three Pareto optimal designs after 100 iterations. Parametric studies were conducted first, and the results of Bayesian optimization were suggested with initial samples obtained via Latin Hypercube Sampling. The Pareto optimal designs revealed significant reduction in size and improvement in productivity. Furthermore, adjusting operating conditions, specifically the steam-to-carbon ratio, showed additional improvements of productivity. This study can contribute to efficiently proposing Pareto optimal designs with potential applications in hydrogen supply for compact and highly efficient systems.

Experimentally validated inverse design of multi-property Fe-Co-Ni alloys

This study presents a machine learning (ML) framework aimed at accelerating the discovery of multi-property optimized Fe-Ni-Co alloys, addressing the time-consuming, expensive, and inefficient nature of traditional methods of material discovery, development, and deployment. We compiled a detailed heterogeneous database of the magnetic, electrical, and mechanical properties of Fe-Co-Ni alloys, employing a novel ML-based imputation strategy to address gaps in property data. Leveraging this comprehensive database, we developed predictive ML models using tree-based and neural network approaches for optimizing multiple properties simultaneously. An inverse design strategy, utilizing multi-objective Bayesian optimization (MOBO), enabled the identification of promising alloy compositions. This approach was experimentally validated using high-throughput methodology, highlighting alloys such as Fe66.8Co28Ni5.2 and Fe61.9Co22.8Ni15.3, which demonstrated superior properties. The predicted properties data closely matched experimental data within 14% accuracy. Our approach can be extended to a broad range of materials systems to predict novel materials with an optimized set of properties.

Physics-constrained multi-objective bayesian optimization to accelerate 3d printing of thermoplastics

The printing outcome of vat photopolymerization (VPP) of thermoplastics largely depends on physicochemical properties of monomers and their compositions in resins, which also greatly determine the material properties, e.g., tensile strength (σT) and toughness (UT)and phase transition temperature (Tg). A methodology for optimizing the resin formulation is of paramount importance in realizing highly printable thermoplastics with balanced σT/UT and target Tg while remaining largely underexplored. Herein, we introduce a multi-objective Bayesian optimization (MOBO) algorithm under two physics informed constraints (printability and Tg) to optimize two conflicting properties: σT and UT. The two constraints are formulated as two machine learning (ML) models, which are trained with weight ratios of the six monomers and physics informed (PI) descriptors derived from their physiochemical parameters. Dimensional reduction analysis reveals that the algorithm avoids recommendation of the monomer ratios that do not pass the two constraints. The printing failure rate is reduced from 16% in the background experiments to 3% in the recommended experiments. Within only 36 iterations (72 samples), the MOBO algorithm successfully identifies five sets of ratios leading to Pareto optimal of σT and UT. Due to the constraint in Tg they show appropriate Tg for shape memory application. The partial dependence analysis indicates that σT and UT depend on both the ratios and physiochemical features of the monomers. These results underscore capability of such a smart decision-making algorithm—with constraints that are not fully understood but can be informed by prior knowledge—in planning the experiments from the vast design space, thus holding a great promise for broader applications in materials design and manufacturing.

Are You Concerned about Limited Function Evaluations: Data-Augmented Pareto Set Learning for Expensive Multi-Objective Optimization

Optimizing multiple conflicting black-box objectives simultaneously is a prevalent occurrence in many real-world applications, such as neural architecture search, and machine learning. These problems are known as expensive multi-objective optimization problems (EMOPs) when the function evaluations are computationally or financially costly. Multi-objective Bayesian optimization (MOBO) offers an efficient approach to discovering a set of Pareto optimal solutions. However, the data deficiency issue caused by limited function evaluations has posed a great challenge to current optimization methods. Moreover, most current methods tend to prioritize the quality of candidate solutions, while ignoring the quantity of promising samples. In order to tackle these issues, our paper proposes a novel multi-objective Bayesian optimization algorithm with a data augmentation strategy that provides ample high-quality samples for Pareto set learning (PSL). Specifically, it utilizes Generative Adversarial Networks (GANs) to enrich data and a dominance prediction model to screen out high-quality samples, mitigating the predicament of limited function evaluations in EMOPs. Additionally, we adopt the regularity model to expensive multi-objective Bayesian optimization for PSL. Experimental results on both synthetic and real-world problems demonstrate that our algorithm outperforms several state-of-the-art and classical algorithms.

Multi-Objective Bayesian Optimization with Active Preference Learning

There are a lot of real-world black-box optimization problems that need to optimize multiple criteria simultaneously. However, in a multi-objective optimization (MOO) problem, identifying the whole Pareto front requires the prohibitive search cost, while in many practical scenarios, the decision maker (DM) only needs a specific solution among the set of the Pareto optimal solutions. We propose a Bayesian optimization (BO) approach to identifying the most preferred solution in the MOO with expensive objective functions, in which a Bayesian preference model of the DM is adaptively estimated by an interactive manner based on the two types of supervisions called the pairwise preference and improvement request. To explore the most preferred solution, we define an acquisition function in which the uncertainty both in the objective function and the DM preference is incorporated. Further, to minimize the interaction cost with the DM, we also propose an active learning strategy for the preference estimation. We empirically demonstrate the effectiveness of our proposed method through the benchmark function optimization and the hyper-parameter optimization problems for machine learning models.

FlexiBO: A Decoupled Cost-Aware Multi-objective Optimization Approach for Deep Neural Networks (Abstract Reprint)

The design of machine learning systems often requires trading off different objectives, for example, prediction error and energy consumption for deep neural networks (DNNs). Typically, no single design performs well in all objectives; therefore, finding Pareto-optimal designs is of interest. The search for Pareto-optimal designs involves evaluating designs in an iterative process, and the measurements are used to evaluate an acquisition function that guides the search process. However, measuring different objectives incurs different costs. For example, the cost of measuring the prediction error of DNNs is orders of magnitude higher than that of measuring the energy consumption of a pre-trained DNN as it requires re-training the DNN. Current state-of-the-art methods do not consider this difference in objective evaluation cost, potentially incurring expensive evaluations of objective functions in the optimization process. In this paper, we develop a novel decoupled and cost-aware multi-objective optimization algorithm, which we call Flexible Multi-Objective Bayesian Optimization (FlexiBO) to address this issue. For evaluating each design, FlexiBO selects the objective with higher relative gain by weighting the improvement of the hypervolume of the Pareto region with the measurement cost of each objective. This strategy, therefore, balances the expense of collecting new information with the knowledge gained through objective evaluations, preventing FlexiBO from performing expensive measurements for little to no gain. We evaluate FlexiBO on seven state-of-the-art DNNs for image recognition, natural language processing (NLP), and speech-to-text translation. Our results indicate that, given the same total experimental budget, FlexiBO discovers designs with 4.8% to 12.4% lower hypervolume error than the best method in state-of-the-art multi-objective optimization.

Preference-Aware Constrained Multi-Objective Bayesian Optimization (Student Abstract)

We consider the problem of constrained multi-objective optimization over black-box objectives, with user-defined preferences, with a largely infeasible input space. Our goal is to approximate the optimal Pareto set from the small fraction of feasible inputs. The main challenges include huge design space, multiple objectives, numerous constraints, and rare feasible inputs identified only through expensive experiments. We present PAC-MOO, a novel preference-aware multi-objective Bayesian optimization algorithm to solve this problem. It leverages surrogate models for objectives and constraints to intelligently select the sequence of inputs for evaluation to achieve the target goal.

Fin-Bayes: A Multi-Objective Bayesian Optimization Framework for Soft Robotic Fingers.

Computational design is a critical tool to realize the full potential of Soft Robotics, maximizing their inherent benefits of high performance, flexibility, robustness, and safe interaction. Practically, computational design entails a rapid iterative search process over a parameterized design space, with assessment using (frequently) computational modeling and (more rarely) physical experimentation. Bayesian approaches work well for these expensive-to-analyze systems and can lead to efficient exploration of design space than comparative algorithms. However, such computational design typically entails weaknesses related to a lack of fidelity in assessment, a lack of sufficient iterations, and/or optimizing to a singular objective function. Our work directly addresses these shortcomings. First, we harness a sophisticated nonlinear Finite Element Modeling suite that explicitly considers geometry, material, and contact nonlinearity to perform rapid accurate characterization. We validate this through extensive physical testing using an automated test rig and printed robotic fingers, providing far more experimental data than that reported in the literature. Second, we explore a significantly larger design space than comparative approaches, with more free variables and more opportunity to discover novel, high performance designs. Finally, we use a multiobjective Bayesian optimizer that allows for the identification of promising trade-offs between two critical objectives, compliance and contact force. We test our framework on optimizing Fin Ray grippers, which are ubiquitous throughout research and industry due to their passive compliance and durability. Results demonstrate the benefits of our approach, allowing for the optimization and identification of promising gripper designs within an extensive design space, which are then 3D printed and usable in reality.

Leveraging trust for joint multi-objective and multi-fidelity optimization

In the pursuit of efficient optimization of expensive-to-evaluate systems, this paper investigates a novel approach to Bayesian multi-objective and multi-fidelity (MOMF) optimization. Traditional optimization methods, while effective, often encounter prohibitively high costs in multi-dimensional optimizations of one or more objectives. Multi-fidelity approaches offer potential remedies by utilizing multiple, less costly information sources, such as low-resolution approximations in numerical simulations. However, integrating these two strategies presents a significant challenge. We propose the innovative use of a trust metric to facilitate the joint optimization of multiple objectives and data sources. Our methodology introduces a modified multi-objective (MO) optimization policy incorporating the trust gain per evaluation cost as one of the objectives of a Pareto optimization problem. This modification enables simultaneous MOMF optimization, which proves effective in establishing the Pareto set and front at a fraction of the cost. Two specific methods of MOMF optimization are presented and compared: a holistic approach selecting both the input parameters and the fidelity parameter jointly, and a sequential approach for benchmarking. Through benchmarks on synthetic test functions, our novel approach is shown to yield significant cost reductions—up to an order of magnitude compared to pure MO optimization. Furthermore, we find that joint optimization of the trust and objective domains outperforms sequentially addressing them. We validate our findings with the specific use case of optimizing particle-in-cell simulations of laser-plasma acceleration, highlighting the practical potential of our method in the Pareto optimization of highly expensive black-box functions. Implementation of the methods in existing Bayesian optimization frameworks is straightforward, with immediate extensions e.g. to batch optimization possible. Given their ability to handle various continuous or discrete fidelity dimensions, these techniques have wide-ranging applicability in tackling simulation challenges across various scientific computing fields such as plasma physics and fluid dynamics.

Fair and green hyperparameter optimization via multi-objective and multiple information source Bayesian optimization

AbstractIt has been recently remarked that focusing only on accuracy in searching for optimal Machine Learning models amplifies biases contained in the data, leading to unfair predictions and decision supports. Recently, multi-objective hyperparameter optimization has been proposed to search for Machine Learning models which offer equally Pareto-efficient trade-offs between accuracy and fairness. Although these approaches proved to be more versatile than fairness-aware Machine Learning algorithms—which instead optimize accuracy constrained to some threshold on fairness—their carbon footprint could be dramatic, due to the large amount of energy required in the case of large datasets. We propose an approach named FanG-HPO: fair and green hyperparameter optimization (HPO), based on both multi-objective and multiple information source Bayesian optimization. FanG-HPO uses subsets of the large dataset to obtain cheap approximations (aka information sources) of both accuracy and fairness, and multi-objective Bayesian optimization to efficiently identify Pareto-efficient (accurate and fair) Machine Learning models. Experiments consider four benchmark (fairness) datasets and four Machine Learning algorithms, and provide an assessment of FanG-HPO against both fairness-aware Machine Learning approaches and two state-of-the-art Bayesian optimization tools addressing multi-objective and energy-aware optimization.

Evolutionary Multi-Objective Bayesian Optimization Based on Multisource Online Transfer Learning

Evolutionary Multi-Objective Bayesian Optimization Based on Multisource Online Transfer Learning

Multi-objective optimization of ViT architecture for efficient brain tumor classification

This study presents an advanced approach to optimizing the Vision Transformer (ViT) network for brain tumor classification in 2D MRI images, utilizing Bayesian Multi-Objective (BMO) optimization techniques. Rather than merely addressing the limitations of the standard ViT model, our objective was to enhance its overall efficiency and effectiveness. The application of BMO enabled us to fine-tune the architectural parameters of the ViT network, resulting in a model that was not only twice as fast but also four times smaller in size compared to the original. In terms of performance, the optimized ViT model achieved notable improvements, with a 1.48 % increase in validation accuracy, a 3.23 % rise in the F1-score, and a 3.36 % improvement in precision. These substantial enhancements highlight the potential of integrating BMO with visual transformer-based models, suggesting a promising direction for future research in achieving high efficiency and accuracy in complex classification tasks.

Bayesian multi-objective optimization of process design parameters in constrained settings with noise: an engineering design application

Bayesian multi-objective optimization of process design parameters in constrained settings with noise: an engineering design application

High-Dimensional Multi-Objective Bayesian Optimization With Block Coordinate Updates: Case Studies in Intelligent Transportation System

Many transportation system problems can be formulated as high-dimensional expensive multi-objective problems. They are challenging for Gaussian process-based Bayesian optimization methods to find the Pareto fronts due to the curse of dimensionality and the boundary issue in the acquisition function optimization. This paper presents a multi-objective Bayesian optimization method with block coordinate updates, Block-MOBO, to solve high-dimensional expensive multi-objective problems. Block-MOBO first partitions the decision variable space into different blocks, each of which includes a low-dimensional multi-objective problem. At each iteration, one block is considered and the decision variables not in this block are approximated by context-vector generation embedded with the Pareto prior knowledge thus promoting convergence. To tackle the boundary issue, we present <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon$</tex-math> </inline-formula> -greedy acquisition function in a Bayesian and multi-objective fashion, which recommends candidates either from the exploitation-exploration trade-off perspective or with probability <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon$</tex-math> </inline-formula> from the Pareto dominance relationship perspective. We verify the effectiveness of Block-MOBO by comparing it with other multi-objective Bayesian methods on two real-world optimization problems in transportation system and three multi-objective synthetic test suites. The experimental results show that Block-MOBO can find more evenly distributed and non-dominated solutions in the whole search space with lower complexity compared with other state-of-the-art approaches. Our analyses illustrate that block coordinate updates and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon$</tex-math> </inline-formula> -greedy acquisition function contribute to computational complexity reduction and convergence-diversity trade-offs, respectively.

AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning

<scp>AutoPEFT</scp>: Automatic Configuration Search for Parameter-Efficient Fine-Tuning