Black-box Optimization Research Articles

Self-learning salp swarm algorithm for global optimization and its application in multi-layer perceptron model training

Optimization problems are common across various fields, and one effective solution is the swarm intelligence algorithm.It is essential for the algorithm to deliver high-quality solutions for problems with varying characteristics. However, most existing swarm intelligence rely on fixed and monotonic search strategies, which limits their ability to handle the diverse and complex situations encountered when solving real-world optimization problems with unknown fitness landscapes. To extend the applicability of swarm intelligence and thus offer users an efficient black-box optimizer for various applications, a novel self-learning mechanism is proposed and applied to the Salp Swarm Algorithm (SSA) to develop the self-learning salp swarm algorithm (SLSSA) in this paper. In SLSSA, four distinct search strategies, including a novel multiple food sources search strategy, are adopted to strengthen the search agents’ abilities to conquer various difficulties in the search space. To improve the efficiency of the search process, the self-learning strategy dynamically determines the execution probability of each search strategy according to the quality of solutions it produced previously. Moreover, a parameter setting method is proposed in this paper, which eliminates the need for a trial-and-error approach and allows for straightforward configuration of the parameters that optimize the performance of SLSSA. In comparison with several highly regarded state-of-the-art peer algorithms, the performance of SLSSA in solving the CEC2014 benchmark functions was thoroughly examined. Subsequently, SLSSA was applied to train multi-layer perceptron classifiers and test on the UCI machine-learning datasets. The experimental results and analysis on benchmark functions and multi-layer perceptron classifier training problems demonstrate that SLSSA outperforms the competing algorithms in terms of solution accuracy, stability, and overall convergence speed. Moreover, computational time comparisons reveal that SLSSA achieves significant performance improvement with only a marginal increase in time cost compared to the original SSA.

Discretization-independent node-based shape optimization with the Vertex Morphing method using design variable scaling

AbstractThe Vertex Morphing method is a node-based shape parameterization that uses an explicit filtering approach to regularize the optimization problem and generate smooth shapes. It has been successfully applied to shape optimization problems of industrial size in recent years. This work investigates in detail how irregular discretizations, design surface boundaries, and complex geometries can influence the progress of a gradient-based optimization using the standard Vertex Morphing formulation. A sensitivity weighting approach based on the available shape morphing functions is presented, which eliminates all of the aforementioned influences. Subsequently, a design variable scaling strategy is developed that transforms the optimization problem into an alternative design space and allows the use of arbitrary, even highly irregular surface discretizations in combination with black-box optimization algorithms for shape optimization with the Vertex Morphing method. Illustrative academic examples and an application case of an additively manufactured part are presented to support the work.

Distributed Evolution Strategies With Multi-Level Learning for Large-Scale Black-Box Optimization

Distributed Evolution Strategies With Multi-Level Learning for Large-Scale Black-Box Optimization

Station-Keeping Control of Stratospheric Balloons Based on Simultaneous Optimistic Optimization in Dynamic Wind

Stratospheric balloons serve as cost-effective platforms for wireless communication. However, these platforms encounter challenges stemming from their underactuation in the horizontal plane. Consequently, controllers must continually identify favorable wind conditions to optimize station-keeping performance while managing energy consumption. This study presents a receding horizon controller based on wind and balloon models. Two neural networks, PredRNN and ResNet, are utilized for short-term wind field forecast. Additionally, an online receding horizon controller, based on simultaneous optimistic optimization (SOO), is developed for action sequence planning and adapted to accommodate various constraints, which is especially suitable due to its gradient-free nature, high efficiency, and effectiveness in black-box function optimization. A reward function is formulated to balance power consumption and station-keeping performance. Simulations conducted across diverse positions and dates demonstrate the superior performance of the proposed method compared with traditional greedy and A* algorithms.

Tail Bounds on the Runtime of Categorical Compact Genetic Algorithm.

The majority of theoretical analyses of evolutionary algorithms in the discrete domain focus on binary optimization algorithms, even though black-box optimization on the categorical domain has a lot of practical applications. In this paper, we consider a probabilistic model-based algorithm using the family of categorical distributions as its underlying distribution and set the sample size as two. We term this specific algorithm the categorical compact genetic algorithm (ccGA). The ccGA can be considered as an extension of the compact genetic algorithm (cGA), which is an efficient binary optimization algorithm. We theoretically analyze the dependency of the number of possible categories K, the number of dimensions D, and the learning rate η on the runtime. We investigate the tail bound of the runtime on two typical linear functions on the categorical domain: categorical OneMax (COM) and KVAL. We derive that the runtimes on COM and KVAL are O(Dln(DK)/η) and Θ(DlnK/η) with high probability, respectively. Our analysis is a generalization for that of the cGA on the binary domain.

Adapting the population size in CMA-ES using nearest-better clustering method for multimodal optimization

The covariance matrix adaptation evolution strategy (CMA-ES) is one of the advanced algorithms for solving continuous single-objective optimization. In this algorithm, the population size is an important parameter which represents the number of candidate points generated in each iteration. A larger population size allows for better exploration of the search space and is typically required for handling multimodal functions, while a smaller population size facilitates quicker convergence. Adapting the population size in this algorithm raises a question about how to increase/decrease/keep it in an efficient way to solve specific problems. In this paper, we propose a novel approach for this purpose based on the information of niches observed during the evolution process. Firstly, the nearest-better clustering (NBC) technique is employed to detect the number of niches in each iteration, then the information will be used to adapt the population size accordingly. Additionally, a quasi-Newton method, which serves as a local search, can be incorporated into the algorithm in order to speed it up. We evaluate the effectiveness of our method through numerical simulations on multimodal test functions, the black-box optimization benchmarking (BBOB) noiseless testbed, and the CEC 2014 and CEC 2017 benchmark testing suites.

CMA-ES with Learning Rate Adaptation

The covariance matrix adaptation evolution strategy (CMA-ES) is one of the most successful methods for solving continuous black-box optimization problems. A practically useful aspect of CMA-ES is that it can be used without hyperparameter tuning. However, the hyperparameter settings still have a considerable impact on performance, especially for difficult tasks, such as solving multimodal or noisy problems. This study comprehensively explores the impact of learning rate on CMA-ES performance and demonstrates the necessity of a small learning rate by considering ordinary differential equations. Thereafter, it discusses the setting of an ideal learning rate. Based on these discussions, we develop a novel learning rate adaptation mechanism for CMA-ES that maintains a constant signal-to-noise ratio. Additionally, we investigate the behavior of CMA-ES with the proposed learning rate adaptation mechanism through numerical experiments and compare the results with those obtained for CMA-ES with a fixed learning rate and with population size adaptation. The results show that CMA-ES with the proposed learning rate adaptation works well for multimodal and/or noisy problems without extremely expensive learning rate tuning.

Cooperative coevolution for non-separable large-scale black-box optimization: Convergence analyses and distributed accelerations

Given the ubiquity of non-separable optimization problems in real worlds, in this paper we analyze and extend the large-scale version of the well-known cooperative coevolution (CC), a divide-and-conquer black-box optimization framework, on non-separable functions. First, we reveal empirical reasons of when decomposition-based methods are preferred or not in practice on some non-separable large-scale problems, which have not been clearly pointed out in many previous CC papers. Then, we formalize CC to a continuous-game model via simplification, but without losing its essential property. Different from previous evolutionary game theory for CC, our new model provides a much simpler but useful viewpoint to analyze its convergence, since only the pure Nash equilibrium concept is needed and more general fitness landscapes can be explicitly considered. Based on convergence analyses, we propose a hierarchical decomposition strategy for better generalization, as for any decomposition, there is a risk of getting trapped into a suboptimal Nash equilibrium. Finally, we use powerful distributed computing to accelerate it under the recent multi-level learning framework, which combines the fine-tuning ability from decomposition with the invariance property of CMA-ES. Experiments on a set of high-dimensional test functions validate both its search performance and scalability (w.r.t. CPU cores) on a clustering computing platform with 400 CPU cores.

Comparative study of optimization methods for building energy consumption and daylighting performance

When striving for sustainable development, designers encounter the formidable task of integrating the advantages of daylighting into energy-efficient designs. The process of investigating a multitude of design variables can be arduous and time-intensive. However, computational simulations provide indispensable insights into forecasting performance. A considerable number of these design challenges do not have explicit mathematical solutions; thus, black-box optimization techniques are favored for the purpose of determining the most effective design solutions. Popular meta-heuristic optimization algorithms, including the Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Whale Optimization Algorithm (WOA), Sine Cosine Algorithm (SCA), Grey Wolf Optimizer (GWO), Ant Lion Optimizer (ALO), and Multi-Verse Optimizer (MVO), are described and thoroughly analyzed in this article. In addition, the present study run empirical studies to compare these optimization algorithms by applying them to tackle five distinct daylighting and energy optimization problems associated with enhancing building performance. In the test cases, the present study results demonstrate that the PSO and MVO algorithms exhibit better performance compared to the GA and WOA algorithms. The selection of the most suitable algorithm is contingent upon the distinctive attributes of the problem, and the intricacy of the optimization landscape.

Tensor product algorithms for inference of contact network from epidemiological data

We consider a problem of inferring contact network from nodal states observed during an epidemiological process. In a black-box Bayesian optimisation framework this problem reduces to a discrete likelihood optimisation over the set of possible networks. The cardinality of this set grows combinatorially with the number of network nodes, which makes this optimisation computationally challenging. For each network, its likelihood is the probability for the observed data to appear during the evolution of the epidemiological process on this network. This probability can be very small, particularly if the network is significantly different from the ground truth network, from which the observed data actually appear. A commonly used stochastic simulation algorithm struggles to recover rare events and hence to estimate small probabilities and likelihoods. In this paper we replace the stochastic simulation with solving the chemical master equation for the probabilities of all network states. Since this equation also suffers from the curse of dimensionality, we apply tensor train approximations to overcome it and enable fast and accurate computations. Numerical simulations demonstrate efficient black-box Bayesian inference of the network.

Large language models for automatic equation discovery of nonlinear dynamics

Equation discovery aims to directly extract physical laws from data and has emerged as a pivotal research domain in nonlinear systems. Previous methods based on symbolic mathematics have achieved substantial advancements, but often require handcrafted representation rules and complex optimization algorithms. In this paper, we introduce a novel framework that utilizes natural language-based prompts to guide large language models (LLMs) in automatically extracting governing equations from data. Specifically, we first utilize the generation capability of LLMs to generate diverse candidate equations in string form and then evaluate the generated equations based on observations. The best equations are preserved and further refined iteratively using the reasoning capacity of LLMs. We propose two alternately iterated strategies to collaboratively optimize the generated equations. The first strategy uses LLMs as a black-box optimizer to achieve equation self-improvement based on historical samples and their performance. The second strategy instructs LLMs to perform evolutionary operations for a global search. Experiments are conducted on various nonlinear systems described by partial differential equations, including the Burgers equation, the Chafee–Infante equation, and the Navier–Stokes equation. The results demonstrate that our framework can discover correct equations that reveal the underlying physical laws. Further comparisons with state-of-the-art models on extensive ordinary differential equations showcase that the equations discovered by our framework possess physical meaning and better generalization capability on unseen data.

Revealing Atomic Structure of Hybrid Octacalcium-Phosphate Derivative.

Octacalcium phosphate (OCP), which is one of the bioactive calcium phosphates, can incorporate various organic molecules in its crystal lattice, forming the organic-inorganic hybrid derivatives. However, detailed atomic arrangements of OCP hybridized with organic molecules such as dicarboxylate are still unknown, although many years have passed since the first discovery of the materials systems. In the present study, some black-box optimization methods combined with first-principles calculations were used to theoretically identify the most stable atomic structure of the OCP with the incorporation of malonate ions as a typical case study. The results showed that the calculated interplanar spacing on the (100) plane of the most stable structure agrees well with experimental data, by taking account of implicit solvent of aqueous solution. An underlying mechanism that realizes the bridging feature of the incorporated malonate ions between the apatitic layers is also discussed. The present methodology can pave the way to accurately explore reliable atomic structures of such complicated organic-inorganic hybrid biomaterials with high structural degrees of freedom.

RetinaViT: Efficient Visual Backbone for Online Video Streams.

In online video understanding, which has a wide range of real-world applications, inference speed is crucial. Many approaches involve frame-level visual feature extraction, which often represents the biggest bottleneck. We propose RetinaViT, an efficient method for extracting frame-level visual features in an online video stream, aiming to fundamentally enhance the efficiency of online video understanding tasks. RetinaViT is composed of efficiently approximated Transformer blocks that only take changed tokens (event tokens) as queries and reuse the already processed tokens from the previous timestep for the others. Furthermore, we restrict keys and values to the spatial neighborhoods of event tokens to further improve efficiency. RetinaViT involves tuning multiple parameters, which we determine through a multi-step process. During model training, we randomly vary these parameters and then perform black-box optimization to maximize accuracy and efficiency on the pre-trained model. We conducted extensive experiments on various online video recognition tasks, including action recognition, pose estimation, and object segmentation, validating the effectiveness of each component in RetinaViT and demonstrating improvements in the speed/accuracy trade-off compared to baselines. In particular, for action recognition, RetinaViT built on ViT-B16 reduces inference time by approximately 61.9% on the CPU and 50.8% on the GPU, while achieving slight accuracy improvements rather than degradation.

Improving derivative-free optimization algorithms through an adaptive sampling procedure

Black-box optimization plays a pivotal role in addressing complex real-world problems where the underlying mathematical model is unknown or expensive to evaluate. In this context, this work presents a method to enhance the performance of derivative-free optimization algorithms by integrating an adaptive sampling process. The proposed methodology aims to overcome the limitations of traditional methods by intelligently guiding the search towards promising regions of the search space. To achieve this, we utilize machine learning models, which effectively substitute first principles models. Furthermore, we employ the error maximization approach to steer the exploration towards areas where the surrogate model deviates significantly from the true model. Moreover, we involve a heuristic method, an adaptive sampling procedure, that repeats calls to a widely-used derivative-free optimization algorithm, SNOBFIT, allowing for the creation of new and improved surrogate models. To evaluate the efficiency of the proposed method, we conduct a comparative analysis across a benchmark set of 776 continuous problems. Our findings indicate that our approach successfully solved 93% of the problems. Notably, for larger problems, our method outperformed the standard SNOBFIT algorithm by achieving a 19% increase in problem-solving rate, and when, we introduced an additional termination criterion to enhance computational efficiency, the proposed method achieved a 31% time reduction compared to SNOBFIT.

Landscape Analysis for Surrogate Models in the Evolutionary Black-Box Context.

Surrogate modeling has become a valuable technique for black-box optimization tasks with expensive evaluation of the objective function. In this paper, we investigate the relationships between the predictive accuracy of surrogate models, their settings, and features of the black-box function landscape during evolutionary optimization by the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) state-of-the-art optimizer for expensive continuous black-box tasks. This study aims to establish the foundation for specific rules and automated methods for selecting and tuning surrogate models by exploring relationships between landscape features and model errors, focusing on the behavior of a specific model within each generation in contrast to selecting a specific algorithm at the outset. We perform a feature analysis process, identifying a significant number of non-robust features and clustering similar landscape features, resulting in the selection of 14 features out of 384, varying with input data selection methods. Our analysis explores the error dependencies of four models across 39 settings, utilizing three methods for input data selection, drawn from surrogate-assisted CMA-ES runs on noiseless benchmarks within the Comparing Continuous Optimizers framework.

Machine learning prediction of the mechanical properties of injection-molded polypropylene through X-ray diffraction analysis

ABSTRACT Predicting the mechanical properties of polymer materials using machine learning is essential for the design of next-generation of polymers. However, the strong relationship between the higher-order structure of polymers and their mechanical properties hinders the mechanical property predictions based on their primary structures. To incorporate information on higher-order structures into the prediction model, X-ray diffraction (XRD) can be used. This study proposes a strategy to generate appropriate descriptors from the XRD analysis of the injection-molded polypropylene samples, which were prepared under almost the same injection molding conditions. To this end, first, Bayesian spectral deconvolution is used to automatically create high-dimensional descriptors. Second, informative descriptors are selected to achieve highly accurate predictions by implementing the black-box optimization method using Ising machine. This approach was applied to custom-built polymer datasets containing data on homo- polypropylene and derived composite polymers with the addition of elastomers. Results show that reasonable accuracy of predictions for seven mechanical properties can be achieved using only XRD.

Using Machine Learning Methods to Assess Module Performance Contribution in Modular Optimization Frameworks.

Modular algorithm frameworks not only allow for combinations never tested in manually selected algorithm portfolios, but they also provide a structured approach to assess which algorithmic ideas are crucial for the observed performance of algorithms. In this study, we propose a methodology for analyzing the impact of the different modules on the overall performance. We consider modular frameworks for two widely used families of derivative-free black-box optimization algorithms, the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) and differential evolution (DE). More specifically, we use performance data of 324 modCMA-ES and 576 modDE algorithm variants (with each variant corresponding to a specific configuration of modules) obtained on the 24 BBOB problems for 6 different runtime budgets in 2 dimensions. Our analysis of these data reveals that the impact of individual modules on overall algorithm performance varies significantly. Notably, among the examined modules, the elitism module in CMA-ES and the linear population size reduction module in DE exhibit the most significant impact on performance. Furthermore, our exploratory data analysis of problem landscape data suggests that the most relevant landscape features remain consistent regardless of the configuration of individual modules, but the influence that these features have on regression accuracy varies. In addition, we apply classifiers that exploit feature importance with respect to the trained models for performance prediction and performance data, to predict the modular configurations of CMA-ES and DE algorithm variants. The results show that the predicted configurations do not exhibit a statistically significant difference in performance compared to the true configurations, with the percentage varying depending on the setup (from 49.1% to 95.5% for mod-CMA and 21.7% to 77.1% for DE).

Federated Many-Task Bayesian Optimization

Bayesian optimization is a powerful surrogate-assisted algorithm for solving expensive black-box optimization problems. While Bayesian optimization was developed for centralized optimization, the availability of massive distributed data has attracted increased interests in exploring federated Bayesian optimization that can use data on multiple clients without leaking the raw data. However, existing federated Bayesian optimization (FBO) approaches assume that either all clients jointly solve the same optimization task, or only one client solves one target optimization task by transferring knowledge from others in a federated way, making them unsuited for many real-world applications. In this paper, we consider FBO for the scenario where multiple related local black-box tasks associated with different clients are jointly optimized by sharing knowledge between tasks without leaking the data privacy. An efficient federated many-task Bayesian optimization framework is proposed to address not independent and identically distributed (non-IID) data while protecting the data privacy in the federated setting. A novel federated knowledge transfer paradigm is developed for dynamic many-task model aggregation according to a dissimilarity matrix. The dissimilarity is measured based on the rank of the predictions and only the hyperparameters in the local Gaussian process models are shared. In addition, a federated ensemble acquisition function is constructed by integrating the predictions of two surrogates using the global and local hyperparameters, respectively, to effectively search for the optimal solution. Experimental results show that our proposed method has reliable performance on both benchmark problems and a real machine learning problem also in the presence of non-IID data.

Optimization of low-loss, high birefringence parameters of a hollow-core anti-resonant fiber with back-propagation neural network assisted hyperplane segmentation algorithm

In engineering, optimizing parameters often involves computationally expensive tasks, especially when dealing with multi-dimensional variables and multiple performance metrics. This falls under the category of multi-objective black-box optimization. To address this, we propose two optimization algorithms for low and medium-dimensional spaces, incorporating relaxation conditions for hyperplane segmentation. For the specific parameter optimization of HC-ARF, we employed a two-stage approach. It combines a BP neural network as a surrogate model with a hyperplane separation optimization algorithm. This method efficiently optimizes both confinement loss (CL) and birefringence, using a weighted sum approach to identify their Pareto sets. We validate the effectiveness and stability of the surrogate model by comparing it with traditional optimization algorithms. Exhaustive experiments confirm the superiority of this algorithm and the results show that our optimized structure achieves impressive performance metrics, including a loss of 0.8 dB/m, a birefringence of 2.2×10−4, and a critical bending radius of 0.5 cm under optimal parameters.

Efficient trust region filter modeling strategies for computationally expensive black-box optimization

The study and application of contemporary optimization techniques considerably enhance the efficiency of chemical research and manufacturing. With the dynamic progression of modern manufacturing technologies, the emergence of numerous black-box models characterized by inaccessible mathematical formulations and high evaluation costs poses new challenges to traditional optimization methods, leading to difficulty in programming and solving. Hence, based on the trust region filter (TRF) method, we define a new framework to elevate optimization efficiency in this study for chemical systems involving computationally expensive black-box functions. Sampling size per iteration is reduced, and sampling efficiency is improved by incorporating the known data beyond the trust region to assist in constructing reduced models, which is achieved by the application of the Gaussian process. Through comparison and validation of benchmark tests and case studies, this approach demonstrates that using the Gaussian process as the reduced model can lower the number of calls to black-box functions by more than half compared to common linear and quadratic models, and convergence to first-order critical points can still be guaranteed.