Large-scale Optimization Problems Research Articles

Cooperative coevolutionary differential evolution with adjacent intensity matrix with linkage identification for large-scale optimization problems in noisy environments

Cooperative coevolutionary differential evolution with adjacent intensity matrix with linkage identification for large-scale optimization problems in noisy environments

Large-scale and cooperative graybox parallel optimization on the supercomputer Fugaku

We design, develop and analyze parallel variants of a state-of-the-art graybox optimization algorithm, namely Drils (Deterministic Recombination and Iterated Local Search), for attacking large-scale pseudo-boolean optimization problems on top of the large-scale computing facilities offered by the supercomputer Fugaku. We first adopt a Master/Worker design coupled with a fully distributed Island-based model, ending up with a number of hybrid OpenMP/MPI implementations of high-level parallel Drils versions. We show that such a design, although effective, can be substantially improved by enabling a more focused iteration-level cooperation mechanism between the core graybox components of the original serial Drils algorithm. Extensive experiments are conducted in order to provide a systematic analysis of the impact of the designed parallel algorithms on search behavior, and their ability to compute high-quality solutions using increasing number of CPU-cores. Results using up to 1024×12-cores NUMA nodes, and NK-landscapes with up to 10,000 binary variables are reported, providing evidence on the relative strength of the designed hybrid cooperative graybox parallel search.

Distributed gradient-free and projection-free algorithm for stochastic constrained optimization

Distributed stochastic zeroth-order optimization (DSZO), in which the objective function is allocated over multiple agents and the derivative of cost functions is unavailable, arises frequently in large-scale machine learning and reinforcement learning. This paper introduces a distributed stochastic algorithm for DSZO in a projection-free and gradient-free manner via the Frank-Wolfe framework and the stochastic zeroth-order oracle (SZO). Such a scheme is particularly useful in large-scale constrained optimization problems where calculating gradients or projection operators is impractical, costly, or when the objective function is not differentiable everywhere. Specifically, the proposed algorithm, enhanced by recursive momentum and gradient tracking techniques, guarantees convergence with just a single batch per iteration. This significant improvement over existing algorithms substantially lowers the computational complexity. Under mild conditions, we prove that the complexity bounds on SZO of the proposed algorithm are O(n/ϵ2)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{O}(n/\\epsilon ^{2})$\\end{document} and O(n(21ϵ))\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{O}(n(2^{\\frac{1}{\\epsilon}}))$\\end{document} for convex and nonconvex cases, respectively. The efficacy of the algorithm is verified on black-box binary classification problems against several competing alternatives.

A two-stage direction-guided evolutionary algorithm for large-scale multiobjective optimization

Large-scale multiobjective optimization problems (LSMOPs) have exponential growth in the search space as the decision variables increase, and the vast search space poses a challenge to the performance of multiobjective evolutionary algorithms (MOEAs). Many current large-scale MOEAs need to consume a large amount of computational resources to get good performance. This paper proposes a two-stage direction-guided evolutionary algorithm for large-scale multiobjective optimization (LMOEA-S2D) to balance the performance and computational resource overhead. The algorithm exploits the Pareto-optimality property of domination and the diversity-preserving property of decomposition to optimize the performance in the two stages, respectively, and designs a corresponding direction-guided mechanism to improve search efficiency. LMOEA-S2D designs global direction search and local direction search in the domination-based stage for efficient exploitation to accelerate population convergence. To promote greater population diversity, a hybrid direction search was devised to aid diversity exploration in the decomposition-based stage, and this facilitates even distribution of candidate solutions across the Pareto optimal frontier. LMOEA-S2D is compared with five state-of-the-art large-scale MOEAs on some large-scale multiobjective test suites with 100 to 5,000 decision variables. The experimental results show that LMOEA-S2D significantly outperformed all compared algorithms under limited computational resources.

A mid-range approximation method assisted by trust region strategy for aerodynamic shape optimization

This paper presents an efficient solution for high-fidelity large-scale aerodynamic shape optimization problems based on several developments in the mid-range approximation method within a trust region optimization framework. The mid-range approximation method is an iterative optimization technique that utilizes mid-range approximations to replace the physical experiments or simulations during the optimization based on the selected trust region strategy. It could transform the original optimization problem into a sequence of approximate sub-optimization problems. In this work, an improved trust region strategy is proposed to contain more optimization states with a flexible and controllable performance to suit different types of problems. A metamodel assembly technique and its gradient-enhanced version are developed to further relax the requirements of computational costs in the mid-range approximation method. Its performance is discussed through a detailed comparison of metamodel performance using a mathematical benchmark case named Vanderplaats scalable beam. The wing only of the Common Research Model is offered to the proposed method for aerodynamic shape optimization. The optimization has 1 design objective, 232 design variables, and 135 design constraints. With all constraints satisfied, the optimized configuration has a 4.85% improvement in wing drag performance. The shock region is greatly reduced and the wing pressure distribution is smooth and nearly parallel. These results show that the proposed method could achieve the design goal successfully within a reasonable computational cost for large-scale problems.

An inexact variable metric variant of incremental aggregated Forward-Backward method for the large-scale composite optimization problem

ABSTRACT This paper focuses on the large-scale composite optimization problem, which is the sum of a finite number of smooth (possibly nonconvex) functions and a convex (possibly nonsmooth) function. A common method to solve this problem is proximal incremental aggregated gradient (PIAG) method (N. D. Vanli, M. Gurbuzbalaban and A. Ozdaglar, SIAM Journal on Optimization, 2018, 28(2): 1282–1300), which exploits the gradient information from previous iterations to approximate the gradient at the current point. However, as many first-order algorithms, it may suffer from slow convergence. To address this issue, this paper proposes an inexact variable metric variant of incremental aggregated Forward-Backward (iVMFB-IAG) method. This methods incorporates a variable metric strategy to accelerate the PIAG method and introduces an inexact criterion to enhance its practicality. Under the Kurdyka-Łojasiewicz (KL) property and the uniformly bounded positive definiteness of the scaling matrix, the sequence generated by the iVMFB-IAG method converges globally to the critical point of the problem. Additionally, the convergence rate is established under the KL framework. Moreover, under certain restrictive conditions, the method exhibits local superlinear convergence. Through the numerical experiments on image reconstruction applications, we demonstrate the effectiveness of the iVMFB-IAG method by restoring two distinct images.

Feature selection by Universum embedding

Feature selection in classification is an important task in machine learning. Inspired by the success of Universum support vector machine proposed by Weston et al. on improving the classification ability of classical support vector machine, this paper considers a special type of Universum and further lets it play its role in both useful feature identification and separating hyperplane construction, aiming to improve both the feature selection ability and classification performance of Universum support vector machine. By introducing this special Universum, a redundant feature can be identified by observing whether some Universum sample is useful. In fact, we prove that by observing the dual solution of the optimization problem, useful features can be selected from a set satisfying some properties. Due to the introduction of these extra Universum samples, it needs to cope with a large-scale optimization problem. To improve the training efficiency, we modify the sequential minimal optimization algorithm and further combine it with the coordinate descent technique to solve the proposed model. Experimental results on artificial datasets, benchmark datasets, and text classification datasets demonstrate that the proposed method improves the classification performance of support vector machine and Universum support vector machine, and also has good feature selection ability.

Optimization of CO2 capture plants with surrogate model uncertainties

CO2 capture plants can help reduce the cost of deploying capture systems across the globe. However, the CO2 variability and model uncertainty represent operational challenges to capture CO2 from different sources. This work proposes a framework for analyzing the optimal plant design considering different flue gas sources. We show a methodology to generate large data sets from optimization runs using rigorous models in Aspen Plus®. The efficiency of the approach allows its application to large-scale optimization problems, with an average CPU time per run of 176 s.We additionally build surrogate models (SMs) for the capital and operating costs of the capture plants, employing an iterative procedure to generate SMs using ALAMO. We systematically reject SMs with high uncertainty in the estimated parameters. This approach results in SMs with favorable bias-variance tradeoffs, enabling their effective application to optimization problems under uncertainty, as demonstrated with a pooling problem of CO2 streams.

Efficient knowledge model for whale optimization algorithm to solve large-scale problems

In the process of digital transformation and development in various industries, there are more and more large-scale optimization problems. Currently, swarm intelligence optimization algorithms are the best method to solve such problems. However, previous experimental research has found that there is still room for improvement in the performance of using existing swarm intelligence optimization algorithms to solve such problems. To obtain the high-precision optimal value of whale optimization algorithm (WOA) for solving large-scale optimization problems, the optimization problem knowledge model is studied to guide the iterative process of WOA algorithm, and a novel whale optimization algorithm based on knowledge model guidance (KMGWOA) is proposed. First, a population update strategy based on multiple elite individuals is proposed to reduce the impact of the local optimal values, and the knowledge model to guide population update is constructed by combining the proposed population update strategy with the population update strategy based on global optimal individual. Second, a collaborative reverse learning knowledge model with multiple elite and poor individuals in the solution space is proposed to prevent long-term non-ideal region search. The above two knowledge models guide the iterative process of WOA algorithm in solving large-scale optimization problems. The performance of the KMGWOA algorithm guided by the proposed knowledge models is tested through the well-known classical test functions. The results demonstrate that the proposed KMGWOA algorithm not only has good search ability for the theoretical optimal value, but also achieves higher accuracy in obtaining the optimal value when it is difficult to obtain the theoretical optimal value. Moreover, KMGWOA algorithm has fast convergence speed and high effective iteration percentage.

A stabilised Benders decomposition with adaptive oracles for large-scale stochastic programming with short-term and long-term uncertainty

Benders decomposition with adaptive oracles was proposed to solve large-scale optimisation problems with a column-bounded block-diagonal structure, where subproblems differ only in the right-hand side and cost coefficients. Adaptive Benders reduces computational effort significantly by iteratively building inexact cutting planes and valid upper and lower bounds. However, Adaptive Benders and standard Benders may suffer severe oscillation when solving degenerate models. Therefore, we propose stabilising Adaptive Benders with the level method and adaptively selecting which subproblems to solve each iteration for more accurate information. In addition, we propose a dynamic level method to improve the robustness of stabilised Adaptive Benders by adjusting the level set each iteration. We compare stabilised Adaptive Benders with the unstabilised versions of Adaptive Benders with one subproblem solved per iteration and standard Benders on a multi-region long-term power system investment planning problem with short-term and long-term uncertainty. The problem is formulated as multi-horizon stochastic programming. Four algorithms were implemented to solve linear programming with up to 1 billion variables and 4.5 billion constraints. The computational results show that: (a) for a 1.00% convergence tolerance, the proposed stabilised method is up to 113.7 times faster than standard Benders and 2.1 times faster than unstabilised Adaptive Benders; (b) for a 0.10% convergence tolerance, the proposed stabilised method is up to 45.5 times faster than standard Benders and unstabilised Adaptive Benders cannot solve the largest instance to convergence tolerance due to severe oscillation and (c) dynamic level method makes stabilisation more robust.

Boosting scalability for large-scale multiobjective optimization via transfer weights

Large-scale multiobjective optimization problems (LSMOPs), which optimize multiple conflicting objectives with hundreds or even thousands of decision variables, demand increasing computational resources to assure satisfactory performance as the decision variables increase. Multiobjective evolutionary algorithms are naturally scalable, but as the dimension increases, the conflict between analyzing enough solutions and the limited number of function evaluations has hindered further improvements in scalability. In this paper, we first define the scalability of multiobjective evolutionary algorithms and design an indicator to quantitatively measure the scalability. Second, to boost scalability when solving LSMOPs, we propose a scalable multiobjective optimization algorithm by transferring weights between solutions to reduce dependency on ever-increasing computational resources as the problem dimension increases. The proposed framework entails constructing a latent decision space to determine evolutionary weights for chosen representative solutions. These computed weights are then transferred to the remaining solutions, enabling evolutionary optimization to proceed without requiring additional evaluations, even as the dimensionality increases. By utilizing the knowledge gained from the source solutions, each solution is customized with an evolutionary weight scheme that not only preserves computational resources but also enhances optimization performance, thereby boosting scalability. We have conducted experiments on LSMOPs to verify the effectiveness and scalability of the proposed algorithm. The proposed method outperforms selected state-of-the-art algorithms and gains scalability boosts in situations where the dimensions increase while the function evaluation remains constant.

Multi-task collaborative method based on manifold optimization for automated test case generation based on path coverage

Automated test case generation based on path coverage is not only a large-scale black-box optimization problem, but also a key scientific problem in software automatic testing technology. Evolutionary algorithms and other search-based algorithms are representative methods for this problem. However, existing research mainly focuses on the multi-function case, where generating test cases for multiple functions is difficult due to the combinatorial explosion of the path number. In this paper, we propose a multi-task collaborative method based on manifold optimization by considering the topological manifold relationship between the test case space (decision space) and the program path space (target space). This method achieves the goal of collaborative optimization for different function coverage tasks by coordinating the allocation of computing resources and knowledge transfer mechanisms among them. To verify the effectiveness of the proposed method, we compare it with general solution methods for single-function automated test case generation based on path coverage, such as the manifold-inspired search-based algorithm. The experimental results show that the proposed method outperforms the compared single-function optimization algorithms especially on programs with strong coding similarities. This study verifies the feasibility of collaborative optimization methods for solving large-scale black-box optimization problems, such as the automated test case generation based on path coverage, and expands their application scenarios.

Stabilized BB projection algorithm for large-scale convex constrained nonlinear monotone equations to signal and image processing problems

The Barzilai–Borwein (BB) method is a popular and efficient gradient method for solving large-scale unconstrained optimization problems. In general, it converges much faster than the steepest descent (Cauchy) method. However, it may not converge, even when the objective function is strongly convex. To overcome this, by virtue of bounding the distance between sequential iterates, [Burdakov et al. (2019)] proposed a stabilized BB method. In this paper, by combining the stabilized BB method with a projection approach, we develop a stabilized BB projection (SBBP) method to solve nonlinear monotone equations with convex constraints. SBBP does not involve computing matrices proving the usefulness for solving large-scale problems. Its global convergence and Q-linear convergence rate are also established. We report numerical experiments on recovering sparse signals and restoring blurred images arising from compressive sensing, demonstrating the usefulness of SBBP.

Taking the human out of decomposition-based optimization via artificial intelligence, Part II: Learning to initialize

The repeated solution of large-scale optimization problems arises frequently in process systems engineering tasks. Decomposition-based solution methods have been widely used to reduce the corresponding computational time, yet their implementation has multiple steps that are difficult to configure. We propose a machine learning approach to learn the optimal initialization of such algorithms which minimizes the computational time. Active and supervised learning is used to learn a surrogate model that predicts the computational performance for a given initialization. We apply this approach to the initialization of Generalized Benders Decomposition for the solution of mixed-integer model predictive control problems. The surrogate models are used to find the optimal initialization, which corresponds to the number of initial cuts that should be added in the master problem. The results show that the proposed approach can lead to a significant reduction in solution time, and active learning can reduce the data required for learning.

Learning-assisted optimization for transmission switching

AbstractThe design of new strategies that exploit methods from machine learning to facilitate the resolution of challenging and large-scale mathematical optimization problems has recently become an avenue of prolific and promising research. In this paper, we propose a novel learning procedure to assist in the solution of a well-known computationally difficult optimization problem in power systems: The Direct Current Optimal Transmission Switching (DC-OTS) problem. The DC-OTS problem consists in finding the configuration of the power network that results in the cheapest dispatch of the power generating units. With the increasing variability in the operating conditions of power grids, the DC-OTS problem has lately sparked renewed interest, because operational strategies that include topological network changes have proved to be effective and efficient in helping maintain the balance between generation and demand. The DC-OTS problem includes a set of binaries that determine the on/off status of the switchable transmission lines. Therefore, it takes the form of a mixed-integer program, which is NP-hard in general. In this paper, we propose an approach to tackle the DC-OTS problem that leverages known solutions to past instances of the problem to speed up the mixed-integer optimization of a new unseen model. Although our approach does not offer optimality guarantees, a series of numerical experiments run on a real-life power system dataset show that it features a very high success rate in identifying the optimal grid topology (especially when compared to alternative competing heuristics), while rendering remarkable speed-up factors.

Optimal fissile distribution in multiplying systems: Illustrative examples with Monte Carlo simulation and Pontryagin’s maximum principle

In multiplying systems, such as nuclear reactors and criticality experiments, it is desirable to place the fissile material in the optimal or ‘best’ way to reduce the critical mass of the system as well as to achieve uniform fuel burnup. This paper considers two methods, namely Pontryagin’s maximum principle (PMP) and Monte Carlo (MC) perturbation for estimating a minimum critical mass configuration. These methods are applied to an elementary multizone model of a pressurized water reactor (PWR) and a criticality experiment to estimate the minimum critical mass. It is found that while two-group diffusion theory with PMP predicts a minimum critical mass, more detailed MC simulations with MCNP5 show a consistent reduction in critical mass when fissile fuel is placed in inner zones. Such a distribution reduces the fissile material requirement but is undesirable due to the higher power peaking. MC simulations show that for a three-zone model of the KORI 1 PWR, a uniform fissile distribution gives criticality for 1.09 atomic percent (at.%) enrichment, whereas non-uniform fissile distribution (0.6, 1.6, 0.6 at.%) reduces the critical mass by 14%. The changes found from MC simulations were subsequently predicted from first- and second-order derivative sampling. It was found that substantial computational savings can be achieved for large-scale optimization problems. In the case of a criticality experiment, MC derivative sampling was also used to estimate optimal fissile distribution for minimizing the critical mass.

Parametric Frequency Divider Based Ising Machines.

We report on a new class of Ising machines (IMs) that rely on coupled parametric frequency dividers (PFDs) as macroscopic artificial spins. Unlike the IM counterparts based on subharmonic-injection locking (SHIL), PFD IMs do not require strong injected continuous-wave signals or applied dc voltages. Therefore, they show a significantly lower power consumption per spin compared to SHIL-based IMs, making it feasible to accurately solve large-scale combinatorial optimization problems that are hard or even impossible to solve by using the current von Neumann computing architectures. Furthermore, using high quality factor resonators in the PFD design makes PFD IMs able to exhibit a nanowatt-level power per spin. Also, it remarkably allows a speedup of the phase synchronization among the PFDs, resulting in shorter time to solution and lower energy to solution despite the resonators' longer relaxation time. As a proof of concept, a 4-node PFD IM has been demonstrated. This IM correctly solves a set of Max-Cut problems while consuming just 600 nanowatts per spin. This power consumption is 2 orders of magnitude lower than the power per spin of state-of-the-art SHIL-based IMs operating at the same frequency.

A New Dai-Liao Conjugate Gradient Method based on Approximately Optimal Stepsize for Unconstrained Optimization

Conjugate gradient methods are a class of very effective iterative methods for large-scale unconstrained optimization. In this paper, a new Dai-Liao conjugate gradient method for solving large-scale unconstrained optimization problem is proposed. Based on the approximately optimal stepsize for the gradient method, we derive three new choices for the important parameters tk in Dai-Liao conjugate gradient method. The search direction satisfies the sufficient descent condition, and the global convergences of the proposed method for uniformly convex and general functions are proved under some mild conditions. Numerical experiments on a set of test problems from the CUTEst library show that the proposed method is superior to some well-known conjugate gradient methods.

Improved Evolutionary Operators for Sparse Large-Scale Multiobjective Optimization Problems

Many critical science, societal, and engineering fields contain large-scale multiobjective optimization problems (LSMOPs), comprised of many decision variables. However, as the number of decision variables increases, optimization algorithms face exponentially large search spaces, thereby exhibiting a degraded performance. Nonetheless, LSMOPs whose optimal solutions correspond to sparse variable vectors can be solved more efficiently by evolutionary multiobjective optimization (EMO) algorithms. Despite the great recent strides in developing generic EMO algorithms for sparse LSMOPs, there is still room for improvement. Specifically, algorithms still struggle to find convergent and diverse Pareto fronts in an acceptable amount of time when solving sparse LSMOPs with thousands of decision variables. To better solve sparse LSMOPs, we propose a novel set of evolutionary operators to adapt small-scale EMO algorithms for sparse LSMOPs. These simple, novel, and effective operators include varied striped sparse population sampling (VSSPS), sparse simulated binary crossover (S-SBX), and sparse polynomial mutation (S-PM). These operators, combined with NSGA-II, make the proposed S-NSGA-II algorithm. S-NSGA-II runs near-universally faster than existing methods for problems containing up to 6,400 decision variables, while performing as well as or better than contemporary sparse LSMOP algorithms with respect to hypervolume, especially with problems with larger than 5,000 decision variables.

Accelerated Double-Sketching Subspace Newton

This paper proposes a second-order stochastic algorithm called Accelerated Double-Sketching Subspace Newton (ADSSN) to solve large-scale optimization problems with high dimensional feature spaces and substantial sample sizes. The proposed ADSSN has two computational superiority. First, ADSSN achieves a fast local convergence rate by exploiting Nesterov’s acceleration technique. Second, by taking full advantage of the double sketching strategy, ADSSN provides a lower computational cost for each iteration than competitive approaches. Moreover, these advantages hold for actually all sketching techniques, which enables practitioners to design custom sketching methods for specific applications. Finally, numerical experiments are carried out to demonstrate the efficiency of ADSSN compared with accelerated gradient descent and two single sketching counterparts.