Large-scale Constrained Optimization Problems Research Articles

Distributed gradient-free and projection-free algorithm for stochastic constrained optimization

Distributed stochastic zeroth-order optimization (DSZO), in which the objective function is allocated over multiple agents and the derivative of cost functions is unavailable, arises frequently in large-scale machine learning and reinforcement learning. This paper introduces a distributed stochastic algorithm for DSZO in a projection-free and gradient-free manner via the Frank-Wolfe framework and the stochastic zeroth-order oracle (SZO). Such a scheme is particularly useful in large-scale constrained optimization problems where calculating gradients or projection operators is impractical, costly, or when the objective function is not differentiable everywhere. Specifically, the proposed algorithm, enhanced by recursive momentum and gradient tracking techniques, guarantees convergence with just a single batch per iteration. This significant improvement over existing algorithms substantially lowers the computational complexity. Under mild conditions, we prove that the complexity bounds on SZO of the proposed algorithm are O(n/ϵ2)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{O}(n/\\epsilon ^{2})$\\end{document} and O(n(21ϵ))\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{O}(n(2^{\\frac{1}{\\epsilon}}))$\\end{document} for convex and nonconvex cases, respectively. The efficacy of the algorithm is verified on black-box binary classification problems against several competing alternatives.

Optimal power flow solution based on gorilla troops optimization technique considering uncertainty of renewable energy sources: A case study of Adrar’s isolated power network

This paper proposes an efficient Gorilla troops-inspired algorithm to cope optimal power flow (OPF) problem considering uncertainty of renewable energy sources (RES). The problem is formulated as large-scale constrained optimization problem with non-linear characteristics. Its degree of complexity increases with incorporation of intermittent energy sources, making it harder to be solved using conventional optimization techniques. However, could be efficiently resolved by nature-inspired optimization algorithms and solvers. The objective function is the overall cost of system, including reserve cost for over-estimation and penalty cost for under-estimation of two types of PV-solar and wind energy. To demonstrate the consistency and robustness of the developed algorithm a case study on the modified IEEE 30-bus system and and Adrar’s power network (isolated grid) is carried out. Simulation results show the capability of GTO to find high quality optimal feasible solutions and ranked first among the compared algorithms, and so, over different function landscapes.

A novel improved teaching and learning-based-optimization algorithm and its application in a large-scale inventory control system

PurposeThe purpose of this paper is to propose a novel improved teaching and learning-based algorithm (TLBO) to enhance its convergence ability and solution accuracy, making it more suitable for solving large-scale optimization issues.Design/methodology/approachUtilizing multiple cooperation mechanisms in teaching and learning processes, an improved TBLO named CTLBO (collectivism teaching-learning-based optimization) is developed. This algorithm introduces a new preparation phase before the teaching and learning phases and applies multiple teacher–learner cooperation strategies in teaching and learning processes. Applying modularization idea, based on the configuration structure of operators of CTLBO, six variants of CTLBO are constructed. For identifying the best configuration, 30 general benchmark functions are tested. Then, three experiments using CEC2020 (2020 IEEE Conference on Evolutionary Computation)-constrained optimization problems are conducted to compare CTLBO with other algorithms. At last, a large-scale industrial engineering problem is taken as the application case.FindingsExperiment with 30 general unconstrained benchmark functions indicates that CTLBO-c is the best configuration of all variants of CTLBO. Three experiments using CEC2020-constrained optimization problems show that CTLBO is one powerful algorithm for solving large-scale constrained optimization problems. The application case of industrial engineering problem shows that CTLBO and its variant CTLBO-c can effectively solve the large-scale real problem, while the accuracies of TLBO and other meta-heuristic algorithm are far lower than CLTBO and CTLBO-c, revealing that CTLBO and its variants can far outperform other algorithms. CTLBO is an excellent algorithm for solving large-scale complex optimization issues.Originality/valueThe innovation of this paper lies in the improvement strategies in changing the original TLBO with two-phase teaching–learning mechanism to a new algorithm CTLBO with three-phase multiple cooperation teaching–learning mechanism, self-learning mechanism in teaching and group teaching mechanism. CTLBO has important application value in solving large-scale optimization problems.

Hybrid limited memory gradient projection methods for box-constrained optimization problems

Gradient projection methods represent effective tools for solving large-scale constrained optimization problems thanks to their simple implementation and low computational cost per iteration. Despite these good properties, a slow convergence rate can affect gradient projection schemes, especially when high accurate solutions are needed. A strategy to mitigate this drawback consists in properly selecting the values for the steplength along the negative gradient. In this paper, we consider the class of gradient projection methods with line search along the projected arc for box-constrained minimization problems and we analyse different strategies to define the steplength. It is well known in the literature that steplength selection rules able to approximate, at each iteration, the eigenvalues of the inverse of a suitable submatrix of the Hessian of the objective function can improve the performance of gradient projection methods. In this perspective, we propose an automatic hybrid steplength selection technique that employs a proper alternation of standard Barzilai–Borwein rules, when the final active set is not well approximated, and a generalized limited memory strategy based on the Ritz-like values of the Hessian matrix restricted to the inactive constraints, when the final active set is reached. Numerical experiments on quadratic and non-quadratic test problems show the effectiveness of the proposed steplength scheme.

Gannet optimization algorithm : A new metaheuristic algorithm for solving engineering optimization problems

Engineering design problems are usually large-scale constrained optimization problems, and metaheuristic algorithms are vital for solving such complex problems. Therefore, this paper introduces a new nature-inspired metaheuristic algorithm: the gannet optimization algorithm (GOA). The GOA mathematizes the various unique behaviors of gannets during foraging and is used to enable exploration and exploitation. GOA’s U-shaped and V-shaped diving patterns are responsible for exploring the optimal region within the search space, with sudden turns and random walks ensuring better solutions are found in this region. In order to verify the ability of the GOA to find the optimal solution, we compared it with other comparison algorithms in multiple dimensions of 28 benchmark functions. We found that the GOA has a shorter running time in high dimensions and can provide a better solution. Finally, we apply the GOA to five engineering optimization problems. The experimental results show that the GOA is suitable for many constrained engineering design problems and can provide better solutions in most cases.

Variable Decomposition for Large-Scale Constrained Optimization Problems Using a Grouping Genetic Algorithm

Several real optimization problems are very difficult, and their optimal solutions cannot be found with a traditional method. Moreover, for some of these problems, the large number of decision variables is a major contributing factor to their complexity; they are known as Large-Scale Optimization Problems, and various strategies have been proposed to deal with them. One of the most popular tools is called Cooperative Co-Evolution, which works through a decomposition of the decision variables into smaller subproblems or variables subgroups, which are optimized separately and cooperate to finally create a complete solution of the original problem. This kind of decomposition can be handled as a combinatorial optimization problem where we want to group variables that interact with each other. In this work, we propose a Grouping Genetic Algorithm to optimize the variable decomposition by reducing their interaction. Although the Cooperative Co-Evolution approach is widely used to deal with unconstrained optimization problems, there are few works related to constrained problems. Therefore, our experiments were performed on a test benchmark of 18 constrained functions under 100, 500, and 1000 variables. The results obtained indicate that a Grouping Genetic Algorithm is an appropriate tool to optimize the variable decomposition for Large-Scale Constrained Optimization Problems, outperforming the decomposition obtained by a state-of-the-art genetic algorithm.

Novel Design of Slim Mould Optimizer for the Solution of Optimal Power Flow Problems Incorporating Intermittent Sources: A Case Study of Algerian Electricity Grid

Nowadays, electrical power grids are facing increased penetration of renewable energy sources (RES), which result in increasing level of randomness and uncertainties for its operational quality. In addition, emerging need for efficient solutions to stochastic optimal power flow (OPF) problem has attracted considerable attention to ensure optimal and reliable grid operations in the presence of generation uncertainty and increasing demand. Therefore, this paper proposes an efficient Slime Mould-inspired Algorithm (SMA) that aims to minimize overall operating cost of main grid by managing the power flow among different generating resources. The problem is formulated as large-scale constrained optimization problem with non-linear characteristics. Its degree of complexity increases with incorporation of intermittent energy sources, making it harder to be solved using conventional optimization techniques. However, could be efficiently resolved by nature-inspired optimization techniques without any modification or approximation into the original-formulation. The objective function is the overall cost of system, including reserve cost for over-estimation and penalty cost for under-estimation of both PV-solar and wind energy. The SMA performance is evaluated on the IEEE 30-bus test system and Algerian power system, DZA 114-bus. The SMA is compared with four optimization algorithms: i) The well-studied meta-heuristics, i.e., Gorilla troops optimizer (GTO), and Orca predation algorithm (OPA), ii) Recently developed meta-heuristics, i.e., Artificial ecosystem optimizer (AEO), Hunger games search (HGS), and Jellyfish search (JS) optimizer, iii) ad high-performance meta-heuristics, Success-History based parameter adaptation for differential evolution method. The overall simulation results reveal that the SMA ranked first among the compared algorithms, and so, over and so, over different function landscapes.

Distributed Primal-Dual Method for Convex Optimization With Coupled Constraints

Distributed primal-dual methods have been widely used for solving large-scale constrained optimization problems. The majority of existing results focus on the problems with decoupled constraints. Some recent works have studied the problems subject to separable globally coupled constraints. This paper considers the distributed optimization problems with globally coupled constraints over networks without requiring the separability of the globally coupled constraints. This is made possible by the local estimates of the constraint violations. For solving such a problem, we propose a primal-dual algorithm in the augmented Lagrangian framework, combining the average consensus technique. We first establish a non-ergodic convergence rate of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {O}(1/k)$</tex-math></inline-formula> in terms of the objective residual for solving a distributed constrained convex optimization problem, where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> is the iteration counter. Specifically, the global objective function is the aggregate of the local convex and possibly non-smooth costs, and the coupled constraint is the sum of the local linear equality constraints. The numerical results illustrate the performance of the proposed method.

Fast quasi-harmonic weights for geometric data interpolation

We propose quasi-harmonic weights for interpolating geometric data, which are orders of magnitude faster to compute than state-of-the-art. Currently, interpolation (or, skinning) weights are obtained by solving large-scale constrained optimization problems with explicit constraints to suppress oscillative patterns, yielding smooth weights only after a substantial amount of computation time. As an alternative, our weights are obtained as minima of an unconstrained problem that can be optimized quickly using straightforward numerical techniques. We consider weights that can be obtained as solutions to a parameterized family of second-order elliptic partial differential equations. By leveraging the maximum principle and careful parameterization, we pose weight computation as an inverse problem of recovering optimal anisotropic diffusivity tensors. In addition, we provide a customized ADAM solver that significantly reduces the number of gradient steps; our solver only requires inverting tens of linear systems that share the same sparsity pattern. Overall, our approach achieves orders of magnitude acceleration compared to previous methods, allowing weight computation in near real-time.

On the solution of linearly constrained optimization problems by means of barrier algorithms

Many practical problems require the solution of large-scale constrained optimization problems for which preserving feasibility is a key issue, and the evaluation of the objective function is very expensive. In these cases it is mandatory to start with a feasible approximation of the solution, the obtention of which should not require objective function evaluations. The necessity of solving this type of problems motivated us to revisit the classical barrier approach for nonlinear optimization, providing a careful implementation of a modern version of this method. This is the main objective of the present paper. For completeness, we provide global convergence results and comparative numerical experiments with one of the state-of-the-art interior-point solvers for continuous optimization.

Complexity and performance of an Augmented Lagrangian algorithm

Algencan is a well established safeguarded Augmented Lagrangian algorithm introduced in [R. Andreani, E. G. Birgin, J. M. Martínez, and M. L. Schuverdt, On Augmented Lagrangian methods with general lower-level constraints, SIAM J. Optim. 18 (2008), pp. 1286–1309]. Complexity results that report its worst-case behaviour in terms of iterations and evaluations of functions and derivatives that are necessary to obtain suitable stopping criteria are presented in this work. In addition, its computational performance considering all problems from the CUTEst collection is presented, which shows that it is a useful tool for solving large-scale constrained optimization problems.

A local cooperative approach to solve large-scale constrained optimization problems

Abstract Cooperative Co-evolutionary algorithms are very popular to solve large-scale problems. A significant part of these algorithms is the decomposition of the problems according to the variables interaction. In this paper, an approach based on a memetic scheme, where its local stage (and not the global stage) is guided by the decomposition method (Local Cooperative Search LoCoS), is presented to solve large-scale constrained optimization problems. Two decomposition methods are tested: the improved version of the Variable Interdependence Identification for Constrained problems and Differential Grouping version 2. A recently-proposed benchmark with eighteen test problems with different features is solved to assess the performance of LoCoS when compared against a similar memetic algorithm but without decomposition and also against a state-of-the-art cooperative co-evolutionary algorithm. The results show a faster convergence, better final results and higher feasibility ratio by LoCosS with respect to the values provided by the compared algorithms.

An augmented Lagrangian proximal alternating method for sparse discrete optimization problems

In this paper, we propose an augmented Lagrangian proximal alternating (ALPA) method for solving two classes of large-scale sparse discrete constrained optimization problems. Specifically, the ALPA method generates a sequence of augmented Lagrangian (AL) subproblems in the out iterations and utilizes a proximal alternating linearized minimization method and sparse projection techniques to solve these AL subproblems. And we study the first-order optimality conditions for these two classes of problems. Under some suitable assumptions, we show that any accumulation point of the sequence generated by the ALPA method satisfies the necessary first-order optimality conditions of these problems or is a local minimizer of these problems. The computational results with practical problems demonstrate that our method can find the suboptimal solutions of the problems efficiently and is competitive with some other local solution methods.

A Level-Set Method for Convex Optimization with a Feasible Solution Path

Large-scale constrained convex optimization problems arise in several application domains. First-order methods are good candidates to tackle such problems due to their low iteration complexity and ...

On the application of an augmented Lagrangian algorithm to some portfolio problems

Algencan is a freely available piece of software that aims to solve smooth large-scale constrained optimization problems. When applied to specific problems, obtaining a good performance in terms of efficacy and efficiency may depend on careful choices of options and parameters. In the present paper, the application of Algencan to four portfolio optimization problems is discussed and numerical results are presented and evaluated.

Primal-Dual Active-Set Methods for Large-Scale Optimization

In this paper, we introduce two primal-dual active-set methods for solving large-scale constrained optimization problems. The first method minimizes a sequence of primal-dual augmented Lagrangian functions subject to bounds on the primal variables and artificial bounds on the dual variables. The basic structure is similar to the well-known optimization package Lancelot (Conn, et al. in SIAM J Numer Anal 28:545---572, 1991), which uses the traditional primal augmented Lagrangian function. Like Lancelot, our algorithm may use gradient projection-based methods enhanced by subspace acceleration techniques to solve each subproblem and therefore may be implemented matrix-free. The artificial bounds on the dual variables are a unique feature of our method and serve as a form of dual regularization. Our second algorithm is a two-phase method. The first phase computes iterates using our primal-dual augmented Lagrangian algorithm, which benefits from using cheap gradient projections and matrix-free linear CG calculations. The final iterate produced during this phase is then used as input for phase two, which is a stabilized sequential quadratic programming method (Gill and Robinson in SIAM J Opt 1---45, 2013). Obtaining superlinear local convergence under weak assumptions is an important benefit of the transition to a stabilized sequential quadratic programming algorithm. Interestingly, the bound-constrained subproblem used in phase one is equivalent to the stabilized subproblem used in phase two under certain assumptions. This fact makes our choice of algorithms a natural one.

An adaptive augmented Lagrangian method for large-scale constrained optimization

We propose an augmented Lagrangian algorithm for solving large-scale constrained optimization problems. The novel feature of the algorithm is an adaptive update for the penalty parameter motivated by recently proposed techniques for exact penalty methods. This adaptive updating scheme greatly improves the overall performance of the algorithm without sacrificing the strengths of the core augmented Lagrangian framework, such as its ability to be implemented matrix-free. This is important as this feature of augmented Lagrangian methods is responsible for renewed interests in employing such methods for solving large-scale problems. We provide convergence results from remote starting points and illustrate by a set of numerical experiments that our method outperforms traditional augmented Lagrangian methods in terms of critical performance measures.

Sequential Quadratic Programming (SQP) for optimal control in direct numerical simulation of turbulent flow

The current work focuses on the development and application of an efficient algorithm for optimization of three-dimensional turbulent flows, simulated using Direct Numerical Simulation (DNS) or Large-Eddy Simulations, and further characterized by large-dimensional optimization-parameter spaces. The optimization algorithm is based on Sequential Quadratic Programming (SQP) in combination with a damped formulation of the limited-memory BFGS method. The latter is suitable for solving large-scale constrained optimization problems whose Hessian matrices cannot be computed and stored at a reasonable cost. We combine the algorithm with a line-search merit function based on an L1-norm to enforce the convergence from any remote point. It is first shown that the proposed form of the damped L-BFGS algorithm is suitable for solving equality constrained Rosenbrock type functions. Then, we apply the algorithm to an optimal-control test problem that consists of finding the optimal initial perturbations to a turbulent temporal mixing layer such that mixing is improved at the end of a simulation time horizon T. The controls are further subject to a non-linear equality constraint on the total control energy. DNSs are used to resolve all turbulent scales of motion, and a continuous adjoint formulation is employed to calculate the gradient of the cost functionals. We compare the convergence speed of the SQP L-BFGS algorithm to a conventional non-linear conjugate-gradient method (i.e. the current standard in DNS-based optimal control), and find that the SQP algorithm is more than an order of magnitude faster than the conjugate-gradient method.

Parallel Variable Distribution Algorithm for Constrained Optimization with Nonmonotone Technique

A modified parallel variable distribution (PVD) algorithm for solving large-scale constrained optimization problems is developed, which modifies quadratic subproblemQPlat each iteration instead of theQPl0of the SQP-type PVD algorithm proposed by C. A. Sagastizábal and M. V. Solodov in 2002. The algorithm can circumvent the difficulties associated with the possible inconsistency ofQPl0subproblem of the original SQP method. Moreover, we introduce a nonmonotone technique instead of the penalty function to carry out the line search procedure with more flexibly. Under appropriate conditions, the global convergence of the method is established. In the final part, parallel numerical experiments are implemented on CUDA based on GPU (Graphics Processing unit).

Efficient computation of nonparametric survival functions via a hierarchical mixture formulation

We propose a new algorithm for computing the maximum likelihood estimate of a nonparametric survival function for interval-censored data, by extending the recently-proposed constrained Newton method in a hierarchical fashion. The new algorithm makes use of the fact that a mixture distribution can be recursively written as a mixture of mixtures, and takes a divide-and-conquer approach to break down a large-scale constrained optimization problem into many small-scale ones, which can be solved rapidly. During the course of optimization, the new algorithm, which we call the hierarchical constrained Newton method, can efficiently reallocate the probability mass, both locally and globally, among potential support intervals. Its convergence is theoretically established based on an equilibrium analysis. Numerical study results suggest that the new algorithm is the best choice for data sets of any size and for solutions with any number of support intervals.