Nonconvex Constrained Optimization Research Articles

Stochastic nested primal-dual method for nonconvex constrained composition optimization

In this paper we study the nonconvex constrained composition optimization, in which the objective contains a composition of two expected-value functions whose accurate information is normally expensive to calculate. We propose a STochastic nEsted Primal-dual (STEP) method for such problems. In each iteration, with an auxiliary variable introduced to track the inner layer function values we compute stochastic gradients of the nested function using a subsampling strategy. To alleviate difficulties caused by possibly nonconvex constraints, we construct a stochastic approximation to the linearized augmented Lagrangian function to update the primal variable, which further motivates to update the dual variable in a weighted-average way. Moreover, to better understand the asymptotic dynamics of the update schemes we consider a deterministic continuous-time system from the perspective of ordinary differential equation (ODE). We analyze the Karush-Kuhn-Tucker measure at the output by the STEP method with constant parameters and establish its iteration and sample complexities to find an ϵ \epsilon -stationary point, ensuring that expected stationarity, feasibility as well as complementary slackness are below accuracy ϵ \epsilon . To leverage the benefit of the (near) initial feasibility in the STEP method, we propose a two-stage framework incorporating a feasibility-seeking phase, aiming to locate a nearly feasible initial point. Moreover, to enhance the adaptivity of the STEP algorithm, we propose an adaptive variant by adaptively adjusting its parameters, along with a complexity analysis. Numerical results on a risk-averse portfolio optimization problem and an orthogonal nonnegative matrix decomposition reveal the effectiveness of the proposed algorithms.

Normalized penalty gradient flow: A continuous-time approach for solving constrained nonconvex nonsmooth optimization problems

To avoid calculating penalty parameters, this paper introduces a continuous-time approach that combines the normalized gradient flow with the penalty method to solve the nonconvex nonsmooth optimization problem with a convex constraint set. Subsequently, this approach is extended to solve nonconvex nonsmooth optimization with a box constraint set and affine equality constraints. Compared with the current methods, the proposed approach does not require calculating the penalty parameters, allows the objective function to be nonconvex or nonsmooth, and allows the initial point to be arbitrarily selected. It is proved that the solution trajectory with any initial point converges to the critical point set of the optimization problem. Finally, the effectiveness of the proposed approach is authenticated through several numerical experiments.

A Nonconvex Proximal Bundle Method for Nonsmooth Constrained Optimization

An implementable algorithm for solving nonsmooth nonconvex constrained optimization is proposed by combining bundle ideas, proximity control, and the exact penalty function. We construct two kinds of approximations to nonconvex objective function; these two approximations correspond to the convex and concave behaviors of the objective function at the current point, which captures precisely the characteristic of the objective function. The penalty coefficients are increased only a finite number of times under the conditions of Slater constraint qualification and the boundedness of the constrained set, which limit the unnecessary penalty growth. The given algorithm converges to an approximate stationary point of the exact penalty function for constrained nonconvex optimization with weakly semismooth objective function. We also provide the results of some preliminary numerical testing to show the validity and efficiency of the proposed method.

Comparison of a Hybrid Firefly–Particle Swarm Optimization Algorithm with Six Hybrid Firefly–Differential Evolution Algorithms and an Effective Cost-Saving Allocation Method for Ridesharing Recommendation Systems

The optimization and allocation of transport cost savings among stakeholders are two important issues that influence the satisfaction of information providers, drivers and passengers in ridesharing recommendation systems. For optimization issues, finding optimal solutions for nonconvex constrained discrete ridesharing optimization problems poses a challenge due to computational complexity. For the allocation of transport cost savings issues, the development of an effective method to allocate cost savings in ridesharing recommendation systems is an urgent need to improve the acceptability of ridesharing. The hybridization of different metaheuristic approaches has demonstrated its advantages in tackling the complexity of optimization problems. The principle of the hybridization of metaheuristic approaches is similar to a marriage of two people with the goal of having a happy ending. However, the effectiveness of hybrid metaheuristic algorithms is unknown a priori and depends on the problem to be solved. This is similar to a situation where no one knows whether a marriage will have a happy ending a priori. Whether the hybridization of the Firefly Algorithm (FA) with Particle Swarm Optimization (PSO) or Differential Evolution (DE) can work effectively in solving ridesharing optimization problems needs further study. Motivated by deficiencies in existing studies, this paper focuses on the effectiveness of hybrid metaheuristic algorithms for solving ridesharing problems based on the hybridization of FA with PSO or the hybridization of FA with DE. Another focus of this paper is to propose and study the effectiveness of a new method to allocate ridesharing cost savings to the stakeholders in ridesharing systems. The developed hybrid metaheuristic algorithms and the allocation method have been compared with examples of several application scenarios to illustrate their effectiveness. The results indicate that hybridizing FA with PSO creates a more efficient algorithm, whereas hybridizing FA with DE does not lead to a more efficient algorithm for the ridesharing recommendation problem. An interesting finding of this study is very similar to what happens in the real world: “Not all marriages have happy endings”.

Distributed Resilient Initialization-Free Jacobi Descent Algorithm for Constrained Optimization Against DoS Attacks

This paper investigates one type of distributed constrained optimization problem, e.g., the economic dispatch problem, in the presence of DoS attacks. Therein, multiple DoS attackers are collaborative to impede the communication transmission and change the communication topology at will. Consequently, the convergence and/or optimality of distributed algorithm may be compromised. To reduce the effect of this kind of DoS attacks, a distributed resilient initialization-free Jacobi descent algorithm is proposed. It is designed with three switched control protocols which enable the proposed algorithm reasonably employing the estimations to replace the missing information when attacks occur. Meanwhile, the proposed method is embedded with second order information, resulting in faster convergence speed. Moreover, theoretical analysis results are provided to show that the proposed algorithm can exponentially converge to the global optimal solution of the studied problem. Finally, simulation results tested in IEEE 30-bus system validate its effectiveness and flexibility. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The economic dispatch is a key issue in smart grid, which can be formulated as a kind of distributed constrained optimization problem. Since the distributed algorithms work under distributed sensor networks, they are easier to undergo DoS attacks. To address this issue, this paper presents a distributed resilient initialization-free Jacobi descent algorithm, which features strong robustness to resist DoS attacks and faster convergence. Meanwhile, the proposed method is shaped for common constrained optimization problem with better expansibility. We conduct the global convergence and optimality proofs, which benefits the practitioners to estimate the convergence performance, e.g., the convergence rate. Simulations further show the correctness and effectiveness of the proposed method. In future, we will pay more attention on the non-convex constrained optimization problem.

A Robust Coevolutionary Neural-Based Optimization Algorithm for Constrained Nonconvex Optimization.

For nonconvex optimization problems, a routine is to assume that there is no perturbation when executing the solution task. Nevertheless, dealing with the perturbation in advance may increase the burden on the system and take up extra time. To remedy this weakness, we propose a robust coevolutionary neural-based optimization algorithm with inherent robustness based on the hybridization between the particle swarm optimization and a class of robust neural dynamics (RND). In this framework, every neural agent guided by the RND supersedes the place of the particle, mutually searches for the optimal solution, and stabilizes itself from different perturbations. The theoretical analysis ensures that the proposed algorithm is globally convergent with probability one. Besides, the effectiveness and robustness of the proposed approach are illustrated by illustrative examples compared with the existing methods. We further apply this proposed algorithm to the source localization and manipulability optimization of the redundant manipulator, simultaneously disposing of perturbation from the internal and exogenous system with satisfactory performance.

Adaptive Beamforming with Sidelobe Level Control for Multiband Sparse Linear Array

Multiband antenna arrays have the capability of effectively working in multiple frequency bands and thus significantly simplify the antenna system. To further reduce the system overhead, this paper discusses the joint design of antenna selection and adaptive beamforming for multiband antenna arrays, where the sidelobe level is also controlled so as to alleviate the effect of unknown sporadic interference. Based on the maximum signal-to-interference-plus-noise ratio (SINR) criterion and sidelobe level constraints, the proposed multiband sparse array design is formulated into a nonconvex constrained nonlinear optimization problem with an l0,2-mixed norm regularization. This problem ensures that the same antenna positions are selected at all operating frequencies while the beamformer weights of each frequency are optimized independently. By exploiting the semi-definite relaxation and the reweighted l1,∞-norm approximation, the problem is converted into a series of convex subproblems and is then effectively solved by the proposed iterative reweighted method. Numerical results show that the proposed multiband sparse array significantly reduces the sidelobe levels in all operating frequencies while maintaining the maximum SINR, so it provides superior performance of interference suppression.

A hybrid quantum-behaved particle swarm optimization solution to non-convex economic load dispatch with multiple fuel types and valve-point effects

Economic dispatch problems (EDPs) can be reduced to non-convex constrained optimization problems, and most of the population-based algorithms are prone to have problems of premature and falling into local optimum when solving EDPs. Therefore, this paper proposes a hybrid quantum-behaved particle swarm optimization (HQPSO) algorithm to alleviate the above problems. In the HQPSO, the Solis and Wets local search method is used to enhance the local search ability of the QPSO so that the algorithm can find solutions that is close to optimal when the constraints are met, and two evolution operators are proposed and incorporated for the purpose of making a better balance between local search and global search abilities at the later search stage. The performance comparison is made among the HQPSO and the other ten population-based random search methods under two different experimental configurations and four different power systems in terms of solution quality, robustness, and convergence property. The experimental results show that the HQPSO improves the convergence properties of the QPSO and finally obtains the best total generation cost without violating any constraints. In addition, the HQPSO outperforms all the other algorithms on 7 cases of all 8 experimental cases in terms of global best position and mean position, which verifies the effectiveness of the algorithm.

Slow Kill for Big Data Learning

Big-data applications often involve a vast number of observations and features, creating new challenges for variable selection and parameter estimation. This paper presents a novel technique called “slow kill,” which utilizes nonconvex constrained optimization, adaptive ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> -shrinkage, and increasing learning rates. The fact that the problem size can decrease during the slow kill iterations makes it particularly effective for large-scale variable screening. The interaction between statistics and optimization provides valuable insights into controlling quantiles, stepsize, and shrinkage parameters in order to relax the regularity conditions required to achieve the desired level of statistical accuracy. Experimental results on real and synthetic data show that slow kill outperforms state-of-the-art algorithms in various situations while being computationally efficient for large-scale data.

Structured Static Output Feedback Mixed ${H}_2/{H}_\infty $ Control for Linear Control Systems

This note considers the design of static output feedback mixed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${H}_2/{H}_\infty $</tex-math></inline-formula> controllers for linear control systems with certain equality and inequality constraints imposed directly on the feedback matrix. Based on the barrier method, we solve an auxiliary minimization problem to obtain an approximate solution to the original nonconvex constrained optimization problem. Necessary conditions for the optimal solution of the auxiliary minimization problem are derived using the Lagrange multiplier method. Subsequently, an iterative steepest descent algorithm is developed to find an approximate optimal solution. Finally, an example is provided to validate the proposed approach.

Exactly Uncorrelated Sparse Principal Component Analysis

Sparse principal component analysis (PCA) aims to find principal components as linear combinations of a subset of the original input variables without sacrificing the fidelity of the classical PCA. Most existing sparse PCA methods produce correlated sparse principal components. We argue that many applications of PCA prefer uncorrelated principal components. However, handling sparsity and uncorrelatedness properties in a sparse PCA method is nontrivial. This article proposes an exactly uncorrelated sparse PCA method named EUSPCA, whose formulation is motivated by original views and motivations of PCA as advocated by Pearson and Hotelling. EUSPCA is a non-smooth constrained non-convex manifold optimization problem. We solve it by combining augmented Lagrangian and non-monotone proximal gradient methods. We observe that EUSPCA produces uncorrelated components and maintains a similar or better level of fidelity based on adjusted total variance through simulated and real data examples. In contrast, existing sparse PCA methods produce significantly correlated components. Supplemental materials for this article are available online.

Perception-aware online trajectory generation for a prescribed manoeuvre of unmanned surface vehicle in cluttered unstructured environment

This paper proposes a perception-aware online trajectory generation system that facilitates prescribed manoeuvres of an unmanned surface vehicle (USV) in a dynamic unstructured environment. The proposed system is developed based on the principles of the inverse dynamics in the virtual domain (IDVD) method and an event-triggered receding horizon control (ETRHC) mechanism. This approach transforms the underlying nonconvex constrained optimization problem into a virtual space with a differentially flat dynamics and uses relatively few decision variables to prototype feasible quasi-optimal trajectories. The closed-loop configuration is provided by a computationally efficient ETRHC mechanism that uses situational awareness of operating environment to trigger trajectory replanning if/when required. This addresses the challenge of continuously updating a closed-loop trajectory which imposes unnecessary computational burden on a system with the limited onboard resources. To investigate the performance of the proposed trajectory generating system, a dynamic unstructured environment including variable and uncertain no-fly zone areas as well as variable current vector fields are modeled. Further, different operating conditions incorporating the uncertainties of environment and sudden failure on the USV propulsion system are introduced to examine the effectiveness, agility, and robustness of the proposed trajectory generating system. A comparative study with benchmark solutions generated by the hp-adaptive Radau pseudo-spectral method is conducted to provide a detailed statistical analysis of the proposed approach robustness, computational complexity, and effectiveness. The simulation results confirm the effectiveness of the proposed trajectory generator and ability to produce a solution for online realization.

An alternating structure-adapted Bregman proximal gradient descent algorithm for constrained nonconvex nonsmooth optimization problems and its inertial variant

An alternating structure-adapted Bregman proximal gradient descent algorithm for constrained nonconvex nonsmooth optimization problems and its inertial variant

Leveraging Power Grid Topology in Machine Learning Assisted Optimal Power Flow

Machine learning assisted optimal power flow (OPF) aims to reduce the computational complexity of these non-linear and non-convex constrained optimization problems by consigning expensive (online) optimization to offline training. The majority of work in this area typically employs fully connected neural networks (FCNN). However, recently convolutional (CNN) and graph (GNN) neural networks have also been investigated, in effort to exploit topological information within the power grid. Although promising results have been obtained, there lacks a systematic comparison between these architectures throughout literature. Accordingly, we introduce a concise framework for generalizing methods for machine learning assisted OPF and assess the performance of a variety of FCNN, CNN and GNN models for two fundamental approaches in this domain: regression (predicting optimal generator set-points) and classification (predicting the active set of constraints). For several synthetic power grids with interconnected utilities, we show that locality properties between feature and target variables are scarce and subsequently demonstrate marginal utility of applying CNN and GNN architectures compared to FCNN for a fixed grid topology. However, with variable topology (for instance, modeling transmission line contingency), GNN models are able to straightforwardly take the change of topological information into account and outperform both FCNN and CNN models.

A new Lagrangian-based first-order method for nonconvex constrained optimization

In this paper, we introduce a new form of Lagrangian function and propose a simple first-order primal-dual algorithm for solving nonconvex optimization with nonlinear equality constraints. We show that the algorithm generates bounded primal-dual iterates, and establish the convergence to KKT points under standard assumptions. The key features of the proposed method are: (i) it does not require boundedness assumptions on dual iterates generated by the algorithm as well as the set of multipliers; (ii) it is a single-loop algorithm that does not involve any penalty subproblems.

Global Convergence of Policy Gradient Primal–Dual Methods for Risk-Constrained LQRs

While the techniques in optimal control theory are often model-based, the policy optimization (PO) approach directly optimizes the performance metric of interest. Even though it has been an essential approach for reinforcement learning problems, there is little theoretical understanding of its performance. In this paper, we focus on the risk-constrained linear quadratic regulator (RC-LQR) problem via the PO approach, which requires addressing a challenging non-convex constrained optimization problem. To solve it, we first build on our earlier result that an optimal policy has a time-invariant affine structure to show that the associated Lagrangian function is coercive, locally gradient dominated, and has local Lipschitz continuous gradient, based on which we establish strong duality. Then, we design policy gradient primal-dual methods with global convergence guarantees in both model-based and sample-based settings. Finally, we use samples of system trajectories in simulations to validate our methods.

A First-Order Primal-Dual Method for Nonconvex Constrained Optimization Based on the Augmented Lagrangian

Nonlinearly constrained nonconvex and nonsmooth optimization models play an increasingly important role in machine learning, statistics, and data analytics. In this paper, based on the augmented Lagrangian function, we introduce a flexible first-order primal-dual method, to be called nonconvex auxiliary problem principle of augmented Lagrangian (NAPP-AL), for solving a class of nonlinearly constrained nonconvex and nonsmooth optimization problems. We demonstrate that NAPP-AL converges to a stationary solution at the rate of [Formula: see text], where k is the number of iterations. Moreover, under an additional error bound condition (to be called HVP-EB in the paper) with exponent [Formula: see text], we further show the global convergence of NAPP-AL. Additionally, if [Formula: see text], then we furthermore show that the convergence rate is in fact linear. Finally, we show that the well-known Kurdyka-Łojasiewicz property and the Hölderian metric subregularity imply the aforementioned HVP-EB condition. We demonstrate that under mild conditions, NAPP-AL can also be interpreted as a variant of the forward-backward operator splitting method in this context. Funding: This work was supported by the National Natural Science Foundation of China [Grant 71871140].

Memory, evolutionary operator, and local search based improved Grey Wolf Optimizer with linear population size reduction technique

Optimization of multi-modal functions is challenging even for evolutionary and swarm-based algorithms as it requires an efficient exploration for finding the promising region of the search space, and effective exploitation to precisely find the global optimum. Grey Wolf Optimizer (GWO) is a recently developed metaheuristic algorithm that is inspired by nature with a relatively small number of parameters for tuning. However, GWO and most of its variants may suffer from the lack of population diversity, premature convergence, and the inability to preserve a good balance between exploratory and exploitative behaviors. To address these limitations, this work proposes a new variant of GWO incorporating memory, evolutionary operators, and a stochastic local search technique. It further integrates Linear Population Size Reduction (LPSR) technique. The proposed algorithm is comprehensively tested on 23 numerical benchmark functions, high dimensional benchmark functions, 13 engineering case studies, four data classifications, and three function approximation problems. The benchmark functions are mostly taken from the CEC 2005 and CEC 2010 special sessions, and they include rotated, shifted functions. The engineering case studies are from the CEC 2020 real-world non-convex constrained optimization problems. The performance of the proposed GWO is compared with popular metaheuristics, namely, particle swarm optimization (PSO), gravitational search algorithm (GSA), slap swarm algorithm (SSA), differential evolution (DE), self-adaptive differential evolution (SADE), basic GWO and its three recently improved variants. Statistical analysis and Friedman tests have been conducted to thoroughly compare their performance. The obtained results demonstrate that the proposed GWO outperforms the algorithms compared for the benchmark functions and engineering case studies tested.

Convergence analysis of an ALF-based nonconvex splitting algorithm with SQP structure

&lt;p style='text-indent:20px;'&gt;In this paper, by combining the splitting method of augmented Lagrange function (ALF) with the sequential quadratic programming (SQP) approximation, a novel ALF-based splitting algorithm with SQP structure is proposed for multi-block linear constrained nonconvex separable optimization. The new algorithm uses ALF-based splitting idea to decompose the original problem into several small-scale subproblems. Meanwhile, the SQP approximation and Armijo-type line search are used to solve some subproblems with smoothness concurrently. Under the conventional weak hypothesis, the decreasing property of ALF as merit function is obtained. Furthermore, the global convergence, strong convergence and convergence rate results of the new algorithm in general sense are given.&lt;/p&gt;

PANTR: A Proximal Algorithm With Trust-Region Updates for Nonconvex Constrained Optimization

This work presents PANTR, an efficient solver for nonconvex constrained optimization problems, that is well-suited as an inner solver for an augmented Lagrangian method. The proposed scheme combines forward-backward iterations with solutions to trust-region subproblems: the former ensures global convergence, whereas the latter enables fast update directions. We discuss how the algorithm is able to exploit exact Hessian information of the smooth objective term through a linear Newton approximation, while benefiting from the structure of box-constraints or ℓ1-regularization. An open-source C++ implementation of PANTR is made available as part of the NLP solver library ALPAQA. Finally, the effectiveness of the proposed method is demonstrated in nonlinear model predictive control applications.