Inertial Accelerated Primal–Dual Algorithms for Non-smooth Convex Optimization Problems with Linear Equality Constraints

A recurrent neural network is proposed for solving the non-smooth convex optimization problem with the convex inequality and linear equality constraints. Since the objective function and inequality constraints may not be smooth, the Clarke's generalized gradients of the objective function and inequality constraints are employed to describe the dynamics of the proposed neural network. It is proved that the equilibrium point set of the proposed neural network is equivalent to the optimal solution of the original optimization problem by using the Lagrangian saddle-point theorem. Under weak conditions, the proposed neural network is proved to be stable, and the state of the neural network is convergent to one of its equilibrium points. Compared with the existing neural network models for non-smooth optimization problems, the proposed neural network can deal with a larger class of constraints and is not based on the penalty method. Finally, the proposed neural network is used to solve the identification problem of genetic regulatory networks, which can be transformed into a non-smooth convex optimization problem. The simulation results show the satisfactory identification accuracy, which demonstrates the effectiveness and efficiency of the proposed approach.

In this paper, a one-layer recurrent neural network is proposed for solving non-smooth convex optimization problems with linear equality constraints. Comparing with the existing neural networks, the proposed neural network has simpler architecture and the number of neurons is the same as that of decision variables in the optimization problems. The global convergence of the neural network can be guaranteed if the non-smooth objective function is convex. Simulation results are provided to show that the state trajectories of the neural network can converge to the optimal solutions of the non-smooth convex optimization problems and show the performance of the proposed neural network.KeywordsNeural NetworkIEEE TransactionGlobal ConvergenceRecurrent Neural NetworkState TrajectoryThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, a two-layer recurrent neural network is proposed to solve the nonsmooth convex optimization problem subject to convex inequality and linear equality constraints. Compared with existing neural network models, the proposed neural network has a low model complexity and avoids penalty parameters. It is proved that from any initial point, the state of the proposed neural network reaches the equality feasible region in finite time and stays there thereafter. Moreover, the state is unique if the initial point lies in the equality feasible region. The equilibrium point set of the proposed neural network is proved to be equivalent to the Karush-Kuhn-Tucker optimality set of the original optimization problem. It is further proved that the equilibrium point of the proposed neural network is stable in the sense of Lyapunov. Moreover, from any initial point, the state is proved to be convergent to an equilibrium point of the proposed neural network. Finally, as applications, the proposed neural network is used to solve nonlinear convex programming with linear constraints and L1 -norm minimization problems.

This chapter is devoted to the blackbox subgradient algorithms with the minimal requirements for the storage of auxiliary results, which are necessary to execute these algorithms. To provide historical perspective this survey starts with the original result of Shor which opened this field with the application to the classical transportation problem. The theoretical complexity bounds for smooth and nonsmooth convex and quasiconvex optimization problems are briefly exposed in what follows to introduce the relevant fundamentals of nonsmooth optimization. Special attention in this section is given to the adaptive step size policy which aims to attain lowest complexity bounds. Nondifferentiability of objective function in convex optimization significantly slows down the rate of convergence in subgradient optimization compared to the smooth case, but there are different modern techniques that allow to solve nonsmooth convex optimization problems faster than dictate theoretical lower complexity bounds. In this work the particular attention is given to Nesterov smoothing technique, Nesterov universal approach, and Legendre (saddle point) representation approach. The new results on universal mirror prox algorithms represent the original parts of the survey. To demonstrate application of nonsmooth convex optimization algorithms to solution of huge-scale extremal problems we consider convex optimization problems with nonsmooth functional constraints and propose two adaptive mirror descent methods. The first method is of primal-dual variety and proved to be optimal in terms of lower oracle bounds for the class of Lipschitz continuous convex objectives and constraints. The advantages of application of this method to the sparse truss topology design problem are discussed in essential details. The second method can be used for solution of convex and quasiconvex optimization problems and it is optimal in terms of complexity bounds. The conclusion part of the survey contains the important references that characterize recent developments of nonsmooth convex optimization.

Extensions of ADMM for separable convex optimization problems with linear equality or inequality constraints

In this paper we first establish a Lagrange multiplier condition characterizing a regularized Lagrangian duality for quadratic minimization problems with finitely many linear equality and quadratic inequality constraints, where the linear constraints are not relaxed in the regularized Lagrangian dual. In particular, in the case of a quadratic optimization problem with a single quadratic inequality constraint such as the linearly constrained trust-region problems, we show that the Slater constraint qualification (SCQ) is necessary and sufficient for the regularized Lagrangian duality in the sense that the regularized duality holds for each quadratic objective function over the constraints if and only if (SCQ) holds. A new theorem of the alternative for systems involving both equality constraints and two quadratic inequality constraints plays a key role. We also provide classes of quadratic programs, including a class of CDT-subproblems with linear equality constraints, where (SCQ) ensures regularized Lagrangian duality.

For a class of nonsmooth composite optimization problems with linear equality constraints, we utilize a Lyapunov-based approach to establish the global exponential stability of the primal-dual gradient flow dynamics based on the proximal augmented Lagrangian. The result holds when the differentiable part of the objective function is strongly convex with a Lipschitz continuous gradient; the non-differentiable part is proper, lower semi-continuous, and convex; and the matrix in the linear constraint is full row rank. Our quadratic Lyapunov function generalizes recent result from strongly convex problems with either affine equality or inequality constraints to a broader class of composite optimization problems with nonsmooth regularizers and it provides a worst-case lower bound of the exponential decay rate. Finally, we use computational experiments to demonstrate that our convergence rate estimate is less conservative than the existing alternatives.

An extended alternating direction method for variational inequality problems with linear equality and inequality constraints

We consider the problem of non-smooth convex optimization with linear equality constraints, where the objective function is only accessible through its proximal operator. This problem arises in many different fields such as statistical learning, computational imaging, telecommunications, and optimal control. To solve it, we propose an Anderson accelerated Douglas-Rachford splitting (A2DR) algorithm, which we show either globally converges or provides a certificate of infeasibility/unboundedness under very mild conditions. Applied to a block separable objective, A2DR partially decouples so that its steps may be carried out in parallel, yielding an algorithm that is fast and scalable to multiple processors. We describe an open-source implementation and demonstrate its performance on a wide range of examples.

A faster augmented Lagrangian method (Faster ALM) with constant inertial parameters for solving convex optimization problems with linear equality constraint is presented in this paper. The proposed faster ALM exhibits linear convergence rates in non-ergodic (the last iteration) sense of the Lagrangian primal–dual gap, the objective function value and the feasibility measure. In addition, we prove that the sequence generated by the faster ALM converges to a saddle point of the linear equality constrained optimization problem. This is the first result that provides both linear non-ergodic convergence rate and the convergence of the iterative sequence when solving linearly constrained convex optimization problems by inertial ALM-type algorithms without additional assumptions such as strong convexity or Lipschitz continuous of gradient. As an extension, we present contracting ALM for solving convex optimization problems with linear equality constraint with O ( 1 / ∑ i = 0 k a i ) ( a i > 0 is an arbitrarily constant) non-ergodic convergence rates of the Lagrangian primal–dual gap, the objective function value and the feasibility measure.

A subgradient-based neurodynamic algorithm to constrained nonsmooth nonconvex interval-valued optimization

This paper deals with robust ε-quasi optimal solutions for a class of nonsmooth optimization problems with uncertain data. Under some mild assumptions, we first establish, by using robust optimization (i.e. worst-case) approach, approximate optimality conditions for this uncertain nonsmooth optimization problem. Then, we introduce a Mixed-type robust approximate dual problem of this uncertain optimization problem, and explore their relationships. Moreover, using a scalarization method, we derive optimality conditions for robust weakly approximate efficient solutions for an uncertain nonsmooth multiobjective optimization problem. We also obtain approximate duality theorems for the uncertain nonsmooth multiobjective optimization problem.

In this paper, we consider the problem of distributed optimisation of a separable convex cost function over a graph, where every edge and node in the graph could carry both linear equality and/or inequality constraints. We show how to modify the primal-dual method of multipliers (PDMM), originally designed for linear equality constraints, such that it can handle inequality constraints as well. The proposed algorithm does not need any slack variables, which is similar to the recent work [1] which extends the alternating direction method of multipliers (ADMM) for addressing decomposable optimisation with linear equality and inequality constraints. Using convex analysis, monotone operator theory and fixed-point theory, we show how to derive the update equations of the modified PDMM algorithm by applying Peaceman-Rachford splitting to the monotonic inclusion related to the lifted dual problem. To incorporate the inequality constraints, we impose a non-negativity constraint on the associated dual variables. This additional constraint results in the introduction of a reflection operator to model the data exchange in the network, instead of a permutation operator as derived for equality constraint PDMM. Convergence for both synchronous and stochastic update schemes of PDMM are provided. The latter includes asynchronous update schemes and update schemes with transmission losses. Experiments show that PDMM converges notably faster than extended ADMM of [1].

LSEQIEQ: a FORTRAN IV subroutine package for the analysis of multiple linear regression problems with possibly deficient pseudorank and linear equality and inequality constraints

The bilevel programming problem (BLP) is an optimization problem with another optimization problem in its constraints. This framework finds utility in modeling decentralized scenarios, which arise in real-world applications such as traffic management, transportation, and economic policy. Differential Evolution (DE) techniques have emerged in literature for addressing such complex problems. However, handling linear equality constraints poses a significant challenge for DE and other metaheuristics. To address this issue, we previously introduced DELEqC, enhancing DE with a mechanism to manipulate the linear equality constraints. A specialized variant, BL-DELEqC, was further proposed specifically for tackling general BLPs. Another variant, DELEqC-III, transforms the original constrained optimization problem into a lower-dimensional unconstrained one, offering applicability to BLPs with linear equality constraints. Thus, we explore in this study the efficacy of DELEqC-III in handling BLPs with linear equality constraints. The proposed BL-DELEqC-III is compared to BL-DELEqC on a selection of benchmark BLPs, demonstrating superior results.