Feasibility Violation Research Articles

A unified differential equation solver approach for separable convex optimization: Splitting, acceleration and nonergodic rate

This paper provides a self-contained ordinary differential equation solver approach for separable convex optimization problems. A novel primal-dual dynamical system with built-in time rescaling factors is introduced, and the exponential decay of a tailored Lyapunov function is established. Then several time discretizations of the continuous model are considered and analyzed via a unified discrete Lyapunov function. Moreover, two families of accelerated primal-dual methods are obtained, and nonergodic optimal mixed-type convergence rates shall be proved for the primal objective residual, the feasibility violation and the Lagrangian gap. Finally, numerical experiments are provided to validate the practical performances of the proposed methods.

Accelerated quadratic penalty dynamic approaches with applications to distributed optimization.

In this paper, we explore accelerated continuous-time dynamic approaches with a vanishing damping α/t, driven by a quadratic penalty function designed for linearly constrained convex optimization problems. We replace these linear constraints with penalty terms incorporated into the objective function, where the penalty coefficient grows to +∞ as t tends to infinity. With appropriate penalty coefficients, we establish convergence rates of O(1/tmin{2α/3,2}) for the objective residual and the feasibility violation when α>0, and demonstrate the robustness of these convergence rates against external perturbation. Furthermore, we apply the proposed dynamic approach to three distributed optimization problems: a distributed constrained consensus problem, a distributed extended monotropic optimization, and a distributed optimization with separated equations, resulting in three variant distributed dynamic approaches. Numerical examples are provided to show the effectiveness of the proposed quadratic penalty dynamic approaches.

Fast convergence rates and trajectory convergence of a Tikhonov regularized inertial primal–dual dynamical system with time scaling and vanishing damping

A Tikhonov regularized inertial primal-dual dynamical system with time scaling and vanishing damping is proposed for solving a linearly constrained convex optimization problem in Hilbert spaces. The system under consideration consists of two coupled second order differential equations and its convergence properties depend upon the decaying speed of the product of the time scaling parameter and the Tikhonov regularization parameter (named the rescaled regularization parameter) to zero. When the rescaled regularization parameter converges slowly to zero, the generated primal trajectory converges strongly to the minimal norm solution of the problem under suitable conditions. When the rescaled regularization parameter converges rapidly to zero, the system enjoys fast convergence rates in the primal–dual gap, the feasibility violation, the objective residual, and the gradient norm of the objective function along the trajectory, and the weak convergence of the trajectory to a primal–dual solution of the linearly constrained convex optimization problem. Finally, numerical experiments are performed to illustrate the theoretical findings.

Tikhonov regularized second-order plus first-order primal-dual dynamical systems with asymptotically vanishing damping for linear equality constrained convex optimization problems

In this paper, in the setting of Hilbert spaces, we consider a Tikhonov regularized second-order plus first-order primal-dual dynamical system with asymptotically vanishing damping for a linear equality constrained convex optimization problem. It is shown that convergence properties of the proposed inertial dynamical system depend upon the choice of the Tikhonov regularization parameter. When the Tikhonov regularization parameter decays rapidly to zero, we derive the fast convergence rates of the primal-dual gap, the objective residual, the feasibility violation and the gradient norm of the objective function along the generated trajectory. When the Tikhonov regularization parameter decreases slowly to zero, we prove the strong convergence of the primal trajectory of the Tikhonov regularized dynamical system to the minimal norm solution of the linear equality constrained convex optimization problem. Numerical experiments are performed to illustrate the efficiency of our approach.

Proximal Alternating Direction Method of Multipliers with Convex Combination Proximal Centers

Proximal alternating direction method of multipliers (PADMM) is a classical primal-dual splitting method for solving separable convex optimization problems with linear equality constraints, which have numerous applications in, e.g., signal and image processing, machine learning, and statistics. In this paper, we propose a new variant of PADMM, called PADMC, whose proximal centers are constructed by convex combinations of the iterates. PADMC is able to take advantage of problem structures and preserves the desirable properties of the classical PADMM. We establish iterate convergence as well as [Formula: see text] ergodic and [Formula: see text] nonergodic sublinear convergence rate results measured by function residual and feasibility violation, where [Formula: see text] denotes the iteration number. Moreover, we propose two fast variants of PADMC, one achieves faster [Formula: see text] ergodic convergence rate when one of the component functions is strongly convex, and the other ensures faster [Formula: see text] nonergodic convergence rate measured by constraint violation. Finally, preliminary numerical results on the LASSO and the elastic-net regularization problems are presented to demonstrate the performance of the proposed methods.

Convergence rates of mixed primal-dual dynamical systems with Hessian driven damping

For a linear equality constrained convex optimization problem, we initially propose a mixed primal-dual dynamical system with Hessian driven damping. This dynamical system comprises a second-order ordinary differential equation (ODE) with damping and Hessian driven damping for the primal variable, and a first-order ODE for the dual variable. Utilizing the Lyapunov analysis approach, we prove that all the Lagrangian residual, the objective residual and the feasibility violation enjoy fast convergence under general scaling coefficients. We further analyse the convergence rate results under specific scaling coefficients. Our mixed dynamical system is extended to solve multi-block optimization problems, and we also consider the discrete case of the mixed dynamical system. This is the first study to explore primal-dual dynamical systems with Hessian driven damping for linear equality constrained problems.

Mini-batch stochastic subgradient for functional constrained optimization

In this paper, we consider finite sum composite optimization problems with many functional constraints. The objective function is expressed as a finite sum of two terms, one of which admits easy computation of (sub)gradients while the other is amenable to proximal evaluations. We assume a generalized bounded gradient condition on the objective which allows us to simultaneously tackle both smooth and nonsmooth problems. We also consider the cases of both with and without a quadratic functional growth property. Further, we assume that each constraint set is given as the level set of a convex but not necessarily a differentiable function. We reformulate the constrained finite sum problem into a stochastic optimization problem for which the stochastic subgradient projection method from Necoara and Singh [Stochastic subgradient projection methods for composite optimization with functional constraints; 2022 Journal of Machine Learning Research, 23, 1–35] specializes in a collection of mini-batch variants, with different mini-batch sizes for the objective function and functional constraints, respectively. More specifically, at each iteration, our algorithm takes a mini-batch stochastic proximal subgradient step aimed at minimizing the objective function and then a subsequent mini-batch subgradient projection step minimizing the feasibility violation. By specializing in different mini-batching strategies, we derive exact expressions for the stepsizes as a function of the mini-batch size and in some cases we also derive insightful stepsize-switching rules which describe when one should switch from a constant to a decreasing stepsize regime. We also prove sublinear convergence rates for the mini-batch subgradient projection algorithm which depend explicitly on the mini-batch sizes and on the properties of the objective function. Numerical results also show a better performance of our mini-batch scheme over its single-batch counterpart.

Consensus-based Dantzig-Wolfe decomposition

Dantzig-Wolfe decomposition (DWD) is a classical algorithm for solving large-scale linear programs whose constraint matrix involves a set of independent blocks coupled with a set of linking rows. The algorithm decomposes such a model into a master problem and a set of independent subproblems that can be solved in a distributed manner. In a typical implementation, the master problem is solved centrally. In certain settings, solving the master problem centrally is undesirable or infeasible, such as in the case of decentralized storage of data, or when independent agents who are responsible for the subproblems desire privacy of information. In this paper, we propose a fully distributed DWD algorithm which relies on solving the master problem using a consensus-based Alternating Direction Method of Multipliers (ADMM) method. We derive error bounds on the optimality gap and feasibility violation of the proposed approach. We provide preliminary computational results for our algorithm using a Message Passing Interface implementation on a delivery planning problem, the multi-commodity network flow problem, and synthetic instances where we obtain high quality solutions. An open-source implementation of the algorithm is available.

A new randomized primal-dual algorithm for convex optimization with fast last iterate convergence rates

We develop a novel unified randomized block-coordinate primal-dual algorithm to solve a class of nonsmooth constrained convex optimization problems, which covers different existing variants and model settings from the literature. We prove that our algorithm achieves and convergence rates in two cases: merely convexity and strong convexity, respectively, where k is the iteration counter and n is the number of block-coordinates. These rates are known to be optimal (up to a constant factor) when n = 1. Our convergence rates are obtained through three criteria: primal objective residual and primal feasibility violation, dual objective residual, and primal-dual expected gap. Moreover, our rates for the primal problem are on the last-iterate sequence. Our dual convergence guarantee requires additionally a Lipschitz continuity assumption. We specify our algorithm to handle two important special cases, where our rates are still applied. Finally, we verify our algorithm on two well-studied numerical examples and compare it with two existing methods. Our results show that the proposed method has encouraging performance on different experiments.

Fast primal–dual algorithm via dynamical system for a linearly constrained convex optimization problem

By time discretization of a second-order primal–dual dynamical system with damping α/t where an inertial construction in the sense of Nesterov is needed only for the primal variable, we propose a fast primal–dual algorithm for a linear equality constrained convex optimization problem. Under a suitable scaling condition, we show that the proposed algorithm enjoys a fast convergence rate for the objective residual and the feasibility violation, and the decay rate can reach O(1/kα−1) at the most. We also study convergence properties of the corresponding primal–dual dynamical system to better understand the acceleration scheme. Finally, we report numerical experiments to demonstrate the effectiveness of the proposed algorithm.

Faster Lagrangian-Based Methods in Convex Optimization

In this paper, we aim at unifying, simplifying, and improving the convergence rate analysis of Lagrangian-based methods for convex optimization problems. We first introduce the notion of nice primal algorithmic map, which plays a central role in the unification and in the simplification of the analysis of most Lagrangian-based methods. Equipped with a nice primal algorithmic map, we then introduce a versatile generic scheme, which allows for the design and analysis of faster Lagrangian (FLAG) methods with new provably sublinear rate of convergence expressed in terms of function values and feasibility violation of the original (nonergodic) generated sequence. To demonstrate the power and versatility of our approach and results, we show that most well-known iconic Lagrangian-based schemes admit a nice primal algorithmic map and hence share the new faster rate of convergence results within their corresponding FLAG.

Graph neural network-based resource allocation strategies for multi-object spectroscopy

Resource allocation problems are often approached with linear programming techniques. But many concrete allocation problems in the experimental and observational sciences cannot or should not be expressed in the form of linear objective functions. Even if the objective is linear, its parameters may not be known beforehand because they depend on the results of the experiment for which the allocation is to be determined. To address these challenges, we present a bipartite graph neural network (GNN) architecture for trainable resource allocation strategies. Items of value and constraints form the two sets of graph nodes, which are connected by edges corresponding to possible allocations. The GNN is trained on simulations or past problem occurrences to maximize any user-supplied, scientifically motivated objective function, augmented by an infeasibility penalty. The amount of feasibility violation can be tuned in relation to any available slack in the system. We apply this method to optimize the astronomical target selection strategy for the highly multiplexed Subaru Prime Focus Spectrograph instrument, where it shows superior results to direct gradient descent optimization and extends the capabilities of the currently employed solver which uses linear objective functions. The development of this method enables fast adjustment and deployment of allocation strategies, statistical analyses of allocation patterns, and fully differentiable, science-driven solutions for resource allocation problems.

A competitive inexact nonmonotone filter SQP method: convergence analysis and numerical results

We propose an inexact nonmonotone successive quadratic programming (SQP) algorithm for solving nonlinear programming problems with equality constraints and bounded variables. Regarding the value of the current feasibility violation and the minimum value of its linear approximation over a trust region, several scenarios are envisaged. In one scenario, a possible infeasible stationary point is detected. In other scenarios, the search direction is computed using an inexact (truncated) solution of a feasible strictly convex quadratic program (QP). The search direction is shown to be a descent direction for the objective function or the feasibility violation in the feasible or infeasible iterations, respectively. A new penalty parameter updating formula is proposed to turn the search direction into a descent direction for an -penalty function. In certain iterations, an accelerator direction is developed to obtain a superlinear local convergence rate of the algorithm. Using a nonmonotone filter strategy, the global convergence of the algorithm and a superlinear local rate of convergence are guaranteed. The main advantage of the algorithm is that the global convergence of the algorithm is established using inexact solutions of the QPs. Furthermore, the use of inexact solutions instead of exact solutions of the subproblems enhances the robustness and efficiency of the algorithm. The algorithm is implemented using MATLAB and the program is tested on a wide range of test problems from the CUTEst library. Comparison of the obtained numerical results with those obtained by testing some similar SQP algorithms affirms the efficiency and robustness of the proposed algorithm.

Random Minibatch Subgradient Algorithms for Convex Problems with Functional Constraints

In this paper we consider non-smooth convex optimization problems with (possibly) infinite intersection of constraints. In contrast to the classical approach, where the constraints are usually represented as intersection of simple sets, which are easy to project onto, in this paper we consider that each constraint set is given as the level set of a convex but not necessarily differentiable function. For these settings we propose subgradient iterative algorithms with random minibatch feasibility updates. At each iteration, our algorithms take a subgradient step aimed at only minimizing the objective function and then a subsequent subgradient step minimizing the feasibility violation of the observed minibatch of constraints. The feasibility updates are performed based on either parallel or sequential random observations of several constraint components. We analyze the convergence behavior of the proposed algorithms for the case when the objective function is strongly convex and with bounded subgradients, while the functional constraints are endowed with a bounded first-order black-box oracle. For a diminishing stepsize, we prove sublinear convergence rates for the expected distances of the weighted averages of the iterates from the constraint set, as well as for the expected suboptimality of the function values along the weighted averages. Our convergence rates are known to be optimal for subgradient methods on this class of problems. Moreover, the rates depend explicitly on the minibatch size and show when minibatching helps a subgradient scheme with random feasibility updates.

Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming

Augmented Lagrangian method (ALM) has been popularly used for solving constrained optimization problems. Practically, subproblems for updating primal variables in the framework of ALM usually can only be solved inexactly. The convergence and local convergence speed of ALM have been extensively studied. However, the global convergence rate of the inexact ALM is still open for problems with nonlinear inequality constraints. In this paper, we work on general convex programs with both equality and inequality constraints. For these problems, we establish the global convergence rate of the inexact ALM and estimate its iteration complexity in terms of the number of gradient evaluations to produce a primal and/or primal-dual solution with a specified accuracy. We first establish an ergodic convergence rate result of the inexact ALM that uses constant penalty parameters or geometrically increasing penalty parameters. Based on the convergence rate result, we then apply Nesterov’s optimal first-order method on each primal subproblem and estimate the iteration complexity of the inexact ALM. We show that if the objective is convex, then $$O(\varepsilon ^{-1})$$ gradient evaluations are sufficient to guarantee a primal $$\varepsilon $$ -solution in terms of both primal objective and feasibility violation. If the objective is strongly convex, the result can be improved to $$O(\varepsilon ^{-\frac{1}{2}}|\log \varepsilon |)$$ . To produce a primal-dual $$\varepsilon $$ -solution, more gradient evaluations are needed for convex case, and the number is $$O(\varepsilon ^{-\frac{4}{3}})$$ , while for strongly convex case, the number is still $$O(\varepsilon ^{-\frac{1}{2}}|\log \varepsilon |)$$ . Finally, we establish a nonergodic convergence rate result of the inexact ALM that uses geometrically increasing penalty parameters. This result is established only for the primal problem. We show that the nonergodic iteration complexity result is in the same order as that for the ergodic result. Numerical experiments on quadratically constrained quadratic programming are conducted to compare the performance of the inexact ALM with different settings.

Randomized Primal–Dual Proximal Block Coordinate Updates

In this paper, we propose a randomized primal–dual proximal block coordinate updating framework for a general multi-block convex optimization model with coupled objective function and linear constraints. Assuming mere convexity, we establish its O(1 / t) convergence rate in terms of the objective value and feasibility measure. The framework includes several existing algorithms as special cases such as a primal–dual method for bilinear saddle-point problems (PD-S), the proximal Jacobian alternating direction method of multipliers (Prox-JADMM) and a randomized variant of the ADMM for multi-block convex optimization. Our analysis recovers and/or strengthens the convergence properties of several existing algorithms. For example, for PD-S our result leads to the same order of convergence rate without the previously assumed boundedness condition on the constraint sets, and for Prox-JADMM the new result provides convergence rate in terms of the objective value and the feasibility violation. It is well known that the original ADMM may fail to converge when the number of blocks exceeds two. Our result shows that if an appropriate randomization procedure is invoked to select the updating blocks, then a sublinear rate of convergence in expectation can be guaranteed for multi-block ADMM, without assuming any strong convexity. The new approach is also extended to solve problems where only a stochastic approximation of the subgradient of the objective is available, and we establish an $$O(1/\sqrt{t})$$ convergence rate of the extended approach for solving stochastic programming.

A Priori Error Estimates for State-Constrained Semilinear Parabolic Optimal Control Problems

We consider the finite element discretization of semilinear parabolic optimization problems subject to pointwise in time constraints on mean values of the state variable. In order to control the feasibility violation induced by the discretization, error estimates for the semilinear partial differential equation are derived. Based upon these estimates, it can be shown that any local minimizer of the semilinear parabolic optimization problems satisfying a weak second-order sufficient condition can be approximated by the discretized problem. Rates for this convergence in terms of temporal and spatial discretization mesh sizes are provided. In contrast to other results in numerical analysis of optimization problems subject to semilinear parabolic equations, the analysis can work with a weak second-order condition, requiring growth of the Lagrangian in critical directions only. The analysis can then be conducted relying solely on the resulting quadratic growth condition of the continuous problem, without the need for similar assumptions on the discrete or time semidiscrete setting.

Stochastic Accelerated Alternating Direction Method of Multipliers with Importance Sampling

In this paper, we incorporate importance sampling strategy into accelerated framework of stochastic alternating direction method of multipliers for solving a class of stochastic composite problems with linear equality constraint. The rates of convergence for primal residual and feasibility violation are established. Moreover, the estimation of variance of stochastic gradient is improved due to the use of important sampling. The proposed algorithm is capable of dealing with the situation, where the feasible set is unbounded. The experimental results indicate the effectiveness of the proposed method.

Convergence Rate of Distributed ADMM Over Networks

We propose a new distributed algorithm based on alternating direction method of multipliers (ADMM) to minimize sum of locally known convex functions using communication over a network. This optimization problem emerges in many applications in distributed machine learning and statistical estimation. Our algorithm allows for a general choice of the communication weight matrix, which is used to combine the iterates at different nodes. We show that when functions are convex, both the objective function values and the feasibility violation converge with rate $O(1/T)$ , where $T$ is the number of iterations. We then show that when functions are strongly convex and have Lipschitz continuous gradients, the sequence generated by our algorithm converges linearly to the optimal solution. In particular, an $\epsilon$ -optimal solution can be computed with $O\left(\sqrt{\kappa _f} \log (1/\epsilon) \right)$ iterations, where $\kappa _f$ is the condition number of the problem. Our analysis highlights the effect of network and communication weights on the convergence rate through degrees of the nodes, the smallest nonzero eigenvalue, and operator norm of the communication matrix.

On fully distributed dual first order methods for convex network optimization

This paper provides distributed optimization methods to carry out fully distributed computations in convex network optimization problems. The network constraints are moved into the cost using the Lagrange multipliers. For solving the dual problem we propose a fully distributed dual fast gradient algorithm having sublinear rate of convergence in terms of dual suboptimality but also in terms of primal suboptimality and feasibility violation for the last generated primal iterate. The convergence analysis combines the Lipschitz property of dual function with fast gradient schemes. The main application we consider is the DC optimal power flow problem from an electric energy system. Our approach allows to carry out an optimal dispatch for each bus to be done independently and in parallel while still achieving the global economical optimum of the whole electric system. The newly developed algorithms can be run distributively, as we show on several numerical simulations using classical IEEE bus test cases.