Primal Variables Research Articles

Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming

Augmented Lagrangian method (ALM) has been popularly used for solving constrained optimization problems. Practically, subproblems for updating primal variables in the framework of ALM usually can only be solved inexactly. The convergence and local convergence speed of ALM have been extensively studied. However, the global convergence rate of the inexact ALM is still open for problems with nonlinear inequality constraints. In this paper, we work on general convex programs with both equality and inequality constraints. For these problems, we establish the global convergence rate of the inexact ALM and estimate its iteration complexity in terms of the number of gradient evaluations to produce a primal and/or primal-dual solution with a specified accuracy. We first establish an ergodic convergence rate result of the inexact ALM that uses constant penalty parameters or geometrically increasing penalty parameters. Based on the convergence rate result, we then apply Nesterov’s optimal first-order method on each primal subproblem and estimate the iteration complexity of the inexact ALM. We show that if the objective is convex, then $$O(\varepsilon ^{-1})$$ gradient evaluations are sufficient to guarantee a primal $$\varepsilon $$ -solution in terms of both primal objective and feasibility violation. If the objective is strongly convex, the result can be improved to $$O(\varepsilon ^{-\frac{1}{2}}|\log \varepsilon |)$$ . To produce a primal-dual $$\varepsilon $$ -solution, more gradient evaluations are needed for convex case, and the number is $$O(\varepsilon ^{-\frac{4}{3}})$$ , while for strongly convex case, the number is still $$O(\varepsilon ^{-\frac{1}{2}}|\log \varepsilon |)$$ . Finally, we establish a nonergodic convergence rate result of the inexact ALM that uses geometrically increasing penalty parameters. This result is established only for the primal problem. We show that the nonergodic iteration complexity result is in the same order as that for the ergodic result. Numerical experiments on quadratically constrained quadratic programming are conducted to compare the performance of the inexact ALM with different settings.

Model order reduction with Galerkin projection applied to nonlinear optimization with infeasible primal‐dual interior point method

SummaryIt is not new that model order reduction (MOR) methods are employed in almost all fields of engineering to reduce the processing time of complex computational simulations. At the same time, interior point methods (IPMs), a method to deal with inequality constraint problems (which is little explored in engineering), can be applied in many fields such as plasticity theory, contact mechanics, micromechanics, and topology optimization. In this work, a MOR based in Galerkin projection is coupled with the infeasible primal‐dual IPM. Such research concentrates on how to develop a Galerkin projection in one field with the interior point method; the combination of both methods, coupled with Schur complement, permits to solve this MOR similar to problems without constraints, leading to new approaches to adaptive strategies. Moreover, this research develops an analysis of error from the Galerkin projection related to the primal and dual variables. Finally, this work also suggests an adaptive strategy to alternate the Galerkin projection operator, between primal and dual variable, according to the error during the processing of a problem.

The Proximal Augmented Lagrangian Method for Nonsmooth Composite Optimization

We study a class of optimization problems in which the objective function is given by the sum of a differentiable but possibly nonconvex component and a nondifferentiable convex regularization term. We introduce an auxiliary variable to separate the objective function components and utilize the Moreau envelope of the regularization term to derive the proximal augmented Lagrangian $-$ a continuously differentiable function obtained by constraining the augmented Lagrangian to the manifold that corresponds to the explicit minimization over the variable in the nonsmooth term. The continuous differentiability of this function with respect to both primal and dual variables allows us to leverage the method of multipliers (MM) to compute optimal primal-dual pairs by solving a sequence of differentiable problems. The MM algorithm is applicable to a broader class of problems than proximal gradient methods and it has stronger convergence guarantees and a more refined step-size update rules than the alternating direction method of multipliers. These features make it an attractive option for solving structured optimal control problems. We also develop an algorithm based on the primal-descent dual-ascent gradient method and prove global (exponential) asymptotic stability when the differentiable component of the objective function is (strongly) convex and the regularization term is convex. Finally, we identify classes of problems for which the primal-dual gradient flow dynamics are convenient for distributed implementation and compare/contrast our framework to the existing approaches.

An Approximate Penalty Method with Descent for Convex Optimization Problems

We propose a penalty method for general convex constrained optimization problems, where each auxiliary penalized problem is solved approximately with a special composite descent method. Direction finding choice in this method is found with the help of an equivalent equilibrium type problem. This allows one to keep the complete structure of the initial problem, although without nonlinear constraints and to simply calculate the descent direction in separable problems. Convergence of the method in primal and dual variables is established under rather weak assumptions.

Lagrangian Relaxation With Incremental Proximal Method for Economic Dispatch With Large Numbers of Wind Power Scenarios

Scenario-based approaches have been widely employed in stochastic economic dispatch with wind power integration. However, due to computational limitations, the number of wind power scenarios usually need to be reduced to a small subset of all available scenarios. The limited number of scenarios could affect the objective performance related to the correlations and uncertainties of wind power, especially when the influence of extreme scenarios needs to be accounted for. To increase the computational efficiency of economic dispatch problem with a large number of scenarios, we propose a Lagrangian relaxation with incremental proximal method to solve the dispatch problem. This approach combines the benefit of Lagrangian relaxation, which allows the problem to be decomposed into a large number of smaller problems and the proximal method, which lead to much faster convergence. In addition, we also propose a multipliers and primal variables initialization method to further reduce the convergence time. Results of case studies show that this framework greatly reduces the computation time and the system cost of economic dispatch compared with the traditional approaches.

Decoding Binary Linear Codes Using Penalty Dual Decomposition Method

In this letter, we utilize the penalty dual decomposition (PDD) framework and develop a novel PDD decoding algorithm for binary linear codes. Instead of relaxing the discrete constraints to continuous ones, we take an alternative by transforming them into equivalent equality constraints. This idea leads to a double-loop parallel algorithm: In the outer loop, we update the dual variables and certain penalty parameters, while in the inner loop, we divide the primal variables into several blocks and employ the block successive upper-bound minimization method to iteratively optimize each block variable in closed form. Every limit point generated by the proposed algorithm is guaranteed to be a stationary point of the maximum likelihood decoding problem. Simulation results demonstrate that the proposed algorithm shows great error rate performance at both low and high signal-to-noise ratios.

Distributed GNE Seeking Under Partial-Decision Information Over Networks via a Doubly-Augmented Operator Splitting Approach

We consider distributed computation of generalized Nash equilibrium (GNE) over networks, in games with shared coupling constraints. Existing methods require that each player has full access to opponents' decisions. In this paper, we assume that players have only partial-decision information, and can communicate with their neighbours over an arbitrary undirected graph. We recast the problem as that of finding a zero of a sum of monotone operators through primal-dual analysis. To distribute the problem, we doubly augment variables, so that each player has local decision estimates and local copies of Lagrangian multipliers. We introduce a single-layer algorithm, fully distributed with respect to both primal and dual variables. We show its convergence to a variational GNE with fixed step-sizes, by reformulating it as a forward-backward iteration for a pair of doubly-augmented monotone operators.

Enriched three-field numerical manifold formulation for dynamics of fractured saturated porous media

In terms of the three-field (u-w-p) formulation of Biot’s theory of saturated porous media with incompressible solid and fluid phases, the numerical manifold method (NMM) models are developed to analyze the fully dynamic consolidation of fractured porous media in this study. The same approximation to fluid velocity and skeleton displacement is constructed which is capable of modeling both incompressible and compressible deformation, while two types of approximations to pore pressure field are established. Since the inertial effect of fluid is not neglected, the proposed model can fully capture the dynamic behavior of porous media, especially under the impact or high-frequency loading condition and, accordingly, exhibits apparent superiority in predicting transient and wave propagation responses of cracked porous media. Moreover, low order interpolation functions for primal variables and the most flexible three-node triangular finite element mesh are used, which both are difficult to implement using other partition of unity (PU) based numerical methods in terms of Biot’s two-field formulation. Also, the discontinuities can be modeled more naturally in the NMM framework in comparison with XFEM or PNM. Meanwhile, an augmented Lagrange multiplier method for stick–slip contact model is first incorporated into fully-dynamic three-field Biot’ formulation. In addition, a mass lumping technique within NMM framework, which turns out to be a unique advantage of NMM over other numerical methods, is employed to suppress unphysical oscillations and increase computational efficiency. Energy balance condition is employed to evaluate the stability and accuracy of the time integration scheme. The robustness and versatility of the proposed models are manifested with several typical examples.

Learning Optimal Resource Allocations in Wireless Systems

This paper considers the design of optimal resource allocation policies in wireless communication systems which are generically modeled as a functional optimization problem with stochastic constraints. These optimization problems have the structure of a learning problem in which the statistical loss appears as a constraint, motivating the development of learning methodologies to attempt their solution. To handle stochastic constraints, training is undertaken in the dual domain. It is shown that this can be done with small loss of optimality when using near-universal learning parameterizations. In particular, since deep neural networks (DNN) are near-universal their use is advocated and explored. DNNs are trained here with a model-free primal-dual method that simultaneously learns a DNN parametrization of the resource allocation policy and optimizes the primal and dual variables. Numerical simulations demonstrate the strong performance of the proposed approach on a number of common wireless resource allocation problems.

Sparse support vector machines with L0 approximation for ultra-high dimensional omics data.

Omics data usually have ultra-high dimension (p) and small sample size (n). Standard support vector machines (SVMs), which minimize the L2 norm for the primal variables, only lead to sparse solutions for the dual variables. L1 based SVMs, directly minimizing the L1 norm, have been used for feature selection with omics data. However, most current methods directly solve the primal formulations of the problem, which are not computationally scalable. The computational complexity increases with the number of features. In addition, L1 norm is known to be asymptotically biased and not consistent for feature selection. In this paper, we develop an efficient method for sparse support vector machines with L0 norm approximation. The proposed method approximates the L0 minimization through solving a series of L2 optimization problems, which can be formulated with dual variables. It finds the optimal solution for p primal variables through estimating n dual variables, which is more efficient as long as the sample size is small. L0 approximation leads to sparsity in both dual and primal variables, and can be used for both feature and sample selections. The proposed method identifies much less number of features and achieves similar performances in simulations. We apply the proposed method to feature selections with metagenomic sequencing and gene expression data. It can identify biologically important genes and taxa efficiently.

On relaxation of some customized proximal point algorithms for convex minimization: from variational inequality perspective

The proximal point algorithm (PPA) is a fundamental method for convex programming. When applying the PPA to solve linearly constrained convex problems, we may prefer to choose an appropriate metric matrix to define the proximal regularization, so that the computational burden of the resulted PPA can be reduced, and sometimes even admit closed form or efficient solutions. This idea results in the so-called customized PPA (also known as preconditioned PPA), and it covers the linearized ALM, the primal-dual hybrid gradient algorithm, ADMM as special cases. Since each customized PPA owes special structures and has popular applications, it is interesting to ask wether we can design a simple relaxation strategy for these algorithms. In this paper we treat these customized PPA algorithms uniformly by a mixed variational inequality approach, and propose a new relaxation strategy for these customized PPA algorithms. Our idea is based on correcting the dual variables individually and does not rely on relaxing the primal variables. This is very different from previous works. From variational inequality perspective, we prove the global convergence and establish a worst-case convergence rate for these relaxed PPA algorithms. Finally, we demonstrate the performance improvements by some numerical results.

Revisiting Approximate Linear Programming: Constraint-Violation Learning with Applications to Inventory Control and Energy Storage

Approximate linear programs (ALPs) are well-known models for computing value function approximations (VFAs) of intractable Markov decision processes (MDPs). VFAs from ALPs have desirable theoretical properties, define an operating policy, and provide a lower bound on the optimal policy cost. However, solving ALPs near-optimally remains challenging, for example, when approximating MDPs with nonlinear cost functions and transition dynamics or when rich basis functions are required to obtain a good VFA. We address this tension between theory and solvability by proposing a convex saddle-point reformulation of an ALP that includes as primal and dual variables, respectively, a vector of basis function weights and a constraint violation density function over the state-action space. To solve this reformulation, we develop a proximal stochastic mirror descent (PSMD) method that learns regions of high ALP constraint violation via its dual update. We establish that PSMD returns a near-optimal ALP solution and a lower bound on the optimal policy cost in a finite number of iterations with high probability. We numerically compare PSMD with several benchmarks on inventory control and energy storage applications. We find that the PSMD lower bound is tighter than a perfect information bound. In contrast, the constraint-sampling approach to solve ALPs may not provide a lower bound, and applying row generation to tackle ALPs is not computationally viable. PSMD policies outperform problem-specific heuristics and are comparable or better than the policies obtained using constraint sampling. Overall, our ALP reformulation and solution approach broadens the applicability of approximate linear programming.This paper was accepted by Yinyu Ye, optimization.

An accelerated primal-dual iterative scheme for the L2-TV regularized model of linear inverse problems

A model with the and total variational (TV) regularization terms for linear inverse problems is considered. The regularized model is reformulated as a saddle-point problem, and the primal and dual variables are discretized in the piecewise affine and piecewise constant finite element spaces, respectively. An accelerated primal-dual iterative scheme with an convergence rate is proposed for the discretized problem, where is the iteration counter. Both the regularization and perturbation errors of the regularized model, and the finite element discretization and iteration errors of the accelerated primal-dual scheme, are estimated. Preliminary numerical results are reported to show the efficiency of the proposed iterative scheme.

Block-proximal methods with spatially adapted acceleration

We study and develop (stochastic) primal--dual block-coordinate descent methods for convex problems based on the method due to Chambolle and Pock. Our methods have known convergence rates for the iterates and the ergodic gap: $O(1/N^2)$ if each block is strongly convex, $O(1/N)$ if no convexity is present, and more generally a mixed rate $O(1/N^2)+O(1/N)$ for strongly convex blocks, if only some blocks are strongly convex. Additional novelties of our methods include blockwise-adapted step lengths and acceleration, as well as the ability to update both the primal and dual variables randomly in blocks under a very light compatibility condition. In other words, these variants of our methods are doubly-stochastic. We test the proposed methods on various image processing problems, where we employ pixelwise-adapted acceleration.

An Energy-Efficient and Fast Convergent Resource Allocation Algorithm in Distributed Wireless Ad Hoc Networks

Since the initial publication of Kelly et al .’s seminal paper on resource allocation in wired networks, many studies based on the cross-layer design philosophy have been conducted in both wired and wireless networks. The Lagrangian duality technique has been widely adopted to solve the cross-layer optimization problem, but it is slowly convergent and sensitive to the iterative step size, especially for large networks. In this paper, a joint congestion control and power allocation second-order algorithm (JCCPA) is proposed, in which the joint optimization problem is modeled as a network utility maximization (NUM) framework, and the primal-dual interior-point method is used to solve the NUM problem. The JCCPA updates the primal variables and dual variables simultaneously according to their Newton directions; as a result,resulting in a faster convergence speed is reached. Moreover, the matrix-splitting technology is utilized to decompose the computation of the Hessian matrix and its inverse into different nodes and links so that the distributed update of source rate and node power can be implemented. The simulation results demonstrate that the JCCPA not only significantly improves energy efficiency but also has a faster convergence speed and is insensitive to step size.

Accelerated Distributed Primal-Dual Dynamics Using Adaptive Synchronization

This paper proposes an adaptive primal-dual dynamics for distributed optimization in multi-agent systems. The proposed dynamics incorporates an adaptive synchronization law that reinforces the interconnection strength between the coupled agents. By strengthening the synchronization between the primal variables of the coupled agents, the given law accelerates the convergence of the proposed dynamics to the saddle-point solution. The resulting dynamics is represented as a feedback-interconnected networked system that proves to be passive. The passivity properties of the proposed dynamics are exploited along with the LaSalle's invariance principle for hybrid systems, to establish asymptotic convergence and stability of the saddle-point solution. Further, the primal dynamics is analyzed for the rate of convergence and stronger convergence bounds are established, it is proved that the primal dynamics achieve accelerated convergence under the adaptive synchronization. The robustness of the proposed dynamics is quantified using $L_{2}$ -gain analysis and the correlation between the rate of convergence and robustness of the proposed dynamics are presented. The effectiveness of the proposed dynamics is demonstrated by applying it to solve distributed least squares and distributed support vector machines problems.

A new error analysis of a mixed finite element method for the quad-curl problem

In this paper, we study a new numerical approach for a quad-curl model problem which arises in the inverse electromagnetic scattering problems and magnetohydrodynamics (MHD). We first split the quad-curl problem with homogeneous boundary conditions into a system of second order equations, and then apply a mixed finite element method to solve the resulting system. The perturbed mixed finite element method is constructed by using edge elements. The well posedness of the numerical scheme is derived. The optimal error estimates in H(curl) and L2 norms for the primal and auxiliary variables are obtained, respectively. The theoretical results are verified by numerical experiments.

A Variant of the Affine-Scaling Method for a Second-Order Cone Program

A linear cone program in which the cone is the direct product of second-order cones (Lorentz cones) is considered. For its solution, we propose a primal affine-scaling type method generalizing the corresponding method used in linear programming. The method can be considered as a special way to solve a system of necessary and sufficient optimality conditions for a pair of mutually dual cone programs. These conditions are used to derive the dependence of the dual variables on the primal variables, and the dependence is substituted into the complementarity condition. The obtained system of equations is solved with respect to the primal variables by the fixed-point iteration method. The starting points in the method belong to the cone but do not necessarily satisfy the linear equality-type constraints. The local linear convergence of the method is proved under the assumption that the solutions of the primal and dual problems are nondegenerate and strictly complementary.

Feedback Synthesis for a Terminal Control Problem

A terminal control problem with linear controlled dynamics on a fixed time interval is considered. A boundary value problem in the form of a linear programming problem is stated in a finite-dimensional terminal space at the right endpoint of the interval. The solution of this problem implicitly determines a terminal condition for the controlled dynamics. A saddle-point approach to solving the problem is proposed, which is reduced to the computation a saddle point of the Lagrangian. The approach is based on saddle-point inequalities in terms of primal and dual variables. These inequalities are sufficient optimality conditions. A method for computing a saddle point of the Lagrangian is described. Its monotone convergence with respect to some of the variables on their direct product is proved. Additionally, weak convergence with respect to controls and strong convergence with respect to phase and adjoint trajectories and with respect to terminal variables of the boundary value problem are proved. The saddle-point approach is used to synthesize a feedback control in the case of control constraints in the form of a convex closed set. This result is new, since, in the classical case of the theory of linear regulators, a similar assertion is proved without constraints imposed on the controls. The theory of linear regulators relies on matrix Riccati equations, while the result obtained is based on the concept of a support function (mapping) for the control set.

A Superconvergent HDG Method for Stokes Flow with Strongly Enforced Symmetry of the Stress Tensor

This work proposes a superconvergent hybridizable discontinuous Galerkin (HDG) method for the approximation of the Cauchy formulation of the Stokes equation using same degree of polynomials for the primal and mixed variables. The novel formulation relies on the well-known Voigt notation to strongly enforce the symmetry of the stress tensor. The proposed strategy introduces several advantages with respect to the existing HDG formulations. First, it remedies the suboptimal behavior experienced by the classical HDG method for formulations involving the symmetric part of the gradient of the primal variable. The optimal convergence of the mixed variable is retrieved and an element-by-element post-process procedure leads to a superconvergent velocity field, even for low-order approximations. Second, no additional enrichment of the discrete spaces is required and a gain in computational efficiency follows from reducing the quantity of stored information and the size of the local problems. Eventually, the novel formulation naturally imposes physical tractions on the Neumann boundary. Numerical validation of the optimality of the method and its superconvergent properties is performed in 2D and 3D using meshes of different element types.