Unconstrained Subproblems Research Articles

Error Forgetting of Bregman Iteration

This short article analyzes an interesting property of the Bregman iterative procedure, which is equivalent to the augmented Lagrangian method, for minimizing a convex piece-wise linear function J(x) subject to linear constraints Ax=b. The procedure obtains its solution by solving a sequence of unconstrained subproblems of minimizing $J(x)+\frac{1}{2}\|Ax-b^{k}\|_{2}^{2}$ , where b k is iteratively updated. In practice, the subproblem at each iteration is solved at a relatively low accuracy. Let w k denote the error introduced by early stopping a subproblem solver at iteration k. We show that if all w k are sufficiently small so that Bregman iteration enters the optimal face, then while on the optimal face, Bregman iteration enjoys an interesting error-forgetting property: the distance between the current point $\bar{x}^{k}$ and the optimal solution set X ? is bounded by ?w k+1?w k ?, independent of the previous errors w k?1,w k?2,?,w 1. This property partially explains why the Bregman iterative procedure works well for sparse optimization and, in particular, for ? 1-minimization. The error-forgetting property is unique to J(x) that is a piece-wise linear function (also known as a polyhedral function), and the results of this article appear to be new to the literature of the augmented Lagrangian method.

Parallel Uzawa Method for Large-Scale Minimization of Partially Separable Functions

A parallel Uzawa-type algorithm, for solving unconstrained minimization of large-scale partially separable functions, is presented. Using auxiliary unknowns, the unconstrained minimization problem is transformed into a (linearly) constrained minimization of a separable function.The augmented Lagrangian of this problem decomposes into a sum of partially separable augmented Lagrangian functions. To take advantage of this property, a Uzawa block relaxation is applied. In every iteration, unconstrained minimization subproblems are solved in parallel before updating Lagrange multipliers. Numerical experiments show that the speed-up factor gained using our algorithm is significant.

On the implementation of a log-barrier progressive hedging method for multistage stochastic programs

A progressive hedging method incorporated with self-concordant barrier for solving multistage stochastic programs is proposed recently by Zhao [G. Zhao, A Lagrangian dual method with self-concordant barrier for multistage stochastic convex nonlinear programming, Math. Program. 102 (2005) 1–24]. The method relaxes the nonanticipativity constraints by the Lagrangian dual approach and smoothes the Lagrangian dual function by self-concordant barrier functions. The convergence and polynomial-time complexity of the method have been established. Although the analysis is done on stochastic convex programming, the method can be applied to the nonconvex situation. We discuss some details on the implementation of this method in this paper, including when to terminate the solution of unconstrained subproblems with special structure and how to perform a line search procedure for a new dual estimate effectively. In particular, the method is used to solve some multistage stochastic nonlinear test problems. The collection of test problems also contains two practical examples from the literature. We report the results of our preliminary numerical experiments. As a comparison, we also solve all test problems by the well-known progressive hedging method.

Bregman Iterative Algorithms for $\ell_1$-Minimization with Applications to Compressed Sensing

We propose simple and extremely efficient methods for solving the basis pursuit problem $\min\{\|u\|_1 : Au = f, u\in\mathbb{R}^n\},$ which is used in compressed sensing. Our methods are based on Bregman iterative regularization, and they give a very accurate solution after solving only a very small number of instances of the unconstrained problem $\min_{u\in\mathbb{R}^n} \mu\|u\|_1+\frac{1}{2}\|Au-f^k\|_2^2$ for given matrix A and vector $f^k$. We show analytically that this iterative approach yields exact solutions in a finite number of steps and present numerical results that demonstrate that as few as two to six iterations are sufficient in most cases. Our approach is especially useful for many compressed sensing applications where matrix-vector operations involving A and $A^\top$ can be computed by fast transforms. Utilizing a fast fixed-point continuation solver that is based solely on such operations for solving the above unconstrained subproblem, we were able to quickly solve huge instances of compressed sensing problems on a standard PC.

Transient Optimization in Natural Gas Compressor Stations for Linepack Operation

One of the key factors in the operation of a natural gas pipeline network is the linepack in the network. The desired operation of the network as derived from estimated receipts and deliveries is expressed in terms of the desired linepack profile that must be maintained. The compressor stations in the pipeline network are then operated in a manner that generates this linepack profile. Generally, the operating points selected for the units in the compressor stations are based on experience and experimentation and are therefore not optimal. In this paper, we present a systematic approach for operating the units of a compressor station to meet a specified linepack profile. The first step in developing this approach is the derivation of a numerical method for analyzing the flow through the pipeline under transient nonisothermal conditions. We have developed and verified a fully implicit finite difference formulation that provides this analysis capability. Next, the optimization of the compressor stations is formulated as a standard nonlinear programing problem in the following form: Find the values in the design variable vector denoted by b=[b1,b2,…,bn]T, to minimize a given objective function F(b), subject to the constraints gj(b)⩽0, j=1,…,m. Here, n is the number of operational parameters whose optimal value is to be determined, while m is the number of operational constraints that must be enforced. In our formulation, the design variables are chosen to be the operating speeds of the units in the compressor stations, while the objective function is taken to be the average fuel consumption rate over the interval of interest, summed over all units. The constraint functions gj(b) are formulated suitably to ensure that operational limits are met at the final solution that is obtained. The optimization problem is then solved using a sequential unconstrained minimization technique (SUMT), in conjunction with a directed grid search method for solving the unconstrained subproblems that are encountered in the SUMT formulation. The evaluation of the objective function and constraint functions at each step of the optimization is done by using the fully implicit analysis method mentioned above. A representative numerical example has been solved by the proposed approach. The results obtained indicate that the method is very effective in finding operating points that are optimal with respect to fuel consumption. The optimization can be done at the level of a single unit, a single compressor station, a set of compressor stations, or an entire network. It should also be noted that the proposed solution approach is fully automated and requires no user involvement in the solution process.

A proximal subgradient projection algorithm for linearly constrained strictly convex problems

We propose a new algorithm for linearly constrained strictly convex problems. This algorithm follows the characterization of saddle points introduced earlier in ref. [Ouorou, A., 2000, A primal-dual algorithm for monotropic programming and its application to network optimization. Computational Optimization and Applications, 15(2), 125–143.], using two different augmented Lagrangian functions defined for the primal problem and its dual. The saddle points may be computed in a variety of ways. We propose a scheme that results in a special implementation of Martinet's proximal algorithm ref. [Martinet, B. 1970, Régularisation d'inéquations variationnelles par approximations successives. Revue Franç'Informatique et de Recherche Opérationnelle, 3, 154–179.]. In the primal space, the resulting algorithm appears as a nonsmooth version of the projection algorithm by Rosen ref. [Rosen, J.B., 1960, The gradient projected method for nonlinear programming, part I: Linear Constraints. Journal of the Society for Industrial and Applied Mathematics, 8, 181–217.]. The dual iterates are generated through an unconstrained subproblem which can be solved efficiently by L-BFGS refs. [Byrd, R., Lu, P., Nocedal, J. and Zhu, C. 1995, A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computation, 16(5), 1190–1208.; Liu, D. and Nocedal, J., 1989, On the limited memory BFGS method for large scale optimization. Mathematical Programming B 45, 503–528.]. We establish convergence with no use of the concept of resolvent of maximal monotone operators. To assess the numerical behaviour of the algorithm, we use some randomly generated quadratic network flow problems and compare it with PPRN, a specialized code for linear and nonlinear cost network flow problems.

Penalty Algorithms in Hilbert Spaces

We analyze the classical penalty algorithm for nonlinear programming in Hilbert spaces and obtain global convergence results, as well as asymptotic superlinear convergence order. These convergence results generalize similar results obtained for finite-dimensional problems. Moreover, the nature of the algorithms allows us to solve the unconstrained subproblems in finite-dimensional spaces.

Alocação de unidades hidrelétricas no problema da programação da operação energética utilizando relaxação lagrangeana e lagrangeano aumentado

O problema da programação da operação energética visa definir quais unidades geradoras devem estar em operação para o atendimento à demanda e às demais restrições do sistema, ao longo do horizonte de estudo, de modo que o mínimo custo de operação seja encontrado. Matematicamente, trata-se de um problema não-linear, inteiro-misto e de grande porte, o que torna a sua solução uma tarefa desafiadora. Este artigo apresenta o uso da Relaxação Lagrangeana para decompor o problema da programação da operação em subproblemas menores e mais simples de serem solucionados. No esquema de decomposição utilizado, os subproblemas têm naturezas distintas e são construídos aproveitando-se as particularidades que cada um deles apresenta. Um dos subproblemas resultante do esquema de relaxação utilizado refere-se à alocação das unidades hidrelétricas. Para resolver esse subproblema, propõe-se um algoritmo de enumeração exaustiva do espaço de estados do problema. Cada combinação consiste na solução de problemas não-lineares restritos, resolvidos aqui por meio do método de Lagrangeano Aumentado. Tal método transforma cada problema restrito em uma série de subproblemas irrestritos, que por sua vez são solucionados por um algoritmo de Quase-Newton. O modelo computacional desenvolvido é aplicado a duas usinas hidrelétricas do sistema brasileiro, demonstrando-se a sua consistência e viabilidade prática.

Some Inexact Hybrid Proximal Augmented Lagrangian Algorithms

In this work, Solodov–Svaiter's hybrid projection-proximal and extragradient-proximal methods [16,17] are used to derive two algorithms to find a Karush–Kuhn–Tucker pair of a convex programming problem. These algorithms are variations of the proximal augmented Lagrangian. As a main feature, both algorithms allow for a fixed relative accuracy of the solution of the unconstrained subproblems. We also show that the convergence is Q-linear under strong second order assumptions. Preliminary computational experiments are also presented.

Parallel Image Restoration with Domain Decomposition

In this paper we present parallel algorithms to solve the problem of image restoration when the Point Spread Function is Space Variant. The problem has a very high computational complexity and it is very hard to solve it on scalar computers. The algorithms are based on the decomposition of the image spatial domain and on the solution of both constrained and unconstrained restoration subproblems of size smaller than the original. The main results can be summarized as follows: (a) the quality of restorations do not depend on the number of subdomains; (b) the unconstrained restoration is scalable and efficient even with a large number of processors while the constrained restoration is efficient for subdomains of more than 50×50 pixels. The numerical tests have been executed on a Cray T3E with 128 processors and on a network of workstations.

Primal-Dual Interior Methods for Nonconvex Nonlinear Programming

This paper concerns large-scale general (nonconvex) nonlinear programming when first and second derivatives of the objective and constraint functions are available. A method is proposed that is based on finding an approximate solution of a sequence of unconstrained subproblems parameterized by a scalar parameter. The objective function of each unconstrained subproblem is an augmented penalty-barrier function that involves both primal and dual variables. Each subproblem is solved with a modified Newton method that generates search directions from a primal-dual system similar to that proposed for interior methods. The augmented penalty-barrier function may be interpreted as a merit function for values of the primal and dual variables. An inertia-controlling symmetric indefinite factorization is used to provide descent directions and directions of negative curvature for the augmented penalty-barrier merit function. A method suitable for large problems can be obtained by providing a version of this factorization that will treat large sparse indefinite systems.

Global convergence of QPFTH method for large-scale nonlinear sparse constrained optimization

A QP-free, truncated hybrid (QPFTH) method was proposed and developed in [6] for solving sparse large-scale nonlinear programming problems. In the hybrid method, a truncated Newton method is combined with the method of multiplier. In every iteration level, either a truncated solution for a symmetric system of linear equations is determined by CG algorithm or an unconstrained subproblem is solved by the limited memory BFGS algorithm such that the hybrid algorithm is suitable to large0scale problems. In this paper, the consistency in the hybrid method and a steplength procedure are discussed and developed. The global convergence of QPFTH method is proved and the two-step Q-quadratic convergence rate is further analyzed.

Numerical Stability and Efficiency of Penalty Algorithms

Penalty algorithms have been somewhat forgotten due to numerical instabilities once believed to be inherent to those methods. One usually has to solve a sequence of such problems, and when the penalty factor decreases too fast, the subproblems may become intractable. Moreover, as the penalty factor decreases, the unconstrained subproblem becomes ill conditioned, and thus difficult to solve. Also, in several intermediate computations, numerical instability may show up. The author proposes remedies to such problems and presents a wide class of numerically stable penalty algorithms. The work is done in the more general context of variational inequality problems, which encompasses optimization problems. The author’s results yield a family of globally convergent, two-step superlinearly convergent, numerically stable algorithms for variational inequality problems. Finally, issues in the numerically stable implementation of intermediate computations within those algorithms are discussed.

Optimal control of constrained problems by the costate coordination structure

A three-level hierarchical computation structure based on the costate coordination technique is developed to efficiently optimize the constrained control problems with nonseparable cost functions. The key idea underlying this development is the effective utilization of the Lagrangian generalized gradients to successfully convert the integrated optimization problem at hand into an interactive process carried out by decentralized calculations. The decentralization is conceived both horizontally, by generating a group of lower order unconstrained subproblems, and vertically through the division of the coordination task into two distinct mechanisms. While the first is mainly concerned with adjusting the system state and control trajectories against constraint violations, the second updates the costate variables in order to provide for both the matching of computationally separated subproblems and overall system optimality.

A column generation algorithm for nonlinear programming

An algorithm using column generation and penalty function techniques is presented. A linear program with a uniformly bounded number of columns, similar to the restricted master in generalized programming, is used to reduce the number of constraints included in forming a penalty function. The penalty function is used as a Lagrangian in an unconstrained subproblem.