Dual Iteration Research Articles

A Highly Efficient Semismooth Newton Augmented Lagrangian Method for Solving Lasso Problems

We develop a fast and robust algorithm for solving large scale convex composite optimization models with an emphasis on the $\ell_1$-regularized least squares regression (Lasso) problems. Despite the fact that there exist a large number of solvers in the literature for the Lasso problems, we found that no solver can efficiently handle difficult large scale regression problems with real data. By leveraging on available error bound results to realize the asymptotic superlinear convergence property of the augmented Lagrangian algorithm, and by exploiting the second order sparsity of the problem through the semismooth Newton method, we are able to propose an algorithm, called {\sc Ssnal}, to efficiently solve the aforementioned difficult problems. Under very mild conditions, which hold automatically for Lasso problems, both the primal and the dual iteration sequences generated by {\sc Ssnal} possess a fast linear convergence rate, which can even be superlinear asymptotically. Numerical comparisons between our approach and a number of state-of-the-art solvers, on real data sets, are presented to demonstrate the high efficiency and robustness of our proposed algorithm in solving difficult large scale Lasso problems.

Ergodic, primal convergence in dual subgradient schemes for convex programming, II: the case of inconsistent primal problems

Consider the utilization of a Lagrangian dual method which is convergent for consistent convex optimization problems. When it is used to solve an infeasible optimization problem, its inconsistency will then manifest itself through the divergence of the sequence of dual iterates. Will then the sequence of primal subproblem solutions still yield relevant information regarding the primal program? We answer this question in the affirmative for a convex program and an associated subgradient algorithm for its Lagrange dual. We show that the primal–dual pair of programs corresponding to an associated homogeneous dual function is in turn associated with a saddle-point problem, in which—in the inconsistent case—the primal part amounts to finding a solution in the primal space such that the Euclidean norm of the infeasibility in the relaxed constraints is minimized; the dual part amounts to identifying a feasible steepest ascent direction for the Lagrangian dual function. We present convergence results for a conditional varepsilon -subgradient optimization algorithm applied to the Lagrangian dual problem, and the construction of an ergodic sequence of primal subproblem solutions; this composite algorithm yields convergence of the primal–dual sequence to the set of saddle-points of the associated homogeneous Lagrangian function; for linear programs, convergence to the subset in which the primal objective is at minimum is also achieved.

A global minimization algorithm for Tikhonov functionals with sparsity constraints

In this paper, we present a globally convergent algorithm for the computation of a minimizer of the Tikhonov functional with sparsity promoting penalty term for nonlinear forward operators in Banach space. The dual TIGRA method uses a gradient descent iteration in the dual space at decreasing values of the regularization parameter , where the approximation obtained with serves as the starting value for the dual iteration with parameter . With the discrepancy principle as a global stopping rule, the method further yields an automatic parameter choice. We prove convergence of the algorithm under suitable step-size selection and stopping rules and illustrate our theoretic results with numerical experiments for the nonlinear autoconvolution problem.

Decentralized Dynamic Optimization Through the Alternating Direction Method of Multipliers

This paper develops the application of the alternating direction method of multipliers (ADMM) to optimize a dynamic objective function in a decentralized multi-agent system. At each time slot, agents in the network observe local functions and cooperate to track the optimal time-varying argument of the sum objective. This cooperation is based on maintaining local primal variables that estimate the value of the optimal argument and auxiliary dual variables that encourage proximity with neighboring estimates. Primal and dual variables are updated by an ADMM iteration that can be implemented in a distributed manner whereby local updates require access to local variables and the most recent primal variables from adjacent agents. For objective functions that are strongly convex and have Lipschitz continuous gradients, the distances between the primal and dual iterates to their corresponding time-varying optimal values are shown to converge to a steady state gap. This gap is explicitly characterized in terms of the condition number of the objective function, the condition number of the network that is defined as the ratio between the largest and smallest nonzero Laplacian eigenvalues, and a bound on the drifts of the optimal primal variables and the optimal gradients. Numerical experiments corroborate theoretical findings and show that the results also hold for non-differentiable and non-strongly convex primal objectives.

Convergence Analysis for a Primal-Dual Monotone + Skew Splitting Algorithm with Applications to Total Variation Minimization

In this paper we investigate the convergence behavior of a primal-dual splitting method for solving monotone inclusions involving mixtures of composite, Lipschitzian and parallel sum type operators proposed by Combettes and Pesquet (in Set-Valued Var. Anal. 20(2):307---330, 2012). Firstly, in the particular case of convex minimization problems, we derive convergence rates for the partial primal-dual gap function associated to a primal-dual pair of optimization problems by making use of conjugate duality techniques. Secondly, we propose for the general monotone inclusion problem two new schemes which accelerate the sequences of primal and/or dual iterates, provided strong monotonicity assumptions for some of the involved operators are fulfilled. Finally, we apply the theoretical achievements in the context of different types of image restoration problems solved via total variation regularization.

Addressing supply-side risk in uncertain power markets: stochastic Nash models, scalable algorithms and error analysis

Increasing penetration of volatile wind-based generation into the fuel mix is leading to growing supply-side volatility. As a consequence, the reliability of the power grid continues to be a source of much concern, particularly since the impact of supply-side risk exposure, arising from aggressive bidding,1 is not felt by risk-seeking generation firms; instead, the system operator is largely responsible for managing shortfalls in the real-time market. We propose an alternate design in which the cost of such risk is transferred to firms responsible for imposing such risk. The resulting strategic problem can be cast as a two-period generalized stochastic Nash game with shared strategy sets. A subset of equilibria is given by a solution to a related stochastic variational inequality, that is shown to be both monotone and solvable. Computing solutions of this variational problem is challenging since the size of the problem grows with the cardinality of the sample space, network size and the number of participating firms. Consequently, direct schemes are inadvisable for most practical problems. Instead, we present a distributed regularized primal–dual scheme and a dual projection scheme where both primal and dual iterates are computed separately. Rates of convergence estimates are provided and error bounds are developed for inexact extensions of the dual scheme. Unlike projection schemes for deterministic problems, here the projection step requires the solution of a possibly massive stochastic programme. By utilizing cutting plane methods, we ensure that the complexity of the projection scheme scales slowly with the size of the sample space. We conclude with a study of a 53-node electricity network that allows for deriving insights regarding market design and operation, particularly for accommodating firms with uncertain generation assets.

Implementation of a block-decomposition algorithm for solving large-scale conic semidefinite programming problems

In this paper, we consider block-decomposition first-order methods for solving large-scale conic semidefinite programming problems given in standard form. Several ingredients are introduced to speed-up the method in its pure form such as: an aggressive choice of stepsize for performing the extragradient step; use of scaled inner products; dynamic update of the scaled inner product for properly balancing the primal and dual relative residuals; and proper choices of the initial primal and dual iterates, as well as the initial parameter for the scaled inner product. Finally, we present computational results showing that our method outperforms the two most competitive codes for large-scale conic semidefinite programs, namely: the boundary-point method introduced by Povh et al. and the Newton-CG augmented Lagrangian method by Zhao et al.

Nonlinear multigrid method for solving the LLT model

Our goal in this paper is to study a nonlinear multigrid method for solving the LLT model. In order to improve the convergence rate of the proposed multigrid method, an improved dual iteration is proposed as its smoother. Furthermore, we give the local Fourier analysis (LFAs) of the Chambolle’s dual iterations and a modified smoother for solving the LLT model, respectively. Numerical results illustrate the efficiency of the proposed method and indicate that such a multigrid method is more suitable to deal with large-sized images.

Infeasible constraint-reduced interior-point methods for linear optimization

Constraint-reduction schemes have been proposed for the solution by means of interior-point methods of linear programs with many more inequality constraints than variables in the standard dual form. Such schemes have been shown to be provably convergent and highly efficient in practice. A critical requirement of these schemes is the availability of an initial dual-feasible point. In this paper, building on a general framework (which encompasses several previously proposed approaches) for dual-feasible constraint-reduced interior-point optimization, for which we prove convergence to a single point of the sequence of dual iterates, we propose a framework for ‘infeasible’ constraint-reduced interior-point optimization. Central to this framework is an exact (ℓ1 or ) penalty function scheme endowed with a mechanism for iterative adjustment of the penalty parameter, which aims at yielding, after a finite number of iterations, a value that guarantees feasibility (for the original problem) of the minimizers. Finiteness of the sequence of penalty parameter adjustments is proved under mild assumptions for all algorithms that fit within the framework, including ‘infeasible’ extensions of a ‘dual’ algorithm proposed in the early 1990s (Dantzig and Ye, A build-up interior-point method for linear programming: Affine scaling form, Working paper, Department of Management Science, University of Iowa, 1991) and of two recently proposed ‘primal–dual’ algorithms (Tits, Absil, and Woessner, Constraint reduction for linear programs with many inequality constraints, SIAM J. Optim. 17 (2006), pp. 119–146; Winternitz, Nicholls, Tits, and O’Leary, A constraint-reduced variant of Mehrotra’s predictor–corrector algorithm, Comput. Optim. Appl. (published on-line as of January 2011), DOI: 10.1007/s10589-010-9389-4). The last one, a constraint-reduced variant of Mehrotra’s Predictor–Corrector algorithm, is then more specifically considered: further convergence results are proved, and numerical results are reported that demonstrate that the approach is of practical interest.

Nonlinear multigrid method for solving the anisotropic image denoising models

In this paper, we study a nonlinear multigrid method for solving a general image denoising model with two L 1-regularization terms. Different from the previous studies, we give a simpler derivation of the dual formulation of the general model by augmented Lagrangian method. In order to improve the convergence rate of the proposed multigrid method, an improved dual iteration is proposed as its smoother. Furthermore, we apply the proposed method to the anisotropic ROF model and the anisotropic LLT model. We also give the local Fourier analysis (LFAs) of the Chambolle's dual iterations and a modified smoother for solving these two models, respectively. Numerical results illustrate the efficiency of the proposed method and indicate that such a multigrid method is more suitable to deal with large-sized images.

Dual Choice and Iteration in an Abstract Algebra of Action

This paper presents an abstract-algebraic formulation of action facilitating reasoning about two opposing agents. Two dual nondeterministic choice operators are formulated abstract-algebraically: angelic (or user) choice and demonic (or system) choice. Iteration operators are also defined. As an application, Hoare-style correctness rules are established by means of the algebra. A negation operator is also discussed.

Robust Force Controller for Industrial Robots: Optimal Design and Real-Time Implementation on a KUKA Robot

In this brief, a new formulation is proposed for the design of a robust force controller for industrial robots with closed architecture controller. A recent model, which takes into account the complete kinematics of the system and the position response of the industrial robot controller, allows this formulation. The proposed design is based on a set of nonlinear matrix inequalities, and solved by the dual iteration algorithm. The robust controller is implemented on a KUKA robot. Good results are obtained with the proposed approach.

A Goal-Oriented Dual LRCF-ADI for Balanced Truncation

AbstractIn this contribution we propose a novel dual Lyapunov solver that computes low rank factors of the reachability and observability Gramians of a control system simultaneously. This is especially helpful in balanced truncation model order reduction applications, where the singular values of the product of these Gramian factors are the basis of the truncation process. For stopping the dual iteration scheme we therefore propose a tailored stopping criterion aiming at the accurate computation of these singular values.

Continuous Multiclass Labeling Approaches and Algorithms

We study convex relaxations of the image labeling problem on a continuous domain with regularizers based on metric interaction potentials. The generic framework ensures existence of minimizers and covers a wide range of relaxations of the original combinatorial problem. We focus on two specific relaxations that differ in flexibility and simplicity—one can be used to tightly relax any metric interaction potential, while the other covers only Euclidean metrics but requires less computational effort. For solving the nonsmooth discretized problem, we propose a globally convergent Douglas-Rachford scheme and show that a sequence of dual iterates can be recovered in order to provide a posteriori optimality bounds. In a quantitative comparison to two other first-order methods, the approach shows competitive performance on synthetic and real-world images. By combining the method with an improved rounding technique for nonstandard potentials, we were able to routinely recover integral solutions within $1\%$-$5\%$ of the global optimum for the combinatorial image labeling problem.

逐次 LMI 化手法と双対反復法の関係について

逐次 LMI 化手法と双対反復法の関係について

An inexact Newton method for nonconvex equality constrained optimization

We present a matrix-free line search algorithm for large-scale equality constrained optimization that allows for inexact step computations. For strictly convex problems, the method reduces to the inexact sequential quadratic programming approach proposed by Byrd et al. [SIAM J. Optim. 19(1) 351–369, 2008]. For nonconvex problems, the methodology developed in this paper allows for the presence of negative curvature without requiring information about the inertia of the primal–dual iteration matrix. Negative curvature may arise from second-order information of the problem functions, but in fact exact second derivatives are not required in the approach. The complete algorithm is characterized by its emphasis on sufficient reductions in a model of an exact penalty function. We analyze the global behavior of the algorithm and present numerical results on a collection of test problems.

A new family of penalties for augmented Lagrangian methods

AbstractWe study a family of penalty functions for augmented Lagrangian methods, and concentrate on a penalty based on the modified logarithmic barrier function. The convex conjugate of this penalty induces a Bregman distance, and the dual iterates associated with the augmented Lagrangian algorithm correspond to the iterates produced by a proximal point algorithm based on this distance. The global convergence of the dual iterates is then proved. Moreover, the level curves of the quadratic approximation of the dual kernels associated with these penalty functions are the Dikin ellipsoids. Copyright © 2008 John Wiley & Sons, Ltd.

A proximal subgradient projection algorithm for linearly constrained strictly convex problems

We propose a new algorithm for linearly constrained strictly convex problems. This algorithm follows the characterization of saddle points introduced earlier in ref. [Ouorou, A., 2000, A primal-dual algorithm for monotropic programming and its application to network optimization. Computational Optimization and Applications, 15(2), 125–143.], using two different augmented Lagrangian functions defined for the primal problem and its dual. The saddle points may be computed in a variety of ways. We propose a scheme that results in a special implementation of Martinet's proximal algorithm ref. [Martinet, B. 1970, Régularisation d'inéquations variationnelles par approximations successives. Revue Franç'Informatique et de Recherche Opérationnelle, 3, 154–179.]. In the primal space, the resulting algorithm appears as a nonsmooth version of the projection algorithm by Rosen ref. [Rosen, J.B., 1960, The gradient projected method for nonlinear programming, part I: Linear Constraints. Journal of the Society for Industrial and Applied Mathematics, 8, 181–217.]. The dual iterates are generated through an unconstrained subproblem which can be solved efficiently by L-BFGS refs. [Byrd, R., Lu, P., Nocedal, J. and Zhu, C. 1995, A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computation, 16(5), 1190–1208.; Liu, D. and Nocedal, J., 1989, On the limited memory BFGS method for large scale optimization. Mathematical Programming B 45, 503–528.]. We establish convergence with no use of the concept of resolvent of maximal monotone operators. To assess the numerical behaviour of the algorithm, we use some randomly generated quadratic network flow problems and compare it with PPRN, a specialized code for linear and nonlinear cost network flow problems.

Low-Order Robust Decentralized Controller Design for a Hvdc Power Link using Successive Centering

In this paper an iterative technique based on the successive centering method is proposed for designing a robust decentralized low-order controller for a High Voltage DC power transmission link. Starting with an arbitrary stabilizing loworder controller obtained via dual iteration, in a three-stage procedure this controller is successively (i) transformed into a decentralized structure, (ii) made robust against admissible parameter variations by imposing a H∞ norm constraint, and (iii) tuned for a fast response by pushing closed-loop poles to the left. Simulation results illustrate the achieved robust performance of the HVDC power link under different operating conditions.

Alternating Direction Method for Maximum Entropy Subject to Simple Constraint Sets

The problem of maximizing the entropy subject to simple constraint sets is reformulated as a structured variational inequality problem by introducing dual variables. A new iterative alternating direction method is then developed that generates alternatively the dual and primal iterates. For some existing maximum entropy problems in the literature, the new dual iterate can be derived from a simple projection and then the new primal iterate can be obtained via solving approximately n separate one-dimensional strong monotone equations. Therefore, the proposed method is very easy to carry out. Preliminary numerical results show that the method is applicable.