Quadratic Proximal Term Research Articles

A proximal fully parallel splitting method with a relaxation factor for separable convex programming

In this paper, we propose a proximal fully parallel splitting method with a relaxation factor for solving separable convex minimization model with linear constraints, where the objective function is the sum of m individual functions without coupled variables. With a full Jacobian decomposition, we decompose the subproblem associated with the augmented Lagrangian method into m smaller subproblems and then add a quadratic proximal term to each decomposed subproblem, which makes the resulting ones easier to solve for many applications. In order to accelerate the numerical performance, we attach a positive relaxation factor to update the Lagrange multiplier, which also allows more flexibility in the design of algorithms. Moreover, we refine the step size of the underrelaxation step, which enlarges several existing ones in the literature. We prove that the proposed method is globally convergent, and show the worst-case O(1/k) convergence rate in a nonergodic sense. Finally, the efficiency and robustness of the proposed method are also demonstrated by solving the ℓ1 norm problem, the ℓ1-regularized least squares problem, the exchange problem and the total variation image restoration problem.

Optimal proximal augmented Lagrangian method and its application to full Jacobian splitting for multi-block separable convex minimization problems

Abstract The augmented Lagrangian method (ALM) is fundamental in solving convex programming problems with linear constraints. The proximal version of ALM, which regularizes ALM’s subproblem over the primal variable at each iteration by an additional positive-definite quadratic proximal term, has been well studied in the literature. In this paper we show that it is not necessary to employ a positive-definite quadratic proximal term for the proximal ALM and the convergence can be still ensured if the positive definiteness is relaxed to indefiniteness by reducing the proximal parameter. An indefinite proximal version of the ALM is thus proposed for the generic setting of convex programming problems with linear constraints. We show that our relaxation is optimal in the sense that the proximal parameter cannot be further reduced. The consideration of indefinite proximal regularization is particularly meaningful for generating larger step sizes in solving ALM’s primal subproblems. When the model under discussion is separable in the sense that its objective function consists of finitely many additive function components without coupled variables, it is desired to decompose each ALM’s subproblem over the primal variable in Jacobian manner, replacing the original one by a sequence of easier and smaller decomposed subproblems, so that parallel computation can be applied. This full Jacobian splitting version of the ALM is known to be not necessarily convergent, and it has been studied in the literature that its convergence can be ensured if all the decomposed subproblems are further regularized by sufficiently large proximal terms. But how small the proximal parameter could be is still open. The other purpose of this paper is to show the smallest proximal parameter for the full Jacobian splitting version of ALM for solving multi-block separable convex minimization models.

Finite convergence of an active signature method to local minima of piecewise linear functions

ABSTRACTWe previously derived first-order (KKT) and second-order (SOSC) optimality conditions for functions defined by evaluation programs involving smooth elementals and absolute values. For this class of problems we showed that the algorithm of successive abs-linear minimization with a proximal term (SALMIN) achieves a linear or even quadratic rate of convergence under suitable assumptions. An implementation of SALMIN called LiPsMin has been implemented and tested. It uses a bundle-type method direction finder for the inner loop, i.e. the minimization of the local piecewise linear model with a quadratic proximal term. As a consequence the inner solver can get stuck at stationary points of the piecewise linearization and the overall algorithm can only guarantee Clarke stationarity of its cluster points. Here we present a new method that computes a local minimizer of the proximal model objective. As a consequence all cluster points of the outer iteration must be first-order minimal, which is also known as criticality in nonsmooth optimization. The active signature strategy employed very much resembles the classical method for convex QOPs and utilizes the same numerical linear algebra techniques. The new custom-made algorithm provides opportunities for structure exploitation like warm starts in the context of the nonlinear, outer loop.

On Lipschitz optimization based on gray-box piecewise linearization

We address the problem of minimizing objectives from the class of piecewise differentiable functions whose nonsmoothness can be encapsulated in the absolute value function. They possess local piecewise linear approximations with a discrepancy that can be bounded by a quadratic proximal term. This overestimating local model is continuous but generally nonconvex. It can be generated in its abs-normal form by a minor extension of standard algorithmic differentiation tools. Here we demonstrate how the local model can be minimized by a bundle-type method, which benefits from the availability of additional gray-box information via the abs-normal form. In the convex case our algorithm realizes the consistent steepest descent trajectory for which finite convergence was established earlier, specifically covering counterexamples where steepest descent with exact line-search famously fails. The analysis of the abs-normal representation and the design of the optimization algorithm are geared toward the general case, whereas the convergence proof so far covers only the convex case.

Entropy-Like Minimization Methods Based On Modified Proximal Point Algorithm

>In this paper we introduce an extension of the proximal point algorithm proposed by Güler for solving convex minimization problems. This extension is obtained by substituting the usual quadratic proximal term by a class of convex nonquadratic entropy-like distances, called φ-divergences. A study of the convergence rate of this new proximal point method under mild assumptions is given, and further it is shown that this estimate rate is better than the available one of proximal-like methods. Some applications are given concerning general convex minimizations, linearly constrained convex programs and variationnal inequalities.

Convergence of a Proximal Point Algorithm for Solving Minimization Problems

We introduce and consider a proximal point algorithm for solving minimization problems using the technique of Güler. This proximal point algorithm is obtained by substituting the usual quadratic proximal term by a class of convex nonquadratic distance‐like functions. It can be seen as an extragradient iterative scheme. We prove the convergence rate of this new proximal point method under mild assumptions. Furthermore, it is shown that this estimate rate is better than the available ones.

Proximal point nonlinear rescaling method for convex optimization

Nonlinear rescaling (NR) methods alternate finding an unconstrained minimizer of the Lagrangian for the equivalent problem in the primal space (which is an infinite procedure) with Lagrange multipliers update. <br> &nbsp; &nbsp We introduce and study a proximal point nonlinear rescaling (PPNR) method that preserves convergence and retains a linear convergence rate of the original NR method and at the same time does not require an infinite procedure at each step.<br> &nbsp; &nbsp The critical component of our analysis is the equivalence of the NR method with dynamic scaling parameter update to the interior quadratic proximal point method for the dual problem in the rescaled from step to step dual space.<br> &nbsp; &nbsp By adding the classical quadratic proximal term to the primal objective function the PPNR step can be viewed as a primal-dual proximal point mapping. This allows analyzing a wide variety of non-quadratic augmented Lagrangian methods from unique and general point of view using tools typical for the classical quadratic proximal-point technique.<br> &nbsp; &nbsp We proved convergence of the primal-dual PPNR sequence under minimum assumptions on the input data and established a $q$-linear rate of convergence under the standard second-order optimality conditions.

A continuation approach for the capacitated multi-facility weber problem based on nonlinear SOCP reformulation

We propose a primal-dual continuation approach for the capacitated multi-facility Weber problem (CMFWP) based on its nonlinear second-order cone program (SOCP) reformulation. The main idea of the approach is to reformulate the CMFWP as a nonlinear SOCP with a nonconvex objective function, and then introduce a logarithmic barrier term and a quadratic proximal term into the objective to construct a sequence of convexified subproblems. By this, this class of nondifferentiable and nonconvex optimization problems is converted into the solution of a sequence of nonlinear convex SOCPs. In this paper, we employ the semismooth Newton method proposed in Kanzow et al. (SIAM Journal of Optimization 20:297---320, 2009) to solve the KKT system of the resulting convex SOCPs. Preliminary numerical results are reported for eighteen test instances, which indicate that the continuation approach is promising to find a satisfying suboptimal solution, even a global optimal solution for some test problems.

The interior proximal extragradient method for solving equilibrium problems

In this article we present a new and efficient method for solving equilibrium problems on polyhedra. The method is based on an interior-quadratic proximal term which replaces the usual quadratic proximal term. This leads to an interior proximal type algorithm. Each iteration consists in a prediction step followed by a correction step as in the extragradient method. In a first algorithm each of these steps is obtained by solving an unconstrained minimization problem, while in a second algorithm the correction step is replaced by an Armijo-backtracking linesearch followed by an hyperplane projection step. We prove that our algorithms are convergent under mild assumptions: pseudomonotonicity for the two algorithms and a Lipschitz property for the first one. Finally we present some numerical experiments to illustrate the behavior of the proposed algorithms.

Regularized Lotka-Volterra Dynamical System as Continuous Proximal-Like Method in Optimization

We introduce and study a new type of dynamical system which combines the continuous gradient method with a nonlinear Lotka-Volterra (LV) type of differential system within a logarithmic-quadratic proximal scheme. We prove a global existence and viability result for the resulting trajectory which holds for a general smooth function. The asymptotic behavior of the produced trajectory is analyzed and global convergence of the trajectory to a minimizer of the convex minimization problem over the nonnegative orthant is established. The implicit discretization which is at the origin of the proposed continuous dynamical system is an interior proximal scheme for minimizing a closed proper convex function, and convergence results and properties of the resulting discrete scheme are also established. We show finally that the trajectories of the family of regularized Lotka-Volterra systems, parametrized by the positive parameter associated with the quadratic proximal term, are uniformly convergent to the solution of the classical LV-dynamical system, as the parameter associated with the proximal term approaches zero.

Convergence Rate Analysis of Nonquadratic Proximal Methods for Convex and Linear Programming

The φ-divergence proximal method is an extension of the proximal minimization algorithm, where the usual quadratic proximal term is substituted by a class of convex statistical distances, called φ-divergences. In this paper, we study the convergence rate of this nonquadratic proximal method for convex and particularly linear programming. We identify a class of φ-divergences for which superlinear convergence is attained both for optimization problems with strongly convex objectives at the optimum and linear programming problems, when the regularization parameters tend to zero. We prove also that, with regularization parameters bounded away from zero, convergence is at least linear for a wider class of φ-divergences, when the method is applied to the same kinds of problems. We further analyze the associated class of augmented Lagrangian methods for convex programming with nonquadratic penalty terms, and prove convergence of the dual generated by these methods for linear programming problems under a weak nondegeneracy assumption.

Partial Proximal Minimization Algorithms for Convex Pprogramming

An extension of the proximal minimization algorithm is considered where only some of the minimization variables appear in the quadratic proximal term. The resulting iterates are interpreted in terms of the iterates of the standard algorithm, and a uniform descent property is shown that holds independently of the proximal terms used. This property is used to give simple convergence proofs of parallel algorithms where multiple processors simultaneously execute proximal iterations using different partial proximal terms. It is also shown that partial proximal minimization algorithms are dual to multiplier methods with partial elimination of constraints, and a relation is established between parallel proximal minimization algorithms and parallel constraint distribution algorithms.

A proximal-based decomposition method for convex minimization problems

This paper presents a decomposition method for solving convex minimization problems. At each iteration, the algorithm computes two proximal steps in the dual variables and one proximal step in the primal variables. We derive this algorithm from Rockafellar's proximal method of multipliers, which involves an augmented Lagrangian with an additional quadratic proximal term. The algorithm preserves the good features of the proximal method of multipliers, with the additional advantage that it leads to a decoupling of the constraints, and is thus suitable for parallel implementation. We allow for computing approximately the proximal minimization steps and we prove that under mild assumptions on the problem's data, the method is globally convergent and at a linear rate. The method is compared with alternating direction type methods and applied to the particular case of minimizing a convex function over a finite intersection of closed convex sets.

A regularized stochastic decomposition algorithm for two-stage stochastic linear programs

In this paper a regularized stochastic decomposition algorithm with master programs of finite size is described for solving two-stage stochastic linear programming problems with recourse. In a deterministic setting cut dropping schemes in decomposition based algorithms have been used routinely. However, when only estimates of the objective function are available such schemes can only be properly justified if convergence results are not sacrificed. It is shown that almost surely every accumulation point in an identified subsequence of iterates produced by the algorithm, which includes a cut dropping scheme, is an optimal solution. The results are obtained by including a quadratic proximal term in the master program. In addition to the cut dropping scheme, other enhancements to the existing methodology are described. These include (i) a new updating rule for the retained cuts and (ii) an adaptive rule to determine when additional reestimation of the cut associated with the current solution is needed. The algorithm is tested on problems from the literature assuming both descrete and continuous random variables.

Interior proximal point algorithm for linear programs

An interior proximal point algorithm for finding a solution of a linear program is presented. The distinguishing feature of this algorithm is the addition of a quadratic proximal term to the linear objective function. This perturbation has allowed us to obtain solutions with better feasibility. Implementation of this algorithm shows that the algorithms. We also establish global convergence and local linear convergence of the algorithm.