Piecewise Linear-quadratic Functions Research Articles

Primal Superlinear Convergence of Sqp Methods in Piecewise Linear-Quadratic Composite Optimization

This paper mainly concerns with the primal superlinear convergence of the quasi-Newton sequential quadratic programming (SQP) method for piecewise linear-quadratic composite optimization problems. We show that the latter primal superlinear convergence can be justified under the noncriticality of Lagrange multipliers and a version of the Dennis-Moré condition. Furthermore, we show that if we replace the noncriticality condition with the second-order sufficient condition, this primal superlinear convergence is equivalent with an appropriate version of the Dennis-Moré condition. We also recover Bonnans’ result in (Appl. Math. Optim. 29, 161–186, 1994) for the primal-dual superlinear of the basic SQP method for this class of composite problems under the second-order sufficient condition and the uniqueness of Lagrange multipliers. To achieve these goals, we first obtain an extension of the reduction lemma for convex Piecewise linear-quadratic functions and then provide a comprehensive analysis of the noncriticality of Lagrange multipliers for composite problems. We also establish certain primal estimates for KKT systems of composite problems, which play a significant role in our local convergence analysis of the quasi-Newton SQP method.

Kurdyka–Łojasiewicz Property of Zero-Norm Composite Functions

This paper focuses on a class of zero-norm composite optimization problems. For this class of nonconvex nonsmooth problems, we establish the Kurdyka–Łojasiewicz property of exponent being a half for its objective function under a suitable assumption and provide some examples to illustrate that such an assumption is not very restricted which, in particular, involve the zero-norm regularized or constrained piecewise linear–quadratic function, the zero-norm regularized or constrained logistic regression function, the zero-norm regularized or constrained quadratic function over a sphere.

Strong Metric (Sub)regularity of Karush–Kuhn–Tucker Mappings for Piecewise Linear-Quadratic Convex-Composite Optimization and the Quadratic Convergence of Newton’s Method

This work concerns the local convergence theory of Newton and quasi-Newton methods for convex-composite optimization: where one minimizes an objective that can be written as the composition of a convex function with one that is continuiously differentiable. We focus on the case in which the convex function is a potentially infinite-valued piecewise linear-quadratic function. Such problems include nonlinear programming, mini-max optimization, and estimation of nonlinear dynamics with non-Gaussian noise as well as many modern approaches to large-scale data analysis and machine learning. Our approach embeds the optimality conditions for convex-composite optimization problems into a generalized equation. We establish conditions for strong metric subregularity and strong metric regularity of the corresponding set-valued mappings. This allows us to extend classical convergence of Newton and quasi-Newton methods to the broader class of nonfinite valued piecewise linear-quadratic convex-composite optimization problems. In particular, we establish local quadratic convergence of the Newton method under conditions that parallel those in nonlinear programming.

A Study of Piecewise Linear-Quadratic Programs

Motivated by a growing list of nontraditional statistical estimation problems of the piecewise kind, this paper provides a survey of known results supplemented with new results for the class of piecewise linear-quadratic programs. These are linearly constrained optimization problems with piecewise linear-quadratic (PLQ) objective functions. Starting from a study of the representation of such a function in terms of a family of elementary functions consisting of squared affine functions, squared plus-composite-affine functions, and affine functions themselves, we summarize some local properties of a PLQ function in terms of their first and second-order directional derivatives. We extend some well-known necessary and sufficient second-order conditions for local optimality of a quadratic program to a PLQ program and provide a dozen such equivalent conditions for strong, strict, and isolated local optimality, showing in particular that a PLQ program has the same characterizations for local minimality as a standard quadratic program. As a consequence of one such condition, we show that the number of strong, strict, or isolated local minima of a PLQ program is finite; this result supplements a recent result about the finite number of directional stationary objective values. Interestingly, these finiteness results can be uncovered by invoking a very powerful property of subanalytic functions; our proof is fairly elementary, however. We discuss applications of PLQ programs in some modern statistical estimation problems. These problems lead to a special class of unconstrained composite programs involving the non-differentiable $\ell_1$-function, for which we show that the task of verifying the second-order stationary condition can be converted to the problem of checking the copositivity of certain Schur complement on the nonnegative orthant.

A linear-time algorithm to compute the conjugate of convex piecewise linear-quadratic bivariate functions

We propose the first algorithm to compute the conjugate of a bivariate Piecewise Linear-Quadratic (PLQ) function in optimal linear worst-case time complexity. The key step is to use a planar graph, called the entity graph, not only to represent the entities (vertex, edge, or face) of the domain of a PLQ function but most importantly to record adjacent entities. We traverse the graph using breadth-first search to compute the conjugate of each entity using graph-matrix calculus, and use the adjacency information to create the output data structure in linear time.

Computation of the Epsilon-Subdifferential of Convex Piecewise Linear-Quadratic Functions in Optimal Worst-Case Time

The $\epsilon$-subdifferential of convex univariate piecewise linear-quadratic functions can be computed in linear worst-case time complexity as the level-set of a convex function. Using dichotomic search, we show how the computation can be performed in logarithmic worst-case time. Furthermore, a new algorithm to compute the entire graph of the $\epsilon$-subdifferential in linear time is presented. Both algorithms are not limited to convex PLQ functions but are also applicable to any convex piecewise-defined function with little restrictions.

Linear convergence of the randomized sparse Kaczmarz method

The randomized version of the Kaczmarz method for the solution of consistent linear systems is known to converge linearly in expectation. And even in the possibly inconsistent case, when only noisy data is given, the iterates are expected to reach an error threshold in the order of the noise-level with the same rate as in the noiseless case. In this work we show that the same also holds for the iterates of the recently proposed randomized sparse Kaczmarz method for recovery of sparse solutions. Furthermore we consider the more general setting of convex feasibility problems and their solution by the method of randomized Bregman projections. This is motivated by the observation that, similarly to the Kaczmarz method, the Sparse Kaczmarz method can also be interpreted as an iterative Bregman projection method to solve a convex feasibility problem. We obtain expected sublinear rates for Bregman projections with respect to a general strongly convex function. Moreover, even linear rates are expected for Bregman projections with respect to smooth or piecewise linear-quadratic functions, and also the regularized nuclear norm, which is used in the area of low rank matrix problems.

Visualization of the $$\varepsilon $$ ε -subdifferential of piecewise linear–quadratic functions

Computing explicitly the {\epsilon}-subdifferential of a proper function amounts to computing the level set of a convex function namely the conjugate minus a linear function. The resulting theoretical algorithm is applied to the the class of (convex univariate) piecewise linear-quadratic functions for which existing numerical libraries allow practical computations. We visualize the results in a primal, dual, and subdifferential views through several numerical examples. We also provide a visualization of the Br{\o}ndsted-Rockafellar Theorem.

Local Linear Convergence of the ADMM/Douglas--Rachford Algorithms without Strong Convexity and Application to Statistical Imaging

We consider the problem of minimizing the sum of a convex function and a convex function composed with an injective linear mapping. For such problems, subject to a coercivity condition at fixed points of the corresponding Picard iteration, iterates of the alternating directions method of multipliers converge locally linearly to points from which the solution to the original problem can be computed. Our proof strategy uses duality and strong metric subregularity of the Douglas--Rachford fixed point mapping. Our analysis does not require strong convexity and yields error bounds to the set of model solutions. We show in particular that convex piecewise linear-quadratic functions naturally satisfy the requirements of the theory, guaranteeing eventual linear convergence of both the Douglas--Rachford algorithm and the alternating directions method of multipliers for this class of objectives under mild assumptions on the set of fixed points. We demonstrate this result on quantitative image deconvolution and deno...

Computing the partial conjugate of convex piecewise linear-quadratic bivariate functions

Piecewise linear-quadratic (PLQ) functions are an important class of functions in convex analysis since the result of most convex operators applied to a PLQ function is a PLQ function. We modify a recent algorithm for computing the convex (Legendre-Fenchel) conjugate of convex PLQ functions of two variables, to compute its partial conjugate i.e. the conjugate with respect to one of the variables. The structure of the original algorithm is preserved including its time complexity (linear time with some approximation and log-linear time without approximation). Applying twice the partial conjugate (and a variable switching operator) recovers the full conjugate. We present our partial conjugate algorithm, which is more flexible and simpler than the original full conjugate algorithm. We emphasize the difference with the full conjugate algorithm and illustrate results by computing partial conjugates, partial Moreau envelopes, and partial proximal averages.

Computing the conjugate of convex piecewise linear-quadratic bivariate functions

We present a new algorithm to compute the Legendre–Fenchel conjugate of convex piecewise linear-quadratic (PLQ) bivariate functions. The algorithm stores a function using a (primal) planar arrangement. It then computes the (dual) arrangement associated with the conjugate by looping through vertices, edges, and faces in the primal arrangement and building associated dual vertices, edges, and faces. Using optimal computational geometry data structures, the algorithm has a linear time worst-case complexity. We present the algorithm, and illustrate it with numerical examples. We proceed to build a toolbox for convex bivariate PLQ functions by implementing the addition, and scalar multiplication operations. Finally, we compose these operators to compute classical convex analysis operators such as the Moreau envelope, and the proximal average.

Second-Order Subdifferential Calculus with Applications to Tilt Stability in Optimization

The paper concerns the second-order generalized differentiation theory of variational analysis and new applications of this theory to some problems of constrained optimization in finitedimensional spaces. The main attention is paid to the so-called (full and partial) second-order subdifferentials of extended-real-valued functions, which are dual-type constructions generated by coderivatives of frst-order subdifferential mappings. We develop an extended second-order subdifferential calculus and analyze the basic second-order qualification condition ensuring the fulfillment of the principal secondorder chain rule for strongly and fully amenable compositions. The calculus results obtained in this way and computing the second-order subdifferentials for piecewise linear-quadratic functions and their major specifications are applied then to the study of tilt stability of local minimizers for important classes of problems in constrained optimization that include, in particular, problems of nonlinear programming and certain classes of extended nonlinear programs described in composite terms.

Convex Hull Algorithms for Piecewise Linear-Quadratic Functions in Computational Convex Analysis

Computing the convex envelope is a core operation in nonsmooth analysis that bridges the convex with the nonconvex world. Although efficient algorithms to compute fundamental transforms of convex analysis have been proposed over the years, they are limited to convex functions until an efficient algorithm becomes available to compute the convex envelope of a piecewise linear-quadratic function (of one variable) efficiently. We present two such algorithms, one based on maximum and conjugate computation that is easy to implement but has quadratic time complexity, and another based on direct computation that requires more work to implement but has optimal (linear time) complexity. We prove their time (and space) complexity, and compare their performances.

The piecewise linear-quadratic model for computational convex analysis

A new computational framework for computer-aided convex analysis is proposed and investigated. Existing computational frameworks are reviewed and their limitations pointed out. The class of piecewise linear-quadratic functions is introduced to improve convergence and stability. A stable convex calculus is achieved using symbolic-numeric algorithms to compute all fundamental transforms of convex analysis. Our main result states the existence of efficient (linear time) algorithms for the class of piecewise linear-quadratic functions. We also recall that such class is closed under convex transforms. We illustrate the results with numerical examples, and validate numerically the resulting computational framework.

First- and Second-Order Epi-Differentiability in Nonlinear Programming

Problems are considered in which an objective function expressible as a max of finitely many ${C^2}$ functions, or more generally as the composition of a piecewise linear-quadratic function with a ${C^2}$ mapping, is minimized subject to finitely many ${C^2}$ constraints. The essential objective function in such a problem, which is the sum of the given objective and the indicator of the constraints, is shown to be twice epi-differentiable at any point where the active constraints (if any) satisfy the Mangasarian-Fromovitz qualification. The epi-derivatives are defined by taking epigraphical limits of classical first-and second-order difference quotients instead of pointwise limits, and they reveal properties of local geometric approximation that have not previously been observed.

First- and second-order epi-differentiability in nonlinear programming

Problems are considered in which an objective function expressible as a max of finitely many C 2 {C^2} functions, or more generally as the composition of a piecewise linear-quadratic function with a C 2 {C^2} mapping, is minimized subject to finitely many C 2 {C^2} constraints. The essential objective function in such a problem, which is the sum of the given objective and the indicator of the constraints, is shown to be twice epi-differentiable at any point where the active constraints (if any) satisfy the Mangasarian-Fromovitz qualification. The epi-derivatives are defined by taking epigraphical limits of classical first-and second-order difference quotients instead of pointwise limits, and they reveal properties of local geometric approximation that have not previously been observed.