Nonsmooth Convex Optimization Research Articles

Exact penalties for decomposable convex optimization problems

We consider a general decomposable convex optimization problem. By using right-hand side allocation technique, it can be transformed into a collection of small dimensional optimization problems. The master problem is a convex non-smooth optimization problem. We propose to apply the exact non-smooth penalty method, which gives a solution of the initial problem under some fixed penalty parameter and provides the consistency of lower level problems. The master problem can be solved with a suitable non-smooth optimization method. The simplest of them is the custom subgradient projection method using the divergent series step-size rule without line-search, whose convergence may be, however, rather low. We suggest to enhance its step-size selection by using a two-speed rule. Preliminary results of computational experiments confirm efficiency of this technique.

Variational Analysis Perspective on Linear Convergence of Some First Order Methods for Nonsmooth Convex Optimization Problems

Variational Analysis Perspective on Linear Convergence of Some First Order Methods for Nonsmooth Convex Optimization Problems

On the rate of convergence of alternating minimization for non-smooth non-strongly convex optimization in Banach spaces

In this paper, the convergence of the fundamental alternating minimization is established for non-smooth non-strongly convex optimization problems in Banach spaces, and novel rates of convergence are provided. As objective function a composition of a smooth, and a block-separable, non-smooth part is considered, covering a large range of applications. For the former, three different relaxations of strong convexity are considered: (i) quasi-strong convexity; (ii) quadratic functional growth; and (iii) plain convexity. With new and improved rates benefiting from both separate steps of the scheme, linear convergence is proved for (i) and (ii), whereas sublinear convergence is showed for (iii).

Randomized Smoothing Variance Reduction Method for Large-Scale Non-smooth Convex Optimization

Randomized Smoothing Variance Reduction Method for Large-Scale Non-smooth Convex Optimization

Strongly convex optimization for joint fractal feature estimation and texture segmentation

The present work investigates the segmentation of textures by formulating it as a strongly convex optimization problem, aiming to favor piecewise constancy of fractal features (local variance and local regularity) widely used to model real-world textures in numerous applications very different in nature. Two objective functions combining these two features are compared, referred to as joint and coupled , promoting either independent or co-localized changes in local variance and regularity. To solve the resulting convex nonsmooth optimization problems, because the processing of large size images and databases are targeted, two categories of proximal algorithms (dual forward-backward and primal-dual), are devised and compared. An in-depth study of the objective functions, notably of their strong convexity, memory and computational costs, permits to propose significantly accelerated algorithms. A class of synthetic models of piecewise fractal texture is constructed and studied. They enable, by means of large-scale Monte-Carlo simulations, to quantify the benefits in texture segmentation of combining local regularity and local variance (as opposed to regularity only) while using strong-convexity accelerated primal-dual algorithms. Achieved results also permit to discuss the gains/costs in imposing co-localizations of changes in local regularity and local variance in the problem formulation. Finally, the potential of the proposed approaches is illustrated on real-world textures taken from a publicly available and documented database.

Iteration complexity analysis of a partial LQP-based alternating direction method of multipliers

In this paper, we consider a prototypical convex optimization problem with multi-block variables and separable structures. By adding the Logarithmic Quadratic Proximal (LQP) regularizer with suitable proximal parameter to each of the first grouped subproblems, we develop a partial LQP-based Alternating Direction Method of Multipliers (ADMM-LQP). The dual variable is updated twice with relatively larger stepsizes than the classical region (0,1+52). Using a prediction-correction approach to analyze properties of the iterates generated by ADMM-LQP, we establish its global convergence and sublinear convergence rate of O(1/T) in the new ergodic and nonergodic senses, where T denotes the iteration index. We also extend the algorithm to a nonsmooth composite convex optimization and establish similar convergence results as our ADMM-LQP.

Distributed Nonsmooth Convex Optimization over Markovian Switching Random Networks with Two Step-Sizes

This paper investigates the distributed convex optimization problem over a multi-agent system with Markovian switching communication networks. The objective function is the sum of each agent’s local nonsmooth objective function, which cannot be known by other agents. The communication network is assumed to switch over a set of weight-balanced directed graphs with a Markovian property. The authors propose a consensus sub-gradient algorithm with two time-scale step-sizes to handle the Markovian switching topologies and the absence of global gradient information. With proper selection of step-sizes, the authors prove the almost sure convergence of all agents’ local estimates to the same optimal solution when the union graph of the Markovian network’ states is strongly connected and the Markovian chain is irreducible. The convergence rate analysis is also given for specific cases. Simulations are given to demonstrate the results.

Momentum Acceleration in the Individual Convergence of Nonsmooth Convex Optimization With Constraints.

Momentum technique has recently emerged as an effective strategy in accelerating convergence of gradient descent (GD) methods and exhibits improved performance in deep learning as well as regularized learning. Typical momentum examples include Nesterov's accelerated gradient (NAG) and heavy-ball (HB) methods. However, so far, almost all the acceleration analyses are only limited to NAG, and a few investigations about the acceleration of HB are reported. In this article, we address the convergence about the last iterate of HB in nonsmooth optimizations with constraints, which we name individual convergence. This question is significant in machine learning, where the constraints are required to impose on the learning structure and the individual output is needed to effectively guarantee this structure while keeping an optimal rate of convergence. Specifically, we prove that HB achieves an individual convergence rate of O(1/√t) , where t is the number of iterations. This indicates that both of the two momentum methods can accelerate the individual convergence of basic GD to be optimal. Even for the convergence of averaged iterates, our result avoids the disadvantages of the previous work in restricting the optimization problem to be unconstrained as well as limiting the performed number of iterations to be predefined. The novelty of convergence analysis presented in this article provides a clear understanding of how the HB momentum can accelerate the individual convergence and reveals more insights about the similarities and differences in getting the averaging and individual convergence rates. The derived optimal individual convergence is extended to regularized and stochastic settings, in which an individual solution can be produced by the projection-based operation. In contrast to the averaged output, the sparsity can be reduced remarkably without sacrificing the theoretical optimal rates. Several real experiments demonstrate the performance of HB momentum strategy.

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

In this paper, we consider a class of constrained nonsmooth convex optimization problems, which have wide applications in signal processing, imaging recovery, machine learning, etc. Recently, theoretical analysis and numerical experiments show that the extrapolation can improve the convergence rate of the algorithms efficiently, which let the proximal gradient algorithm with extrapolation have significant advantages in solving large scale optimization problems. Motivated by the smoothing techniques, and the fast iterative shrinkage-thresholding algorithm (FISTA) proposed by Beck and Teboulle, we bring forward a new accelerated algorithm for solving the considered constrained nonsmooth convex optimization problems. We prove that any accumulated point of the algorithm is an optimal solution of the problem. In the analysis of the algorithm, we consider several different update methods for the smoothing parameter, and give the global convergence rate $O(\\ln~k/k)$ on the objective function values. Moreover, we analyze the convergence rate of the difference on the iterates $\\lim_{k\\rightarrow}\\|x^{k+1}-x^k\\|=0$. Finally, our numerical experiments illustrate the good performance of the proposed algorithm in solving two classes of sparse optimization problems and the positive influence of the extrapolation on the convergence rate.

Fast proximal algorithms for nonsmooth convex optimization

In the lines of our previous approach to devise proximal algorithms for nonsmooth convex optimization by applying Nesterov fast gradient concept to the Moreau–Yosida regularization of a convex function, we develop three new proximal algorithms for nonsmooth convex optimization. In these algorithms, the errors in computing approximate solutions for the Moreau–Yosida regularization are not fixed beforehand, while preserving the complexity estimates already established. We report some preliminary computational results to give a first estimate of their performance.

Extending interior‐point methods to nonlinear second‐order cone programming: Application to finite‐strain elastoplasticity

AbstractInterior‐point methods (IPMs) are well suited for solving convex nonsmooth optimization problems which arise for instance in problems involving plasticity or contact conditions. This work attempts at extending their field of application to optimization problems involving either smooth but nonconvex or nonsmooth but convex objectives or constraints. A typical application for such kind of problems is finite‐strain elastoplasticity which we address using a total Lagrangian formulation based on logarithmic strain measures. The proposed interior‐point algorithm is implemented and tested on 3D examples involving plastic collapse and geometrical changes. Comparison with classical, Newton‐Raphson/return mapping methods show that the IPM exhibits good computational performance, especially in terms of convergence robustness. Similar to what is observed for convex small‐strain plasticity, the IPM is able to converge for much larger load steps than classical methods.

A new method based on the proximal bundle idea and gradient sampling technique for minimizing nonsmooth convex functions

In this paper, we combine the positive aspects of the gradient sampling (GS) and bundle methods, as the most efficient methods in nonsmooth optimization, to develop a robust method for solving unconstrained nonsmooth convex optimization problems. The main aim of the proposed method is to take advantage of both GS and bundle methods, meanwhile avoiding their drawbacks. At each iteration of this method, to find an efficient descent direction, the GS technique is utilized for constructing a local polyhedral model for the objective function. If necessary, via an iterative improvement process, this initial polyhedral model is improved by some techniques inspired by the bundle and GS methods. The convergence of the method is studied, which reveals that the global convergence property of our method is independent of the number of gradient evaluations required to establish and improve the initial polyhedral models. Thus, the presented method needs much fewer gradient evaluations in comparison to the original GS method. Furthermore, by means of numerical simulations, we show that the presented method provides promising results in comparison with GS methods, especially for large scale problems. Moreover, in contrast with some bundle methods, our method is not very sensitive to the accuracy of supplied gradients.

Non-Smooth Projected Primal-Dual Dynamical Approach to Solve the Extended Fermat-Torricelli Problem

The problem of finding a point in R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> , from which the sum-of-distances to a finite number of nonempty, closed and convex sets is minimum is called generalized Fermat-Torricelli Problem (FTP). In applications, along with the point that minimizes sum-of-distances, it is important to know the points in the convex sets at which the minimum sum-of-distances is achieved. Various formulations existing in literature do not involve finding the optimal points in the convex sets. In this letter, we formulate a non-smooth convex optimization problem, with both the point/set of points which yields the minimum sum-of-distances as well as the corresponding points in the convex sets as primal variables. We term this problem as extended FTP (eFTP). We adopt non-smooth projected primal-dual dynamical approach to solve this problem. The proposed dynamical system can exhibit a continuum of equilibria. Hence we show semistability of the set of optimal points, which is the pertinent notion of stability for such systems. A distributed implementation of the primal-dual dynamical system is also presented in this letter. Four illustrative examples are considered for the simulation based validation of the solution proposed for eFTP.

Wei-Yao-Liu Conjugate Gradient Algorithm for Nonsmooth Convex Optimization Problems

This paper presents a Wei-Yao-Liu conjugate gradient algorithm for nonsmooth convex optimization problem. The proposed algorithm makes use of approximate function and gradient values of the Moreau-Yosida regularization function instead of the corresponding exact values. Under suitable conditions, the global convergence property could be established for the proposed conjugate gradient method. Finally, some numerical results are reported to show the efficiency of our algorithm.

An ADMM-based SQP method for separably smooth nonconvex optimization

This work is about a splitting approach for solving separably smooth nonconvex linearly constrained optimization problems. Based on the ideas from two classical methods, namely the sequential quadratic programming (SQP) and the alternating direction method of multipliers (ADMM), we propose an ADMM-based SQP method. We focus on decomposing the quadratic programming (QP) subproblem of the primal problem into small-scale QP subproblems, which further embedded with Bregman distances can be solved effectively and followed by a dual ascent type update for the Lagrangian multipliers. Under suitable conditions as well as the crucial Kurdyka–Łojasiewicz property, we establish the global and strong convergence properties of the proposed method.

A multi-step doubly stabilized bundle method for nonsmooth convex optimization

In this paper, by incorporating a multi-step scheme into the doubly stabilized bundle method (DSBM) recently developed by Oliveira and Solodov (2016), we propose a multi-step doubly stabilized bundle method (MDSBM) for solving nonsmooth convex optimization problems. In contrast to a single sequence generated by DSBM, the MDSBM generates three related iteration sequences. One is used to build the cutting-planes model of the objective function, another is served as the stability centers, and the third is the sequence of solutions to our new doubly stabilized subproblems. In addition, we present a new descent test criterion, aiming to take advantage of the multi-step scheme. We establish global convergence of the proposed method, and finally present some promising numerical results.

Distributed continuous-time algorithm for nonsmooth optimal consensus without sharing local decision variables

A distributed continuous-time algorithm is proposed for constrained nonsmooth convex optimization. A distinct feature of our algorithm is that it does not require agents to share their local decision variables to the network, and it still achieves the optimal solution. With the help of Lagrangian functions, exact penalty techniques, differential inclusions with maximal monotone maps and saddle-point dynamics, we prove the convergence of the proposed algorithm and show that it achieves an O(1/t) convergence rate. Numerical example also illustrates the effectiveness of the proposed method.

Non-stationary First-Order Primal-Dual Algorithms with Faster Convergence Rates

In this paper, we propose two novel non-stationary first-order primal-dual algorithms to solve non-smooth composite convex optimization problems. Unlike existing primal-dual schemes where the param...

Parallel computing proximal method for nonsmooth convex optimization with fixed point constraints of quasi-nonexpansive mappings

Parallel computing proximal method for nonsmooth convex optimization with fixed point constraints of quasi-nonexpansive mappings

Distributed Control Enforcing Group Sparsity in Smart Grids

In modern smart grids, charging of local energy storage devices is coordinated within the distribution grid to compensate the volatile aggregated power demand on the time interval of interest. However, this results in a perpetual usage of all batteries which in return reduces their lifetime. In this paper, we enforce group sparsity by using an ℓp,q-regularization on the control to counteract this phenomenon. This leads to a non-smooth convex optimization problem, for which a tailored Alternating Direction Method of Multipliers algorithm is proposed. Furthermore, the algorithm is embedded in a Model Predictive Control framework. Numerical simulations show that the proposed scheme yields sparse control while achieving reasonable overall peak shaving.