Convex Nonsmooth Function Research Articles

Linear Convergence of the Derivative-Free Proximal Bundle Method on Convex Nonsmooth Functions, with Application to the Derivative-Free VU\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{VU}$\\end{document}-Algorithm

Proximal bundle methods are a class of optimisation algorithms that leverage the proximal operator to address nonsmoothness in the objective function efficiently. This study focuses on a derivative-free (DFO) proximal bundle method and one of its applications called the DFO VU\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{VU}$\\end{document}-algorithm. These algorithms incorporate approximate proximal points as subprocedures in order to optimise convex nonsmooth functions based on approximated subdifferential information. Interestingly, the classical VU\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{VU}$\\end{document}-algorithm, which operates on true subgradient values, achieves superlinear convergence. At each iteration, the algorithm divides the whole space into two: the smooth U\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{U}$\\end{document}-space and the nonsmooth V\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{V}$\\end{document}-space. It takes a Newton-like step on the U\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{U}$\\end{document}-space and a proximal-point step on the V\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{V}$\\end{document}-space, enabling it to handle both smooth and nonsmooth parts effectively and converge faster. In this work, we reveal the worst possible convergence rate for the DFO VU\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal{VU}$\\end{document}-method by showing the linear convergence of the DFO proximal bundle method. This will be done by presenting a suitable framework and using the subdifferential-based error bound on the distance to critical points.

A Bregman proximal subgradient algorithm for nonconvex and nonsmooth fractional optimization problems

In this paper, we study a class of nonconvex and nonsmooth fractional optimization problem, where the numerator of which is the sum of a nonsmooth and nonconvex function and a relative smooth nonconvex function, while the denominator is relative weakly convex nonsmooth function. We propose a Bregman proximal subgradient algorithm for solving this type of fractional optimization problems. Under moderate conditions, we prove that the subsequence generated by the proposed algorithm converges to a critical point, and the generated sequence globally converges to a critical point when the objective function satisfies the Kurdyka-Łojasiewicz property. We also obtain the convergence rate of the proposed algorithm. Finally, two numerical experiments illustrate the effectiveness and superiority of the algorithm. Our results give a positive answer to an open problem proposed by Bot et al. [14].

Proximal stochastic recursive momentum algorithm for nonsmooth nonconvex optimization problems

In this paper, we mainly consider a class of nonconvex nonsmooth optimization problems, whose objective function is the sum of a smooth function with a Lipschitz continuous gradient and a convex nonsmooth function. We first propose a proximal stochastic recursive momentum algorithm(ProxSTORM) with mini-batch for solving the optimization problems and consider its convergence behaviour. Then, based on the Polyak–Łojasiewicz inequality, we establish the global linear convergence rate of ProxSTORM. Finally, some numerical experiments have been conducted to illustrate the efficiency of our method.

Selective linearization for multi-block statistical learning

We consider the problem of minimizing a sum of several convex non-smooth functions and discuss the selective linearization method (SLIN), which iteratively linearizes all but one of the functions and employs simple proximal steps. The algorithm is a form of multiple operator splitting in which the order of processing partial functions is not fixed, but rather determined in the course of calculations. SLIN is globally convergent for an arbitrary number of component functions without artificial duplication of variables. We report results from extensive numerical experiments in two statistical learning settings such as large-scale overlapping group Lasso and doubly regularized support vector machine. In each setting, we introduce novel and efficient solutions for solving sub-problems. The numerical results demonstrate the efficacy and accuracy of SLIN.

On the Convergence to Stationary Points of Deterministic and Randomized Feasible Descent Directions Methods

This paper studies the class of nonsmooth nonconvex problems in which the difference between a continuously differentiable function and a convex nonsmooth function is minimized over linear constraints. Our goal is to attain a point satisfying the stationarity necessary optimality condition, defined as the lack of feasible descent directions. Although elementary in smooth optimization, this condition is nontrivial when the objective function is nonsmooth, and, correspondingly, there are very few methods that obtain stationary points in such settings. We prove that stationarity in our model can be characterized by a finite number of directions and develop two methods, one deterministic and one random, that use these directions to obtain stationary points. Numerical experiments illustrate the benefit of obtaining a stationary point and the advantage of using the random method to do so.

“Active-set complexity” of proximal gradient: How long does it take to find the sparsity pattern?

Proximal gradient methods have been found to be highly effective for solving minimization problems with non-negative constraints or L1-regularization. Under suitable nondegeneracy conditions, it is known that these algorithms identify the optimal sparsity pattern for these types of problems in a finite number of iterations. However, it is not known how many iterations this may take. We introduce the notion of the "active-set complexity", which in these cases is the number of iterations before an algorithm is guaranteed to have identified the final sparsity pattern. We further give a bound on the active-set complexity of proximal gradient methods in the common case of minimizing the sum of a strongly-convex smooth function and a separable convex non-smooth function.

A Selective Linearization Method For Multiblock Convex Optimization

We consider the problem of minimizing a sum of several convex nonsmooth functions. We introduce a new algorithm called the selective linearization method, which iteratively linearizes all but one of the functions and employs simple proximal steps. The algorithm is a form of multiple operator splitting in which the order of processing partial functions is not fixed, but rather determined in the course of calculations. Global convergence is proved and estimates of the convergence rate are derived. Specifically, the number of iterations needed to achieve solution accuracy $\varepsilon$ is of order $\mathcal{O}(\ln(1/\varepsilon)/\varepsilon)$.

A minimization method on the basis of embedding the feasible set and the epigraph

We propose a conditional minimization method of the convex nonsmooth function which belongs to the class of cutting-plane methods. During constructing iteration points a feasible set and an epigraph of the objective function are approximated by the polyhedral sets. In this connection, auxiliary problems of constructing iteration points are linear programming problems. In optimization process there is some opportunity of updating sets which approximate the epigraph. These updates are performed by periodically dropping of cutting planes which form embedding sets. Convergence of the proposed method is proved, some realizations of the method are discussed.

Aggregate codifferential method for nonsmooth DC optimization

A new algorithm is developed based on the concept of codifferential for minimizing the difference of convex nonsmooth functions. Since the computation of the whole codifferential is not always possible, we use a fixed number of elements from the codifferential to compute the search directions. The convergence of the proposed algorithm is proved. The efficiency of the algorithm is demonstrated by comparing it with the subgradient, the truncated codifferential and the proximal bundle methods using nonsmooth optimization test problems.

Primal and dual alternating direction algorithms for ℓ 1-ℓ 1-norm minimization problems in compressive sensing

In this paper, we propose, analyze and test primal and dual versions of the alternating direction algorithm for the sparse signal reconstruction from its major noise contained observation data. The algorithm minimizes a convex non-smooth function consisting of the sum of l 1-norm regularization term and l 1-norm data fidelity term. We minimize the corresponding augmented Lagrangian function alternatively from either primal or dual forms. Both of the resulting subproblems admit explicit solutions either by using a one-dimensional shrinkage or by an efficient Euclidean projection. The algorithm is easily implementable and it requires only two matrix-vector multiplications per-iteration. The global convergence of the proposed algorithm is established under some technical conditions. The extensions to the non-negative signal recovery problem and the weighted regularization minimization problem are also discussed and tested. Numerical results illustrate that the proposed algorithm performs better than the state-of-the-art algorithm YALL1.

Smoothing and First Order Methods: A Unified Framework

We propose a unifying framework that combines smoothing approximation with fast first order algorithms for solving nonsmooth convex minimization problems. We prove that independently of the structure of the convex nonsmooth function involved, and of the given fast first order iterative scheme, it is always possible to improve the complexity rate and reach an $O(\varepsilon^{-1})$ efficiency estimate by solving an adequately smoothed approximation counterpart. Our approach relies on the combination of the notion of smoothable functions that we introduce with a natural extension of the Moreau-infimal convolution technique along with its connection to the smoothing mechanism via asymptotic functions. This allows for clarification and unification of several issues on the design, analysis, and potential applications of smoothing methods when combined with fast first order algorithms.

An iterative method for minimizing a convex nonsmooth function on a convex smooth surface

An iterative algorithm is proposed for the constrained minimization of a convex nonsmooth function on a set given as a convex smooth surface. The convergence of the algorithm in the sense of necessary conditions for a local minimum is proved.

A multipoint method of minimization of a convex function

The problem of minimization of a convex nonsmooth function in a finite-dimensional space is considered. The method employs the Moreau-Yosida regularization. To accelerate the computation process, the proximate function is constructed using quasi-Newton matrices.

Quadratic Approximations in Convex Nondifferentiable Optimization

An implementable descent method for the unconstrained minimization of convex nonsmooth functions of several variables is described. The algorithm is characterized by the use of a set of quadratic approximations of the objective function in order to compute the search direction. The resulting direction finding subproblem is shown to be equivalent to a structured parametric quadratic programming problem. The convergence of the algorithm to the minimum is proved, and numerical experience is reported.

Decomposition method of descent for minimizing the sum of convex nonsmooth functions

This paper presents a descent method for minimizing a sum of possibly nonsmooth convex functions. Search directions are found by solving subproblems obtained by replacing all but one of the component functions with their polyhedral approximations and adding a quadratic term. The algorithm is globally convergent and terminates when the objective function happens to be polyhedral. It yields a new decomposition method for solving large-scale linear programs with dual block-angular structure.

A bundle type approach to the unconstrained minimization of convex nonsmooth functions

A numerical method for the unconstrained minimization of a convex nonsmooth function of several variables is presented. It is closely related to the ‘bundle type’ approach and to the conjugate subgradient method. A way is suggested to reduce the amount of information to be stored during the computational procedure. Global convergence of the method to the minimum is proved.