Nonsmooth Convex Optimization Problems Research Articles

A new computational framework for log-concave density estimation

AbstractIn statistics, log-concave density estimation is a central problem within the field of nonparametric inference under shape constraints. Despite great progress in recent years on the statistical theory of the canonical estimator, namely the log-concave maximum likelihood estimator, adoption of this method has been hampered by the complexities of the non-smooth convex optimization problem that underpins its computation. We provide enhanced understanding of the structural properties of this optimization problem, which motivates the proposal of new algorithms, based on both randomized and Nesterov smoothing, combined with an appropriate integral discretization of increasing accuracy. We prove that these methods enjoy, both with high probability and in expectation, a convergence rate of order 1/T up to logarithmic factors on the objective function scale, where T denotes the number of iterations. The benefits of our new computational framework are demonstrated on both synthetic and real data, and our implementation is available in a github repository (Log-Concave Computation).

Inertial self-adaptive algorithms for solving non-smooth convex optimization problems

Inertial self-adaptive algorithms for solving non-smooth convex optimization problems

Accelerated Projection Algorithm Based on Smoothing Approximation for Distributed Nonsmooth Optimization

In this paper, a distributed smoothing accelerated projection algorithm (DSAPA) is proposed to address constrained non-smooth convex optimization problems over undirected multi-agent networks in a distributed manner, where the objective function is free of the assumption of Lipschitz gradient or strong convexity. First, based on a distributed exact penalty method, the original optimization problem is translated to a problem of standard assignment without consensus constraints. Then, a novel DSAPA by combining the smoothing approximation with Nesterov's accelerated schemes is proposed. In addition, we provide a systematic analysis to derive an upper bound on the convergence rate in terms of the objective function based on penalty function and to choose the optimal step size accordingly. Our results demonstrate that the proposed DSAPA can reach <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$O(\frac{\log (k)}{k})$</tex-math></inline-formula> when the optimal step size is chosen. Finally, the effectiveness and correctness of the proposed algorithm are verified by numerical and practical application examples.

Enhancing neurodynamic approach with physics-informed neural networks for solving non-smooth convex optimization problems

This paper proposes a deep learning approach for solving non-smooth convex optimization problems (NCOPs), which have broad applications in computer science, engineering, and physics. Our approach combines neurodynamic optimization with physics-informed neural networks (PINNs) to provide an efficient and accurate solution. We first use neurodynamic optimization to formulate an initial value problem (IVP) that involves a system of ordinary differential equations for the NCOP. We then introduce a modified PINN as an approximate state solution to the IVP. Finally, we develop a dedicated algorithm to train the model to solve the IVP and minimize the NCOP objective simultaneously. Unlike existing numerical integration methods, a key advantage of our approach is that it does not require the computation of a series of intermediate states to produce a prediction of the NCOP. Our experimental results show that this computational feature results in fewer iterations being required to produce more accurate prediction solutions. Furthermore, our approach is effective in finding feasible solutions that satisfy the NCOP constraint.

Adaptive proximal SGD based on new estimating sequences for sparser ERM

Estimating sequences introduced by Nesterov is an efficient trick to accelerate gradient descent (GD). The stochastic version of estimating sequences is also successfully used to speed up stochastic gradient descent (SGD). In solving the non-smooth convex optimization problems, the convergence rate of SGD with stochastic estimating sequences is O(1/k). Here k is the number of iterations. In this paper, we present a new way of constructing estimating sequences. The characteristic of the new estimating sequences is to replace the subgradient with the proximal stochastic gradient. The novelty of the new estimating sequences is to replace the fixed learning rate with the adaptive learning rate. The adaptive learning rate is calculated by the exponential moving average of past squared stochastic gradients. Based on the new estimating sequences, we propose an adaptive proximal SGD algorithm, called ES-APSGD, for solving the large-scale ℓ1-norm regularized empirical risk minimization (ERM). The proposed ES-APSGD simplifies the calculation and can obtain a convergence rate of O(1/k2). The significant advantage of ES-APSGD is to strengthen the sparsity of solution by adaptively adjusting the threshold magnitude. Experimental results on Lasso and ℓ1-norm regularized logistic regression show that ES-APSGD speeds up convergence and obtains the sparser optimal solutions.

An inexact restoration-nonsmooth algorithm with variable accuracy for stochastic nonsmooth convex optimization problems in machine learning and stochastic linear complementarity problems

We study unconstrained optimization problems with nonsmooth and convex objective function in the form of a mathematical expectation. The proposed method approximates the expected objective function with a sample average function using Inexact Restoration-based adapted sample sizes. The sample size is chosen in an adaptive manner based on Inexact Restoration. The algorithm uses line search and assumes descent directions with respect to the current approximate function. We prove the a.s. convergence under standard assumptions. Numerical results for two types of problems, machine learning loss function for training classifiers and stochastic linear complementarity problems, prove the efficiency of the proposed scheme.

Spectral projected subgradient method for nonsmooth convex optimization problems

We consider constrained optimization problems with a nonsmooth objective function in the form of mathematical expectation. The Sample Average Approximation (SAA) is used to estimate the objective function and variable sample size strategy is employed. The proposed algorithm combines an SAA subgradient with the spectral coefficient in order to provide a suitable direction which improves the performance of the first order method as shown by numerical results. The step sizes are chosen from the predefined interval and the almost sure convergence of the method is proved under the standard assumptions in stochastic environment. To enhance the performance of the proposed algorithm, we further specify the choice of the step size by introducing an Armijo-like procedure adapted to this framework. Considering the computational cost on machine learning problems, we conclude that the line search improves the performance significantly. Numerical experiments conducted on finite sum problems also reveal that the variable sample strategy outperforms the full sample approach.

A new randomized primal-dual algorithm for convex optimization with fast last iterate convergence rates

We develop a novel unified randomized block-coordinate primal-dual algorithm to solve a class of nonsmooth constrained convex optimization problems, which covers different existing variants and model settings from the literature. We prove that our algorithm achieves and convergence rates in two cases: merely convexity and strong convexity, respectively, where k is the iteration counter and n is the number of block-coordinates. These rates are known to be optimal (up to a constant factor) when n = 1. Our convergence rates are obtained through three criteria: primal objective residual and primal feasibility violation, dual objective residual, and primal-dual expected gap. Moreover, our rates for the primal problem are on the last-iterate sequence. Our dual convergence guarantee requires additionally a Lipschitz continuity assumption. We specify our algorithm to handle two important special cases, where our rates are still applied. Finally, we verify our algorithm on two well-studied numerical examples and compare it with two existing methods. Our results show that the proposed method has encouraging performance on different experiments.

From the Ravine Method to the Nesterov Method and Vice Versa: A Dynamical System Perspective

We revisit the Ravine method of Gelfand and Tsetlin from a dynamical system perspective, study its convergence properties, and highlight its similarities and differences with the Nesterov accelerated gradient method. The two methods are closely related. They can be deduced from each other by reversing the order of the extrapolation and gradient operations in their definitions. They benefit from similar fast convergence of values and convergence of iterates for general convex objective functions. We will also establish the high resolution ODE of the Ravine and Nesterov methods and reveal an additional geometric damping term driven by the Hessian for both methods. This will allow us to prove fast convergence toward zero of the gradients not only for the Ravine method but also for the Nesterov method for the first time. In the strongly convex case, we show linear convergence for the Ravine method at an optimal rate. We also highlight connections to other algorithms resulting from more subtle discretization schemes and finally describe a Ravine version of the proximal-gradient algorithms for general structured smooth + nonsmooth convex optimization problems.

A globally convergent proximal Newton-type method in nonsmooth convex optimization

The paper proposes and justifies a new algorithm of the proximal Newton type to solve a broad class of nonsmooth composite convex optimization problems without strong convexity assumptions. Based on advanced notions and techniques of variational analysis, we establish implementable results on the global convergence of the proposed algorithm as well as its local convergence with superlinear and quadratic rates. For certain structured problems, the obtained local convergence conditions do not require the local Lipschitz continuity of the corresponding Hessian mappings that is a crucial assumption used in the literature to ensure a superlinear convergence of other algorithms of the proximal Newton type. The conducted numerical experiments of solving the \(l_1\) regularized logistic regression model illustrate the possibility of applying the proposed algorithm to deal with practically important problems.

Linear Convergence of Prox-SVRG Method for Separable Non-smooth Convex Optimization Problems under Bounded Metric Subregularity

Linear Convergence of Prox-SVRG Method for Separable Non-smooth Convex Optimization Problems under Bounded Metric Subregularity

A General Destriping Framework for Remote Sensing Images Using Flatness Constraint

Removing stripe noise, i.e., destriping, from remote sensing images is an essential task in terms of visual quality and subsequent processing. Most existing destriping methods are designed by combining a particular image regularization with a stripe noise characterization that cooperates with the regularization, which precludes us to examine and activate different regularizations to adapt to various target images. To resolve this, two requirements need to be considered: a general framework that can handle a variety of image regularizations in destriping, and a strong stripe noise characterization that can consistently capture the nature of stripe noise, regardless of the choice of image regularization. To this end, this article proposes a general destriping framework using a newly introduced stripe noise characterization, named <i>flatness constraint (FC)</i>, where we can handle various regularization functions in a unified manner. Specifically, we formulate the destriping problem as a nonsmooth convex optimization problem involving a general form of image regularization and the FC. The constraint mathematically models that the intensity of each stripe is constant along one direction, resulting in a strong characterization of stripe noise. For solving the optimization problem, we also develop an efficient algorithm based on a diagonally preconditioned primal-dual splitting algorithm (DP-PDS), which can automatically adjust the step sizes. The effectiveness of our framework is demonstrated through destriping experiments, where we comprehensively compare combinations of a variety of image regularizations and stripe noise characterizations using hyperspectral images (HSIs) and infrared (IR) videos.

Variational Analysis Perspective on Linear Convergence of Some First Order Methods for Nonsmooth Convex Optimization Problems

Variational Analysis Perspective on Linear Convergence of Some First Order Methods for Nonsmooth Convex Optimization Problems

On the rate of convergence of alternating minimization for non-smooth non-strongly convex optimization in Banach spaces

In this paper, the convergence of the fundamental alternating minimization is established for non-smooth non-strongly convex optimization problems in Banach spaces, and novel rates of convergence are provided. As objective function a composition of a smooth, and a block-separable, non-smooth part is considered, covering a large range of applications. For the former, three different relaxations of strong convexity are considered: (i) quasi-strong convexity; (ii) quadratic functional growth; and (iii) plain convexity. With new and improved rates benefiting from both separate steps of the scheme, linear convergence is proved for (i) and (ii), whereas sublinear convergence is showed for (iii).

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

In this paper, we consider a class of constrained nonsmooth convex optimization problems, which have wide applications in signal processing, imaging recovery, machine learning, etc. Recently, theoretical analysis and numerical experiments show that the extrapolation can improve the convergence rate of the algorithms efficiently, which let the proximal gradient algorithm with extrapolation have significant advantages in solving large scale optimization problems. Motivated by the smoothing techniques, and the fast iterative shrinkage-thresholding algorithm (FISTA) proposed by Beck and Teboulle, we bring forward a new accelerated algorithm for solving the considered constrained nonsmooth convex optimization problems. We prove that any accumulated point of the algorithm is an optimal solution of the problem. In the analysis of the algorithm, we consider several different update methods for the smoothing parameter, and give the global convergence rate $O(\\ln~k/k)$ on the objective function values. Moreover, we analyze the convergence rate of the difference on the iterates $\\lim_{k\\rightarrow}\\|x^{k+1}-x^k\\|=0$. Finally, our numerical experiments illustrate the good performance of the proposed algorithm in solving two classes of sparse optimization problems and the positive influence of the extrapolation on the convergence rate.

A new method based on the proximal bundle idea and gradient sampling technique for minimizing nonsmooth convex functions

In this paper, we combine the positive aspects of the gradient sampling (GS) and bundle methods, as the most efficient methods in nonsmooth optimization, to develop a robust method for solving unconstrained nonsmooth convex optimization problems. The main aim of the proposed method is to take advantage of both GS and bundle methods, meanwhile avoiding their drawbacks. At each iteration of this method, to find an efficient descent direction, the GS technique is utilized for constructing a local polyhedral model for the objective function. If necessary, via an iterative improvement process, this initial polyhedral model is improved by some techniques inspired by the bundle and GS methods. The convergence of the method is studied, which reveals that the global convergence property of our method is independent of the number of gradient evaluations required to establish and improve the initial polyhedral models. Thus, the presented method needs much fewer gradient evaluations in comparison to the original GS method. Furthermore, by means of numerical simulations, we show that the presented method provides promising results in comparison with GS methods, especially for large scale problems. Moreover, in contrast with some bundle methods, our method is not very sensitive to the accuracy of supplied gradients.

Non-Smooth Projected Primal-Dual Dynamical Approach to Solve the Extended Fermat-Torricelli Problem

The problem of finding a point in R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> , from which the sum-of-distances to a finite number of nonempty, closed and convex sets is minimum is called generalized Fermat-Torricelli Problem (FTP). In applications, along with the point that minimizes sum-of-distances, it is important to know the points in the convex sets at which the minimum sum-of-distances is achieved. Various formulations existing in literature do not involve finding the optimal points in the convex sets. In this letter, we formulate a non-smooth convex optimization problem, with both the point/set of points which yields the minimum sum-of-distances as well as the corresponding points in the convex sets as primal variables. We term this problem as extended FTP (eFTP). We adopt non-smooth projected primal-dual dynamical approach to solve this problem. The proposed dynamical system can exhibit a continuum of equilibria. Hence we show semistability of the set of optimal points, which is the pertinent notion of stability for such systems. A distributed implementation of the primal-dual dynamical system is also presented in this letter. Four illustrative examples are considered for the simulation based validation of the solution proposed for eFTP.

Wei-Yao-Liu Conjugate Gradient Algorithm for Nonsmooth Convex Optimization Problems

This paper presents a Wei-Yao-Liu conjugate gradient algorithm for nonsmooth convex optimization problem. The proposed algorithm makes use of approximate function and gradient values of the Moreau-Yosida regularization function instead of the corresponding exact values. Under suitable conditions, the global convergence property could be established for the proposed conjugate gradient method. Finally, some numerical results are reported to show the efficiency of our algorithm.

A multi-step doubly stabilized bundle method for nonsmooth convex optimization

In this paper, by incorporating a multi-step scheme into the doubly stabilized bundle method (DSBM) recently developed by Oliveira and Solodov (2016), we propose a multi-step doubly stabilized bundle method (MDSBM) for solving nonsmooth convex optimization problems. In contrast to a single sequence generated by DSBM, the MDSBM generates three related iteration sequences. One is used to build the cutting-planes model of the objective function, another is served as the stability centers, and the third is the sequence of solutions to our new doubly stabilized subproblems. In addition, we present a new descent test criterion, aiming to take advantage of the multi-step scheme. We establish global convergence of the proposed method, and finally present some promising numerical results.

Non-stationary First-Order Primal-Dual Algorithms with Faster Convergence Rates

In this paper, we propose two novel non-stationary first-order primal-dual algorithms to solve non-smooth composite convex optimization problems. Unlike existing primal-dual schemes where the param...