Nonsmooth Optimization Problems Research Articles

Proximal Splitting Algorithms for Convex Optimization: A Tour of Recent Advances, with New Twists

Convex nonsmooth optimization problems, whose solutions live in very high dimensional spaces, have become ubiquitous. To solve them, the class of first-order algorithms known as proximal splitting algorithms is particularly adequate: they consist of simple operations, handling the terms in the objective function separately. In this overview, we demystify a selection of recent proximal splitting algorithms: we present them within a unified framework, which consists in applying splitting methods for monotone inclusions in primal-dual product spaces, with well-chosen metrics. Along the way, we easily derive new variants of the algorithms and revisit existing convergence results, extending the parameter ranges in several cases. In particular, we emphasize that when the smooth term in the objective function is quadratic, e.g., for least-squares problems, convergence is guaranteed with larger values of the relaxation parameter than previously known. Such larger values are usually beneficial for the convergence speed in practice.

Adaptive proximal SGD based on new estimating sequences for sparser ERM

Estimating sequences introduced by Nesterov is an efficient trick to accelerate gradient descent (GD). The stochastic version of estimating sequences is also successfully used to speed up stochastic gradient descent (SGD). In solving the non-smooth convex optimization problems, the convergence rate of SGD with stochastic estimating sequences is O(1/k). Here k is the number of iterations. In this paper, we present a new way of constructing estimating sequences. The characteristic of the new estimating sequences is to replace the subgradient with the proximal stochastic gradient. The novelty of the new estimating sequences is to replace the fixed learning rate with the adaptive learning rate. The adaptive learning rate is calculated by the exponential moving average of past squared stochastic gradients. Based on the new estimating sequences, we propose an adaptive proximal SGD algorithm, called ES-APSGD, for solving the large-scale ℓ1-norm regularized empirical risk minimization (ERM). The proposed ES-APSGD simplifies the calculation and can obtain a convergence rate of O(1/k2). The significant advantage of ES-APSGD is to strengthen the sparsity of solution by adaptively adjusting the threshold magnitude. Experimental results on Lasso and ℓ1-norm regularized logistic regression show that ES-APSGD speeds up convergence and obtains the sparser optimal solutions.

Nonsmooth interval-valued multiobjective optimization problems and generalized variational inequalities on Hadamard manifolds

Nonsmooth interval-valued multiobjective optimization problems and generalized variational inequalities on Hadamard manifolds

A Projection Neural Network to Nonsmooth Constrained Pseudoconvex Optimization.

In this article, a single-layer projection neural network based on penalty function and differential inclusion is proposed to solve nonsmooth pseudoconvex optimization problems with linear equality and convex inequality constraints, and the bound constraints, such as box and sphere types, in inequality constraints are processed by projection operator. By introducing the Tikhonov-like regularization method, the proposed neural network no longer needs to calculate the exact penalty parameters. Under mild assumptions, by nonsmooth analysis, it is proved that the state solution of the proposed neural network is always bounded and globally exists, and enters the constrained feasible region in a finite time, and never escapes from this region again. Finally, the state solution converges to an optimal solution for the considered optimization problem. Compared with some other existing neural networks based on subgradients, this algorithm eliminates the dependence on the selection of the initial point, which is a neural network model with a simple structure and low calculation load. Three numerical experiments and two application examples are used to illustrate the global convergence and effectiveness of the proposed neural network.

A First-Order Primal-Dual Method for Nonconvex Constrained Optimization Based on the Augmented Lagrangian

Nonlinearly constrained nonconvex and nonsmooth optimization models play an increasingly important role in machine learning, statistics, and data analytics. In this paper, based on the augmented Lagrangian function, we introduce a flexible first-order primal-dual method, to be called nonconvex auxiliary problem principle of augmented Lagrangian (NAPP-AL), for solving a class of nonlinearly constrained nonconvex and nonsmooth optimization problems. We demonstrate that NAPP-AL converges to a stationary solution at the rate of [Formula: see text], where k is the number of iterations. Moreover, under an additional error bound condition (to be called HVP-EB in the paper) with exponent [Formula: see text], we further show the global convergence of NAPP-AL. Additionally, if [Formula: see text], then we furthermore show that the convergence rate is in fact linear. Finally, we show that the well-known Kurdyka-Łojasiewicz property and the Hölderian metric subregularity imply the aforementioned HVP-EB condition. We demonstrate that under mild conditions, NAPP-AL can also be interpreted as a variant of the forward-backward operator splitting method in this context. Funding: This work was supported by the National Natural Science Foundation of China [Grant 71871140].

A nonsmooth optimization approach for time-dependent hemivariational inequalities

This paper is devoted to the study of time-dependent hemivariational inequality. We prove the existence and uniqueness of its solution, provide a fully discrete scheme, and reformulate this scheme as a series of nonsmooth optimization problems. The introduced theory is later applied to a sample quasistatic contact problem that describes a viscoelastic body in frictional contact with a foundation. This contact is governed by a nonmonotone friction law with dependence on the normal component of displacement and the tangential component of velocity. Finally, computational simulations are performed to illustrate the obtained results.

Properties of the Quadratic Transformation of Dual Variables

We investigate a solution of a convex programming problem with a strongly convex objective function based on the dual approach. A dual optimization problem has constraints on the positivity of variables. We study the methods and properties of transformations of dual variables that enable us to obtain an unconstrained optimization problem. We investigate the previously known method of transforming the components of dual variables in the form of their modulus (modulus method). We show that in the case of using the modulus method, the degree of the degeneracy of the function increases as it approaches the optimal point. Taking into account the ambiguity of the gradient in the boundary regions of the sign change of the new dual function variables and the increase in the degree of the function degeneracy, we need to use relaxation subgradient methods (RSM) that are difficult to implement and that can solve non-smooth non-convex optimization problems with a high degree of elongation of level surfaces. We propose to use the transformation of the components of dual variables in the form of their square (quadratic method). We prove that the transformed dual function has a Lipschitz gradient with a quadratic method of transformation. This enables us to use efficient gradient methods to find the extremum. The above properties are confirmed by a computational experiment. With a quadratic transformation compared to a modulus transformation, it is possible to obtain a solution of the problem by relaxation subgradient methods and smooth function minimization methods (conjugate gradient method and quasi-Newtonian method) with higher accuracy and lower computational costs. The noted transformations of dual variables were used in the program module for calculating the maximum permissible emissions of enterprises (MPE) of the software package for environmental monitoring of atmospheric air (ERA-AIR).

Second-order optimality conditions in nonsmooth vector optimization

Abstract In this paper, we introduce new classes of nonsmooth second-order cone-convex functions and respective generalizations in terms of first and second-order directional derivative. These classes encapsulate several already existing classes of cone-convex functions and their weaker variants. Second-order KKT type sufficient optimality conditions and duality results for a nonsmooth vector optimization problem are proved using these functions. The results have been supported by examples.

A Bundle Trust Region Algorithm for Minimizing Locally Lipschitz Functions

We propose a new algorithm for minimizing locally Lipschitz functions that combines both the bundle and trust region techniques. Based on the bundle methods the objective function is approximated by a piecewise linear working model which is updated by adding cutting planes at unsuccessful trial steps. The algorithm defines, at each iteration, a new trial point by solving a subproblem that employs the working model in the objective function subject to a region, which is called the trust region. The algorithm is studied from both theoretical and practical points of view. Under a assumption on the objective function, global convergence of it is verified to stationary points. In order to demonstrate the reliability and efficiency of the proposed algorithm, a MATLAB implementation of it is prepared and numerical experiments have been made using some academic nonsmooth test problems. Computational results show that the developed method is efficient for solving nonsmooth and nonconvex optimization problems.

Smooth over-parameterized solvers for non-smooth structured optimization

Non-smooth optimization is a core ingredient of many imaging or machine learning pipelines. Non-smoothness encodes structural constraints on the solutions, such as sparsity, group sparsity, low-rank and sharp edges. It is also the basis for the definition of robust loss functions and scale-free functionals such as square-root Lasso. Standard approaches to deal with non-smoothness leverage either proximal splitting or coordinate descent. These approaches are effective but usually require parameter tuning, preconditioning or some sort of support pruning. In this work, we advocate and study a different route, which operates a non-convex but smooth over-parameterization of the underlying non-smooth optimization problems. This generalizes quadratic variational forms that are at the heart of the popular Iterative Reweighted Least Squares. Our main theoretical contribution connects gradient descent on this reformulation to a mirror descent flow with a varying Hessian metric. This analysis is crucial to derive convergence bounds that are dimension-free. This explains the efficiency of the method when using small grid sizes in imaging. Our main algorithmic contribution is to apply the Variable Projection method which defines a new formulation by explicitly minimizing over part of the variables. This leads to a better conditioning of the minimized functional and improves the convergence of simple but very efficient gradient-based methods, for instance quasi-Newton solvers. We exemplify the use of this new solver for the resolution of regularized regression problems for inverse problems and supervised learning, including total variation prior and non-convex regularizers.

Steering exact penalty DCA for nonsmooth DC optimisation problems with equality and inequality constraints

We propose and study a version of the DCA (Difference-of-Convex functions Algorithm) using the penalty function for solving nonsmooth DC optimisation problems with nonsmooth DC equality and inequality constraints. The method employs an adaptive penalty updating strategy to improve its performance. This strategy is based on the so-called steering exact penalty methodology and relies on solving some auxiliary convex subproblems to determine a suitable value of the penalty parameter. We present a detailed convergence analysis of the method and illustrate its practical performance by applying the method to two nonsmooth discrete optimal control problem.

On optimality conditions and duality theorems for approximate solutions of nonsmooth infinite optimization problems

On optimality conditions and duality theorems for approximate solutions of nonsmooth infinite optimization problems

Short Paper - A note on the Frank–Wolfe algorithm for a class of nonconvex and nonsmooth optimization problems

Frank and Wolfe’s celebrated conditional gradient method is a well-known tool for solving smooth optimization problems for which minimizing a linear function over the feasible set is computationally cheap. However, when the objective function is nonsmooth, the method may fail to compute a stationary point. In this work, we show that the Frank–Wolfe algorithm can be employed to compute Clarke-stationary points for nonconvex and nonsmooth optimization problems consisting of minimizing upper-C 1,α functions over convex and compact sets. Furthermore, under more restrictive assumptions, we propose a new algorithm variant with stronger stationarity guarantees, namely directional stationarity and even local optimality.

A novel sequential optimality condition for smooth constrained optimization and algorithmic consequences

In the smooth constrained optimization setting, this work introduces the Domain Complementary Approximate Karush–Kuhn–Tucker (DCAKKT) condition, inspired by a sequential optimality condition recently devised for non-smooth constrained optimization problems. It is shown that the augmented Lagrangian method can generate limit points satisfying DCAKKT, and it is proved that such a condition is not related to previously established sequential optimality conditions. An essential characteristic of the DCAKKT is to capture the asymptotic potential increasing of the Lagrange multipliers using a single parameter. Besides that, DCAKKT points satisfy the Strong Approximate Gradient Projection (SAGP) condition. Due to the intrinsic features of DCAKKT, which combine strength and generality, this novel and genuine sequential optimality condition may shed some light upon the practical performance of algorithms that are yet to be devised.

A subgradient-based neurodynamic algorithm to constrained nonsmooth nonconvex interval-valued optimization

In this paper, a subgradient-based neurodynamic algorithm is presented to solve the nonsmooth nonconvex interval-valued optimization problem with both partial order and linear equality constraints, where the interval-valued objective function is nonconvex, and interval-valued partial order constraint functions are convex. The designed neurodynamic system is constructed by a differential inclusion with upper semicontinuous right-hand side, whose calculation load is reduced by relieving penalty parameters estimation and complex matrix inversion. Based on nonsmooth analysis and the extension theorem of the solution of differential inclusion, it is obtained that the global existence and boundedness of state solution of neurodynamic system, as well as the asymptotic convergence of state solution to the feasible region and the set of LU-critical points of interval-valued nonconvex optimization problem. Several numerical experiments and the applications to emergency supplies distribution and nondeterministic fractional continuous static games are solved to illustrate the applicability of the proposed neurodynamic algorithm.

Robust amplitude method with <inline-formula><tex-math id="M1">$ L_{1/2} $</tex-math></inline-formula>-regularization for compressive phase retrieval

Compressive phase retrieval aims to recover a sparse signal from the amplitude of its linear measurements when the number of measurements required is far less than the dimension of the signal. In real applications, the measurements are often corrupted by outliers and asymmetrical distribution noise. In this paper, we introduce a novel method for compressive phase retrieval which consists of least absolute deviation loss function and an $ L_{1/2}- $regularization term. It is a nonconvex, nonsmooth, and non-Lipschitz optimization problem. We design an efficient alternating direction method of multipliers(ADMM) to solve the problem and establish its convergence. Extensive numerical experiments demonstrate that the proposed method has higher success probability when the number of measurements is relatively less and is robust to outliers, dense bounded noise as well as Laplace noise.

An abstract convergence framework with application to inertial inexact forward–backward methods

In this paper we introduce a novel abstract descent scheme suited for the minimization of proper and lower semicontinuous functions. The proposed abstract scheme generalizes a set of properties that are crucial for the convergence of several first-order methods designed for nonsmooth nonconvex optimization problems. Such properties guarantee the convergence of the full sequence of iterates to a stationary point, if the objective function satisfies the Kurdyka-Lojasiewicz property. The abstract framework allows for the design of new algorithms. We propose two inertial-type algorithms with implementable inexactness criteria for the main iteration update step. The first algorithm, i$^2$Piano, exploits large steps by adjusting a local Lipschitz constant. The second algorithm, iPila, overcomes the main drawback of line-search based methods by enforcing a descent only on a merit function instead of the objective function. Both algorithms have the potential to escape local minimizers (or stationary points) by leveraging the inertial feature. Moreover, they are proved to enjoy the full convergence guarantees of the abstract descent scheme, which is the best we can expect in such a general nonsmooth nonconvex optimization setup using first-order methods. The efficiency of the proposed algorithms is demonstrated on two exemplary image deblurring problems, where we can appreciate the benefits of performing a linesearch along the descent direction inside an inertial scheme.

Strong stationarity for a highly nonsmooth optimization problem with control constraints

<p style='text-indent:20px;'>This paper deals with a control constrained optimization problem governed by a nonsmooth elliptic PDE in the presence of a non-differentiable objective. The nonsmooth non-linearity in the state equation is locally Lipschitz continuous and directionally differentiable, while one of the nonsmooth terms appearing in the objective is convex. Since these mappings are not necessarily Gâteaux-differentiable, the application of standard adjoint calculus is excluded. Based on their limited differentiability properties, we derive a strong stationary optimality system, i.e., an optimality system which is equivalent to the purely primal optimality condition saying that the directional derivative of the reduced objective in feasible directions is nonnegative.</p>

A redistributed proximal bundle method for nonsmooth nonconvex functions with inexact information

In this paper, we propose a redistributed proximal bundle method for a class of nonconvex nonsmooth optimization problems with inexact information, i.e., we consider the problem of computing the approximate critical points when only the inexact information about the function values and subgradients are available and show that reasonable convergence properties are obtained. We assume that the errors in the computation of functions and subgradients are only bounded and in principle do not have to vanish within the limits. For the nonconvex functions, we design the convexification technique, which ensures that the linearization error of its augmentation function is nonnegative. Meanwhile, for the inexact information, we utilize noise management strategies and update approximate parameters to reduce the impact of inexact information. Based on this method, we can obtain the approximate solution.

Necessary optimality conditions for approximate Pareto efficient solutions of nonsmooth fractional interval-valued multiobjective optimization problems

This paper deals with approximate Pareto efficient solutions of a nonsmooth fractional interval-valued multiobjective optimization. We first introduce some types of approximate Pareto efficient solutions of the considered problem by considering the lower-upper interval order relation. Then we apply some advanced tools of variational analysis and generalized differentiation to establish necessary optimality conditions of Karush–Kuhn–Tucker (KKT)-type for these solutions.