Nonsmooth Nonconvex Optimization Problems Research Articles

Normalized penalty gradient flow: A continuous-time approach for solving constrained nonconvex nonsmooth optimization problems

To avoid calculating penalty parameters, this paper introduces a continuous-time approach that combines the normalized gradient flow with the penalty method to solve the nonconvex nonsmooth optimization problem with a convex constraint set. Subsequently, this approach is extended to solve nonconvex nonsmooth optimization with a box constraint set and affine equality constraints. Compared with the current methods, the proposed approach does not require calculating the penalty parameters, allows the objective function to be nonconvex or nonsmooth, and allows the initial point to be arbitrarily selected. It is proved that the solution trajectory with any initial point converges to the critical point set of the optimization problem. Finally, the effectiveness of the proposed approach is authenticated through several numerical experiments.

Solving a Class of Nonsmooth Nonconvex Optimization Problems Via Proximal Alternating Linearization Scheme with Inexact Information

For optimization problems minimizing the sum of two nonconvex and nonsmooth functions, we propose an alternate linearization method with inexact data. In many practical optimization applications, only the inexact information of the function can be obtained. The core idea of this method is to add a quadratic function term to the nonconvex function(called local convexification of nonconvex function), and then to construct an approximate proximal point model. In each iteration, a series of iteration points are obtained by solving subproblems alternately. It can be proved that, in the sense of inexact oracles, these iteration points converge to the stable point of the original problem, and theoretically show that the algorithm has good convergent properties.

A Primal-Dual Smoothing Framework for Max-Structured Non-Convex Optimization

We propose a primal-dual smoothing framework for finding a near-stationary point of a class of nonsmooth nonconvex optimization problems with max-structure. We analyze the primal and dual gradient complexities of the framework via two approaches, that is, the dual-then-primal and primal-the-dual smoothing approaches. Our framework improves the best-known oracle complexities of the existing method, even in the restricted problem setting. As an important part of our framework, we propose a first-order method for solving a class of (strongly) convex-concave saddle-point problems, which is based on a newly developed non-Hilbertian inexact accelerated proximal gradient algorithm for strongly convex composite minimization that enjoys duality-gap convergence guarantees. Some variants and extensions of our framework are also discussed. Funding: R. Zhao’s research is partially supported by AFOSR [Grant FA9550-22-1-0356].

Solving nonsmooth interval optimization problems based on interval-valued symmetric invexity

This paper focuses on a nonsmooth nonconvex interval-valued optimization problem. For this, we propose interval-valued symmetric invexity, interval-valued symmetric pseudo-invexity and interval-valued symmetric quasi-invexity in terms of the symmetric gH-differentiable interval-valued functions. Some important properties of these generalized convexities are also discussed. By utilizing these new concepts, we establish sufficient Karush–Kuhn–Tucker conditions for the considered problem. Further, the Wolfe and Mond–Weir type dual problems are associated and weak, strong and strict converse duality results have been derived. Finally, we apply the developed theory to a binary classification problem of interval data by support vector machine.

A Lagrange–Newton algorithm for tensor sparse principal component analysis

This paper is concerned with the tensor sparse principal component analysis (TSPCA) by obtaining principal components which are linear combinations of a small subset of the original features for tensorial data. The core mathematical model can be formulated as a nonsmooth nonconvex optimization problem with a polynomial objective function, and with the sparsity constraint and the unit Euclidean spherical constraint. By employing the tools in tensor analysis, along with the variational properties for the involved -norm, the optimality condition of TSPCA is analysed in terms of stationary points. To well resolve the problem, we reformulate the stationary conditions into the Lagrange stationary equation system via the property of the projection operator onto the sparsity constraint set. With special emphasis on the Jacobian nonsingularity of the corresponding nonlinear system, we propose the Lagrange–Newton algorithm for pursuing the stationary point, which serves as a promising approximation of the optimal solution to TSPCA. The locally quadratic convergence rate is also established under mild conditions. Numerical experiments illustrate the effectiveness of our proposed TSPCA approach in terms of the solution accuracy as well as the computation time.

An alternating structure-adapted Bregman proximal gradient descent algorithm for constrained nonconvex nonsmooth optimization problems and its inertial variant

An alternating structure-adapted Bregman proximal gradient descent algorithm for constrained nonconvex nonsmooth optimization problems and its inertial variant

A proximal subgradient algorithm with extrapolation for structured nonconvex nonsmooth problems

In this paper, we consider a class of structured nonconvex nonsmooth optimization problems, in which the objective function is formed by the sum of a possibly nonsmooth nonconvex function and a differentiable function with Lipschitz continuous gradient, subtracted by a weakly convex function. This general framework allows us to tackle problems involving nonconvex loss functions and problems with specific nonconvex constraints, and it has many applications such as signal recovery, compressed sensing, and optimal power flow distribution. We develop a proximal subgradient algorithm with extrapolation for solving these problems with guaranteed subsequential convergence to a stationary point. The convergence of the whole sequence generated by our algorithm is also established under the widely used Kurdyka–Łojasiewicz property. To illustrate the promising numerical performance of the proposed algorithm, we conduct numerical experiments on two important nonconvex models. These include a compressed sensing problem with a nonconvex regularization and an optimal power flow problem with distributed energy resources.

Low tensor train and low multilinear rank approximations of 3D tensors for compression and de-speckling of optical coherence tomography images

Objective. Many methods for compression and/or de-speckling of 3D optical coherence tomography (OCT) images operate on a slice-by-slice basis and, consequently, ignore spatial relations between the B-scans. Thus, we develop compression ratio (CR)-constrained low tensor train (TT)—and low multilinear (ML) rank approximations of 3D tensors for compression and de-speckling of 3D OCT images. Due to inherent denoising mechanism of low-rank approximation, compressed image is often even of better quality than the raw image it is based on. Approach. We formulate CR-constrained low rank approximations of 3D tensor as parallel non-convex non-smooth optimization problems implemented by alternating direction method of multipliers of unfolded tensors. In contrast to patch- and sparsity-based OCT image compression methods, proposed approach does not require clean images for dictionary learning, enables CR as high as 60:1, and it is fast. In contrast to deep networks based OCT image compression, proposed approach is training free and does not require any supervised data pre-processing. Main results. Proposed methodology is evaluated on twenty four images of a retina acquired on Topcon 3D OCT-1000 scanner, and twenty images of a retina acquired on Big Vision BV1000 3D OCT scanner. For the first dataset, statistical significance analysis shows that for CR ≤ 35, all low ML rank approximations and Schatten-0 (S 0) norm constrained low TT rank approximation can be useful for machine learning-based diagnostics by using segmented retina layers. Also for CR ≤ 35, S 0-constrained ML rank approximation and S 0-constrained low TT rank approximation can be useful for visual inspection-based diagnostics. For the second dataset, statistical significance analysis shows that for CR ≤ 60 all low ML rank approximations as well as S 0 and S 1/2 low TT ranks approximations can be useful for machine learning-based diagnostics by using segmented retina layers. Also, for CR ≤ 60, low ML rank approximations constrained with S p , p ∊ {0, 1/2, 2/3} and one surrogate of S 0 can be useful for visual inspection-based diagnostics. That is also true for low TT rank approximations constrained with S p , p ∊ {0, 1/2, 2/3} for CR ≤ 20. Significance. Studies conducted on datasets acquired by two different types of scanners confirmed capabilities of proposed framework that, for a wide range of CRs, yields de-speckled 3D OCT images suitable for clinical data archiving and remote consultation, for visual inspection-based diagnosis and for machine learning-based diagnosis by using segmented retina layers.

Distributed Proximal Gradient Algorithm for Nonconvex Optimization Over Time-Varying Networks

This paper studies the distributed non-convex optimization problem with non-smooth regularization, which has wide applications in decentralized learning, estimation and control. The objective function is the sum of local objective functions, which consist of differentiable (possibly non-convex) cost functions and non-smooth convex functions. This paper presents a distributed proximal gradient algorithm for the non-smooth non-convex optimization problem. Over time-varying multi-agent networks, the proposed algorithm updates local variable estimates with a constant step-size at the cost of multiple consensus steps, where the number of communication rounds increases over time. We prove that the generated local variables achieve consensus and converge to the set of critical points. Finally, we verify the efficiency of the proposed algorithm by numerical simulations.

Properties of the Quadratic Transformation of Dual Variables

We investigate a solution of a convex programming problem with a strongly convex objective function based on the dual approach. A dual optimization problem has constraints on the positivity of variables. We study the methods and properties of transformations of dual variables that enable us to obtain an unconstrained optimization problem. We investigate the previously known method of transforming the components of dual variables in the form of their modulus (modulus method). We show that in the case of using the modulus method, the degree of the degeneracy of the function increases as it approaches the optimal point. Taking into account the ambiguity of the gradient in the boundary regions of the sign change of the new dual function variables and the increase in the degree of the function degeneracy, we need to use relaxation subgradient methods (RSM) that are difficult to implement and that can solve non-smooth non-convex optimization problems with a high degree of elongation of level surfaces. We propose to use the transformation of the components of dual variables in the form of their square (quadratic method). We prove that the transformed dual function has a Lipschitz gradient with a quadratic method of transformation. This enables us to use efficient gradient methods to find the extremum. The above properties are confirmed by a computational experiment. With a quadratic transformation compared to a modulus transformation, it is possible to obtain a solution of the problem by relaxation subgradient methods and smooth function minimization methods (conjugate gradient method and quasi-Newtonian method) with higher accuracy and lower computational costs. The noted transformations of dual variables were used in the program module for calculating the maximum permissible emissions of enterprises (MPE) of the software package for environmental monitoring of atmospheric air (ERA-AIR).

A subgradient-based neurodynamic algorithm to constrained nonsmooth nonconvex interval-valued optimization

In this paper, a subgradient-based neurodynamic algorithm is presented to solve the nonsmooth nonconvex interval-valued optimization problem with both partial order and linear equality constraints, where the interval-valued objective function is nonconvex, and interval-valued partial order constraint functions are convex. The designed neurodynamic system is constructed by a differential inclusion with upper semicontinuous right-hand side, whose calculation load is reduced by relieving penalty parameters estimation and complex matrix inversion. Based on nonsmooth analysis and the extension theorem of the solution of differential inclusion, it is obtained that the global existence and boundedness of state solution of neurodynamic system, as well as the asymptotic convergence of state solution to the feasible region and the set of LU-critical points of interval-valued nonconvex optimization problem. Several numerical experiments and the applications to emergency supplies distribution and nondeterministic fractional continuous static games are solved to illustrate the applicability of the proposed neurodynamic algorithm.

A redistributed proximal bundle method for nonsmooth nonconvex functions with inexact information

In this paper, we propose a redistributed proximal bundle method for a class of nonconvex nonsmooth optimization problems with inexact information, i.e., we consider the problem of computing the approximate critical points when only the inexact information about the function values and subgradients are available and show that reasonable convergence properties are obtained. We assume that the errors in the computation of functions and subgradients are only bounded and in principle do not have to vanish within the limits. For the nonconvex functions, we design the convexification technique, which ensures that the linearization error of its augmentation function is nonnegative. Meanwhile, for the inexact information, we utilize noise management strategies and update approximate parameters to reduce the impact of inexact information. Based on this method, we can obtain the approximate solution.

An abstract convergence framework with application to inertial inexact forward–backward methods

In this paper we introduce a novel abstract descent scheme suited for the minimization of proper and lower semicontinuous functions. The proposed abstract scheme generalizes a set of properties that are crucial for the convergence of several first-order methods designed for nonsmooth nonconvex optimization problems. Such properties guarantee the convergence of the full sequence of iterates to a stationary point, if the objective function satisfies the Kurdyka-Lojasiewicz property. The abstract framework allows for the design of new algorithms. We propose two inertial-type algorithms with implementable inexactness criteria for the main iteration update step. The first algorithm, i$^2$Piano, exploits large steps by adjusting a local Lipschitz constant. The second algorithm, iPila, overcomes the main drawback of line-search based methods by enforcing a descent only on a merit function instead of the objective function. Both algorithms have the potential to escape local minimizers (or stationary points) by leveraging the inertial feature. Moreover, they are proved to enjoy the full convergence guarantees of the abstract descent scheme, which is the best we can expect in such a general nonsmooth nonconvex optimization setup using first-order methods. The efficiency of the proposed algorithms is demonstrated on two exemplary image deblurring problems, where we can appreciate the benefits of performing a linesearch along the descent direction inside an inertial scheme.

Bregman iterative regularization using model functions for nonconvex nonsmooth optimization

In this paper, we propose a new algorithm called ModelBI by blending the Bregman iterative regularization method and the model function technique for solving a class of nonconvex nonsmooth optimization problems. On one hand, we use the model function technique, which is essentially a first-order approximation to the objective function, to go beyond the traditional Lipschitz gradient continuity. On the other hand, we use the Bregman iterative regularization to generate solutions fitting certain structures. Theoretically, we show the global convergence of the proposed algorithm with the help of the Kurdyka-Łojasiewicz property. Finally, we consider two kinds of nonsmooth phase retrieval problems and propose an explicit iteration scheme. Numerical results verify the global convergence and illustrate the potential of our proposed algorithm.

Autonomous Tracking and State Estimation With Generalized Group Lasso

We address the problem of autonomous tracking and state estimation for marine vessels, autonomous vehicles, and other dynamic signals under a (structured) sparsity assumption. The aim is to improve the tracking and estimation accuracy with respect to the classical Bayesian filters and smoothers. We formulate the estimation problem as a dynamic generalized group Lasso problem and develop a class of smoothing-and-splitting methods to solve it. The Levenberg-Marquardt iterated extended Kalman smoother-based multiblock alternating direction method of multipliers (LM-IEKS-mADMMs) algorithms are based on the alternating direction method of multipliers (ADMMs) framework. This leads to minimization subproblems with an inherent structure to which three new augmented recursive smoothers are applied. Our methods can deal with large-scale problems without preprocessing for dimensionality reduction. Moreover, the methods allow one to solve nonsmooth nonconvex optimization problems. We then prove that under mild conditions, the proposed methods converge to a stationary point of the optimization problem. By simulated and real-data experiments, including multisensor range measurement problems, marine vessel tracking, autonomous vehicle tracking, and audio signal restoration, we show the practical effectiveness of the proposed methods.

A class of infeasible proximal bundle methods for nonsmooth nonconvex multi-objective optimization problems

A class of infeasible proximal bundle methods for nonsmooth nonconvex multi-objective optimization problems

Perturbed Iterate SGD for Lipschitz Continuous Loss Functions

This paper presents an extension of stochastic gradient descent for the minimization of Lipschitz continuous loss functions. Our motivation is for use in non-smooth non-convex stochastic optimization problems, which are frequently encountered in applications such as machine learning. Using the Clarke \(\epsilon \)-subdifferential, we prove the non-asymptotic convergence to an approximate stationary point in expectation for the proposed method. From this result, a method with non-asymptotic convergence with high probability, as well as a method with asymptotic convergence to a Clarke stationary point almost surely are developed. Our results hold under the assumption that the stochastic loss function is a Carathéodory function which is almost everywhere Lipschitz continuous in the decision variables. To the best of our knowledge, this is the first non-asymptotic convergence analysis under these minimal assumptions.

Unified Robust Necessary Optimality Conditions for Nonconvex Nonsmooth Uncertain Multiobjective Optimization.

This paper is concerned with nonconvex nonsmooth uncertain multiobjective optimization problems, in which the decision variable of both objective and constraint functions is defined on Banach space while uncertain parameters are defined on arbitrary nonempty (may not be compact) sets. We employ the Stone–C̆ech compactification of uncertainty sets and the upper semicontinuous regularization of original functions with respect to uncertain parameters, giving rise to unified robust necessary optimality conditions for the local robust weakly efficient solution of the considered problem. Moreover, we derive weak and strong KKT robust necessary conditions via the constraint qualification and the regularity condition, respectively. Several examples are provided to illustrate the validity of our results.

Newton acceleration on manifolds identified by proximal gradient methods

Proximal methods are known to identify the underlying substructure of nonsmooth optimization problems. Even more, in many interesting situations, the output of a proximity operator comes with its structure at no additional cost, and convergence is improved once it matches the structure of a minimizer. However, it is impossible in general to know whether the current structure is final or not; such highly valuable information has to be exploited adaptively. To do so, we place ourselves in the case where a proximal gradient method can identify manifolds of differentiability of the nonsmooth objective. Leveraging this manifold identification, we show that Riemannian Newton-like methods can be intertwined with the proximal gradient steps to drastically boost the convergence. We prove the superlinear convergence of the algorithm when solving some nondegenerated nonsmooth nonconvex optimization problems. We provide numerical illustrations on optimization problems regularized by \(\ell _1\)-norm or trace-norm.

Proximal stochastic recursive momentum algorithm for nonsmooth nonconvex optimization problems

In this paper, we mainly consider a class of nonconvex nonsmooth optimization problems, whose objective function is the sum of a smooth function with a Lipschitz continuous gradient and a convex nonsmooth function. We first propose a proximal stochastic recursive momentum algorithm(ProxSTORM) with mini-batch for solving the optimization problems and consider its convergence behaviour. Then, based on the Polyak–Łojasiewicz inequality, we establish the global linear convergence rate of ProxSTORM. Finally, some numerical experiments have been conducted to illustrate the efficiency of our method.