First-order Stationary Points Research Articles

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models.

In deep learning, different kinds of deep networks typically need different optimizers, which have to be chosen after multiple trials, making the training process inefficient. To relieve this issue and consistently improve the model training speed across deep networks, we propose the ADAptive Nesterov momentum algorithm, Adan for short. Adan first reformulates the vanilla Nesterov acceleration to develop a new Nesterov momentum estimation (NME) method, which avoids the extra overhead of computing gradient at the extrapolation point. Then Adan adopts NME to estimate the gradient's first- and second-order moments in adaptive gradient algorithms for convergence acceleration. Besides, we prove that Adan finds an ϵ-approximate first-order stationary point within O(ϵ-3.5) stochastic gradient complexity on the non-convex stochastic problems (e.g.deep learning problems), matching the best-known lower bound. Extensive experimental results show that Adan consistently surpasses the corresponding SoTA optimizers on vision, language, and RL tasks and sets new SoTAs for many popular networks and frameworks, egResNet, ConvNext, ViT, Swin, MAE, DETR, GPT-2, Transformer-XL, and BERT. More surprisingly, Adan can use half of the training cost (epochs) of SoTA optimizers to achieve higher or comparable performance on ViT, GPT-2, MAE, etc, and also shows great tolerance to a large range of minibatch size, e.g.from 1k to 32k. Code is released at https://github.com/sail-sg/Adan, and has been used in multiple popular deep learning frameworks or projects.

Estimation of sparse covariance matrix via non-convex regularization

Estimation of high-dimensional sparse covariance matrix is one of the fundamental and important problems in multivariate analysis and has a wide range of applications in many fields. This paper presents a novel method for sparse covariance matrix estimation via solving a non-convex regularization optimization problem. We establish the asymptotic properties of the proposed estimator and develop a multi-stage convex relaxation method to find an effective estimator. The multi-stage convex relaxation method guarantees any accumulation point of the sequence generated is a first-order stationary point of the non-convex optimization. Moreover, the error bounds of the first two stage estimators of the multi-stage convex relaxation method are derived under some regularity conditions. The numerical results show that our estimator outperforms the state-of-the-art estimators and has a high degree of sparsity on the premise of its effectiveness.

A constraint dissolving approach for nonsmooth optimization over the Stiefel manifold

Abstract This paper focuses on the minimization of a possibly nonsmooth objective function over the Stiefel manifold. The existing approaches either lack efficiency or can only tackle prox-friendly objective functions. We propose a constraint dissolving function named NCDF and show that it has the same first-order stationary points and local minimizers as the original problem in a neighborhood of the Stiefel manifold. Furthermore, we show that the Clarke subdifferential of NCDF is easy to achieve from the Clarke subdifferential of the objective function. Therefore, various existing approaches for unconstrained nonsmooth optimization can be directly applied to nonsmooth optimization problems over the Stiefel manifold. We propose a framework for developing subgradient-based methods and establishing their convergence properties based on prior works. Furthermore, based on our proposed framework, we can develop efficient approaches for optimization over the Stiefel manifold. Preliminary numerical experiments further highlight that the proposed constraint dissolving approach yields efficient and direct implementations of various unconstrained approaches to nonsmooth optimization problems over the Stiefel manifold.

PersA-FL: personalized asynchronous federated learning

We study the personalized federated learning problem under asynchronous updates. In this problem, each client seeks to obtain a personalized model that simultaneously outperforms local and global models. We consider two optimization-based frameworks for personalization: (i) Model-Agnostic Meta-Learning (MAML) and (ii) Moreau Envelope (ME). MAML involves learning a joint model adapted for each client through fine-tuning, whereas ME requires a bi-level optimization problem with implicit gradients to enforce personalization via regularized losses. We focus on improving the scalability of personalized federated learning by removing the synchronous communication assumption. Moreover, we extend the studied function class by removing boundedness assumptions on the gradient norm. Our main technical contribution is a unified proof for asynchronous federated learning with bounded staleness that we apply to MAML and ME personalization frameworks. For the smooth and non-convex functions class, we show the convergence of our method to a first-order stationary point. We illustrate the performance of our method and its tolerance to staleness through experiments for classification tasks over heterogeneous datasets.

Distributed Stochastic Gradient Tracking Algorithm With Variance Reduction for Non-Convex Optimization.

This article proposes a distributed stochastic algorithm with variance reduction for general smooth non-convex finite-sum optimization, which has wide applications in signal processing and machine learning communities. In distributed setting, a large number of samples are allocated to multiple agents in the network. Each agent computes local stochastic gradient and communicates with its neighbors to seek for the global optimum. In this article, we develop a modified variance reduction technique to deal with the variance introduced by stochastic gradients. Combining gradient tracking and variance reduction techniques, this article proposes a distributed stochastic algorithm, gradient tracking algorithm with variance reduction (GT-VR), to solve large-scale non-convex finite-sum optimization over multiagent networks. A complete and rigorous proof shows that the GT-VR algorithm converges to the first-order stationary points with O(1/k) convergence rate. In addition, we provide the complexity analysis of the proposed algorithm. Compared with some existing first-order methods, the proposed algorithm has a lower O(PMε⁻¹) gradient complexity under some mild condition. By comparing state-of-the-art algorithms and GT-VR in numerical simulations, we verify the efficiency of the proposed algorithm.

Communication Compression for Distributed Nonconvex Optimization

This paper considers distributed nonconvex optimization with the cost functions being distributed over agents. Noting that information compression is a key tool to reduce the heavy communication load for distributed algorithms as agents iteratively communicate with neighbors, we propose three distributed primal–dual algorithms with compressed communication. The first two algorithms are applicable to a general class of compressors with bounded relative compression error and the third algorithm is suitable for two general classes of compressors with bounded absolute compression error. We show that the proposed distributed algorithms with compressed communication have comparable convergence properties as state-of-the-art algorithms with exact communication. Specifically, we show that they can find first-order stationary points with sublinear convergence rate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {O}(1/T)$</tex-math></inline-formula> when each local cost function is smooth, where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$T$</tex-math></inline-formula> is the total number of iterations, and find global optima with linear convergence rate under an additional condition that the global cost function satisfies the Polyak–Łojasiewicz condition. Numerical simulations are provided to illustrate the effectiveness of the theoretical results.

On Geometric Connections of Embedded and Quotient Geometries in Riemannian Fixed-Rank Matrix Optimization

In this paper, we propose a general procedure for establishing the geometric landscape connections of a Riemannian optimization problem under the embedded and quotient geometries. By applying the general procedure to the fixed-rank positive semidefinite (PSD) and general matrix optimization, we establish an exact Riemannian gradient connection under two geometries at every point on the manifold and sandwich inequalities between the spectra of Riemannian Hessians at Riemannian first-order stationary points (FOSPs). These results immediately imply an equivalence on the sets of Riemannian FOSPs, Riemannian second-order stationary points (SOSPs), and strict saddles of fixed-rank matrix optimization under the embedded and the quotient geometries. To the best of our knowledge, this is the first geometric landscape connection between the embedded and the quotient geometries for fixed-rank matrix optimization, and it provides a concrete example of how these two geometries are connected in Riemannian optimization. In addition, the effects of the Riemannian metric and quotient structure on the landscape connection are discussed. We also observe an algorithmic connection between two geometries with some specific Riemannian metrics in fixed-rank matrix optimization: there is an equivalence between gradient flows under two geometries with shared spectra of Riemannian Hessians. A number of novel ideas and technical ingredients—including a unified treatment for different Riemannian metrics, novel metrics for the Stiefel manifold, and new horizontal space representations under quotient geometries—are developed to obtain our results. The results in this paper deepen our understanding of geometric and algorithmic connections of Riemannian optimization under different Riemannian geometries and provide a few new theoretical insights to unanswered questions in the literature. Funding: X. Li was partially supported by National Key R&amp;D Program of China [Grants 2020YFA0711900, 2020YFA0711901], the National Natural Science Foundation of China [Grants 12271107, 62141407], the Young Elite Scientists Sponsorship Program by CAST [Grant 2019QNRC001], the “Chenguang Program” by the Shanghai Education Development Foundation and Shanghai Municipal Education Commission [Grant 19CG02], the Shanghai Science and Technology Program [21JC1400600]. Y. Luo and A. R. Zhang were partially supported by the National Science Foundation [CAREER-2203741].

Derivative-free separable quadratic modeling and cubic regularization for unconstrained optimization

We present a derivative-free separable quadratic modeling and cubic regularization technique for solving smooth unconstrained minimization problems. The derivative-free approach is mainly concerned with building a quadratic model that could be generated by numerical interpolation or using a minimum Frobenius norm approach, when the number of points available does not allow to build a complete quadratic model. This model plays a key role to generate an approximated gradient vector and Hessian matrix of the objective function at every iteration. We add a specialized cubic regularization strategy to minimize the quadratic model at each iteration, that makes use of separability. We discuss convergence results, including worst case complexity, of the proposed schemes to first-order stationary points. Some preliminary numerical results are presented to illustrate the robustness of the specialized separable cubic algorithm.

High-Dimensional Covariance Estimation via Constrained Lq-Type Regularization

High-dimensional covariance matrix estimation is one of the fundamental and important problems in multivariate analysis and has a wide range of applications in many fields. In practice, it is common that a covariance matrix is composed of a low-rank matrix and a sparse matrix. In this paper we estimate the covariance matrix by solving a constrained Lq-type regularized optimization problem. We establish the first-order optimality conditions for this problem by using proximal mapping and the subspace method. The proposed stationary point degenerates to the first-order stationary points of the unconstrained Lq regularized sparse or low-rank optimization problems. A smoothing alternating updating method is proposed to find an estimator for the covariance matrix. We establish the convergence of the proposed calculation method. The numerical simulation results show the effectiveness of the proposed approach for high-dimensional covariance estimation.

Zeroth-Order Non-Convex Optimization for Cooperative Multi-Agent Systems with Diminishing Step Size and Smoothing Radius

We study a class of zeroth-order distributed optimization problems, where each agent can control a partial vector and observe a local cost that depends on the joint vector of all agents, and the agents can communicate with each other with time delay. We propose and study a gradient descent-based algorithm using two-point gradient estimators with diminishing smoothing parameters and diminishing step-size and we establish the convergence rate to a first-order stationary point for general nonconvex problems. A byproduct of our proposed method with diminishing step size and smoothing parameters, as opposed to the fixed-parameter scheme, is that our proposed algorithm does not require any information regarding the local cost functions. This makes the solution appealing in practice as it allows for optimizing an unknown (black-box) global function without prior knowledge of its smoothness parameters. At the same time, the performance will adaptively match the problem instance parameters.

Faster Stochastic Quasi-Newton Methods.

Stochastic optimization methods have become a class of popular optimization tools in machine learning. Especially, stochastic gradient descent (SGD) has been widely used for machine learning problems, such as training neural networks, due to low per-iteration computational complexity. In fact, the Newton or quasi-newton (QN) methods leveraging the second-order information are able to achieve a better solution than the first-order methods. Thus, stochastic QN (SQN) methods have been developed to achieve a better solution efficiently than the stochastic first-order methods by utilizing approximate second-order information. However, the existing SQN methods still do not reach the best known stochastic first-order oracle (SFO) complexity. To fill this gap, we propose a novel faster stochastic QN method (SpiderSQN) based on the variance reduced technique of SIPDER. We prove that our SpiderSQN method reaches the best known SFO complexity of O(n+n1/2ϵ-2) in the finite-sum setting to obtain an ϵ -first-order stationary point. To further improve its practical performance, we incorporate SpiderSQN with different momentum schemes. Moreover, the proposed algorithms are generalized to the online setting, and the corresponding SFO complexity of O(ϵ-3) is developed, which also matches the existing best result. Extensive experiments on benchmark data sets demonstrate that our new algorithms outperform state-of-the-art approaches for nonconvex optimization.

Stochastic cubic-regularized policy gradient method

Policy-based reinforcement learning methods have achieved great achievements in real-world decision-making problems. However, the theoretical understanding of policy-based methods is still limited. Specifically, existing works mainly focus on first-order stationary point policies (FOSPs); in some very special reinforcement learning settings (e.g., tabular case and function approximation with restricted parametric policy classes) some works consider globally optimal policy. It is well-known that FOSPs could be undesirable local optima or saddle points, and obtaining a global optimum is generally NP-hard. In this paper, we propose a policy gradient method that provably converges to second-order stationary point policies (SOSPs) for any differentiable policy classes. The proposed method is computationally efficient, and it judiciously uses cubic-regularized subroutines to escape saddle points while at the same time minimizing the Hessian-based computations. We prove that the method enjoys the sample complexity of O˜(ϵ−3.5), which improves upon the current optimal complexity O˜ϵ−4.5. Finally, experimental results are provided to demonstrate the effectiveness of the method.

DESTRESS: Computation-Optimal and Communication-Efficient Decentralized Nonconvex Finite-Sum Optimization

Emerging applications in multi-agent environments such as internet-of-things, networked sensing, autonomous systems and federated learning, call for decentralized algorithms for finite-sum optimizations that are resource-efficient in terms of both computation and communication. In this paper, we consider the prototypical setting where the agents work collaboratively to minimize the sum of local loss functions by only communicating with their neighbors over a predetermined network topology. We develop a new algorithm, called DEcentralized STochastic REcurSive gradient methodS (DESTRESS) for nonconvex finite-sum optimization, which matches the optimal incremental first-order oracle (IFO) complexity of centralized algorithms for finding first-order stationary points, while maintaining communication efficiency. Detailed theoretical and numerical comparisons corroborate that the resource efficiencies of DESTRESS improve upon prior decentralized algorithms over a wide range of parameter regimes. DESTRESS leverages several key algorithm design ideas including randomly activated stochastic recursive gradient updates with mini-batches for local computation, gradient tracking with extra mixing (i.e., multiple gossiping rounds) for per-iteration communication, together with careful choices of hyper-parameters and new analysis frameworks to provably achieve a desirable computation-communication trade-off.

Convergence rate analysis of proximal iteratively reweighted $$\ell _1$$ methods for $$\ell _p$$ regularization problems

In this paper, we focus on the local convergence rate analysis of the proximal iteratively reweighted \(\ell _1\) algorithms for solving \(\ell _p\) regularization problems, which are widely applied for inducing sparse solutions. We show that if the Kurdyka–Łojasiewicz property is satisfied, the algorithm converges to a unique first-order stationary point; furthermore, the algorithm has local linear convergence or local sublinear convergence. The theoretical results we derived are much stronger than the existing results for iteratively reweighted \(\ell _1\) algorithms.

Policy Optimization with Stochastic Mirror Descent

Improving sample efficiency has been a longstanding goal in reinforcement learning. This paper proposes VRMPO algorithm: a sample efficient policy gradient method with stochastic mirror descent. In VRMPO, a novel variance-reduced policy gradient estimator is presented to improve sample efficiency. We prove that the proposed VRMPO needs only O(ε−3) sample trajectories to achieve an ε-approximate first-order stationary point, which matches the best sample complexity for policy optimization. Extensive empirical results demonstrate that VRMP outperforms the state-of-the-art policy gradient methods in various settings.

A Proximal Quasi-Newton Trust-Region Method for Nonsmooth Regularized Optimization

We develop a trust-region method for minimizing the sum of a smooth term (f) and a nonsmooth term (h), both of which can be nonconvex. Each iteration of our method minimizes a possibly nonconvex model of (f + h) in a trust region. The model coincides with (f + h) in value and subdifferential at the center. We establish global convergence to a first-order stationary point when (f) satisfies a smoothness condition that holds, in particular, when it has a Lipschitz-continuous gradient, and (h) is proper and lower semicontinuous. The model of (h) is required to be proper, lower semi-continuous and prox-bounded. Under these weak assumptions, we establish a worst-case (O(1/\epsilon^2)) iteration complexity bound that matches the best known complexity bound of standard trust-region methods for smooth optimization. We detail a special instance, named TR-PG, in which we use a limited-memory quasi-Newton model of (f) and compute a step with the proximal gradient method, resulting in a practical proximal quasi-Newton method. We establish similar convergence properties and complexity bound for a quadratic regularization variant, named R2, and provide an interpretation as a proximal gradient method with adaptive step size for nonconvex problems. R2 may also be used to compute steps inside the trust-region method, resulting in an implementation named TR-R2. We describe our Julia implementations and report numerical results on inverse problems from sparse optimization and signal processing. Both TR-PG and TR-R2 exhibit promising performance and compare favorably with two linesearch proximal quasi-Newton methods based on convex models.

ARES: a fast and accurate tool for the identification of plasma stationary points and separatrix

In this work, triangular C 1 interpolation based on Argyris element (AE) is used to develop the ARgyris element based searching (ARES) algorithm, an efficient tool for the accurate identification of stationary points (SPs), e.g. X-point, magnetic axis, snowflake point, etc and separatrix in magnetic confinement fusion application. AE-based interpolation allows the formulation of the nonlinear problem of SP identification on the triangle as a vectorial root-finding problem, addressed via different Newton-like algorithms. The proposed method is able to detect both first-order SP (e.g. X-points and magnetic axis, where the norm of the gradient vanishes) and second-order SP (e.g. snowflake points, where both the norm of the gradient and of the second derivatives vanishes). The separatrix is detected inside the triangles via the novel zero line searching algorithm, and is described by means of quintic spline interpolant of its components. Thanks to its modularity and efficiency, ARES is suitable to run on structured and unstructured triangular meshes, and can be readily integrated with any kind of Grad–Shafranov solver based on 2D triangular finite element method, or used for real-time SP and separatrix identification for control oriented purposes.

Moreau Envelope Augmented Lagrangian Method for Nonconvex Optimization with Linear Constraints

The augmented Lagrangian method (ALM) is one of the most useful methods for constrained optimization. Its convergence has been well established under convexity assumptions or smoothness assumptions, or under both assumptions. ALM may experience oscillations and divergence when the underlying problem is simultaneously nonconvex and nonsmooth. In this paper, we consider the linearly constrained problem with a nonconvex (in particular, weakly convex) and nonsmooth objective. We modify ALM to use a Moreau envelope of the augmented Lagrangian and establish its convergence under conditions that are weaker than those in the literature. We call it the Moreau envelope augmented Lagrangian (MEAL) method. We also show that the iteration complexity of MEAL is $$o(\varepsilon ^{-2})$$ to yield an $$\varepsilon $$ -accurate first-order stationary point. We establish its whole sequence convergence (regardless of the initial guess) and a rate when a Kurdyka–Łojasiewicz property is assumed. Moreover, when the subproblem of MEAL has no closed-form solution and is difficult to solve, we propose two practical variants of MEAL, an inexact version called iMEAL with an approximate proximal update, and a linearized version called LiMEAL for the constrained problem with a composite objective. Their convergence is also established.

Low-rank traffic matrix completion with marginal information

Accurate spatio-temporal traffic data is crucial to intelligent transportation systems. Missing traffic data is an important problem to solve. Low-rank matrix completion provides an effective way to find the missing data. The completion aims to obtain a low-rank matrix that can approximate the known entries as far as possible. Meanwhile, some linear constraint marginal information of the matrix can also be observed in the real application. In this paper, we utilize such marginal information to largely improve the performance of common matrix completion algorithms and propose an alternating direction method of multipliers (ADMM) and conjugate gradient descent method (CGD) based SoftImpute alternative least square (ALS) algorithm. We analyze their convergence rates and prove that the model can always converge to a first-order stationary point. We also utilize ADMM and CGD to largely accelerate the subproblem and make its complexity of each iteration at the same level as the popular SoftImpute-ALS matrix completion algorithm. Furthermore, this algorithm can be used in distributed computation, suitable for large-scale problems. In the numerical experiments, we demonstrate its outstanding matrix completion performance and high speed in several traffic matrix datasets.

Block Bregman Majorization Minimization with Extrapolation

In this paper, we consider a class of nonsmooth nonconvex optimization problems whose objective is the sum of a block relative smooth function and a proper and lower semicontinuous block separable function. Although the analysis of block proximal gradient (BPG) methods for the class of block $L$-smooth functions have been successfully extended to Bregman BPG methods that deal with the class of block relative smooth functions, accelerated Bregman BPG methods are scarce and challenging to design. Taking our inspiration from Nesterov-type acceleration and the majorization-minimization scheme, we propose a block alternating Bregman Majorization-Minimization framework with Extrapolation (BMME). We prove subsequential convergence of BMME to a first-order stationary point under mild assumptions, and study its global convergence under stronger conditions. We illustrate the effectiveness of BMME on the penalized orthogonal nonnegative matrix factorization problem.