Nesterov's Method Research Articles

Deep bidirectional hierarchical matrix factorization model for hyperspectral unmixing

Existing deep nonnegative matrix factorization-based approaches treat shallow and deep layers equally or with similar strategies, neglecting the heterogeneous physical structures between the shallow and progressively deeper layers, thus failing to explore the latent different sub-manifold structures in the hyperspectral image. In this paper, we propose a deep nonnegative matrix factorization model with bidirectional constraints to achieve hyperspectral unmixing. The sub-manifold structures in hyperspectral image are fully exploited by filtering and penalizing the shallow abundance layer with a denoised regularizer and a manifold regularizer. In contrast to the shallow abundance layer, the remaining layers are constrained by an extremely common regularizer to avoid over-denoising and maintain fidelity. In this way, the fine cues between different substances are exploited to a large extent. Additionally, the overall reconstruction error can be well controlled because the performance of the designed feedback mechanism can be fine-tuned by the inverse hierarchical constraints. Finally, we employ Nesterov's optimal gradient method to solve the optimization problem effectively. Experiment results are conducted on both synthetic datasets and real datasets, and all results show that the proposed method is superior to recent canonical unmixing methods.

Numerical Solution of the Cauchy Problem for the Helmholtz Equation Using Nesterov’s Accelerated Method

In this paper, the Cauchy problem for the Helmholtz equation, also known as the continuation problem, is considered. The continuation problem is reduced to a boundary inverse problem for a well-posed direct problem. A generalized solution to the direct problem is obtained and an estimate of its stability is given. The inverse problem is reduced to an optimization problem solved using the gradient method. The convergence of the Landweber method with respect to the functionals is compared with the convergence of the Nesterov method. The calculation of the gradient in discrete form, which is often used in the numerical solutions of the inverse problem, is described. The formulation of the conjugate problem in discrete form is presented. After calculating the gradient, an algorithm for solving the inverse problem using the Nesterov method is constructed. A computational experiment for the boundary inverse problem is carried out, and the results of the comparative analysis of the Landweber and Nesterov methods in a graphical form are presented.

A reconstructed feasible solution-based safe feature elimination rule for expediting multi-task lasso

Multi-task lasso (MTL) is an effective algorithm to handle multi-task problems. By introducing ℓ2,1-norm, it can realize joint feature selection across a group of related tasks. But it is time-consuming when handling high-dimensional problems. Motivated by the row sparsity of the optimal solution, an improved safe feature elimination rule termed IEDPP is proposed to accelerate the training process. It could delete most of the redundant features before we solve the problem. Then the computational efficiency will be improved since only a reduced problem should be solved. Moreover, the properties of the projection operator and the reconstructed feasible solutions ensure the safety of the proposed method. That is to say, it will derive an identical optimal solution to the original problem both in theory and in practice. But our IEDPP could only be used once before solving, there are still some redundant features that are not deleted. Therefore, we further propose an integrated IIEG-ML rule by combining IEDPP with GAP. Then, more and more redundant features could be deleted as the algorithm converges. Moreover, by embedding IIEG-ML into the grid search method, the whole training process will be accelerated. Finally, an improved Nesterov's method is used to solve the reduced problems. Experimental results on different datasets confirm the effectiveness of our method.

Nesterov's Method for Convex Optimization

While Nesterov's algorithm for computing the minimum of a convex function is now over forty years old, it is rarely presented in texts for a first course in optimization. This is unfortunate since for many problems this algorithm is superior to the ubiquitous steepest descent algorithm, and it is equally simple to implement. This article presents an elementary analysis of Nesterov's algorithm that parallels that of steepest descent. It is envisioned that this presentation of Nesterov's algorithm could easily be covered in a few lectures following the introductory material on convex functions and steepest descent included in every course on optimization.

Education

The Education section in this issue presents two contributions. In `"Nesterov's Method for Convex Optimization," Noel J. Walkington proposes a teaching guide for a first course in optimization of this well-known algorithm for computing the minimum of a convex function. This algorithm, first proposed in 1983 by Yuri Nesterov, though well recognized in computational optimization in the presence of large data as a more efficient tool than the steepest descent method, is still absent in most modern textbooks on optimization. The author of the present article develops an elementary analysis of Nesterov's first order algorithm that parallels that of steepest descent but with an additional requirement proposed by Nesterov. Two cases are discussed. The first concerns an unconstrained minimization problem, while the second includes closed convex constraints represented using infinite penalization of the cost. More generally, the cost function becomes the sum of a smooth convex function and a lower semicontinuous convex function. Several student-level exercises are included in this paper. Results are nicely illustrated by an example of a signal recovery problem and a discussion of the Uzawa algorithm for optimization problems with constraints defined by inequalities involving convex functions. The second paper, "A Comprehensive Proof of Bertrand's Theorem," is presented by Patrick De Leenheer, John Musgrove, and Tyler Schimleck. It concerns the behavior of the solutions of the classical two-body problem and states that, among all possible gravitational laws, there are only two exhibiting the property that all bounded orbits are closed: Newtonian gravitation and Hookean gravitation. Historically, even if Newton was aware that there are to specific gravitational laws having the above property, it was only two centuries later, in 1873, that Bertrand realized that these are the only ones. Bertrand's theorem, due to its important consequences, has been integrated into the undergraduate curriculum in theoretical mechanics, but its proof, accessible to undergraduate mathematics or physics students, seems to be absent from modern textbooks. Although Bertrand's original paper did not contain a precise proof, V. Arnold proposed a sketch of it based on six subproblems. Among other contributions, this article provides a complete proof of the sixth subproblem under a specific assumption imposed on the magnitude of the force in the motion model. Under this assumption, a complete proof of Bertrand's theorem is then given, incorporating also earlier contributions by other authors. Still, comprehensive does not mean simple here, and this paper may be used to conceive several research projects for advanced-level undergraduate students in mathematics or physics.

Multi-angle lensless ptychographic imaging via adaptive correction and the Nesterov method.

Lensless systems based on ptychographic imaging can simultaneously achieve a large field of view and high resolution while having the advantages of small size, portability, and low cost compared to traditional lensed imaging. However, lensless imaging systems are susceptible to environmental noise and have a lower resolution of individual images than lens-based imaging systems, which means that they require a longer time to obtain a good result. Therefore, in this paper, to improve the convergence rate and robustness of noise in lensless ptychographic imaging, we propose an adaptive correction method, in which we add an adaptive error term and noise correction term in lensless ptychographic algorithms to reach convergence faster and create a better suppression effect on both Gaussian noise and Poisson noise. The Wirtinger flow and the Nesterov algorithms are used in our method to reduce computational complexity and improve the convergence rate. We applied the method to phase reconstruction for lensless imaging and demonstrated the effectiveness of the method by simulation and experiment. The method can be easily applied to other ptychographic iterative algorithms.

Solving strongly convex-concave composite saddle-point problems with low dimension of one group of variable

Algorithmic methods are developed that guarantee efficient complexity estimates for strongly convex-concave saddle-point problems in the case when one group of variables has a high dimension, while another has a rather low dimension (up to 100). These methods are based on reducing problems of this type to the minimization (maximization) problem for a convex (concave) functional with respect to one of the variables such that an approximate value of the gradient at an arbitrary point can be obtained with the required accuracy using an auxiliary optimization subproblem with respect to the other variable. It is proposed to use the ellipsoid method and Vaidya's method for low-dimensional problems and accelerated gradient methods with inexact information about the gradient or subgradient for high-dimensional problems. In the case when one group of variables, ranging over a hypercube, has a very low dimension (up to five), another proposed approach to strongly convex-concave saddle-point problems is rather efficient. This approach is based on a new version of a multidimensional analogue of Nesterov's method on a square (the multidimensional dichotomy method) with the possibility to use inexact values of the gradient of the objective functional. Bibliography: 28 titles.

Accelerated Distributed Projected Gradient Descent for Convex Optimization with Clique-wise Coupled Constraints

This paper addresses a distributed convex optimization problem with a class of coupled constraints, which arises in a multi-agent system composed of multiple communities modeled by cliques. First, we propose a fully distributed gradient-based algorithm with a novel operator inspired by the convex projection, called the clique-based projection. Next, we scrutinize the convergence properties for both diminishing and fixed step sizes. For diminishing ones, we show the convergence to an optimal solution under the assumptions of the smoothness of an objective function and the compactness of the constraint set. Additionally, when the objective function is strongly monotone, the exact convergence to the unique solution is proved without the assumption of compactness. For fixed step sizes, we prove the convergence rate of O(1/k) concerning an approximated objective residual under the assumption of the smoothness of the objective function. Furthermore, we apply Nesterov's acceleration method to the proposed algorithm and establish the convergence rate of O(1/k2). Numerical experiments illustrate the effectiveness of the proposed method.

From the Ravine Method to the Nesterov Method and Vice Versa: A Dynamical System Perspective

We revisit the Ravine method of Gelfand and Tsetlin from a dynamical system perspective, study its convergence properties, and highlight its similarities and differences with the Nesterov accelerated gradient method. The two methods are closely related. They can be deduced from each other by reversing the order of the extrapolation and gradient operations in their definitions. They benefit from similar fast convergence of values and convergence of iterates for general convex objective functions. We will also establish the high resolution ODE of the Ravine and Nesterov methods and reveal an additional geometric damping term driven by the Hessian for both methods. This will allow us to prove fast convergence toward zero of the gradients not only for the Ravine method but also for the Nesterov method for the first time. In the strongly convex case, we show linear convergence for the Ravine method at an optimal rate. We also highlight connections to other algorithms resulting from more subtle discretization schemes and finally describe a Ravine version of the proximal-gradient algorithms for general structured smooth + nonsmooth convex optimization problems.

Climate-Induced Fire Hazard in Forests in the Volga Federal District of European Russia during 1992–2020

This paper shows the relevance of the problem of fire hazard in the forests of the Volga Federal District (VFD) of European Russia. The Nesterov index and the Selyaninov hydrothermal coefficient (HTC) are considered as indicators of fire hazard. The changes in climatic conditions in the VFD during 1955–2018 are shown; a trend towards warming and an increase in aridity in the study region were revealed. The repeatability of various fire hazard classes from May to September was calculated using the Nesterov method. It is shown that in July, the most dangerous situation was in the south of the VFD, where the repeatability of class IV fire hazard reached 27%. Using the HTC index, the degree of aridity of the district in the summer period was estimated. The frequency of the most arid conditions (HTC < 0.5) increases from the north to the south of the district, from 6% (Kirov Region) to 47% (Orenburg Region). Using the TT index, the potential thunderstorm danger in the VFD was assessed. With the help of the constructed maps, the hotspots of the most probable occurrence of thunderstorms were detected. The use of Rosstat data on the number of forest fires from 1992 to 2020 made it possible to consider the spatiotemporal distribution of forest fires in 14 administrative regions of the VFD. The distribution of the number of fires by the regions is shown depending on their forest cover and season. The peak of the number of fires was revealed in 2010, when the entire territory of the study region was covered by a severe drought, as a result of which the area of forests covered by fire increased many times over. In recent years (since 2017), there has been an increase in the area of burned forest due to the active phase of climate warming.

Differentially Private Accelerated Optimization Algorithms

We present two classes of differentially private optimization algorithms derived from the well-known accelerated first-order methods. The first algorithm is inspired by Polyak's heavy ball method and employs a smoothing approach to decrease the accumulated noise on the gradient steps required for differential privacy. The second class of algorithms are based on Nesterov's accelerated gradient method and its recent multi-stage variant. We propose a noise dividing mechanism for the iterations of Nesterov's method in order to improve the error behavior of the algorithm. The convergence rate analyses are provided for both the heavy ball and the Nesterov's accelerated gradient method with the help of the dynamical system analysis techniques. Finally, we conclude with our numerical experiments showing that the presented algorithms have advantages over the well-known differentially private algorithms.

修正加速勾配法を用いた摩擦係数の最小化を目的としたテクスチャの形状最適化解析

In this study, we proposed a new acceleration gradient method that introduces the concept of Taylor expansion into Nesterov's acceleration gradient method. The proposed method is called the modified acceleration gradient method. A texture shape optimization analysis is performed using the proposed method to minimize the friction coefficient. The gradient of the Lagrange function with respect to the design variables used in the proposed method can be obtained by the adjoint variable method. The equation of state used in the analysis is the Reynolds equation that determines the pressure on the oil film. As a result, the number of iterations can be significantly reduced in comparison to Nesterov's acceleration gradient method. The optimal oil film thickness distribution in the textured area was also calculated. The texture shape after convergence was asymmetric in the flow direction. This pattern was similar to that observed in shape optimization using Nesterov's acceleration gradient method. Numerical experiments performed to validate the proposed method are described.

Gradient flows and proximal splitting methods: A unified view on accelerated and stochastic optimization.

Optimization is at the heart of machine learning, statistics, and many applied scientific disciplines. It also has a long history in physics, ranging from the minimal action principle to finding ground states of disordered systems such as spin glasses. Proximal algorithms form a class of methods that are broadly applicable and are particularly well-suited to nonsmooth, constrained, large-scale, and distributed optimization problems. There are essentially five proximal algorithms currently known, each proposed in seminal work: Forward-backward splitting, Tseng splitting, Douglas-Rachford, alternating direction method of multipliers, and the more recent Davis-Yin. These methods sit on a higher level of abstraction compared to gradient-based ones, with deep roots in nonlinear functional analysis. In this paper we show that all of these methods are actually different discretizations of a single differential equation, namely, the simple gradient flow which dates back to Cauchy (1847). An important aspect behind many of the success stories in machine learning relies on "accelerating" the convergence of first-order methods. However, accelerated methods are notoriously difficult to analyze, counterintuitive, and without an underlying guiding principle. We show that similar discretization schemes applied to Newton's equation with an additional dissipative force, which we refer to as accelerated gradient flow, allow us to obtain accelerated variants of all these proximal algorithms-the majority of which are new although some recover known cases in the literature. Furthermore, we extend these methods to stochastic settings, allowing us to make connections with Langevin and Fokker-Planck equations. Similar ideas apply to gradient descent, heavy ball, and Nesterov's method which are simpler. Our results therefore provide a unified framework from which several important optimization methods are nothing but simulations of classical dissipative systems.

Distributed Generalized Nash Equilibrium Seeking for Energy Sharing Games in Prosumers

With the proliferation of distributed generators and energy storage systems, traditional passive consumers in power systems have been gradually evolving into the so-called “prosumers,” i.e., proactive consumers, which can both produce and consume power. To encourage energy exchange among prosumers, energy sharing is increasingly adopted, which is usually formulated as a generalized Nash game (GNG). In this paper, a distributed approach is proposed to seek the Generalized Nash equilibrium (GNE) of the energy sharing game. To this end, we first prove the strong monotonicity of the game. Then, the GNG is converted into an equivalent optimization problem. An algorithm based on Nesterov's methods is thereby devised to solve the equivalent problem and consequently find the GNE in a distributed manner. The convergence of the proposed algorithm is proved rigorously based on the nonexpansive operator theory. The performance of the algorithm is validated by experiments with three prosumers, and the scalability is tested by simulations using 1888 prosumers.

Continuous-Time Accelerated Methods via a Hybrid Control Lens

Treating optimization methods as dynamical systems can be traced back centuries ago in order to comprehend the notions and behaviors of optimization methods. Lately, this mindset has become the driving force to design new optimization methods. Inspired by the recent dynamical system viewpoint of Nesterov's fast method, we propose two classes of fast methods, formulated as hybrid control systems, to obtain prespecified exponential convergence rate. Alternative to the existing fast methods, which are parametric-in-time second-order differential equations, we dynamically synthesize feedback controls in a state-dependent manner. Namely, in the first class, the damping term is viewed as the control input, while in the second class the amplitude with which the gradient of the objective function impacts the dynamics serves as the controller. The objective function requires to satisfy the so-called Polyak–Łojasiewicz inequality, which effectively implies no local optima and a certain gradient-domination property. Moreover, we establish that both hybrid structures possess Zeno-free solution trajectories. We finally provide a mechanism to determine the discretization step size to attain an exponential convergence rate.

Resilient Desynchronization for Decentralized Medium Access Control

In Wireless Sensor Networks (WSNs), equally spaced timing for Medium Access Control (MAC) is fundamental to guarantee throughput maximization from all nodes. This motivated the so called desynchronization problem and its solution based on the fast Nesterov method. In this letter, we tackle the problem of constructing centralized and distributed versions of the optimal fixed-parameter Nesterov that are resilient to attacks to a subset of nodes. By showing a relationship between the variance of the attacker signal and how further away a node is, we are able to present a distributed algorithm that has minimal added complexity and performs the detection and isolation of the faulty node. Simulations are provided illustrating the successful detection and highlighting that without a correction mechanism (dependent on additional assumptions), there is a residual error that is not eliminated.

Robustness of Accelerated First-Order Algorithms for Strongly Convex Optimization Problems

We study the robustness of accelerated first-order algorithms to stochastic uncertainties in gradient evaluation. Specifically, for unconstrained, smooth, strongly convex optimization problems, we examine the mean-squared error in the optimization variable when the iterates are perturbed by additive white noise. This type of uncertainty may arise in situations where an approximation of the gradient is sought through measurements of a real system or in a distributed computation over a network. Even though the underlying dynamics of first-order algorithms for this class of problems are nonlinear, we establish upper bounds on the mean-squared deviation from the optimal solution that are tight up to constant factors. Our analysis quantifies fundamental tradeoffs between noise amplification and convergence rates obtained via any acceleration scheme similar to Nesterov's or heavy-ball methods. To gain additional analytical insight, for strongly convex quadratic problems, we explicitly evaluate the steady-state variance of the optimization variable in terms of the eigenvalues of the Hessian of the objective function. We demonstrate that the entire spectrum of the Hessian, rather than just the extreme eigenvalues, influences robustness of noisy algorithms. We specialize this result to the problem of distributed averaging over undirected networks and examine the role of network size and topology on the robustness of noisy accelerated algorithms.

Optimal Distributed Convex Optimization on Slowly Time-Varying Graphs

We study optimal distributed first-order optimization algorithms when the network (i.e., communication constraints between the agents) changes with time. This problem is motivated by scenarios where agents experience network malfunctions. We provide a sufficient condition that guarantees a convergence rate with optimal (up lo logarithmic terms) dependencies on the network and function parameters if the network changes are constrained to a small percentage $\alpha$ of the total number of iterations. We call such networks slowly time-varying networks. Moreover, we show that Nesterov's method has an iteration complexity of $\Omega \big( \big(\sqrt{\kappa_\Phi \cdot \bar{\chi}} + \alpha \log(\kappa_\Phi \cdot \bar{\chi})\big) \log(1 / \varepsilon)\big)$ for decentralized algorithms, where $\kappa_\Phi$ is condition number of the objective function, and $\bar\chi$ is a worst case bound on the condition number of the sequence of communication graphs. Additionally, we provide an explicit upper bound on $\alpha$ in terms of the condition number of the objective function and network topologies.

A Nesterov-Like Gradient Tracking Algorithm for Distributed Optimization Over Directed Networks

In this article, we concentrate on dealing with the distributed optimization problem over a directed network, where each unit possesses its own convex cost function and the principal target is to minimize a global cost function (formulated by the average of all local cost functions) while obeying the network connectivity structure. Most of the existing methods, such as push-sum strategy, have eliminated the unbalancedness induced by the directed network via utilizing column-stochastic weights, which may be infeasible if the distributed implementation requires each unit to gain access to (at least) its out-degree information. In contrast, to be suitable for the directed networks with row-stochastic weights, we propose a new directed distributed Nesterov-like gradient tracking algorithm, named as D-DNGT, that incorporates the gradient tracking into the distributed Nesterov method with momentum terms and employs nonuniform step-sizes. D-DNGT extends a number of outstanding consensus algorithms over strongly connected directed networks. The implementation of D-DNGT is straightforward if each unit locally chooses a suitable step-size and privately regulates the weights on information that acquires from in-neighbors. If the largest step-size and the maximum momentum coefficient are positive and small sufficiently, we can prove that D-DNGT converges linearly to the optimal solution provided that the cost functions are smooth and strongly convex. We provide numerical experiments to confirm the findings in this article and contrast D-DNGT with recently proposed distributed optimization approaches.

Stability analysis of ground states in a one-dimensional trapped spin-1 Bose gas

In this work we study the stability properties of the ground states of a spin-1 Bose gas in presence of a trapping potential in one spatial dimension. To set the stage we first map out the phase diagram for the trapped system by making use of a, so-called, continuous-time Nesterov method. We present an extension of the method, which has been previously applied to one-component systems, to our multi-component system. We show that it is a powerful and robust tool for finding the ground states of a physical system without the need of an accurate initial guess. We subsequently solve numerically the Bogoliubov de-Gennes equations in order to analyze the stability of the ground states of the trapped spin-1 system. We find that the trapping potential retains the overall structure of the stability diagram, while affecting the spectral details of each of the possible ground state waveforms. It is also found that the peak density of the trapped system is the characteristic quantity describing dynamical instabilities in the system. Therefore replacing the homogeneous density with the peak density of the trapped system leads to good agreement of the homogeneous Bogoliubov predictions with the numerically observed maximal growth rates of dynamically unstable modes. The stability conclusions in the one-dimensional trapped system are independent of the spin coupling strength and the normalized trap strength over several orders of magnitude of their variation.