Inexact Gradient Research Articles

A new inexact gradient descent method with applications to nonsmooth convex optimization

The paper proposes and develops a novel inexact gradient method (IGD) for minimizing C 1 -smooth functions with Lipschitzian gradients, i.e. for problems of C 1 , 1 optimization. We show that the sequence of gradients generated by IGD converges to zero. The convergence of iterates to stationary points is guaranteed under the Kurdyka-Łojasiewicz (KL) property of the objective function with convergence rates depending on the KL exponent. The newly developed IGD is applied to designing two novel gradient-based methods of nonsmooth convex optimization such as the inexact proximal point methods (GIPPM) and the inexact augmented Lagrangian method (GIALM) for convex programs with linear equality constraints. These two methods inherit global convergence properties from IGD and are confirmed by numerical experiments to have practical advantages over some well-known algorithms of nonsmooth convex optimization.

An inexact semismooth Newton method with application to adaptive randomized sketching for dynamic optimization

In many applications, one can only access the inexact gradients and inexact hessian times vector products. Thus it is essential to consider algorithms that can handle such inexact quantities with a guaranteed convergence to solution. An inexact adaptive and provably convergent semismooth Newton method is considered to solve constrained optimization problems. In particular, dynamic optimization problems, which are known to be highly expensive, are the focus. A memory efficient semismooth Newton algorithm is introduced for these problems. The source of efficiency and inexactness is the randomized matrix sketching. Applications to optimization problems constrained by partial differential equations are also considered.

Accelerated Iterated Filtering

Simulation-based inferences have attracted much attention in recent years, as the direct computation of the likelihood function in many real-world problems is difficult or even impossible. Iterated filtering (Ionides, Bretó, and King 2006; Ionides, Bhadra, Atchadé,and King 2011) enables maximization of likelihood function via model perturbations and approximation of the gradient of loglikelihood through sequential Monte Carlo filtering. By an application of Stein’s identity, Doucet, Jacob, and Rubenthaler (2013) developed asecond-order approximation of the gradient of log-likelihood using sequential Monte Carlo smoothing. Based on these gradient approximations, we develop a new algorithm for maximizing the likelihood using the Nesterov accelerated gradient. We adopt the accelerated inexact gradient algorithm (Ghadimi and Lan 2016) to iterated filtering framework, relaxing the unbiased gradient approximation condition. We devise a perturbation policy for iterated filtering, allowing the new algorithm to converge at an optimal rate for both concave and non-concave log-likelihood functions. It is comparable to the recently developed Bayes map iterated filtering approach and outperforms the original iterated filtering approach.

Stopping Rules for Gradient Methods for Non-convex Problems with Additive Noise in Gradient

We study the gradient method under the assumption that an additively inexact gradient is available for, generally speaking, non-convex problems. The non-convexity of the objective function, as well as the use of an inexactness specified gradient at iterations, can lead to various problems. For example, the trajectory of the gradient method may be far enough away from the starting point. On the other hand, the unbounded removal of the trajectory of the gradient method in the presence of noise can lead to the removal of the trajectory of the method from the desired global solution. The results of investigating the behavior of the trajectory of the gradient method are obtained under the assumption of the inexactness of the gradient and the condition of gradient dominance. It is well known that such a condition is valid for many important non-convex problems. Moreover, it leads to good complexity guarantees for the gradient method. A rule of early stopping of the gradient method is proposed. Firstly, it guarantees achieving an acceptable quality of the exit point of the method in terms of the function. Secondly, the stopping rule ensures a fairly moderate distance of this point from the chosen initial position. In addition to the gradient method with a constant step, its variant with adaptive step size is also investigated in detail, which makes it possible to apply the developed technique in the case of an unknown Lipschitz constant for the gradient. Some computational experiments have been carried out which demonstrate effectiveness of the proposed stopping rule for the investigated gradient methods.

A parallel non-convex approximation framework for risk parity portfolio design

In this paper, we propose a parallel non-convex approximation framework (NCAQ) for optimization problems whose objective is to minimize a convex function plus the sum of non-convex functions. Based on the structure of the objective function, our framework transforms the non-convex constraints to the logarithmic barrier function and approximates the non-convex problem by a parallel quadratic approximation scheme, which will allow the original problem to be solved by accelerated inexact gradient descent in the parallel environment. Moreover, we give a detailed convergence analysis for the proposed framework. The numerical experiments show that our framework outperforms the state-of-art approaches in terms of accuracy and computation time on the high dimension non-convex Rosenbrock test functions and the risk parity problems. In particular, we implement the proposed framework on CUDA, showing a more than 25 times speed-up ratio and removing the computational bottleneck for non-convex risk-parity portfolio design. Finally, we construct the high dimension risk parity portfolio which can consistently outperform the equal weight portfolio in the application of Chinese stock markets.

Channel capacity and power allocation of MIMO visible light communication system

In this paper, the channel capacity of the multiple-input multiple-output (MIMO) visible light communication (VLC) system is investigated under the peak, average optical and electrical power constraints. Finding the channel capacity of MIMO VLC is shown to be a mixed integer programming problem. To address this open problem, we propose an inexact gradient projection method to find the channel capacity-achieving discrete input distribution and the channel capacity of MIMO VLC. Also we derive both upper and lower bounds of the capacity of MIMO VLC with the closed-form expressions. Furthermore, by considering practical discrete constellation inputs, we develop the optimal power allocation scheme to maximize transmission rate of MIMO VLC system. Simulation results show that more discrete points are needed to achieve the channel capacity as SNR increases. Both the upper and lower bounds of channel capacity are tight at low SNR region. In addition, comparing the equal power allocation, the proposed power allocation scheme can significantly increase the rate for the low-order modulation inputs.

A BFGS method using inexact gradient for general nonlinear equations

A globally and superlinearly convergent BFGS methods is introduced to solve general nonlinear equations without computing exact gradient. Compared with existing Gauss-Newton-based BFGS type methods, the proposed method does not require conditions such as sysmmetry on the underlying function. Moreover, it can be suitably adjusted to solve nonlinear least squares problems and still guarantee global convergence. Some numerical results are reported are reported to show its efficiency.

Smoothed Variable Sample-Size Accelerated Proximal Methods for Nonsmooth Stochastic Convex Programs

We consider the unconstrained minimization of the function F, where F = f + g, f is an expectation-valued nonsmooth convex or strongly convex function, and g is a closed, convex, and proper function. (I) Strongly convex f. When f is μ-strongly convex in x, traditional stochastic subgradient schemes ( SSG ) often display poor behavior, arising in part from noisy subgradients and diminishing steplengths. Instead, we apply a variable sample-size accelerated proximal scheme (VS-APM) on F, the Moreau envelope of F; we term such a scheme as ( mVS-APM ) and in contrast with ( SSG ) schemes, ( mVS-APM ) utilizes constant steplengths and increasingly exact gradients. We consider two settings. (a) Bounded domains. In this setting, ( mVS-APM ) displays linear convergence in inexact gradient steps, each of which requires utilizing an inner ( prox-SSG ) scheme. Specically, ( mVS-APM ) achieves an optimal oracle complexity in prox-SSG steps of [Formula: see text] with an iteration complexity of [Formula: see text] in inexact (outer) gradients of F to achieve an ϵ-accurate solution in mean-squared error, computed via an increasing number of inner (stochastic) subgradient steps; (b) Unbounded domains. In this regime, under an assumption of state-dependent bounds on subgradients, an unaccelerated variant ( mVS-APM ) is linearly convergent where increasingly exact gradients ∇xF(x) are approximated with increasing accuracy via ( SSG ) schemes. Notably, ( mVS-APM ) also displays an optimal oracle complexity of [Formula: see text]; (II) Convex f. When f is merely convex but smoothable, by suitable choices of the smoothing, steplength, and batch-size sequences, smoothed (VS-APM) (or sVS-APM) achieves an optimal oracle complexity of [Formula: see text] to obtain an ϵ-optimal solution. Our results can be specialized to two important cases: (a) Smooth f. Since smoothing is no longer required, we observe that (VS-APM) admits the optimal rate and oracle complexity, matching prior ndings; (b) Deterministic nonsmooth f. In the nonsmooth deterministic regime, (sVS-APM) reduces to a smoothed accelerated proximal method (s-APM) that is both asymptotically convergent and optimal in that it displays a complexity of [Formula: see text], matching the bound provided by Nesterov in 2005 for producing ϵ-optimal solutions. Finally, (sVS-APM) and (VS-APM) produce sequences that converge almost surely to a solution of the original problem. Funding: The first and second authors would like to acknowledge support from NSF CMMI-1538605, DOE ARPA-E award DE-AR0001076, and the Gary and Sheila Bello Chair funds.

Exact gradient methods with memory

ABSTRACT The Inexact Gradient Method with Memory (IGMM) is able to considerably outperform the Gradient Method by employing a piece-wise linear lower model on the smooth part of the objective. However, the auxiliary problem can only be solved within a fixed tolerance at every iteration. The need to contain the inexactness narrows the range of problems to which IGMM can be applied and degrades the worst-case convergence rate. In this work, we show how a simple modification of IGMM removes the tolerance parameter from the analysis. The resulting Exact Gradient Method with Memory (EGMM) is as broadly applicable as the Bregman Distance Gradient Method/NoLips and has the same worst-case rate of , the best for its class. Under necessarily stricter assumptions, we can accelerate EGMM without error accumulation yielding an Accelerated Gradient Method with Memory (AGMM) possessing a worst-case rate of . In our preliminary computational experiments EGMM displays excellent performance, sometimes surpassing accelerated methods. When the model discards old information, AGMM also consistently exceeds the Fast Gradient Method.

Optimal Discrete Constellation Inputs for Aggregated LiFi-WiFi Networks

In this paper, we investigate the performance of a practical aggregated LiFi-WiFi system with the discrete constellation inputs from a practical view. We derive the achievable rate expressions of the aggregated LiFi-WiFi system for the first time. Then, we study the rate maximization problem via optimizing the constellation distribution and power allocation jointly. Specifically, a multilevel mercy-filling power allocation scheme is proposed by exploiting the relationship between the mutual information and minimum mean-squared error (MMSE) of discrete inputs. Meanwhile, an inexact gradient descent method is proposed for obtaining the optimal probability distributions. To strike a balance between the computational complexity and the transmission performance, we further develop a framework that maximizes the lower bound of the achievable rate where the optimal power allocation can be obtained in closed forms and the constellation distributions problem can be solved efficiently by Frank-Wolfe method. Extensive numerical results show that the optimized strategies are able to provide significant gains over the state-of-the-art schemes in terms of the achievable rate.

A Note on the Optimal Convergence Rate of Descent Methods with Fixed Step Sizes for Smooth Strongly Convex Functions

Based on a result by Taylor et al. (J Optim Theory Appl 178(2):455–476, 2018) on the attainable convergence rate of gradient descent for smooth and strongly convex functions in terms of function values, an elementary convergence analysis for general descent methods with fixed step sizes is presented. It covers general variable metric methods, gradient-related search directions under angle and scaling conditions, as well as inexact gradient methods. In all cases, optimal rates are obtained.

Derivatives for Eigenvalues and Eigenvectors via Analytic Reverse Algorithmic Differentiation

Eigenvalue and eigenvectors are essential metrics to characterize dynamic system behavior and stability. When performing gradient-based design optimization, derivatives of these metrics are required. Analytic forward algorithmic differentiation (FAD) for a self-adjoint generalized eigenproblem has been a useful technique. However, reverse algorithmic differentiation (RAD) is preferred over FAD because it scales more favorably with the number of design variables. We propose two RAD formulas based on their mode-based FAD counterparts that project the derivative onto a reduced eigenvector space. One challenge for the mode-based derivative is that the reduced eigenvectors yield inexact gradient results. An approximation technique mitigates this issue. We verify the proposed methods by implementing a reverse Lanczos iteration and the adjoint of an Euler–Bernoulli beam test case.

Convergence of Distributed Accelerated Algorithm Over Unbalanced Directed Networks

In this article, the problem of the distributed convex optimization is investigated, where the target is to collectively minimize a sum of local convex functions over an unbalanced directed multiagent network. Each agent in the network possesses only its private local objective function, and the sum of all local objective functions constitutes the global objective function. We particularly consider the scenario, where the underlying interaction network is strongly connected and the relevant weight matrix is row stochastic. To collectively figure out the optimization problem, a distributed accelerated convergence algorithm where agents utilize uncoordinated step-sizes is presented by incorporating consensus of multiagent networks into distributed inexact gradient tracking technique. Most of the existing methods require all agents to possess the out-degree information of their in-neighbors, which is impractical and hardly inevitable as interpreted in this article. By utilizing the small-gain theorem, we prove that if the maximum step-size is positive and sufficiently small (constrained by a specific upper bound), the proposed algorithm, termed as SGT-FROST, converges geometrically to the optimal solution given that the objective functions are smooth and strongly convex. A certain convergence rate is also shown. Simulations confirm the findings in this article.

A frequency-domain analysis of inexact gradient methods

We study robustness properties of some iterative gradient-based methods for strongly convex functions, as well as for the larger class of functions with sector-bounded gradients, under a relative error model. Proofs of the corresponding convergence rates are based on frequency-domain criteria for the stability of nonlinear systems. Applications are given to inexact versions of gradient descent and the Triple Momentum Method. To further emphasize the usefulness of frequency-domain methods, we derive improved analytic bounds for the convergence rate of Nesterov’s accelerated method (in the exact setting) on strongly convex functions.

Stochastic analysis of an adaptive cubic regularization method under inexact gradient evaluations and dynamic Hessian accuracy

We here adapt an extended version of the adaptive cubic regularization method with dynamic inexact Hessian information for nonconvex optimization in Bellavia et al. [Adaptive cubic regularization methods with dynamic inexact hessian information and applications to finite-sum minimization. IMA Journal of Numerical Analysis. 2021;41(1):764–799] to the stochastic optimization setting. While exact function evaluations are still considered, this novel variant inherits the innovative use of adaptive accuracy requirements for Hessian approximations introduced in the just quoted paper and additionally employs inexact computations of the gradient. Without restrictions on the variance of the errors, we assume that these approximations are available within a sufficiently large, but fixed, probability and we extend, in the spirit of Cartis and Scheinberg [Global convergence rate analysis of unconstrained optimization methods based on probabilistic models. Math Program Ser A. 2018;159(2):337–375], the deterministic analysis of the framework to its stochastic counterpart, showing that the expected number of iterations to reach a first-order stationary point matches the well-known worst-case optimal complexity. This is, in fact, still given by , with respect to the first-order ϵ tolerance. Finally, numerical tests on nonconvex finite-sum minimization confirm that using inexact first- and second-order derivatives can be beneficial in terms of the computational savings.

Gradient Methods With Dynamic Inexact Oracles

We present a framework for generalizing the primal-dual gradient method, also known as the gradient descent ascent method, for solving convex-concave minimax problems. The framework is based on the observation that the primal-dual gradient method can be viewed as an inexact gradient method applied to the primal problem. Unlike the setting of traditional inexact gradient methods, the inexact gradient is computed by a dynamic inexact oracle, which is a discrete-time dynamical system whose output asymptotically approaches the exact gradient. For minimax problems, dynamic inexact oracles are capable of modeling a range of first-order methods for computing the gradient of the primal objective, which relies on solving the inner maximization problem. We provide a unified convergence analysis of gradient methods with dynamic inexact oracles and demonstrate its use in creating new accelerated primal-dual algorithms.

Accelerated and Unaccelerated Stochastic Gradient Descent in Model Generality

A new method for deriving estimates of the rate of convergence of optimal methods for solving problems of smooth (strongly) convex stochastic optimization is described. The method is based on the results of stochastic optimization derived from results on the convergence of optimal methods under the conditions of inexact gradients with small noises of nonrandom nature. In contrast to earlier results, all estimates in the present paper are obtained in model generality.

An Inexact Interior-Point Lagrangian Decomposition Algorithm with Inexact Oracles

We develop a new inexact interior-point Lagrangian decomposition method to solve a wide range class of constrained composite convex optimization problems. Our method relies on four techniques: Lagrangian dual decomposition, self-concordant barrier smoothing, path-following, and proximal-Newton technique. It also allows one to approximately compute the solution of the primal subproblems (called the slave problems), which leads to inexact oracles (i.e., inexact gradients and Hessians) of the smoothed dual problem (called the master problem). The smoothed dual problem is nonsmooth, we propose to use an inexact proximal-Newton method to solve it. By appropriately controlling the inexact computation at both levels: the slave and master problems, we still estimate a polynomial-time iteration-complexity of our algorithm as in standard short-step interior-point methods. We also provide a strategy to recover primal solutions and establish complexity to achieve an approximate primal solution. We illustrate our method through two numerical examples on well-known models with both synthetic and real data and compare it with some existing state-of-the-art methods.

Achieving Channel Capacity of Visible Light Communication

In this article, we propose an efficient inexact gradient descent method to obtain the channel capacity-achieving input distribution of visible light communication (VLC). Under both a peak and an average optical power constraints, finding the channel capacity of VLC is formulated as a mixed continuous-discrete optimization problem without an analytical expression of the objective function. To solve this challenging problem, we first adopt the numerical integration method to approximate the objective function and its gradient. Here, we prove that the gaps between the original functions and the approximations can be arbitrarily small. Then, based on the approximated functions, we describe the method, and theoretically show that the obtained solution sequence converges to the channel capacity-achieving discrete distribution of VLC. We also provide simulation results to verify the effectiveness and optimality of the proposed method. More importantly, the simulations numerically reveal that on-off keying (OOK) modulation achieves the capacity of VLC channel at low signal-to-noise ratio (SNR), while pulse amplitude modulation (PAM) achieves the capacity at high SNR.

Event-Triggering Interaction Scheme for Discrete-Time Decentralized Optimization With Nonuniform Step Sizes.

In this article, we study the discrete-time decentralized optimization problems of multiagent systems by an event-triggering interaction scheme, in which each agent privately knows its local convex cost function, and collectively minimizes the total cost functions. The underlying interaction and the corresponding weight matrix are required to be undirected connected and doubly stochastic, respectively. To resolve this optimization problem collaboratively, we propose a decentralized event-triggering algorithm (DETA) that is based on the consensus theory and inexact gradient tracking technique. DETA involves each agent interacting with its neighboring agents only at some independent event-triggering sampling time instants. Under the assumptions that the global convex cost function is coercive and has Lipschitz continuous gradient, we prove that DETA steers all agents' states to an optimal solution even with nonuniform constant step sizes. Moreover, our analysis also shows that DETA converges at a rate of O(1/√t) if the step sizes are uniform and do not exceed some upper bounds. We illustrate the effectiveness of DETA on a canonical simple decentralized parameter estimation problem.