First-order Algorithm Research Articles

S-NEAR-DGD: A Flexible Distributed Stochastic Gradient Method for Inexact Communication

We present and analyze a stochastic distributed method (S-NEAR-DGD) that can tolerate inexact computation and inaccurate information exchange to alleviate the problems of costly gradient evaluations and bandwidth-limited communication in large-scale systems. Our method is based on a class of flexible, distributed first-order algorithms that allow for the tradeoff of computation and communication to best accommodate the application setting. We assume that the information exchanged between nodes is subject to random distortion and that only stochastic approximations of the true gradients are available. Our theoretical results prove that the proposed algorithm converges linearly in expectation to a neighborhood of the optimal solution for strongly convex objective functions with Lipschitz gradients. We characterize the dependence of this neighborhood on algorithm and network parameters, the quality of the communication channel and the precision of the stochastic gradient approximations used. Finally, we provide numerical results to evaluate the empirical performance of our method.

Meta-Learning a Cross-lingual Manifold for Semantic Parsing

Abstract Localizing a semantic parser to support new languages requires effective cross-lingual generalization. Recent work has found success with machine-translation or zero-shot methods, although these approaches can struggle to model how native speakers ask questions. We consider how to effectively leverage minimal annotated examples in new languages for few-shot cross-lingual semantic parsing. We introduce a first-order meta-learning algorithm to train a semantic parser with maximal sample efficiency during cross-lingual transfer. Our algorithm uses high-resource languages to train the parser and simultaneously optimizes for cross-lingual generalization to lower-resource languages. Results across six languages on ATIS demonstrate that our combination of generalization steps yields accurate semantic parsers sampling ≤10% of source training data in each new language. Our approach also trains a competitive model on Spider using English with generalization to Chinese similarly sampling ≤10% of training data.1

Distributed State Fusion Estimation of Multi-Source Localization Nonlinear Systems.

For the state estimation problem of a multi-source localization nonlinear system with unknown and bounded noise, a distributed sequential ellipsoidal intersection fusion estimation algorithm based on the dual set-membership filtering method is proposed to ensure the reliability of the localization system. First, noise with unknown and bounded characteristics is modeled by using bounded ellipsoidal regions. At the same time, local estimators are designed at the sensor link nodes to filter out the noise interference in the localization system. The local estimator is designed using the dual set-membership filtering algorithm. It uses the dual principle to find the minimizing ellipsoid that can contain the nonlinear function by solving the optimization problem with semi-infinite constraints, and a first-order conditional gradient algorithm is used to solve the optimization problem with a low computational complexity. Meanwhile, the communication confusion among multiple sensors causes the problem of unknown correlation. The obtained estimates of local filters are fused at the fusion center by designing a distributed sequential ellipsoid intersection fusion estimation algorithm to obtain more accurate fusion localization results with lower computational cost. Finally, the stability and reliability of the proposed distributed fusion algorithm are verified by designing a simulation example of a multi-source nonlinear system.

Integrated photovoltaic storage joint smoothing strategy based on simultaneous perturbation stochastic approximation algorithm

In order to realize the real-time control of photovoltaic power generation smoothly connected to the grid under the condition that the energy storage equipment can operate safely, a control strategy combining the simultaneous perturbation stochastic approximation (SPSA) algorithm with rule control is designed. Firstly, the photovoltaic data are processed to extract the data characteristics of the power ramp, and then the grid-connected reference power is obtained through SPSA algorithm. Secondly, considering the state of charge (SOC) of energy storage equipment and the safe operating power range of hydrogen storage equipment, 24 hybrid energy storage power allocation rules are formulated by using the rule control method. Finally, according to the sampling data of every 10 s interval in typical day, the simulation is carried out on MATLAB/simulink platform. The results show that, compared with the first-order low-pass filtering algorithm and recursive fuzzy neural network (RFNN) algorithm, the SPSA algorithm has stronger smoothing ability, and the rule control can also complete the allocation according to the characteristics of the hybrid storage device while ensuring its normal operation.

Low-Rank Room Impulse Response Estimation

In this paper we consider low-rank estimation of room impulse responses (RIRs). Inspired by a physics-driven room-acoustical model, we propose an estimator of RIRs that promotes a low-rank structure for a matricization, or reshaping, of the estimated RIR. This low-rank prior acts as a regularizer for the inverse problem of estimating an RIR from input-output observations, preventing overfitting and improving estimation accuracy. As directly enforcing a low rank of the estimate results is an NP-hard problem, we consider two different relaxations, one using the nuclear norm, and one using the recently introduced concept of quadratic envelopes. Both relaxations allow for implementing the proposed estimator using a first-order algorithm with convergence guarantees. When evaluated on both synthetic and recorded RIRs, it is shown that under noisy output conditions, or when the spectral excitation of the input signal is poor, the proposed estimator outperforms comparable existing methods. The performance of the two low-rank relaxations methods is similar, but the quadratic envelope has the benefit of superior robustness to the choice of regularization hyperparameter in the case when the signal-to-noise ratio is unknown. The performance of the proposed method is compared to that of ordinary least squares, Tikhonov least squares, as well as the Cramér-Rao lower bound (CRLB).

Space-Time Electromagnetic Inverse Scattering Tomography of Objects With Affine Deformation

In this article, an electromagnetic inverse scattering problem (EISP) method is implemented successfully to capture the dynamic phenomena of the scene with affine deformation. In the presented method, a new variational model has been suggested which is based on two error functions of the current and state equations. Moreover, two types of regularizers have been adopted in the proposed scheme. In the first type, two separated spatial total variation (TV) and smooth, temporal regularization terms are used. In the second kind, a TV regularization, being a function of both spatial and temporal variables, is used. It is shown that the second type gives better results compared to the first type. The minimization problem is solved by the first-order primal-dual optimization algorithm of Chambolle and Pock.

Ultra-Low-Complexity Algorithms With Structurally Optimal Multi-Group Multicast Beamforming in Large-Scale Systems

In this work, we propose ultra-low-complexity design solutions for multi-group multicast beamforming in large-scale systems. For the quality-of-service (QoS) problem, by utilizing the optimal multicast beamforming structure obtained recently in [2], we convert the original problem into a non-convex weight optimization problem of a lower dimension and propose two fast first-order algorithms to solve it. Both algorithms are based on successive convex approximation (SCA) and provide fast iterative updates to solve each SCA subproblem. The first algorithm uses a saddle point reformulation in the dual domain and applies the extragradient method with an adaptive step-size procedure to find the saddle point with simple closed-form updates. The second algorithm adopts the alternating direction method of multipliers (ADMM) method by converting each SCA subproblem into a favorable ADMM structure. The structure leads to simple closed-form ADMM updates, where the problem in each update block can be further decomposed into parallel subproblems of small sizes, for which closed-form solutions are obtained. We also propose efficient initialization methods to obtain favorable initial points that facilitate fast convergence. Furthermore, taking advantage of the proposed fast algorithms, for the max-min fair (MMF) problem, we propose a simple closed-form scaling scheme that directly uses the solution obtained from the QoS problem, avoiding the conventional computationally expensive method that iteratively solves the inverse QoS problem. We further develop lower and upper bounds on the performance of this scaling scheme. Simulation results show that the proposed algorithms offer near-optimal performance with substantially lower computational complexity than the state-of-the-art algorithms for large-scale systems.

On the effect of perturbations in first-order optimization methods with inertia and Hessian driven damping

<p style='text-indent:20px;'>Second-order continuous-time dissipative dynamical systems with viscous and Hessian driven damping have inspired effective first-order algorithms for solving convex optimization problems. While preserving the fast convergence properties of the Nesterov-type acceleration, the Hessian driven damping makes it possible to significantly attenuate the oscillations. To study the stability of these algorithms with respect to perturbations, we analyze the behaviour of the corresponding continuous systems when the gradient computation is subject to exogenous additive errors. We provide a quantitative analysis of the asymptotic behaviour of two types of systems, those with implicit and explicit Hessian driven damping. We consider convex, strongly convex, and non-smooth objective functions defined on a real Hilbert space and show that, depending on the formulation, different integrability conditions on the perturbations are sufficient to maintain the convergence rates of the systems. We highlight the differences between the implicit and explicit Hessian damping, and in particular point out that the assumptions on the objective and perturbations needed in the implicit case are more stringent than in the explicit case.</p>

Methodology and first-order algorithms for solving nonsmooth and non-strongly convex bilevel optimization problems

Simple bilevel problems are optimization problems in which we want to find an optimal solution to an inner problem that minimizes an outer objective function. Such problems appear in many machine learning and signal processing applications as a way to eliminate undesirable solutions. In our work, we suggest a new approach that is designed for bilevel problems with simple outer functions, such as the \(l_1\) norm, which are not required to be either smooth or strongly convex. In our new ITerative Approximation and Level-set EXpansion (ITALEX) approach, we alternate between expanding the level-set of the outer function and approximately optimizing the inner problem over this level-set. We show that optimizing the inner function through first-order methods such as proximal gradient and generalized conditional gradient results in a feasibility convergence rate of O(1/k), which up to now was a rate only achieved by bilevel algorithms for smooth and strongly convex outer functions. Moreover, we prove an \(O(1/\sqrt{k})\) rate of convergence for the outer function, contrary to existing methods, which only provide asymptotic guarantees. We demonstrate this performance through numerical experiments.

A new class of composite objective multistep estimating sequence techniques

A plethora of problems arising in signal processing, machine learning and statistics can be cast as large-scale optimization problems with a composite objective structure. Such problems are typically solved by utilizing iterative first-order algorithms. In this work, we devise a new accelerated gradient-based estimating sequence technique for solving large-scale optimization problems with composite objective structure. Specifically, we introduce a new class of estimating functions, which are obtained by utilizing both a tight lower bound on the objective function, as well as the gradient mapping technique. Then, using the proposed estimating functions, we construct a class of Composite Objective Multi-step Estimating-sequence Techniques (COMET), which are endowed with an efficient line-search procedure. We prove that our proposed COMET enjoys the accelerated convergence rate, and our newly established convergence results allow for step-size adaptation. Our theoretical findings are supported by extensive computational experiments on various problem types and real-world datasets. Moreover, our numerical results show evidence of the robustness of the proposed method to the imperfect knowledge of the smoothness and strong convexity parameters.

Distributed control strategy for voltage and frequency restoration and accurate reactive power-sharing for islanded microgrid

For a conventional droop-controlled islanded microgrid, the poor reactive power-sharing and drop of bus voltage and frequency caused by line impedance mismatch and droop characteristics are inevitable. So far, consensus-based distributed control algorithms, in which a secondary control layer with the integration of voltage and frequency restoration and reactive power-sharing compensation mechanisms, has been treated as a promising solution. However, for most of the currently available distributed algorithms, it is the active power-sharing rather than the reactive power-sharing that was made the control goal, which is opposite to the fact that accurate power-sharing could always hold considering the integration effect in the active power control loop. Although a few recently published distributed algorithms added the reactive power-sharing goal in the second control layer, the system efficiency and reliability may deteriorate because a large circulating current could be induced at the malfunction of the communication network since there is no virtual impedance injected at the bottom control layer to limit that. This paper proposes a voltage and frequency restoration method based on the dynamic average consensus algorithm and an accurate reactive power-sharing method based on a first-order consensus algorithm and virtual impedance method. A simple pseudo-unidirectional communication network with a narrow communication band is adopted for the proposed distributed control to lower the communication burden and system complexity. The operational principle is first analyzed in detail. The proposed strategy is easily implementable and features not breaking droop characteristics. Then simulations and hardware-in-the-loop (HIL) real-time experiments verify the proposed approach.

Achieving Online Regression Performance of LSTMs With Simple RNNs.

Recurrent neural networks (RNNs) are widely used for online regression due to their ability to generalize nonlinear temporal dependencies. As an RNN model, long short-term memory networks (LSTMs) are commonly preferred in practice, as these networks are capable of learning long-term dependencies while avoiding the vanishing gradient problem. However, due to their large number of parameters, training LSTMs requires considerably longer training time compared to simple RNNs (SRNNs). In this article, we achieve the online regression performance of LSTMs with SRNNs efficiently. To this end, we introduce a first-order training algorithm with a linear time complexity in the number of parameters. We show that when SRNNs are trained with our algorithm, they provide very similar regression performance with the LSTMs in two to three times shorter training time. We provide strong theoretical analysis to support our experimental results by providing regret bounds on the convergence rate of our algorithm. Through an extensive set of experiments, we verify our theoretical work and demonstrate significant performance improvements of our algorithm with respect to LSTMs and the other state-of-the-art learning models.

On the Divergence of Decentralized Nonconvex Optimization

We study a generic class of decentralized algorithms in which $N$ agents jointly optimize the non-convex objective $f(u):=1/N\sum_{i=1}^{N}f_i(u)$, while only communicating with their neighbors. This class of problems has become popular in modeling many signal processing and machine learning applications, and many efficient algorithms have been proposed. However, by constructing some counter-examples, we show that when certain local Lipschitz conditions (LLC) on the local function gradient $\nabla f_i$'s are not satisfied, most of the existing decentralized algorithms diverge, even if the global Lipschitz condition (GLC) is satisfied, where the sum function $f$ has Lipschitz gradient. This observation raises an important open question: How to design decentralized algorithms when the LLC, or even the GLC, is not satisfied? To address the above question, we design a first-order algorithm called Multi-stage gradient tracking algorithm (MAGENTA), which is capable of computing stationary solutions with neither the LLC nor the GLC. In particular, we show that the proposed algorithm converges sublinearly to certain $\epsilon$-stationary solution, where the precise rate depends on various algorithmic and problem parameters. In particular, if the local function $f_i$'s are $Q$th order polynomials, then the rate becomes $\mathcal{O}(1/\epsilon^{Q-1})$. Such a rate is tight for the special case of $Q=2$ where each $f_i$ satisfies LLC. To our knowledge, this is the first attempt that studies decentralized non-convex optimization problems with neither the LLC nor the GLC.

Nonlinear Decoration Control Based on Perturbation of Ship Longitudinal Motion Model

This paper proposes a nonlinear decoration control based on perturbation of the ship longitudinal Motion Model to reduce energy loss by achieving a good anti-pitch control effect. Even when the model is perturbed, it can continue to maintain a good robust performance and energy-saving effect for multiple-input multiple-output (MIMO) unstable systems. This proposed method uses the root locus shaping technology and the weighting matrix of mathematical tools to pull the root locus of the transfer function mathematical model with unstable zero-pole and zero-pole processes to the left half-plane of the teaching practice ship "YUKUN" of Dalian Maritime University used as the research object. The closed-loop gain shaping algorithm used to design the controller is then introduced, followed by the introduction of the first-order closed-loop gain shaping algorithm to design the robust controller with the shaped model. Finally, the nonlinear power function to decorate and improve the robust controller is presented. The simulation results show that when the model is perturbed, the nonlinear decoration control improves the control effects of the ship's heave and pitch, reduces control energy, and is critical for the ship's smooth and efficient navigation.

New First-Order Algorithms for Stochastic Variational Inequalities

In this paper, we propose two new solution schemes to solve the stochastic strongly monotone variational inequality (VI) problems: the stochastic extra-point solution scheme and the stochastic extra-momentum solution scheme. The first one is a general scheme based on updating the iterative sequence and an auxiliary extra-point sequence. In the case of a deterministic VI model, this approach includes several state-of-the-art first-order methods as its special cases. The second scheme combines two momentum-based directions: the so-called heavy-ball direction and the optimism direction, where only one projection per iteration is required in its updating process. We show that if the variance of the stochastic oracle is appropriately controlled, then both schemes can be made to achieve optimal iteration complexity of $\mathcal{O}\left(\kappa\ln\left(\frac{1}{\epsilon}\right)\right)$ to reach an $\epsilon$-solution for a strongly monotone VI problem with condition number $\kappa$. As a specific application to stochastic VI, we demonstrate how to incorporate a zeroth-order approach for solving stochastic minimax saddle-point problems in our schemes, where only noisy and biased samples of the objective can be obtained, with a total sample complexity of $\mathcal{O}\left(\frac{\kappa}{\epsilon}\right)$.

Joint optimization of scoring and thresholding models for online multi-label classification

Existing online multi-label classification works cannot well handle the online label thresholding problem and lack regret analysis for their online algorithms. This paper proposes a novel framework of joint optimization of scoring and thresholding models for online multi-label classification, with the aim to overcome the above drawbacks. The key feature of our framework is that both scoring and thresholding models are included as important components of the online multi-label classifier and are incorporated into one online optimization problem. Based on this framework, we present two adaptive label thresholding algorithms and two fixed thresholding algorithms. For each type of algorithms, a first-order method and a second-order one are provided for updating the online multi-label classifier. Both methods enjoy a closed-form update. Our proposed algorithms are proved to achieve a sub-linear regret. Using Mercer kernels, two first-order algorithms can be extended to handle nonlinear multi-label prediction tasks. Experiments show the advantage of the adaptive and the fixed thresholding algorithms, in terms of various multi-label performance metrics.

Riemannian Conjugate Gradient Methods: General Framework and Specific Algorithms with Convergence Analyses

This paper proposes a novel general framework of Riemannian conjugate gradient methods, that is, conjugate gradient methods on Riemannian manifolds. The conjugate gradient methods are important first-order optimization algorithms both in Euclidean spaces and on Riemannian manifolds. While various types of conjugate gradient methods are studied in Euclidean spaces, there have been fewer studies on those on Riemannian manifolds. In each iteration of the Riemannian conjugate gradient methods, the previous search direction must be transported to the current tangent space so that it can be added to the negative gradient of the objective function at the current point. There are several approaches to transport a tangent vector to another tangent space. Therefore, there are more variants of the Riemannian conjugate gradient methods than the Euclidean case. In order to investigate them in more detail, the proposed framework unifies the existing Riemannian conjugate gradient methods such as ones utilizing a vector transport or inverse retraction and also develops other methods that have not been covered in previous studies. Furthermore, sufficient conditions for the convergence of a class of algorithms in the proposed framework are clarified. Moreover, the global convergence properties of several specific types of algorithms are extensively analyzed. The analyses provide the theoretical results for some algorithms in a more general setting than the existing studies and completely new developments for the other algorithms. Numerical experiments are performed to confirm the validity of the theoretical results. The results also compare the performances of several specific algorithms in the proposed framework.

Generalized damped Newton algorithms in nonsmooth optimization via second-order subdifferentials

The paper proposes and develops new globally convergent algorithms of the generalized damped Newton type for solving important classes of nonsmooth optimization problems. These algorithms are based on the theory and calculations of second-order subdifferentials of nonsmooth functions with employing the machinery of second-order variational analysis and generalized differentiation. First we develop a globally superlinearly convergent damped Newton-type algorithm for the class of continuously differentiable functions with Lipschitzian gradients, which are nonsmooth of second order. Then we design such a globally convergent algorithm to solve a structured class of nonsmooth quadratic composite problems with extended-real-valued cost functions, which typically arise in machine learning and statistics. Finally, we present the results of numerical experiments and compare the performance of our main algorithm applied to an important class of Lasso problems with those achieved by other first-order and second-order optimization algorithms.

Biomechanical Trajectory Optimization of Human Sit-to-Stand Motion With Stochastic Motion Planning Framework

Trajectory optimization has been an important approach in biomechanics for the analysis and prediction of the limb movement. Such approaches have paved the way for the motion planning of biped and quadruped robots as well. Most of these methods are deterministic, utilizing first-order iterative gradient-based algorithms incorporating the constrained differentiable objective functions. However, the limitation of prevailing methods concerning differentiability hinders the implementation of non-differentiable objective functions such as metabolic energy expenditure (MEE) function, which is highly relevant for physiological systems and can even be implemented across the muscular space. This paper consolidates the implementation of the prevalent direct collocation-based optimal control method with the stochastic trajectory optimization method based on Policy Improvement with Path Integral (PI2) for comprehending the human sit-to-stand (STS) motion. PI2 method, which utilizes reinforcement learning of Dynamic Movement Primitive (DMP) to learn a goal-based trajectory is implemented and validated by comparing with the experimental result in joint-space and muscle-space.

On the Nonexpansive Operators Based on Arbitrary Metric: A Degenerate Analysis

We in this paper study the nonexpansive operators equipped with arbitrary metric and investigate the connections between firm nonexpansiveness, cocoerciveness and averagedness. The convergence of the associated fixed-point iterations is discussed with particular focus on the case of degenerate metric, since the degeneracy is often encountered when reformulating many existing first-order operator splitting algorithms as a metric resolvent. This work paves a way for analyzing the generalized proximal point algorithm with a non-trivial relaxation step and degenerate metric.