Stochastic Approximation Method Research Articles

Implementation of unknown parameter estimation procedure for hybrid and discrete non‐linear systems

AbstractThe application of the hybrid extended Kalman filter (HEKF), hybrid unscented Kalman filter (HUKF), hybrid particle filter (HPF), and hybrid extended Kalman particle filter (HEKPF) is discussed for hybrid non‐linear filter problems, when prediction equations are continuous‐time and the update equations are discrete‐time, and also the discrete extended Kalman filter (DEKF), discrete unscented Kalman filter (DUKF), discrete particle filter (DPF), and discrete extended Kalman particle filter (DEKPF) for discrete‐time non‐linear filter problems, when prediction equations and update equations are discrete‐time. In order to assess the performance of the filters, the authors consider the non‐linear dynamics for a re‐entry vehicle. The filters are used in two hybrid and discrete states to estimate the position, velocity, and drag parameter associated with the re‐entry vehicle. Theoretical topics concerning estimating the drag parameter of a vehicle in re‐entry phase have been dealt with. Drag parameter estimation is carried out using a combination of hybrid filters and discrete filters as an effective estimator and fixed value, forgetting factor, and Robbins‐Monro stochastic approximation methods as the noise covariance matrix adjuster of the parameter.

Optimization of nonlinear turbulence in stellarators

We present new stellarator equilibria that have been optimized for reduced turbulent transport using nonlinear gyrokinetic simulations within the optimization loop. The optimization routine involves coupling the pseudo-spectral GPU-native gyrokinetic code GX with the stellarator equilibrium and optimization code DESC. Since using GX allows for fast nonlinear simulations, we directly optimize for reduced nonlinear heat fluxes. To handle the noisy heat flux traces returned by these simulations, we employ the simultaneous perturbation stochastic approximation (SPSA) method that only uses two objective function evaluations for a simple estimate of the gradient. We show several examples that optimize for both reduced heat fluxes and good quasi-symmetry as a proxy for low neoclassical transport. Finally, we run full transport simulations using the T3D stellarator transport code to evaluate the changes in the macroscopic profiles.

A 0–1 linear relaxation and stochastic approximation method for real-time pricing in a smart grid with non-convex constraints

In this article, real-time pricing (RTP) for a smart grid with complicated non-convex and/or non-smooth constraints is explored. First, piecewise linear functions are adopted to approximate any kinds of utility function and RTP is formulated into bilevel non-convex programming. Then, a 0–1 relaxation method is used to linearize sparse constraints and piecewise linear utility functions, to obtain mixed integer linear programming. Finally, a distributed simultaneous perturbation stochastic approximation (DSPSA) method is designed to solve the bilevel non-convex programming. The RTP strategy is generalizable because sparse constraints and non-differentiable and/or non-concave utility functions are widespread in practice, and the piecewise linear functions approximation method is applicable to all kinds of utility function, not just to non-smooth and/or non-concave utility functions. In addition, the 0–1 linear relaxation and stochastic approximation method make the bilevel non-convex programming easily solvable by DSPSA, which converges rapidly while maintaining the high accuracy of solutions.

Distributed Localization for Multi-Agent Systems With Random Noise Based on Iterative Learning.

This article is concerned with the real-time localization problem for the dynamic multi-agent systems with measurement and communication noises under directed graphs. The barycentric coordinates are introduced to describe the relative position between agents. A novel robust distributed localization estimation algorithm based on iterative learning is proposed. The relative-distance unbiased estimator constructed from the historical iterative information is used to suppress the measurement noise. The designed stochastic approximation method with two iterative-varying gains is used to inhibit the communication noise. Under the zero-mean and independent distributed conditions on the measurement and communication noises, the asymptotic convergence of the proposed methods is derived. The numerical simulation and the QBot-2e robot experiment are conducted to test and verify the effectiveness and the practicability of the proposed methods.

A Lyapunov Theory for Finite-Sample Guarantees of Markovian Stochastic Approximation

The stochastic approximation (SA) method stands as the foundational mathematical tool for modern large-scale optimization and machine learning. Therefore, gaining a theoretical understanding of SA algorithms is of fundamental interest. In their paper titled “A Lyapunov Theory for Finite-Sample Guarantees of Markovian Stochastic Approximation,” Chen et al. present a unified Lyapunov framework for the finite-sample analysis of a Markovian SA algorithm under a contractive operator with respect to an arbitrary norm. The key novelty lies in the construction of a smooth Lyapunov function called the generalized Moreau envelope. The authors demonstrate the effectiveness of their SA results in the context of reinforcement learning (RL), specifically through popular algorithms such as variants of temporal difference (TD) learning and Q-learning. As byproducts, the results provide theoretical insights into the efficiency of bootstrapping in TD learning with eligibility traces and the bias-variance tradeoff in off-policy learning.

Approximate Kalman filtering by both M-robustified dynamic stochastic approximation and statistical linearization methods

The problem of designing a robustified Kalman filtering technique, insensitive to spiky observations, or outliers, contaminating the Gaussian observations has been presented in the paper. Firstly, a class of M-robustified dynamic stochastic approximation algorithms is derived by minimizing at each stage a specific time-varying M-robust performance index, that is, general for a family of algorithms to be considered. The gain matrix of a particular algorithm is calculated at each stage by minimizing an additional criterion of the approximate minimum variance type, with the aid of the statistical linearization method. By combining the proposed M-robust estimator with the one-stage optimal prediction, in the minimum mean-square error sense, a new statistically linearized M-robustified Kalman filtering technique has been derived. Two simple practical versions of the proposed M-robustified state estimator are derived by approximating the mean-square optimal statistical linearization coefficient with the fixed and the time-varying factors. The feasibility of the approaches has been analysed by the simulations, using a manoeuvring target radar tracking example, and the real data, related to an object video tracking using short-wave infrared camera.

Threshold Optimized CSWPL Behavioral Model for RF Power Transistors Based on Particle Swarm Algorithm

In this letter, the particle swarm optimization (PSO) is employed to optimize the threshold values of the canonical section-wise piecewise linear (CSWPL) function-based behavioral model of power transistors. The effectiveness of the proposed method is validated through measurements carried out on a 10-W GaN power transistor produced by Wolfspeed. Compared with the existing simultaneous perturbation stochastic approximation (SPSA) method, the developed modeling technique not only provides superior prediction performance across different input power levels but also finds the optimal thresholds much more efficiently.

Finite-Time Convergence Rates of Decentralized Stochastic Approximation With Applications in Multi-Agent and Multi-Task Learning

We study a decentralized variant of stochastic approximation, a data-driven approach for finding the root of an operator under noisy measurements. A network of agents, each with its own operator and data observations, cooperatively find the fixed point of the aggregate operator over a decentralized communication graph. Our main contribution is to provide a finite-time analysis of this decentralized stochastic approximation method when the data observed at each agent are sampled from a Markov process; this lack of independence makes the iterates biased and (potentially) unbounded. Under fairly standard assumptions, we show that the convergence rate of the proposed method is essentially the same as if the samples were independent, differing only by a log factor that accounts for the mixing time of the Markov processes. The key idea in our analysis is to introduce a novel Lyapunov-Razumikhin function, motivated by the one used in analyzing the stability of delayed ordinary differential equations. We also discuss applications of the proposed method on a number of interesting learning problems in multi-agent systems.

Archive-based coronavirus herd immunity algorithm for optimizing weights in neural networks.

The success of the supervised learning process for feedforward neural networks, especially multilayer perceptron neural network (MLP), depends on the suitable configuration of its controlling parameters (i.e., weights and biases). Normally, the gradient descent method is used to find the optimal values of weights and biases. The gradient descent method suffers from the local optimal trap and slow convergence. Therefore, stochastic approximation methods such as metaheuristics are invited. Coronavirus herd immunity optimizer (CHIO) is a recent metaheuristic human-based algorithm stemmed from the herd immunity mechanism as a way to treat the spread of the coronavirus pandemic. In this paper, an external archive strategy is proposed and applied to direct the population closer to more promising search regions. The external archive is implemented during the algorithm evolution, and it saves the best solutions to be used later. This enhanced version of CHIO is called ACHIO. The algorithm is utilized in the training process of MLP to find its optimal controlling parameters thus empowering their classification accuracy. The proposed approach is evaluated using 15 classification datasets with classes ranging between 2 to 10. The performance of ACHIO is compared against six well-known swarm intelligence algorithms and the original CHIO in terms of classification accuracy. Interestingly, ACHIO is able to produce accurate results that excel other comparative methods in ten out of the fifteen classification datasets and very competitive results for others.

Online data‐driven control of variable speed wind turbines using the simultaneous perturbation stochastic approximation approach

AbstractThis article presents a data‐driven control algorithm for a variable‐speed wind turbine in the partial load region. The controller is designed and optimized online to capture maximum energy from the wind. The online optimization is based on the simultaneous perturbation stochastic approximation method (SPSA) that is data‐driven, has a low computational load, and converges rapidly. In the control structure, the generator torque control is activated, and the aerodynamical models and the wind speed value are not required. A suitable cost function is defined, and a time window optimization problem is solved via the SPSA algorithm. The efficiency of the proposed method is evaluated on a 4.8 MW wind turbine benchmark.

Block coordinate type methods for optimization and learning

We study nonconvex (composite) optimization and learning problems where the decision variables can be split into blocks of variables. Random block projection is a popular technique to handle this kind of problem for its remarkable reduction of the computational cost from the projection. This powerful method has not been well proposed for the situation that first-order information is prohibited and only zeroth-order information is available. In this paper, we propose to develop different classes of zeroth-order stochastic block coordinate type methods. Zeroth-order block coordinate descent (ZS-BCD) is proposed for solving unconstrained nonconvex optimization problem. For composite optimization, we establish the zeroth-order stochastic block mirror descent (ZS-BMD) and its associated two-phase method to achieve the complexity bound for the composite optimization problem. Furthermore, we also establish zeroth-order stochastic block coordinate conditional gradient (ZS-BCCG) method for nonconvex (composite) optimization. By implementing ZS-BCCG method, in each iteration, only (approximate) linear programming subproblem needs to be solved on a random block instead of a rather costly projection subproblem on the whole decision space, in contrast to the existing traditional stochastic approximation methods. In what follows, an approximate ZS-BCCG method and corresponding two-phase ZS-BCCG method are proposed. This is also the first time that a two-phase BCCG method has been developed to carry out the complexity analysis of nonconvex composite optimization problem.

Discrete Stochastic Optimization for Public Health Interventions with Constraints

Many public health threats exist, motivating the need to find optimal intervention strategies. Given the stochastic nature of the threats (e.g., the spread of pandemic influenza, the occurrence of drug overdoses, and the prevalence of alcohol-related threats), deterministic optimization approaches may be inappropriate. In this paper, we implement a stochastic optimization method to address aspects of the 2009 H1N1 and the COVID-19 pandemics, with the spread of disease modeled by the open-source Monte Carlo simulations, FluTE, and Covasim, respectively. Without testing every possible option, the objective of the optimization is to determine the best combination of intervention strategies so as to result in minimal economic loss to society. To reach our objective, this application-oriented paper uses the discrete simultaneous perturbation stochastic approximation method (DSPSA), a recursive simulation-based optimization algorithm, to update the input parameters in the disease simulation software so that the output iteratively approaches minimal economic loss. Assuming that the simulation models for the spread of disease (FluTE for H1N1 and Covasim for COVID-19 in our case) are accurate representations for the population being studied, the simulation-based strategy we present provides decision makers a powerful tool to mitigate potential human and economic losses from any epidemic. The basic approach is also applicable in other public health problems, such as opioid abuse and drunk driving.

A SIEVE STOCHASTIC GRADIENT DESCENT ESTIMATOR FOR ONLINE NONPARAMETRIC REGRESSION IN SOBOLEV ELLIPSOIDS.

The goal of regression is to recover an unknown underlying function that best links a set of predictors to an outcome from noisy observations. in nonparametric regression, one assumes that the regression function belongs to a pre-specified infinite-dimensional function space (the hypothesis space). in the online setting, when the observations come in a stream, it is computationally-preferable to iteratively update an estimate rather than refitting an entire model repeatedly. inspired by nonparametric sieve estimation and stochastic approximation methods, we propose a sieve stochastic gradient descent estimator (Sieve-SGD) when the hypothesis space is a Sobolev ellipsoid. We show that Sieve-SGD has rate-optimal mean squared error (MSE) under a set of simple and direct conditions. The proposed estimator can be constructed with a low computational (time and space) expense: We also formally show that Sieve-SGD requires almost minimal memory usage among all statistically rate-optimal estimators.

Spectrum Sharing Among Multiple-Seller and Multiple-Buyer Operators of a Mobile Network: A Stochastic Geometry Approach

Sharing the spectrum among mobile network operators (MNOs) is a promising approach to improve the spectrum utilization and to increase the monetary income of MNOs. In this paper, we model a nonorthogonal spectrum sharing system for a large-scale cellular network where multiple seller MNOs lease their licensed sub-bands to several buyer MNOs. We first analyze the per-user expected rate and the per-MNO expected profit using stochastic geometry. Then, we formulate the joint problem of power control and licensed sub-band sharing to maximize the expected profit of all MNOs as a multiobjective optimization problem (MOOP) under the users' quality of service requirement and the nonnegative return on investment constraints. To transform the MOOP into a single objective form, we use a combination of the $\epsilon$-constraint and weighted sum methods. However, the transformed problem is nonconvex because of the presence of binary variables and nonconvex rate functions in the objective function and constraints. We address this problem by using a penalty function and approximating the nonconvex rate functions by a constrained stochastic successive convex approximation method. Finally, the numerical results show the correctness and performance of the proposed algorithm under various conditions.

Stochastic Approximation Methods for the Two-Stage Stochastic Linear Complementarity Problem

The two-stage stochastic linear complementarity problem (TSLCP), which can be regarded as a special and important reformulation of two-stage stochastic linear programming, has arisen in various fields, such as stochastic programming, game theory, traffic equilibrium, and theoretical economics. Considerable effort has been devoted to designing numerical methods for solving TSLCPs. A popular approach is to integrate the progressive hedging algorithm (PHA) as a sub-algorithm into a discretization framework. In this paper, aiming to solve large-scale TSLCPs, we propose two kinds of stochastic methods: the stochastic approximation method based on projection (SAP) and the dynamic sampling SAP (DS-SAP), both of which offering more direct and improved control of the computational costs of the involved subproblems, especially compared with the PHA. In particular, the linear complementarity subproblems are solved inexactly during each iteration, and the convergence analysis of both SAP and DS-SAP with an inexactness criterion is presented. Moreover, numerical implementations and practical applications demonstrate the efficiency of our proposed methods.

SlimTrain---A Stochastic Approximation Method for Training Separable Deep Neural Networks

Deep neural networks (DNNs) have shown their success as high-dimensional function approximators in many applications; however, training DNNs can be challenging in general. DNN training is commonly phrased as a stochastic optimization problem whose challenges include nonconvexity, nonsmoothness, insufficient regularization, and complicated data distributions. Hence, the performance of DNNs on a given task depends crucially on tuning hyperparameters, especially learning rates and regularization parameters. In the absence of theoretical guidelines or prior experience on similar tasks, this requires solving a series of repeated training problems which can be time-consuming and demanding on computational resources. This can limit the applicability of DNNs to problems with nonstandard, complex, and scarce datasets, e.g., those arising in many scientific applications. To remedy the challenges of DNN training, we propose \tt slimTrain, a stochastic optimization method for training DNNs with reduced sensitivity to the choice of hyperparameters and fast initial convergence. The central idea of slimTrain is to exploit the separability inherent in many DNN architectures; that is, we separate the DNN into a nonlinear feature extractor followed by a linear model. This separability allows us to leverage recent advances made for solving large-scale, linear, ill-posed inverse problems. Crucially, for the linear weights, \tt slimTrain does not require a learning rate and automatically adapts the regularization parameter. In our numerical experiments using function approximation tasks arising in surrogate modeling and dimensionality reduction, slimTrain outperforms existing DNN training methods with the recommended hyperparameter settings and reduces the sensitivity of DNN training to the remaining hyperparameters. Since our method operates on mini-batches, its computational overhead per iteration is modest and savings can be realized by reducing the number of iterations (due to quicker initial convergence) or the number of training problems that need to be solved to identify effective hyperparameters.

A Stochastic Approximation Method for Simulation-Based Quantile Optimization

We present a gradient-based algorithm for solving a class of simulation optimization problems in which the objective function is the quantile of a simulation output random variable. In contrast with existing quantile (quantile derivative) estimation techniques, which aim to eliminate the estimator bias by gradually increasing the simulation sample size, our algorithm incorporates a novel recursive procedure that only requires a single simulation sample at each step to simultaneously obtain quantile and quantile derivative estimators that are asymptotically unbiased. We show that these estimators, when coupled with the standard gradient descent method, lead to a multitime-scale stochastic approximation type of algorithm that converges to an optimal quantile value with probability one. In our numerical experiments, the proposed algorithm is applied to optimal investment portfolio problems, resulting in new solutions that complement those obtained under the classical Markowitz mean-variance framework. History: Accepted by Alice E. Smith, Editor-in-Chief; Bruno Tuffin, Area Editor for Simulation. Funding: The work of Y. Peng was supported in part by the National Natural Science Foundation of China (NSFC) [Grants 72022001, 92146003, and 71901003], and by the Key Research and Development Programof Beijing Municipal Science and Technology Commission. Supplemental Material: The e-companion is available at https://doi.org/10.1287/ijoc.2022.1214 .

A stochastic approximation method for convex programming with many semidefinite constraints

In this paper, we consider a type of semidefinite programming problem (MSDP), which involves many (not necessarily finite) of semidefinite constraints. MSDP can be established in a wide range of applications, including covering ellipsoids problem and truss topology design. We propose a random method based on a stochastic approximation technique for solving MSDP, without calculating and storing the multiplier. Under mild conditions, we establish the almost sure convergence and expected convergence rates of the proposed method. A variety of simulation experiments are carried out to support our theoretical results.

Probabilistic programming with stochastic variational message passing

Stochastic approximation methods for variational inference have recently gained popularity in the probabilistic programming community since these methods are amenable to automation and allow online, scalable, and universal approximate Bayesian inference. Unfortunately, common Probabilistic Programming Languages (PPLs) with stochastic approximation engines lack the efficiency of message passing-based inference algorithms with deterministic update rules such as Belief Propagation (BP) and Variational Message Passing (VMP). Still, Stochastic Variational Inference (SVI) and Conjugate-Computation Variational Inference (CVI) provide principled methods to integrate fast deterministic inference techniques with broadly applicable stochastic approximate inference. Unfortunately, implementation of SVI and CVI necessitates manually driven variational update rules, which does not yet exist in most PPLs. In this paper, we cast SVI and CVI explicitly in a message passing-based inference context. We provide an implementation for SVI and CVI in ForneyLab, which is an automated message passing-based probabilistic programming package in the open source Julia language. Through a number of experiments, we demonstrate how SVI and CVI extends the automated inference capabilities of message passing-based probabilistic programming.

SimSR: Simple Distance-Based State Representations for Deep Reinforcement Learning

This work explores how to learn robust and generalizable state representation from image-based observations with deep reinforcement learning methods. Addressing the computational complexity, stringent assumptions and representation collapse challenges in existing work of bisimulation metric, we devise Simple State Representation (SimSR) operator. SimSR enables us to design a stochastic approximation method that can practically learn the mapping functions (encoders) from observations to latent representation space. In addition to the theoretical analysis and comparison with the existing work, we experimented and compared our work with recent state-of-the-art solutions in visual MuJoCo tasks. The results shows that our model generally achieves better performance and has better robustness and good generalization.