Stochastic Approximation Algorithm Research Articles

A Unified Lyapunov Framework for Finite-Sample Analysis of Reinforcement Learning Algorithms

Reinforcement learning (RL) is a paradigm where an agent learns to accomplish tasks by interacting with the environment, similar to how humans learn. RL is therefore viewed as a promising approach to achieve artificial intelligence, as evidenced by the remarkable empirical successes. However, many RL algorithms are theoretically not well-understood, especially in the setting where function approximation and off-policy sampling are employed. My thesis [1] aims at developing thorough theoretical understanding to the performance of various RL algorithms through finite-sample analysis.Since most of the RL algorithms are essentially stochastic approximation (SA) algorithms for solving variants of the Bellman equation, the first part of thesis is dedicated to the analysis of general SA involving a contraction operator, and under Markovian noise. We develop a Lyapunov approach where we construct a novel Lyapunov function called the generaled Moreau envelope. The results on SA enable us to establish finite-sample bounds of various RL algorithms in the tabular setting (cf. Part II of the thesis) and when using function approximation (cf. Part III of the thesis), which in turn provide theoretical insights to several important problems in the RL community, such as the efficiency of bootstrapping, the bias-variance trade-off in off-policy learning, and the stability of off-policy control.The main body of this document provides an overview of the contributions of my thesis.

Calibrating stochastic traffic simulation models for safety and operational measures based on vehicle conflict distributions obtained from aerial and traffic camera videos

Proper calibration process is of considerable importance for traffic safety evaluations using simulation models. Allowing for a pure with and without comparison under identical circumstances that is not directly testable in the field, microsimulation-based approach has drawn considerable attention for the performance evaluation of emerging technologies, such as connected vehicle (CV) safety applications. Different from the traditional approaches to evaluate mobility impacts, safety evaluations of such applications demand the simulation models to be well calibrated to match real-world safety conditions. This paper proposes a novel calibration framework which combines traffic conflict techniques and multi-objective stochastic optimization so that the operational and safety measures can be calibrated simultaneously. The conflict distribution of different severity levels categorized by time-to-collision (TTC) is applied as the safety performance measure. Simultaneous perturbation stochastic approximation (SPSA) algorithm, which can efficiently approximate the gradient of the multi-objective stochastic loss function, is used for model parameters optimization that minimizes the total simulation error of both operational and safety performance measures. The proposed calibration methodology is implemented using an open-source software SUMO on a simulation network of the Flatbush Avenue corridor in Brooklyn, NY. 17 key parameters are calibrated using the SPSA algorithm and are compared with the real-world traffic conflicts extracted using vehicle trajectories from 14 h’ high-resolution aerial and traffic surveillance videos. Representative days are identified to create variation envelopes for performance measures. Four acceptability criteria, including control for time-variant outliers and inliers, bounded dynamic absolute and system errors are adopted for results analysis. The results show that the calibrated parameters can significantly improve the performance of the simulation model to represent real-world safety conditions (i.e., traffic conflicts) as well as operational conditions. The case study also demonstrates the usefulness of aerial imagery and the applicability of the proposed model calibration framework, so the calibrated model can be used to evaluate the safety benefits of CV applications more accurately.

Mixed-integer simulation optimization for multi-echelon inventory problems with lost sales

We propose a mixed-integer simulation optimization framework for solving multi-echelon inventory problems with lost sales. We want to seek optimal settings of the order-up-to levels and the review intervals for warehouse and retailers. The aim is to minimize the total expected costs including the inventory holding cost, the ordering cost and the penalty cost. The proposed optimization method represents a complementary combination of ranking-and-selection procedures and stochastic-approximation algorithms for both integer-valued and real-valued variables. We provide a proof for the finite-time statistical validity of the developed algorithm. We also discuss the convergence conditions for the asymptotic optimality of our algorithm. The algorithmic performance is examined with experiments under different parameter settings and stopping conditions. During the experiments, our algorithm performs favorably in comparison to the popular Arena optimization tool, OptQuest.

On stochastic accelerated gradient with convergence rate

Abstract This article studies the regression learning problem from given sample data by using stochastic approximation (SA) type algorithm, namely, the accelerated SA. We focus on problems without strong convexity, for which all well-known algorithms achieve a convergence rate for function values of O ( 1 / n ) O\left(1\hspace{0.1em}\text{/}\hspace{0.1em}n) . We consider and analyze accelerated SA algorithm that achieves a rate of O ( 1 / n ) O\left(1\hspace{0.1em}\text{/}\hspace{0.1em}n) for classical least-square regression and logistic regression problems, respectively. Comparing with the well-known results, we only need fewer conditions to obtain the tight convergence rate for least-square regression and logistic regression problems.

Antenna array calibration methods based on simultaneous perturbation

Antenna arrays have gained significant interest in millimetre-wave communication systems as an enabling technology to achieve higher capacity and mitigate the high propagation loss. Such arrays with a large bandwidth need to be efficiently calibrated to maximise their performance. An antenna array calibration method based on a stochastic approximation algorithm and simultaneous perturbation has been developed and the procedures to implement it in both frequency and time domains have been presented. The approaches to define objective functions and establish gradient approximations to fulfill a successful convergence for acquiring calibration coefficients in both domains have been explored. In the time domain implementation, only a fraction of the measurement time was required to calibrate an antenna array of ultrawide bandwidth compared with other methods using a perturbation technique. The effectiveness of the proposed method has been validated via numerical experiments in both domains.

Finite-sample analysis of nonlinear stochastic approximation with applications in reinforcement learning

Motivated by applications in reinforcement learning (RL), we study a nonlinear stochastic approximation (SA) algorithm under Markovian noise, and establish its finite-sample convergence bounds under various stepsizes. Specifically, we show that when using constant stepsize (i.e., αk≡α), the algorithm achieves exponential fast convergence to a neighborhood (with radius O(αlog(1/α))) around the desired limit point. When using diminishing stepsizes with appropriate decay rate, the algorithm converges with rate O(log(k)/k). Our proof is based on Lyapunov drift arguments, and to handle the Markovian noise, we exploit the fast mixing of the underlying Markov chain. To demonstrate the generality of our theoretical results on Markovian SA, we use it to derive the finite-sample bounds of the popular Q-learning algorithm with linear function approximation, under a condition on the behavior policy. Importantly, we do not need to make the assumption that the samples are i.i.d., and do not require an artificial projection step in the algorithm. Numerical simulations corroborate our theoretical results.

Convergence of the Kiefer–Wolfowitz algorithm in the presence of discontinuities

AbstractIn this paper we estimate the expected error of a stochastic approximation algorithm where the maximum of a function is found using finite differences of a stochastic representation of that function. An error estimate of the order $n^{-1/5}$ for the nth iteration is achieved using suitable parameters. The novelty with respect to previous studies is that we allow the stochastic representation to be discontinuous and to consist of possibly dependent random variables (satisfying a mixing condition).

Mixed-Timescale Deep-Unfolding for Joint Channel Estimation and Hybrid Beamforming

In massive multiple-input multiple-output (MIMO) systems, hybrid analog-digital beamforming is an essential technique for exploiting the potential array gain without using a dedicated radio frequency chain for each antenna. However, due to the large number of antennas, the conventional channel estimation and hybrid beamforming algorithms generally require high computational complexity and signaling overhead. In this work, we propose an end-to-end deep-unfolding neural network (NN) joint channel estimation and hybrid beamforming (JCEHB) algorithm to maximize the system sum rate in time-division duplex (TDD) massive MIMO. Specifically, the recursive least-squares (RLS) algorithm and stochastic successive convex approximation (SSCA) algorithm are unfolded for channel estimation and hybrid beamforming, respectively. In order to reduce the signaling overhead, we consider a mixed-timescale hybrid beamforming scheme, where the analog beamforming matrices are optimized based on the channel state information (CSI) statistics offline, while the digital beamforming matrices are designed at each time slot based on the estimated low-dimensional equivalent CSI matrices. We jointly train the analog beamformers together with the trainable parameters of the RLS and SSCA induced deep-unfolding NNs based on the CSI statistics offline. During data transmission, we estimate the low-dimensional equivalent CSI by the RLS induced deep-unfolding NN and update the digital beamformers. In addition, we propose a mixed-timescale deep-unfolding NN where the analog beamformers are optimized online, and extend the framework to frequency-division duplex (FDD) systems where channel feedback is considered. Simulation results show that the proposed algorithm can significantly outperform conventional algorithms with reduced computational complexity and signaling overhead.

Adaptive Singular Evolutive Interpolated Kalman filter and its application to data assimilation in 2D Water pollution model

This study promotes a new algorithm for estimating the water pollution propagation with the primary goal of providing more reliable and high quality estimates to decision makers. To date, the widely used variational method suffers from the large computational burden which limits its application in practice. Moreover, this method, considering the initial state as a control variable, is very sensitive in specifying initial error, especially for unstable dynamical systems. The adaptive filter (AF), proposed in this paper, is aimed at overcoming these two drawbacks in the variational method: by its nature, the AF is sequential (no large batch assimilation window used) and stable even for unstable dynamics, with the gain parameters as control variables. The AF, developed in this paper, is an adaptive version of the Singular Evolutive Interpolated Kalman Filter (SEIKF). One of the new versions of this AF is that it uses a time-varying structure of the gain of SEIKF. To deal with the uncertainty of the system parameters and of the noise covariance, the proposed adaptive SEIKF (ASEIKF) makes use of the covariance of reduced rank iterated during assimilation process and of some pertinent gain parameters tuned adaptively to yield the minimum prediction error for the system output. The computational burden in implementation of the ASEIKF is reduced drastically due to applying the optimization tool known as a simultaneous perturbation stochastic approximation algorithm, which requires only two integrations of the numerical model. No iterative loop is required at each assimilation instant as usually happens with the standard gradient descent optimization algorithms. Data assimilation experiment, carried out by the SEIKF and ASEIKF, is implemented for the Thanh Nhan Lake in Hanoi and the performance comparison between the ASEIKF and SEIKF is given to show the high effectiveness of the proposed ASEIKF.

Concentration of Contractive Stochastic Approximation and Reinforcement Learning

Using a martingale concentration inequality, concentration bounds “from time n0 on” are derived for stochastic approximation algorithms with contractive maps and both martingale difference and Markov noises. These are applied to reinforcement learning algorithms, in particular to asynchronous Q-learning and TD(0). Funding: V. S. Borkar was supported in part by a S. S. Bhatnagar Fellowship from the Council of Scientific and Industrial Research, Government of India.

Incentive design and profit sharing in multi-modal transportation networks

We consider the situation where multiple transportation service providers cooperate to offer an integrated multi-modal platform to enhance the convenience to the passengers through ease in multi-modal journey planning, payment, and first and last mile connectivity. This market structure allows the multi-modal platform to coordinate profits across modes and also provide incentives to the passengers. Accordingly, in this paper, we use cooperative game theory coupled with the hyperpath-based stochastic user equilibrium framework to study such a market. We assume that the platform sets incentives (price discount or excess charge on passengers) along every edge in the transportation network. We derive the continuity and monotonicity properties of the equilibrium flow with respect to the incentives along every edge. The optimal incentives that maximize the profit of the platform are obtained through a two time-scale stochastic approximation algorithm. We use the asymmetric Nash bargaining solution to design a fair profit sharing scheme among the service providers. We show that the profit for each service provider increases after cooperation on such a platform. Finally, we complement the theoretical results through two numerical simulations.

Stochastic Optimization of the CPL-Function-Based Model for RF Power Transistors

In this article, the simultaneous perturbation stochastic approximation (SPSA) algorithm is proposed to optimize the threshold of the canonical piecewise linear (CPL) function-based models for radio frequency (RF) power transistors. Compared with the existing standard CPL-function-based model, the proposed optimization technique can greatly improve model performance. Two different CPL-based models, the CPL Y/quadratic Y (QY) model and the canonical section-wise piecewise linear (CSWPL) model, are used to verify the proposed technique. Experimental test of a 10-W GaN device from Cree is presented, showing the effectiveness of the proposed technique.

УПРАВЛЕНИЕ КОМПЬЮТЕРНЫМ УЗЛОМ С ИСПОЛЬЗОВАНИЕМ БЕСПРОЕКТОРНЫХ АЛГОРИТМОВ СТОХАСТИЧЕСКОЙ АППРОКСИМАЦИИ ПРИ НЕСАНКЦИОНИРОВАННЫХ ИЗМЕНЕНИЯХ ВХОДНОГО ПОТОКА

The aim of the work is to propose a decision-making approach for managing a computer node in the face of unauthorized input traffic modifications. The research methods are analytical modelling of queuing systems and stochastic approximation algorithms. The results of the research and novelty are to obtain an approach based on the stochastic approximation algorithms for adaptive decision making under a priori uncertainty of input data. Findings of the obtained approach allow increasing the efficiency of computer systems in case of unauthorized intrusions.

Gradient Temporal Difference with Momentum: Stability and Convergence

Gradient temporal difference (Gradient TD) algorithms are a popular class of stochastic approximation (SA) algorithms used for policy evaluation in reinforcement learning. Here, we consider Gradient TD algorithms with an additional heavy ball momentum term and provide choice of step size and momentum parameter that ensures almost sure convergence of these algorithms asymptotically. In doing so, we decompose the heavy ball Gradient TD iterates into three separate iterates with different step sizes. We first analyze these iterates under one-timescale SA setting using results from current literature. However, the one-timescale case is restrictive and a more general analysis can be provided by looking at a three-timescale decomposition of the iterates. In the process we provide the first conditions for stability and convergence of general three-timescale SA. We then prove that the heavy ball Gradient TD algorithm is convergent using our three-timescale SA analysis. Finally, we evaluate these algorithms on standard RL problems and report improvement in performance over the vanilla algorithms.

Stationary Behavior of Constant Stepsize SGD Type Algorithms

Stochastic approximation (SA) and stochastic gradient descent (SGD) algorithms are work-horses for modern machine learning algorithms. Their constant stepsize variants are preferred in practice due to fast convergence behavior. However, constant stepsize SA algorithms do not converge to the optimal solution, but instead have a stationary distribution, which in general cannot be analytically characterized. In this work, we study the asymptotic behavior of the appropriately scaled stationary distribution, in the limit when the constant stepsize goes to zero. Specifically, we consider the following three settings: (1) SGD algorithm with a smooth and strongly convex objective, (2) linear SA algorithm involving a Hurwitz matrix, and (3) nonlinear SA algorithm involving a contractive operator. When the iterate is scaled by 1/√α, where α is the constant stepsize, we show that the limiting scaled stationary distribution is a solution of an implicit equation. Under a uniqueness assumption (which can be removed in certain settings) on this equation, we further characterize the limiting distribution as a Gaussian distribution whose covariance matrix is the unique solution of an appropriate Lyapunov equation. For SA algorithms beyond these cases, our numerical experiments suggest that unlike central limit theorem type results: (1) the scaling factor need not be 1/√α, and (2) the limiting distribution need not be Gaussian. Based on the numerical study, we come up with a heuristic formula to determine the right scaling factor, and make a connection to the Euler-Maruyama discretization scheme for approximating stochastic differential equations.

CTM-based traffic signal optimization of mixed traffic flow with connected automated vehicles and human-driven vehicles

This paper proposes a cell transmission model (CTM)-based traffic signal timing model of mixed traffic flow composed of connected automated vehicles (CAVs) and human-driven vehicles (HDVs). Firstly, the CTM of mixed traffic flow is derived from considering the influence of the market penetration rates (MPRs) of CAVs. Secondly, the dynamic evolution is developed to capture the queue accumulation and the congestion dissipation at the entrance of the intersection. Then, the optimization model is proposed based on the constraints of traffic signals and the relationship of flow transmission between adjacent cells. Moreover, the simultaneous perturbation stochastic approximation (SPSA) algorithm is adopted to solve the proposed model. The evolution laws of the density of each entrance with time and space are compared under the fixed and the optimized traffic signals. Finally, the vehicle’s delay is selected as the evaluation index, and the superiority of the optimization model is discussed. The results show that the proposed model can effectively reduce the range and dissipation time of traffic congestion. The average dissipation efficiency of each entrance is increased by 11.11%. Furthermore, the traffic delay gradually decreases with the MPRs of CAVs, and the delay of homogeneous CAVs is 14.81% lower than that of homogeneous HDVs traffic flow. Therefore, the large-scale application of CAVs can alleviate traffic congestion and improve the traffic capacity of the signalized intersection.

SPSA Algorithm for Matching of Historical Data for Complex Non-Gaussian Geological Models

History matching is the process of integrating dynamic production data in the reservoir model. It consists in estimation of uncertain model parameters such that oil or water production data from flow simulation become close to observed dynamic data. Various optimization methods can be used to estimate the model parameters. Simultaneous perturbation stochastic approximation (SPSA) is one of the stochastic approximation algorithms. It requires only two objective function measurements for gradient approximation per iteration. Also parameters estimated by this algorithm might converge to their true values under arbitrary bounded additive noise, while many other optimization algorithms require the noise to have zero mean. SPSA algorithm has not been well explored for history matching problems and has been applied only to simple Gaussian models. In this paper, we applied SPSA to history matching of binary channelized reservoir models. We also used SPSA in combination with parameterization method CNN-PCA. And we considered the case of complex noise in observed production data and with objective function that does not require assumptions of normality of the observations, which is common in history matching literature. We experimentally showed that SPSA method can be successfully used for history matching of non-Gaussian geological models with different types of noise in observations and outperforms Particle Swarm Optimization by convergence speed.

Stochastic Hybrid Approximation for Uncertainty Management in Gas-Electric Systems

Gas-fired generators, with their ability to quickly ramp up and down their electricity production, play an important role in managing renewable energy variability. However, these changes in electricity production translate into variability in the consumption of natural gas, and propagation of uncertainty from the electric grid to the natural gas system. To ensure that both systems are operating safely, there is an increasing need for coordination and uncertainty management among the electricity and gas networks. A challenging aspect of this coordination is the consideration of natural gas dynamics, which play an important role in intra-day operation, but give rise to a set of non-linear and non-convex equations that are hard to optimize even in the deterministic case. Ideally, the problem is formulated as a stochastic problem but many conventional methods for stochastic optimization are numerically intractable because they either incorporate a large number of scenarios directly or require the underlying problem to be convex. To address these challenges, we propose using a Stochastic Hybrid Approximation algorithm to more efficiently solve these problems and investigate the efficacy of several different variants of this algorithm. Our case study demonstrates that the proposed technique is able to quickly obtain high quality solutions and outperforms existing benchmarks such as Generalized Benders Decomposition. We demonstrate that coordinated uncertainty management that accounts for the gas system can significantly reduce both electric and gas system load shed in stressed conditions.

Bernstein polynomial of recursive regression estimation with censored data

In this paper, we deal with the problem of the regression estimation near the edges under censoring. For this purpose, we consider a new recursive estimator based on the stochastic approximation algorithm and Bernstein polynomials of the regression function when the response random variable is subject to random right censoring. We give the central limit theorem and the strong pointwise convergence rate for our proposed nonparametric recursive estimators under some mild conditions. Finally, we provide pointwise moderate deviation principles (MDP) for the proposed estimators. We corroborate these theoretical results through simulations as well as the analysis of a real data set.

The Stochastic Approximation Method for Recursive Kernel Estimation of the Conditional Extreme Value Index

In this research paper, we apply the stochastic approximation method to define a class of recursive kernel estimator of the conditional extreme value index. We investigate the properties of the proposed recursive estimator and compare them to those pertaining to Hill’s non-recursive kernel estimator. We show that using some optimal parameters, the proposed recursive estimator defined by the stochastic approximation algorithm proves to be very competitive to Hill’s non-recursive kernel estimator. Finally, the theoretical results are confirmed through simulation experiments and illustrated using a real dataset about Malaria in Senegalese children.