Stochastic Approximation Algorithm Research Articles

Stochastic search for a parametric cost function approximation: Energy storage with rolling forecasts

Rolling forecasts have been almost overlooked in the renewable energy storage literature. In this paper, we provide a new approach for handling uncertainty not just in the accuracy of a forecast, but in the evolution of forecasts over time. Our approach shifts the focus from modeling the uncertainty in a lookahead model to accurate simulations in a stochastic base model. We develop a robust policy for making energy storage decisions by creating a parametrically modified lookahead model, where the parameters are tuned in the stochastic base model. Since computing unbiased stochastic gradients with respect to the parameters require restrictive assumptions, we propose a simulation-based stochastic approximation algorithm based on numerical derivatives to optimize these parameters. While numerical derivatives, calculated based on the noisy function evaluations, provide biased gradient estimates, an online variance reduction technique built in the framework of our proposed algorithm, will enable us to control the accumulated bias errors and establish the finite-time rate of convergence of the algorithm. Our numerical experiments show the performance of this algorithm in finding policies outperforming the deterministic benchmark policy.

Secrecy Rate Maximization of RIS-Assisted SWIPT Systems: A Two-Timescale Beamforming Design Approach

Reconfigurable intelligent surfaces (RISs) achieve high passive beamforming gains for signal enhancement or interference nulling by dynamically adjusting their reflection coefficients. Their employment is particularly appealing for improving both the wireless security and the efficiency of radio frequency (RF)-based wireless power transfer. Motivated by this, we conceive and investigate a RIS-assisted secure simultaneous wireless information and power transfer (SWIPT) system designed for information and power transfer from a base station (BS) to an information user (IU) and to multiple energy users (EUs), respectively. Moreover, the EUs are also potential eavesdroppers that may overhear the communication between the BS and IU. We adopt <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">two-timescale</i> transmission for reducing the signal processing complexity as well as channel training overhead, and aim for maximizing the average worst-case secrecy rate achieved by the IU. This is achieved by jointly optimizing the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">short-term</i> transmit beamforming vectors at the BS (including information and energy beams) as well as the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">long-term</i> phase shifts at the RIS, under the energy harvesting constraints considered at the EUs and the power constraint at the BS. The stochastic optimization problem formulated is non-convex with intricately coupled variables, and is non-smooth due to the existence of multiple EUs/eavesdroppers. No standard optimization approach is available for this challenging scenario. To tackle this challenge, we propose a smooth approximation aided stochastic successive convex approximation (SA-SSCA) algorithm. Furthermore, a low-complexity heuristic algorithm is proposed for reducing the computational complexity without unduly eroding the performance. Simulation results show the efficiency of the RIS in securing SWIPT systems. The significant performance gains achieved by our proposed algorithms over the relevant benchmark schemes are also demonstrated.

Approximate Kalman filtering by both M-robustified dynamic stochastic approximation and statistical linearization methods

The problem of designing a robustified Kalman filtering technique, insensitive to spiky observations, or outliers, contaminating the Gaussian observations has been presented in the paper. Firstly, a class of M-robustified dynamic stochastic approximation algorithms is derived by minimizing at each stage a specific time-varying M-robust performance index, that is, general for a family of algorithms to be considered. The gain matrix of a particular algorithm is calculated at each stage by minimizing an additional criterion of the approximate minimum variance type, with the aid of the statistical linearization method. By combining the proposed M-robust estimator with the one-stage optimal prediction, in the minimum mean-square error sense, a new statistically linearized M-robustified Kalman filtering technique has been derived. Two simple practical versions of the proposed M-robustified state estimator are derived by approximating the mean-square optimal statistical linearization coefficient with the fixed and the time-varying factors. The feasibility of the approaches has been analysed by the simulations, using a manoeuvring target radar tracking example, and the real data, related to an object video tracking using short-wave infrared camera.

Neural networks singular evolutive interpolated Kalman filter and its application to data assimilation for 2D water pollution model

Abstract The present study promotes a new algorithm for estimating the water pollution propagation with the primary goal of providing more reliable and high quality estimates to decision makers. To date, the widely used variational method suffers from the large computational burden, which limits its application in practice. Moreover, this method, considering the initial state as a control variable, is very sensitive in specifying initial error, especially for unstable dynamical systems. The Neural Network Filter (NNF), proposed in the present paper, is aimed at overcoming these two drawbacks in the variational method: by its nature, the NNF is sequential (no batch large assimilation window used) and stable even for unstable dynamics, with the gain parameters as control variables. The NNF, developed in the present paper, is a Neural Network Filter (NNF) version of the Singular Evolutive Interpolated Kalman Filter (SEIKF). One of the new versions of this NNF is that it uses structure of the gain of SEIKF0 taken by the SEIKF at the first time moment of correction process. To deal with the uncertainty of the system parameters and of the noise covariance, the proposed Neural Network SEIKF0 named by NNSEIKF0 makes use of the covariance of a reduced rank iterated during assimilation process and of some pertinent gain parameters tuned adaptively to yield the minimum prediction error for the system output. The computational burden in implementation of the NNSEIKF0 is reduced drastically due to applying the optimization tool known as a simultaneous perturbation stochastic approximation (SPSA) algorithm, which requires only two integrations of the numerical model. No iterative loop is required at each assimilation instant as usually happens with the standard gradient descent optimization algorithms. Data assimilation experiment, carried out by the SEIKF0 and NNSEIKF0, is implemented for the Thanh Nhan Lake in Hanoi and the performance comparison between the NNSEIKF0 and SEIKF0 is given to show the high efficiency of the proposed NNSEIKF0.

Simultaneous perturbation stochastic approximation: towards one-measurement per iteration

When measuring the value of a function to be minimized is not only expensive but also with noise, the popular simultaneous perturbation stochastic approximation (SPSA) algorithm requires only two function values in each iteration. In this paper, we present a method requiring only one function measurement value per iteration in the average sense. We prove the strong convergence and asymptotic normality of the new algorithm. Limited experimental results demonstrate the effectiveness and potential of our algorithm for solving low-dimensional problems.

Annealing Optimization for Progressive Learning With Stochastic Approximation

In this work, we introduce a learning model designed to meet the needs of applications in which computational resources are limited, and robustness and interpretability are prioritized. Learning problems can be formulated as constrained stochastic optimization problems, with the constraints originating mainly from model assumptions that define a trade-off between complexity and performance. This trade-off is closely related to over-fitting, generalization capacity, and robustness to noise and adversarial attacks, and depends on both the structure and complexity of the model, as well as the properties of the optimization methods used. We develop an online prototype-based learning algorithm based on annealing optimization that is formulated as an online gradient-free stochastic approximation algorithm. The learning model can be viewed as an interpretable and progressively growing competitive-learning neural network model to be used for supervised, unsupervised, and reinforcement learning. The annealing nature of the algorithm contributes to minimal hyper-parameter tuning requirements, poor local minima prevention, and robustness with respect to the initial conditions. At the same time, it provides online control over the performance-complexity trade-off by progressively increasing the complexity of the learning model as needed, through an intuitive bifurcation phenomenon. Finally, the use of stochastic approximation enables the study of the convergence of the learning algorithm through mathematical tools from dynamical systems and control, and allows for its integration with reinforcement learning algorithms, constructing an adaptive state-action aggregation scheme.

A tutorial introduction to reinforcement learning

In this paper, we present a brief survey of reinforcement learning, with particular emphasis on stochastic approximation (SA) as a unifying theme. The scope of the paper includes Markov reward processes, Markov decision processes, SA algorithms, and widely used algorithms such as temporal difference learning and Q-learning.

High Dimensional Origin Destination Calibration Using Metamodel Assisted Simultaneous Perturbation Stochastic Approximation

The huge traffic data generated by intelligent transportation system (ITS) leads to the development of many advanced traffic models. These traffic models consist of many adjustable parameters which need to be calibrated before they are used in practice. This paper focuses on the offline calibration of origin-destination (OD) input parameters of a simulation-based traffic model to match its output with sensor data. An improved Simultaneous Perturbation Stochastic Approximation (SPSA) algorithm called Metamodel Assisted Simultaneous Perturbation Stochastic Approximation (MSPSA) is proposed in this paper to calibrate high-dimensional OD parameters within a tight computational budget. The proposed MSPSA combines the gradient of SPSA with the gradient of a differentiable metamodel function to improve the calibration efficiency. An integer program is also used to fine-tune the OD estimates. The proposed MSPSA algorithm is tested on a simple synthetic toy network and complex road network of Kuala Lumpur (KL), Malaysia. The proposed MSPSA algorithm is compared against SPSA and state-of-the-art Weighted Simultaneous Perturbation Stochastic Approximation (WSPSA) in both transportation networks. For KL network, synthetic and real-world sensor measurements are used as ground truth references to evaluate the performance of each approach. Based on the simulation results, the proposed MSPSA algorithm is able to gain at least 50% of improvement as compared to SPSA and WSPSA in both synthetic and real-world scenarios.

Adaptive Regulation of Block-Oriented Nonlinear Systems Using Binary Sensors With Applications to Automotive Engine Control

In this article, adaptive regulation of block-oriented nonlinear systems, i.e., Hammerstein and Wiener systems, with binary-valued measurements of the regulation errors is considered. Compared with the classical framework for stochastic adaptive control, the new feature here is that only binary-valued observations of regulation errors are available to the controller. An adaptive regulator based on the stochastic approximation algorithm is proposed and it is proved that the regulator is optimal in the sense that it minimizes the long-run average of the squared regulation errors almost surely. Numerical examples as well as real applications of the proposed algorithms to automotive engine control are given.

A distributed stochastic approximation algorithm for stochastic LQ control with unknown uncertainty

This paper studies a discrete-time stochastic control problem with linear quadratic criteria over an infinite-time horizon. We focus on control systems whose system matrices are associated with random parameters involving unknown statistical properties. We design a distributed stochastic approximation algorithm to tackle the Riccati equation and derive the optimal controller stabilizing the system. The convergence analysis is provided.

Stochastic approximation with discontinuous dynamics, differential inclusions, and applications

This work develops new results for stochastic approximation algorithms. The emphases are on treating algorithms and limits with discontinuities. The main ingredients include the use of differential inclusions, set-valued analysis, nonsmooth analysis, and stochastic differential inclusions. Under broad conditions, it is shown that a suitably scaled sequence of the iterates has a differential inclusion limit. In addition, it is shown for the first time that a centered and scaled sequence of the iterates converges weakly to a stochastic differential inclusion limit. The results are then used to treat several application examples including Markov decision processes, Lasso algorithms, Pegasos algorithms, support vector machine classification, and learning. Some numerical demonstrations are also provided.

A Two-Timescale Stochastic Algorithm Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic

This paper analyzes a two-timescale stochastic algorithm framework for bilevel optimization. Bilevel optimization is a class of problems which exhibits a two-level structure, and its goal is to minimize an outer objective function with variables which are constrained to be the optimal solution to an (inner) optimization problem. We consider the case when the inner problem is unconstrained and strongly convex, while the outer problem is constrained and has a smooth objective function. We propose a two-timescale stochastic approximation (TTSA) algorithm for tackling such a bilevel problem. In the algorithm, a stochastic gradient update with a larger step size is used for the inner problem, while a projected stochastic gradient update with a smaller step size is used for the outer problem. We analyze the convergence rates for the TTSA algorithm under various settings: when the outer problem is strongly convex (resp. weakly convex), the TTSA algorithm finds an -optimal (resp. -stationary) solution, where is the total iteration number. As an application, we show that a two-timescale natural actor-critic proximal policy optimization algorithm can be viewed as a special case of our TTSA framework. Importantly, the natural actor-critic algorithm is shown to converge at a rate of in terms of the gap in expected discounted reward compared to a global optimal policy.

Convergence of stochastic approximation via martingale and converse Lyapunov methods

In this paper, we study the almost sure boundedness and the convergence of the stochastic approximation (SA) algorithm. At present, most available convergence proofs are based on the ODE method, and the almost sure boundedness of the iterations is an assumption and not a conclusion. In Borkar and Meyn (SIAM J Control Optim 38:447–469, 2000), it is shown that if the ODE has only one globally attractive equilibrium, then under additional assumptions, the iterations are bounded almost surely, and the SA algorithm converges to the desired solution. Our objective in the present paper is to provide an alternate proof of the above, based on martingale methods, which are simpler and less technical than those based on the ODE method. As a prelude, we prove a new sufficient condition for the global asymptotic stability of an ODE. Next we prove a “converse” Lyapunov theorem on the existence of a suitable Lyapunov function with a globally bounded Hessian, for a globally exponentially stable system. Both theorems are of independent interest to researchers in stability theory. Then, using these results, we provide sufficient conditions for the almost sure boundedness and the convergence of the SA algorithm. We show through examples that our theory covers some situations that are not covered by currently known results, specifically Borkar and Meyn (2000).

Formulation and solution for calibrating boundedly rational activity-travel assignment: An exploratory study

Parameter calibration of the traffic assignment models is vital to travel demand analysis and management. As an extension of the conventional traffic assignment, boundedly rational activity-travel assignment (BR-ATA) combines activity-based modeling and traffic assignment endogenously and can capture the interdependencies between high dimensional choice facets along the activity-travel patterns. The inclusion of multiple episodes of activity participation and bounded rationality behavior enlarges the choice space and poses a challenge for calibrating the BR-ATA models. In virtue of the multi-state supernetwork, this exploratory study formulates the BR-ATA calibration as an optimization problem and analyzes the influence of the two additional components on the calibration problem. Considering the temporal dimension, we also propose a dynamic formulation of the BR-ATA calibration problem. The simultaneous perturbation stochastic approximation algorithm is adopted to solve the proposed calibration problems. Numerical examples are presented to calibrate the activity-based travel demand for illustrations. The results demonstrate the feasibility of the solution method and show that the parameter characterizing the bounded rationality behavior has a significant effect on the convergence of the calibration solutions.

Integrated photovoltaic storage joint smoothing strategy based on simultaneous perturbation stochastic approximation algorithm

In order to realize the real-time control of photovoltaic power generation smoothly connected to the grid under the condition that the energy storage equipment can operate safely, a control strategy combining the simultaneous perturbation stochastic approximation (SPSA) algorithm with rule control is designed. Firstly, the photovoltaic data are processed to extract the data characteristics of the power ramp, and then the grid-connected reference power is obtained through SPSA algorithm. Secondly, considering the state of charge (SOC) of energy storage equipment and the safe operating power range of hydrogen storage equipment, 24 hybrid energy storage power allocation rules are formulated by using the rule control method. Finally, according to the sampling data of every 10 s interval in typical day, the simulation is carried out on MATLAB/simulink platform. The results show that, compared with the first-order low-pass filtering algorithm and recursive fuzzy neural network (RFNN) algorithm, the SPSA algorithm has stronger smoothing ability, and the rule control can also complete the allocation according to the characteristics of the hybrid storage device while ensuring its normal operation.

State estimation and optimal control of an inverted pendulum on a cart system with stochastic approximation approach

&lt;abstract&gt; &lt;p&gt;In this paper, optimal control of an inverted pendulum on a cart system is studied. Since the nonlinear structure of the system is complex, and in the presence of random disturbances, optimization and control of the motion of the system become more challenging. For handling this system, a discrete-time stochastic optimal control problem for the system is described, where the external force is considered as the control input. By defining a loss function, namely, the mean squared errors to be minimized, the stochastic approximation (SA) approach is applied to estimate the state dynamics. In addition, the Hamiltonian function is defined, and the first-order necessary conditions are derived. The gradient of the cost function is determined so that the SA approach is employed to update the control sequences. For illustration, considering the values of the related parameters in the system, the discrete-time stochastic optimal control problem is solved iteratively by using the SA algorithm. The simulation results show that the state estimation and the optimal control law design are well performed with the SA algorithm, and the motion of the inverted pendulum cart is addressed satisfactorily. In conclusion, the efficiency of the SA approach for solving the inverted pendulum on a cart system is verified.&lt;/p&gt; &lt;/abstract&gt;

Tracking Control for Nonlinear Nonparametric Systems Based on the Stochastic Approximation Algorithm

In this article, we study the tracking control problem for nonlinear nonparametric systems, and additive random observation noise is also taken into account. The dynamical function is allowed to be time varying and possess an arbitrary growth rate at control input. The control algorithm is designed on the basis of the stochastic approximation algorithm with expanding truncations, and it is found that there is a tradeoff between the growth rate of the dynamical function at control input and that of the truncation bound sequence to be chosen. We prove that the average tracking or strong tracking is asymptotically achieved for a class of reference state sequences, which can be strongly averaged. Finally, numerical simulations given in this article justify the theoretical assertions.

A simultaneous perturbation stochastic approximation algorithm for broadband noise control.

Simultaneous perturbation stochastic approximation (SPSA) algorithm, an algorithm without secondary path modeling, has been applied to active noise control by some researchers. Some extended versions of this algorithm have been also developed to improve its performance. However, these existing algorithms are mostly dedicated to controlling the periodic noise instead of the broadband noise. In particular, background noise is not taken into account when SPSA algorithms are applied to control broadband noise. In this paper, an algorithm combining the cost function with the SPSA algorithm to control broadband noise has been proposed. The suggested cost function is an inner product of the estimated cross-correlation function between a reference vector and the error signal. The elements of the reference vector are composed of the reference signals at different times. Moreover, the algorithm analysis is performed and the numerical simulations are carried out to demonstrate the validity of the proposed algorithm. The results illustrate that the proposed algorithm can effectively reduce broadband noise when interference noise exists in the control system. Furthermore, the proposed algorithm has better convergence performance than other SPSA algorithms.

Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Streaming Data

We introduce a streaming framework for analyzing stochastic approximation/optimization problems. This streaming framework is analogous to solving optimization problems using time-varying mini-batches that arrive sequentially. We provide non-asymptotic convergence rates of various gradientbased algorithms; this includes the famous Stochastic Gradient (SG) descent (a.k.a. Robbins-Monro algorithm), mini-batch SG and time-varying mini-batch SG algorithms, as well as their iterated averages (a.k.a. Polyak-Ruppert averaging). We show (i) how to accelerate convergence by choosing the learning rate according to the time-varying mini-batches, (ii) that Polyak-Ruppert averaging achieves optimal convergence in terms of attaining the Cramer-Rao lower bound, and (iii) how time-varying mini-batches together with Polyak-Ruppert averaging can provide variance reduction and accelerate convergence simultaneously, which is advantageous for many learning problems, such as online, sequential, and large-scale learning. We further demonstrate these favorable effects for various time-varying minibatches.

Optimization of the Pressure Drop Prediction Model of Wellbore Multiphase Flow Based on Simultaneous Perturbation Stochastic Approximation

In the process of gas lift design and condition diagnosis, the accuracy and timeliness of wellbore multiphase flow model prediction results are the basis for all subsequent work. However, for the commonly used wellbore multiphase flow pressure drop prediction models, there is a big deviation between the predicted value and the measured one, and the optimization based on the measured data is time-consuming, and it is difficult to obtain the optimal parameters of the model. Therefore, based on the well bore pressure distribution data measured quickly in area R of an oil field in Kazakhstan, a better prediction model of multiphase flow in the well bore was selected at first. Then, the simultaneous perturbation stochastic approximation (SPSA) algorithm was incorporated in the wellbore multiphase flow model to optimize the liquid holdup, which is the leading factor in the model. After repeated single well optimization and greedy selection, the optimal parameters suitable for the whole block were obtained. The example shows that the optimization speed is 10 times faster than that of Particle Swarm Optimization (PSO). After that, the optimized model was used to predict the wellbore pressure distribution, and it was found that the relative error between the measured value and the predicted one was less than 15%.