Dynamic Stochastic Approximation Research Articles

Approximate Kalman filtering by both M-robustified dynamic stochastic approximation and statistical linearization methods

The problem of designing a robustified Kalman filtering technique, insensitive to spiky observations, or outliers, contaminating the Gaussian observations has been presented in the paper. Firstly, a class of M-robustified dynamic stochastic approximation algorithms is derived by minimizing at each stage a specific time-varying M-robust performance index, that is, general for a family of algorithms to be considered. The gain matrix of a particular algorithm is calculated at each stage by minimizing an additional criterion of the approximate minimum variance type, with the aid of the statistical linearization method. By combining the proposed M-robust estimator with the one-stage optimal prediction, in the minimum mean-square error sense, a new statistically linearized M-robustified Kalman filtering technique has been derived. Two simple practical versions of the proposed M-robustified state estimator are derived by approximating the mean-square optimal statistical linearization coefficient with the fixed and the time-varying factors. The feasibility of the approaches has been analysed by the simulations, using a manoeuvring target radar tracking example, and the real data, related to an object video tracking using short-wave infrared camera.

Distributed dynamic stochastic approximation algorithm over time-varying networks

In this paper, a distributed stochastic approximation algorithm is proposed to track the dynamic root of a sum of time-varying regression functions over a network. Each agent updates its estimate by using the local observation, the dynamic information of the global root, and information received from its neighbors. Compared with similar works in optimization area, we allow the observation to be noise-corrupted, and the noise condition is much weaker. Furthermore, instead of the upper bound of the estimate error, we present the asymptotic convergence result of the algorithm. The consensus and convergence of the estimates are established. Finally, the algorithm is applied to a distributed target tracking problem and the numerical example is presented to demonstrate the performance of the algorithm.

Dynamic stochastic approximation for multi-stage stochastic optimization

In this paper, we consider multi-stage stochastic optimization problems with convex objectives and conic constraints at each stage. We present a new stochastic first-order method, namely the dynamic stochastic approximation (DSA) algorithm, for solving these types of stochastic optimization problems. We show that DSA can achieve an optimal $${{\mathcal {O}}}(1/\epsilon ^4)$$ rate of convergence in terms of the total number of required scenarios when applied to a three-stage stochastic optimization problem. We further show that this rate of convergence can be improved to $${{\mathcal {O}}}(1/\epsilon ^2)$$ when the objective function is strongly convex. We also discuss variants of DSA for solving more general multi-stage stochastic optimization problems with the number of stages $$T > 3$$ . The developed DSA algorithms only need to go through the scenario tree once in order to compute an $$\epsilon $$ -solution of the multi-stage stochastic optimization problem. As a result, the memory required by DSA only grows linearly with respect to the number of stages. To the best of our knowledge, this is the first time that stochastic approximation type methods are generalized for multi-stage stochastic optimization with $$T \ge 3$$ .

Robust Real-Time Identification Based on an Optimal Residuals Nonlinear Transformation

A robust identification algorithm for nonlinear dynamic systems described by a NARX model with stochastically time-varying parameters is proposed. The method is based on the general formulation of dynamic stochastic approximation schemes characterised by an optimal nonlinear residuals transformation (in the maximum likelihood estimation sense), as well as a step-by-step optimisation with respect to the weighting matrix within the algorithm. This method has the advantage of directly estimating the error distribution from the data itself, and is not based on any generalised assumption about the error distribution as is done in other robust approaches. As a result, this is a flexible and non-parametric approach. Wavelet-based density estimation is employed in this approach while a Monte Carlo simulation study is used to compare the proposed method with several previous identification methods.

Convergence analysis of dynamic stochastic approximation

Stochastic approximation method concerns the sequential estimation of the root or the extreme of a function observed with noise. The idea is then extended to the moving root case by Dupac̀, and it is called as the dynamic stochastic approximation method. Its convergence properties are derived here by using a randomly varying truncation technique under weaker conditions in comparison with the previous work: the growth rate restriction on the function has been removed and the condition on the observation noises has been weakened to the possibly weakest ones.

Asymptotically Optimal Dynamic Stochastic Approximation

Stochastic approximation method originated by Robbins and Monro, and Kiefer and Wolfowitz concerns the sequential estimation of the root or the maximum of a regression function, and the idea is extended to the moving root case by Dupač. It is called as the dynamic stochastic approximation method. Here the asymptotic variance of the estimation error of the dynamic stochastic approximation method is discussed. First the asymptotic variance is derived for more general approximation method, and then derive a nonlinear transformation function minimizing the variance. However, the construction of optimal transformation requires the knowledge of probability distribution of observation noise. And hence, a construction method of the asymptotically optimal transformation function from the estimates is developed.

Performance management of multiple access communication networks

This paper focuses on conceptual design, development, and implementation of a performance management tool for computer communication networks to serve large-scale integrated systems. The objective is to improve the network performance in handling various types of messages by on-line adjustment of protocol parameters. The techniques of perturbation analysis of Discrete Event Dynamic Systems (DEDS), stochastic approximation (SA), and learning automata have been used in formulating the algorithm of performance management. The efficacy of the performance management tool has been demonstrated on a network testbed. The conceptual design presented offers a step forward to bridging the gap between management standards and users' demands for efficient network operations since most standards such as ISO (International Standards Organization) and IEEE address only the architecture, services, and interfaces for network management. The proposed concept for performance management can also be used as a general framework to assist design, operation, and management of various DEDS such as computer integrated manufacturing and battlefield C/sup 3/ (Command, Control, and Communications).< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Reinforcement learning is direct adaptive optimal control

Neural network reinforcement learning methods are described and considered as a direct approach to adaptive optimal control of nonlinear systems. These methods have their roots in studies of animal learning and in early learning control work. An emerging deeper understanding of these methods is summarized that is obtained by viewing them as a synthesis of dynamic programming and stochastic approximation methods. The focus is on Q-learning systems, which maintain estimates of utilities for all state-action pairs and make use of these estimates to select actions. The use of hybrid direct/indirect methods is briefly discussed. >

Robust real-time identification for a class of linear time-varying discrete systems

The problem of robust real-time identification of linear single-input-single-output dynamic systems with stochastically time-varying parameters is considered. Two ways of constructing robust algorithms that are able to handle outliers contaminating the gaussian observation disturbance samples are discussed. The first way is based on the general formulation of dynamic stochastic approximation schemes characterized by an adequate non-linear residual transformation, as well as the step-by-step optimization with respect to the weighting matrix of the algorithm. The second way is based on the formulation of one-step optimal estimates. Monte Carlo simulation results illustrate the discussion and show the efficiency of the proposed algorithms in the presence of outliers.

On dynamic stochastic approximation

The behaviour of the dynamic stochastic approximation algorithm suggested by DUPA[Cbreve] [4, 5] is examined under various conditions. Essentially, convergence in q.m. is proved and the rate of convergence is obtained for three classes of regression functions. Necessary and sufficient conditions are established for the convergence in q.m. of the algorithm in terms of the algorithm parameters for the stationary (nondynamic) case.

On the dynamic stochastic approximation

On the dynamic stochastic approximation

A New Dynamic Stochastic Approximation Procedure

This paper considers Robbins-Monro stochastic approximation when the regression function changes with time. At time $n$, one can select $X_n$ and observe an unbiased estimator of the regression function evaluated at $X_n$. Let $\theta_n$ be the root of the regression function at time $n$. Our goal is to select the sequence $X_n$ so that $X_n - \theta_n$ converges to 0. It is assumed that $\theta_n = f(s_n)$ for $s_n$ known at time $n$ and $f$ an unknown element of a class of functions. Under certain conditions on this class and on the sequence of regression functions, we obtain a random sequence $X_n$ such that $|X_n - \theta_n|$ converges to 0 in Cesaro mean with probability 1. Under more stringent conditions, $X_n - \theta_n$ converges to 0 with probability 1.

Comparative Analysis of a class of Robust Real-Time Identification Methods

In this paper the problem of robust real-time identification of 1inear dynamic systems with time varying parameters is considered. Supposing two classes of system disturbance distributions several robust identification algorithms are constructed either on the basis of the formulation of minimum variance estimates or the optimization of dynamic stochastic approximation schemes. The comparative analysis of the algorithms is done by Monte Carlo simulations, covering both constant and time-varying parameter cases. The analysis shows successful applications of the robust algorithms and indicates the most suitable methods for practical applications. Some connections between the adaptive and the robust estimations are illustrated.

On asymptotic properties of real-time identification algorithms based on dynamic stochastic approximation

The algorithms of dynamic stochastic approximation type are proposed for real-time identification of multivariable linear dynamic discrete-time systems with stochastic parameters. These algorithms can be considered as the general representatives of a class of gradient-type equation-error recursive identification methods. The analysis of their asymptotic properties is presented. It is proved that the algorithms converge either in the mean-square sense, or in the sense of keeping the mean-square error bounded, depending on system parameter properties. Convergence conditions are expressed in terms of inherent system characteristics, e.g., properties of the impulse response matrices and their realizations. A large class of input random processes is supposed.

Some Generalizations of Dynamic Stochastic Approximation Processes

Some generalizations of Dupac's dynamic stochastic approximation have been worked out to the more general cases of time variation. Sufficient conditions for convergence in the mean square and with probability one are given in case of deterministic trend and convergence with a bound is proved for the random trend case, using the estimation scheme $x_{n+1} = g_n(x_n) + a_n(\alpha - y_{n+1}(g_n(x_n)))$. This estimation procedure seems to be of practical use to a variety of problems in estimation, prediction and control.

Convergence conditions of dynamic stochastic approximation method for nonlinear stochastic discrete-time dynamic systems

Sufficient conditions are presented under which the dynamic stochastic approximation estimate converges with probability 1 to the true state vector of a nonlinear stochastic discrete-time dynamic system. These conditions are stated in terms of system dynamics, measurement function, and noise statistics; and are closely related to the concept of on-line stochastic observability.

Stochastic Learning of Time-Varying Parameters in Random Environment

The problem of learning in nonstationary environment is formulated as that of estimating time-varying parameters of a probability distribution which characterizes the process under study. Dynamic stochastic approximation algorithms are proposed to estimate the unknown time-varying parameters in a recursive fashion. Both supervised and nonsupervised learning schemes are discussed and their convergence properties are investigated. An accelerated scheme for the possible improvement of the dynamic algorithm is given. Numerical examples and an application of the proposed algorithm to a problem in weather forecasting are presented.

Convergent algorithms for pattern recognition in nonlinearly evolving nonstationary environment

Blaydon and Ho have recently proposed two algorithms to determine the probability p(A/x) that a sample with the set of attributes x belongs to a pattern class A, assuming a fixed p(A/x). The present letter modifies these algorithms to allow p(A/x) ≡ p i (A/x), i= 1, 2, ..., to evolve (not necessarily linearly) with i. Dynamic stochastic approximation arguments are used.