On the convergence of stochastic approximation procedures under markov noise in the measurements

Abstract

The problem is considered of the convergence of stochastic approximation procedures /1,2/ for seeking the zero of a function under the condition that the values of this function, accessible to measurement, contain both external as well as internal perturbations. The statement of this problem differs from the most prevalent ones in that the assumptions on the independence and additivity of the noise are waived. The proof of the convergence is based on the use of stochastic Liapunov functions /3 –6/. Discrete stochastic approximation procedures under dependent measurements were examined, for instance, in /7,8/ with another way of accounting for the perturbations and by other methods. The majority of papers on the study of stochastic programming /9/ and stochastic approximation procedures assume the independence of the measurements and the additivity of the noise. Without disparaging such an approach, it should be emphasized that it does not exhaust all varieties of problems whose study might lead to stochastic approximation procedures. In particular, if the measurements are made sufficiently often or, even more so, continuously, then the assumption of dependence of the measurements proves to be very natural, specially if the noise realize parametric perturbations of the system. Other examples, not covered in the scheme of independent measurements, are the problems of adaptive control, of observation, of estimation /10/. There are comparatively few papers (see /7,8/, for example) where the convergence of gradient procedures of extremum search is proved in the presence of additive Markov noise. Conditions are formulated in the present paper on the convergence of stochastic approximation procedures under the condition that the measurements contain both additive as well as nonadditive (internal) Markov perturbations. The analysis is restricted to procedures of the Robbins-Monroe type, mainly in the continuous version.