Strong consistency of the regularized least-squares estimates of infinite autoregressive models

Abstract

Our main interest is on-line parameter estimation of infinite AR models with exponentially decaying coefficients. The practical importance of the problem follows from the fact that the class of such models includes (but not limited to) all causal invertible ARMA( p , q ) models. On-line parameter estimation means that the length of the observed data sample is not known a priori and may indefinitely increase. Hence, the parameter estimates should be refined upon arrival of every new observation. So use of the maximum likelihood (ML) method is not feasible due to the high computational burden, and recursive estimation procedures are preferable. We usually assume that the true underlying model of the observed process is a finite AR, MA or a mixed ARMA equation. However, this assumption can be rarely justified in practice. A common approach is to approximate the true model by finite AR models whose orders are chosen by information criterions such as AIC, BIC, PLS and whose parameters may be estimated by the ordinary least-squares method, the Yule–Walker method and others. In this paper we investigate a limiting case for approximating by finite AR models, i.e. an AR model of infinite order. We focus on the strong consistency properties of the regularized least-squares estimates. The regularizer may be interpreted as the smoothing operator applied to the number of the AR coefficients being estimated, and constitutes a link to the model selection criterions. The rate of a.s. convergence of the infinite LS estimates is established. In addition, a complimentary result on the convergence of almost semi-martingales, which is also of independent interest, is presented. The proposed identification procedure is evaluated by numerical examples.