Estimating the Model Order in Exponential Families

Abstract

The problem of estimating the model order of a statistical model has been widely studied in the literature of time series analysis, information theory and automatic control. Most of the known order estimation schemes (AIC, BIC, CAT, FPE, MDL, etc.), although based on reasonable ideas, are heuristic in the sense that no particular risk function (involving the true order and its estimate) is optimized. Rather, these methods are derived from various extensions of the maximum likelihood principle. In this talk, a new approach to the model order estimation problem is presented: Estimators are sought which accomplish higher exponential rate of decrease in the underestimation probability, while keeping the exponential rate of the overestimation probability at a certain prescribed level. This criterion, which is an extension to the Neyman-Pearson criterion, enables to control between overestimation and underestimation probabilities, in a way that is easy and well understood. For the class of statistical models from the exponential family, an order estimator is suggested and shown to be optimal in the above defined sense, that is, it provides the best tradeoff between the asymptotic exponential rates of overestimation and underestimation probabilities. The suggested method is strongly related to the gene-realized likelihood ratio test (GLRT), which is widely used for composite hypothesis testing problems. Several examples of specific models from the exponential family are given: The Gaussian linear regression model, the Gaussian autoregressive model, and the finite alphabet Markov model. It is also demonstrated that several well known composite hypothesis testing problems can be formalized in the model order estimation framework and then solved as special cases. The results generalize to models where there are more than one order to estimate (e.g. ARMA(p,q) model). It is demonstrated that the computation time is significantly smaller than those of other model order estimation schemes. Another direction of extending the results is that of estimating the number of states of a general finite-state source, which not necessarily belongs to the exponential family. An interesting relation between the proposed scheme and universal data compression schemes will be pointed out: It can be shown that efficient data compression algorithms can be used as tools for efficient order estimation in the above described approach.