Consistent Order Estimation and Minimal Penalties

Abstract

Consider an i.i.d. sequence of random variables whose distribution <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f*</i> lies in one of the nested families of models <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Mq</i> , <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">q</i> ≥ 1. The smallest index <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">q*</i> such that <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Mq*</i> contains <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f*</i> is called the model order. The aim of this paper is to explore the consistency properties of penalized likelihood model order estimators such as Bayesian information criterion. We show in a general setting that the minimal strongly consistent penalty is of order η( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">q</i> )loglog <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> , where η( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">q</i> ) is a dimensional quantity. In contrast to previous work, an a priori upper bound on the model order is not assumed. The results rely on a sharp characterization of the pathwise fluctuations of the generalized likelihood ratio statistic under entropy assumptions on the model classes. Our results are applied to the geometrically complex problem of location mixture order estimation, which is widely used but poorly understood.