Towards robust model selection using estimation and approximation error bounds

Abstract

One of the main problems in machine learning and statistical inference is selecting an appropriate model by which a set of data can be explained. In the absense of any structured prior information aa to the data generating mechanism, one is often forced to consider a range of models, attempting to select the model which best explains the data, based on some quality criterion. While there have been many proposals for various criteria for model selection, most of these approaches suffer from some form of bias built into the construction of the criterion. Moreover, many of the standard methods are only guaranteed to work well asymptotically, leaving their behavior in the face of a finite amount of data completely unknown. In this paper we extend on previous work [17] and introduce a novel model selection criterion, based on combining two recent chains of thought. In particular we make use of the powerful framework of uniform convergence of empirical processes pioneered by Vapnik and Chernovenkins [23], combined with recent results concerning the approximation ability of non-linear manifolds of functions, focusing in particular on feedforward neural networks. The main contributions of this work are twofold: (i) Conceptual elucidating a coherent and robust framework for model selection, (ii) Technical the main contribution here is a lower bound on the approximation error (Theorem 10), which holds in a well specified sense for most functions of interest. As far as we are aware, this result is new in the field of function approximation.