Finite sample performance guarantees of fusers for function estimators

Abstract

Abstract An independent and identically distributed sample of an unknown function generated according to an unknown distribution is given. Several function estimators are computed based on the sample by minimizing the empirical error over function families. These estimators provide performance guarantees based on best available cover sizes for the respective function families. Traditionally, the estimator that performs the best on the training data or provides the best performance guarantee is often selected and others are discarded when no other information – such as additional examples – is available. We consider a fuser trained with the outputs of the individual estimators by minimizing empirical error over a fuser class. If the fuser class satisfies a simple isolation property and has a smaller cover size compared to individual estimators, we show that the performance guarantee of the fuser is at least as good as that of the empirical best estimator. Several well-known fusers such as linear combinations, special potential functions, and certain feedforward piecewise-linear networks satisfy the isolation property. In the first two cases, the fuser class forms a vector space for which we derive more detailed conditions. We also derive conditions in terms of the natural parameters when the estimators are feedforward sigmoidal networks with bounded weights. We present simulation results to show the effectiveness of the fuser.