Empirical risk minimization is optimal for the convex aggregation problem

Abstract

Let $F$ be a finite model of cardinality $M$ and denote by $\operatorname{conv}(F)$ its convex hull. The problem of convex aggregation is to construct a procedure having a risk as close as possible to the minimal risk over $\operatorname{conv} (F)$. Consider the bounded regression model with respect to the squared risk denoted by $R(\cdot)$. If ${ \widehat{f}}_{n}^{\mathit{ERM}\mbox{-}C}$ denotes the empirical risk minimization procedure over $\operatorname{conv}(F)$, then we prove that for any $x>0$, with probability greater than $1-4\exp(-x)$, \[R(\widehat{f}_{n}^{\mathit{ERM}\mbox{-}C})\leq\min_{f\in\operatorname{conv}(F)}R(f)+c_{0}\max\biggl(\psi_{n}^{(C)}(M),\frac{x}{n}\biggr),\] where $c_{0}>0$ is an absolute constant and $\psi_{n}^{(C)}(M)$ is the optimal rate of convex aggregation defined in (In Computational Learning Theory and Kernel Machines (COLT-2003) (2003) 303–313 Springer) by $\psi _{n}^{(C)}(M)=M/n$ when $M\leq\sqrt{n}$ and $\psi _{n}^{(C)}(M)=\sqrt{\log (\mathrm{e}M/\sqrt{n})/n}$ when $M>\sqrt{n}$.