Degrees of freedom for off-the-grid sparse estimation

Abstract

The degrees of freedom quantify the number of parameters of an estimator and is central to various risk minimization procedures. Its computation and properties are well understood when dealing with finite dimensional linear models, possibly regularized using sparsity through the celebrated Lasso estimator. However, for some applications (such as super-resolution in imaging or training multi-layer perceptrons with a single hidden layer) it makes sense to rather consider “continuous” methods. For these applications, these methods avoid the discretization of the parameter space and lead to more efficient numerical solvers. Training these continuous models with a sparsity inducing prior can be achieved by solving a convex optimization problem over the infinite dimensional space of measures, which is often called the Beurling Lasso (Blasso), and is the continuous counterpart of the Lasso. Previous works (Ann. Statist. 35 (2007) 2173–2192; Statist. Sinica 23 (2013) 809–828) show that the size of the smallest solution support is an unbiased estimator of the degrees of freedom of the Lasso. Our main contribution is a proof of a continuous counterpart to this result for the Blasso. In contrast to the Lasso, our new formula shows that the empirical degrees of freedom of the Blasso is not proportional to the size of the support. Our second contribution is a detailed study of the case of sampling Fourier coefficients in 1D, which corresponds to a super-resolution problem. We show that our formula for the unbiased estimation of the degrees of freedom is valid outside of a set of measure zero of observations, which in turn justifies its use to compute an unbiased estimator of the prediction risk using the Stein Unbiased Risk Estimator (SURE). We also report numerical results for both the case of Fourier sampling and the learning of a multilayers perceptron with a single hidden layer.