Degrees of Freedom: Search Cost and Self-consistency

Abstract

Model degrees of freedom ( df ) is a fundamental concept in statistics because it quantifies the flexibility of a fitting procedure and is indispensable in model selection. To investigate the gap between df and the number of independent variables in the fitting procedure, Tibshirani (2015) introduced the search degrees of freedom ( sdf ) concept to account for the search cost during model selection. However, this definition has two limitations: it does not consider fitting procedures in augmented spaces and does not use the same fitting procedure for sdf and df . We propose a modified search degrees of freedom ( msdf ) to directly account for the cost of searching in either original or augmented spaces. We check this definition for various fitting procedures, including classical linear regressions, spline methods, adaptive regressions (the best subset and the lasso), regression trees, and multivariate adaptive regression splines (MARS). In many scenarios when sdf is applicable, msdf reduces to sdf . However, for certain procedures like the lasso, msdf offers a fresh perspective on search costs. For some complex procedures like MARS, the df has been pre-determined during model fitting, but the df of the final fitted procedure might differ from the pre-determined one. To investigate this discrepancy, we introduce the concepts of nominal df and actual df , and define the property of self-consistency, which occurs when there is no gap between these two df ’s. We propose a correction procedure for MARS to align these two df ’s, demonstrating improved fitting performance through extensive simulations and two real data applications.