Scale-Invariant and consistent Bayesian information criterion for order selection in linear regression models

Abstract

The Bayesian information criterion (BIC) is one of the most well-known criterion used for model order estimation in linear regression models. However, in its popular form, BIC is inconsistent as the noise variance tends to zero given that the sample size is small and fixed. Several modifications of the original BIC have been proposed that takes into account the high-SNR consistency, but it has been recently observed that the performance of the high-SNR forms of BIC highly depends on the scaling of the data. This scaling problem is a byproduct of the data dependent penalty design, which generates irregular penalties when the data is scaled and often leads to greater underfitting or overfitting losses in some scenarios when the noise variance is too small or large. In this paper, we present a new form of the BIC for order selection in linear regression models where the parameter vector dimension is small compared to the sample size. The proposed criterion eliminates the scaling problem and at the same time is consistent for both large sample sizes and high-SNR scenarios.