A post-screening diagnostic study for ultrahigh dimensional data

Abstract

We propose a consistent lack-of-fit test to assess whether replacing the original ultrahigh dimensional covariates with a given number of linear combinations results in a loss of regression information. To attenuate the spurious correlations that may inflate type-I error rates in high dimensions, we suggest to randomly split the observations into two parts. In the first part, we screen out as many irrelevant covariates as possible. This screening step helps to reduce the ultrahigh dimensionality to a moderate scale. In the second part, we perform a lack-of-fit test for conditional independence in the context of sufficient dimension reduction. In case that some important covariates are missed with a non-ignorable probability in the first screening stage, we introduce a multiple splitting procedure. We further propose a new statistic to test for conditional independence, which is shown to be n-consistent under the null and root-n-consistent under the alternative. We develop a consistent bootstrap procedure to approximate the asymptotic null distribution. The performances of our proposal are evaluated through comprehensive simulations and an empirical analysis of GDP data.