Comparing pivotal and REML-based confidence intervals for heritability

Abstract

Heritability quantifies the extent to which a physical characteristic is passed from one generation to the next. From a statistical perspective, heritability is the proportion of the phenotypic variance attributable to (additive) genetic effects and is equal to a function of variance components in linear mixed models. Relying on normal distribution assumptions, one can compute exact confidence intervals for heritability using a pivotal quantity procedure. Alternatively, large-sample properties of the restricted maximum likelihood (REML) estimator can be used to construct asymptotic confidence intervals for heritability. Exact and asymptotic intervals are compared loineye muscle area measurements and balanced one-way random effects models having groups of correlated responses. In some cases the two interval methods yield vastly different results and the REML-based confidence interval does not maintain the nomiral coverate value even for seemingly large sample sizes. For finite sample size applications, the validity of the REML-based procedure depends on the correlation structure of the data.