Comment: Bayesian Checking of the Second Level of Hierarchical Models: Cross-Validated Posterior Predictive Checks Using Discrepancy Measures

Abstract

We compliment Bayarri and Castellanos (BC) on producing an interesting and insightful paper on model checking applied to the second level of hierarchical models. Distributions of test statistics (functions of the observed data not involving parameters) for judg ing appropriateness of hierarchical models typically in volve nuisance (i.e., unknown) parameters. BC (2007) focus on ways to remove the dependency on nui sance parameters so that test statistics can be used to assess models, either through p-values or Berger's relative predictive surprise (RPS). They demonstrate shortcomings in terms of very low power of posterior predictive checks and a posterior empirical Bayesian method. They also demonstrate better performance of their partial posterior predictive (ppp) method over a prior empirical Bayesian method. Methods of Dey et al. (1998), O'Hagan (2003) and Marshall and Spiegel halter (2003) also are compared. Methods are contrasted in terms of whether they re quire proper prior distributions, how many measures of surprise (one per group or one total) are produced, and the degree to which data are used twice in estimation and testing. Their preferred method (ppp) can use im proper prior distributions, which are referred to as ob jective, produces a single measure of surprise for each test statistic, and avoids double use of the data. For the models and statistics considered, in comparison to the alternatives presented, ppp has a more uniform null dis tribution of p -values and more power versus alterna tives.