Exploratory Inference: Localizing Relevant Effects with Confidence

Abstract

Modern statistical analysis is often concerned with the exploration of data in the hope of finding patterns of effects congruent with domain knowledge. We propose an approach that uses domain knowledge at two levels: to define the minimum effect size of real interest, and to drive the user’s exploration of the data. In this search for patterns guided by the interplay between domain knowledge and statistical evidence, we will take into account the effect of selection with a simultaneous inference paradigm. We illustrate the concepts and methods with a neuroimaging data set, by providing simultaneous confidence bounds for the number of true discoveries in any selected set of voxels. Notably, these bounds are simultaneously valid for all possible selections. This allows a truly interactive approach to post-selection inference, which does not impose any limits on the way the researcher chooses to perform the selection.