Learning, Time-variation, and the Factor Zoo

Abstract

Even in low-signal-to-noise settings, conventional frequentist power analysis still identifies effects 5% of the time. Furthermore, significant estimates have the wrong sign up to 50% of the time and their magnitudes are often inflated. In comparable settings, Bayesian inference rarely identifies effects, but any identified effects are large (but not inflated) and have the wrong sign less than 2.5% of the time. Bayesian inference therefore does not require significance threshold correction for multiple testing, taming the factor zoo. More generally, the Type-S (sign) error setting yields more reliable frequency properties for multiple testing inferences than the Type-1 setting.