Confounds in multivariate pattern analysis: Theory and rule representation case study

Abstract

Multivariate pattern analysis (MVPA) is a relatively recent innovation in functional magnetic resonance imaging (fMRI) methods. MVPA is increasingly widely used, as it is apparently more effective than classical general linear model analysis (GLMA) for detecting response patterns or representations that are distributed at a fine spatial scale. However, we demonstrate that widely used approaches to MVPA can systematically admit certain confounds that are appropriately eliminated by GLMA. Thus confounds rather than distributed representations may explain some cases in which MVPA produced positive results but GLMA did not. The issue is that it is common practice in MVPA to conduct group tests on single-subject summary statistics that discard the sign or direction of underlying effects, whereas GLMA group tests are conducted directly on single-subject effects themselves. We describe how this common MVPA practice undermines standard experiment design logic that is intended to control at the group level for certain types of confounds, such as time on task and individual differences. Furthermore, we note that a simple application of linear regression can restore experimental control when using MVPA in many situations. Finally, we present a case study with novel fMRI data in the domain of rule representations, or flexible stimulus-response mappings, which has seen several recent MVPA publications. In our new dataset, as with recent reports, standard MVPA appears to reveal rule representations in prefrontal cortex regions, whereas GLMA produces null results. However, controlling for a variable that is confounded with rule at the individual-subject level but not the group level (reaction time differences across rules) eliminates the MVPA results. This raises the question of whether recently reported results truly reflect rule representations, or rather the effects of confounds such as reaction time, difficulty, or other variables of no interest.