Assessing the practical differences between model selection methods in inferences about choice response time tasks

Nathan J Evans

doi:10.3758/s13423-018-01563-9

Abstract

Evidence accumulations models (EAMs) have become the dominant modeling framework within rapid decision-making, using choice response time distributions to make inferences about the underlying decision process. These models are often applied to empirical data as “measurement tools”, with different theoretical accounts being contrasted within the framework of the model. Some method is then needed to decide between these competing theoretical accounts, as only assessing the models on their ability to fit trends in the empirical data ignores model flexibility, and therefore, creates a bias towards more flexible models. However, there is no objectively optimal method to select between models, with methods varying in both their computational tractability and theoretical basis. I provide a systematic comparison between nine different model selection methods using a popular EAM—the linear ballistic accumulator (LBA; Brown & Heathcote, Cognitive Psychology 57(3), 153–178 2008)—in a large-scale simulation study and the empirical data of Dutilh et al. (Psychonomic Bulletin and Review, 1–19 2018). I find that the “predictive accuracy” class of methods (i.e., the Akaike Information Criterion [AIC], the Deviance Information Criterion [DIC], and the Widely Applicable Information Criterion [WAIC]) make different inferences to the “Bayes factor” class of methods (i.e., the Bayesian Information Criterion [BIC], and Bayes factors) in many, but not all, instances, and that the simpler methods (i.e., AIC and BIC) make inferences that are highly consistent with their more complex counterparts. These findings suggest that researchers should be able to use simpler “parameter counting” methods when applying the LBA and be confident in their inferences, but that researchers need to carefully consider and justify the general class of model selection method that they use, as different classes of methods often result in different inferences.

Highlights

Over the past several decades, formalized cognitive models have become a dominant way of expressing theoretical explanations and analyzing empirical data
The linear ballistic accumulator (LBA) is arguably the simplest Evidence accumulations models (EAMs) that provides inferences based on the entire choice response time distributions (Brown & Heathcote, 2008; though see Wagenmakers, Van Der Maas, & Grasman, 2007 and Grasman, Wagenmakers, & Van Der Maas, 2009 for simpler EAMs based on summary statistics), allowing the simulation study to be performed within a feasible computational time-frame
The current study provided a systematic assessment of how different model selection methods perform in practical situations for the analysis of response time data using evidence accumulation models (EAMs)

Summary

Introduction

Over the past several decades, formalized cognitive models have become a dominant way of expressing theoretical explanations and analyzing empirical data. One of the primary uses of EAMs has been as a “measurement tool” for empirical data analysis, in order to provide more nuanced answers to research questions in terms of the decision-making process, rather than in terms of observed variables (Donkin, Averell, Brown, & Heathcote, 2009). These applications have been widespread, and have greatly improved our understanding of psychological theory in many instances. A common finding in the aging literature has been that older adults

Methods

Results

Conclusion