The Limits of Empiricism: A Critique of Data-Driven Theory Development

Abstract

The abundance of data available to researchers has led to increasing interest in data-derived theoretical development. Although this is a valid method of deriving theoretical models, it is subject to numerous limitations and hazards that may threaten the validity and usefulness of the models. The purpose of this paper is to critique empirically driven theoretical development. Our goal is to offer a cautionary tale about the limits of derivation of theory from empirical analysis in the hopes that our analysis and critique can strengthen empirical derivation of theory. In this paper, we use the empirical derivation of the Unified Model of Information Security Policy Compliance (UMISPC) as a research case study to illustrate some of these limitations and risks. For example, we critique the opportunistic dropping of theoretical paths based on statistical results, cautioning that doing so is insufficient for forming new theory. We also report several attempts at validating UMISPC through replication, including our own, which used data from a survey of 525 employed American adults. Comparison of the replications and original model indicates a general failure to replicate substantial portions of the original paper. We discuss five specific pitfalls associated with empirically driven model development and make recommendations for future studies that use inductive, data-driven approaches to derive theoretical models.