Things you Wanted to Know about Bias in Evaluations but Never Dared to Think

Abstract

The thrust for evidence‐based policymaking has paid little attention to problems of bias. Statistical evidence is more fragile than generally understood, and false positives are all too likely given the incentives of policymakers and academic and professional evaluators. Well‐known cognitive biases make bias likely for not dissimilar reasons in qualitative and mixed methods evaluations. What we term delinquent organisational isomorphism promotes purportedly scientific evaluations in inappropriate institutional contexts, intensifying motivated reasoning and avoidance of cognitive dissonance. This leads to states of denial with regard to the validity of much evaluation activity. Independent replications, revisits and restudies, together with codes of ethics that relate to professional integrity, may mitigate these problems.