Comparative Judgement for advancing research in applied linguistics

Abstract

Comparative judgement (CJ) is a data collection method in which judges are presented with two items, side-by-side, and asked to decide which is “better”. By compiling the results of many such decisions, a scale can be developed to rank each item from best to worst. Though most commonly used for educational assessment, CJ is fundamentally a method for generating holistic, perceptually grounded measurements of hard-to-define constructs. This capability gives CJ broad potential in the field of applied linguistics, as it can address the need for more accurate measurement and definition of various applied linguistic constructs. In this tutorial, we provide a step-by-step guide on how to set up CJ studies and analyse the resulting data. We also discuss some of the method's strengths and weaknesses, and explore ways in which it might enhance and broaden the methodological toolkit of applied linguistic research.