The FCE Speaking test: using rater reports to help interpret test scores

Abstract

Interpreting test scores depends on understanding what they represent. This article reports research using oral examiner verbal reports which attempts to gain insights into the rating process in the Cambridge First Certificate in English Speaking test. Raters' verbal reports of the decision-making process were analysed, and heeded aspects of the test performances identified, with a view to better understanding how test scores could be interpreted. In fact, it was found that raters did not heed the same aspects of the assessment criteria, and heeded a wide range of non-criterion relevant information. One consequence was that raters produced a range of scores, but another was that raters giving the same score still perceived the performance quite differently. The varied nature of the raters’ perceptions, with regard to what was heeded, and how it was judged, suggests that in normal circumstances it would be impossible to say how any one Speaking score had been reached. The validity of the interpretations that test users might wish to make of the results is thus brought into question.