The First Validation Study: Comparing OSM and PBM in English Language HKCEE (Year 11) Essays Using Classical Statistics and Rasch Measurement

Abstract

This chapter describes the first OSM validation study. It compares paper-based and onscreen marking in Hong Kong using both classical statistics and Multi-faceted Rasch measurement (MFRM). The study was unlike those which usually compare experienced paper-based markers’ efforts when marking on screen in that it compares both experienced PBM markers and new markers who had only ever marked onscreen. In addition, it was innovative in that it investigated PBM and OSM in the context of the large scale implementation of OSM for all public examinations in Hong Kong at Year 11. The study consisted of an examination of English language essay scripts drawn from the live 2007 Hong Kong Certificate of Education Examination (HKCEE) Year 11 English Language Writing Paper, Part 1B2. In order to compare the two different modes of marking, it was arranged that of the total of 196 markers of the 2007 HKCEE Writing paper, 30 with good marker statistics would mark on paper (i.e., re-mark) 100 scripts – scripts that they had marked onscreen 9 months previously. Subsequent to the rating, they completed a questionnaire providing feedback on the exercise. An analysis of the data suggested that technologically, markers had no problems with OSM. Statistical results – correlations between the two forms of rating, the number of discrepancy scripts (where a third rating is required) – suggested no bias favouring either form of rating. Attitudinal differences surfaced, however, between new markers who had solely rated on screen compared with veteran markers, whose previous rating experiences had only been with paper-based rating. New markers felt that having to travel to a special marking centre was less of an inconvenience than did old markers. New markers, additionally, expressed a preference to mark on screen rather than on paper. The study concluded by predicting that as OSM became more prevalent, it would be accepted as the norm, with prejudice about fairness to candidates or convenience associated with PBM being preferable – for whatever reasons – gradually fading.