Validating onscreen marking in Hong Kong

Abstract

This paper details an investigation into onscreen marking (OSM) in Hong Kong—where paper-based marking is being phased out, being completely superseded by OSM. The study described involved 30 raters who had previously rated English language essay scripts on screen in the 2007 Hong Kong Certificate of Education examination. In the current study, the 30 raters re-rated (unbeknown to them) 100 scripts they had rated in the 2007 examination. Using multi-faceted Rasch measurement, a five-faceted design was employed, modelling raters, test takers, input prompt materials, rating scales, and, especially, the marking medium. Results showed that all factors generally exhibited good data fit. With the method of marking, the major facet for investigation, logit values emerged which were both very close to zero. The hypothesis that the method of marking does not intrude on scores awarded to test takers when scripts are marked on screen or on paper was therefore accepted. Given that in 2012, all public examinations in Hong Kong will be marked solely on screen, results from the current study suggest that the scores returned from the OSM marking system will be able to be considered as reliable as those obtained from paper-based marking.