Assessing computer program reading comprehension skill in pre-service teachers: The development and piloting of a screening instrument in Scratch

Abstract

The recent addition of computer science and computer programming to primary school curricula worldwide introduces the challenge of bringing primary school teachers up to speed in this new learning domain. Assessments are needed to determine their knowledge and competencies in this area. The aim of this study was to develop and pilot a program reading comprehension test (the “SRCT”) in Scratch, a frequently used programming language in primary schools. The test consists of 12 items that can be administered within 30 minutes via a Google Forms interface. We piloted the test with two cohorts of Swiss, German-speaking pre-service teachers. The first cohort (n=94) filled out the test during class at the end of semester and the second cohort (n=74) from home and at the beginning of semester. In both contexts, the test showed an internal consistency of .68. Such internal consistency is considered acceptable for a screening instrument and removal of items neither consistently (between the samples), nor substantially increased the internal consistency. We found no significant differences in performance between men and women. We also examined validity by comparing the performance pattern on the test with that on the Scratch exam assignment. However, more specific research, with one-on-one links between individual test performance and a larger variety of tests are needed to draw conclusions about this aspect of the SCRT.