Possible biases in observation systems when applied across contexts: conceptualizing, operationalizing, and sequencing instructional quality

Abstract

Capturing and measuring instructional patterns by using standardized observation manuals has become increasingly popular in classroom research. While researchers argue that a common vocabulary of teaching is necessary for the field of classroom research to move forward, instructional features vary across classrooms and contexts, which poses serious measuring challenges. In this article, we argue that potential biases embedded in observation systems have to be identified and addressed in order for interpretations of results across different classrooms and contexts to be valid and relevant. We identify three aspects of possible systematic biases (related to the grain size of conceptualization, operationalization, and sequencing of lessons) and how these may influence ratings of instructional quality when an established observation system (the Protocol for Language Arts Teaching Observations [PLATO]) is applied in the contexts of Nordic mathematics classrooms. We discuss implications of such possible biases and make suggestions for how they may be addressed.