Fast-and-frugal means to assess reflection-related reasoning processes in teacher training—Development and evaluation of a scalable machine learning-based metric

Abstract

Reflection is hypothesized to be a key component for teachers’ professional development and is often assessed and facilitated through written reflections in university-based teacher education. Empirical research shows that reflection-related competencies are domain-dependent and multi-faceted. However, assessing reflections is complex. Given this complexity, novel methodological tools such as non-linear, algorithmic models can help explore unseen relationships and better determine quality correlates for written reflections. Consequently, this study utilized machine learning methods to explore quality correlates for written reflections in physics on a standardized teaching situation. N = 110 pre- and in-service physics teachers were instructed to reflect upon a standardized teaching situation in physics displayed in a video vignette. The teachers’ written reflections were analyzed with a machine learning model which classified sentences in the written reflections according to elements in a reflection-supporting model. A quality indicator called level of structure (LOS) was devised and further used to validate machine learning classifications against experts’ judgements. Analyses show that LOS is positively correlated with experts’ judgements on reflection quality. We conclude that LOS of a written reflection is one important indicator for high-quality written reflections which is able to exclude typical quality correlates such as text length. With the help of the machine learning model, LOS can be useful to assess pre-service physics teachers written reflections.