Machine and expert judgments of student perceptions of teaching behavior in secondary education: Added value of topic modeling with big data

Abstract

Research shows that effective teaching behavior is important for students' learning and outcomes, and scholars have developed various instruments for measuring effective teaching behavior domains. Although student assessments are frequently used for evaluating teaching behavior, they are mainly in Likert-scale or categorical forms, which precludes students from freely expressing their perceptions of teaching. Drawing on an open-ended questionnaire from large-scale student surveys, this study uses a machine learning tool aiming to extract teaching behavior topics from large-scale students’ open-ended answers and to test the convergent validity of the outcomes by comparing them with theory-driven manual coding outcomes based on expert judgments. We applied a latent Dirichlet allocation (LDA) topic modeling analysis, together with a visualization tool (LDAvis), to qualitative data collected from 173,858 secondary education students in the Netherlands. This data-driven machine learning analysis yielded eight topics of teaching behavior domains: Clear explanation, Student-centered supportive learning climate, Lesson variety, Likable characteristics of the teacher, Evoking interest, Monitoring understanding, Inclusiveness and equity, Lesson objectives and formative assessment. In addition, we subjected 864 randomly selected student responses from the same dataset to manual coding, and performed theory-driven content analysis, which resulted in nine teaching behavior domains and 19 sub-domains. Results suggest that the relation between machine learning and human analysis is complementary. By comparing the bottom-up (machine learning analysis) and top-down (content analysis), we found that the proposed topic modeling approach reveals unique domains of teaching behavior, and confirmed the validity of the topic modeling outcomes evident from the overlapping topics.