Contribution of open-ended questions in student evaluation of teaching

Abstract

ABSTRACT Student evaluation of teaching (SET) is applied in the vast majority of universities and higher education institutions. They are used to design professor training programs, evaluate teaching performance, and show evidence of performance to different stakeholders. SET surveys typically include an open-ended question which is not always considered in the analysis of the results. This study aims to show the contribution of analyzing the students’ comments by means of the Latent Dirichlet Allocation methodology to factor them into the analysis of the quantitative part of the survey. For this purpose, a sample of 737 courses taught during 2017 and 2018 in an undergraduate program at a Chilean university is used. The results show that both the number of comments and the topics that can be extracted from them contribute significantly to the analysis of the professors’ teaching performance. The topics extracted are more specific than the quantitative dimensions of the survey, which allows obtaining very concrete feedback for professors and for designing training programs. Around half of the topics extracted are actionable and do not depend on the intrinsic characteristics of the professors, which allows for effective improvements in teaching. Additionally, the extracted topics can be grouped into dimensions that have a correspondence with the quantitative dimensions of the survey, although they only cover a subset of the latter. This result provides insights to improve the survey design and adjust the weighting of its different dimensions.