Who is the expert? Construct and criteria validity of student and teacher ratings of instruction

Abstract

In this article, we examine the construct and criteria validity of student and teacher ratings as indicators of instructional features. As parsimonious measures, student and teacher reports are frequently used to assess aspects of the learning environment. From a methodological perspective, however, both approaches have been questioned. Whereas student ratings are occasionally criticised as being undifferentiated and easily influenced by personal preferences, teacher reports are sometimes considered to be biased by self-serving strategies or teaching ideals. Instead of pitting one method against the other, our study aimed at establishing the specific value of each approach. The analyses draw on data from a German extension to the 2003 PISA assessment, in which 288 mathematics teachers and their students completed questionnaires using identical items to tap various aspects of instruction. Mean student ratings were computed for each class. Exploratory and confirmatory factor analyses revealed specific conceptual structures for student and teacher ratings, with teachers elaborating on the use of tasks and methods, and students focusing on their teacher’s support in personal and learning matters. Three shared dimensions were also established: occurrence of classroom management problems, degree of cognitive autonomy during lessons, and tempo of interaction. Teacher/student agreement on these dimensions varied across constructs, with considerable agreement on classroom management, low agreement on cognitive autonomy, and no significant agreement on whether or not the interaction tempo was appropriate. Accordingly, correlations with external criteria (student motivation, teacher motivation, mathematical achievement scores, and characteristics of the tasks set in class) also varied across constructs. We conclude that student and teacher ratings are best suited to tapping different aspects of the learning environment, and recommend that care be taken in choosing a data source appropriate for the construct to be measured.