Classroom Observation Scores Research Articles

Improving the Precision of Classroom Observation Scores Using a Multi-Rater and Multi-Timepoint Item Response Theory Model

ABSTRACT Classroom observational protocols, in which raters observe and score the quality of teachers’ instructional practices, are often used to evaluate teachers for consequential purposes despite evidence that scores from such protocols are frequently driven by factors, such as rater and temporal effects, that have little to do with teacher quality. In this study, we present an item response theory model that improves precision of observation scores by accounting for multiple sources of measurement error. Unlike the sum-scoring approach typically used in practice, the model presented here identifies raters and protocol items that are more reliable and should therefore be weighted more in scores. We then show that teachers identified as substandard using our model versus the sum-scoring approach are oftentimes very different. Our results suggest that accountability systems could improve observation scores by increasing the scoring weights of items and raters that provide greater information.

Investigating the presence of mathematics and the levels of cognitively demanding mathematical tasks in integrated STEM units

Effective K-12 integrated STEM education should reflect an intentional effort to adequately represent and facilitate each of its component disciplines in a meaningful way. However, most research in this space has been conducted within the context of science classrooms, ignoring mathematics. Also missing from the literature is research that examines the level of cognitive demand required from mathematical tasks present within integrated STEM lessons. In order to seek insight pertaining to this gap in the literature, we sought to better understand how science teachers use mathematics within K-12 integrated STEM instruction. We used an explanatory sequential mixed methods research design to explore the enactment of mathematics in integrated STEM lessons that focus on physical, earth, and life science content. We first examined 2030 sets of video-recorded classroom observation scores generated from the 10-item STEM Observation Protocol (STEM-OP) designed for observing integrated STEM education in K-12 classrooms. We compared the STEM-OP scores of classroom observations that included mathematics with those that did not. This quantitative analysis was followed by a closer, more in-depth qualitative examination of how mathematics was employed, focusing on the degree of cognitive demand. To do this, we coded and analyzed transcripts from video-recorded classroom observations in which mathematical content was present. Our study yielded two main findings about mathematics in integrated STEM lessons: (1) the presence of mathematical content resulted in higher STEM-OP scores on nearly all items, and (2) mathematical tasks within these lessons were categorized as requiring mainly low levels of cognitive demand from students. This study highlights the need for the increased inclusion of mathematical tasks in integrated STEM teaching. Implications for including higher-order mathematical thinking within integrated STEM teaching are discussed.

The Word Nerds project: Findings from a research–practice partnership focused on spelling instruction

Learning to spell in English requires the integration of general and specific word knowledge. This paper describes the ‘Word Nerds’ project, a research–practice partnership consisting of two researchers from a large public university and 17 elementary teachers in seven school districts in the United States. The collaboration was formed to study variation in instructional practice among teachers using theWords Their Wayprogramme and address teacher‐generated questions related to how children learn to spell words. This paper describes how stakeholders worked together to understand (1) what grouping and organisational structures teachers use for spelling instruction, (2) the extent to which elementary students use analogy to spell unknown words that share spelling patterns with known words and (3) the affordances and challenges of a partnership approach to educational research. Data collection took place over one school year. Data sources included teacher focus groups, classroom observations and student formative spelling assessment data (n = 178). Analyses included descriptive statistics,t‐tests and multilevel modelling, nesting spelling items in students in classrooms. Findings indicated that classroom observation scores for grouping, teacher talk, student‐to‐student talk, reflection and student engagement during spelling lessons varied depending on format (whole group, small group or pair/individual). Students showed evidence of spelling by analogy, but this strategy was associated in part with the semantic difficulty of the target words. Instructional practices for spelling vary based on teachers' priorities, and teachers need support to implement and prioritise evidence‐based practices. Conducting research in partnership allowed the researchers to observe and understand variation in implementation of a popular spelling programme and to develop and answer research questions that mattered to teachers; however, it also limited the size, scope and location of the project.

Measuring Teaching Practices at Scale: A Novel Application of Text-as-Data Methods

Valid and reliable measurements of teaching quality facilitate school-level decision-making and policies pertaining to teachers. Using nearly 1,000 word-to-word transcriptions of fourth- and fifth-grade English language arts classes, we apply novel text-as-data methods to develop automated measures of teaching to complement classroom observations traditionally done by human raters. This approach is free of rater bias and enables the detection of three instructional factors that are well aligned with commonly used observation protocols: classroom management, interactive instruction, and teacher-centered instruction. The teacher-centered instruction factor is a consistent negative predictor of value-added scores, even after controlling for teachers’ average classroom observation scores. The interactive instruction factor predicts positive value-added scores. Our results suggest that the text-as-data approach has the potential to enhance existing classroom observation systems through collecting far more data on teaching with a lower cost, higher speed, and the detection of multifaceted classroom practices.

What Explains the Race Gap in Teacher Performance Ratings? Evidence From Chicago Public Schools

Racial gaps in teacher performance ratings have emerged nationwide across newly implemented educator evaluation systems. Using Chicago Public Schools data, we quantify the magnitude of the race gap in teachers’ classroom observation scores, examine its determinants, and describe the potential implications for teacher diversity. Between-school differences explain most of the race gap and within-school classroom-level differences—poverty, incoming achievement, and prior-year misconduct of a teacher’s students—explain the remainder of the race gap. Teachers’ value-added scores explain none of the race gap. Leveraging within-teacher variation in the teacher–evaluator race match, we find that racial mismatch does not influence observation scores. Adjusting observation scores for classroom and school context will generate more equitable ratings of teacher performance and mitigate potential adverse consequences for teacher diversity.

Comparing teaching practices, teacher content knowledge and pay in Punjab

This study utilises a unique school based survey of public, private and Public Partnership Programme (PPP) schools in Punjab, Pakistan to identify the correlates of: teacher behaviour in the classroom, teacher content knowledge mastery, and teacher pay. The study finds that private school teachers are associated with lower classroom observation scores than public school teachers, and both private and PPP school teachers are less likely to exhibit content knowledge mastery in Urdu than public school teachers (however, this is not the case for math and English). Public school teachers recruited after the introduction of test-based teacher recruitment are 30 percentage points more likely to demonstrate math content knowledge mastery than teachers recruited prior to test-based recruitment. The study also finds evidence of a pay gap between male and female teachers in both public and PPP schools after controlling for other characteristics. Teachers’ math content knowledge scores were found to be statistically significant correlates of teacher pay for private school teachers, but not for PPP or public school teachers. For female PPP school teachers, having higher academic qualifications is associated with higher wages however, this is not the case for male teachers in these schools.

An experimental evaluation of three teacher quality measures: Value-added, classroom observations, and student surveys

Nearly every state evaluates teacher performance using multiple measures, but evidence has largely shown that only one such measure—teachers’ effects on student achievement (i.e., value-added)—captures teachers’ causal effects. We conducted a random assignment experiment in 66 fourth- and fifth-grade mathematics classrooms to evaluate the predictive validity of three measures of teacher performance: value-added, classroom observations, and student surveys. Combining our results with those from two previous random assignment experiments, we provide additional experimental evidence that value-added measures are unbiased predictors of teacher performance. Though results for the other two measures are less precise, we find that classroom observation scores are predictive of teachers’ performance after random assignment while student surveys are not. These results thus lend support to teacher evaluation systems that use value-added and classroom observations, but suggest practitioners should proceed with caution when considering student survey measures for teacher evaluation.

Strategic Retention: Principal Effectiveness and Teacher Turnover in Multiple-Measure Teacher Evaluation Systems

Studies link principal effectiveness to lower average rates of teacher turnover. However, principals need not target retention efforts equally to all teachers. Instead, strong principals may seek to strategically influence the composition of their school’s teaching force by retaining high performers and not retaining lower performers. We investigate such strategic retention behaviors with longitudinal data from Tennessee. Using multiple measures of teacher and principal effectiveness, we document that indeed more effective principals see lower rates of teacher turnover, on average. Moreover, this lower turnover is concentrated among high-performing teachers. In contrast, turnover rates of the lowest-performing teachers, as measured by classroom observation scores, increase substantially under higher-rated principals. This pattern is more apparent in advantaged schools and schools with stable leadership.

Rater Performance Standards for Classroom Observation Instruments

Raters must score accurately and consistently for classroom observation scores to be valid. This requires (a) a standard defining when scoring is accurate and consistent enough and (b) measuring and remediating rater performance against that standard. Current practice has focused on this second problem to the exclusion of the first. My goal here is to start a discussion about identifying a clear, explicit standard that ensures observation scores reflect a consistent view of teaching quality, rather than raters’ idiosyncratic perspectives. In doing so, I connect current certification test cut-scores, the current practice most analogous to a standard, to explicit rater standards, highlighting both the inadequacy of cut-scores and the low standards implicit to current practice.

Using Classroom Observation Scores Instead of Test Scores as Criterion in the Estimation of Discrimination Index

Item discrimination index, which indicates the ability of the item to distinguish whether or not the individuals have acquired the qualities that are evaluated, is basically a validity measure and it is estimated by examining the fit between item score and the test score. Based on the definition of item discrimination index, classroom observation scores were used in this study instead of test scores as the indication of having the tested quality.In the framework of the study, a 25-item multiple-choice test prepared in the context of 8th grade Mathematics Unit "Multipliers and Multiples" was administered to a total of 109 8th graders (44 females, 65 males) studying in 4 separate classrooms of Ömer Nasuhi Bilmen Secondary School in Şanlıurfa Province. Furthermore, these students’ Mathematics teachers were asked to observe and score students during the unit and the obtained observation scores were used as external criterion in estimating the discrimination index. By using this criterion, fit values estimated with the help of upper and lower groups consisting of 27% from the extremes of the criterion score distribution and biserial correlation were compared with the traditional conditions where test scores were utilized. It was found that item discrimination indices based on classroom observations were higher than those based on test scores in both of the discrimination indices estimated via upper-lower 27% groups and biserial correlation. This finding was discussed to be related to the fact that while classroom observation scores were an external validity criterion, test scores were composed of items whose discrimination values were calculated. The finding also demonstrated that classroom observation scores were more rigid and eliminative than test scores in terms of discrimination.

Classroom Composition and Measured Teacher Performance

As states and districts implement more rigorous teacher evaluation systems, measures of teacher performance are increasingly being used to support instruction and inform retention decisions. Classroom observations take a central role in these systems, accounting for the majority of teacher ratings upon which accountability decisions are based. Using data from the Measures of Effective Teaching study, we explore the extent to which classroom composition influences measured teacher performance based on classroom observation scores. The context in which teachers work—most notably, the incoming academic performance of their students—plays a critical role in determining teachers’ measured performance. Furthermore, the intentional sorting of teachers to students has a significant influence on measured performance. Implications for high-stakes teacher accountability policies are discussed.

Examining Teacher Effectiveness Using Classroom Observation Scores

Despite policy efforts to encourage multiple measures of performance in newly developing teacher evaluation systems, practical constraints often result in evaluations based predominantly on formal classroom observations. Yet there is limited knowledge of how these observational measures relate to student achievement. This article leverages the random assignment of teachers to classrooms from the Measures of Effective Teaching (MET) study to identify teacher effectiveness using scores from the Framework for Teaching (FFT) instrument, one of the most widely used classroom observation protocols. While our evidence suggests that teacher performance, as measured by the FFT, is correlated with student achievement, noncompliance with randomization and the modest year-to-year correlation of a teacher’s FFT scores constrain our ability to causally identify effective teachers. Implications for policy and practice are discussed.

Uncovering Multivariate Structure in Classroom Observations in the Presence of Rater Errors

We examine the factor structure of scores from the CLASS‐S protocol obtained from observations of middle school classroom teaching. Factor analysis has been used to support both interpretations of scores from classroom observation protocols, like CLASS‐S, and the theories about teaching that underlie them. However, classroom observations contain multiple sources of error, most predominantly rater errors. We demonstrate that errors in scores made by two raters on the same lesson have a factor structure that is distinct from the factor structure at the teacher level. Consequently, the “standard” approach of analyzing on teacher‐level average dimension scores can yield incorrect inferences about the factor structure at the teacher level and possibly misleading evidence about the validity of scores and theories of teaching. We consider alternative hierarchical estimation approaches designed to prevent the contamination of estimated teacher‐level factors. These alternative approaches find a teacher‐level factor structure for CLASS‐S that consists of strongly correlated support and classroom management factors. Our results have implications for future studies using factor analysis on classroom observation data to develop validity evidence and test theories of teaching and for practitioners who rely on the results of such studies to support their use and interpretation of the classroom observation scores.

Measuring characteristics of teacher professional development

The primary purpose of the study was to create and psychometrically test an instrument which measured teachers’ perceptions of characteristics of professional development. The sample consisted of elementary teachers from five school districts in Washington State participating in a district improvement initiative. Results of exploratory factor analysis resulted in a five-component solution which provided evidence of construct validity. All components/subscales had moderate to strong internal consistency as measured by Cronbach’s alpha. The secondary purpose was to determine if characteristics of professional development predicted teachers’ use of new knowledge and skills, and student learning outcomes, thus contributing to the predictive validity of the instrument. There was a slight, but significant, correlation between Active Learning in Classroom and teachers’ use of new knowledge and skills, as measured by classroom observation scores. A small negative correlation emerged between Collective Participation and student learning, as measured by student scores on the state’s mathematics assessment. Potential uses of the instrument, called Characteristics of Teacher Professional Development (CTPD), are discussed.

Trends in Classroom Observation Scores.

Observations and ratings of classroom teaching and interactions collected over time are susceptible to trends in both the quality of instruction and rater behavior. These trends have potential implications for inferences about teaching and for study design. We use scores on the Classroom Assessment Scoring System-Secondary (CLASS-S) protocol from 458 middle school teachers over a 2-year period to study changes over time in (a) the average quality of teaching for the population of teachers, (b) the average severity of the population of raters, and (c) the severity of individual raters. To obtain these estimates and assess them in the context of other factors that contribute to the variability in scores, we develop an augmented G study model that is broadly applicable for modeling sources of variability in classroom observation ratings data collected over time. In our data, we found that trends in teaching quality were small. Rater drift was very large during raters' initial days of observation and persisted throughout nearly 2 years of scoring. Raters did not converge to a common level of severity; using our model we estimate that variability among raters actually increases over the course of the study. Variance decompositions based on the model find that trends are a modest source of variance relative to overall rater effects, rater errors on specific lessons, and residual error. The discussion provides possible explanations for trends and rater divergence as well as implications for designs collecting ratings over time.

The Instructional Challenge in Improving Teaching Quality: Lessons from a Classroom Observation Protocol

Background/Context Teacher evaluation is a major policy initiative intended to improve the quality of classroom instruction. This study documents a fundamental challenge to using teacher evaluation to improve teaching and learning. Purpose Using an observation instrument (CLASS-S), we evaluate evidence on different aspects of instructional practice in algebra classrooms to consider how much scores vary, how well observers are able to judge practice, and how well teachers are able to evaluate their own practice. Participants The study includes 82 Algebra I teachers in middle and high schools. Five observers completed almost all observations. Research Design Each classroom was observed 4–5 times over the school year. Each observation was coded and scored live and by video. All videos were coded by two independent observers, as were 36% of the live observations. Observers assigned scores to each of 10 dimensions. Observer scores were also compared with master coders for a subset of videos. Participating teachers also completed a self-report instrument (CLASS-T) to assess their own skills on dimensions of CLASS-S. Data Collection and Analysis For each lesson, data were aggregated into three domain scores, Emotional Support, Classroom Organization, and Instructional Support, and then averaged across lessons to create scores for each classroom. Findings/Results Classroom Observation scores fell in the high range of the protocol. Scores for Emotional Support were in the midlevel range, and the lowest scores were for Instructional Support. Scores for each domain were clustered in narrow ranges. Observers were more consistent over time and agreed more when judging Classroom Organization than the other two domains. Teacher ratings of their own strengths and weaknesses were positively related to observation scores for Classroom Organization and unrelated to observation scores for Instructional Support. Conclusions/Recommendations This study identifies a critical challenge for teacher evaluation policy if it is to improve teaching and learning. Aspects of teaching and learning in the observation protocol that appear most in need of improvement are those that are the hardest for observers to agree on, and teachers and external observers view most differently. Reliability is a marker of common understanding about important constructs and observation protocols are intended to provide a common language and structure to inform teaching practice. This study suggests the need to focus our efforts on the instructional and interactional aspects of classrooms through shared conversations and clear images of what teaching quality looks like.