A process for establishing and maintaining inter-rater reliability for two observation instruments as a fidelity of implementation measure: A large-scale randomized controlled trial perspective

Abstract

In this study, we presented a practical framework to establish and monitor inter-rater reliability (IRR) of classroom observation instruments as a measure of fidelity of implementation (FOI) in the context of a large-scale, federally-funded, yearly-long randomized study targeting English learners and their teachers’ instruction. We utilized two different instruments and demonstrated the following steps: (a) establishing initial IRR, (b) continuously monitoring IRR, and (c) re-establishing IRR when it fell below the pre-determined cut-off value at each FOI checkpoint from 116 bilingual classrooms across seven school districts in Texas. The on-going training, coding procedures, and calculations of IRR across both treatment and control conditions were closely examined. While this project was equipped with extensive resources and a sophisticated study design, we have also addressed guidelines for establishing and maintaining IRR that can be applied across various designs in under-resourced programs to measure FOI through classroom observation.