Increasing the Rigor of Procedural Fidelity Assessment: An Empirical Comparison of Direct Observation and Permanent Product Review Methods

Abstract

Although it is widely accepted that procedural fidelity data are important for making valid decisions about intervention effectiveness, there is little empirical guidance for researchers and practitioners regarding how to assess procedural fidelity. A first step in moving procedural fidelity assessment research forward is to develop a comprehensive understanding of the relation between currently recommended assessment methods, systematic direct observation (SDO) and permanent product review (PPR). To this end, we first discuss methodological issues related to procedural fidelity assessment via SDO and PPR, and how they are evidenced in the behavioral intervention outcome literature. Second, we present a single-case design study in which teachers’ implementation of behavior support plans was assessed via SDO and PPR. Third, we analyze the SDO and PPR data with regard to (a) relation between the methods; (b) methodological issues including definitions of procedural fidelity and intervention steps, representativeness of intervention steps, and collection of an adequate sample; and (c) association with student outcomes. Finally, we discuss implications for research and practice.