Advancing the Science of Direct Observation in Emotional and/or Behavioral Disorders Research: Reliability and Unified Validity

Abstract

* Evidence of functional and causal relationships between independent and dependent variables is contingent upon accurate, valid, and reliable measurement (Kazdin, 2011; Shadish, Cook, & Campbell, 2002). Standards for reliability and validity are well established in education (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 1999), but the application of many of those standards to direct observation is limited (Hintze, 2005), perhaps for reason (Baer, Harrison, Fradenburg, Peterson, & Milla, 2005). As an example of differences in psychometric criteria between direct observation and other generalized measurement approaches (e.g., rating scales, knowledge tests), review the special education quality indicators for single-subject design research, which is based in principles of applied behavior analysis and relies on systematic direct observation (Horner et al., 2005), and group experimental and quasiexperimental design research (Gersten, Baker, Haager, & Graves, 2005). Horner and colleagues indicate that dependent variables must be operationally defined and assessed for consistency using interobserver agreement, while Gersten and colleagues indicate that dependent variables must have evidence of internal consistency and concurrent validity, constructs rarely addressed in direct observation studies.The validity and reliability of scores resulting from either direct observation or other measurement approaches are necessary requirements of good science regardless of experimental design (i.e., single-subject, group experimental). A mismatch between established professional standards of generalized outcomes and direct observation, for example the within-subject repeated single observation scores that violate assumptions of traditional psychometric methods (e.g., assumption of independence inherent in many inferential approaches), is not license to apply direct observation without evidence of accuracy, reliability, and validity. As a remedy, the field of applied behavior analysis established criteria for determining reliability and validity of direct observation methods decades ago, namely interobserver agreement, transparent procedural descriptions, face validity based on complete reporting of operational definitions, and social validity of target behavior (Cooper, Heron, & Heward, 2008; Kazdin, 2011), and the field continues to refine and advance those methods (e.g., Wirth, Slaven, & Taylor, 2014). These criteria are accepted across disciplines and meet federal requirements of high quality research (e.g., Kratochwill et al., 2010), yet challenges remain with direct observation approaches in natural settings, particularly classrooms, including item scaling, observation procedures, and instrumentation (Horovitz & Matson, 2012; McComas, Vollmer, & Kennedy, 2009). To date, little has been written about these concerns or the advances that have been made addressing them in direct observation approaches in research supporting the needs of students with emotional and/or behavioral disorders (EBD) in schools and classrooms.This special issue of Behavioral Disorders is dedicated to addressing challenges and concerns related to direct observation measurement systems in school and classroom settings supporting the needs of students with EBD and describing advances in validity and reliability of direct observation results. The papers herein have been selected as means of highlighting three basic goals for this issue. The initial goal is to present an overview of basic issues related to direct observations in natural settings with students with or at risk for EBD and to provide general recommended considerations for reliable, valid, and generalizaba outcomes. The second goal is to provide a summation of the work that has been done in this area and to draw conclusions as to what we know and where research needs to go. …