Measurement Of Implementation Fidelity Research Articles

Data based individualization in early writing: the importance and measurement of implementation fidelity

In this paper we describe the process of monitoring fidelity of implementation for a teacher-implemented early writing intervention. As part of a large, federally funded project, teachers who worked with students in grades 1 through 3 in schools across two states in the US were recruited and then randomly assigned to implementation and control conditions. Using Data-Based Individualization (DBI) as a framework for best practice in assessment and intervention, teachers in the implementation group received professional development on early writing intervention and assessment and then implemented these practices with their students who had significant writing challenges. Coaches, who were part of the research project, supported teachers and also observed teachers in both the implementation and control conditions at least twice during the course of the 20-week study. This paper focuses on the results of the fidelity measures that were administered throughout the project. An overview of the importance of fidelity checks is followed by a description of the fidelity tools used, as well as data from those tools. Areas of strength and challenge for teachers when implementing early writing assessment and intervention and engaging in data-based decision making with fidelity are discussed, along with recommendations regarding the practical and research importance of fidelity checks.

How Do Intervention Studies Measure the Relation between Implementation Fidelity and Students’ Reading Outcomes?

Implementation fidelity is often thought of as a necessary condition to achieve internal validity and as having a relation to student outcomes. This relation is increasingly modeled; we reviewed reading intervention studies for students in pre-K–12 in which measures of implementation fidelity were included in final data analysis. A systematic search yielded 50 studies using mostly measures of dosage, adherence, and quality. We provide a detailed, narrative description of the construction of the measures, revealing a wide variety of conceptualizations of implementation fidelity, in terms of content, measurement approaches, and variable construction. The original studies reported varied estimates of the relation between implementation fidelity and student outcomes. To better study the relation in the future, we recommend researchers incorporate measurement of implementation fidelity in the design phase of research projects.

A Synthesis of the Conceptualization and Measurement of Implementation Fidelity in Mathematics Intervention Research

In intervention studies, high rates of implementation fidelity are important markers of a study’s success; however, the definition of implementation fidelity is both complex and dynamic. In this synthesis, we examined the dimensions of implementation fidelity measured and reported in 99 studies in which researchers utilized a mathematics intervention for elementary students (i.e., Grades 1–5). We examined implementation fidelity following recommendations made by Dane and Schneider (1998), O’Donnell (2008), and DeFouw et al. (2009) to capture a comprehensive representation of the implementation fidelity data collected and reported within mathematics intervention studies. We organized our conceptualization of implementation fidelity into four overarching categories and nine dimensions within those categories: intervention design (i.e., theories of change and logistics), fidelity of implementor (i.e., adherence, quality of delivery, dosage, and implementor knowledge or experience), student engagement, and treatment analysis (i.e., treatment differentiation and analysis of implementation fidelity). Overall, findings indicate many author teams reported adherence data and dosage data, but significantly fewer studies reported quality of delivery data, student engagement data, or treatment differentiation data. In addition, author teams were more likely to report at least one form of logistics, such as implementor support, than theories of change. Implications for research and practice are discussed.

Preliminary development and validation of the Interconnected Systems Framework-Implementation Inventory (ISF-II).

As schools increasingly implement multitiered systems of support, there is a pressing need to develop psychometrically sound implementation fidelity measures. The interconnected systems framework (ISF) is a multitiered model blending systems of positive behavioral interventions and supports with promotion, prevention, and intervention strategies of school mental health. The ISF is being implemented in communities across the United States with ongoing evaluation in several randomized controlled trials. The ISF-Implementation Inventory (ISF-II) was developed to measure fidelity of the ISF within a school building. We conducted a national validation study including completion of the ISF-II by 398 educators in 49 schools, 16 school districts, and 9 states. Results indicate the ISF-II produces scores that are internally consistent and structurally valid when items are organized into a three-tiered model. Additionally, the ISF-II was rated as feasible, acceptable, and beneficial. Limitations of the study, including the need for additional psychometric testing, are discussed in light of these results that suggest educators and researchers, alike, should feel confident in using the ISF-II as a measure of ISF implementation quality. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

An examination of the association between MTSS implementation fidelity measures and student outcomes

PBIS is has been consistently correlated with reductions in student exclusion including suspensions, expulsions, poor attendance, and high school dropout rates. However, school-wide strategies that do not specifically involve effective instruction in academic areas are unlikely to result in increased academic achievement. To address this reality, multi-tiered systems of support (MTSS) involving tiered intervention for both academic and behavior have become commonplace. The Academic and Behavior Response to Intervention School Assessment (ASA) was developed to assess the fidelity with which schools are implementing MTSS for Reading, Mathematics, and behavior. Using the ASA to assess MTSS fidelity across 29 schools and four years, analyses were conducted to determine the predictive validity of sub-group domain scores. The question was whether ASA scores were predictive of student outcomes in terms of suspension and of state academic achievement scores in the areas of reading, math, and language. Results show that schools with higher fidelity in the behavior domain had significantly fewer suspension events than matched comparison schools. In comparison, higher fidelity in the reading domain was associated with more students at or above proficient on both the Language Mechanics measure and the Mathematics measure, but not in Reading; and higher fidelity in the math domain was also associated with more students at proficient or above on the Language Mechanics, but not in math or reading. Results are discussed in terms of implications for the further development of fidelity assessments and future research.

Examining the Relative Utility of PBIS Implementation Fidelity Scores in Relation to Student Outcomes

There are several widely used Positive Behavioral Interventions and Supports (PBIS) implementation fidelity measures, including the Schoolwide Evaluation Tool (SET), but there is limited empirical investigation substantiating the association between specific fidelity cut point scores and student outcomes. This article examined different SET cut points in reference to school-level outcomes across 180 elementary, middle, and high schools participating in randomized trials of PBIS. Specifically, we examined the implementation fidelity achieved in the sample, as well as within each school level (i.e., elementary, middle, and high). Using the binary complier index, we examined the association of each of three SET cut points with behavioral and academic outcomes using t tests and receiver operating characteristic (ROC) curve analyses. Although the t tests did not provide confirmation of one specific cut point, post hoc analyses suggested that the cut point may need to be tailored to school levels. There was moderate convergence between implementation on one scale (i.e., responses to behavioral violations) and academic achievement, based on ROC curve analyses for the 70% and 80% cut points. Implications for research and practice are discussed.

Is implementation of evidence-based interventions in schools related to pupil outcomes? A systematic review

AimsThe growing influence of implementation science has resulted in educational researchers exploring what occurs within schools to support intervention effectiveness. This paper provides an overview of existing research so that practitioners can understand the extent to which measures of implementation are associated with the outcomes of school-based interventions.MethodThis paper systematically identified studies which correlated or directly compared the implementation of school-based interventions with pupil outcomes. Effect-sizes are reported and the strength of evidence appraised using a weight-of-evidence framework.FindingsThe 13 studies reviewed reported 32 quantified effect sizes which represented the strength and direction of the relationship between measures of implementation and intervention outcomes in schools. The review also identified gaps in current evidence which have implications for further research and practice.LimitationsThis review did not explore factors which supported staff to implement interventions effectively. As such, this review focusses on the effects of implementation, rather than detailed practices.ConclusionsThis review found that educational researchers rarely measured fidelity of programme implementation. When fidelity is measured, there are indications that proper execution and co-ordination of evidence-based interventions is positively related to pupil outcomes. However, the measurement of implementation fidelity can be undermined when data is transformed into arbitrary categories, such as ‘good’ and ‘bad’. The practicalities of effectively transporting evidence-based interventions into school settings are discussed.

Teacher-Led Math Inquiry: A Cluster Randomized Trial in Belize

Teacher professional development and in-class mentors were used to support structured inquiry with math manipulatives. Twenty-four primary schools (n = 6,628 students) were randomly assigned to treatment and control groups as an experimental field trial to examine the effectiveness of this instructional approach in a scaled-up application in Belize for the duration of a school year. Implementation fidelity measures were collected permitting evaluation of two separate multilevel models: intention-to-treat and test-of-treatment. Both quantitative and qualitative evidence suggest students within this culture respond well to this relatively simple and inexpensive intervention that departs from traditional, expository math instruction in many developing countries. Policy implications are discussed that supported nationwide rollout of the intervention.

Examining the Impact of ABRACADABRA on Early Literacy in Northern Australia: An Implementation Fidelity Analysis

ABSTRACTTo address students’ poor literacy outcomes, an intervention using a computer-based literacy tool, ABRACADABRA, was implemented in 6 Northern Australia primary schools. A pretest, posttest parallel group, single blind multisite randomized controlled trial was conducted with 308 students between the ages of 4 and 8 years old (M age = 5.8 years, SD = 0.8 years). Findings suggested that computer-based instruction under controlled conditions can improve student literacy, especially for Indigenous students at risk of reading difficulties. The authors examine the fidelity with which the computer-based literacy tool was implemented and the impact of implementation fidelity measures on student outcomes. Student exposure to and use of the literacy tool, and quality of instruction and lesson delivery, were analyzed for their influence on students’ literacy outcomes. Implementation fidelity measures accounted for between 1.8% and 15% of the variance of intervention students’ scores.

News from the NIH: research to evaluate “natural experiments” related to obesity and diabetes

Obesity is a major contributor to many serious health conditions that increase morbidity and reduce quality of life. For example, obesity is a significant risk factor for diabetes, cardiovascular disease, and certain forms of cancer. The prevalence of obesity in children and adults in the USA has dramatically increased in the past four decades [1, 2]. Although some cities and states are showing reductions in rates of childhood obesity, the prevalence is still far above 1970 levels [3–5]. Diabetes currently affects an estimated 25.8 million people in the USA and another 79 million Americans are estimated to be at greatly increased risk of developing diabetes in the next several years [6]. Further, most adults with diabetes in the USA are not meeting the recommended goals for diabetes care [7]. Overweight, obesity, and/or excessive weight gain during pregnancy are also contributing to rising rates of gestational diabetes mellitus which in turn increases risk of future type 2 diabetes in the mother and child. In addition to the considerable impairments to health and quality of life with these conditions, there are also serious economic consequences. The estimated current annual cost of diabetes alone in the USA is $245 billion dollars per year with $176 billion in direct medical costs and the remainder related to reduced productivity [8]. The increasing challenges of obesity and its related health conditions in our society also coincide with a dynamic time for health care in the USA, including changing consumer demands (employers and individuals) and the pressing need to deliver evidence-based care, improve health outcomes for all Americans, and control costs. In communities and in health-care settings, many of the policy and programmatic changes are evidence-informed, but often little is known about population-level effects. Rigorous scientific evaluation of these “natural experiments” can help to more rapidly build an evidence base to inform key stakeholders and policy makers. Evaluation research generally, and rigorous evaluation of natural experiments specifically, is increasingly recognized as an important and appropriate approach to health-related research, particularly in the context of policy and environmental [9, 10] change. The term experiment in natural experiments is something of a misnomer as the implementation of these public health and health-care system policies and programs are not often designed as true experiments. Generally, the term natural experiment refers to an exposure or change that is not directly manipulated by the researcher, but rather the result of policy or program interventions that are varied in their implementation along a number of possible dimensions, such as time, geography, or content. Research that evaluates the real-world implementation of polices and program has some challenges such as the potential for bias, confounding, and threats to casual inference. Despite these limitations, the data from natural experiment research offers unique opportunities to enrich the evidence base and can offer advantages not provided by other research designs, such as improved external validity. This research can provide important data about the impact of a policy or program in real-world settings and populations; see for example the well-known study on the impact of education on health outcomes by Lleras-Muney [11]. Often, these are data that would be very difficult, potentially unethical, and often cost prohibitive to collect in the context of an investigator-initiated randomized experiment [12]. Natural experiment research also allows for the assessment of potential unintended consequences or subgroup differences in response to the program or policy and measurement of implementation fidelity, variability, and process. Since 2010, several institutes and centers (ICs) at the National Institutes of Health have jointly issued funding opportunity announcements intended to support rigorous evaluation of natural experiments related to obesity and/or diabetes outcomes, including the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK); National Cancer Institute (NCI); Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD); National Heart, Lung, and Blood Institute (NHLBI); National Institute on Aging (NIA); and the Office of Behavioral and Social Sciences Research (OBSSR). In these funding opportunity announcements, policies are broadly defined to include public policies at local, state, and federal levels of government and organizational level policies, such as those implemented by large organizations, worksites, or school districts. Polices can be formal such as laws and regulations or informal such as guidelines and procedures. Program is defined as a set of activities initiated by governmental or other organizational bodies to enhance obesity prevention and control.

Parenting Intervention Outcome Studies: Research Design Considerations

SYNOPSIS This tutorial provides an overview of major considerations in the design and implementation of research on outcomes of parenting interventions. It is offered in the context of escalating interest in the use of rigorous research designs for determining effective strategies for improving parents’ contributions to the well-being of their children. The tutorial provides guidance on preliminary work for conducting an outcome study, sampling, research designs that permit causal inference, options for comparison conditions, the measurement of implementation fidelity and parent/child outcomes, and basic considerations in outcome analyses.

Methodological Issues in Child Welfare and Children’s Mental Health Implementation Research

Methodological Issues in Child Welfare and Children’s Mental Health Implementation Research

Toward the Effective and Efficient Measurement of Implementation Fidelity

Implementation science in mental health is informed by other academic disciplines and industries. Conceptual and methodological territory charted in psychotherapy research is pertinent to two elements of the conceptual model of implementation posited by Aarons and colleagues (2010)--implementation fidelity and innovation feedback systems. Key characteristics of scientifically validated fidelity instruments, and of the feasibility of their use in routine care, are presented. The challenges of ensuring fidelity measurement methods are both effective (scientifically validated) and efficient (feasible and useful in routine care) are identified as are examples of implementation research attempting to balance these attributes of fidelity measurement.

Dosage Effect and the Efficacy of a Video-Based Teamwork-Building Series with Urban Elementary School Children

This study examines the role of implementation dosage in the effectiveness of a school-based social and emotional skills-building curriculum, the Talking with TJ video series, among urban, predominantly minority elementary school children. It was hypothesized that children in higher dosage schools will exhibit greater improvements in their social skills and reductions in their problematic behaviors than children from low dosage schools. It was also expected that children in higher dosage schools will report more positive feelings about their self-concept. Results from 655 second and third graders from 6 elementary schools in an urban district confirmed the first hypothesis. After controlling for pre-assessment differences between the two dosage groups, ANCOVAs confirmed that children in high implementation groups had higher ratings of social skills and lower ratings of problem behaviors during post-assessment period while their counterparts received higher ratings of problem behaviors and lower ratings of social competence. Findings indicate differential effects on social skills and problem behaviors. There were no significant differences in students' self-concept between the two dosage groups. It was concluded that subsequent research on school-based social competence programs should incorporate implementation fidelity measures, such as implementation dosage in assessing a program's effectiveness.