Evaluating the Implementation Quality of a Social and Emotional Learning Program: A Mixed Methods Approach

Katherine Dowling,Margaret M Barry

doi:10.3390/ijerph17093249

Abstract

School-based social and emotional learning (SEL) programs have been shown to be effective in producing positive outcomes for adolescents. However, variability in implementation quality can have a negative impact on these program effects. The aim of this current study is to examine the variability in implementation quality for schools implementing the MindOut program and to identify factors that were likely to contribute to this variability. Employing a mixed methods approach, quantitative and qualitative implementation data were collected from teachers (n = 16) and students (n = 280) who participated in the MindOut program. Quantitative indicators were used to score schools’ implementation quality across four dimensions (dosage, adherence/fidelity, quality of delivery and participant responsiveness), and these were averaged to determine overall level of implementation (high/low). Qualitative data identified factors that contributed to implementation quality, and factors were then analyzed in accordance with the schools’ implementation level grouping. Findings indicated that variability in implementation quality existed both between and within schools. A total of eight schools were assigned as high implementers and another eight as low implementers. Influencing factors were categorized into five themes: (i) program factors, (ii) participant factors, (iii) teacher factors, (iv) school contextual factors, and (v) organizational capacity factors. Several differences between high and low implementers were found in relation to these influencing factors. The findings contribute to the evidence on implementation quality in schools by advancing knowledge on measuring implementation quality across multiple dimensions and informants successfully. These findings can also inform practitioners of the main influencing factors in schools so that strategies can be developed to optimize implementation quality in the future.

Highlights

Well-designed evidence-based social and emotional learning (SEL) programs in schools have been shown to be effective in promoting positive outcomes for students [1,2,3,4]
This study set out to assess the variability in implementation quality of schools delivering the MindOut program and to investigate the factors that most likely contributed to this variability, in relation to schools’ implementation level
This study demonstrated an effective way of assessing implementation quality of a school-based program by scoring schools on a range of indicators and combining these scores to produce a total implementation score for each school

Summary

Introduction

Well-designed evidence-based social and emotional learning (SEL) programs in schools have been shown to be effective in promoting positive outcomes for students [1,2,3,4]. Variability and fragmentation in implementation quality can lead to inconsistencies in outcome achievement, especially when evidence-based programs are scaled up and delivered outside of controlled research conditions [5,6,7,8]. In order to accurately interpret the effectiveness of a program, it is necessary to understand how implementation quality varies between sites by answering questions on how much, how well and which aspects of the program were delivered. Public Health 2020, 17, 3249; doi:10.3390/ijerph17093249 www.mdpi.com/journal/ijerph

Objectives

Methods

Results

Discussion

Conclusion