Evaluating the implementation and sustainability of an evidence-based intervention: delivering Triple P within Indigenous child welfare agencies

Abstract

In both Indigenous and non-Indigenous populations, research clearly links evidence-based parenting programs to decreases in child behavior problems, dysfunctional parenting, and child abuse and maltreatment. However, Indigenous families often do not have adequate opportunities to access these programs. This is due to a number of compounding program, service organization, process and interaction factors that are reviewed and evaluated in this thesis. Chapter 1 provides a rationale and overview for evaluating factors that impact initial and long-term use of evidence-based programs (EBPs). It presents the case for developing and evaluating a framework of supports for sustainment of EBPs developed specifically for providers working in disadvantage communities. The research plan presented involves: a systematic review and conceptual framework of barriers and enablers to implementation and sustainment in real-world practice settings; development and validation of a scale measuring inhibitors and enablers to program sustainment; and evaluation of outcomes for implementation and sustainment for providers who are trained in the Triple P – Positive Parenting Program, as an example EBP, both internationally in varied service settings and specifically for Indigenous Australians working in the child protection sector. Chapter 2 presents a review of the literature (submitted for publication) pertaining to provider implementation and sustainment of EBPs with families and communities experiencing disadvantage. Important factors that facilitate success and create barriers to program sustainment are synthesized into key themes. These themes are drawn on to develop the Sustained Implementation Support Framework for EBPs. The need to develop a conceptual framework and a measure to guide and evaluate EBP implementation in community settings is established. Chapter 3 outlines the preliminary validation of a measure, the Sustained Implementation Support Scale, of enablers and inhibitors (program benefits, program burden, workplace support, workplace cohesion and leadership style) to EBP sustainment (submitted for publication). Exploratory and confirmatory factor analysis with a sample of 593 Triple P trained practitioners showed that the model had good fit to the data [χ2 (340) = 736.27, p < .001; CFI = .914; SRMR = .053; RMSEA = .062 90% (CI .056 - .068)] and led to a 28-item scale with good reliability, and good convergent, discriminant and predictive validity. The combined set of predictors explained between 8.3% – 8.9%e (Cox and Snell R2) of the variance in sustained program implementation. This study demonstrated that practitioners sustaining implementation at least three years post training were more likely to have supervision/peer support, reported higher levels of program benefit, workplace support and positive leadership style, and lower program burden compared to practitioners who were non-sustainers. Workplace cohesion was not significantly related to sustained implementation. This highlights the potential benefit of using an evaluation measure to assess service provider perceptions of sustainment inhibitors and enablers to enhance capacity to sustain EBPs. Chapter 4 consists of a paper (submitted for publication) reporting on the use of the Sustained Implementation Support Scale (validated in Chapter 3), to evaluate key workplace, program and process and interaction factors identified in the Sustained Implementation Support Framework (developed in Chapter 2), with (N=35) Aboriginal and Torres Strait Islander family support workers using Triple P in real-world settings. Correlation analyses and binary logistic regression were used to assess the associations between key factors and program implementation (at 18 months) and sustainment (at 36 months) when Triple P is used with Indigenous parents involved with child protection services in Queensland, Australia. This study demonstrated that for implementation at 18 months, as predicted, there was a trend for implementing practitioners to report higher levels of partnership support, program benefit, workplace support and workplace cohesion (weak positive relationship). However, the only significant (moderate positive) relationship was with partnership support (r=.31 p<.05), and the regression analysis indicated that none of the independent variables made a significant contribution to the program implementation model. For sustained implementation at 36 months, as predicted, practitioners that received supportive coaching [OR = 15.63, 95% CI (1.98 – 123.68), p = 0.009] were more likely to sustain the intervention, however the hypothesized relationship between the remaining four factors (program characteristics, workplace support, supervision and peer support and sustainability planning) was not significant. Overall, this suggests further exploration of program burden and perceived program benefit, workplace support and cohesion, and provides evidence for ensuring partnership support and supportive coaching are available to improve the likelihood of EBP program implementation and sustainment in Indigenous child protection services. Chapter 5 concludes by drawing together the findings. The major conclusions are presented, limitations and directions for future research are outlined along with implications for researchers, practice and policy. It is argued that program, workplace and process and interaction factors, including perceptions of program burden, program benefit, workplace support, cohesion and leadership style, partnership support, supportive coaching, and supervision and peer support are important factors to increase the likelihood of EBP implementation and sustainment for communities with disadvantage.