PROTOCOL: Strategies for scaling up the implementation of interventions in social welfare: protocol for a systematic review.

Abstract

Social welfare organisations provide care, support and protection to children or adults at risk of, or with needs arising from, mental illness, disability, age and poverty. There is increasing social welfare sector interest in integrating effective practices into the array of services that are provided to citizens by both public and private provider agencies (Montero, 2015; Supplee & Metz, 2014). Yet maximising the benefits of effective interventions addressing the needs of children, adults and families in adversity requires these interventions to be implemented at scale, providing high quality interventions to large numbers of individuals who require various types of support. Social welfare interventions, even if found to have a treatment effect, can fail due to poor implementation. That is, recipients of a potentially effective service can only benefit from a service they actually receive. However, the implementation of potentially effective interventions in social welfare is still sporadic, localised, and can have considerable geographic variation (Walker et al., 2016). There are numerous reasons for uneven uptake. Barriers in social service organisations and their workforce may hamper the systematic uptake of effective practices due to a lack of training and support (McBeath et al., 2015; McBeath & Austin, 2014). Existing attitudes towards and habits for utilising research in social work practice (Wutzke et al., 2016; Kreisberg & Marsh, 2016; Knight, 2013; Smith, 2013) may have the same effect due to the general complexity that characterises processes of evidence integration in routine service settings (Carnochan et al., 2017). There is also evidence to suggest that high quality implementation of even modestly effective programs can result in better outcomes than the poor implementation of programs that have been found to be highly effective (Lipsey, 2009; Lipsey, Howell, Kelly, Chapman & Carvin, 2010). Understanding how interventions can be scaled up from single trials and local innovation projects to fully implemented practices and programs that reach their intended entire population is a priority within social welfare. ‘Scaling up’ or the concept of ‘scalability’ has been variously defined in the literature. Perhaps the most widely used definition of ‘scaling up’ is that provided by the World Health Organization (WHO) in 2009 and since adopted by other agencies within health and human services including the recently published European Scaling Up Strategy in Active and Healthy Ageing (European Commission, 2015). Specifically, WHO defines ‘scaling up’ as “deliberate efforts to increase the impact of health service innovations successfully tested in pilot or experimental projects to benefit more people and to foster policy and programme development on a lasting basis” (WHO, 2009: 1). Scalability is defined as the the ability of a health intervention shown to be efficacious on a small scale and / or under controlled conditions to be expanded under real world conditions to reach a greater proportion of the eligible population, while retaining effectiveness (Milat et al., 2012). At its core, the concept includes an explicit intent to expand the reach of an intervention. Implementation – “the use of strategies to adopt and integrate evidence-based health interventions and to change practice patterns within specific settings” (Glasgow et al., 2012, p.1275) – is a key feature of the process of scaling up as sufficient implementation is necessary for scaled interventions to be beneficial. As such, all implementation strategies could be considered relevant to scaling interventions. The Expert Recommendation for Implementing Change Project, for example lists 73 implementation strategies including providing technical assistance using an implementation advisor, and mandating change (Powell et al., 2015). However, the process of scaling interventions includes consideration of a range of pre-conditions and strategies that typically precede implementation efforts, including constituency building and realigning resources and infrastructure to enable delivery at scale (Milat et al., 2015). There have been a number of examples of attempts to scale-up implementation of social welfare interventions. The Parent Management Training Oregon Model is a curriculum-based training program for parents that seeks to reduce or prevent child externalising behaviour problems and is typically provided over 25 one-hour sessions by trained therapists. Following positive findings from two randomised trials, PMTO was scaled up across Norway. Implementation was supported by investment in a national centre to establish implementation infrastructure. Implementation support strategies included therapist training, the establishment of regional therapist networks, on-site coaching, monitoring, accreditation processes and technical support. Evaluation of the initiative suggests that the program fidelity was maintained, and effects of the program were comparable with those reporting in efficacy trials (Tommeraas et al., 2017). Brown and colleagues (2014) report a randomised trial of comparing two different approaches to the implementation of Multidimensional Treatment Foster Care - an evidence-based intervention involving the placement of children in supported community-based foster care rather than aggregated care settings. Implementation strategies delivered to one group included technical support, face to face stakeholder meetings, the development of action plans, training of administrators, supervisors and therapists and parents, and on-going consultation. A comparison group received the same implementation support strategies but also received additional technical assistance from program consultants during peer to peer meetings and monthly conference calls and facilitating communities of practice (quality improvement collaboratives). The trial identified little difference in implementation between groups at follow-up (Brown et al., 2014). Several frameworks have been published describing approaches to scale interventions. Such frameworks demonstrate that interventions to implement innovations at scale are complex, and involve the consideration of a number of individual, organisational, social, political and other contextual factors (Barker et al., 2016; Milat et al., 2016; Seay et al., 2015; Atkinson et al., 2013; Kohl & Cooley, 2003). Interventions to achieve implementation at scale, therefore, are not uniform, and are likely most effective if they consider and address factors that impede achievement of large scale implementation. While some already consider the small-scale transport of effective interventions into real life settings (e.g., through a single project or trial site) as a part of scaling up effective practice (Dunlap et al., 2009), this review will focus on system-, state-, nation-based or global scale up efforts aiming to integrate effective interventions within social welfare to entire relevant populations. Implementation science is the study of methods and strategies to promote the uptake of interventions that have proven effective into routine practice, with the aim of improving population health1. ‘Scaling-up’ is a broadly acknowledged concept within implementation science (Dymnicki et al., 2017; Hoagwood et al., 2014; Milat et al., 2012; Norton & Mittman, 2010), and several strategies have been suggested as effective by different researchers based on multiple studies. Among scale up strategies mirrored in this literature are e.g., ‘research-practice collaborations’ (Chamberlain et al., 2012), advocacy and stakeholder activation; resource allocation; capacity building (Resnick and Rosenheck, 2009; Hurlburt et al., 2014); system restructuring (Eaton et al., 2011), use of business practices and technologies; quality assessment and evaluation (Hoagwood et al., 2014); interagency collaboration (Aarons et al., 2014; Hurlburt et al., 2014) and the adaptation and maximisation of fit between intervention and context (Klingner, Boardman & Macmaster, 2013). However, a set of commonly agreed upon strategies or actions necessary to include in effective scaling efforts has not emerged from this literature. Nonetheless, frameworks consolidating behavioural and organisational theories that have been applied to improve the scale of implementation of interventions have identified a number of factors that can impede or facilitate scaling. For example, a recent review of frameworks (Atkinson et al., 2013) identified a range of factors that need to be in place prior to larger scale roll-out of interventions including leadership structures, infrastructure, and alignment between intervention goals and organisational priorities. During the implementation phase, however, facilitators include attention to changing organisational culture., equipping frontline staff with tools for problem solving, and implementation monitoring (ibid.). While the mechanisms of effects of scale-up strategies differ based on the contexts and determinants of implementation, best practice approaches recommend formative evaluation to identify scale-up barriers and developing strategies to address them, thereby enhancing the likeliness of increasing their impact. The heterogeneity mirrored by just these few articles on questions of scale up, highlights that it is difficult at the current developmental stage of implementation science to – a priori – identify a standardised set of individual, organisational and system strategies that researchers have agreed upon as necessary to include in any type of scaling attempt. A systematic review aiming to synthesise the current best evidence on the effectiveness of strategies to scale up potentially effective social welfare interventions therefore can help inform both practice and research environments with a strong focus on the implementation of effective services. A small number of systematic reviews address implementation in general within health and education without addressing particular questions of scaling (Greenhalgh et al., 2004; Francke et al., 2008; Chaudoir et al., 2013; Gibson et al., 2015; Naylor et al., 2015). Fewer still specifically focus on scale up. In examining the literature on effective dissemination and implementation interventions to support cancer prevention, Rabin et al. (2010) point to the high level of heterogeneity in language, study population & design, measures and other study characteristics, making it difficult to draw firm conclusions about effective strategies. Through a rapid review, Atkinson et al. (2013) succeeded in identifying 21 frameworks for facilitating large-scale changes in health services. Based on the seven of these that had been applied in quality improvement initiatives, the authors derived 13 key factors important to large scale change. Milat et al. (2015) use a similar approach in conducting a narrative review of eight frameworks for scaling health interventions. They point to several common cross-framework characteristics, one of which is the requirement to have a well-defined scale up strategy. However, the steps involved in developing such a strategy are again highly different from framework to framework. With a focus on the UK, Pearson et al. (2015) examine the conditions and actions supporting the implementation of health promotion programmes in schools. They conclude that while the literature provides insights into some aspects of programme implementation, the knowledge about several implementation practices remains underdeveloped, among others because studies often do not reach the scaling stage of implementation due to limitations put on funding. Finally, strategies to effectively implement policies and practices to improve child health through childcare services were the focus of a systematic review by Wolfenden et al. (2015). Like other publications, the authors point to weak and inconsistent evidence in this area, making it impossible to point to particularly effective strategies for high quality large scale implementation. Similarly, very few systematic reviews in social welfare focus on dissemination and/or implementation in general (Novins et al., 2013; Leeman et al. 2015). We are not aware of any systematic review, completed or in progress, that specifically focuses on scale up in a social welfare context. Further, social welfare agencies have specific values, governance structures, resources, workforce capacity and cultures that are different from those present in health services. As such, contextual factors are important considerations in the process of scaling effective interventions (Milat et al., 2015), and the findings of existing reviews may not generalise to the social welfare sector. Given this lack of clarity around scaling terminology and the scarcity of knowledge about effective scaling, this systematic review has the potential to contribute to: The primary objective of this review is to assess the effectiveness of strategies aiming to support the scale-up of interventions in social welfare. Hence, the research question guiding this systematic review is: Among social welfare services, how effective are strategies seeking to improve the implementation of effective interventions, programs or services at scale? Additionally, the review seeks to describe the context in which implementation at scale occurs. Given the potentially complex nature of evaluation studies of implementation trials, we will include a broad range of study designs in this review. While randomized controlled trials (RCTs) are the most internally valid design to assess the effectiveness of a scaling strategy, such designs may not always be the most appropriate research designs for evaluating the impact of such strategies. For example, there may be too few allocation units available for baseline equivalence when measuring scaling strategies that target large geographic regions, such as provinces, counties, states or nations. Furthermore, given that implementation science (more broadly) and scale up (more specifically) are relatively new fields of science, large numbers of randomised trials are unlikely, and the inclusion of non-randomised trials may increase the pool of available studies, providing a more comprehensive evidence-base for policy and practice decision-making. We therefore will include any study that uses one of the following designs: Qualitative and other uncontrolled observational studies may be useful for understanding why strategies succeed or fail, but this review is focused on establishing whether strategies are effective. As such, we will only include quantitative studies that Participants could include any social welfare organisation that provides care, support and protection services to children, adults, families and communities that are at risk of or already require support due to adversities arising from mental illness, disability, age or poverty. This includes social welfare organisations operating in the areas of child welfare and child protection; mental health and substance abuse; juvenile justice; housing; aged care and employment. Included are both the internal stakeholders to social welfare organisations - their staff, clients, and administrators - and their external stakeholders responsible for e.g. their financing, regulation and development. This group includes representatives for e.g. government bodies, regulatory agencies or for intermediaries that have a capacity to provide service agencies with professional supports and technical assistance. Studies in which only a subset of the sample is eligible for inclusion – e.g. if a study covers both social welfare and health organisations – will be excluded. A number of reviews have been conducted in hospitals and primary care settings to assess strategies to scale evidence-based health services. However, operating contexts, processes and structures of medical settings differ considerably from social welfare organisations, and as such, the effects of implementation approaches may not generalise across these settings. This systematic review, therefore, focuses on strategies aiming to scale up the implementation of discrete, potentially effective social welfare interventions in federal, state, community and individual settings including social assistance offices, community based mental health clinics, neighbourhood initiatives, individual households, and individuals within households. Studies of strategies to scale interventions in medical settings such as hospitals or general practice will be excluded, as will those in educational settings such as schools or universities. We will include trials of any strategy that seeks to increase the scale of implementation of social welfare interventions. A range of potential strategies could be used to improve implementation of social welfare interventions at scale. (Powell et al., 2012 & 2015). For example, the Cochrane Effective Practice and Organisation of Care Taxonomy (EPOC, 2016) lists strategies pertaining to delivery arrangements (e.g. co-ordination of different providers), financial arrangements (e.g. pay for performance – targeted payments), governance arrangements (e.g. decentralisation of authority for health services) and implementation strategies (e.g. audit and feedback). Implementation strategies have also been characterised by the Expert Recommendation for Implementing Change (ERIC) Project, and scale-up frameworks suggest a range of strategies that can be enacted, before during and following the implementation phases of the scale-up process. We will include both individual strategies and combinations of strategies. Additionally, to be included, studies are required to: Potentially effective programs of services will be defined as those that have been evaluated in at least one randomised controlled trial where at least one primary trial outcome had a statistically significant (P<0.05), positive effect favouring the intervention and no substantial adverse effects of the intervention (i.e., not the scale up) were reported. Randomised designs were selected as they provide good evidence to support effectiveness in evidence hierarchies (Evans, 2003). If the intervention, service or program has been subject to repeated assessment using randomised designs, pooled effect estimates on the primary trial outcome of the social welfare intervention must represent a significant improvement on the primary trial outcome (p>.05) relative to control or comparison group. Or - if the pooled estimate was unavailable or impossible to be calculated - the findings of the trial judged to represent the most valid estimate of effect will be used, based on consensus of two review authors and using the Cochrane Risk of Bias assessment tool. To be eligible, trials must include a measure of implementation fidelity – the primary outcome of this review. From such studies, data pertaining to secondary trial outcomes will also be assessed. We will include trials reporting post intervention follow-up data only if baseline values can be assumed to be zero, as would be the case for trials attempting to scale up implementation of an intervention, program or service that was not available prior to study commencement, or if the trial employed a randomised design, where by baseline equivalence can be assumed (or differ only due to chance). There will be no exclusion criteria on the source of outcome data. Data for the primary and secondary outcome measures can be obtained from any course including institutional records, direct observations, surveys or questionnaires completed by social welfare organisation staff or clients. The included primary and secondary outcome measures and their operational definitions have been based on the heuristic and working taxonomy of outcomes proposed by Proctor et al. (2011). The included measures are also inclusive of factors recommended in different evaluation frameworks, including the RE-AIM (Reach Effectiveness Adoption Implementation Maintenance) framework (Glasgow et al., 2009; Gaglio et al., 2013). Primary outcomes: Secondary outcomes: We will include any measure of unintended adverse effects from strategies to increase the scale of implementation of potentially effective social welfare interventions for either individuals, or social welfare organisations. These could include adverse changes to the moral of staff or their working conditions, displacement or defunding of other potentially effective interventions with greater empirical support or worsening of conditions of service recipients due to poor, incorrect or unsafe implementation practices. All adverse effects described in eligible studies will be included in the synthesis. There will be no restriction on the length of the study follow-up period. This systematic review will focus on the scaling of social welfare interventions in federal, state, community and individual settings including social assistance offices, community based mental health clinics, neighbourhood initiatives, individual households, and individuals within households (see also 3.1.2). Studies of strategies to scale interventions in medical settings such as hospitals or general practice will be excluded, as will those in educational settings such as schools or universities. There will be no restriction on the language of publication. We will identify suitably qualified translators for studies identified published in non-English languages. We will include studies in the peer reviewed and grey literature. Any changes in eligibility criteria will be agreed prospectively between the members of the review team. These will be documented and reported as a discrepancy from protocol in the manuscript. In the advent of a change in eligibility, we will re-screen citations. The following electronic databases will be searched: Medline, Embase, PsycInfo, Cochrane Central Register of Controlled Trials (CENTRAL), CINAHL, Education Resources Information Center (ERIC), International Bibliography of the Social Sciences, Applied Social Science Index and Abstracts (ASSIA), Sociological Abstracts, Social Services Abstracts, Web of Science incl. Social Sciences Citation Index and Conference Proceedings Citation Index- Social Science & Humanities, Campbell Collaboration, and Criminal Justice Abstracts. Search terms will be developed based on terminology representative of implementation and dissemination research (Rabin, 2008) and include search filters used in previous reviews (Rabin 2010; Wolfenden 2014). The search strategy for Medline is presented in Appendix 1 and will be adapted for the other databases by an experienced librarian, who also will conduct the searches. There will be no date of publication or language restrictions. Protocols for all included trials will be sourced. Reference lists of all included trials will be hand-searched for citations of other potentially relevant trials. Previous reviews on the topic will be screened for relevant studies. Targeted hand searches of all publications in the journal Implementation Science will be conducted. Forward citation searches of included studies will be performed using Google Scholar. Furthermore, we will conduct searches of the grey literature. We will develop a separate search strategy including all of the following elements: We will document all steps of the search process in sufficient detail to ensure future replicability and correct reporting. This will include a PRISMA flowchart, registration of excluded studies and dates at which the search was conducted. If the initial search date is more than 12 months from the intended publication date, we will rerun searches and fully incorporate new eligible studies. A broad range of methods can be expected to be used as part of the studies to be identified for this review. They may include the collection of data through archives and administrative databases, surveys, standardised inventories and measures, documents, interviews etc. Two reviewers, who will not be blind to the author or journal information, will independently screen abstracts and titles. Screening of studies will be conducted using a standardised screening tool developed based on the resources for authors of systematic reviews available through the Cochrane EPOC group (EPOC, 2015), and will be piloted before use. The full texts of manuscripts will be obtained for all potentially eligible trials for further examination. For all manuscripts, information regarding the primary reason for exclusion will be recorded and documented in the excluded studies table. The remaining eligible trials will be included in the review. For these, any relevant retraction statements and errata will be examined for information. Discrepancies between reviewers regarding study eligibility will be resolved by consensus. In instances where the study eligibility cannot be resolved via consensus, a third reviewer will make the final decision. Two review authors, also unblinded to author or journal information, will independently extract information from the included trials. This information will be recorded in a data-extraction form that will be piloted before initiation of the review. Discrepancies between reviewers regarding data extraction will be resolved by consensus or if required via a third reviewer. One reviewer will transcribe information from data extraction forms into Rev Man meta-analysis software and included study tables. Data transcription will be checked by a second reviewer. The following information will be extracted: As part of data extraction, we will check the accuracy of all numeric data in the review. Where information is unavailable from published reports we will contact study authors to obtain such data. Multiple reports of the same study will be collated to ensure that each study rather than each report is the unit of interest in the review. Risk of bias will be assessed independently by two reviewers using the risk of bias tool described in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins & Green, 2011). The tool provides an overall risk of bias (‘high’, ‘low’ or ‘unclear’) assessment for each included study based on consideration of study methodological characteristics, random sequence generation, allocation concealment, protection against contamination, blinding of outcome assessment, baseline outcome, baseline characteristics, selective outcome reporting, missing outcome data and other risks of bias. Judgements made will be justified with information included in studies or related documents. If these documents are not publicly available, this will be explicitly stated. If required, a third reviewer will adjudicate discrepancies regarding the risk of bias that cannot be resolved via consensus. An additional criterion ‘potential confounding’ will be included for the assessment of the risk of bias in non-randomised trial designs (Higgins & Green 2011). Risk of bias for included studies will be documented in a ‘Risk of Bias’ table. In accordance with Campbell and Cochrane guidelines, we will try to maximise the likelihood to quantitatively synthesize studies. For the primary and secondary outcomes, attempts will be made to conduct meta-analysis using data from included trials. For binary outcomes, the standard estimation of the risk ratio and a 95% confidence interval will be calculated. For continuous data, the mean difference will be calculated where a consistent measure of outcome is used in included trials. Where different measures are used to examine the primary outcome, the standardised mean difference will be calculated where possible. Where data from the same outcome are reported, in some studies as dichotomous data and in others as continuous data, we will transform these to enable pooled estimates of effect if it is appropriate to do so. Where outcome data are not presented in 2X2 tables or are not presented in means and standard deviations, we will attempt to transform available data into a usable effect size using the online calculator Practical Meta-Analysis Effect Size Calculator (Wilson, n.d.). If studies using different scales are combined, we will ensure that higher scores for continuous outcomes all have the same meaning for any particular outcome. Specifically, we will explain the direction of interpretation and report when reversing scores to align direction is done. Finally, we will check continuous outcome measures for skewness and, if substantial departures from normality are observed, we will transform these data prior to meta-analysis. If we are unsuccessful at transforming the data, we will attempt to contact the author of the study and request additional data. The overall quality of evidence for each outcome will be rated using the GRADE system (Gyatt et al., 2011) by two reviewers, with any disagreements resolved via consensus, or if required by a third reviewer. The GRADE system defines the quality of the body of evidence for each review outcome, describing the extent to which one can be confident in the review findings. The GRADE system requires an assessment of methodological quality, directness of evidence, heterogeneity, and precision of effect estimates and risk of publication bias. GRADE quality ratings (from ‘very low’ to ‘high’) will be used to describe the quality of the body of evidence for each review outcome. All assessments of the quality of the body of evidence (e.g. downgrading or upgrading) will be justified and documented. Clustered trials will be examined for unit of analysis errors. Trials with unit of analysis error will be identified in the risk of bias table. For cluster-randomised trials that have performed analyses at a different level to that of allocation, without appropriate statistical adjustment for clustering, the trials' effective sample size will be calculated for use in meta-analysis. The intra-cluster correlation co-effic