Realist Evaluation Research Articles

S7 - Realist Evaluation of the Impact, Viability, and Transferability of an Alcohol Harm Reduction Support Program Based on Mental Health Recovery: The Vitae Study: First Results

S7 - Realist Evaluation of the Impact, Viability, and Transferability of an Alcohol Harm Reduction Support Program Based on Mental Health Recovery: The Vitae Study: First Results

Exploring how and why a sports-based multi-component intervention works for disengaged students: a longitudinal realist evaluation

ABSTRACT School dropout is associated with numerous detrimental consequences including prolonged unemployment, poverty, a wide range of psychological and physical health problems, and premature mortality. As such, designing and implementing interventions to prevent school dropout and ensure successful school completion is crucial. The purpose of the current study was to conduct a longitudinal realist evaluation to understand how, and under which circumstances, a multi-component intervention delivered through the charitable foundation of a professional sports team may impact the developmental outcomes of disengaged students. The intervention comprised one-to-one mentoring, classroom-based learning, sport and physical activity. Participant observations and fifty-two interviews were conducted with teachers and students over a ten-month period to form context-mechanism-outcome configurations (CMOCs) and to refine initial programme theories. The CMOCs developed highlighted the importance of students developing healthy conflict resolution skills and emotional regulation strategies, the potential of sporting content to re-ignite interest in academic learning, the powerful effects of deviant peer contagion, the synergistic impact of a multi-component intervention, and the role of pre-existing and ongoing contextual factors in determining whether interventions can create sustainable and lasting desirable outcomes among students. The findings provide practical recommendations for future sport-framed intervention design, implementation, and evaluation.

Women’s engagement with community perinatal mental health services: a realist evaluation

BackgroundIn recognition of the burden of Perinatal Mental Health problems, NHS England invested £365 million to transform women’s access to mental health care, including investment in Community Perinatal Mental Health Services. This study examined how elements of provider care affected women’s engagement with these services.MethodsSemi-structured interviews were conducted with 139 women and explored their experiences of care from 10 different Community Perinatal Mental Health Teams; including which service components participants believed made a difference to their initial and continued engagement. Realist analysis was used to create context-mechanism-outcome configurations (CMOCs) across interviews, since not all parts of the configurations were always articulated within singular interviews.ResultsFour key pillars for engagement were identified: perinatal competence, relationship building, accurate reassurance, and reliability. The way perinatal competencies were relayed to women mattered; compassion, understanding and consistency were critical interactional styles. The extent to which these factors affected women’s engagement varied by their context and personal characteristics.ConclusionsAs mental health problems increase, disproportionately affecting vulnerable populations, it is critical to continue to ensure support is not only available, but appropriately meets the needs of those individuals. Our findings suggest that key staff behaviours applied at the right time can support women’s engagement and potentially contribute to better treatment outcomes.

How does mentoring occupational therapists improve intervention fidelity in a randomised controlled trial? A realist evaluation

BackgroundIntegrating complex interventions within healthcare settings can be challenging. Mentoring can be embedded within a randomised controlled trial (RCT) to upskill and support those delivering the intervention. This study aimed to understand, from a realist perspective, how mentoring worked to support implementation fidelity for occupational therapists (OTs) delivering a vocational rehabilitation (VR) intervention within the context of an RCT.MethodsA realist evaluation using secondary data (emails, mentoring record forms, interviews) collected as part of an RCT. Three researchers coded the data following content analysis, focused on refining or refuting an initial programme theory by exploring the interactions between context, mechanisms, and outcomes. The research team met to further refine the programme theories.ResultsData from 584 emails, 184 mentoring record forms, and 25 interviews were analysed following a realist approach. We developed a programme theory consisting of two contexts (trial set-up, ongoing mentoring), nine mechanisms (collective understanding, monitoring, timely support, positive reinforcement, reflective practice, support data completeness, facilitation strategy, shared learning experience, management of research and clinical duties), and three outcomes (improved confidence, improved fidelity, reduced contamination).ConclusionsOffering mentoring support to OTs delivering a VR intervention as part of an RCT improves intervention fidelity and reduces the risk of contamination. It improves OTs’ understanding of the differences between their clinical and research roles and increases their confidence and competence in trial paperwork completion and identification of potential contamination issues.

Using PURPOSE-T in clinical practice: A realist evaluation

AimTo explain how the clinical and organisational context influenced the way the Pressure Ulcer Risk Primary or Secondary Evaluation Tool (PURPOSE-T) is used by nursing staff to support their clinical judgement and decision making about care planning and delivery. MethodsA realist process evaluation was undertaken in a large acute hospital trust using mixed methods incorporating organisational policy review, staff semi-structured, ethnographic observation of clinical care and patient record review. Approximately 75 h of ethnographic field work involving 72 patients, 15 patient record reviews and 16 staff interviews were undertaken on 4 wards. FindingsFindings suggest PURPOSE-T assisted nurses differently depending on their level of experience. Those with less experience use it as an educational guide, while those with more experience made an initial clinical judgement and used PURPOSE-T as a safety net to ensure they hadn't missed anything. Nurses were concerned about demonstrating good documentation of assessment, care planning and delivery in order to underpin consistent communication about care and because they had an underlying fear of being blamed if things went wrong. There is an array of other contextual features that impact the planning and delivery of pressure area care that go beyond the use of PURPOSE-T alone, including systematic equipment provision, competing patient safety initiatives and rehabilitation requirements. ConclusionThe findings reinforce the assertion that PU-RAIs are complex interventions and could inform the development of a more integrated system of care which takes into account the contextual features associated with PU prevention in modern hospitals.

Realist process evaluation of the knowledge translation programme of a patient‐oriented research network

AbstractRationaleThe Knowledge Translation (KT) Programme of a pan‐Canadian strategic patient‐oriented research network focused on brain‐based developmental disabilities aimed to mobilize knowledge relevant to the network members. The programme also promotes and studies integrated Knowledge Translation (iKT) approaches involving different interested parties, such as researchers, patient‐partners and decision‐makers, in all parts of the knowledge creation process.Aims and ObjectivesThe objective of this study is to advance research programme evaluation methods through a realist evaluation of the process of implementing iKT activities.MethodsRealist process evaluation included: (1) development of initial programme theories (using the partnership synergy theory); (2) data collection and analysis; (3) synthesis and refinement of theories through engagement with literature; and (4) presentation of findings in context‐mechanism‐outcome (C‐M‐O) configurations. A range of project documentation records were reviewed for analysis, and three co‐leads, a programme coordinator, and a senior research associate were consulted to contextualize the implementation process of relevant KT activities.ResultsBased on the developed C‐M‐O configurations, we identified five key mechanisms of generating synergy in the iKT processes: (1) Visible shared leadership that embodies what iKT looks like; (2) Researchers' readiness for iKT; (3) Adaptation and flexible allocation of resources to emerging needs; (4) Power sharing to create practical and creative knowledge; and (5) Collective voice for potential transformative impacts at the policy level.ConclusionsThe current realist evaluation demonstrated how partnerships between researchers, patient‐partners and other interested parties can synergistically generate new ways of thinking among all interested parties, actionable strategies to integrate users in research, and solutions to disseminate knowledge. In particular, we identified a pivotal role for patient‐partners to act as equal decision‐maker helps building and maintaining partnerships and consolidating KT strategies.

Strategic and operational issues in sustaining community-based dementia support groups: the Get Real with Meeting Centres realist evaluation part 2

Objectives Support for people with dementia in their communities is neither robust nor consistent in the UK, often bolstered by third sector/grass-roots initiatives facing formidable challenges in sustaining long-term. The Get Real with Meeting Centres project explored factors involved in sustaining one such form of community-based support. This is the second of two linked articles outlining learning from this realist evaluation of Meeting Centres (MCs) for people with dementia and carers, which focusses on findings regarding their operational and strategic running. Method Semi-structured interviews and focus group discussions were conducted with 77 participants across three MC sites in England and Wales, including people living with dementia, informal carers, staff, volunteers, trustees, and supporting professionals/practitioners. Data were themed, then analysed using soft systems methodology and realist logic of analysis. Results Forty-two ‘context-mechanism-outcome’ statements were generated, explaining how background circumstances might trigger responses/processes to produce wanted or unwanted outcomes regarding three key areas for MC sustainability: External relationships and collaboration; Internal relationships and practices; and Finances and funding. Conclusion Collaboration is essential to sustaining community-based initiatives such as MCs, particularly between local community and regional level. MCs need to be vigilant in mitigating pressures that create ‘mission drift’, as targeting a gap in the care pathway and maintaining a person-centred ethos are central to MCs’ appeal. Stable, ongoing funding is needed for stable, ongoing community dementia support. More formal recognition of the value of social model community-based initiatives, helped by improved data collection, would encourage more robust and consistent community dementia support.

Unboxing the blackbox of human-centric and inclusive smart city initiatives: a theory based evaluation framework

ABSTRACT The concept of smart cities has evolved from a technology and data-driven approach to a more human-centred and inclusive model. This paper explores the shift in smart city development towards a focus on human-centric and inclusive solutions while suggesting a novel evaluation framework for these smart city initiatives. Through a comprehensive literature review, the study identifies key challenges in evaluating smart city initiatives including the lack of standardized evaluation frameworks, limited focus on social and environmental outcomes, and the need for citizen participation. To address these challenges, the paper proposes a theoretical framework grounded in a realist evaluation approach. This framework incorporates a Theory of Change (ToC) that outlines the causal pathways and contextual factors underlying smart city initiatives. The proposed framework is designed to guide the evaluation of smart city initiatives by providing specific evaluation criteria and sub-criteria that are aligned with the objectives of inclusive and sustainable urban development. By highlighting the importance of human-centric and inclusive approaches and providing a practical evaluation framework, the paper aims to contribute to the ongoing discourse on smart cities and their evaluation.

Working mechanisms of the use and acceptability of ecological momentary interventions: a realist evaluation of a guided self-help ecological momentary intervention targeting self-esteem

BackgroundTechnology improves accessibility of psychological interventions for youth. An ecological momentary intervention (EMI) is a digital intervention geared toward intervening in daily life to enhance the generalizability and ecological validity, and to be able to intervene in moments most needed. Identifying working mechanisms of the use of ecological momentary interventions might generate insights to improve interventions.MethodsThe present study investigates the working mechanisms of the use and acceptability of an ecological momentary intervention, named SELFIE, targeting self-esteem in youth exposed to childhood trauma, and evaluates under what circumstances these mechanisms of use and acceptability do or do not come into play. A realist evaluation approach was used for developing initial program theories (data: expert interviews and a stakeholders focus group), and subsequently testing (data: 15 interviews with participants, a focus group with therapists, debriefing questionnaire), and refining them.ResultsThe SELFIE intervention is offered through a smartphone application enabling constant availability of the intervention and thereby increasing accessibility and feasibility. When the intervention was offered on their personal smartphone, this enhanced a sense of privacy and less hesitance in engaging with the app, leading to increased disclosure and active participation. Further, the smartphone application facilitates the practice of skills in daily life, supporting the repeated practice of exercises in different situations leading to the generalizability of the effect. Buffering against technical malfunction seemed important to decrease its possible negative effects.ConclusionsThis study enhanced our understanding of possible working mechanisms in EMIs, such as the constant availability supporting increased accessibility and feasibility, for which the use of the personal smartphone was experienced as a facilitating context. Hereby, the current study contributes to relatively limited research in this field. For the field to move forward, mechanisms of use, and acceptability of EMIs need to be understood. It is strongly recommended that alongside efficacy trials of an EMI on specific target mechanisms, a process evaluation is conducted investigating the working mechanisms of use.Trial registrationThe current paper reports on a realist evaluation within the SELFIE trial (Netherlands Trial Register NL7129 (NTR7475)).

Let’s Go! 2022-2023 Educational Programs: Developing a Standardized Obesity Medicine Curriculum through a Realist Evaluation

Let’s Go! 2022-2023 Educational Programs: Developing a Standardized Obesity Medicine Curriculum through a Realist Evaluation

Identifying mechanisms of youth mental health promotion: A realist evaluation of the Agenda Gap programme

Promoting youth mental health is a critical public health priority and merits robust policy and practice responses, inclusive of youth-centred and upstream interventions that address the root factors contributing to mental health outcomes. To that end, non-familial youth-adult relationships can powerfully impact youths’ healthy development, mental health and well-being, and capacities for enacting change within home, school, and policy contexts. Agenda Gap is a youth mental health promotion programme based on this principle, in which adult facilitators support youth (aged 15–24) in co-exploring activities focused on mental health promotion and policy advocacy, while fostering supportive, trusting intergenerational relationships. This study presents a realist evaluation of Agenda Gap, drawing on realist qualitative interviews with youth participants (n = 18) and adult collaborators (n = 4). We constructed 15 initial programme theories in four theoretical areas that we subsequently explored via analysis of the realist interview data. The analysis used the middle-range theories of ‘Third Space’ and ‘Third Place’ to conceptualize and articulate how causal mechanisms were produced from the non-familial youth-adult relationships that are foundational to the programme. Results are presented across 10 context-mechanism-outcome (CMO) configurations in three sections: (1) Agenda Gap Facilitation Activates ‘Third-Space/Third-Place’ Mechanisms; (2) Youth Enhance Personal Mental Health while also Supporting Mental Health of Friends and Family; (3) Youth Become Inspired and Informed to Act as Advocates and Policy Change Agents. Taken together, these findings provide insights into the important mechanisms of non-familial intergenerational trust building and demonstrate how a strengths-based conceptualization of youth mental health supports mental health maintenance, promotion, and advocacy for this population.

A realist evaluation of community champion and participatory action approaches during the COVID-19 pandemic

A realist evaluation of community champion and participatory action approaches during the COVID-19 pandemic

Realist process evaluation of occupational performance coaching: protocol

IntroductionA cluster randomised controlled trial, the Meaning, Agency and Nurturing Autonomy (MANA) study, is underway comparing the effects of occupational performance coaching (OPC) and usual care on the social participation,...

Guiding Through Turbulent Times: Coaching During Merger and Acquisition

ABSTRACT Mergers and acquisitions put organizations through significant disruptions, often taking a toll on employee well-being. This qualitative study aims to analyze how coaching supports employees in coping with the impact of mergers. Eleven semi-structured interviews were conducted with employees and managers. Using realist evaluation, four context-mechanism-outcome configurations (CMOCs) were developed, providing evidence for the following working mechanisms: (1) Increased self-awareness and assertiveness led to greater self-efficacy, empowering participants to advocate for themselves. (2) Breaking down big things into manageable steps helped participants cope with overwhelming issues. (3) Leaders’ support and guidance played a crucial role in helping employees navigate change. And (4) managing a high workload led to increased self-efficacy and well-being at work. These CMOCs demonstrate the diverse working mechanisms of coaching interventions implemented in a disruptive context. The study's insights into these mechanisms offer valuable guidance for designing future interventions to support employees during organizational changes, such as mergers and acquisitions. By tailoring interventions to address employees’ individual challenges and needs, organizations can enhance the effectiveness of these initiatives and promote employee well-being during times of transition. MAD statement This study aims to Make A Difference (MAD) by revealing how coaching works in the context of mergers and acquisitions; more specifically, it identifies the working mechanisms contributing to the intervention's success. By its detailed identification of the interplay of context, mechanism and outcome, it highlights the multifaceted impacts of coaching, ranging from enhanced self-efficacy to coping strategies. It adds to the theoretical discussion on the effectiveness of workplace interventions by not only analyzing what works but also how they work under specific circumstances. As such, it can inform employers on how to support employees during times of organizational change.

Dementia care coordinators in primary care: a realist evaluation.

A dementia care coordinator (DCC) service has recently been implemented across Kent and Medway Integrated Care Board (ICB). DCCs are provided at the primary care network (PCN) level and work closely with GP practice teams. The service is intended to help service users navigate the care system and provide proactive support to mitigate crisis points. However, the value of this service from key stakeholder perspectives is not yet known. To understand: 1) how the DCC service works, for whom, and in what circumstances; and 2) identify outcomes, resource implications, and costs. The study uses mixed-methods realist evaluation with an embedded economic component. Data collection includes: questionnaires, service provider metrics, and qualitative interviews with healthcare practitioners and service users. Interpretive comparative analysis and narrative synthesis including evaluation of service costs against outcomes will produce a refined final programme theory. Patient and public involvement have been consulted throughout. Preliminary results indicate that when embedded in a local practice, healthcare practitioners including GPs welcome DCC involvement. DCCs act as community lynchpins, bridging service users to appropriate care pathways. The service may be capable of increasing diagnosis rates and reducing crisis points. However, caseloads are currently untamed. Therefore, the service and its workforce are becoming over-stretched. This evaluation describes preliminary findings from a recently implemented DCC service in Kent and Medway ICB. There is a crucial need to enhance the support of this workforce in order to ensure sustainability of this service. These findings will inform service development and future investment decisions.

How do oilcloth sessions work? A realist evaluation approach to exploring ripple effects in an implementation strategy

PurposeTo explore the mechanisms of the implementation strategy, “oilcloth sessions” and understand and explain the ripple effects of oilcloth sessions as a strategy to implement a new emergency department.Design/methodology/approachA qualitative design was used whereby data were collected using field notes from an ethnographic study of the oilcloth sessions and follow-up semi-structured interviews with staff, managers and key employees who participated in the oilcloth sessions. The data analysis was inspired by the realist evaluation approach of generative causality proposed by Pawson and Tilley.FindingsThe primary ripple effect was that the oilcloth sessions were used for different purposes than the proposed program theory, including being used as: (1) a stage, (2) a battlefield, (3) a space for imagination and (4) a strategic management tool influencing the implementation outcomes. The results bring essential knowledge that may help to explain why and how a well-defined implementation strategy has unplanned outcomes.Originality/valueUnintended outcomes of implementation strategies are an underexplored issue. This study may help implementation researchers rethink the activities required to reduce unintended negative outcomes or explore potential unplanned outcomes and, in this way, hinder or enhance outcomes, effectiveness and sustainability. Future studies within implementation research should incorporate attention to unintended outcomes to fully understand the impact of implementation strategies.

IMplementation of the Preterm Birth Surveillance PAthway: a RealisT evaluation (The IMPART Study)

BackgroundIn the UK, 7.6% of babies are born preterm, which the Department of Health aims to decrease to 6% by 2025. To advance this, NHS England released Saving Babies Lives Care Bundle Version 2 Element 5, recommending the Preterm Birth Pathway for women at risk of preterm birth. The success of this new pathway depends on its implementation. The IMPART (IMplementation of the Preterm Birth Surveillance PAthway: a RealisT evaluation) study aimed to research how, why, for whom, to what extent and in what contexts the prediction and prevention aspects of Preterm Birth Surveillance Pathway is implemented through a realist evaluation. Realist implementation studies are growing in popularity.MethodsInitial programme theories were developed through a realist informed literature scope, interviews with developers of the NHS England guidance, and a national questionnaire of current practice. Implementation theory was utilised in developing the programme theories. Data (interviews and observations with staff and women) were undertaken in 3 case sites in England to ‘test’ the programme theories. Substantive theory was utilised during data analysis to interpret and refine the theories on how implementation could be improved.ResultsThree explanatory areas were developed: risk assessing and referral; the preterm birth surveillance clinic; and women centred care. Explanatory area 1 dealt with the problems in correct risk assessment and referral to a preterm clinic. Explanatory area 2 focused on how once a correct referral has been made to a preterm clinic, knowledgeable and supported clinicians can deliver a well-functioning clinic. Explanatory area 3 concentrated on how the pathway delivers appropriate care to women.ConclusionsThe IMPART study provides several areas where implementation could be improved. These include educating clinicians on knowledge of risk factors and the purpose of the preterm clinic, having a multidisciplinary preterm team (including a preterm midwife) with specialist preterm knowledge and skills (including transvaginal cervical scanning skills), and sites actively working with their local network. This multidisciplinary preterm team are placed to deliver continuity of care for women at high-risk of preterm birth, being attentive to their history but also ensuring they are not defined by their risk status.Trial registrationISRCTN57127874.

Reaching people and managing membership in community-based dementia support groups: the Get Real with Meeting Centres realist evaluation part 1

Objectives There is a need to improve the provision and reach of community services for people living with dementia, a goal in which community-based support groups can play a key role. The Get Real with Meeting Centres project aimed to explore factors involved in the success and sustainability of Meeting Centres (MCs) a form of community-based support proliferating in the UK. This is the first of two linked articles outlining learning from this realist evaluation of MCs, which focusses on findings around reach and membership. Method Semi-structured interviews and focus group discussions were conducted with 77 participants across three case study MC sites in England and Wales, including people living with dementia, informal carers, staff, volunteers, trustees, and supporting professionals/practitioners. Data were themed, then analysed using both soft systems methodology and realist logic of analysis. Results Fifty-two ‘context-mechanism-outcome’ statements were generated, explaining how background circumstances might trigger responses/processes to produce wanted or unwanted outcomes regarding four key areas for MC sustainability: Referrals and the dementia care pathway; Reaching people and membership; Carer engagement and benefit; and Venue and location. Conclusion Strong links with formal services and a well-functioning dementia care pathway are essential to sustaining community-based group support such as MCs; group support is also well-placed to assist work to improve pathway issues. Clarity of offer (including benefit to carers), and a wide range of activities, are key to appeal and reach; transport to, and use of, venue are challenges, as are pressures to support people with more advanced dementia.

A protocol for a critical realist systematic synthesis of interventions to promote pupils' wellbeing by improving the school climate in low- and middle-income countries.

The review described in this protocol will be the first critical realist review of the literature reporting on the impact of interventions to promote pupils' wellbeing by improving the school climate in Low- and Middle-Income Countries. The review is being carried out to inform the programme theory for a critical realist evaluation of a whole school mindfulness intervention in Ethiopia and Rwanda to improve pupils' mental wellbeing. Our initial programme theory hypothesises that pupils' (and teachers') responses to the mindfulness intervention as well as changing the behaviour and attitudes of individual pupils and teachers, will change the 'school climate' in ways that have a positive impact on mental wellbeing. This literature review will facilitate the identification of mechanisms for change working at the level of the whole school climate, something which is only infrequently discussed in evaluations of mindfulness interventions. A critical realist review methodology will be used to provide a causal interdisciplinary understanding of how school climate can promote the wellbeing of pupils. This will be done through a systematic literature review and extrapolating context, agency, intervention, mechanisms, and outcome configurations and synthesising these to provide a conceptual understanding of the impact of interventions to improve school climate. The review findings will inform a critical realist evaluation of a mindfulness intervention in schools that we will be carrying out. The findings from the review will enable us to focus more precisely and transparently on what policymakers and other stakeholders need to know about how school climate changes due to introducing mindfulness to the curriculum and how this impacts pupils' wellbeing [and for which pupils]. We will publish the findings from the review in academic and professional publications, policy briefs, workshops, conferences, and social media. PROSPERO registration number: CRD42023417735.

Understanding the impact of a residential housing program for people living with HIV/AIDS: results from a realist evaluation

BackgroundIn Canada, the Public Health Agency surveillance of new HIV cases has demonstrated annual increasing rates since 2020. The rates of new HIV cases are highest in the province of...