Principles Of Realist Evaluation Research Articles

The Implementation of an Innovative Hydration Monitoring App in Care Home Settings: A Qualitative Study.

BackgroundIn response to marked concern regarding inadequate fluid intake recording in care homes, an innovative mobile hydration app was collaboratively developed. “Hydr8” aimed to facilitate accurate recording and communication of residents’ fluid intake and ultimately increase care quality and patient safety.ObjectiveThe aim of this study was to examine the implementation of Hydr8 in a sample of care homes in one area in England.MethodsThe principles of Realist Evaluation and Action research were drawn upon throughout the study. Overall, 5 care homes participated in this study, 3 interview-only sites and 2 case-study sites, where interviews and observations were conducted at 3 time-points. Furthermore, 28 staff members participated, including care staff, management, a registered nurse, and administrative staff.ResultsFindings suggest that Hydr8 benefits practice, enhancing the understanding of hydration and person-centered care and improving staff communication. However, technical glitches hindered the seamless embedding of Hydr8 into everyday practice, and enthusiasm for long-term use was dependent on the resolution of issues. In addition, Hydr8 heightened perceptions of personal accountability, and while managers viewed this as positive, some staff members were apprehensive. However, individuals were enthusiastic about the long-term use and potential of Hydr8.ConclusionsUtilizing the findings of this study to further develop and adapt Hydr8 indicates the long-term use of Hydr8 as promising. Although perceptions of Hydr8 were primarily positive, setbacks in its implementation and use created difficulties in normalizing the solution into everyday practice. This study highlights the need for education related to hydration practice and a change of infrastructure in care home settings to implement technical solutions and changes to care.

Midwifery continuity of carer: Developing a realist evaluation framework to evaluate the implementation of strategic change in Scotland

Midwifery continuity of carer (MCC) models result in better clinical outcomes for women and offer midwives a superior way of working when compared to other models of maternity care. Implementing a MCC model, a key recommendation of the Scottish Government Maternity and Neonatal Strategy Best Start, requires significant restructuring of maternity services and changes to midwives’ roles. Careful evaluation is therefore required to monitor and understand how the policy affects care providers and users. Realist evaluation is an appropriate methodology for evaluating programmes of change set within complex social organisations, such as health services, and can help to understand variations in outcomes and experiences. This paper presents the approach taken using the principles of realist evaluation to identify key programme theories, which then informed an evaluation framework and a midwives’ evaluation tool. The comprehensive survey-tool developed for midwives has the potential to be used more widely to evaluate comparable strategic change in this area.

Early warning systems and rapid response to the deteriorating patient in hospital: A realist evaluation.

To test the Rapid Response Systems programme theory against actual practice components of the Rapid Response Systems implemented to identify those contexts and mechanisms which have an impact on the successful achievement of desired outcomes in practice. Rapid Response Systems allow deteriorating patients to be recognized using Early Warning Systems, referred early via escalation protocols and managed at the bedside by competent staff. Realist evaluation. The research design was an embedded multiple case study approach of four wards in two hospitals in Northern Ireland which followed the principles of Realist Evaluation. We used various mixed methods including individual and focus group interviews, observation of nursing practice between June-November 2010 and document analysis of Early Warning Systems audit data between May-October 2010 and hospital acute care training records over 4.5years from 2003-2008. Data were analysed using NiVivo8 and SPPS. A cross-case analysis highlighted similar patterns of factors which enabled or constrained successful recognition, referral and response to deteriorating patients in practice. Key enabling factors were the use of clinical judgement by experienced nurses and the empowerment of nurses as a result of organizational change associated with implementation of Early Warning System protocols. Key constraining factors were low staffing and inappropriate skill mix levels, rigid implementation of protocols and culturally embedded suboptimal communication processes. Successful implementation of Rapid Response Systems was dependent on adopting organizational and cultural changes that facilitated staff empowerment, flexible implementation of protocols and ongoing experiential learning.

Substituting physicians with nurse practitioners, physician assistants or nurses in nursing homes: protocol for a realist evaluation case study

IntroductionIn developed countries, substituting physicians with nurse practitioners, physician assistants and nurses (physician substitution) occurs in nursing homes as an answer to the challenges related to the ageing population and...

Public-private partnerships to improve primary healthcare surgeries: clarifying assumptions about the role of private provider activities.

PPPs were developed to improve the quality of care and patient satisfaction. However, evidence of their effectiveness in delivering health benefits is limited. A qualitative study design was employed. A total of 25 interviews with public sector staff (n=23) and private sector managers (n=2) were conducted to understand their interpretations of assumptions in the activities of private investors and service contractors participating in Local Improvement Finance Trust (LIFT) partnerships. Realist evaluation principles were applied in the data analysis to interpret the findings. Six thematic areas of assumed health benefits were identified: (i) quality improvement; (ii) improved risk management; (iii) reduced procurement costs; (iv) increased efficiency; (v) community involvement; and (vi) sustainable investment. Primary Care Trusts that chose to procure their surgeries through LIFT were expected to support its implementation by providing an environment conducive for the private participants to achieve these benefits. Private participant activities were found to be based on a range of explicit and tacit assumptions perceived helpful in achieving government objectives for LIFT. The success of PPPs depended upon private participants' (i) capacity to assess how PPP assumptions added value to their activities, (ii) effectiveness in interpreting assumptions in their expected activities, and (iii) preparedness to align their business principles to government objectives for PPPs. They risked missing some of the expected benefits because of some factors constraining realization of the assumptions. The ways in which private participants preferred to carry out their activities also influenced the extent to which expected benefits were achieved. Giving more discretion to public than private participants over critical decisions may help in ensuring that assumptions in PPP activities result in outcomes that match the anticipated health benefits.

An environmental social marketing intervention in cultural heritage tourism: a realist evaluation

ABSTRACT Following Pawson and Tilley's principles of realist evaluation and the context–mechanism–outcome (CMO) framework, this paper conducts a process evaluation of an environmental social marketing intervention in a heritage tourism organisation. Social marketing and employee environmental interventions have received relatively scant attention in tourism. Additionally, prior literature mostly focused on the evaluation of intervention outcomes (i.e. how far the intervention produces precise targeted outcomes) and ignores the importance of process evaluation (i.e. identifying what works, for whom, under which circumstances and how, plus issues of intervention maintenance). This paper fills this literature gap using realist evaluation theory and academic perspectives, as well as via the reflections of practitioners involved in intervention design and delivery. Findings suggest that a good understanding of the tourism and organisational context (regarding the dimensions of structure, culture, agency and relations) and the use of tailored, action-focused mechanisms (for each context dimension) are critical to achieving transformational outcomes in environmental interventions in cultural heritage organisations. Based on these findings, it is concluded that the CMO is a useful framework for assessing environmental social marketing interventions in tourism (both for heritage and other tourism organisations). Implications for tourism practice and further research directions are also discussed.

A Framework for Developing the Knowledge and Competencies of the Outside School Hours Services Workforce

Abstract A skilled workforce is critical to the delivery of quality school age care services. A strategy called Core Knowledge and Competency (CKC) Framework was designed to build the skills and knowledge of after school care educators as they provided before and after school programs in Queensland, Australia. The strategy was to help educators meet the Australian National Quality Standard for Early Childhood Education and Care and School Age Care (National Quality Standards) (NQS). This article describes the use of realist evaluation principles to examine the implementation of this strategy. Improvements to the quality of the workforce were reported including less staff turnover, increased capacity and competence of educators which would result in enhanced outcomes for children. However the implementation of the strategy relied on highly competent leaders within the services. Keywords: workforce development, mentors, school-aged care, leadership, child development ----- Bibl iography: Cartmel, Jennifer/Brannelly, Kylie: A Framework for Developing the Knowledge and Competencies of the Outside School Hours Services Workforce, IJREE, Vol. 4, Issue 2-2016, pp. 17-33. https://doi.org/10.3224/ijree.v4i2.25779

Realist complex intervention science: Applying realist principles across all phases of the Medical Research Council framework for developing and evaluating complex interventions.

The integration of realist evaluation principles within randomised controlled trials (‘realist RCTs’) enables evaluations of complex interventions to answer questions about what works, for whom and under what circumstances. This allows evaluators to better develop and refine mid-level programme theories. However, this is only one phase in the process of developing and evaluating complex interventions. We describe and exemplify how social scientists can integrate realist principles across all phases of the Medical Research Council framework. Intervention development, modelling, and feasibility and pilot studies need to theorise the contextual conditions necessary for intervention mechanisms to be activated. Where interventions are scaled up and translated into routine practice, realist principles also have much to offer in facilitating knowledge about longer-term sustainability, benefits and harms. Integrating a realist approach across all phases of complex intervention science is vital for considering the feasibility and likely effects of interventions for different localities and population subgroups.

Evaluating the implementation of a national clinical programme for diabetes to standardise and improve services: a realist evaluation protocol.

BackgroundOver the last three decades in response to the growing burden of diabetes, countries worldwide have developed national and regional multifaceted programmes to improve the monitoring and management of diabetes and to enhance the coordination of care within and across settings. In Ireland in 2010, against a backdrop of limited dedicated strategic planning and engrained variation in the type and level of diabetes care, a national programme was established to standardise and improve care for people with diabetes in Ireland, known as the National Diabetes Programme (NDP). The NDP comprises a range of organisational and service delivery changes to support evidence-based practices and policies. This realist evaluation protocol sets out the approach that will be used to identify and explain which aspects of the programme are working, for whom and in what circumstances to produce the outcomes intended.Methods/designThis mixed method realist evaluation will develop theories about the relationship between the context, mechanisms and outcomes of the diabetes programme. In stage 1, to identify the official programme theories, documentary analysis and qualitative interviews were conducted with national stakeholders involved in the design, development and management of the programme. In stage 2, as part of a multiple case study design with one case per administrative region in the health system, qualitative interviews are being conducted with frontline staff and service users to explore their responses to, and reasoning about, the programme’s resources (mechanisms). Finally, administrative data will be used to examine intermediate implementation outcomes such as service uptake, acceptability, and fidelity to models of care.DiscussionThis evaluation is using the principles of realist evaluation to examine the implementation of a national programme to standardise and improve services for people with diabetes in Ireland. The concurrence of implementation and evaluation has enabled us to produce formative feedback for the NDP while also supporting the refinement and revision of initial theories about how the programme is being implemented in the dynamic and unstable context of the Irish healthcare system.Electronic supplementary materialThe online version of this article (doi:10.1186/s13012-016-0464-9) contains supplementary material, which is available to authorized users.

The Well Organised Working Environment: A mixed methods study

BackgroundThe English National Health Service Institute for Innovation and Improvement designed a series of programmes called The Productive Series. These are innovations designed to help healthcare staff reduce inefficiency and improve quality, and have been implemented in healthcare organisations in at least 14 different countries. This paper examines an implementation of the first module of the Productive Community Services programme called ‘The Well Organised Working Environment’. ObjectiveThe quantitative component aims to identify the quantitative outcomes and impact of the implementation of the Well Organised Working Environment module. The qualitative component aims to describe the contexts, mechanisms and outcomes evident during the implementation, and to consider the implication of these findings for healthcare staff, commissioners and implementation teams. DesignMixed methods explanatory sequential design. SettingsCommunity Healthcare Organisation in East Anglia, England. ParticipantsFor the quantitative data, participants were 73 staff members that completed End of Module Assessments. Data from 25 services that carried out an inventory of stock items stored were also analysed. For the qualitative element, participants were 45 staff members working in the organisation during the implementation, and four members of the Productive Community Services Implementation Team. MethodsStaff completed assessments at the end of the module implementation, and the value of items stored by clinical services was recorded. After the programme concluded, semi-structured interviews with staff and a focus group with members of the Productive Community Services implementation team were analysed using Framework Analysis employing the principles of Realist Evaluation. Results62.5% respondents (n=45) to the module assessment reported an improvement in their working environment, 37.5% (n=27) reported that their working environment stayed the same or deteriorated. The reduction of the value of items stored by services ranged from £4 to £5039 across different services. Results of the qualitative analysis suggests explanations for why the programme worked in some contexts and not others, for instance due to varying levels of management support, and varying levels of resources allocated to carrying out or sustaining the improvement work. ConclusionsQuantitative analysis of data generated during healthcare improvement initiatives can give an impression of the benefits realised, but additional qualitative analysis also provides opportunity for learning to improve future implementations. Targets set by commissioners for innovation should focus on sustaining improvement rather demonstrating one-off benefits, and implementation teams should not let their preconceptions of what will and what will not work prevent them from trying interventions that may benefit staff.

Explaining how unexploded ordnance clearance enhances livelihoods in the Lao PDR

BackgroundFollowing violent conflict, the continued presence of landmines and unexploded ordnance pose a barrier to rebuilding livelihoods. Mine action removes these explosive remnants of conflict to enable communities to safely return contaminated land to productive use. There is limited understanding, however, of how, why, in what context and in what respects mine action contributes to livelihoods. Yet, such information is required for effective resource allocation, checking underlying program assumptions, understanding benefits and potential harms. MethodsThe evaluation was undertaken in the Lao People's Democratic Republic. It used an interpretive case study design and applied the principles of realist evaluation. Program staff and local government authorities were interviewed (N=37) and program beneficiaries. In total, 38 individual interviews with program beneficiaries were conducted and eighteen focus group interviews (9 with males, 9 with females), each with 6–9 participants. ResultsThe evaluation identified two main mechanisms through which the program ‘worked’: (1) communication pre- and post-clearance and (2) the delivery of the product (cleared land). ConclusionThe realist approach helped to refine the program theory, highlighted the role of self- and task-efficacy and community communication, assisted in identifying contextual factors that influence outcomes and suggested a revision of expected outcomes.

Conditions for production of interdisciplinary teamwork outcomes in oncology teams: protocol for a realist evaluation.

BackgroundInterdisciplinary teamwork (ITW) is designed to promote the active participation of several disciplines in delivering comprehensive cancer care to patients. ITW provides mechanisms to support continuous communication among care providers, optimize professionals’ participation in clinical decision-making within and across disciplines, and foster care coordination along the cancer trajectory. However, ITW mechanisms are not activated optimally by all teams, resulting in a gap between desired outcomes of ITW and actual outcomes observed. The aim of the present study is to identify the conditions underlying outcome production by ITW in local oncology teams.MethodsThis retrospective multiple case study will draw upon realist evaluation principles to explore associations among context, mechanisms and outcomes (CMO). The cases are nine interdisciplinary cancer teams that participated in a previous study evaluating ITW outcomes. Qualitative data sources will be used to construct a picture of CMO associations in each case. For data collection, reflexive focus groups will be held to capture patients’ and professionals’ perspectives on ITW, using the guiding question, ‘What works, for whom, and under what circumstances?’ Intra-case analysis will be used to trace associations between context, ITW mechanisms, and patient outcomes. Inter-case analysis will be used to compare the different cases’ CMO associations for a better understanding of the phenomenon under study.DiscussionThis multiple case study will use realist evaluation principles to draw lessons about how certain contexts are more or less likely to produce particular outcomes. The results will make it possible to target more specifically the actions required to optimize structures and to activate the best mechanisms to meet the needs of cancer patients. This project could also contribute significantly to the development of improved research methods for conducting realist evaluations of complex healthcare interventions. To our knowledge, this study is the first to use CMO associations to improved empirical and theoretical understanding of interdisciplinary teamwork in oncology, and its results could foster more effective implementation in clinical practice.

Towards a programme theory for fidelity in the evaluation of complex interventions.

This paper addresses the challenge of investigating fidelity in the implementation of a complex rehabilitation intervention designed to increase the level of independence in personal activities of daily living of stroke patients living in UK care homes. A programme theory of intervention fidelity was constructed to underpin a process evaluation running alongside a cluster randomized trial of the rehabilitation intervention. The programme theory has been constructed drawing on principles of realist evaluation. Using data from in-depth semi-structured interviews (n = 17) with all occupational therapists (OTs) and critical incident reports from the trial (n = 20), and drawing from frameworks for implementation, the programme theory was developed. The programme theory incorporates four potential mechanisms through which fidelity within the trial can be investigated. These four programme theory areas are (1) the balancing of research and professional requirements that therapists performed in a number of areas while delivering the study interventions; (2) the OTs rapport building with care home staff; (3) the work focused on re-engineering the personal environments of care home patients; and (4) the learning about the intervention within the context of the trial and its impacts over time. These findings characterize the real-world nature of fidelity within intervention research, and specifically the negotiated nature of implementation within clinical settings, including individual patients' needs. This research adds to the evidence base because current frameworks for fidelity neglect the importance of learning over time of individuals and across the time span of a trial.

A realist logic model of the links between nurse staffing and the outcomes of nursing

Background: There has been a long-standing debate over the definition and nature of the quality of healthcare and factors that influence and enhance quality. Within nursing, the challenge is to identify the outcomes that are measurable and amenable to change as a result of nursing care. Arising originally from concerns over potential nurse staffing shortages and nurse retention within the United States, an extensive literature has developed in the acute sector, exploring nurse staffing and its consequences. All of these studies raise the generic question of what potential causal mechanisms might link nurse staffing levels and skill mix to issues of patient safety and outcome. Objectives: To generate a tentative logic model to understand existing findings and to elucidate possible ways in which nurse staffing and skill mix may affect patient and nurse outcomes. Methods: This study was grounded within the principles of realist evaluation, realist review and logic modelling. The existing literature was reviewed to bring to light the underlying rationale suggested by the authors of this study on how nursing care might affect patient outcomes. A step-by-step process was followed to demonstrate the generation of a tentative logic model of how nurse staffing might influence patient, and nursing, outcomes. Results: The final logic model depicts staffing adequacy as having a complex link with patient outcomes. This is mediated at a general level through factors in the process of care (for example, nurse surveillance, clinical judgement, level of education, level of nurse training and length of nursing experience) and tasks left undone. These operate in conjunction with working with other nurses who are clinically competent, having good nurse–physician relationships and communication, supportive nurse manager/supervisor and good teamwork. Conclusions: This study extends the understanding of the mechanisms through which nurse staffing levels may result in adverse patient outcomes in the acute sector. Key intervening variables are the application of nurse intuition, operation of clinical judgement and missed nursing care/tasks left undone. The tentative logic model can be used to draw up areas and hypotheses to guide the direction of future research and to aid interpretation of existing research.

Realist methods in medical education research: what are they and what can they contribute?

Education is a complex intervention which produces different outcomes in different circumstances. Education researchers have long recognised the need to supplement experimental studies of efficacy with a broader range of study designs that will help to unpack the 'how' and 'why' questions and illuminate the many, varied and interdependent mechanisms by which interventions may work (or fail to work) in different contexts. One promising approach is realist evaluation, which seeks to establish what works, for whom, in what circumstances, in what respects, to what extent, and why. This paper introduces the realist approach and explains why it is particularly suited to education research. It gives a brief introduction to the philosophical assumptions underlying realist methods and outlines key principles of realist evaluation (designed for empirical studies) and realist review (the application of realist methods to secondary research). The paper warns that realist approaches are not a panacea and lists the circumstances in which they are likely to be particularly useful.

Implementing accountability for reasonableness framework at district level in Tanzania: a realist evaluation

BackgroundDespite the growing importance of the Accountability for Reasonableness (A4R) framework in priority setting worldwide, there is still an inadequate understanding of the processes and mechanisms underlying its influence on legitimacy and fairness, as conceived and reflected in service management processes and outcomes. As a result, the ability to draw scientifically sound lessons for the application of the framework to services and interventions is limited. This paper evaluates the experiences of implementing the A4R approach in Mbarali District, Tanzania, in order to find out how the innovation was shaped, enabled, and constrained by the interaction between contexts, mechanisms and outcomes.MethodsThis study draws on the principles of realist evaluation -- a largely qualitative approach, chiefly concerned with testing and refining programme theories by exploring the complex interactions of contexts, mechanisms, and outcomes. Mixed methods were used in data collection, including individual interviews, non-participant observation, and document reviews. A thematic framework approach was adopted for the data analysis.ResultsThe study found that while the A4R approach to priority setting was helpful in strengthening transparency, accountability, stakeholder engagement, and fairness, the efforts at integrating it into the current district health system were challenging. Participatory structures under the decentralisation framework, central government's call for partnership in district-level planning and priority setting, perceived needs of stakeholders, as well as active engagement between researchers and decision makers all facilitated the adoption and implementation of the innovation. In contrast, however, limited local autonomy, low level of public awareness, unreliable and untimely funding, inadequate accountability mechanisms, and limited local resources were the major contextual factors that hampered the full implementation.ConclusionThis study documents an important first step in the effort to introduce the ethical framework A4R into district planning processes. This study supports the idea that a greater involvement and accountability among local actors through the A4R process may increase the legitimacy and fairness of priority-setting decisions. Support from researchers in providing a broader and more detailed analysis of health system elements, and the socio-cultural context, could lead to better prediction of the effects of the innovation and pinpoint stakeholders' concerns, thereby illuminating areas that require special attention to promote sustainability.

How do you modernize a health service? A realist evaluation of whole-scale transformation in london.

Large-scale, whole-systems interventions in health care require imaginative approaches to evaluation that go beyond assessing progress against predefined goals and milestones. This project evaluated a major change effort in inner London, funded by a charitable donation of approximately $21 million, which spanned four large health care organizations, covered three services (stroke, kidney, and sexual health), and sought to "modernize" these services with a view to making health care more efficient, effective, and patient centered. This organizational case study draws on the principles of realist evaluation, a largely qualitative approach that is centrally concerned with testing and refining program theories by exploring the complex and dynamic interaction among context, mechanism, and outcome. This approach used multiple data sources and methods in a pragmatic and reflexive manner to build a picture of the case and follow its fortunes over the three-year study period. The methods included ethnographic observation, semistructured interviews, and scrutiny of documents and other contemporaneous materials. As well as providing ongoing formative feedback to the change teams in specific areas of activity, we undertook a more abstract, interpretive analysis, which explored the context-mechanism-outcome relationship using the guiding question "what works, for whom, under what circumstances?" In this example of large-scale service transformation, numerous projects and subprojects emerged, fed into one another, and evolved over time. Six broad mechanisms appeared to be driving the efforts of change agents: integrating services across providers, finding and using evidence, involving service users in the modernization effort, supporting self-care, developing the workforce, and extending the range of services. Within each of these mechanisms, different teams chose widely differing approaches and met with differing success. The realist analysis of the fortunes of different subprojects identified aspects of context and mechanism that accounted for observed outcomes (both intended and unintended). This study was one of the first applications of realist evaluation to a large-scale change effort in health care. Even when an ambitious change program shifts from its original goals and meets unforeseen challenges (indeed, precisely because the program morphs and adapts over time), realist evaluation can draw useful lessons about how particular preconditions make particular outcomes more likely, even though it cannot produce predictive guidance or a simple recipe for success. Noting recent calls by others for the greater use of realist evaluation in health care, this article considers some of the challenges and limitations of this method in the light of this experience and suggests that its use will require some fundamental changes in the worldview of some health services researchers.

Evidence and Policy and Naming and Shaming

The debate on the potential of evidence-based policy is now in full flow, though opinions differ on whether we have reached zenith or nadir in our capacity to transform public data into public service. Clearly, we are not short of 'evidence'; there is a plethora of programme evaluators and policy analysts out there, and a surfeit of their publications - official, academic and grey. What is at issue is the method of synthesizing that evidence, a research strategy that goes by the name of 'systematic review'. In particular, there is considerable doubt about how to handle the variegated and contested evidence base that typifies most sectors of public policy. Can it ever be made to yield to the formal statistical techniques of meta-analysis? And if not, can evidence-based policy ever match the precision of the demanding blueprint represented by evidence-based medicine? This paper seeks to go beyond these old antagonisms by proposing a novel model of evidence synthesis based on the principles of realist evaluation. Under this approach, systematic review changes from a data-driven exercise to a process of theory development and refinement. The core of the paper is a demonstration project, consisting of an embryonic review of public disclosure initiatives, a policy better known as 'naming and shaming'. Six initiatives are examined in order to develop an elementary theory of the conditions under which shaming sanctions might work. The paper concludes with reflections on how such a process of 'realist synthesis' may feed into policy making.