Understanding Realist Reviews for Medical Education.

Abstract

A realist review is a theory-driven approach to literature synthesis that seeks to evaluate the effectiveness of interventions in a contextualized way. Realist research, which includes review and program evaluation, is a relatively recent development. Pioneered by Pawson,1-3 this approach has proliferated in health services because realist research acknowledges that program interventions are complex, and context is fundamental to understanding how outcomes are mediated. The foundational premise of realism is that interventions work differently in different contexts and for different people. In medical education, realist reviews have been conducted to understand internet-based medical education,4 balancing health professional education and patient care,5 productive research environments,6 clinical reasoning,7 and interprofessional programs.8 (See the Box for a case example of an interprofessional education realist review.)The purpose of a realist review is to build theory and new understandings of causal mechanisms from existing research by focusing on the relationship between context and outcomes. A critical feature of complex interventions is that “as they are delivered, they are embedded in social systems. It is through the workings of entire systems of social relationships that any changes in behaviours, events and social conditions are effected.”3 Complex interventions include multiple interrelated components, which occur over time, in intricate environments, with multiple stakeholders. In a realist review, context is more than a setting or a community; it includes individual, institution, practice, social, technical, and other layers, as described by Bates and Ellaway.9The key realist research question is typically: What intervention works, for whom, how, and in what circumstances?Realist reviews differ from other reviews in approach to causality and attention to mechanisms of change and human action. Given that interventions are embedded within social systems, causality is understood as dependent on the whole context of an intervention.11 How an intervention or program causes an outcome is not simple, linear, or deterministic. Programs will have multiple mechanisms that lead to different outcomes, both positive and negative.Understanding mechanisms is the crux of understanding realist reviews. While there is no consensus on a single definition, broadly speaking, mechanisms cause things to happen. Mechanisms can be defined as the “underlying entities, processes, or structures which operate in particular contexts to generate outcomes of interest.”12 A mechanism may cause changes in individual beliefs, values, or reasoning or may change the social structures and resources available to individuals.13 Mechanisms are not necessarily visible but can mediate effects in certain contexts. For example, trust might be a mechanism by which feedback could lead a trainee to change their behavior in the context of a supervisory relationship. Motivation might be another mechanism for how feedback interventions work. It is not difficult to accept that both can operate in any single feedback intervention, which is why multiple mechanisms may be identified in a single review.Realist reviews seek to unpack mechanisms that mediate outcomes within specific contexts. These context-mechanism-outcome (CMO) configurations may occur in regular patterns, also called demi-regularities, which is another defining feature of realist reviews. The term demi-regularity is used to indicate that these patterns of human choice and agency manifest in a semipredictable manner.3 While the role of context is essential, generalizable abstractions are also required to implement changes that may work in other contexts. Therefore, if researchers can identify abstract mechanisms that work within and possibly across contexts, then policy makers can develop interventions and policies for real world effects (see the Table).CMO configurations should help to explain why particular interventions succeed or fail, and how they influence outcomes. These theoretical explanations of influence are referred to as middle-range theories, that is, ones which involve abstraction yet remain close to the observed data.15 Such theoretical explanations connect empirical evidence, the rationale for practice, and contextual issues through a review of the literature. According to Pawson,1 there are 3 characteristics of middle-range theories: (1) sufficient abstraction advancing beyond descriptions or empirical generalizations; (2) logical derivation, that is, making transparent connections to empirical evidence; and (3) adaptive, cumulative explanations, which means accepting that theory will evolve and change with circumstances and new evidence.When designing an education program, educators consider what the program will achieve and how, but may not base decisions on formal educational theories such as constructivism. Realist researchers seeking to understand how an intervention or program works start by articulating underlying explanations for how the interventions (feedback for example) are expected to work. These are referred to as preliminary explanatory theories, initial program theory, or initial rough theory. Researchers then interrogate the existing evidence to adjudicate between initial theories and ascertain whether they are relevant in understanding mechanisms and observed outcomes. Hence, primary research is examined for its contribution to the developing theory.16For example, in a recent realist review undertaken by the authors,14 we sought to explain how feedback programs and interventions might lead to changes in behavior in learners. Our preliminary program theory took account of 3 theories for their potential explanatory powers: self-regulated learning (SRL),17 the educational alliance,18 and self-determination theory (SDT).19 All 3 theories have been used to explain feedback processes and tend to focus on what the trainee does. They also fit with our assumption that feedback is not solely due to the input of teachers, but how it is interpreted also influences learning. SRL might help us understand how feedback could support trainee goal construction, monitoring of performance, information seeking, and closing the gap. The educational alliance might offer different mechanisms for how feedback might influence some trainees within a certain context (eg, through relationship, dialogue, and co-construction). SDT as a theory of motivation might attune us to the conditions that promote trainees' internal motivations to engage with feedback processes. As you can see, all 3, at face value, offer a way to understand how feedback might work, for whom, and under what circumstances. The job of the realist reviewer is to test and refine these initial theories, by seeking evidence in the literature, understanding the relationships between different components based on existing research, and extending program theory in relation to the phenomenon under study.Realism as advanced by Pawson and Tilley2 is the underlying philosophy of realist research. Pawson1 translates Bhaskar's critical realism to a more pragmatic and operational perspective that enables empirical research into how a program works. According to Pawson,3 realism occupies a midpoint between constructivism and post-positivism. Pawson holds that there is a concrete view of reality—a material reality. In other words, reality exists but can only be imperfectly known as processed through human senses, brains, language, and culture. Realism holds that “while our knowledge will always be partial and imperfect, it can accrue over time.”20Pawson and colleagues21 describe the key steps for undertaking a realist review. Like other qualitative literature syntheses, a realist review is not a technical process that follows a set protocol, rather, judgements are made about the relevance and robustness of specific data for the purposes of answering the research question.20The initial phase of scoping the literature is iterative and non-linear. It can include grey literature or advance along different pathways. An important aspect of scoping is developing the program theory—an explanation for why and how a program works. Initial program theories might be developed through brainstorming, speaking with experts in the field, familiarity with the literature, and initial scoping of literature. These are then tested and refined through the analysis process.Searching the literature can be iterative and subsets of educational programs might be reviewed. Questions may be narrow or broad. For example, a review of effective research environments might include searches of specific interventions such as building research capacity, mentoring, or protected time. Resources, time, funding, etc will limit the breadth of the search; therefore, it is necessary to contain or focus the review by deciding on priorities for the lines of investigation. Issues of scale that might need to be managed include time frames, cultures, and countries. As with other reviews, snowballing (ie, identification of references from included papers) forms part of the search strategy. Librarian input is advised to guide researchers through the multitude of databases and the development of a parsimonious search strategy.Relevance and rigor are 2 commonly used criteria for appraising the literature. Relevance is a determination of whether the paper contributes to theory building. Rigor refers to the quality and robustness of the methods. Authors may use holistic appraisal of relevance and rigor or specific analytical tools such as the Critical Appraisal Skills Programme checklists (https://casp-uk.net/casp-tools-checklists/). Software such as Covidence can make this step more manageable across a research team.This step focuses on identifying CMO configurations. Realist analysis involves applying a realist philosophical lens to the data. This may be done in an Excel spreadsheet, comments in a PDF, or with software such as NVivo or Covidence. Data extraction is often qualitative and can be inductive and deductive.A researcher will seek out the CMO configurations in each paper, then seek to identify CMO configurations across the data set. As mentioned, there may be a series of mechanisms required to achieve outcomes. As you can imagine, this leads to multiple circular conversations about what constitutes the CMO configuration. Returning to our feedback example, if trust (M) exists between a supervisor and trainee (C), then feedback (intervention) leads to improved performance (O). Further, CMO configurations are not linear. For example, having feedback conversations can lead to increased trust and motivation. Therefore, we also can think of these causal links as reciprocal.22This stage relates to articulating an explanation of the patterns of CMO configurations identified in the previous stage. At issue is whether the identified CMO configuration can be used to justify, refute, or extend the program theory.When a particular explanatory program theory fails to explain the data, new ones are sought. In keeping with the example above in relation to feedback, an overwhelming number of demi-regularities supported SDT as an explanatory theory. Given that the context of this review was undergraduate-level, open-ended, written tasks, it is possible that a realist review of trainee feedback in clinical environments might lead to a different refined program theory.Publication standards have been established for the dissemination of realist reviews.15 A guideline priority is the need for transparency of process and reasoning, which can be shown through document flow diagrams such as PRISMA,15 tables, quotations, or diagrams showing the preliminary and refined program theories.The strength of realist reviews is clear: their ability to build and refine middle-range theory to explain how a program works and why it might not. Realist reviews enable the drawing of connections across different contexts.Working with qualitative and quantitative source papers is both a strength and a challenge. Together they strengthen theory development, which brings richness to the analysis. However, researchers need to be able to judge the quality of both approaches and to interpret findings in relation to the review research question. Quantitative studies can identify effects of a program on outcomes but not necessarily elucidate the mechanisms that mediate the effect. Alternatively, by their nature qualitative studies might highlight context and mechanisms, but not quantitative outcome measures.There are multiple limitations to consider when undertaking a realist review. Initial program theories are drawn from multiple sources. The breadth and depth of the researchers' knowledge of theory will affect the direction of the review. In addition, the data analysis and synthesis stage of a realist review is particularly time consuming, as is the challenge of identifying CMO configurations.Realist inquiry seeks to unpack the patterns of context-mechanism-outcome relationships which might explain why particular interventions succeed or fail, and how they influence outcomes in complex, open, adaptive systems. Put simply, realist inquiry asks what it is about the program or intervention that generates change. Thus, realist synthesis plays a key role in advancing theoretical explanations of interventions in medical education.