Realist Reviews: A Brief How-To.

Abstract

Realist reviews in medical education are designed to add to our understanding of what intervention works, for whom, how, and in what circumstances. The foundational premise of realism is that interventions work differently in different contexts and for different people.1 Realist reviews add to our understanding of how context and mechanisms relate to educational outcomes, referred to as context-mechanism-outcome (CMO) configurations.The steps below introduce key elements when considering a realist review.Context matters: education programs that are effective in one context may not be effective elsewhere. Educational literature reviews tend to explore if a program works; a realist review seeks to understand how a program works and for whom. A realist review seeks to understand why a program might be effective in some contexts and for some learners but not others, and it is useful when seeking to understand both intended and unintended outcomes.A realist review is theory driven. Researchers will propose an initial program theory (or theories) as a potential explanation for how a program works, which is then refined iteratively as the review progresses. The initial program theory may come from the formal program documentation, education literature, expert stakeholders, and/or the researchers' own experiences. A refined theory is the result of a realist review and would be expected to influence the design and implementation of education programs in the future.2Familiarize yourself with the formal guidelines for conducting a realist review3 and realist review reporting standards.4 Strategically assemble a research team to include both content and realist method expertise. Consider seeking input from a librarian or information specialist. Develop initial program theories through brainstorming, speaking with experts in the field, familiarity with the literature, and initial scoping of literature.Initial scoping of the literature is required before determining the area of focus. Then a formal literature search is undertaken. A later search may also be required to seek out additional studies to refine the program theory as the theory develops. Literature selection is not limited to a particular study design type but is determined by its usefulness and contributions to theory. Literature is sought that provides information about CMO configurations. A document flow diagram is useful to show the number of articles assessed for eligibility and included in the review, with reasons for exclusion at each stage.Literature is assessed for inclusion based on relevance (paper's ability to contribute to theory building or theory testing) and rigor (methods used to generate the data are credible and trustworthy). A realist review does not seek to exclude literature based on the traditional hierarchy of evidence, as multiple methods may contribute usefully to theory building. However, papers judged as low quality may be excluded especially when dealing with large data sets. Alternatively, they may be included to inform understandings of context even if not considered for answering whether a complex intervention is effective. Thus, when studies are deemed low in quality, they may be retained for theory development or for testing in a subsequent realist evaluation.Each manuscript is coded for CMO configurations and for its potential to inform program theory. Both quantitative and qualitative data may be considered relevant and extracted. Deductive and inductive coding are relevant when seeking to understand reasons for positive and negative outcomes. Coding for both positive and negative outcomes enables further exploration of where and why programs might not work and for whom. Data to inform a program theory may be found in any manuscript section, not just the results section. Starting with the outcomes and working backward to identify the mechanism and context can be helpful. Another strategy is to try to articulate the CMO configuration as an “if, then” statement.The purpose of synthesis is to refine the program theory. This may involve testing the initial theory, adjudicating between different theories, or reviewing how the theory holds within different settings. In this process researchers find and align the evidence to demonstrate that particular mechanisms generate particular outcomes. This involves looking for CMO configurations that occur across the literature. Expert stakeholders may again be useful to contribute to building and confirming the program theory.Realist reviews can extend our understanding of how a program works, with statements that make explicit the C and M which will contribute to an educational O. Quantitative and qualitative data, both confirming and disconfirming, from primary studies are sought and used to support the theoretical explanation. This new explanation can inform future educational interventions. Contexts where a program is unsuccessful or mechanisms that are key to success should be highlighted. Please see Box 1 for further resources and Box 2 for examples of realist reviews in medical education.