Realist randomised controlled trials: A new approach to evaluating complex public health interventions

BackgroundRandomised clinical trials with a factorial design may assess the effects of multiple interventions in the same population. Factorial trials are carried out under the assumption that the trial interventions have no interactions on outcomes. Here, we present a protocol for a simulation study investigating the consequences of different levels of interactions between the trial interventions on outcomes for the future 2×2×2 factorial designed randomised clinical Sedation, TEmperature, and Pressure after Cardiac Arrest and REsuscitation (STEPCARE) trial in comatose patients after out-of-hospital cardiac arrest.MethodsBy simulating a multisite trial with 50 sites and 3278 participants, and a presumed six-month all-cause mortality of 60% in the control population, we will investigate the validity of the trial results with different levels of interaction effects on the outcome. The primary simulation outcome of the study is the risks of type-1 and type-2 errors in the simulated scenarios, i.e. at what level of interaction is the desired alpha and beta level exceeded. When keeping the overall risk of type-1 errors ≤ 5% and the risk of type-2 errors ≤ 10%, we will quantify the maximum interaction effect we can accept if the planned sample size is increased by 5% to take into account possible interaction between the trial interventions. Secondly, we will assess how interaction effects influence the minimal detectable difference we may confirm or reject to take into account 5% (small interaction effect), 10% (moderate), or 15% (large) positive interactions in simulations with no ‘true’ intervention effect (type-1 errors) and small (5%), moderate (10%), or large negative interactions (15%) in simulations with ‘true’ intervention effects (type-2 errors). Moreover, we will investigate how much the sample size must be increased to account for a small, moderate, or large interaction effects.DiscussionThis protocol for a simulation study will inform the design of a 2×2×2 factorial randomised clinical trial of how potential interactions between the assessed interventions might affect conclusions. Protocolising this simulation study is important to ensure valid and unbiased results.Trial registrationNot relevant

Realist reviews in medical education are designed to add to our understanding of what intervention works, for whom, how, and in what circumstances. The foundational premise of realism is that interventions work differently in different contexts and for different people.1 Realist reviews add to our understanding of how context and mechanisms relate to educational outcomes, referred to as context-mechanism-outcome (CMO) configurations.The steps below introduce key elements when considering a realist review.Context matters: education programs that are effective in one context may not be effective elsewhere. Educational literature reviews tend to explore if a program works; a realist review seeks to understand how a program works and for whom. A realist review seeks to understand why a program might be effective in some contexts and for some learners but not others, and it is useful when seeking to understand both intended and unintended outcomes.A realist review is theory driven. Researchers will propose an initial program theory (or theories) as a potential explanation for how a program works, which is then refined iteratively as the review progresses. The initial program theory may come from the formal program documentation, education literature, expert stakeholders, and/or the researchers' own experiences. A refined theory is the result of a realist review and would be expected to influence the design and implementation of education programs in the future.2Familiarize yourself with the formal guidelines for conducting a realist review3 and realist review reporting standards.4 Strategically assemble a research team to include both content and realist method expertise. Consider seeking input from a librarian or information specialist. Develop initial program theories through brainstorming, speaking with experts in the field, familiarity with the literature, and initial scoping of literature.Initial scoping of the literature is required before determining the area of focus. Then a formal literature search is undertaken. A later search may also be required to seek out additional studies to refine the program theory as the theory develops. Literature selection is not limited to a particular study design type but is determined by its usefulness and contributions to theory. Literature is sought that provides information about CMO configurations. A document flow diagram is useful to show the number of articles assessed for eligibility and included in the review, with reasons for exclusion at each stage.Literature is assessed for inclusion based on relevance (paper's ability to contribute to theory building or theory testing) and rigor (methods used to generate the data are credible and trustworthy). A realist review does not seek to exclude literature based on the traditional hierarchy of evidence, as multiple methods may contribute usefully to theory building. However, papers judged as low quality may be excluded especially when dealing with large data sets. Alternatively, they may be included to inform understandings of context even if not considered for answering whether a complex intervention is effective. Thus, when studies are deemed low in quality, they may be retained for theory development or for testing in a subsequent realist evaluation.Each manuscript is coded for CMO configurations and for its potential to inform program theory. Both quantitative and qualitative data may be considered relevant and extracted. Deductive and inductive coding are relevant when seeking to understand reasons for positive and negative outcomes. Coding for both positive and negative outcomes enables further exploration of where and why programs might not work and for whom. Data to inform a program theory may be found in any manuscript section, not just the results section. Starting with the outcomes and working backward to identify the mechanism and context can be helpful. Another strategy is to try to articulate the CMO configuration as an "if, then" statement.The purpose of synthesis is to refine the program theory. This may involve testing the initial theory, adjudicating between different theories, or reviewing how the theory holds within different settings. In this process researchers find and align the evidence to demonstrate that particular mechanisms generate particular outcomes. This involves looking for CMO configurations that occur across the literature. Expert stakeholders may again be useful to contribute to building and confirming the program theory.Realist reviews can extend our understanding of how a program works, with statements that make explicit the C and M which will contribute to an educational O. Quantitative and qualitative data, both confirming and disconfirming, from primary studies are sought and used to support the theoretical explanation. This new explanation can inform future educational interventions. Contexts where a program is unsuccessful or mechanisms that are key to success should be highlighted. Please see Box 1 for further resources and Box 2 for examples of realist reviews in medical education.

What is complexity and what do you do with it? Reflections on use of controlled trials to assess complex public health interventions

Global evidence synthesis and UK idiosyncrasy: why have recent UK trials had no significant effects on breastfeeding rates?

Een Program Theory benadering voor het theoretisch onderbouwen van sociale interventies: een casestudie van vijf Nederlandse maatjesprojecten

Female genital mutilation/cutting (FGM/C) is a traditional practice that involves the partial or total removal or other injury to the female genital organs for cultural or other non-therapeutic reasons (WHO, 2008). The current classification describes four types of FGM/C: Type 1, clitoridectomy involves partial or total removal of the clitoris and/or the prepuce. Type 2, excision, involves partial or total removal of the clitoris and the labia minora, with or without excision of the labia majora. Type 3, infibulation, involves narrowing of the vaginal orifice with creation of a covering seal by cutting and appositioning the labia minora and/or the labia majora, with or without excision of the clitoris. Infibulation is considered the most invasive type of FGM/C. Defibulation, opening of the covering seal, is often necessary prior to childbirth. Reinfibulation refers to the recreation of an infibulation after defibulation. Type 4, other, involves all other harmful procedures to the female genitalia for non-medical purposes, for example: pricking, piercing, incising, scraping and cauterizing (WHO, 1997). FGM/C is practised in more than 28 countries in Africa, usually on girls under the age of 15 years, and in some countries in the Middle East and Asia. FGM/C is also practised by immigrant communities in a number of other countries, including Australia, Canada, France, New Zealand, Norway, Sweden, Switzerland, the United Kingdom, and the United States (HRP, 2006). Recent figures for African countries show a prevalence of FGM/C of more than 70 percent in Burkina Faso, Djibouti, Egypt, Eritrea, Ethiopia, Guinea, Mali, Mauritania, Northern Sudan, and Somalia (Yoder & Kahn, 2008). However, there is great variation in prevalence between and within countries, reflecting ethnicity and tradition. Therefore, UNICEF (2005) has proposed that countries be categorized in three groups according to FGM/C prevalence rates: Group 1, 80 percent or higher prevalence, e.g. Ethiopia and Somalia, Group 2, 25-79 percent, e.g. Senegal and Kenya, and Group 3, 1-24 percent, e.g. Nigeria (Table 1). The practice of FGM/C is rooted in religious, personal and societal beliefs within a frame of psycho-sexual and social reasons such as control of women's sexuality and family honour, which is enforced by community mechanisms (WHO, 1999). While reasons for the practice vary across cultural groups, social reasons may include FGM/C as an initiation act for girls into womanhood, as an act of social integration and for the maintenance of social cohesion. Socio-economic reasons include beliefs that FGM/C is a prerequisite for marriage or an economic necessity in cases where women are largely dependent on men. Religious reasons rest on the belief that it is a religious requirement. Hygienic and aesthetic reasons for FGM/C include beliefs that the female genitalia are dirty and unsightly, and health reasons include beliefs that FGM/C enhances fertility and child survival. FGM/C may also be an important source of income for circumcisers (UNFPA, 2007). Girls exposed to FGM/C are at risk of immediate physical consequences, such as severe pain, bleeding, and shock, difficulty in passing urine and faeces, and infections. Long term consequences can include chronic pain and infections (WHO, 2008). In general, the consequences are similar for FGM/C Type I, II, and III, but they tend to be more severe and more prevalent the more extensive the procedure (WHO, 2008). A systematic review of the health complications of FGM/C (WHO, 2000) identified a range of obstetrical problems, the most common being prolonged labour and/or obstruction, episiotomies and perineal tears, post partum haemorrhage, and maternal and foetal death. A recent study investigating 28,393 women attending 28 obstetric centres in several African countries (Banks et al, 2006) concluded that women with FGM/C are significantly more likely than those without to have adverse obstetric outcomes such as a caesarean, postpartum blood loss 500 mL, extended materna I hospital stay, birth weight <2500 g, infant resuscitation, and inpatient perinatal death. The authors also concluded that the risks seemed to be greater with more extensive FGM/C. For many girls and women, undergoing FGM/C is a traumatic experience that leaves a lasting psychological mark and may adversely affect their mental health. In fact, several psychological and psychosomatic disorders such as disordered eating and sleeping habits have been attributed to FGM/C (HRP, 2006). There are also reports of posttraumatic stress disorder, anxiety, and depression associated with FGM/C (WHO, 2008). FGM/C is a deeply entrenched social convention among some ethnic groups and as such carries consequences both when it is and when it is not practised. When girls and families conform to the practice they acquire social status and respect. For girls, undergoing FGM/C promotes honour and her full acceptance in the community, as well as imparts a sense of pride and of coming of age (UNICEF, 2005). In some societies, the link between FGM/C and value is explicit: girls who undergo FGM/C often receive rewards in the form of celebrations and gifts, and the bride price for a girl who has been cut is much higher than that for one who has not (Wheeler, 2003). For families, fulfilling the cultural expectation that girls should be cut assigns status and community membership. Conversely, failure to conform leads to difficulty in finding a husband for the girl, shame, stigmatization, as well as loss of social status, honour and protection, resulting in the family's social exclusion in the community (UNICEF, 2005). Sexual consequences of FGM/C were summarized in a non-systematic literature review (Obermayer, 2005), which concluded that the available evidence does not support the notion that FGM/C automatically precludes sexual activity or the enjoyment of sexual relations. The results from a recent systematic review by the authors of the present project proposal suggest that women with FGM/C experience pain and reduction in sexual satisfaction and desire compared to women without FGM/C. The evidence base is however insufficient to draw causal conclusions about the consequences of FGM/C (Berg et al, 2010a). Efforts to abandon the practice of FGM/C in Africa have used several different approaches which, in turn, have had implication for interventions. These approaches include those based on human rights frameworks, legal mechanisms, health risks, alternative rites, positive deviance, training health workers as change agents, training and converting circumcisers, and the use of comprehensive social development processes. Interventions based on these approaches have targeted stakeholders at individual, interpersonal, community and national levels (Muteshi & Sass, 2005). In 2007, the Population Reference Bureau (PRB) published their results of an extensive survey of current intervention projects taking place in African countries (Feldman-Jacobs & Ryniak, 2007). In total, PRB identified 92 projects, 27 of which were evaluated, mostly by observational designs. Only four of the 27 evaluated projects (15%) used a controlled before-and-after design, and about a dozen of the evaluations used cross-sectional or pre-post intervention questionnaires or interviews without a comparison group. While contributing great understanding about the range of interventions initiated to curb the prevalence of FGM/C, the overview did not reach any conclusions about the effectiveness of interventions. More recently, the authors of the proposed project specifically examined the effectiveness of interventions to reduce the prevalence of FGM/C in a systematic review (Denison et al, 2009). Through our literature search of February 2009 we identified a total of seven controlled studies, six of which could be obtained in full text. All six studies were controlled before-and-after studies carried out in African countries. In contrast to the PRB overview (Feldman-Jacobs & Ryniak, 2007), we included only controlled studies, i.e. studies with reference to a non-intervention comparison group, and we concluded that while the evidence base is insufficient to draw definite conclusions, there are possible advantageous developments as a result of interventions. Notably, the review highlighted the uncertainties regarding relevance of the interventions (e.g. regarding objectives, intervention targets, activities). That is, since it was not a focus of the systematic review, we were unable to provide any assessment of the degree to which the interventions were appropriate responses to the populations' needs with respect to FGM/C, including the degree to which factors that contribute to the perpetuation of the practice were taken into account in the interventions. It is apparent that the degree of relevance of the intervention exerts a considerable influence on an intervention's effectiveness in reaching its designated goals, and may to a large extent help explain variation in behavioral and other outcomes among members of groups. In sum, two recent publications have examined aspects of interventions designed to reduce the prevalence of FGM/C. However, the effectiveness of interventions in the context of relevance has yet to be explored. FGM/C is a long-standing tradition that has become inseparable from ethnic and social identity among many groups (UNICEF, 2005). Disaggregation of data from the Demographic and Health Surveys (DHS)2 shows that the practice of FGM/C varies by demographic variables such as age, urban-rural residence, and region or province, and also by variables such as education, ethnicity and religion (Yoder et al, 2004). Further analysis of DHS data by UNICEF suggests that educational attainment, a woman's own circumcision status and ethnicity have the greatest influence in explaining support or opposition to the practice (UNICEF, 2005). Thus, programmes designed to reduce the prevalence of FGM/C should be country specific and adapted to reflect regional, ethnic and socio-economic variances while also taking into account the diverse reasons why FGM/C is practised among a given ethnic or cultural group (UNICEF, 2005). The review will summarize data relating to 1) key intervention program features, targeted participants, main outcomes, and estimates of intervention effectiveness and 2) factors related to the continuance and discontinuance of FGM/C, such as demographic factors, the frequency and strength of various stakeholders' cognitions and behaviours related to FGM/C, and stakeholders' lived understanding of the persistence of the practice. This will allow us to analyse not only effectiveness of interventions but also their relevance, i.e. the extent to which intervention programs have heeded and built upon factors related to the continuance and discontinuance of FGM/C, the extent to which interventions have been provided to the most appropriate stakeholder groups, and which forces have been overlooked as critical program elements. Overall, we will conduct the review according to the guidelines in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins & Green, 2009) and guidelines for systematic reviews in the social sciences (Petticrew & Roberts, 2006). Answering research question 2 will involve synthesizing quantitative and qualitative evidence and we will follow guidance by Pope et al (2007). Briefly, we will proceed with the review in the following manner: search for empirical studies, screen studies, extract data, summarize data, perform analyses, and write up results. Selection of primary studies will be based on the inclusion criteria described above. The Reference Manager database containing the search results will be used to keep track of references identified through the electronic database search (to be screened for inclusion). Screening of literature will proceed at two levels. For level 1 screening, two reviewers (RB and ED) will independently perform an assessment of the identified records by reading the title, and when available, abstract. The pre-developed inclusion questions for level 1 are based on the inclusion criteria described in 3.1.1 and 3.1.2. They are listed in Appendix 1. The reviewers answer each question "Yes" (= promote), or "Can't tell" (= promote), or "No" (=exclude, do not promote). The reviewers then compare and discuss their judgments. Differences in opinion at level 1 screening will be resolved by promoting the record to level 2 screening. Records that unmistakably fail to meet the inclusion criteria will be excluded, such as editorials and commentaries. Records not excluded at level 1 are promoted to level 2 screening, and ordered in full text. At level 2 screening, two reviewers (RB and ED) will independently evaluate the full text of each record promoted from level 1 screening for inclusion, in accordance with Cochrane guidelines. They will use a pre-developed inclusion form (Appendix 1) based on the inclusion criteria described in 3.1.1 and 3.1.2. There will be a separate set of screening questions for records describing the effect of interventions and records reporting on reasons for the perpetuation of FGM/C (Appendix 1)). The reviewers answer each question "Yes" (= Include), "No" (= Exclude) or "Can't tell" (=Discuss). The reviewers then compare and discuss their assessments. Differences in assessment at level 2 screening will be discussed until consensus is reached. If consensus can't be reached, a third review member (SL or JOJ) will be asked to resolve disagreements. The reference is included when the reviewers agree to score "Yes" to all questions. The reference is excluded when the reviewers agree to score "No" to any one question. If the reviewers score "Can't tell" to any one question, the inclusion question will be resolved by re-reading of the text, discussion and consensus (or resolved by a third person if consensus can't be reached). The main reason for exclusion at this stage will be recorded for each record, and a list of excluded records (with reasons) will be created. These steps are in accordance with the Cochrane Handbook (Higgins & Green, 2009) In the interest of time, the reviewers will not at any screening level be blinded to the authors or other information about the record when assessing the studies. When there is more than one record of the same study, we will include all records meeting the inclusion criteria, but use the most relevant one, i.e. the publication containing the most complete data set, as the main record. Once included, we will group all included studies according to their methodological focus into three main study types: 1) effectiveness studies, 2) quantitative views studies, 3) qualitative views studies. The primary method of study identification will be electronic searches, as advised by the Cochrane Handbook (Higgins & Green, 2009)). For the recently completed systematic review of the effectiveness of interventions designed to prevent the prevalence of FGM/C we searched systematically for relevant literature up to February 2009 in 13 international databases: African Index Medicus, Anthropology Plus, British Nursing Index and Archive, The Cochrane Library (CENTRAL, Cochrane Database of systematic Reviews, Database of Abstracts of Reviews of Effects), EMBASE, EPOC, MEDLINE, PILOTS, POPLINE, PsychINFO, Social Services Abstracts, Sociological Abstracts, and WHOLIS. Under the guidance of one reviewer (ED) a research librarian performed the searches using a strategy incorporating subject headings (for example MeSH terms in MEDLINE, see http://www.nlm.nih.gov/mesh/) and text words (in title and abstract) relating to FGM/C and the four classifications thereof. No method filters were applied as we were more concerned about sensitivity than specificity and prepared to screen a large number of references. We did not restrict our searches by country or language. The MEDLINE search strategy served as the model for the other database searches using appropriate controlled vocabulary as applicable. Our search strategy for MEDLINE is shown: We will include databases for 'grey' literature (defined here as "reports that are produced by all levels of government, academics, business and industry in print and electronic formats but that are not controlled by commercial publishers" [Higgins & Green 2009]): Demographic and Health Surveys, British Library for Development Studies, IDEAS, JOLIS, Google Scholar, and Google. Theses and dissertations will be included. In addition to the electronic database search, we will perform bibliographic back-referencing to identify new leads. We will also supplement the electronic search with searches in databases of six international organizations that are engaged in projects regarding FGM/C: Centre for Development and Population Activities (CEDPA), Population Council, Population Reference Bureau (PRB), The United Nations Children's Fund (UNICEF), The United Nations Population Fund, and The World Health Organization (WHO). Notably, Popline contains ministerial reports on FGM/C. We will also ask for suggestions for literature we may have missed from FGM/C experts, e. g. the research director of Population Council. Forward citation tracking will be performed through the ISI Web of Knowledge in order to identify further studies. Lastly, we will conduct a hand search of relevant journals (African Journal of Reproductive Health, Social Science & Medicine). Two authors (RB and ED) will independently extract data from the published sources using a pre-designed data extraction form, as recommended in the Cochrane Handbook. Data to be extracted from studies included for answering Research question 1: publication identification details (author, year, and source), study design, study setting, population, intervention details, comparisons, outcomes, follow-up details, attrition and missing data details, outcome data (for dichotomous data: number of events, number of persons in the groups, and p-values; for continuous data: means, standard deviations, and p-values), and effect estimates. Intervention details will be classified according to a taxonomy of behaviour change techniques used in interventions proposed by Abraham and Michie (2008). The data extraction form for RQ 1 appears in Appendix 4. Data to be extracted from studies included for answering Research question 2: publication identification details (author, year, and source), study design, study setting, population, quantitative or qualitative data related to the continuance and discontinuance of FGM/C, i.e. demographic factors, the frequency and strength of various stakeholders' cognitions and behaviours related to FGM/C, and stakeholders' lived understanding of the practice. With respect to qualitative-based studies, the reviewers will read the texts independently of each other and extract all text data related to views about FGM/C in light of stakeholders' reasoning for its continuance and discontinuance. Our extraction of data will be inclusive (Briggs et al 2007). We will copy all in the form of or text to with reasons for continuance and discontinuance our pre-designed data extraction The data extraction form for RQ 2 appears in Appendix extraction of data, the two reviewers will compare their and resolve any by discussion and the text. Two authors (RB and ED) will independently the of studies, using appropriate for included study they will compare and discuss their assessments. If consensus can't be reached, a third person will be asked to resolve disagreements. included for answering Research question 1 will be by The Health for (Appendix The Cochrane Handbook this for reviews in the health (Higgins & Green, Reviews in health and health The the following study design, data and intervention and A are each given a of or according to a of or is according to the following with than four and one or more A of or methodological will be upon by the reviewers after there was a with respect to the and will be in studies included for answering Research question 2 will be according to guidelines for prevalence studies The guidelines and analysis which have been into a seven criteria by at The Knowledge Centre for the Health Services (Appendix A of or study is given according to the all criteria are change the of the criteria are not or not may change criteria are or the may be criteria The reviewers will agree on a of or methodological (or after there was a with respect to the seven studies included for answering Research question 2 will be by a designed by the 2006). The questions the and relevance of qualitative studies. Two questions are screening questions to it is with the questions. The questions research design, data of data and value of the The does not guidance as to the of studies. We will the same procedure as for the assessment of prevalence studies. For the of effectiveness evidence question 1, 1, Appendix the studies will be by and key intervention features, outcomes and effect estimates will be described in will be used to the of the effect estimates in to of studies. With respect to analyses, we will present dichotomous data for the outcomes listed in the inclusion criteria in results when and post for both intervention and comparison groups are by study We will of interventions in two we effect by the risk in which the pre-post change score (in in the comparison group will be from the pre-post change score (in in the intervention group. is to be large is a we also into we will effect by the risk and percent based on The of using this is that effect can be at be to of the effect studies we have identified is we will to use this studies be identified in the search and included in the review we will these is that some of of will be present when conclusions are from the results. We will present continuous data with and of effect for each of outcomes will be or will be used to sources of If of studies are identified of studies with one dichotomous will be used to perform variables are intervention and of (e.g. educational training health workers as change agents, comprehensive social stakeholder and The of quantitative and qualitative data describing factors will to the understanding of results from the intervention studies, and to help interventions more in to their question We will an evidence quantitative evidence has methodological Data extraction and of quantitative and qualitative evidence will largely be completed in separate 1, Appendix Data from cross-sectional survey studies will be with data from studies which examined various stakeholders' of factors the continuance and discontinuance of FGM/C Our evidence is largely based on published and guidelines from the for and and Centre et et & 2008). The will be et 2006) and focus on data by similar data from the quantitative studies and the qualitative studies. we will the two of evidence is from the quantitative data set will be used as for the qualitative data the analysis we will place most weight on the quantitative such that the qualitative results are under the quantitative results and the qualitative results the results from the quantitative With respect to the quantitative studies 2, 1, Appendix we will use a 2009) to the extracted quantitative data across studies. for each study and each group of we will review the extracted and according to our such as reasons for FGM/C. We will then the of these in order to a list of factors, up with one list for each stakeholder group in each Thus, we will be to both data across studies and With respect to the qualitative evidence 3, 1, Appendix our analysis is and on published studies et et 2006). analysis involves or in the literature and the of the different studies under headings 2005). The data for of qualitative studies are in text In order to we will copy all into a These data include both from research and of the by the We will the from each study according to the factors are continuance or discontinuance factors, for each stakeholder group we will the of each study in and to the will be without the of the data and to and of each or For we will the finding perform to reduce a sexual under continuance factors as women's the the reviewers will for and between in order to we will group into This will be based on of as well as frequency and strength of various stakeholders' cognitions about FGM/C, for each stakeholder group. with it involves taking from one study and the same in study, they may not have been using by analysis can be data i.e. by the identified in the studies that are included, or i.e. by identified through assessment of the For this systematic review, given quantitative evidence will be given methodological we will by using both a from the included quantitative studies to out evidence from the qualitative as well as to from the qualitative to with related will be identified and into to which we such as sexual and community We

A realist review is a theory-driven approach to literature synthesis that seeks to evaluate the effectiveness of interventions in a contextualized way. Realist research, which includes review and program evaluation, is a relatively recent development. Pioneered by Pawson,1-3 this approach has proliferated in health services because realist research acknowledges that program interventions are complex, and context is fundamental to understanding how outcomes are mediated. The foundational premise of realism is that interventions work differently in different contexts and for different people. In medical education, realist reviews have been conducted to understand internet-based medical education,4 balancing health professional education and patient care,5 productive research environments,6 clinical reasoning,7 and interprofessional programs.8 (See the Box for a case example of an interprofessional education realist review.)The purpose of a realist review is to build theory and new understandings of causal mechanisms from existing research by focusing on the relationship between context and outcomes. A critical feature of complex interventions is that "as they are delivered, they are embedded in social systems. It is through the workings of entire systems of social relationships that any changes in behaviours, events and social conditions are effected."3 Complex interventions include multiple interrelated components, which occur over time, in intricate environments, with multiple stakeholders. In a realist review, context is more than a setting or a community; it includes individual, institution, practice, social, technical, and other layers, as described by Bates and Ellaway.9The key realist research question is typically: What intervention works, for whom, how, and in what circumstances?Realist reviews differ from other reviews in approach to causality and attention to mechanisms of change and human action. Given that interventions are embedded within social systems, causality is understood as dependent on the whole context of an intervention.11 How an intervention or program causes an outcome is not simple, linear, or deterministic. Programs will have multiple mechanisms that lead to different outcomes, both positive and negative.Understanding mechanisms is the crux of understanding realist reviews. While there is no consensus on a single definition, broadly speaking, mechanisms cause things to happen. Mechanisms can be defined as the "underlying entities, processes, or structures which operate in particular contexts to generate outcomes of interest."12 A mechanism may cause changes in individual beliefs, values, or reasoning or may change the social structures and resources available to individuals.13 Mechanisms are not necessarily visible but can mediate effects in certain contexts. For example, trust might be a mechanism by which feedback could lead a trainee to change their behavior in the context of a supervisory relationship. Motivation might be another mechanism for how feedback interventions work. It is not difficult to accept that both can operate in any single feedback intervention, which is why multiple mechanisms may be identified in a single review.Realist reviews seek to unpack mechanisms that mediate outcomes within specific contexts. These context-mechanism-outcome (CMO) configurations may occur in regular patterns, also called demi-regularities, which is another defining feature of realist reviews. The term demi-regularity is used to indicate that these patterns of human choice and agency manifest in a semipredictable manner.3 While the role of context is essential, generalizable abstractions are also required to implement changes that may work in other contexts. Therefore, if researchers can identify abstract mechanisms that work within and possibly across contexts, then policy makers can develop interventions and policies for real world effects (see the Table).CMO configurations should help to explain why particular interventions succeed or fail, and how they influence outcomes. These theoretical explanations of influence are referred to as middle-range theories, that is, ones which involve abstraction yet remain close to the observed data.15 Such theoretical explanations connect empirical evidence, the rationale for practice, and contextual issues through a review of the literature. According to Pawson,1 there are 3 characteristics of middle-range theories: (1) sufficient abstraction advancing beyond descriptions or empirical generalizations; (2) logical derivation, that is, making transparent connections to empirical evidence; and (3) adaptive, cumulative explanations, which means accepting that theory will evolve and change with circumstances and new evidence.When designing an education program, educators consider what the program will achieve and how, but may not base decisions on formal educational theories such as constructivism. Realist researchers seeking to understand how an intervention or program works start by articulating underlying explanations for how the interventions (feedback for example) are expected to work. These are referred to as preliminary explanatory theories, initial program theory, or initial rough theory. Researchers then interrogate the existing evidence to adjudicate between initial theories and ascertain whether they are relevant in understanding mechanisms and observed outcomes. Hence, primary research is examined for its contribution to the developing theory.16For example, in a recent realist review undertaken by the authors,14 we sought to explain how feedback programs and interventions might lead to changes in behavior in learners. Our preliminary program theory took account of 3 theories for their potential explanatory powers: self-regulated learning (SRL),17 the educational alliance,18 and self-determination theory (SDT).19 All 3 theories have been used to explain feedback processes and tend to focus on what the trainee does. They also fit with our assumption that feedback is not solely due to the input of teachers, but how it is interpreted also influences learning. SRL might help us understand how feedback could support trainee goal construction, monitoring of performance, information seeking, and closing the gap. The educational alliance might offer different mechanisms for how feedback might influence some trainees within a certain context (eg, through relationship, dialogue, and co-construction). SDT as a theory of motivation might attune us to the conditions that promote trainees' internal motivations to engage with feedback processes. As you can see, all 3, at face value, offer a way to understand how feedback might work, for whom, and under what circumstances. The job of the realist reviewer is to test and refine these initial theories, by seeking evidence in the literature, understanding the relationships between different components based on existing research, and extending program theory in relation to the phenomenon under study.Realism as advanced by Pawson and Tilley2 is the underlying philosophy of realist research. Pawson1 translates Bhaskar's critical realism to a more pragmatic and operational perspective that enables empirical research into how a program works. According to Pawson,3 realism occupies a midpoint between constructivism and post-positivism. Pawson holds that there is a concrete view of reality—a material reality. In other words, reality exists but can only be imperfectly known as processed through human senses, brains, language, and culture. Realism holds that "while our knowledge will always be partial and imperfect, it can accrue over time."20Pawson and colleagues21 describe the key steps for undertaking a realist review. Like other qualitative literature syntheses, a realist review is not a technical process that follows a set protocol, rather, judgements are made about the relevance and robustness of specific data for the purposes of answering the research question.20The initial phase of scoping the literature is iterative and non-linear. It can include grey literature or advance along different pathways. An important aspect of scoping is developing the program theory—an explanation for why and how a program works. Initial program theories might be developed through brainstorming, speaking with experts in the field, familiarity with the literature, and initial scoping of literature. These are then tested and refined through the analysis process.Searching the literature can be iterative and subsets of educational programs might be reviewed. Questions may be narrow or broad. For example, a review of effective research environments might include searches of specific interventions such as building research capacity, mentoring, or protected time. Resources, time, funding, etc will limit the breadth of the search; therefore, it is necessary to contain or focus the review by deciding on priorities for the lines of investigation. Issues of scale that might need to be managed include time frames, cultures, and countries. As with other reviews, snowballing (ie, identification of references from included papers) forms part of the search strategy. Librarian input is advised to guide researchers through the multitude of databases and the development of a parsimonious search strategy.Relevance and rigor are 2 commonly used criteria for appraising the literature. Relevance is a determination of whether the paper contributes to theory building. Rigor refers to the quality and robustness of the methods. Authors may use holistic appraisal of relevance and rigor or specific analytical tools such as the Critical Appraisal Skills Programme checklists (https://casp-uk.net/casp-tools-checklists/). Software such as Covidence can make this step more manageable across a research team.This step focuses on identifying CMO configurations. Realist analysis involves applying a realist philosophical lens to the data. This may be done in an Excel spreadsheet, comments in a PDF, or with software such as NVivo or Covidence. Data extraction is often qualitative and can be inductive and deductive.A researcher will seek out the CMO configurations in each paper, then seek to identify CMO configurations across the data set. As mentioned, there may be a series of mechanisms required to achieve outcomes. As you can imagine, this leads to multiple circular conversations about what constitutes the CMO configuration. Returning to our feedback example, if trust (M) exists between a supervisor and trainee (C), then feedback (intervention) leads to improved performance (O). Further, CMO configurations are not linear. For example, having feedback conversations can lead to increased trust and motivation. Therefore, we also can think of these causal links as reciprocal.22This stage relates to articulating an explanation of the patterns of CMO configurations identified in the previous stage. At issue is whether the identified CMO configuration can be used to justify, refute, or extend the program theory.When a particular explanatory program theory fails to explain the data, new ones are sought. In keeping with the example above in relation to feedback, an overwhelming number of demi-regularities supported SDT as an explanatory theory. Given that the context of this review was undergraduate-level, open-ended, written tasks, it is possible that a realist review of trainee feedback in clinical environments might lead to a different refined program theory.Publication standards have been established for the dissemination of realist reviews.15 A guideline priority is the need for transparency of process and reasoning, which can be shown through document flow diagrams such as PRISMA,15 tables, quotations, or diagrams showing the preliminary and refined program theories.The strength of realist reviews is clear: their ability to build and refine middle-range theory to explain how a program works and why it might not. Realist reviews enable the drawing of connections across different contexts.Working with qualitative and quantitative source papers is both a strength and a challenge. Together they strengthen theory development, which brings richness to the analysis. However, researchers need to be able to judge the quality of both approaches and to interpret findings in relation to the review research question. Quantitative studies can identify effects of a program on outcomes but not necessarily elucidate the mechanisms that mediate the effect. Alternatively, by their nature qualitative studies might highlight context and mechanisms, but not quantitative outcome measures.There are multiple limitations to consider when undertaking a realist review. Initial program theories are drawn from multiple sources. The breadth and depth of the researchers' knowledge of theory will affect the direction of the review. In addition, the data analysis and synthesis stage of a realist review is particularly time consuming, as is the challenge of identifying CMO configurations.Realist inquiry seeks to unpack the patterns of context-mechanism-outcome relationships which might explain why particular interventions succeed or fail, and how they influence outcomes in complex, open, adaptive systems. Put simply, realist inquiry asks what it is about the program or intervention that generates change. Thus, realist synthesis plays a key role in advancing theoretical explanations of interventions in medical education.

The application of realist synthesis review methods in public health economics

Despite its low profile on the child survival agenda, diarrhoea is the second leading killer of children under 5 years of age and accounts for 10% of child deaths every year.1 But although diarrhoeal disease is preventable and can be managed with low-cost interventions, progress in reducing its incidence in children has been slow in recent years.2 In 2009, the United Nations Children’s Fund (UNICEF) and the World Health Organization (WHO) jointly published Diarrhoea: why children are still dying and what can be done, a report intended to raise awareness of the issue and to lay out a comprehensive plan of action for reducing the incidence of childhood diarrhoea and its associated mortality.3 In June 2012, UNICEF issued another call to action in Pneumonia and diarrhoea: tackling the deadliest diseases for the world's poorest children and urged a refocusing of efforts to reduce preventable deaths caused by diarrhoea.4 This most recent UNICEF report underscores the need to intensify global commitment and funding for the fight against childhood diarrhoea and argues that scaling up key interventions among the poorest children would save lives. Key preventive interventions include an improved water supply and the promotion of community-wide sanitation and hand washing with soap, as well as vaccination against rotavirus infection and measles, promotion of breastfeeding and vitamin A supplementation.3 Key therapeutic interventions for children with diarrhoea include continued feeding, the use of zinc tablets and fluid replacement therapy with low-osmolarity oral rehydration salts (ORS). With the approach of the deadline for attaining the health-related Millennium Development Goals, the point has been repeatedly made that we know what interventions work.3,5 This message is simple and strategic, but it fails to make clear that although proven preventive and therapeutic interventions do exist, we are not always able to deliver them. If further reductions in the burden of childhood diarrhoea and its associated mortality are to be realized, we need more than a catalogue of proven interventions. We need strategies for delivering them with high acceptability and coverage. Our experience with the treatment of childhood diarrhoea illustrates the difficulty of translating knowledge into effective interventions. The use of ORS, which cost a mere US$ 0.10 per sachet, has been widely promoted since the 1970s and endorsed by national health programmes across the developing world, yet today only 34% of children under 5 years of age with diarrhoea receive ORS.4 Many factors account for this disappointing coverage, including ORS’ unpleasant taste and inconvenient one-litre packaging, which have made traditional formulations unpopular with children and caregivers alike. After years of stagnating ORS coverage, innovative delivery strategies are emerging. Flavoured varieties, smaller sachets for children and new distribution mechanisms are being promoted to at last improve the acceptability and uptake of this life-saving product. Vaccination against rotavirus infection is another promising new component of diarrhoea prevention. Rotavirus is the leading cause of severe diarrhoea in young children and vaccination against rotavirus disease could play a crucial role in the control of childhood diarrhoea throughout the world. WHO now recommends including rotavirus vaccination in all national routine immunization programmes. Furthermore, the GAVI Alliance plans to support the vaccine’s introduction in more than 40 countries by 2015. Despite this high-level support, the product’s presentation could make widespread vaccine delivery difficult. The available rotavirus vaccines require a cold chain capacity much greater than that needed for Expanded Programme on Immunization (EPI) vaccines6 and beyond what the public health infrastructure can sustain in some resource-limited settings. In fact, early experiences in Latin America and computational models for the Niger suggest that introducing the rotavirus vaccine where refrigeration and transport are inadequate could create bottlenecks that would reduce the flow of all vaccines.7 Vaccination against rotavirus infection certainly holds promise for reducing diarrhoea incidence and mortality, but without innovations in vaccine delivery, including the development of more EPI-friendly presentations, vaccination may become so difficult that its potential impact may be undercut. We echo the call of others for greater attention to delivery strategies3,8and take UNICEF’s appeal for stronger global commitment in the fight against childhood diarrhoea as an opportunity to underscore the necessity to bridge the gap between knowing what interventions work and delivering them to all those in need. Progress in the fight against diarrhoeal disease will stagnate unless better intervention delivery strategies are developed. We commit ourselves and call upon our partners to critically assess the delivery strategies currently in place, identify local barriers to intervention uptake and respond with appropriate measures for improving outcomes in children with diarrhoea. Only by delivering what we know to be effective can we further reduce childhood diarrhoea throughout the world.

The prevalence of bullying worldwide is high (UNESCO, 2018). Over the past decades, many anti-bullying interventions have been developed to remediate this problem. However, we lack insight into for whom these interventions work and what individual intervention components drive the total intervention effects. We conducted a large-scale individual participant data (IPD) meta-analysis using data from 39,793 children and adolescents aged five to 20 years (Mage = 12.58, SD = 2.34) who had participated in quasi-experimental or randomized controlled trials of school-based anti-bullying interventions (i.e., 10 studies testing nine interventions). Multilevel logistic regression analyses showed that anti-bullying interventions significantly reduced self-reported victimization (d = − 0.14) and bullying perpetration (d = − 0.07). Anti-bullying interventions more strongly reduced bullying perpetration in younger participants (i.e., under age 12) and victimization for youth who were more heavily victimized before the intervention. We did not find evidence to show that the inclusion of specific intervention components was related to higher overall intervention effects, except for an iatrogenic effect of non-punitive disciplinary methods–which was strongest for girls. Exploratory analyses suggested that school assemblies and playground supervision may have harmful effects for some, increasing bullying perpetration in youth who already bullied frequently at baseline. In conclusion, school-based anti-bullying interventions are generally effective and work especially well for younger children and youth who are most heavily victimized. Further tailoring of interventions may be necessary to more effectively meet the needs and strengths of specific subgroups of children and adolescents.

Competency-based medical education (CBME) aims to bring about the sequential acquisition of competencies required for practice. Although it is being adopted in centers of medical education around the globe, there is little evidence concerning whether, in comparison with traditional methods, CBME produces physicians who are better prepared for the practice environment and contributes to improved patient outcomes. Consequently, the authors, an international group of collaborators, wrote this article to provide guidance regarding the evaluation of CBME programs.CBME is a complex service intervention consisting of multiple activities that contribute to the achievement of a variety of outcomes over time. For this reason, it is difficult to apply traditional methods of program evaluation, which require conditions of control and predictability, to CBME. To address this challenge, the authors describe an approach that makes explicit the multiple potential linkages between program activities and outcomes. Referred to as contribution analysis (CA), this theory-based approach to program evaluation provides a systematic way to make credible causal claims under conditions of complexity. Although CA has yet to be applied to medical education, the authors describe how a six-step model and a postulated theory of change could be used to examine the link between CBME, physicians' preparation for practice, and patient care outcomes.The authors argue that adopting the methods of CA, particularly the rigor in thinking required to link program activities, outcomes, and theory, will serve to strengthen understanding of the impact of CBME over time.

Despite advance care planning (ACP) being associated with positive outcomes for residents in long-term care facilities (LTCFs), the causal pathways between ACP and these outcomes are context-specific and less understood. This lack of clarity can hinder the cultural adaptation and evaluation of ACP interventions. This study aimed to develop a programme theory that outlines the causal pathways through which the ACP is hypothesised to achieve impacts in Chinese LTCFs, with a focus on understanding its implementation, processes and outcomes. Exploratory qualitative design incorporating Theory of Change (ToC) methodology. Two ToC workshops (one face-to-face and one online) were held with 37 participants experienced in caring for residents or older people. The process was informed by a realist review and primary qualitative study. A programme theory was developed through thematic analysis, generating a ToC map depicting implementation, processes and outcomes of ACP in LTCFs. The programme theory was constructed to outline the causal pathways of ACP in LTCFs, populating five 'precondition' domains: (1) buy-in from government and facility leadership, (2) availability of external and internal resource, (3) adequate training and awareness for public and facility, (4) identification of residents who are ready for ACP and (5) culturally sensitive communication. Nine intervention components were identified that target preconditions, such as raising ACP awareness and providing staff training and mentoring. The potential impacts of ACP were identified, for example, fostering public attitudes towards a 'good death' and increasing public awareness and acceptance of palliative care. Our mid-range programme theory can serve as a heuristic tool, adaptable for context-specific ACP interventions in other countries, enhancing the likelihood of achieving intended impacts. In particular, intervention components focused on family involvement can be transferable to East Asian regions, where relational autonomy and family-centred decision-making are emphasised. The programme theory is ready for feasibility testing for residents in Chinese LTCFs. We were guided by patient and public involvement members including two residents and one family member of a resident throughout the study. They supported the overall development of programme theory, including reviewing the theory and interpreting findings.

BackgroundIn this methodological paper we document the interpretation of a mixed methods study and outline an approach to dealing with apparent discrepancies between qualitative and quantitative research data in a pilot study evaluating whether welfare rights advice has an impact on health and social outcomes among a population aged 60 and over.MethodsQuantitative and qualitative data were collected contemporaneously. Quantitative data were collected from 126 men and women aged over 60 within a randomised controlled trial. Participants received a full welfare benefits assessment which successfully identified additional financial and non-financial resources for 60% of them. A range of demographic, health and social outcome measures were assessed at baseline, 6, 12 and 24 month follow up. Qualitative data were collected from a sub-sample of 25 participants purposively selected to take part in individual interviews to examine the perceived impact of welfare rights advice.ResultsSeparate analysis of the quantitative and qualitative data revealed discrepant findings. The quantitative data showed little evidence of significant differences of a size that would be of practical or clinical interest, suggesting that the intervention had no impact on these outcome measures. The qualitative data suggested wide-ranging impacts, indicating that the intervention had a positive effect. Six ways of further exploring these data were considered: (i) treating the methods as fundamentally different; (ii) exploring the methodological rigour of each component; (iii) exploring dataset comparability; (iv) collecting further data and making further comparisons; (v) exploring the process of the intervention; and (vi) exploring whether the outcomes of the two components match.ConclusionThe study demonstrates how using mixed methods can lead to different and sometimes conflicting accounts and, using this six step approach, how such discrepancies can be harnessed to interrogate each dataset more fully. Not only does this enhance the robustness of the study, it may lead to different conclusions from those that would have been drawn through relying on one method alone and demonstrates the value of collecting both types of data within a single study. More widespread use of mixed methods in trials of complex interventions is likely to enhance the overall quality of the evidence base.

What is known and objectiveThe recently conducted Medication Actions to Reduce hospital admissions through a collaboration between Community and Hospital pharmacists (MARCH) transitional care programme, which aimed to test the effectiveness of a transitional care programme on the occurrence of ADEs post‐discharge, did not show a significant effect. To clarify whether this non‐significant effect was due to poor implementation or due to ineffectiveness of the intervention as such, a process evaluation was conducted. The aim of the study was to gain more insight into the implementation fidelity of MARCH.MethodsA mixed methods design and the modified Conceptual Framework for Implementation Fidelity was used. For evaluation, the implementation fidelity and moderating factors of four key MARCH intervention components (teach‐back, the pharmaceutical discharge letter, the post‐discharge home‐visit and the transitional medication review) were assessed. Quantitative data were collected during and after the intervention. Qualitative data were collected using semi‐structured interviews with MARCH healthcare professionals (community pharmacists, clinical pharmacists, pharmacy assistants and pharmaceutical consultants) and analysed using thematic analysis.Results and DiscussionNot all key intervention components were implemented as intended. Teach‐back was not always performed. Moreover, 63% of the pharmaceutical discharge letters, 35% of the post‐discharge home‐visits and 44% of the transitional medication reviews were not conducted within their planned time frames. Training sessions, structured manuals and protocols with detailed descriptions facilitated implementation. Intervention complexity, time constraints and the multidisciplinary coordination were identified as barriers for the implementation.What is new and ConclusionOverall, the implementation fidelity was considered to be moderate. Not all key intervention components were carried out as planned. Therefore, the non‐significant results of the MARCH programme on ADEs may at least partly be explained by poor implementation of the programme. To successfully implement transitional care programmes, healthcare professionals require full integration of these programmes in the standard work‐flow including IT improvements as well as compensation for the time investment.

BackgroundStandards for the reporting of factorial randomised trials remain to be established. We aimed to review the quality of reporting of methodological aspects of published factorial trials of complex interventions in community settings.MethodsWe searched MEDLINE, EMBASE, PsychInfo and the Cochrane Controlled Trials Register to identify factorial randomised trials of complex interventions in community settings from January 2000 to August 2009. We also conducted a citation search of two review papers published in 2003. Data were extracted by two reviewers on 22 items relating to study design, analysis and presentation.ResultsWe identified 5941 unique titles, from which 116 full papers were obtained and 76 were included in the review. The included trials reflected a broad range of target conditions and types of intervention. The median sample size was 400 (interquartile range 191-1001). Most (88%) trials employed a 2 × 2 factorial design. Few trials (21%) explicitly stated the rationale for using a factorial design. Reporting of aspects of design, analysis or presentation specific to factorial trials was variable, but there was no evidence that reporting of these aspects was different for trials published before or after 2003. However, for CONSORT items that apply generally to the reporting of all trials, there was some evidence that later studies were more likely to report employing an intention-to-treat (ITT) approach (78% vs 52%), present appropriate between-group estimates of effect (88% vs 63%), and present standard errors or 95% confidence intervals for such estimates (78% vs 56%). Interactions between interventions and some measure of the precision associated with such effects were reported in only 14 (18%) trials.ConclusionsReports of factorial trials of complex interventions in community settings vary in the amount of information they provide regarding important methodological aspects of design and analysis. This variability supports the extension of CONSORT guidelines to include the specific reporting of factorial trials.