Theory-driven Evaluations Research Articles

Evaluating population health interventions:The contributions of theory-driven evaluations

Population health intervention research has been characterized by the deployment of scientific methods designed to produce knowledge on policy and program interventions directly or indirectly involving the health sector, and potentially affecting population health. The proposed solutions encompass a multitude of interventions of variable types, scale, focus and implementation, rendering them particularly complex and difficult to understand. This complexity raises major conceptual and methodological issues because in reality, we assess not an intervention, but rather a set of interventional elements interacting with those specific to the context in which they appear. It is the interactions that produce effects, to the extent that it makes little sense to speak of an intervention, and more sense to consider an interventional system defined by the interactions. To grasp the numerous elements brought into play, it behooves us to amalgamate evaluation paradigms and approaches. In a precise context, theory-driven evaluations are of pronounced interest. This article presents the main principles of this type of evaluation by focusing on its capacity to shed light on the stakes involved in intervention/context interplay, and by putting forward conclusions transferable to population health research.

Learning from experiences of evaluators implementing theory-driven evaluations in diverse settings: Building on the contributions of John Mayne

The papers in this volume grapple with various issues related to the use of theories of change in program evaluation. This introductory paper reviews some of the critical challenges that arise in developing and learning from theory-driven evaluations. These challenges include the relationship between theories of change and ecologies of evidence, the need for epistemic fluency in learning, and coming to terms with the initial incompleteness of knowledge in program mechanisms. The nine papers that follow, which represent a geographically diverse set of evaluations and evaluators (including Scotland, India, Canada, USA), help develop these and other themes. This volume of papers also serves as a celebration of the work of John Mayne, one of the foremost theory-driven evaluators of the last few decades. John passed away in December of 2020. This volume is intended to honor his legacy while also identifying challenging issues that call for further development.

Making implementation science more real

Implementation science in healthcare aims to understand how to get evidence into practice. Once this is achieved in one setting, it becomes increasingly difficult to replicate elsewhere. The problem is often attributed to differences in context that influence how and whether implementation strategies work. We argue that realist research paradigms provide a useful framework to express the effect of contextual factors within implementation strategy causal processes. Realist studies are theory-driven evaluations that focus on understanding how and why interventions work under different circumstances. They consider the interaction between contextual circumstances, theoretical mechanisms of change and the outcomes they produce, to arrive at explanations of conditional causality (i.e., what tends to work, for whom, under what circumstances). This Commentary provides example applications using preliminary findings from a large realist implementation study of system-wide value-based healthcare initiatives in New South Wales, Australia. If applied judiciously, realist implementation studies may represent a sound approach to help optimise delivery of the right care in the right setting and at the right time.

Evaluating the Test Validity of the Comprehensive Assessment of Psychopathic Personality Symptom Rating Sale (CAPP SRS)

The Comprehensive Assessment of Psychopathic Personality Symptom Rating Scale (CAPP SRS) is a relatively new measure of psychopathic personality disorder (PPD) based on the CAPP concept map of psychopathy. To investigate the CAPP SRS, we identified the most plausible formal test structure for the test using the framework proposed by Slaney and Maraun, identified an appropriate quantitative characterization of that test structure, and then statistically evaluated it based on analysis of CAPP SRS data collected from a multisite sample of 314 adult male correctional offenders and secure hospital patients in Scotland and England. Overall, the CAPP SRS survived falsification when observed test data were compared to expectations based on the unidimensional monotone latent variable or UMLV model of Holland and Rosenbaum. CAPP SRS composite scores calculated consistent with the UMLV model had good measurement precision and good external validity with respect to scores on an established test of PPD. The findings provide provisional support for the test validity of the CAPP SRS and highlight the importance of theory-driven evaluations of test validity.

Using path analysis to test theory of change: a quantitative process evaluation of the MapSan trial

BackgroundAlthough theory-driven evaluations should have empirical components, few evaluations of public health interventions quantitatively test the causal model made explicit in the theory of change (ToC). In the context of a shared sanitation trial (MapSan) in Maputo, Mozambique, we report findings of a quantitative process evaluation assessing intervention implementation, participant response and impacts on hypothesised intermediary outcomes on the pathway to trial health outcomes. We examine the utility of path analysis in testing intervention theory using process indicators from the intervention’s ToC.MethodsProcess data were collected through a cross-sectional survey of intervention and control compounds of the MapSan trial > 24-months post-intervention, sampling adult residents and compound leaders. Indicators of implementation fidelity (dose received, reach) and participant response (participant behaviours, intermediary outcomes) were compared between trial arms. The intervention’s ToC (formalised post-intervention) was converted to an initial structural model with multiple alternative pathways. Path analysis was conducted through linear structural equation modelling (SEM) and generalised SEM (probit model), using a model trimming process and grouped analysis to identify parsimonious models that explained variation in outcomes, incorporating demographics of respondents and compounds.ResultsAmong study compounds, the MapSan intervention was implemented with high fidelity, with a strong participant response in intervention compounds: improvements were made to intermediary outcomes related to sanitation ‘quality’ – latrine cleanliness, maintenance and privacy – but not to handwashing (presence of soap / soap residue). These outcomes varied by intervention type: single-cabin latrines or multiple-cabin blocks (designed for > 20 users). Path analysis suggested that changes in intermediary outcomes were likely driven by direct effects of intervention facilities, with little contribution from hygiene promotion activities nor core elements expected to mediate change: a compound sanitation committee and maintenance fund. A distinct structural model for two compound size subgroups (≤ 20 members vs. > 20 members) explained differences by intervention type, and other contextual factors influenced specific model parameters.ConclusionsWhile process evaluation found that the MapSan intervention achieved sufficient fidelity and participant response, the path analysis approach applied to test the ToC added to understanding of possible ‘mechanisms of change’, and has value in disentangling complex intervention pathways.Trial registrationMapSan trial registration: NCT02362932 Feb-13-2015.

Multi-Level Evolution of Shooter Levels

This paper introduces a search-based generative process for first person shooter levels. Genetic algorithms evolve the level's architecture and the placement of powerups and player spawnpoints, generating levels with one floor or two floors. The evaluation of generated levels combines metrics collected from simulations of artificial agents competing in the level and theory-based heuristics targeting general level design patterns. Both simulation-based and theory-driven evaluations target player balance and exploration, while resulting levels emergently exhibit several popular design patters of shooter levels.

Program Logic Foundations: Putting the Logic Back into Program Logic

Background: Program logic is one of the most used tools by the public policy evaluator. There is, however, little explanation in the evaluation literature about the logical foundations of program logic or discussion of how it may be determined if a program is logical. This paper was born on a long journey that started with program logic and ended with the logic of evaluation. Consistent throughout was the idea that the discipline of program evaluation is a pragmatic one, concerned with applied social science and effective action in complex, adaptive systems. It gradually became the central claim of this paper that evidence-based policy requires sound reasoning more urgently than further development and testing of scientific theory. This was difficult to reconcile with the observation that much evaluation was conducted within a scientific paradigm, concerned with the development and testing of various types of theory.
 Purpose: This paper demonstrates the benefits of considering the core essence of a program to be a proposition about the value of a course of action. This contrasts with a research-based paradigm in which programs are considered to be a type of theory, and in which experimental and theory-driven evaluations are conducted. Experimental approaches focus on internal validity of knowledge claims about programs and on discovering stable cause and effect relationships—or, colloquially, ‘what works?’. Theory-driven approaches tend to focus on external validity and in the case of the realist approach, the search for transfactual causal mechanisms—extending the ‘what works’ mantra to include ‘for whom and in what circumstances’. On both approaches, evaluation aspires to be a scientific pursuit for obtaining knowledge of general laws of phenomena, or in the case of realists, replicable context-mechanism-outcome configurations. This paper presents and seeks to justify an approach rooted in logic, and that supports anyone to engage in a reasonable and democratic deliberation about the value of a course of action.
 It is consistent with systems thinking, complexity and the associated limits to certainty for determining the value of a proposed, or actual, course of action in the social world. It suggests that evaluation should learn from the past and have an eye toward the future, but that it would be most beneficial if concerned with evaluating in the present, in addressing the question ‘is this a good idea here and now?
 Setting: Not applicable.
 Intervention: Not applicable
 Research design: Not applicable.
 Findings: In seeking foundations of program logic, this paper exposes roots that extend far deeper than the post-enlightenment, positivist and post-positivist social science search for stable cause and effect relationships. These roots lie in the 4th century BCE with Aristotle’s ‘enthymeme’. The exploration leads to conclusions about the need for a greater focus on logic and reasoning in the design and evaluation of programs and interventions for the public good. Science and research are shown to play a crucial role in providing reasons or warrants to support a claim about the value of a course of action; however, one subordinate to the alpha-discipline of logical evaluation and decision making that must consider what is feasible given the context, capability and capacity available, not to mention values and ethics. Program Design Logic (PDL) is presented as an accessible and incremental innovation that may be used to determine if a program makes sense ‘on paper’ in the design stage as well as ‘in reality’ during delivery. It is based on a configurationalist theory of causality and the concepts of ‘necessary’ and ‘sufficient’ conditions. It is intended to guide deliberation and decision making across the life cycle of any intervention intended for the public good.
 Keywords: Program logic; program theory; theory of change; program design logic; logic of evaluation; theory of causality; INUS condition

Dealing with context in logic model development: Reflections from a realist evaluation of a community health worker programme in Nigeria

Logic models (LMs) have been used in programme evaluation for over four decades. Current debate questions the ability of logic modelling techniques to incorporate contextual factors into logic models. We share experience of developing a logic model within an ongoing realist evaluation which assesses the extent to which, and under what circumstances a community health workers (CHW) programme promotes access to maternity services in Nigeria. The article contributes to logic modelling debate by: i) reflecting on how other scholars captured context during LM development in theory-driven evaluations; and ii) explaining how we explored context during logic model development for realist evaluation of the CHW programme in Nigeria. Data collection methods that informed our logic model development included documents review, email discussions and teleconferences with programme stakeholders and a technical workshop with researchers to clarify programme goals and untangle relationships among programme elements. One of the most important findings is that, rather than being an end in itself, logic model development is an essential step for identifying initial hypotheses for tentative relevant contexts, mechanisms and outcomes (CMOs) and CMO configurations of how programmes produce change. The logic model also informed development of a methodology handbook that is guiding verification and consolidation of underlying programme theories.

Evaluating Comprehensive State Tobacco Prevention and Control Programs Using an Outcome Indicator Framework.

The outcome indicator framework helps tobacco prevention and control programs (TCPs) plan and implement theory-driven evaluations of their efforts to reduce and prevent tobacco use. Tobacco use is the single-most preventable cause of morbidity and mortality in the United States. The implementation of public health best practices by comprehensive state TCPs has been shown to prevent the initiation of tobacco use, reduce tobacco use prevalence, and decrease tobacco-related health care expenditures. Achieving and sustaining program goals require TCPs to evaluate the effectiveness and impact of their programs. To guide evaluation efforts by TCPs, the Centers for Disease Control and Prevention's Office on Smoking and Health developed an outcome indicator framework that includes a high-level logic model and evidence-based outcome indicators for each tobacco prevention and control goal area. In this article, we describe how TCPs and other community organizations can use the outcome indicator framework in their evaluation efforts. We also discuss how the framework is used at the national level to unify tobacco prevention and control efforts across varying state contexts, identify promising practices, and expand the public health evidence base.

Combining Theory-Driven Evaluation and Causal Loop Diagramming for Opening the 'Black Box' of an Intervention in the Health Sector: A Case of Performance-Based Financing in Western Uganda.

Increased attention on “complexity” in health systems evaluation has resulted in many different methodological responses. Theory-driven evaluations and systems thinking are two such responses that aim for better understanding of the mechanisms underlying given outcomes. Here, we studied the implementation of a performance-based financing intervention by the Belgian Technical Cooperation in Western Uganda to illustrate a methodological strategy of combining these two approaches. We utilized a systems dynamics tool called causal loop diagramming (CLD) to generate hypotheses feeding into a theory-driven evaluation. Semi-structured interviews were conducted with 30 health workers from two districts (Kasese and Kyenjojo) and with 16 key informants. After CLD, we identified three relevant hypotheses: “success to the successful”, “growth and underinvestment”, and “supervision conundrum”. The first hypothesis leads to increasing improvements in performance, as better performance leads to more incentives, which in turn leads to better performance. The latter two hypotheses point to potential bottlenecks. Thus, the proposed methodological strategy was a useful tool for identifying hypotheses that can inform a theory-driven evaluation. The hypotheses are represented in a comprehensible way while highlighting the underlying assumptions, and are more easily falsifiable than hypotheses identified without using CLD.

Toward a Better Understanding of Adherence to Micronutrient Powders: Generating Theories to Guide Program Design and Evaluation Based on a Review of Published Results

BackgroundThe Global Alliance for Improved Nutrition is conducting theory-driven process evaluations of micronutrient powder (MNP) programs. ObjectiveThe aim was to generate preliminary theories about factors affecting adherence to recommendations with regard to point-of-use fortification of foods with MNPs. MethodsA literature search was conducted to identify documents with content related to adherence to MNPs as an intervention provided at home to children 6–59 mo of age. Thirty-five studies and 6 program descriptions were identified. We used thematic analyses to generate a comprehensive list of factors that could influence adherence, followed by content analysis to quantify the results. We developed a Program Impact Pathway to concretize the points at which the factors identified affect the process of adherence. ResultsIn the set of documents reviewed (n = 41), the most influential factors, measured by number of documents reporting the factor having effect, were 1) caregivers’ perception of positive changes as a result of MNP use (n = 14), 2) caregivers’ perceived child acceptance of food with MNPs (n = 12), and 3) caregivers’ forgetfulness (n = 11). Behavior change communication channels (n = 13) and messages (n = 12) were the most frequently reported program design features influencing caregiver knowledge and subsequent adherence. Administration regimen (n = 10), which may be related to caregivers’ capacity to remember to give MNPs, was also a frequently cited program design feature affecting adherence. ConclusionsThe preponderance of knowledge and perception factors may reflect an underlying theoretical bias among researchers as to what they measure. To achieve programs that support greater adherence, we need to adopt a cultural-ecological perspective to inform program design in order to address a broader set of determinants. Studies that assess progress across the impact pathway, particularly from adherence to biological outcomes, would also provide guidance for evaluation studies, particularly when time or other constraints limit the potential to measure biological outcomes.

Ecosystemic Needs Assessment for Children with Developmental Coordination Disorder in Elementary School: Multiple Case Studies

ABSTRACTThis study explored the needs of children with developmental coordination disorder (DCD) from an ecosystemic viewpoint as part of a theory-driven program evaluation process. A multiple case study needs assessment was conducted. Participants included ten children with DCD, their parents (n = 12), teachers (n = 9), and service providers (n = 6). Data collection involved semi-structured interviews, validated questionnaires, and a review of the children's records. The results support the relevance of using an ecosystemic model to assess the needs of children with DCD in their life and social contexts. More specifically, the results highlight the need to provide additional services at school, such as occupational therapy and special education, as well as information and training regarding DCD for parents and teachers. The results also point to the relevant variables to consider in an intervention program based on theory-driven evaluations. This study shows how employing an ecosystemic frame of reference provides a better understanding of the needs of children with DCD. Future research should document the ecosystemic profiles and evolution of the needs of children with DCD with a larger sample from diverse socioeconomic backgrounds using a longitudinal study design.

Bringing Rigor to Evaluations of Large-Scale Programs to Improve Infant and Young Child Feeding and Nutrition: The Evaluation Designs for the Alive & Thrive Initiative

The evidence base on the impact of large-scale infant and young child feeding (IYCF) and nutrition programs is limited, partly due to the challenges of rigorously evaluating complex programs including multiple interventions. To describe the process used to design Alive & Thrive's impact evaluations in the three target countries and discuss the feasibility of developing contextually relevant designs adapted to the country-specific programmatic context. The evaluation designs for Alive & Thrive needed to address several challenges. These included the selection of intervention components to evaluate rigorously; the identification of appropriate comparison groups in the context of rapidly scaling-up programs; the choice of impact indicators; addressing measurement challenges related to evaluating the impact of interventions targeted during the first 2 years of life on stunting; and developing methods and tools to assess implementation, utilization, and program impact pathways within evolving program portfolios. In Bangladesh and Vietnam, cluster-randomized probability designs are used for the impact evaluations; in Ethiopia, the impact evaluation uses an adequacy design. In all three countries, repeated cross-sectional surveys, 4 years apart, are used to measure impact, and appropriate age groups are sampled separately to capture change in the main impact indicators. In addition, theory-driven process evaluations are used to study factors that facilitate or prevent achievement of impact and scale. We conclude that robust impact and process evaluations of complex, large-scale nutrition programs are feasible, but that early implementer-evaluator engagement and shared vision and motivation to establishing meaningful evaluations are essential.

When a whole practice model is the intervention: developing fidelity evaluation components using program theory-driven science for an integrative medicine primary care clinic.

Integrative medicine (IM) is a clinical paradigm of whole person healthcare that combines appropriate conventional and complementary medicine (CM) treatments. Studies of integrative healthcare systems and theory-driven evaluations of IM practice models need to be undertaken. Two health services research methods can strengthen the validity of IM healthcare studies, practice theory, and fidelity evaluation. The University of Arizona Integrative Health Center (UAIHC) is a membership-supported integrative primary care clinic in Phoenix, AZ. A comparative effectiveness evaluation is being conducted to assess its clinical and cost outcomes. A process evaluation of the clinic's practice theory components assesses model fidelity for four purposes: (1) as a measure of intervention integrity to determine whether the practice model was delivered as intended; (2) to describe an integrative primary care clinic model as it is being developed and refined; (3) as potential covariates in the outcomes analyses, to assist in interpretation of findings, and for external validity and replication; and (4) to provide feedback for needed corrections and improvements of clinic operations over time. This paper provides a rationale for the use of practice theory and fidelity evaluation in studies of integrative practices and describes the approach and protocol used in fidelity evaluation of the UAIHC.

Towards an evidence base of theory-driven evaluations: Some questions for proponents of theory-driven evaluation

This article discusses the further development of theory-driven evaluation approaches that are informed by contribution analysis. Using an illustrative example of an ongoing dance/physical activity programme for health promotion, a number of challenges are identified when applying a theory-driven evaluation approach. These challenges are reformulated as questions that need to be answered to make further progress with theory-driven evaluation including contribution analysis. Questions include: What is a ‘good enough’ programme theory? How does one arrive at expectations of programme impacts? How does the programme theory incorporate heterogeneous mechanisms that programme recipients might need? What does causality mean for complex interventions? What are structures that can facilitate learning from evaluations? How does the application of theory-driven evaluation approaches help generate an ‘ecology of evidence’? Discussion of these questions leads to a ‘roadmap’ for how contribution analysis might be further tested and refined.

The role of theory in research to develop and evaluate the implementation of patient safety practices

Theories provide a way of understanding and predicting the effects of patient safety practices (PSPs), interventions intended to prevent or mitigate harm caused by healthcare or risks of such harm....

How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa

BackgroundThis paper presents the development of a study design built on the principles of theory-driven evaluation. The theory-driven evaluation approach was used to evaluate an adolescent sexual and reproductive health intervention in Mali, Burkina Faso and Cameroon to improve continuity of care through the creation of networks of social and health care providers.Methods/designBased on our experience and the existing literature, we developed a six-step framework for the design of theory-driven evaluations, which we applied in the ex-post evaluation of the networking component of the intervention. The protocol was drafted with the input of the intervention designer. The programme theory, the central element of theory-driven evaluation, was constructed on the basis of semi-structured interviews with designers, implementers and beneficiaries and an analysis of the intervention's logical framework.DiscussionThe six-step framework proved useful as it allowed for a systematic development of the protocol. We describe the challenges at each step. We found that there is little practical guidance in the existing literature, and also a mix up of terminology of theory-driven evaluation approaches. There is a need for empirical methodological development in order to refine the tools to be used in theory driven evaluation. We conclude that ex-post evaluations of programmes can be based on such an approach if the required information on context and mechanisms is collected during the programme.

A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009

Although the general conceptual basis appeared far earlier, theory-driven evaluation came to prominence only a few decades ago with the appearance of Chen’s 1990 book Theory-Driven Evaluations. Since that time, the approach has attracted many supporters as well as detractors. In this paper, 45 cases of theory-driven evaluations, published over a twenty-year period, are systematically examined to ascertain how closely theory-driven evaluation practices comport with the key tenants of theory-driven evaluation as described and prescribed by prominent theoretical writers. Evidence derived from this review to repudiate or substantiate many of the claims put forth both by critics of and advocates for theory-driven forms of evaluation are presented and an agenda for future research on the approach is recommended.

Using a systems orientation and foundational theory to enhance theory-driven human service program evaluations

This paper offers a framework for using a systems orientation and "foundational theory" to enhance theory-driven evaluations and logic models. The framework guides the process of identifying and explaining operative relationships and perspectives within human service program systems. Self-Determination Theory exemplifies how a foundational theory can be used to support the framework in a wide range of program evaluations. Two examples illustrate how applications of the framework have improved the evaluators' abilities to observe and explain program effect. In both exemplars improvements involved addressing and organizing into a single logic model heretofore seemingly disparate evaluation issues regarding valuing (by whose values); the role of organizational and program context; and evaluation anxiety and utilization.

Trends and Developments in Program Evaluation in General and Criminal Justice Programs in Particular

The paper first describes general trends in evaluation in (mainly) western societies. Why is evaluation growing, what are the characteristics of this ‘growth industry’ and what developments are occurring outside western societies? Trends in the evaluation of criminal justice programs in the USA, the UK and the Netherlands are then discussed. Two important developments are therafter highlighted: experimental evaluations and theory-driven evaluations. Both approaches are discussed, and some pros and cons are listed. Finally, the paper outlines some challenges for future work in the evaluation of criminal justice programs, stressing the importance of combining good designs with both program and social science theories.