Theory-driven Evaluation Research Articles

Exploring the value of integrated findings in a multiphase mixed methods evaluation of the continuous assessment program in the Republic of Trinidad and Tobago

AbstractContinuous assessment programs are externally directed, curriculum-based assessment schemes used for both summative and formative purposes by teachers within classrooms. Since the 1970s, continuous assessment schemes have been implemented within several developing countries, often in an attempt to improve teacher assessment practice and minimize the deleterious effect of high stakes public examinations. To date, few evaluations of continuous assessment in these contexts have employed a formal model or an explicit mixed methods research design. This paper reported on a theory-driven evaluation, which frames a mixed methods investigation of the Trinidad and Tobago Continuous Assessment Program (CAP). The qual→ QUAN→ qual multiphase mixed methods design is briefly described, but the focus is upon the construction and use of integrated findings. Integrated findings do appear to have greater policy relevance and provide deeper insight into the complex interplay among contexts, beliefs, and assessment p...

Engaging primary care practitioners in quality improvement: making explicit the program theory of an interprofessional education intervention

BackgroundThe scientific literature continues to advocate interprofessional collaboration (IPC) as a key component of primary care. It is recommended that primary care groups be created and configured to meet the healthcare needs of the patient population, as defined by patient demographics and other data analyses related to the health of the population being served. It is further recommended that the improvement of primary care services be supported by the delivery of feedback and performance measurements. This paper describes the theory underlying an interprofessional educational intervention developed in Quebec’s Montérégie region (Canada) for the purpose of improving chronic disease management in primary care. The objectives of this study were to explain explicitly the theory underlying this intervention, to describe its components in detail and to assess the intervention’s feasibility and acceptability.MethodA program impact theory-driven evaluation approach was used. Multiple sources of information were examined to make explicit the theory underlying the education intervention: 1) a literature review and a review of documents describing the program’s development; 2) regular attendance at the project’s committee meetings; 3) direct observation of the workshops; 4) interviews of workshop participants; and 5) focus groups with workshop facilitators. Qualitative data collected were analysed using thematic analysis.ResultsThe theoretical basis of the interprofessional education intervention was found to be work motivation theory and reflective learning. Five themes describing the workshop objectives emerged from the qualitative analysis of the interviews conducted with the workshop participants. These five themes were the importance of: 1) adopting a regional perspective, 2) reflecting, 3) recognizing gaps between practice and guidelines, 4) collaborating, and 5) identifying possible practice improvements. The team experienced few challenges implementing the intervention. However, the workshop’s acceptability was found to be very good.ConclusionOur observation of the workshop sessions and the interviews conducted with the participants confirmed that the objectives of the education intervention indeed targeted the improvement of interprofessional collaboration and quality of care. However, it is clear that a three-hour workshop alone cannot lead to major changes in practice. Long-term interventions are needed to support this complex change process.

The Reciprocal Relationship between Implementation Theory and Program Theory in Assisting Program Design and Decision-Making

Abstract: The focus of this article is how Theory Driven Evaluation (TDE) and two of its central tenets—program theory and implementation theory—can be simultaneously used to inform and assist programmatic decision-making. The article argues there is a paucity of evaluation literature demonstrating how program theory can be beneficial to the design and interpretation of implementation theory. A case example is used to illustrate the importance of program theory in developing and interpreting implementation theory.

Conceptualizing Programme Evaluation

The main thrust of this paper deals with the conceptualization of theory-driven evaluation pertaining to a tutor training programme. Conceptualization of evaluation, in this case, is an integration between a conceptualization model as well as a theoretical framework in the form of activity theory. Existing examples of frameworks of programme evaluation from the literature have been conjugated to yield a conceptualization model for the evaluation of the tutor training programme at the Cape Peninsula University of Technology in South Africa. In order to take the study to a higher level of deep analysis, this paper argues for the application of activity theory in the research design and subsequent data collection and analysis. Since evaluation can be a daunting and complex task, the adoption of theory was intended to provide a fulcrum for the study and therefore, this paper argues, becomes an integral part of evaluation. In addition, the application of activity theory in programme evaluation has not been well explored elsewhere and therefore, this paper transcends this limitation by making a case for the use of activity theory in programme evaluation, in particular, tutor training programmes.

Evaluating human resource interventions

Orientation: Programme evaluation is a transdiscipline, which examines whether a programme has merit or not. A programme is a coherent set of activities aimed at bringing about a change in people or their circumstances.Research purpose: The purpose of this special edition is to introduce readers to the evaluation of human resource (HR) programmes.Motivation for the study: There are few comprehensive evaluations of HR programmes despite many publications on functional efficiency measures of HR (i.e. measures of cost, time, quantity, error and quality).Research design, approach and method: This article provides a value chain for HR activities and introduces the reader to programme theory-driven evaluation.Main findings: In summarising all of the contributions in this edition, one of the main findings was the lack of programme evaluation experience within HR functions and the difficulty this posed for the evaluators.Practical/managerial implications: This introductory article presents answers to two simple questions: What does HR do? and, What is programme evaluation? These answers will enable practitioners to understand what programme evaluators mean when we say that programme evaluation seeks to determine the merit of a programme.Contribution/value-add: The main contribution of this introductory article is to set the scene for the HR evaluations that follow. It alerts the reader to the rich theory contribution in HR literature and how to apply this in a theory-driven evaluation.

A theory-driven evaluation of a wellness initiative

Orientation: By reporting on an evaluation of a wellness initiative, this article brings together an element of organisational development (employee wellness) with an approach to programme evaluation (programme theory-driven evaluation). Research purpose: Two questions were addressed: ‘What is the causal logic of the wellness initiative?’ and ‘Is this a plausible programme theory according to social science research and literature?’Motivation for the study: A study that could demonstrate the usefulness of the theory-driven evaluation approach, especially in the local human resource (HR) domain, was considered to be valuable. In addition, this evaluation provided a careful consideration of how plausible it is for such interventions to achieve what they set out to do.Research design, approach and method: The evaluation relied mainly on qualitative methods (the examination of secondary data and interviewing) to extract programme theory, and on literature to assess plausibility.Main findings: The study had two main outcomes: the finalisation of a model of how the programme is supposed to work according to programme staff, and the conclusion that the model is plausible, provided it is implemented at full strength.Practical/managerial implications: Programme staff are advised to pay particular attention to implementation fidelity, especially to employee participation and involvement in the programme’s activities. A number of strategies are recommended to strengthen the effect of the model.Contribution/value-add: This evaluation showed the importance of conducting a theory- driven evaluation, not only in order to understand the programme and its context, but also to provide a basis for an implementation and outcome evaluation.

Towards an evidence base of theory-driven evaluations: Some questions for proponents of theory-driven evaluation

This article discusses the further development of theory-driven evaluation approaches that are informed by contribution analysis. Using an illustrative example of an ongoing dance/physical activity programme for health promotion, a number of challenges are identified when applying a theory-driven evaluation approach. These challenges are reformulated as questions that need to be answered to make further progress with theory-driven evaluation including contribution analysis. Questions include: What is a ‘good enough’ programme theory? How does one arrive at expectations of programme impacts? How does the programme theory incorporate heterogeneous mechanisms that programme recipients might need? What does causality mean for complex interventions? What are structures that can facilitate learning from evaluations? How does the application of theory-driven evaluation approaches help generate an ‘ecology of evidence’? Discussion of these questions leads to a ‘roadmap’ for how contribution analysis might be further tested and refined.

The realist approach to evaluation research: an introduction

Background: The realist approach is a form of theory-driven evaluation developed to strengthen the explanatory power of evaluation studies and contribute to evidence-based policy and practice. It is a generic approach that can be applied to many fields of research, including health and social care. Content: This article introduces the reader to the realist approach and outlines its application in empirical evaluation studies and literature syntheses. It describes how the realist researcher evaluates complex social interventions not by asking whether a particular programme or intervention ‘works’, but, recognising that all interventions only ever work for certain people in certain circumstances, by asking instead ‘what works for whom in what circumstances’? Realist researchers then try to articulate the theories (termed ‘mechanisms’) that underpin the programme and explore how contexts influence the mechanisms to produce different programme outcomes. Conclusions: Realist evaluation and realist synthesis are an advanced approach to evaluation research, but they offer an innovative and insightful means by which to further our understanding of complex social interventions.

Formal Theory Versus Stakeholder Theory

Health promotion and social betterment program interventions are based on either formal theory from academia or stakeholder theory from stakeholders' observations and experiences in working with clients. Over time, formal theory-based interventions have acquired high prestige, while stakeholder theory-based interventions have been held in low esteem. Here we examine the assumption that formal theory-based interventions are superior to stakeholder-based interventions in addressing community problems. The article elaborates on these ideas via a case study of a community-based, environmental tobacco smoke prevention program evaluation. The authors conclude that although both types of interventions had their strengths and limitations in the real world, the stakeholder theory-based intervention was more viable and effective than the formal theory-based intervention in this case due to implementation reasons. Findings are useful in understanding these two intervention types, in developing better strategies to address community problems, and in advancing program theory and theory-driven evaluation.

Assistive technologies at home and in the workplace—a field of research for exercise science and human movement science

Since its existence, the European Review of Aging and Physical Activity (EURAPA) used different forms and contents of editorial articles. Beside referencing and summarizing the current issue, introductory overviews on research subjects are presented, thus drawing attention to fields considered in EURAPA, to stimulate potential research and submission of manuscripts to the journal. Such topics were, e.g. peak exercise [16], socioeconomic perspectives [14], or theory-driven evaluation [4]. The present editorial addresses technical assistance for elderly persons. During the last years, the term “ambient assisted living” (AAL) has been established for this field of research, mostly by national and European Union (EU) funding agencies, including science and technology [1, 19]. This field is of a highly interdisciplinary nature, oriented towards practical solutions, and it includes behavioural and technical aspects. This corresponds to the nature of exercise science and human movement science, which has been described to be “technological” from a theory of science view [3]. In addition, physical activity is an integral part of daily living. Therefore, AAL seems to be an interesting and stimulating field. In the following sections, firstly, the AAL concept is introduced. Secondly, a sketch on concepts to improve and prolong working life is presented, including examples for research questions from a sport and exercise science point of view. Some concluding remarks contain practical information on calls and research frameworks.

Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research

This overview aims to stimulate conceptual and practical discussions to help unlock the full potential of realist evaluation in health systems research. Based on a structured literature search, this review maps how the concepts of realist evaluation are applied in health systems research and which methodological problems are encountered. We found a great diversity in the depth of application of the philosophical concepts, use of terminology and scope of application in the research process. Terms of theory-driven evaluation, theories of change and realist evaluation are often used interchangeably. Diverging views exist regarding the nature of ‘mechanism’ and the difference between mechanism and essential context condition. A lack of methodological guidance was highlighted by many authors. Realist evaluation is slowly gaining traction in health systems research, but more clarity is needed concerning the definitions of mechanisms and context and how the configuration of context, mechanism and intervention can be described and assessed.

Evaluating Public and Community Health Programs

This commentary provides a comprehensive review of key issues related to evaluating public and community health programs. This book is a valuable resource introducing a participatory model for evaluation and describes the approaches and methods for evaluating community health programs and policy interventions. Topics in the chapters are logically sequenced with sufficient details to orient public health and community health students as well as practitioners to program evaluation. The book’s emphasis on participatory approaches throughout the program evaluation process is congruent with the current trend of encouraging community-based participatory research in public health and health promotion. Two primary evaluation frameworks that are most common in public and community health are integrated and discussed: the Donaldson three-step program theory-driven evaluation approach and the Centers for Disease Control and Prevention’s six-step Framework for Program Evaluation in Public Health. The book consists of a detailed survey of methods in community assessment, planning, program design, quantitative and qualitative data collection, data analysis, and reporting of findings to provide readers with a systematic, step-by-step process to program evaluation. It is a timely and well-written resource for faculty and students interested in participatory research to develop community-validated evaluation research.

Theory-driven evaluation of exercise programs: often recommended but still only a few hits in literature databases

Among the papers submitted to and published by the European Review of Aging and Physical Activity, there are usually some dealing with exercise programs. In this editorial, some thoughts on the evaluation of such programs are shared. The editorial closes with some news regarding the journal. Research-based activities in order to develop, to design or to change some real-world object or aspect (e.g., by an exercise program), are considered technological science, as expanded on in a recent editorial of this journal [1]. The outcome of this kind of research should and usually is in fact assessed and rated (evaluation). Researchers, funders, representatives of the target group, and other individuals being affected by such a program (stakeholders) increasingly call for evaluation. Often, the term evaluation is used as a synonym for assessment of effectivity. Compared with that, theory-driven evaluation includes also planning and implementation of a program. Theory-driven evaluation is a framework (in fact, there are several frameworks) with rising acceptance during the last 20 years—and seems to be helpful for health-oriented exercise programs. In the following sections, the reader will find

How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa

BackgroundThis paper presents the development of a study design built on the principles of theory-driven evaluation. The theory-driven evaluation approach was used to evaluate an adolescent sexual and reproductive health intervention in Mali, Burkina Faso and Cameroon to improve continuity of care through the creation of networks of social and health care providers.Methods/designBased on our experience and the existing literature, we developed a six-step framework for the design of theory-driven evaluations, which we applied in the ex-post evaluation of the networking component of the intervention. The protocol was drafted with the input of the intervention designer. The programme theory, the central element of theory-driven evaluation, was constructed on the basis of semi-structured interviews with designers, implementers and beneficiaries and an analysis of the intervention's logical framework.DiscussionThe six-step framework proved useful as it allowed for a systematic development of the protocol. We describe the challenges at each step. We found that there is little practical guidance in the existing literature, and also a mix up of terminology of theory-driven evaluation approaches. There is a need for empirical methodological development in order to refine the tools to be used in theory driven evaluation. We conclude that ex-post evaluations of programmes can be based on such an approach if the required information on context and mechanisms is collected during the programme.

A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009

Although the general conceptual basis appeared far earlier, theory-driven evaluation came to prominence only a few decades ago with the appearance of Chen’s 1990 book Theory-Driven Evaluations. Since that time, the approach has attracted many supporters as well as detractors. In this paper, 45 cases of theory-driven evaluations, published over a twenty-year period, are systematically examined to ascertain how closely theory-driven evaluation practices comport with the key tenants of theory-driven evaluation as described and prescribed by prominent theoretical writers. Evidence derived from this review to repudiate or substantiate many of the claims put forth both by critics of and advocates for theory-driven forms of evaluation are presented and an agenda for future research on the approach is recommended.

Institucionalização da avaliação na administração pública

Este artigo aborda o tema da avaliação sob as perspectivas do potencial em promover transformações nas organizações, das condições à sua institucionalização e das estratégias a serem utilizadas com este fim. Parte das premissas da crença em 'mudar situações', da contribuição à 'compreensão da realidade', e dos 'sentidos dos atores' sobre a tomada de decisão. Embasa a abordagem na emergência de um novo profissionalismo na gestão dos serviços e organizações; na complexidade crescente dos problemas e no papel das 'redes' e sua relação com a governança, ressaltando a legitimidade que a avaliação necessita para influenciar decisões. Destaca três tipos de avaliação no plano da prática e do conhecimento: orientada para as metas, baseada na teoria e a que privilegia a perspectiva participativa e, reflete sobre a incapacidade de resposta aos problemas ao se utilizar apenas um tipo de avaliação. Apresenta como desafios à sua institucionalização: capacidades para realizar a avaliação, estratégia sólida de difusão dos resultados e capacidades à sua utilização. Finaliza constatando que a institucionalização da avaliação supõe mudanças importantes no uso do poder para tomar decisões nas administrações públicas e recomenda o questionamento permanente das possibilidades que os trabalhos dos avaliadores têm de serem utilizados corretamente e sobre os seus benefícios para a sociedade e as instituições.

Maslow's Needs Hierarchy as a Framework for Evaluating Hospitality Houses' Resources and Services

As hospitality houses welcome greater numbers of families and families requiring longer stays, they do so in the absence of a widely accepted theory to guide their understanding of guests' needs and evaluations of how well they meet those needs. We propose A. Maslow's (1970) Hierarchy of Needs as a conceptual framework for understanding what makes a hospitality house a home for families of pediatric patients and for guiding the activities of hospitality houses' boards of directors, staff, volunteers, and donors. This article presents findings from a theory-driven evaluation of one hospitality house's ability to meet guests' needs, describes the house's best practice standards for addressing guests' needs, and suggests areas for future research.

Effectiveness of Interventions Co-Financed by the Eu Structural Funds in Wielkopolska in the Years 2004-2006

Effectiveness of Interventions Co-Financed by the Eu Structural Funds in Wielkopolska in the Years 2004-2006 The aim of this article is to analyse the effectiveness of interventions from the EU Structural Funds in Wielkopolska carried out for investments realised in the years 2004-2006, i.e. in the first period of implementing Community regional policy in Poland. The research procedure is composed of two fundamental stages. In the first, projects co-financed from the EU Structural Funds within the framework of the Regional Component of the Integrated Regional Operational Programme were systematised by the criterion of intervention direction within each of the three dimensions of cohesion: economic, social and territorial. In the second stage, selected diagnostic measures for each dimension of cohesion were subjected to effectiveness analysis. The procedure rested on a questions/methods-oriented model applied in line with the theory-driven evaluation approach, a quasiexperimental design and the difference-in-differences technique, as well as methods of econometric analysis. The results made it possible to assess public intervention in Poland in terms of its effectiveness, and to identify challenges that have to be faced very soon.

Evaluation Training to Build Capability in the Community and Public Health Workforce

Increasingly, staff members in community and public health programs and projects are required to undertake evaluation activities. There is, however, limited capacity for, and understanding of, evaluation within this workforce. Building the capability of individual workers and thereby contributing to the overall capacity among the community and public health workforce has been advocated as one solution. In this article, the Easy Evaluation initiative, which seeks to increase the evaluation capability of individual workers and the capacity of the community and public health workforce in New Zealand, is discussed. The theoretical foundations of the evaluation approach (theory-driven evaluation) taught and of the teaching philosophy (adult learning and experiential learning) used are described, along with course content, teaching activities, and participant activities. Although our observations and experiences are that the workshop model is successful in increasing knowledge and skills of participants (building capability), additional support is required for organizations to build evaluation capacity.

Constructing and Verifying Program Theory Using Source Documentation

Abstract: Making the program theory explicit is an essential first step in Theory Driven Evaluation (TDE). Once explicit, the program logic can be established making necessary links between the program theory, activities, and outcomes. Despite its importance evaluators often encounter situations where the program theory is not explicitly stated. Under such circumstances evaluators require alternatives to generate a program theory with limited time and resources. Using source documentation (e.g., lesson plans, mission statements) to develop program theory is discussed in the evaluation literature as a viable alternative when time and resources do not permit a priori program theory development. Unfortunately, the evaluation literature is devoid of methodology illustrating how to translate source documentation into an explicitly stated program theory. The article describes the steps in using source documentation to develop and verify a program theory and illustrates the application of these steps. It concludes with a discussion about the feasibility and limitations of this methodology.