Program Logic Model Research Articles

A Study of Outcomes Determination Strategy in Non-profit Health and Welfare Programs Evaluation

This article considers the findings of a survey of evaluators practising in Australian health and welfare organisations to learn about strategy in outcomes evaluation design. Outcomes evaluation is identified as an opportunity for health and welfare programs to learn if and how different practice interventions are contributing to the achievement of desired outcomes, for the purpose of building an evidence base for advancing the wellbeing of people. Results indicate that current strategy to determine client outcomes in these sectors is typified by reliance on program logic models and funding specifications, with limited translation of best practice evidence or client wellbeing preferences. We suggest that clarity of purpose is an important consideration for non-profit programs embarking on evaluation design, bringing attention to the need for skill development and better resourcing of non-profit organisations to support the production of evaluations capable of contributing to best practice evidence.

공공인재학부 운영성과에 관한 연구 : 프로그램 논리모형을 활용한 교육사업 성과평가 항목・지표 개발 및 적용을 중심으로

공공인재학부 운영성과에 관한 연구 : 프로그램 논리모형을 활용한 교육사업 성과평가 항목・지표 개발 및 적용을 중심으로

What is the difference between comprehensive and selective primary health care? Evidence from a five-year longitudinal realist case study in South Australia

BackgroundSince the WHO’s Alma Ata Declaration on Primary Health Care (PHC) there has been debate about the advisability of adopting comprehensive or selective PHC. Proponents of the latter argue that...

Abstract WP382: Primary Stroke Centers - What Should We Measure?

Background and Issues: Primary stroke centers are struggling to find their voice in the spectrum of stroke care. It is not feasible for all of us grow into Comprehensive Stroke Centers or even offer endovascular intervention. Victims of our own success, 7 of the 8 stroke Core Measures are discontinued by CMS. Meanwhile we must find other measures to continue to improve the effectiveness of stroke care and our value to our local population, EMS providers, internal stakeholders and the Joint Commission. Purpose: The purpose of this project at our 433-bed community hospital was to shift the focus from micro-management of outdated metrics to a macro-view of our program and demonstrate an effective method for internal evaluation. Methods: We followed the CDC guidelines for public health program evaluation. We reviewed the evidence for all aspects of our program, and developed a program logic model (Figure 1) and theory-based evaluation questions. We used a mixed-methods approach to analyze data for each question. We engaged our leaders in focus groups to review the findings. Results: Table 1 shows examples of our findings. Eleven program leaders, including the hospital medical director, the ED medical director, and nursing directors, identified four priorities: expanding our community outreach, improving ED stroke processes, increasing the use of stroke order sets, and improving post-discharge follow-up recommendations. Conclusions: Structured program evaluation is a means for primary stroke centers to continue to improve outcomes for stroke patients and justify their valuable role in the continuum of stroke care.

Users\u2019 involvement in mental health services: programme logic model of an innovative initiative in integrated care

BackgroundCollaboration and partnership are key issues for modern health systems seeking to implement quality integrated care that meets the needs of the population. The Carrefour Communautaire-Institutionnel-Usagers (Connecting Community organisations-Institutions-Users, CCIU), involving community- and institution-based mental health workers, carers and users, is an innovative normative integrated care group (group for shared values, culture and vision) established by the Canadian Mental Health Association-Montreal Branch. A programme evaluation approach was used to conduct a logic analysis of the CCIU in order to understand the relationships between its resources, activities and outcomes, build a common understanding and, allow for its replication.MethodsFive steps were involved in the creation of a programme logic model. A non-exhaustive literature search for similar initiatives, a review of documents related to the CCIU process and direct observations led to the development of a first model. Then, following a participatory and reflexive process, this model was validated with CCIU participants.ResultsA comprehensive model and a simplified model were created. Participants’ experiential knowledge and scientific knowledge helped to identify the essential components of the successful operation of the CCIU.ConclusionsThe CCIU, with its eight essential components, including relations based on equality and mutual respect, corresponds to an essential step in normative integration and integrated care that lead to improved quality services.

Health promotion and logic models

The first phase of interventional research in the health of populations is the development of its conceptual basis. Programme logic models are used to describe the way in which the results should be generated. The programme theory is used to develop intermediary objectives and to increase the chances of success of public health actions. This approach requires the coproduction of knowledge between researchers and partners. The teams can benefit from the application of evaluation to the theory in order to clarify the results obtained.

An early evaluation of an integrated, recovery orientated approach to severe and persisting mental health problems

Introduction: Partners in Recovery (PIR) is a national program run in 48 Australian local areas from 2012 until 2016. PIR aims to better support people with severe and persistent mental illness who have chronic and complex needs, through integrating health and other services in a recovery orientated approach aimed to be embedded in future practice through systems change. Recovery orientated practice recognises the strengths of service users and empowers them within the mental health system, emphasising shared decision making and respecting the self-determined goals of the consumer.We undertook a multiple methods evaluation of the early implementation of this program in an area to the west of Sydney. We worked with a Reference Group to develop a Program Logic Model (PLM) and assigned indictors. Using a combination of document review, surveys and interviews with consumers, carers and service providers and managers, we measured success in achieving the agreed indicators.Short description of practice change implementedWith funding from the Australian government, the Nepean Blue Mountains Medicare Local (a primary health care organisation west of Sydney) joined in a collaboration with partner organisations as well as consumers with severe and persisting mental health problems and their carers. PIR aims to improve outcomes through better coordination and integration of health and other services. A key longer term aim of the program is enduring systems change, both in terms of promoting recovery focussed care and improved and lasting collaborations across organisations to better achieve the consumers’ goals for recovery.Memoranda of Understanding and inter-organisational agreements assisted in aligning goals and priorities between the different agencies and regular meetings facilitated planning and review of activities across partner organisations. Care Facilitators were employed to assist consumers to access the required support available through the various participating organisations. Training was provided for staff of all partner organisations.Key findings: At the time of our evaluation PIR had been in operation for two years. Organisational structures had been developed and staff were employed according to the needs identified. Enrolment in the program initially occurred more rapidly than anticipated as consumers responded to promotional activities. Many of those enrolled reported improved access to services, although transport, housing, social opportunities and access to some specialist mental health services were noted particularly by staff, to remain challenges for some. Certain population groups in the area, including those living at a greater distance from urban centres, the homeless, those with drug and alcohol as well as mental health diagnoses and culturally diverse groups including Aboriginal and Torres Strait islander peoples appeared to be less well served by PIR.Highlights: This approach to integration of services for those with severe and persisting mental health problems was facilitated by the strong leadership provided by the Primary Care Organisation and the alignment of vision between most of the partner organisations. Challenges occurred where partner organisations were less familiar with and committed to a recovery approach to care. Though training was provided, organisational culture made change difficult in some organisations. Support Facilitators from one organisation will be placed in another organisation in an attempt to influence culture and understandings of participating organisations. Consumers and carers reported improved access to services and were keen to participate in leadership and peer support roles in the future. Staff and management of partner organisations reported improved access to services for their clients as a result of enhanced referral pathways and establishment of collaborative partnerships. However PIR was said to be a “work in progress” and many expressed concern about the short term nature of the current program.Conclusion: PIR in the Nepean Blue Mountains area of Sydney has provided an integrated client-centered service which has relevance in other settings. The PLM approach provided a useful framework for describing the program as well as for conducting an early evaluation of the program and indicating potential opportunities for further development. Our evaluation continues to inform program development.

Developing online accreditation education resources for health care services: An Australian Case Study.

In 2013, 'National Safety and Quality Health Service Standards' accreditation became mandatory for most health care services in Australia. Developing and maintaining accreditation education is challenging for health care services, particularly those in regional and rural settings. With accreditation imminent, there was a need to support health care services through the process. A needs analysis identified limited availability of open access online resources for national accreditation education. A standardized set of online accreditation education resources was the agreed solution to assist regional and rural health care services meet compulsory requirements. Education resources were developed over 3 months with project planning, implementation and assessment based on a program logic model. Resource evaluation was undertaken after the first 3 months of resource availability to establish initial usage and stakeholder perceptions. From 1 January 2015 to 31 March 2015, resource usage was 20 272, comprising 12 989 downloads, 3594 course completions and 3689 page views. Focus groups were conducted at two rural and one metropolitan hospital (n = 16), with rural hospitals reporting more benefits. Main user-based recommendations for future resource development were automatic access to customizable versions, ensuring suitability to intended audience, consistency between resource content and assessment tasks and availability of short and long length versions to meet differing users' needs. Further accreditation education resource development should continue to be collaborative, consider longer development timeframes and user-based recommendations.

An approach to measuring and encouraging research translation and research impact.

BackgroundResearch translation, particularly in the biomedical area, is often discussed but there are few methods that are routinely used to measure it or its impact. Of the impact measurement methods that are used, most aim to provide accountability – to measure and explain what was generated as a consequence of funding research. This case study reports on the development of a novel, conceptual framework that goes beyond measurement. The Framework To Assess the Impact from Translational health research, or FAIT, is a platform designed to prospectively measure and encourage research translation and research impact. A key assumption underpinning FAIT is that research translation is a prerequisite for research impact.MethodsThe research impact literature was mined to understand the range of existing frameworks and techniques employed to measure and encourage research translation and research impact. This review provided insights for the development of a FAIT prototype. A Steering Committee oversaw the project and provided the feedback that was used to refine FAIT.ResultsThe outcome of the case study was the conceptual framework, FAIT, which is based on a modified program logic model and a hybrid of three proven methodologies for measuring research impact, namely a modified Payback method, social return on investment, and case studies or narratives of the process by which research translates and generates impact.ConclusionAs funders increasingly seek to understand the return on their research investments, the routine measurement of research translation and research impact is likely to become mandatory rather than optional. Measurement of research impact on its own is insufficient. There should also be a mechanism attached to measurement that encourages research translation and impact – FAIT was designed for this task.

A Systems Approach to Social Entrepreneurship

Currently, disabilities organizations are increasingly being challenged by the requirement for individualized service, expectations to show personal outcomes, and the need to base their services on evidence-based practices. Social entrepreneurship (SE) is being put forward as an innovative approach for dealing with these challenges. This article presents a systems approach to SE based on a program logic model. This model identifies the input factors (a strong social vision, exploiting opportunities, maximizing resources), throughput strategies (entrepreneurial orientation, critical thinking skills, networking, capacity building), and outcome components (improving people’s lives, community-building, improving society) of SE at the micro, meso, and macro level. Also, the importance of planning for contextual changes as a social entrepreneur is discussed. The article concludes with presenting three inspiring practices regarding SE in the field of disabilities organizations.

Implementation of best manufacturing practices using logic models and system dynamics: project design and project assessment views

In today’s global competitive environment, the need for continuous improvement is a matter of considerable importance within manufacturing enterprises. To this end, project managers, and managers in general, design and assess different projects with the purpose of achieving efficient processes, reducing costs and waste, increasing product and service quality, developing new products and services, enhancing customer relationship management, optimising enterprise resources, and so on. However, it is well-known that managing enterprise resources in order to accomplish effective completion of projects is a complex task to carry out. Furthermore, it has been recognised that the way staff actually understands the purpose of a project, the way they perform different project activities, and how they are able to influence project design and assessment are key factors for influencing the success of a project. This paper presents a systemic methodology to design and assess projects more effectively and efficiently based on program logic models and system dynamics with the aim of facilitating a clear understanding of the needs, purposes, goals, activities and tasks of a project among its stakeholders towards achieving success.

Cultural adaptations of the Strengthening Families Programme 10-14 in the US Pacific Northwest: A qualitative evaluation

Objective: Most reports of adaptations to evidence-based prevention programmes for delivery to specific cultural groups describe formal adaptation procedures. In this paper, we report on how practitioners identify and manage issues of perceived cultural mismatch when delivering a scripted, evidence-based intervention. Design: We used grounded systems theory methodology to plan, conduct and interpret facilitator narratives. Method: Interviews and focus groups with 16 facilitators who delivered the Strengthening Families Programme 10-14, a widely disseminated intervention targeting families with young adolescents, to Latino and American Indian families in the Pacific Northwest region of the USA. Setting: Interviews and focus groups were conducted at programme facilitators’ worksites or other location of their choice. Results: Facilitators described important cultural adaptations related to programme format and timing, family composition and dynamics, acculturation challenges and language. Nearly all adaptations reported were consistent with the programme’s stated goals and were designed to retain families or clarify programme concepts. Conclusion: We recommend training practitioners in the importance and understanding of programme logic models and in culturally sensitive approaches to programme adaptation, including an awareness of differences between assimilation and acculturation.

Outcome Trajectories among Homeless Individuals with Mental Disorders in a Multisite Randomised Controlled Trial of Housing First.

Housing First (HF) has been shown to improve housing stability, on average, for formerly homeless adults with mental illness. However, little is known about patterns of change and characteristics that predict different outcome trajectories over time. This article reports on latent trajectories of housing stability among 2140 participants (84% followed 24 months) of a multisite randomised controlled trial of HF. Data were analyzed using generalised growth mixture modeling for the total cohort. Predictor variables were chosen based on the original program logic model and detailed reviews of other qualitative and quantitative findings. Treatment group assignment and level of need at baseline were included in the model. In total, 73% of HF participants and 43% of treatment-as-usual (TAU) participants were in stable housing after 24 months of follow-up. Six trajectories of housing stability were identified for each of the HF and TAU groups. Variables that distinguished different trajectories included gender, age, prior month income, Aboriginal status, total time homeless, previous hospitalizations, overall health, psychiatric symptoms, and comorbidity, while others such as education, diagnosis, and substance use problems did not. While the observed patterns and their predictors are of interest for further research and general service planning, no set of variables is yet known that can accurately predict the likelihood of particular individuals benefiting from HF programs at the outset.

P146: Designing better continuing education for rural emergency physicians

Introduction / Innovation Concept: Rural emergency physicians often work alone, and identify higher needs for continuing professional development in emergency medicine (EM) than urban doctors. We have offered the Community Emergency Medicine Outreach program (CEMO) at 12 rural hospitals in Eastern Ontario since 2009. Each emergency team selects topics in Adult EM for discussion at half-day outreach sessions at their local hospital. Methods: The CEMO program director participated in a Masters of Health Professions Education program. Newly learned concepts were applied to further the development of CEMO. Curriculum, Tool, or Material: Five important lessons learned, and their impacts on CEMO: First, curriculum design is a dynamic process. While CEMO was originally developed for physicians, the program has attracted many participants from other disciplines including nurses, administrators, pharmacists, and learners. Content and delivery have been redesigned to enhance inter-professional learning, which promotes team harmony, local problem solving, and knowledge translation into practice. Second, learning must be highly relevant to the local context to be effective. The content of each CEMO session is tailored to each group’s perceived and ascribed learning needs. CEMO is informed by sociocultural, transformative, experiential and cognitivist learning theories. Teaching strategies include interactive discussion of locally encountered clinical cases, and simulation. Third, it is more effective to integrate new technologies into a larger curriculum than to offer them as stand-alone modules. CEMO incorporates innovative presentation software, screencasts, procedural videos, and online audience response systems to engage participants. Fourth, learning effectiveness is best measured using multiple sources of assessment, and multiple assessments over time. CEMO’s learner assessment strategies include self-reflection at sessions, and months later. Participants consider CEMO’s effects on their practice, including reactions of co-workers and patients to their new skills, knowledge and behaviours. Finally, program evaluation may take many forms, and begins with defining evaluation goals and questions. We have developed a program logic model for CEMO, and a combined process and outcome evaluation is in progress. Conclusion: The application of important educational concepts promotes the design of effective continuing education in emergency medicine for rural health professionals.

Conceptual and Methodological Issues in Evaluations of Road Safety Countermeasures

Researchers have a long history in the conduct of evaluations of road safety countermeasures. However, despite the strengths of some evaluative road safety evaluations that align with previous and current thinking on program evaluation, few published road safety evaluations have followed standard conceptualization and methodology outlined in numerous program evaluation textbooks, journal articles and Web-based handbooks. However, conceptual and methodological challenges inherent in many evaluations of road safety countermeasures can affect causal attribution. Valid determination of causal attribution is enhanced by use of relevant theory or hypotheses on the putative mechanisms or pathways of change and by the use of a process evaluation to assess the actual implementation process. This article provides a detailed description of the constructs of causal chain, program logic models and process evaluation. This article provides an example of how these standard methods of theory-driven evaluation can improve the interpretation of outcomes and enhance causal attribution of a road safety countermeasure.

PC-FACS

PC-FACS

The Perceived Importance of Youth Educator’s Confidence in Delivering Leadership Development Programming

A successful component of programs designed to deliver youth leadership develop programs are youth educators who understand the importance of utilizing research-based information and seeking professional development opportunities. The purpose of this study was to determine youth educator’s perceived confidence in leading youth leadership development programs. Study objectives included describing types of youth leadership development training received by youth educators, describing the number of hours of youth leadership development training received by youth educators, describing the perceived importance of youth leadership development training received by youth educators, describing youth leadership development training delivery mode preferences of youth educators, determining if the addition of hours of training received and the perception of the importance of youth leadership training improved the prediction of perceived confidence beyond that provided by differences in selected demographic variables (gender, age, years of service, education level, office location). The target population for this study was southern region 4-H youth development educators. Results of this study indicate that perceived importance of youth leadership development training is predictive of youth educator’s confidence level in teaching youth leadership development. Youth leadership development programs are a cornerstone of many youth serving organizations. The importance of youth leadership development to the core goals of programs is illustrated by its explicit inclusion in program logic models as well as its implicit presence when older youth serve in teaching and mentoring roles for younger youth. The success or failure of a youth leadership development program may be linked to many factors, one of which is the educator who leads the program. Youth educators influence program success in many ways. The focus of this study is to explore educators’ prior experiences as well as their perceived confidence in delivering youth leadership development programming. Given the strong connections between a person’s beliefs and behaviors, it is important to understand youth educators’ beliefs within the context of their prior experiences (Ajzen, Joyce, Sheikh, & Cote, 2011). This study seeks to expand the literature about youth educators’ youth leadership development training beliefs. It has practical implications for developing professional development trainings for youth educators as well as theoretical implications for advancing a deeper understanding of educators’ core beliefs about training specifically related to youth leadership development.

Le protocole de l’intervention mapping : un processus méthodique pour élaborer, implanter et évaluer des programmes en promotion de la santé

BackgroundHealth promotion programs are expected to improve population health and reduce social inequalities in health. However, their theoretical foundations are frequently ill-defined, and their implementation faces many obstacles. The aim of this article is to describe the intervention mapping protocol in health promotion programs planning, used recently in several countries. MethodsThe challenges of planning health promotion programs are presented, and the six steps of the intervention mapping protocol are described with an example. Based on a literature review, the use of this protocol, its requirements and potential limitations are discussed. ResultsThe intervention mapping protocol has four essential characteristics: an ecological perspective (person–environment), a participative approach, the use of theoretical models in human and social sciences and the use of scientific evidence. It comprises six steps: conduct a health needs assessment, define change objectives, select theory-based change techniques and practical applications, organize techniques and applications into an intervention program (logic model), plan for program adoption, implementation, and sustainability, and generate an evaluation plan. This protocol was used in different countries and domains such as obesity, tobacco, physical activity, cancer and occupational health. Although its utilization requires resources and a critical stance, this protocol was used to develop interventions which efficacy was demonstrated. ConclusionThe intervention mapping protocol is an integrated process that fits the scientific and practical challenges of health promotion. It could be tested in France as it was used in other countries, in particular to reduce social inequalities in health.

Using the Program Logic Model to Evaluate ¡Cuídate!: A Sexual Health Program for Latino Adolescents in a School-Based Health Center.

Addressing the disparities in pregnancy and sexually transmitted infection (STI) rates for Latino youth in the United States requires an evidence-based approach. Although randomized controlled trials for sexual risk reduction interventions have shown promise in improving short-term outcomes, program sustainability has not been extensively examined in school settings where teens spend time. Latina teen pregnancy rates are nearly twice the national average. Adolescents comprise only 17% of the U.S. population, yet they account for 50% of STIs. ¡Cuídate! is a high-impact sexual health group program endorsed by the Centers for Disease Control and Prevention that has not been replicated in a real-world setting. The program logic model (PLM) was used as the systematic approach to plan, implement, and evaluate a sustainable model of sexual health group programing (¡Cuídate!) in a U.S. high school with a large Latino student population. The PLM provided the framework for the evaluation of outputs, outcome, and impact. A multivariate repeated measures analysis of variance was used as the means to evaluate the participant outcomes immediately postprogram and at 8-12 weeks. ¡Cuídate! was executed within an existing school structure and time constraints, below cost projections, and with high participant retention (95.8%). Three cohorts (N = 24) of female teens demonstrated significant increases in STI or HIV knowledge, self-efficacy, and intention to use condoms (p < .01). Condom use increased postprogram. No participants initiated sexual behavior, nor were there any reported pregnancies or STIs. An evidence-based intervention previously tested in randomized controlled trials can be sustained in a school-based health center with similar results of efficacy. Our success served as a platform for a sustainable program. We continue to extend the impact of the program by delivering ¡Cuídate! in the school setting using a community health worker.

A qualitative evaluation of the 2005–2011 National Academic Centers of Excellence in Youth Violence Prevention Program

The Centers for Disease Control and Prevention's Division of Violence Prevention (DVP) funded eight National Academic Centers of Excellence (ACEs) in Youth Violence Prevention from 2005 to 2010 and two Urban Partnership Academic Centers of Excellence (UPACEs) in Youth Violence Prevention from 2006 to 2011. The ACEs and UPACEs constitute DVP's 2005–2011 ACE Program. ACE Program goals include partnering with communities to promote youth violence (YV) prevention and fostering connections between research and community practice. This article describes a qualitative evaluation of the 2005–2011 ACE Program using an innovative approach for collecting and analyzing data from multiple large research centers via a web-based Information System (ACE-IS). The ACE-IS was established as an efficient mechanism to collect and document ACE research and programmatic activities. Performance indicators for the ACE Program were established in an ACE Program logic model. Data on performance indicators were collected through the ACE-IS biannually. Data assessed Centers’ ability to develop, implement, and evaluate YV prevention activities. Performance indicator data demonstrate substantial progress on Centers’ research in YV risk and protective factors, community partnerships, and other accomplishments. Findings provide important lessons learned, illustrate progress made by the Centers, and point to new directions for YV prevention research and programmatic efforts.