Logic Models for Program Evaluation in Emergency Nursing

Abstract

Helpful Online Tools and Resources for Developing Your Logic Model1.Centers for Disease Control and Prevention’s Program Performance and Evaluation Office (PPEO) Logic Models: https://www.cdc.gov/eval/logicmodels/2.W.K. Kellogg Foundation Logic Model Development Guide: https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide (provide name and email to download)3.University of Kansas Community Toolbox, Developing a Logic Model or Theory of Change: https://ctb.ku.edu/en/table-of-contents/overview/models-for-community-health-and-development/logic-model-development/main4.University of Wisconsin-Madison Program Development and Evaluation, Designing Programs: https://fyi.extension.wisc.edu/programdevelopment/designing-programs/In our work in emergency nursing, we often develop new and innovative programs. Examples of emergency nurse-led programs include the following: (1) bundled improvement interventions to implement new practice guidelines with multiple strategies and activities, (2) educational programs, and (3) injury prevention outreach programs. Careful program planning will include clear documentation of both the overall purpose and components of the program, as well as link specific program activities to the intended results. Programs need to be evaluated to make decisions about whether to adopt, modify, or abandon them. If successful, these programs need to be shared and replicated, and perhaps become models of best practice. Program planning and evaluation, similar to research, need a framework that others can follow to understand and replicate the project. One such framework is a logic model. 1.Centers for Disease Control and Prevention’s Program Performance and Evaluation Office (PPEO) Logic Models: https://www.cdc.gov/eval/logicmodels/2.W.K. Kellogg Foundation Logic Model Development Guide: https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide (provide name and email to download)3.University of Kansas Community Toolbox, Developing a Logic Model or Theory of Change: https://ctb.ku.edu/en/table-of-contents/overview/models-for-community-health-and-development/logic-model-development/main4.University of Wisconsin-Madison Program Development and Evaluation, Designing Programs: https://fyi.extension.wisc.edu/programdevelopment/designing-programs/ A logic model, sometimes called a program model or theory of change, is a planning tool and visual representation of the resources available for a project, the activities that will be carried out to complete a project, and the expected outcomes or results of a project. Think of a logic model as a diagram that shows the flow of the project along with the results to be measured and the expected program impact. Logic models are also excellent strategic planning and communication tools when developing a budget, generating buy-in, or applying for funding.1Demarco F.R. Healey-Walsh J. Community and Public Health Nursing: Evidence for Practice.3rd ed. Wolters Kluwer, Philadelphia2020: 200Google Scholar Creating a logic model is easy, and there are many templates available. An excellent resource can be found at: https://www.cdc.gov/eval/tools/logic_models/index.html. More resources, with blank templates and tutorials, are listed in the Box. A simple logic model can be created using the table feature of your word processor. Basic logic models can be broken down into depicting your planned work and your intended results. Your planned work will be listed in the first 2 columns as (1) inputs and (2) activities. Your intended results will be listed in the last 3 columns as (3) outputs/measures, (4) outcomes, and (5) impact. Overall, the first row of your table should contain columns that may be titled as follows: inputs, activities, outputs/measures, outcomes, and impact. These titles and the number of columns may vary depending on your program. Each column should link to the next in a logical if-then relationship. Table 1 includes definitions and examples for each column to demonstrate how a logic model might be set up. Table 2 illustrates how a logic model could be applied to a program described in this issue of the Journal of Emergency Nursing (JEN), entitled, “Advanced Placement Paramedic Education for Health Care Professionals: A Descriptive Evaluation.”2Leggio W.J. Miller M.G. Panchal A.R. Advanced placement paramedic education for health care professionals: A descriptive evaluation.J Emerg Nurs. 2020; 46: 44-50https://doi.org/10.1016/j.jen.2019.10.010Google ScholarTable 1Sample logic model template with definitions and descriptionsPlanned workIntended resultsInputsActivitiesOutputs/ MeasuresOutcomesImpactDefinition: program resources and/or infrastructure.Examples: materials, personnel, expertise, partnerships, regulations, funding, and practice guidelines.Definition: specific interventions.Examples: action steps in education, research, practice, and/or leadership. List all the activities that will be performed to complete the project.Definition: immediate, short-term process indicators or evidence that activities were performed as planned. Types, level, and targets of services.Examples: number of classes, duration of intervention, and the number of participants.Note: this must be determined before the project begins.Definition: measurable changes, effects, consequences, and results that came about as a direct or indirect result of the activities.Examples: specific changes in participant behavior, knowledge, skill, attitude, and short-term change in health or functioning status.Definition: overarching and long-term difference or change in the organization, system, or community.Examples: long-term reduction in unit morbidity or mortality, sustained culture change, national workforce capacity or proportion with specialty certification, and sustained improvement in workforce retention. Open table in a new tab Table 2Example of a logic model applied to non-EMS health care professionals in EMS Bridge ProgramInputsActivitiesMeasuresOutcomesImpactAdvisory boardProgram faculty and staffAlumniCommunity partnersUniversity classroom and simulation spaceLocal fire-based ALS EMSOther EMS agenciesBudgetQualified applicantsGap analysis between paramedic and nursing curriculaDevelop admission criteria and select participantsCreate self-assessmentDevelop didactic and simulation content for a 2-week course on university campusComplete affiliation agreements with EMS systemsDevelop and administer an 180-item comprehensive written examinationSchedule 10 shifts with local ALS EMS (per student)Schedule time to complete a total of 50 out-of-hospital capstone experiencesDemographic profile of students accepted into programLicensure typeDegrees heldYears of experienceAreas of clinical experience before the courseNumber of students who attended the courseHours to completionScore on comprehensive final examinationNumber of students who completed the capstone courseNumber of affiliation agreements for capstone experiencesStudent satisfaction levels with the courseStudent satisfaction levels with the program facultyNumber of students who passed the comprehensive final examinationNumber of students who completed all curricular requirementsNumber of students who took the National Registry of Emergency Medical Technicians Paramedic CertificationNumber of attempts to pass the National Registry of Emergency Medical Technicians Paramedic CertificationNumber of students who pass the National Registry of Emergency Medical Technicians Paramedic Certification examination in 3 attempts or lessNumber of students who passed the paramedic psychomotor examinationIncreased proportion of health care professionals in the national EMS workforceProportion of students who attained the National Registry of Emergency Medical Technicians Paramedic CertificationNumber of students who have completed requirements to function as a paramedic in their state (if National Paramedic Certification is not required)Proportion of students who maintain current National Paramedic CertificationNumber of alumni employed in EMS roleEMS, emergency medical services; ALS, advanced life support.Example logic model for Leggio WJ, Miller MG, Panchal AR. Advanced placement paramedic education for health care professionals: A descriptive evaluation. J Emerg Nurs. 2020;46:44-50. https://doi.org/10.1016/j.jen.2019.10.010. Open table in a new tab EMS, emergency medical services; ALS, advanced life support. Example logic model for Leggio WJ, Miller MG, Panchal AR. Advanced placement paramedic education for health care professionals: A descriptive evaluation. J Emerg Nurs. 2020;46:44-50. https://doi.org/10.1016/j.jen.2019.10.010. Logic models are often used in program evaluation, which systematically examines a total program.3Polit D.F. Beck C.T. Essential of Nursing Research: Appraising Evidence for Nursing Practice.8th ed. Wolters Kluwer/Lippincott/Williams & Wilkins Health, Philadelphia2014: 344Google Scholar When a program evaluation is conducted, 2 broad areas are evaluated; process and impact. Process evaluation looks at how the program is implemented and how it actually functions. In process analysis, questions might be asked about how people felt about the program or what they thought were barriers to the successful implementation of the program. In essence, a process evaluation reviews whether the program is being implemented as planned. Alternatively, impact analysis (or outcome evaluation) examines the impact or effects of the intervention. Because the program and program activities are interventions, generalizable intervention research designs may also be used to evaluate the impact of a program. The outcome research or evaluation design will depend on several factors.4Coly A. Parry G. Evaluating complex health interventions: A guide to rigorous research designs.https://www.academyhealth.org/files/AH_Evaluation_Guide_FINAL.pdfDate: 2017Date accessed: November 11, 2019Google Scholar If the evaluation research team can randomize sites or participants into the program, randomized trials or stepped wedge design may be ideal. If there are multiple data points gathered on the outcomes over time from both before and after implementation, interrupted time series or regression discontinuity designs may be applied. If contemporaneous controls or comparisons are available, a controlled before-and-after design can be considered. Lastly, and less rigorous, if there are no contemporaneous controls or comparisons, a before-and-after study that either compares the program participants with their own baseline or with other historic controls may be used. If the team conducting the evaluation had no control over implementing the program, observational designs and secondary data analysis may be considered. Single group quasi-experimental studies, observational designs and secondary data analyses limit the ability to draw causal conclusions that the program activities were directly responsible for the outcomes. Emergency nurses have vital perspective, knowledge, and skills for developing innovative programs that improve care. Logic models provide tools for emergency nurses to plan, design, and evaluate programs. JEN welcomes and encourages manuscript submissions focused on program development and evaluation with implications for emergency nursing.