Support Program Evaluation Research Articles

Supporting clinicians post exposure to potentially traumatic events: Emergency department peer support program evaluation.

Workers in EDs are regularly exposed to potentially traumatic events. Since the COVID-19 pandemic, there has been exponential interest in peer support programs (PSPs) in a range of settings. We describe a PSP implemented in 2017 at University Hospital Geelong (UHG) ED together with results of a survey. To describe the program such that others can replicate it in their settings in addition to feedback evaluation. Method involved a survey emailed to the ED doctors. Thirty responses from 96 emails with a range of feedback. ED Doctors place high value on the PSP.

As our population changes, so must we: how race and ethnicity data merging has evolved over time in the Wisconsin Administrative Data Core

The Institute for Research on Poverty (IRP), centered at the University of Wisconsin-Madison in Wisconsin, USA, began collaboration with various Wisconsin state agencies in 1985 to link data between disparate state agencies. These initial efforts have since evolved into the Wisconsin Administrative Data Core (WADC) which is an individual-level population dataset connecting extensive administrative data spanning decades of Wisconsin social services program history. Since its purpose is to facilitate evidence-driven policymaking and support program evaluation, it is essential that our data linking efforts accurately represent our Wisconsin population. Since its inception, the WADC has adapted to changes in how race and ethnicity are collected and reported. While Wisconsin has a racial makeup that is generally less diverse than the national population, the number of non-white and multi-racial identifying residents has increased over time. Furthermore, varying state agency demographic collection and reporting methods have fueled the need for our linking methods to change, too. IRP initially implemented standard data linkage methods using a mixture of probabilistic and deterministic matching of personal identifying information. But as the racial composition of Wisconsin residents changed over time, our linking assumptions induced implicit biases. In response, we updated our matching algorithm to account for the cultural nuances in how personally identifying information is assigned and reported within various racial and ethnic subgroups. Furthermore, we prioritized values from data sources that were self-reported rather than assigned from a third-party. These methodological modifications have improved the WADC’s reliability and representativeness of the population.

Engaging stakeholders to retrospectively discern implementation strategies to support program evaluation: Proposed method and case study

BackgroundAvailability of evidence-based practices (EBPs) is critical for improving health care outcomes, but diffusion can be challenging. Implementation activities increase the adoption of EBPs and support sustainability. However, when implementation activities are a part of quality improvement processes, evaluation of the time and cost associated with these activities is challenged by the need for a correct classification of these activities to a known taxonomy of implementation strategies by implementation actors. DesignObservational study of a four-stage, stakeholder-engaged process for identifying implementation activities and estimating the associated costs. ResultsA national initiative in the Veterans Health Administration (VHA) to improve Advance Care Planning (ACP) via Group Visits (ACP-GV) for rural veterans identified 49 potential implementation activities. Evaluators translated and reduced these to 14 strategies used across three groups with the aid of implementation actors. Data were collected to determine the total implementation effort and applied cost estimates to estimate the budget impact of implementation for VHA. LimitationsRecall bias may influence the identification of potential implementation activities. ConclusionsThis process improved understanding of the implementation effort and allowed estimation of ACP-GV 's budget impact. ImplicationsA four-stage, stakeholder-engaged methodology can be applied to other initiatives when a pragmatic evaluation of implementation efforts is needed.

네트워크 분석을 통한 학사경고자 대상 연구 동향

The purpose of this paper is to examine the trends of academic probation research through keyword network analysis, and to suggest the direction of academic probation research in the future. The key words were selected using Textom program, and the 1-mode network matrix was constructed and visualized. The analysis results are as follows. 1)The papers were published in 96 journals in five research over a 10-year period. 2)Research on academic probation is mainly focused on ‘program development and analysis’, ‘participation in programs of academic warnings’, ‘experience of academic warnings’, and ‘the impact of academic warnings on college life. 3)In the first period, many studies were conducted to identify the characteristics, status, and cause of academic warnings. In the second period, research on the process of overcoming academic warnings and counseling experiences was conducted, and in the third period, the most research was conducted on the development and evaluation of support programs for academic warnings.

Synthesizing frameworks and tools to develop a plan for evaluating an online data utilization curriculum for teachers

This paper presents an overview of how a plan was developed for evaluating Develop Your Data Mindset: Essentials of Educational Data Use, an online curriculum intended to improve in-service and pre-service educators’ knowledge and skills relevant to data utilization. Five categories of assessment, a program evaluation framework, logic model, theory of change, project plan, metrics spreadsheet, and the A+ Inquiry model are synthesized to demonstrate how they supported the development of an evaluation plan. Types of evidence are described to provide an overview of the need for improved instructional supports relevant to educator data utilization, how the activities to develop and implement an online curriculum are appropriate to meet the need, how well the activities are implemented as planned, and the extent to which intended outcomes of the curriculum are achieved. Applying the methods outlined in this paper yielded a plan to help ensure the curriculum would be assessed through a comprehensive evaluation lens. Limitations are acknowledged. Methods described in this paper could be adapted to support program evaluation planning and implementation beyond the scope of this project.

Developing a dashboard for program evaluation in competency-based training programs: a design-based research project.

BackgroundCanadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires the assessment of entrustable professional activities (EPAs). Dashboards could be used to track the completion of EPAs to support program evaluation.MethodsUsing a design-based research process, we identified program evaluation needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. We interviewed leaders from the emergency medicine program and postgraduate medical education office at the University of Saskatchewan. Two investigators thematically analyzed interview transcripts to identify program evaluation needs that were audited by two additional investigators. Identified needs were described using quotes, analytics, and visualizations.ResultsBetween July 1, 2019 and April 6, 2021 we conducted 17 interviews with six participants (two program leaders and four institutional leaders). Four needs emerged as themes: tracking changes in overall assessment metrics, comparing metrics to the assessment plan, evaluating rotation performance, and engagement with the assessment metrics. We addressed these needs by presenting analytics and visualizations within a dashboard.ConclusionsWe identified program evaluation needs related to EPA assessments and designed dashboard elements to meet them. This work will inform the development of other CBME assessment dashboards designed to support program evaluation.

Professional and Practical Considerations for the Program Evaluation Dissertation

This essay discusses the emergence of the program evaluation dissertation in our doctoral program as the preferred dissertation in practice (DiP) option. We also outline important considerations that must be reviewed with students when considering this approach. Our students are professional educational leaders in the settings where they conduct their dissertation research, emphasizing the importance of our doctoral faculty in addressing the potential implications of blending dissertation research with professional practice. Using the utilization-focused evaluation approach as a framework, we address professional and practical considerations to ensure effective evaluation designs to examine a specific problem of practice. These considerations include the student's relationship to the program being evaluated, the impact on their professional position, support for the evaluation, access to data collection, potential consent concerns, and the utilization of findings. We conclude with additional ethical considerations to be considered when supporting program evaluation dissertation work.

The New Brunswick Act Respecting Research Three Years Later: A Data Trickle Turns into a Flood

IntroductionIn New Brunswick Canada (NB) in 2017, the Provincial Government passed a bill called the Act Respecting Research. This Act took the form of an omnibus bill modifying 20 pieces of legislation to define a legal authority for the New Brunswick Institute for Research, Data and Training (NB-IRDT) to receive prepared, linkable microdata from multiple public bodies in NB. This was followed in 2019 by a second Act Respecting Research that broadened the scope of the first. Together this legislation has helped to expand NB-IRDT’s data holdings from its first dataset – hospital records received in 2015 – to over 40 different linkable datasets as of March 2020. These datasets are underpinning ambitious research partnerships between NB-IRDT and the Provincial Government.
 Objectives and ApproachThis presentation will detail the rapid progress made since the first Act was passed in 2017 and presented at IPDLN-2018. It will outline enabling factors, including secured funding, the central role of the Department of Health, engagement with senior decisionmakers, and interaction with other provincial data centres and national networks including HDRN Canada. Ongoing and new challenges arising from the rapid increase in the scale of data collection and their resolution will be discussed.
 ResultsIn addition to a wide range of provincial health administrative and clinical datasets, recent datasets include school records (report cards, standardized testing, attendance), income support data, workers compensation claims data, higher education program and graduation data, court appearances, adult training/retraining programs and immigration landing records. All files are linkable at the individual level. Multi-year research projects to support program evaluation are underway, with departments now able to access other agency data through NB-IRDT.
 Conclusion / ImplicationsNB-IRDT and the experience in NB offer important lessons for other jurisdictions aiming to expand access to linkable multi-agency data for research and evaluation.

PP391 Economic Analysis Of Treatment For Spinal Muscular Atrophy: A Scoping Review

IntroductionSpinal muscular atrophy (SMA) is a rare, life-threatening, and seriously debilitating neuromuscular disorder, which has a heavy burden on patients, caregivers and the health system. Technological advances have improved clinical effect, but have also increased the financial burden. There is limited information in the literature on the resource utilization and economic burden of SMA. Our research aims to summarize the current literature on resource use, cost and economic evaluations of treatments for SMA, to inform further research and policy decision making.MethodsDatabases, including PubMed, Embase, Cochrane Library and CRD Database, were searched from inception. Two reviewers undertook title and abstract screening followed by full-text screening, and any disagreement was resolved in consensus. Data extraction was conducted using a customized form. Included studies were summarized using narrative synthesis structured around general and economic characteristics. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were adhered to where applicable.ResultsWe reviewed 552 abstracts and included twenty-six from 2015 to 2019. Four-fifths were published in the United States and Europe. Five full economic evaluations and one budget impact analysis compared nusinersen with AVXS-101 or best supportive care, and the remaining evaluated the economic burden of SMA. The most common outcomes were healthcare resource utilization and direct medical costs, only a few studies evaluated direct non-medical costs or indirect cost.ConclusionsSMA patients have significant medical expenditures and high utilization of healthcare services, including nusinersen-treated patients. The results highlight the substantial burden of treatment for SMA, not only for patients but also for their caregivers. SMA represents a significant hidden cost that society should be made aware of, and that should be considered in the design, implementation and evaluation of support programs for people who suffer from this disease and their families, as well as in the economic evaluation of new treatments.

HOW TO EVALUATE A PROGRAM WRITTEN BY CHILDREN: A PROGRAM EVALUATION SUPPORT TOOL FOR ELEMENTARY SCHOOL TEACHERS

Programming education will be an obligation at primary school in Japan starting in 2020. This leads elementary school teachers evaluate programs that are created by the students. However, a clear standard of evaluation which is necessary for teachers as an evaluation guidance has not been stipulated yet. The programming obligation will surely give a big burden to the them. Therefore, we are developing an application that support program evaluation, which can support teachers in evaluating programs created by children and enable to connect with the future guidance.A Program created by the children using Scratch will be read as an image and analyzed according to the evaluation item using template matching. By presenting materials to support evaluation using an application to the teachers, it can reduce their burden. In addition, by converting the results into data, it is easier to judge whether or not the children understand the content of the lesson, the teachers can respond to the children who cannot follow the class at an early stage.

Evaluating Dancing With Parkinson's: Reflections from the perspective of a community organization

In 2015, Dancing With Parkinson's (DWP), a Toronto-based community organization, participated in the Ontario Brain Institute's (OBI) newly launched Evaluation Support Program. This paper reflects on that experience. In particular, we identify the key lessons derived from the OBI initiative, discuss how these lessons have informed DWP practice going forward, and highlight what we consider to be the most valuable aspects of the Evaluation Support Program. While we now recognize the need to establish an evaluation culture within DWP, we find that there are significant challenges associated with both building and sustaining evaluation capacity in the context of a small community-based organization. Whereas DWP has built considerable strengths in terms of informal evaluation capacity, on its own, such capacity is insufficient to, for example, demonstrate DWP's impact to outside audiences or successfully scale up the program.

The theory of change of the evaluation support program: Enhancing the role of community organizations in providing an ecology of care for neurological disorders

This paper discusses the Ontario Brain Institute’s theory of change for the Evaluation Support Program, a program designed to enhance the role of community organizations in providing care and services for people living with a brain disorder. This is done by helping community organizations build evaluation capacity and foster the use of evidence to inform their activities and services. Helping organizations to build capacities to track the ‘key ingredients’ of their successes will help ensure that successes are replicated and services can be improved to maximize the benefit that people receive from them. This paper describes the hypothesized outcomes and early impacts of the Evaluation Support Program, as well as how the program will contribute to the field of evaluation capacity building.

Putting evaluation capacity building in context: Reflections on the Ontario Brain Institute’s Evaluation Support Program

This article, in three parts, reflects on the content of the six articles included in the forum. It begins with a description of the Evaluation Support Program, emphasizing its key attributes. Next, it raises two points regarding ECB theory: (1) the need to become clearer about the concepts and terms used to describe and study this phenomenon, and (2) the potential value of social science theory to understand ECB and improve its practice. The article concludes with practical ideas for improving ECB: (1) framing it as an educative act, which assigns the evaluator the critical role of evaluation teacher/coach; and (2) the importance of never assuming that an ECB effort begins in unchartered territory, but rather that it builds on people’s knowledge, skills, attitudes, and previous experiences.

Experiments in evaluation capacity building: Enhancing brain disorders research impact in Ontario

This paper is the introductory paper on a forum on evaluation capacity building for enhancing impacts of research on brain disorders. It describes challenges and opportunities of building evaluation capacity among community-based organizations in Ontario involved in enhancing brain health and supporting people living with a brain disorder. Using an example of a capacity building program called the “Evaluation Support Program”, which is run by the Ontario Brain Institute, this forum discusses multiple themes including evaluation capacity building, evaluation culture and evaluation methodologies appropriate for evaluating complex community interventions. The goal of the Evaluation Support Program is to help community-based organizations build the capacity to demonstrate the value that they offer in order to improve, sustain, and spread their programs and activities. One of the features of this forum is that perspectives on the Evaluation Support Program are provided by multiple stakeholders, including the community-based organizations, evaluation team members involved in capacity building, thought leaders in the fields of evaluation capacity building and evaluation culture, and the funders.

Reflections on experiential learning in evaluation capacity building with a community organization, Dancing With Parkinson’s

This paper discusses what was learned about evaluation capacity building with community organizations who deliver services to individuals with neurological disorders. Evaluation specialists engaged by the Ontario Brain Institute Evaluation Support Program were paired with community organizations, such as Dancing With Parkinson’s. Some of the learning included: relationship building is key for this model of capacity building; community organizations often have had negative experiences with evaluation and the idea that evaluations can be friendly tools in implementing meaningful programs is one key mechanism by which such an initiative can work; community organizations often need evaluation most to be able to demonstrate their value; a strength of this initiative was that the focus was not just on creating products but mostly on developing a learning process in which capacities would remain; evaluation tools and skills that organizations found useful were developing a theory of change and the concept of heterogeneous mechanisms (informed by a realist evaluation lens).

Assessing Individual Health Insurance Coverage and Utilization Before and After the Patient Protection and Affordable Care Act

The adoption of the Patient Protection and Affordable Care Act (PPACA) in 2010 with the intent to improve the U.S. health care delivery system by expanding health insurance coverage and controlling health care costs has generated intense debate regarding its implementation. Marketplaces known as insurance exchanges have been established to provide coverage for Americans who otherwise could not get affordable health care benefits. These exchanges have been plagued with financial losses and other challenges leading to several large insurance providers discontinuing participation in the program. There are many possible remedies under consideration to make the program work better. This research seeks to support program evaluation as well as potential modifications to the law by providing baseline data to compare access and costs in states with state-based exchanges compared to states with federal exchanges. The authors perform an analysis by state for the years 2012 and 2013 (pre-PPACA implementation) using data from the Current Population Survey (U.S. Census) as well as de-identified claims data from Inovalon, Inc.

Transcatheter aortic valve replacement program development: recommendations for best practice.

Transcatheter aortic valve replacement (TAVR) is an increasingly available therapy for the management of aortic stenosis in higher risk populations. Beyond addressing the procedural challenges, centers must attend to the unique requirements of developing TAVR programs from referral to follow-up. The aim of this article is to outline the recommendations for best practice for program development from centers with early and extensive experience. The guideline-recommended Heart Team approach requires interdisciplinary agreements, delineation of roles and responsibilities, and the development of the role of the TAVR Coordinator. To support appropriate case selection, the screening and evaluation must be organized in a comprehensive clinic visit. In addition to the multimodality imaging tests, the assessment of functional status and frailty is pivotal to the eligibility decision. Throughout the TAVR trajectory, careful attention must be afforded to the integration of geriatric best practices. Pre-procedure care requires patient and family education to manage expectations and facilitate early discharge planning. Peri-procedural care planning, including equipment requirements, monitoring protocols, and emergency intervention agreements, contributes to procedural success. The aims of post-procedure care are to monitor the recovery, facilitate the rapid return to baseline status, and optimize length of stay. TAVR programs require data management strategies to facilitate and monitor program growth, support program evaluation, and meet the requirements for submission to national registries. TAVR represents a paradigm shift in the management of structural heart disease. Programmatic success and patient outcomes depend on the development of a comprehensive and collaborative program tailored to TAVR.

A systematic review and summarization of the recommendations and research surrounding Curriculum-Based Measurement of oral reading fluency (CBM-R) decision rules

Research and policy have established that data are necessary to guide decisions within education. Many of these decisions are made within problem solving and response to intervention frameworks for service delivery. Curriculum-Based Measurement in Reading (CBM-R) is a widely used data collection procedure within those models of service delivery. Although the evidence for CBM-R as a screening and benchmarking procedure has been summarized multiple times in the literature, there is no comprehensive review of the evidence for its application to monitor and evaluate individual student progress. The purpose of this study was to identify and summarize the psychometric and empirical evidence for CBM-R as it is used to monitor and evaluate student progress. There was an emphasis on the recommended number of data points collected during progress monitoring and interpretive guidelines. The review identified 171 journal articles, chapters, and instructional manuals using online search engines and research databases. Recommendations and evidence from 102 documents that met the study criteria were evaluated and summarized. Results indicate that most decision-making practices are based on expert opinion and that there is very limited psychometric or empirical support for such practices. There is a lack of published evidence to support program evaluation and progress monitoring with CBM-R. More research is required to inform data collection procedures and interpretive guidelines.

Levers supporting program evaluation culture and capacity in Romanian public administration: The role of leadership

Program evaluation culture and capacity is at the very beginning of its development in Romania. In this article we highlight one of the fundamental, but not always obvious, connections that support sustainable evaluation culture and capacity building and development: the link between leadership and program evaluation. If properly used, program evaluation results can be a strong instrument in leadership, just as leadership can fundamentally encourage the development of evaluation culture and capacity. More precisely, we identify the ways in which different power sources can help leaders in developing evaluation culture and capacity.

Facilitated Sensemaking

Family members of intensive care unit patients may develop anxiety, depression, and/or posttraumatic stress syndrome. Approaches to prevention are not well defined. Before testing preventive measures, it is important to evaluate which interventions the family will accept, use, and value. The purpose of this study was to evaluate the feasibility of an intervention for support for families of mechanically ventilated adults, grounded in a new midrange nursing theory titled "Facilitated Sensemaking." Families were provided a kit of supplies and the primary investigator coached families on how to obtain information, interpret surroundings, and participate in care. Participants were asked to complete an adapted Critical Care Family Needs Inventory and Family Support Program evaluation. Family members of 30 patients consented to participate; 22 participants completed the surveys. Internal consistency reliability of the adapted Critical Care Family Needs Inventory was high (alpha = .96). Results validated the importance of informational needs and provided a score indicating the family member's perception of how well each need was met, weighted by importance, which identified performance improvement opportunities for use by clinical managers. The program evaluation confirmed that families will use this format of support and find it helpful. Personal care supplies (eg, lotion, lip balm) were universally well received. Forty-two referrals to ancillary service were made. Operational issues to improve services were identified. As proposed in the Facilitated Sensemaking model, family members welcomed interventions targeted to help make sense of the new situation and make sense of their new role as caregiver. Planned supportive interventions were perceived as helpful.