Results Of Program Evaluation Research Articles

Networking in eHealth research: results of the IDRC SEARCH program evaluation.

BackgroundThe IDRC ‘Strengthening Equity through Applied Research Capacity building in eHealth’ (SEARCH) funded seven research projects in Bangladesh, Burkina Faso, Ethiopia, Kenya, Lebanon, Peru and Vietnam that sought to answer questions or test solutions related to the use of Internet or mobile phone technology in strengthening health systems. The evaluation accompanied these projects over two years to answer, among others, the question how cross-grant learning interactions influenced project outcomes.MethodsThe evaluation team conducted repeated interviews and on-line questionnaire surveys with the research teams and analysed the information exchanges among researchers on a SharePoint site established by IDRC.ResultsThe expectations of the SEARCH program in terms of cross-project learning were only partially realized. The diversity of themes, language barriers and differences in context were cited as main reasons. Non-facilitated active cross-grant networking was only observed between two teams working in English on thematically similar issues. However, networking among all projects was active during two program workshops organized by IDRC.ConclusionsNetworking among research teams can increase the quality and the applicability of health systems research and potentially promote knowledge translation. Spontaneous networking across language barriers is, however, difficult. Effective global research networks require dedicated human and financial resources to keep them vibrant and alive.Keywordse-health, refugees

Development and Testing of Program Evaluation Instruments for the iCook 4-H Curriculum

ObjectiveTo develop and test the validity of program outcome evaluation instruments for cooking, eating, and playing together for obesity prevention during iCook 4-H. DesignInstrument development for both youth and adults through pre-post testing of items newly constructed and compiled to address key curriculum constructs. Testing occurred throughout program intervention and dissemination to determine dimensionality, internal consistency and test-retest reliability, and validity. SettingA 5-state out-of-school program in cooperative extension and other community sites. ParticipantsYouths aged 9–10 years; adults were main food preparers; the first phase involved 214 dyads and the second phase, 74 dyads. Main Outcome Measure(s)Youth measures were cooking skills, culinary self-efficacy, physical activity, and openness to new foods. Adult measures were cooking together, physical activity, and eating together. AnalysisExploratory factor analysis to determine initial scale structure and confirmatory factor analysis to confirm factor structures. Longitudinal invariance tests to see whether the factor structure held over time. Test-retest reliability was determined by Pearson r and internal consistency was determined by coefficient Ω and Cronbach α. Validity testing was determined by Pearson r correlations. ResultsYouth cooking skills, openness to new foods, and adult eating together and cooking together showed strong evidence for dimensionality, reliability, and validity. Youth physical activity and adult physical activity measures showed strong evidence for dimensionality and validity but not reliability. The youth culinary self-efficacy measure showed strong evidence for reliability and validity but weaker evidence for dimensionality. Conclusions and ImplicationsProgram outcome evaluation instruments for youths and adults were developed and tested to accompany the iCook 4-H curriculum. Program leaders, stakeholders, and administrators may monitor outcomes within and across programs and generate consistent reporting.

Regional Outcome Evaluation Program: inequities in access to health care in Lazio region of Italy.

Regional Outcome Evaluation Program: inequities in access to health care in Lazio region of Italy.

Regional Outcome Evaluation Program: inequities in access to health care in Lazio region of Italy.

Regional Outcome Evaluation Program: inequities in access to health care in Lazio region of Italy.

Measures of hospital competition and their impact on early mortality for congestive heart failure, acute myocardial infarction and cardiac surgery.

To measure competition amongst providers and to examine whether a correlation exists with hospitals mortality for congestive heart failure (CHF), acute myocardial infarction (AMI), isolated-coronary artery bypass graft (CABG) or valve surgery. Cross-sectional study based on publically available data from the National Outcome Evaluation Program (Edition 2016) of the Italian Agency for Regional Health Services. Patients discharged during 2015 for CHF or AMI, and between 2014 and 2015 for cardiac surgery (respectively, from 662, 395 and 91 hospitals). Risk-adjusted mortality rates at 30 days and measures of hospital competition for areas centred on hospital' location (fixed-radius 50-150 km, variable-radius to capture 10-30 hospitals and 6-10% of national volume). Competition was estimated as number of providers and Herfindahl-Hirschman Index (HHI). Indicators of competitions varied by condition and were sensitive to method used for the area definition. Hospital mortality after AMI and valve surgery increased with competition in areas identified by the variable-radius method (higher rates for a greater number of hospitals or lower HHIs). In area with fixed radius of 100-150 km, competition reduced mortality after CABG procedures (lower rates for a greater number of hospitals or smaller HHIs). Neither the number of hospitals nor HHI correlated with outcomes in CHF. The measures of hospital competition changed according to definition of local market and results in mortality correlations varied among conditions. Understanding the relationship between hospital competition and outcomes is important to identify strategies to improve quality of care.

Remunerated patient care services and injections by pharmacists: An international update

ObjectivesRecognizing pharmacists' increasing roles as primary care providers, programs offering remuneration for patient care services, and the administration of injections by pharmacists continue to be implemented. The objective of this article is to provide an update on remuneration programs available to pharmacists internationally for nondispensing services. Data sourcesSystematic searches for relevant articles published from January 2013 to February 2018 across Pubmed (Medline), Embase, International Pharmaceutical Abstracts, Cochrane Library, Econlit, Scopus, and Web of Science. Gray literature searches, including targeted searches of websites of payers and pharmacy associations, were also performed. Study selectionPrograms were included if they were newly introduced or had changes to patient eligibility criteria and fees since previously published reviews and if they were established programs offered by third-party payers for activities separate from dispensing. Data extractionDescriptive information on each program was extracted, including the program's jurisdiction (country and state, provincial, or regional level, as applicable), payer, service description, patient eligibility criteria, and fee structure. ResultsOver the 5-year period studied, 95 new programs for noninjection patient care services and 37 programs for pharmacist-administered injections were introduced. Large ranges in fees offered for similar programs were observed across programs, even within the same country or region, at an average of $US 71 for an initial medication review, $19 for follow-ups to these reviews, $18 for prescription adaptations, and $13 for injection administration. Apart from some smoking cessation programs in England, which offered incentive payments for successful quits, all services were remunerated on a fee-for-service basis, often in the form of a flat fee regardless of the time spent providing the service. ConclusionAlthough funding for pharmacists' activities continues to show growth, concerns identified in previous reviews persist, including the great variability in remunerated activities, patient eligibility, and fees. These issues may limit opportunities for multijurisdictional program and service outcome evaluation.

Assessment of Programs Aimed to Decrease or Prevent Mistreatment of Medical Trainees

Mistreatment of medical students is pervasive and has negative effects on performance, well-being, and patient care. To document the published programmatic and curricular attempts to decrease the incidence of mistreatment. PubMed, Scopus, ERIC, the Cochrane Library, PsycINFO, and MedEdPORTAL were searched. Comprehensive searches were run on "mistreatment" and "abuse of medical trainees" on all peer-reviewed publications until November 1, 2017. Citations were reviewed for descriptions of programs to decrease the incidence of mistreatment in a medical school or hospital with program evaluation data. A mistreatment program was defined as an educational effort to reduce the abuse, mistreatment, harassment, or discrimination of trainees. Studies of the incidence of mistreatment without description of a program, references to a mistreatment program without outcome data, or a program that has never been implemented were excluded. Authors independently reviewed all retrieved citations. Articles that any author found to meet inclusion criteria were included in a full-text review. The data extraction form was developed based on the guidelines for Best Evidence in Medical Education. An assessment of the study quality was conducted using a conceptual framework of 6 elements essential to the reporting of experimental studies in medical education. A descriptive review of the interventions and outcomes is presented along with an analysis of the methodological quality of the studies. A separate review of the MedEdPORTAL mistreatment curricula was conducted. Of 3347 citations identified, 10 studies met inclusion criteria. Of the programs included in the 10 studies, all were implemented in academic medical centers. Seven programs were in the United States, 1 in Canada, 1 in the United Kingdom, and 1 in Australia. The most common format was a combination of lectures, workshops, and seminars over a variable time period. Overall, quality of included studies was low and only 1 study included a conceptual framework. Outcomes were most often limited to participant survey data. The program outcome evaluations consisted primarily of surveys and reports of mistreatment. All of the included studies evaluated participant satisfaction, which was mostly qualitative. Seven studies also included the frequency of mistreatment reports; either surveys to assess perception of the frequency of mistreatment or the frequency of reports via official reporting channels. Five mistreatment program curricula from MedEdPORTAL were also identified; of these, only 2 presented outcome data. There are very few published programs attempting to address mistreatment of medical trainees. This review identifies a gap in the literature and provides advice for reporting on mistreatment programs.

Use of community perceptions to evaluate and adapt coastal resource management practices in the Philippines

Coastal resource management (CRM) programs have been implemented in the Philippines since the 1980s with the specific intent to protect and rehabilitate coastal habitats and enhance the sustainability of coastal communities. However, the implementation of these programs alone does not guarantee the success of program objectives. Monitoring and evaluation of program outcomes is essential for determining if programs are effective in reaching management goals. The purpose of this research was to evaluate long-term CRM programs using community perceptions of coastal resource condition, management practices, and program outcomes. Coastal residents in Baybay City, Leyte were surveyed and asked to rank a series of questions related to resource condition and 20 previously described management performance indicators. Respondents reported a decline in coastal resource condition over the past decade, even though they perceive positive outcomes of management programs aimed at enhancing resource condition. The sustainability and efficiency of management outcomes were perceived positively, with mixed views on equity outcomes. Results suggest that lack of full inclusion, low management oversight, and threat to coastal resources are concerns of the coastal community. Socioeconomic data collected from respondents yielded a pattern indicating that gender, self-reported level of environmental knowledge, and management cluster were significant contextual variables associated with perceptions of respondents. Overall, CRM programs are perceived to have a positive impact and there is a high level of interest for participating in future management activities. This study exemplifies how perceptions and associated contextual information of the community provides invaluable insights into the effectiveness of coastal resource management and be incorporated into the adaptive management cycle.

External Mentor Program: A Pathway to Career Advancement for Women in STEM

Mentor availability helps reduce the barriers facing women, providing skills and confidence necessary to successfully navigate different career stages in academia, and especially in Science, Technology, Engineering, and Math (STEM) related areas. For example, mentoring was noted to increase women’s persistence in engineering, and presumably other STEM fields (Frehill et al., 2006). Washington State University (WSU) began an External Mentor Program under the National Science Foundation ADVANCE Institutional Transformation award, and because of its success, this program is now sustained internally. Results of the WSU External Mentor Program evaluation (N=40) indicate that participating women faculty in STEM disciplines endorsed high levels of overall satisfaction with the External Mentor Program, reported that their expectations were met, and expressed consensus regarding recommending participation to their colleagues. All but one participant generated external grant proposals after taking part in the External Mentor Program. In-depth follow-up interviews revealed additional benefits, principally falling within the following domains: validation, collaboration, access and networking, and a “trickle down” effect - benefits conferred to the students, and participants’ departments more broadly. We conclude that External Mentor programming has emerged as an important tool in the arsenal of institutional strategies aimed at enhancing retention and advancement of women faculty in STEM fields.

The Kindergarten Early Language Intervention (KELI) Program: Multiculturalism in action

The Kindergarten Early Language Intervention (KELI) Program is an innovative program designed to provide early oral language intervention for young students from schools with high socio-economic needs, in order to prevent or alleviate literacy, academic, and social problems. The KELI Program is provided in addition to the regular kindergarten class for students who have not been formally identified for special education services. The program is based on social-pragmatic theory and research indicating that children develop oral language through natural social interactions in meaningful situations, and incorporates an intercultural orientation. The KELI Program has proven itself to be a strength-based, innovative, dynamic, and successful language intervention program that is responsive to the cultural and linguistic needs of diverse students. This article presents the rationale for the development of the program, description of the program, results of program evaluation, and suggestions for implementation of a similar initiative.

Metrics and outcomes of conservation education: a quarter century of lessons learned

We conducted a systematic literature review to analyze evaluations of conservation education programs on a global scale in order to better understand (1) temporal and spatial trends in conservation education program evaluations over the last 25 years, (2) patterns in the types of conservation-related issues addressed through these programs, (3) metrics that indicate effectiveness of conservation education programs, and (4) methods and timeframes used to draw conclusions about program outcomes. Findings indicated that there is a need to better connect the types of issues addressed through conservation education programs with metrics that would indicate success in addressing these issues and the actual outcomes measured and reported. As well, there is an opportunity to employ a variety of metrics and methods for evaluating program outcomes, particularly in developing countries, by focusing on cognitive and behavioral components as well as social and ecological ones. Finally, shifting to a more comprehensive strategy for evaluating multiple outcomes in different cultural contexts would provide opportunities for utilizing mixed methods and qualitative approaches in partnership with community stakeholders.

Regional Outcome Evaluation Program (P.Re.Val.E.): Reduction of inequality in access to effective health care in the Lazio region of Italy (2012-2015).

BackgroundInequalities in health among groups of various socio-economic status (as measured by education, occupation, and income) constitute one of the main challenges for public health. Since 2006, the Lazio Regional Outcome Evaluation Program (P.Re.Val.E.), presents a set of indicators of hospital performance based on quality standards driven by strong clinical recommendations, and measures the variation in the access to effective health care for different population groups and providers in the Lazio Region. One of the aims of the program was to compare population subgroups in order to promote equity in service provision. Since June 2013, a new management strategy has been put in place that assigned specific goals based on performance assessment to the chief executive officers of the hospitals.AimTo evaluate whether, in recent years, there has been a reduction in the differential access to effective health care, among individuals with different educational levels.MethodsWe enrolled all patients discharged from both public and private hospitals of the Lazio region between 2012 and 2015, living in Lazio region. We analysed the proportion of patients with ST-elevation myocardial infarction (STEMI) treated with percutaneous coronary intervention within 90 minutes (primary PCI), the proportion of patients with hip fracture (HF) who underwent surgery within 2 days, and the proportion of women with primary C-section. We applied multivariate logistic regression models to assess the effect of educational level on health outcomes, adjusting for demographic characteristics and comorbidities that could affect the outcomes. For each year of the study period, we compared adjusted proportions of outcomes for the highest and the lowest level of education by using percentage differences.ResultsIn the Lazio region, 44.6% of STEMI patients (N = 3,299) were treated with primary PCI, 54.4% of patients with hip fractures (N = 6,602) underwent surgery within 2 days, and 27.7% of women without a previous C-section (N = 34,718) delivered via C-section, in 2015. The corresponding proportions in 2012 were 27.8%, 31.3% and 31.5%, respectively. By comparing the adjusted proportions in patients with the highest education level (a university degree or higher) to those with the lowest level education level (None/Primary school), a decrease in the percentage difference was observed during the study period. In STEMI and delivery cohorts, the improvement of outcomes involved the least and the most educated patients, respectively, and the difference between the two educational levels was close to zero in 2015, whereas for hip patients, the improvement was more evident among the less educated patients.ConclusionsIn the Lazio region, we observed a reduction in the differential access to effective heath care by educational level, in different clinical areas. Different factors might explain these results. On top of the public disclosure of outcome data, the management strategy applied in mid-2013 might have driven the overall improvement of the health system for the considered conditions, helping to achieve a fairer access to health.

Pilot Study Results of a Longitudinal Integrated Clerkship in a Single Institution

The longitudinal integrated clerkship is an innovative approach in medical education that emphasizes continuity as a key principle to provide student-centered experiences and learning. Seoul National University College of Medicine decided to adapt longitudinal integrated clerkships to the new curriculum beginning in 2018, and therefore conducted pilot studies in 2016 and 2017. This study aimed to analyze the program evaluation results of the two pilot programs and discuss issues related to the successful implementation of longitudinal integrated clerkships in Korea. We conducted a focus group interview with nine students who participated in the 2016 pilot program and 13 students who participated in the 2017 pilot program. We also conducted a focus group interview with 11 faculty members who either participated in the pilot program or will participate in the main program. From the 2016 experience, we found that it is not appropriate to assign each patient to a single student and let the students contact their patients personally for feasibility and safety reasons. In the 2017 pilot program, we assigned each patient to a group of students, which made it more feasible for the students to follow-up with their patients. The students were satisfied with their new experience of longitudinal patient follow-up and regular meetings in the pilot program. Faculty emphasized the importance of establishing the course objectives and holding an orientation for the students and the faculty. Further study is planned to evaluate the early outcomes of the main longitudinalintegrated clerkship.

A Systematic Review of Global Surgery Partnerships and a Proposed Framework for Sustainability

Background: Building surgical capacity through global surgery partnerships (GSPs) between high and low-middle income countries (LMICs) is a rising global health focus. This systematic review characterizes strategies employed by GSPs to build capacity and promote sustainability. Methods: A systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA). PubMed, Embase, Medline, and African Journals Online databases were searched to identify all peer reviewed articles from 2000-2016 describing GSPs between the US/Canada and LMICs. Descriptive features were analyzed, with focus on attributes that promote sustainability. Findings: 3580 abstracts were gathered and independently reviewed by four authors. 128 papers (3·6%) met inclusion criteria describing GSPs in 68 countries on 5 continents. Excluded papers were nonsurgical, unilateral, or military initiatives. 21·9% demonstrated community engagement and 51·6% included multidisciplinary collaboration. Surgical training or education was provided in 81·3% of GSPs. Whereas 64·8% of GSPs had data collection, 53·1% reported project-related outcomes. 55·5% had bilateral authorship, and 28·9% demonstrated multisource funding. Only one GSP fulfilled all six author-proposed criteria of sustainability. Interpretation: Features of GSPs' capacity building and sustainability fell into six pillars: community engagement, multidisciplinary collaboration, education and training, outcomes reporting, bilateral authorship, and multisource funding. We propose that future global surgery partnerships should build on a foundation of bilateral ideas and expertise exchange, defined and measurable objectives, continuous evaluation of program outcomes, and a thoughtful and transparent approach to sustained capacity building. Funding: None. Declaration of Interest: None to declare. Ethical Approval: Not required as per UBC Research Ethics Board eligibility criteria.

Pharmacist training in suicide prevention

ObjectiveSuicide in the United States is a major preventable public health problem. Pharmacists need to be educated on suicide prevention strategies so that they can increase their own awareness and identify patients at-risk. A training program for pharmacists was used to provide skills necessary to recognize a crisis and the warning signs of suicide. The program's effect on the participant's general perception, self-efficacy, and attitude towards suicide prevention was examined. SettingVarious academic, health care, and professional meetings throughout San Diego County. Practice innovationFirst Question, Persuade, and Refer training program targeting pharmacists. EvaluationA self-administered presurvey, postsurvey and, Program Outcome Evaluation were given to participants of the suicide training program. Items included demographics, general perception, self-efficacy, and attitude toward suicide prevention. Descriptive statistics were used to describe participants’ demographics. t tests were used to compare general perception, attitudes, and self-efficacy scores between pretest and post-program evaluation survey responses. Nonparametric Wilcoxon signed rank analyses for matched pairs were used to compare survey responses that asked about attitudes before and after trainings. Regression analyses were conducted to assess factors associated with general perception, self-efficacy, and attitudes. ResultsParticipants were more likely to update knowledge after training and reported more confidence to make an intervention for a patient at risk for suicide. ConclusionOur findings suggest that a suicide prevention training program helped pharmacist respondents build confidence in several self-efficacy areas relating to detection of suicide signs, response to patients with suicidal thoughts, reassurance for patients, and provision of resources and referrals.

Backpack Programs and the Crisis Narrative of Child Hunger—A Critical Review of the Rationale, Targeting, and Potential Benefits and Harms of an Expanding but Untested Model of Practice

AbstractIn recent years, school-based food backpack programs (BPPs) have come into national prominence as a response to a perceived crisis of child hunger in America. Distributing bags of free food directly to schoolchildren for their own personal consumption each weekend, BPPs bring together private donors, faith communities, and public schools around an intuitively appealing project: children are hungry, and so we give them food. Perhaps because of their intuitive appeal, BPPs have expanded rapidly, without rigorous evaluation to determine their impacts on children, families, and schools. This Perspective aims to open up thinking about BPPs, first articulating the implicit conceptual model that undergirds BPPs, drawing on documentation offered by major program providers and on our own experience working with several schools implementing BPPs, to provide a window into what BPPs do and how and why they do it. We focus in particular on how the crisis narrative of child hunger has shaped the BPP model and on the related interplay between public sympathy and the neoliberal climate in which structural solutions to family poverty are eschewed. We then assess the BPP model in light of existing knowledge, concluding that BPPs fit poorly with the needs of the majority of children living in food-insecure households in the United States and consequently put children at risk of negative consequences associated with worry, shame, stigma, and disruptions to family functioning. Finally, we provide recommendations for practice and research, emphasizing the importance of 1) responding to children's actual needs throughout program implementation, 2) avoiding unnecessary risks by effective targeting of services to only those children who need them, and 3) rigorously evaluating program outcomes and unintended consequences to determine whether, even for the small number of US children who experience hunger, the benefits of the BPP model outweigh its psychosocial costs.

Systematic Plan of Evaluation Part II: Assessment of Program Outcomes

As an accrediting agency recognized by the U.S. Department of Education (USDE) and the Council for Higher Education Accreditation (CHEA), the Accreditation Commission for Education in Nursing (ACEN) has established Accreditation Standards and Criteria for the evaluation of nursing programs, including the evaluation of outcomes. This article focuses on the essential components and processes for systematic evaluation of program outcomes, including licensure examination pass rate, program completion rate, and job placement rate.

Patient-centeredness of integrated care programs for people with multimorbidity. Results from the European ICARE4EU project

IntroductionThis paper aims to support the implementation of patient-centered care for people with multimorbidity in Europe, by providing insight into ways in which patient-centeredness is currently shaped in integrated care programs for people with multimorbidity in European countries. MethodsIn 2014, expert organizations in 31 European countries identified 200 integrated care practices (‘programs’) in 25 countries of which 123 were included in our study. Managers of 112 programs from 24 countries completed a questionnaire about characteristics and results of the program, including questions on elements of patient-centeredness. Eight programs that were considered especially innovative or promising were analyzed in depth. ResultsPrograms used various methodologies to involve people with multimorbidity in decision-making, such as motivational interviewing and narrative counseling techniques. In 79 programs individual care plans were developed together with patients. Few programs had already been systematically evaluated, but in one program it was shown that working with individual care plans based on patients’ goals and resources resulted in increased patient satisfaction with care. Various barriers to deliver patient-centered care were reported, including inadequate knowledge and skills of both patients and professionals. ConclusionIn many European countries innovative approaches are applied to increase patient-centeredness of care for people with multimorbidity. To assess their potential benefits and conditions for implementation, thorough process and outcome evaluations of programs are urgently needed.

Use of the PRECIS-II instrument to categorize reports along the efficacy-effectiveness spectrum in an hepatitis C virus care continuum systematic review and meta-analysis

There is increasing recognition of the importance of the distinction between efficacy and effectiveness research in the design, conduct, and evaluation of interventions and program outcomes. There is a concurrent increase in the application of systematic reviews and meta-analyses. These two lines of inquiry are only beginning to meet. There is an emerging need for systematic reviews and meta-analyses to account for differences in degrees to which included studies reflect either efficacy or effectiveness design. Based on ongoing work on a formal systematic review of the hepatitis C virus care continuum, this paper describes and discusses the rationale for, and how the PRECIS-II instrument can be used on, and modestly adapted to, studies included in the systematic review examining the extent to which studies include elements of efficacy or effectiveness or a combination of the two. We also highlight that use of such an instrument may have general applicability to and value in the conduct of systematic reviews and meta-analysis.

Common components analysis: An adapted approach for evaluating programs

Common Components Analysis (CCA) summarizes the results of program evaluations that utilize randomized control trials and have demonstrated effectiveness in improving their intended outcome(s) into their key elements. This area of research has integrated and modified the existing CCA approach to provide a means of evaluating components of programs without a solid evidence-base, across a variety of target outcomes. This adapted CCA approach (a) captures a variety of similar program characteristics to increase the quality of the comparison within components; (b) identifies components from four primary areas (i.e., content, process, barrier reduction, and sustainability) within specific programming domains (e.g., vocation, social); and (c) proposes future directions to test the extent to which the common components are associated with changes in intended program outcomes (e.g., employment, job retention). The purpose of this paper is to discuss the feasibility of this adapted CCA approach. To illustrate the utility of this technique, researchers used CCA with two popular employment programs that target successful Veteran reintegration but have limited program evaluation – Hire Heroes USA and Hire Our Heroes. This adapted CCA could be applied to longitudinal research designs to identify all utilized programs and the most promising components of these programs as they relate to changes in outcomes.