Rigorous Program Evaluation Research Articles

The Development of a Service-Learning Program for First-Year Students Based on the Hallmarks of High Quality Service-learning and Rigorous Program Evaluation

We describe six hallmarks of high quality service-learning and explain how these considerations guided the development of a Transitional Coaching Program (TCP) during the first three years of implementation. We have demonstrated that the TCP is acceptable, feasible, and sustainable. Improvements have been seen in the degree of impact on learning objectives, but statistically significant change has not yet been achieved. This project highlights the importance of looking beyond satisfaction and engaging in rigorous assessment of learning objectives and ongoing quality improvement through attention to best practices and evidence-based, continuous quality improvement.

The impact of medical insurance for the poor in Georgia: a regression discontinuity approach

Improving access to health care and financial protection of the poor is a key concern for policymakers in low- and middle-income countries, but there have been few rigorous program evaluations. The Medical Insurance Program for the Poor in the republic of Georgia provides a free and extensive benefit package and operates through a publicly funded voucher program, enabling beneficiaries to choose their own private insurance company. Eligibility is determined by a proxy means test administered to applicant households. The objective of this study is to evaluate the program's impact on key outcomes including utilization, financial risk protection, and health behavior and management. A dedicated survey of approximately 3500 households around the thresholds was designed to minimize unobserved heterogeneity by sampling clusters with both beneficiary and non-beneficiary households. The research design exploits the sharp discontinuities at two regional eligibility thresholds to estimate local average treatment effects. Results suggest that the program did not affect utilization of health services but decreased mean out-of-pocket expenditures for some groups and reduced the risk of high inpatient expenditures. There are no systematic impacts on health behavior, management of chronic illnesses, and patient satisfaction.

A methodological template for the design and evaluation of evidence-based injury prevention fact sheets

In Canada, public health agencies often employ fact sheets or brochures as means of communicating injury prevention messaging to target groups; these are used because they are cost-effective and can...

Columbia University College of Physicians and Surgeons

Columbia University College of Physicians and Surgeons

Warm glow, information, and inefficient charitable giving

More than 200 donors participated in a framed field experiment which consisted of a series of decisions about how to divide a gift between a set of similar charities. Most subjects simultaneously gave to multiple charities with similar missions even when the social benefit of gifts to different charities were not equal, as proxied by the matching rates applied to subjects' gifts. Taking each subject's preferences over the set of charities as given, these choices resulted in substantial inefficiencies: subjects forfeited social surplus (matching funds) equal to 25% of the value of their gifts. Suggestive evidence indicates that warm glow utility derived from the act of making a gift, which can lead to a love of variety even among similar charities, and risk aversion over the social value of charitable gifts are both important factors motivating donors who make socially inefficient gifts. Additionally, few subjects were willing to pay for information that could have enabled them to increase the social benefit of their gifts, although many of these subjects also forfeited potential personal gains in an investment decision, casting some doubt on this interpretation. The possibility that the personal value of information might not be equal to the social value might help explain why there are so few rigorous evaluations of aid programs: such evaluations are costly to charities and might not be valued by donors.

Study Designs and Evaluation Models for Emergency Department Public Health Research

Abstract Public health research requires sound design and thoughtful consideration of potential biases that may influence the validity of results. It also requires careful implementation of protocols and procedures that are likely to translate from the research environment to actual clinical practice. This article is the product of a breakout session from the 2009 Academic Emergency Medicine consensus conference entitled "Public Health in the ED: Screening, Surveillance, and Intervention" and serves to describe in detail aspects of performing emergency department (ED)-based public health research, while serving as a resource for current and future researchers. In doing so, the authors describe methodologic features of study design, participant selection and retention, and measurements and analyses pertinent to public health research. In addition, a number of recommendations related to research methods and future investigations related to public health work in the ED are provided. Public health investigators are poised to make substantial contributions to this important area of research, but this will only be accomplished by employing sound research methodology in the context of rigorous program evaluation.

Evaluation of home lead remediation in an Australian mining community

In 1994 a comprehensive program was established to reduce children's blood lead levels in Broken Hill, NSW, Australia. Home remediation (abatement of lead hazards in a child's home) was included as part of a case management strategy for children with blood lead levels ≥ 15 µg/dL. Children with blood lead levels ≥ 30 µg/dL were offered immediate home remediation. Children with blood lead levels of 15–29 µg/dL were allocated to ‘immediate’ or ‘delayed’ home remediation; a subset of these participated in a randomized controlled trial (RCT) to evaluate the effectiveness of home remediation for reducing blood lead levels. One hundred and seventeen children received home remediation. One hundred and thirteen returned for follow-up blood tests, 88 of whom participated in the RCT. On average children's blood lead levels decreased by 1.7 µg/dL (10%) in the 6 months after remediation and by 2.2 µg/dL (13%) in the 6–12 months after remediation. However, remediation did not significantly change the rate of decline in blood lead levels ( P = 0.609). There was no evidence of association between change in children's blood lead levels and changes in lead loading in their homes. The results are consistent with the published literature, which suggests that home remediation does not reduce children's exposure to lead sufficiently to cause a moderate or greater decrease in their blood lead level. In communities where lead is widely dispersed, the study suggests that it is important to assess potential sources and pathways by which children are exposed to lead when developing an intervention plan, and the need for multiple interventions to effectively reduce blood lead levels. The findings reinforce the ongoing need for rigorous epidemiological evaluation of lead management programs to improve the evidence base, and for effective primary prevention to avoid children being exposed to lead in the first place.

Reduction of Shape and Weight Concern in Young Adolescents: A 30-Month Controlled Evaluation of a Media Literacy Program

ObjectiveRecent eating disorder prevention efforts have targeted high-risk females in late adolescence (>15 years). Methodologically rigorous evaluations of prevention programs directed to younger, mixed-sex, universal audiences are largely absent. The primary objective was to evaluate a theoretically informed media literacy program delivered to a mixed-sex, universal, young adolescent audience. MethodFive hundred forty Grade 8 students (mean age 13.62 years, SD 0.37 years) from 4 schools participated with a total of 11 classes receiving the 8-lesson media literacy program (126 girls and 107 boys) and 13 comparison classes receiving their normal school lessons (147 girls and 160 boys). Shape and weight concern (primary outcome variable) and seven additional eating disorder risk factors (e.g., dieting, media internalization) were measured with validated questionnaires at baseline, postprogram, and 6- and 30-month follow-up. ResultsLinear mixed model analyses were conducted using a 2 (group: media literacy program, control) × 3 (time: postprogram, 6-month follow-up, 30-month follow-up) × 2 (sex: girls, boys) mixed within-between design, with baseline entered as a covariate. Main effects for group, favoring the media literacy program, were found for shape and weight concern (effect size [ES] = 0.29), dieting (ES = 0.26), body dissatisfaction (ES = 0.20), ineffectiveness (ES = 0.23), and depression (ES = 0.26). Conclusion: Media literacy can be an effective intervention for reducing shape and weight concern and other eating disorder risk factors long-term in a universal mixed-sex, young adolescent population. More evaluations of methodologically sound prevention programs are required with this demographic. J. Am. Acad. Child Adolesc. Psychiatry, 2009;48(6):652-661.

Internet‐Based Psychological and Oral Medical Treatment Compared to Psychological Treatment Alone for ED

Sexual function through pregnancy and the postpartum period is an important aspect of quality of life. Despite this, prospective studies are limited, and the impact of body image on sexual function has not been explored. The current study reports on a pilot study that evaluated the effectiveness of a combined Internet-based psychological treatment program and oral medication compared to an Internet-based psychological program. Twelve men with ED participated in the study. Assessments were made pre and postintervention on ED, relationship satisfaction, sexual satisfaction, self-esteem, and quality of life. Multivariate analyses of variance demonstrated that both treatment conditions showed improvements from pretest to posttest in sexual functioning, relationships, and personal well-being. This study indicates the importance of targeting psychological and relationship variables in the treatment of ED. Further research with larger numbers of participants is necessary for a more rigorous evaluation of treatment programs that combine psychological and medication interventions for ED.

POLICYMAKING TO IMPROVE MATERNAL HEALTH

We read with interest the article by Behague and Storeng titled Collapsing the Vertical-Horizontal Divide: An Ethnographic Study of Evidence-Based Policymaking in Maternal Health.1 The authors should be applauded for their efforts to analyze the complicated and competing factors that have limited advances in policy and programs that improve the health and survival of women in developing countries. The authors address a number of important points in their article. Here we would like to comment on three: (1) the methods used (2) how policymakers choose the evidence that determines how programs are designed and (3) the use of randomized controlled trials in generating data to use for new maternal health programs. The first comment is how the data were gathered. In the Methods section the authors reported that they interviewed 67 professionals identified opportunistically through professional networking publications and conference proceedings.1(p644) The authors did not report that they used a systematic process to generate the list of people interviewed for their study nor did they report whether they gave different weight to responses from interviewees from different groups such as advocates or researchers. Thus the methods used-which may be primarily from key informants in a limited circle of maternal health advocates and researchers-may generate results that are biased or limited in views regarding maternal health research and policy. The second comment is in regards to the authors observation that some advocacy efforts for maternal health would benefit from additional evidence to guide programs and policy that they might advocate.2 We agree. In a perfect world policymaker decisions should be based on an appropriate balance of advocacy and sound evidence and policymakers would support the attainment of evidence to try to achieve this ideal balance. Our final comment is about the authors observation that randomized controlled trials are not necessarily the final gold standard when it comes to developing evidence to be used to design the most effective maternal health programs. As the authors point out randomized controlled trials can only be planned to address the most focused of questions. Although this experimental approach represents a standard to attain they are often not well suited to the messy reality of realworld service-delivery programs in developing countries. Therefore we concur with the authors call for alternate models to generate evidence useful in devising better health care programs. These models include approaches such as nonequivalent control group designs (quasi-experimental designs) and rigorous program evaluation approaches. (full-text)

A Longitudinal Study of the Impact of Village Health Education on Environmental Sanitation

Health educators working in developing nations commonly encounter administrative constraints and methodological problems which prevent rigorous program evaluation. Using a time-series design, this study followed approximately 600 families in two Guatemalan villages during a four-year period to measure the impact of community health education on seven indicators of change in environmental sanitation. Adoption of new behavior was accompanied by widespread change in village attitudes and perceptions of health benefits. At the level of the family, change in perception was not predictive of change in behavior. A strong relationship between participation in planned educational events and the adoption of sanitation changes suggests that the formative educational strategy was responsible for stimulating change for five of seven indicators. Suggestions are made about the usefulness of the methodology, ways that private behavioral changes may be measured are discussed, and the advantages of the time-series design for health education evaluation and monitoring are noted.

An Evaluation of Theoretical Frameworks for Studying Physically Active Leisure

Recent reviews demonstrate participation in active recreation and the development of an active lifestyle is important to a broad spectrum of society. Reviews further highlight the presence of an underdeveloped knowledge base for active leisure. Theory development is required to guide researchers, practitioners, and policy makers. Specifically, evaluating the theoretical frameworks applied to active recreation is needed to foster collaboration and enhance the research-practice relation as well as to allow for rigorous testing and evaluation of theories, policies, programs, and interventions. Our paper responds to this call by developing five criteria to assess a selection of theoretical frameworks. The assessment demonstrates the limitations and advantages of various frameworks. Five propositions are presented to illustrate future theory development.

Coordinated School Health Programs and Academic Achievement: A Systematic Review of the Literature

Few evaluations of school health programs measure academic outcomes. K-12 education needs evidence for academic achievement to implement school programs. This article presents a systematic review of the literature to examine evidence that school health programs aligned with the Coordinated School Health Program (CSHP) model improve academic success. A multidisciplinary panel of health researchers searched the literature related to academic achievement and elements of the CSHP model (health services, counseling/social services, nutrition services, health promotion for staff, parent/family/community involvement, healthy school environment, physical education, and health education) to identify scientifically rigorous studies of interventions. Study designs were classified according to the analytic framework provided in the Guide developed by the Community Preventive Services Task Force. The strongest evidence from scientifically rigorous evaluations exists for a positive effect on some academic outcomes from school health programs for asthmatic children that incorporate health education and parental involvement. Strong evidence also exists for a lack of negative effects of physical education programs on academic outcomes. Limited evidence from scientifically rigorous evaluations support the effect of nutrition services, health services, and mental health programs, but no such evidence is found in the literature to support the effect of staff health promotion programs or school environment interventions on academic outcomes. Scientifically rigorous evaluation of school health programs is challenging to conduct due to issues related to sample size, recruitment, random assignment to condition, implementation fidelity, costs, and adequate follow-up time. However, school health programs hold promise for improving academic outcomes for children.

Do Knowledge and Experience Have Specific Roles in Triage Decision‐making?

Accuracy of triage decisions is a major influence on patient outcomes. Triage nurses' knowledge and experience have been cited as influential factors in triage decision-making. The aim of this article is to examine the independent roles of factual knowledge and experience in triage decisions. All of the articles cited in this review were research papers that examined the relationship between triage decisions and knowledge and/or experience of triage nurses. Numerous studies have shown that factual knowledge is an important factor in improving triage decisions. Although a number of studies have examined the role of experience as an independent influence on triage decisions, none have found a significant relationship between experience and triage decision-making. Factual knowledge appears to be more important than years of emergency nursing or triage experience in triage decision accuracy. Many triage education programs are underpinned by the assumption that knowledge acquisition will result in improved triage decisions. A better understanding of the relationships between clinical decisions, knowledge, and experience is pivotal for the rigorous evaluation of education programs.

Can regulation and governance make a difference?

Can regulation and governance make a difference?

Tobacco education: have New Zealand primary schools done their homework?

To describe the content, delivery and evaluation of tobacco-related (smoke-free) education to pre-teen students and to compare this education with current best practice. Postal survey to staff of 136 New Zealand (NZ) schools with primary-aged children. More than 90% of schools offered some form of education about the harmful effects of tobacco use and many relied on external providers for this education. Although most schools offered some form of smoke-free education, there was a lack of rigorous program evaluation, particularly of programs provided by external agencies. Smoking remains a significant health problem in NZ and improving the quality of smoke-free education for pre-teen students, before most young people initiate smoking, would be a positive step towards better health.

Implementing “Corrections of Place” Ideas

Restorative justice has recently become popular among practitioners, but there have been few rigorous evaluations of programs. For approximately 5 years, Ventura County, California, probation staff and other local organizations worked together on the South Oxnard Challenge Project (SOCP), a juvenile probation program designed to implement Clear’s “corrections of place” (COP) ideas. The program is evaluated using a randomized experimental design and the results of surveys with youth clients, staff, and the youths’ caregivers regarding the effectiveness of the SOCP in reaching COP-related goals are reported. Results reflect that there were few differences between SOCP clients and routine probationers’ perceptions of their experiences. SOCP staff members were significantly more likely to believe they were reaching COP goals than were routine probation officers. Caregivers believed the SOCP was more helpful than routine probation.

Integrating the Learning of Mathematics and Science Using Interactive Teaching and Learning Strategies

To help students grasp the intimate connections that exist between mathematics and its applications in other disciplines a library of interactive learning modules was developed. This library covers the mathematical areas normally studied by undergraduate students and is used in science courses at all levels. Moreover, the library is designed not just to provide critical connections across disciplines but to also provide longitudinal subject reinforcement as students progress in their studies. In the process of developing the modules a complete editing and publishing system was constructed that is optimized for automated maintenance and upgradeability of materials. The result is a single integrated production system for web-based educational materials. Included in this is a rigorous assessment program, involving both internal and external evaluations of each module. As will be seen, the formative evaluation obtained during the development of the library resulted in the modules successfully bridging multiple disciplines and breaking down the disciplinary barriers commonly found in their math and non-math courses.

Faith-based efforts to improve prisoner reentry: Assessing the logic and evidence

Prisoner reentry constitutes one of the central criminal justice challenges confronting U.S. society. Coinciding with this emerging social problem has been increased policymaker interest in faith-based programs to improve outcomes for vulnerable populations, including released prisoners. Critical questions about the nature and effects of faith-based reentry programs remain largely unaddressed, however: (1) What is a “faith-based” program? (2) How does or could such a program reduce recidivism and improve other behavioral outcomes among released offenders? (3) What is the evidence concerning the impacts of faith-based reentry programs? (4) What are critical implementation issues that may affect the operations and impacts of such programs? This article examines each of these questions and identifies critical conceptual, theoretical, and research gaps in the literature. It highlights that the term “faith-based” is used inconsistently, that the precise causal relationship, if any, between various measures of faith and crime remains in question, and that few rigorous evaluations of faith-based reentry programs exist. It then discusses recommendations for improving knowledge and practice.

The Effectiveness of Counter‐Terrorism Strategies

The objective of this Campbell systematic review is to determine the effectiveness of counter‐terrorism strategies from the available social scientific research literature using systematic review methods.From over 20,000 studies we located on terrorism, we found only seven which contained moderately rigorous evaluations of counterterrorism programs. We conclude that there is little scientific knowledge about the effectiveness of most counter‐terrorism interventions. Further, from the evidence we were able to locate, it appears that some evaluated interventions either didn't work or sometimes increased the likelihood of terrorism and terrorism‐related harm.The findings of this review dramatically emphasize the need for government leaders, policy makers, researchers, and funding agencies to include and insist on evaluations of the effectiveness of these programs in their agendas. These agendas would include identifying ways to overcome methodological and data challenges often associated with terrorism research, increasing funding to evaluate existing programs through methodologically rigorous evaluation designs, and paying attention to existing evaluations of programs when implementing them. Further, programs should be assessed to establish if they cause more harm than good or if they create unanticipated consequences.