Program Impact Evaluation Research Articles

Charting the Impact of Federal Spending for Education Research

Impact evaluation plays a critical role in determining whether federally funded research programs in science, technology, engineering, and mathematics are wise investments. This paper develops quantitative methods for program evaluation and applies this approach to a flagship National Science Foundation–funded education research program, Research and Evaluation on Education in Science and Engineering (REESE). Results of three different bibliometric analyses all point to the same conclusion: REESE is an interdisciplinary research program that attracts highly productive investigators who exhibit an additional increase in their productivity rate as a result of receiving REESE funding. Limitations of the bibliometric approach are discussed, and directions are provided for the future of impact evaluations of research programs intended to serve the public good.

Synthesis and review: delivering on conservation promises: the challenges of managing and measuring conservation outcomes

Growing threats and limited resources have always been the financial realities of biodiversity conservation. As the conservation sector has matured, however, the accountability of conservation investments has become an increasingly debated topic, with two key topics being driven to the forefront of the discourse: understanding how to manage the risks associated with our conservation investments and demonstrating that our investments are making a difference through evidence-based analyses. A better understanding of the uncertainties associated with conservation decisions is a central component of managing risks to investments that is often neglected. This focus issue presents both theoretical and applied approaches to quantifying and managing risks. Furthermore, transparent and replicable approaches to measuring impacts of conservation investments are noticeably absent in many conservation programs globally. This focus issue contains state of the art conservation program impact evaluations that both demonstrate how these methods can be used to measure outcomes as well as directing future investments. This focus issue thus brings together current thinking and case studies that can provide a valuable resource for directing future conservation investments.

The Long-Run Effects of a Public Policy on Alcohol Tastes and Mortality

We study the long-run effects of Russia’s anti-alcohol campaign, which dramatically altered the relative supply of hard and light alcohol in the late 1980s. We find that this policy shifted young men’s long-run preferences from hard to light alcohol decades later, and we estimate the age at which consumers form their tastes. We show that the large beer market expansion in the late 1990s had similar effects on young consumers’ tastes, while older consumers’ tastes remained largely unchanged. We then link these long-run changes in alcohol consumption patterns to changes in male mortality. The shift from hard to light alcohol reduced incidences of binge drinking substantially, leading to fewer alcohol-related deaths. We conclude that the resulting large cohort differences in current alcohol consumption shares explain a significant part of the recent decrease in male mortality. Simulations suggest that mortality will continue to decrease by another 23 percent over the next 20 years due to persistent changes in consumer tastes. Program impact evaluations that focus only on contemporaneous effects can therefore severely underestimate the total effect of such public policies that change preferences for goods. (JEL D12, H25, H31, I12, I18, L66, P36)

Effect of a food aid program on BMI/A of Mexican children, mediated by diet (251.3)

Objective: To evaluate the effect of a Mexican Food Aid Program (Programa de Apoyo Alimentario, PAL) on Body Mass Index for Age (BMI/A) and the role of diet of < 5 y. Design: Secondary analysis of a program impact evaluation using a cluster randomized control effectiveness trial design. Setting: Poor and rural households in southern Mexico. Subjects: We analyzed a sample of 2,100 beneficiary children < 5 y from 206 communities that were randomly assigned to receive monthly transfers of either a food basket, with (FBE) or without (FB) an education (E) component, or cash with E (~14 USD/month) vs. a control group. Results: PAL did not contribute to an unhealthy weight of beneficiary children < 5 y as expressed as BMI/A z‐scores. Children in the FBE group had lower BMI/A z‐scores than those children in the control group (p=0.027). This effect is explained by a higher fruit and vegetables consumption and by a smaller sweet consumption in children belonging to the FBE group.

Family planning impact evaluation: the evolution of techniques

This paper is a slightly revised version of a paper prepared for the seminar on methods for impact evaluation of family planning programs held in Jaco, Costa Rica, May 14-16, 1997. The seminar was sponsored by the International Union for the Scientific Study of Population (IUSSP), the United States Agency for International Development (USAID), the Carolina Population Center of the University of North Carolina, and the Central American Population Program of the University of Costa Rica. The goal of the seminar was to look at current methodological problems facing careful evaluation of the impact of programs, to examine some of the new methods that have been developed to address persistent issues, and to assess the methodological challenges posed by the expanded goals of many programs following the 1994 Cairo International Conference on Population and Development. This paper was designed to serve as the background to discussions of current methodologies and issues by tracing the development and nature of methods for assessing impact that started soon after the first programs were initiated in the 1950s. The techniques discussed include standardization and trend analysis, the analyses of acceptor data, experimental designs, multivariate areal analysis, population-based surveys, and multilevel strategies. The intent of the program sponsors and coordinators was to publish the collected papers but various contingencies intervened to make this infeasible. A description of the seminar and many of the papers are maintained on the University of Costa Rica website: http://ccp.ucr.ac.cr/noticias/plani/iusspi.htm. As a background chapter, the original version contained references to many of the other chapters planned for the volume. As many of these papers appear on the website, relevant references are given to the authors and this website throughout the paper.

Impact Evaluation of Innovation and Linkage Development Programs in Costa Rica: The Cases of Propyme and Cr Provee

This paper estimates the impact of two productive development programs (PDPs) in Costa Rica: PROPYME and CR Provee. The first seeks to increase the capacity of small and medium-sized firms (SMEs) to innovate, and the second aims to increase backward linkages between Costa Rican SMEs and multinational companies operating in the country. The impacts of each program were measured in terms of three result variables: real average wages, employment demand, and the probability of exporting. A combination of fixed effects and propensity score matching techniques was used in estimations to correct for any selection bias. The results show that both PROPYME and CR Provee have positive and significant impacts on SME performance. PROPYME?s beneficiaries performed better than other firms in terms of labor demand and their probability of exporting, while firms treated by CR Provee showed higher average wages, labor demand, and chances of exporting than untreated firms. Firms treated simultaneously by both programs performed better in terms of average wages than those that were only treated by CR Provee. This result is of special interest to policymakers since it indicates the importance of bundling in the implementation of PDPs. The findings suggest that policies aimed at overcoming the weaknesses of these two programs are important for obtaining higher real wages, generating more employment, and increasing the probability of exporting by Costa Rican SMEs.

Manajemen Isu dan Tantangan Masa Depan: Pendekatan Public Relations

Abstract : Issues Management is the management process whose goal is to help preserve markets, reduce risk, create opportunities and manage image as an organizational asset for the benefit of both an organization and its primary stakeholders. This is accomplished by: anticipating, researching and prioritizing issues; assessing the impact of issues on the organization: recommending policies and strategies to minimize risk and seize opportunities, participating and implementing strategy; evaluating program impact. Issues management both as a science and managerial practice has developed dynamically in the past three decades. This paper examines approaches to issues management as a science, corporate social responsibility as a pre-emptive policy of issues management, and issues management as skill for public relations practitioners.

Trachoma prevalence in Niger: results of 31 district-level surveys

The leading cause of preventable blindness worldwide is trachoma, a condition caused by an infection of the inner eyelid. In Niger, a landlocked republic in Western Africa, surveys in 1988-89 identified trachoma as endemic in all but one region and, as a result, there is a National Prevention of Blindness Program plan to eliminate trachoma by 2015. Thirty-one districts in eastern and western Niger were surveyed for trachoma prevalence from May 2009 to March 2012 as part of routine program impact evaluations. Prevalence surveys were implemented independently in each district using a two-stage cluster random design. Probability proportional to size was used to randomly select villages and 25 households were selected in each cluster. The prevalence of trachoma of clinical grade trachomatous follicular (TF) was estimated in children aged 1-9 years, and the prevalence of blinding trachoma, trachomatous trichiasis (TT), was measured in adults aged ≥15 years. A total of 14 211 households was surveyed; 58 617 individuals were evaluated for clinical signs of trachoma, of whom 27 087 were children aged 1-9 years. District-wide implementation of the full SAFE strategy is warranted in 16 districts where TF prevalence exceeds 10% and targeted implementation of the SAFE strategy (surgery for trichiasis; antibiotic therapy to control transmission; facial cleanliness for hygiene promotion; environmental change for improvements in access to water and sanitation) is recommended in the remaining 15 districts. The prevalence of TT among adults exceeded 1% in nine districts, suggesting that surgical services to treat TT should be implemented district-wide. These results establish the need for continued SAFE strategy implementation throughout Niger.

Evaluating Different Types of Enterprise Support Programs Using Panel Firm Data: Evidence from the Mexican Manufacturing Sector

(ProQuest: ... denotes formulae omitted.)In most countries, small and medium-sized enterprises (SMEs) make up the vast majority of firms, account for a substantial share of gross domestic product, and involve the bulk of the workforce. However, SMEs often lag behind larger firms in many performance dimensions. This lag is widely believed to result from constraints that SMEs face, including decreased access to finance, weak managerial and workforce skills, an inability to exploit scale economies in production, and imperfect information about market opportunities, new technologies, and methods of organization.In Mexico, microenterprises and SMEs make up 99 percent of firms, employ about 64 percent of the workforce, and account for more than 40 percent of GDP. Given the importance of SMEs in the economy, governments in Mexico over the past twenty years have established a wide variety of SME support programs. How effective these SME programs have been in achieving their objectives is unclear. In Mexico, impact evaluations of SME programs are rare. Most evaluations are qualitative in nature and narrow in scope, usually measuring beneficiary satisfaction with either support services or program coverage.This paper evaluates SME support programs in Mexico using a panel of firm-level data for two groups of firms-a treatment group that participated in SME programs and a control group that did not. The panel data have been created by linking SME program participation information to a large panel of annual industrial surveys (1994-2005) maintained by Mexico's National Institute of Statistics and Geography.The panel data provide an opportunity to address several issues that have plagued impact evaluations of SME programs in most countries, including Mexico. First, asking program participation questions in the firm survey allows the identification of firms that have never participated in any program and firms that have participated in different types of SME support programs such as business development services (BDS) or programs to support research and development (R&D).1 Second, the availability of multiple years of information on the characteristics and performance of participant firms-both before and after program participation-allows us to estimate the impacts of support programs that address selection biases arising from differences between the treatment and control groups in observable attributes and in unobserved heterogeneity. Our findings suggest that program participation in certain types of BDS and R&D support programs is associated with a higher value added per worker and increases in employment and exports. The positive impact associated with firm participation is strongest and most robust in the R&D support programs, which include the Sector Promotions Program (PROSEC) of the Ministry of Economy and the Fiscal Incentives and Technological Innovation (FITI) program of the National Science and Technology Council. These programs showed positive and statistically significant impacts on firm performance, ranging from increases of 9-13 percent on sales-and, in the case of PROSEC, increases of 12 percent in value added per worker and 10 percent in production per worker. Both R&D programs increased exports between 16 and 18 percent. The results also indicate that some outcomes, such as employment and value added per worker, showed positive effects only after the third or fourth year of program participation, although the effect increases as time goes on. As for BDS programs, we estimate positive effects on sales, fixed assets, and productivity measures from the National Environmental Audit Program (PNAA). We were unable to identify positive effects of this program on other outcomes or positive effects of the Comprehensive Quality and Modernization Program for Training the Industrial Workforce (CIMO-PAC).An Overview of SME Programs in MexicoThe Mexican federal government supports the development and competitiveness of Mexican firms by improving the overall business environment and through interventions to support individual firms or groups of firms, especially SMEs. …

Contemporary Social Policy Analysis Methods: An Incorporation of Ethical Principals and Implementation Processes

This article focuses on how policy analysts address complex social issues and problems by organizing them into ethic-based analytic components and utilizing an array of quantitative and qualitative methods, including program outcome and impact evaluation, cost-benefit analysis, and assessments of local political influence on implementation process. Four articles generated by doctoral students under the guidance of their faculty instructor demonstrate how the strategies of connecting social ethical values in policy analysis enlighten the valuation of the policy process, its history, and outcomes.

Evaluating Different Types of Enterprise Support Programs Using Panel Firm Data: Evidence from the Mexican Manufacturing Sector

Evaluating Different Types of Enterprise Support Programs Using Panel Firm Data: Evidence from the Mexican Manufacturing Sector Gladys López-Acevedo (bio) and Mónica Tinajero-Bravo (bio) In most countries, small and medium-sized enterprises (SMEs) make up the vast majority of firms, account for a substantial share of gross domestic product, and involve the bulk of the workforce. However, SMEs often lag behind larger firms in many performance dimensions. This lag is widely believed to result from constraints that SMEs face, including decreased access to finance, weak managerial and workforce skills, an inability to exploit scale economies in production, and imperfect information about market opportunities, new technologies, and methods of organization. In Mexico, microenterprises and SMEs make up 99 percent of firms, employ about 64 percent of the workforce, and account for more than 40 percent of GDP. Given the importance of SMEs in the economy, governments in Mexico over the past twenty years have established a wide variety of SME support programs. How effective these SME programs have been in achieving their objectives is unclear. In Mexico, impact evaluations of SME programs are rare. Most evaluations are qualitative in nature and narrow in scope, usually [End Page 1] measuring beneficiary satisfaction with either support services or program coverage. This paper evaluates SME support programs in Mexico using a panel of firm-level data for two groups of firms—a treatment group that participated in SME programs and a control group that did not. The panel data have been created by linking SME program participation information to a large panel of annual industrial surveys (1994–2005) maintained by Mexico’s National Institute of Statistics and Geography. The panel data provide an opportunity to address several issues that have plagued impact evaluations of SME programs in most countries, including Mexico. First, asking program participation questions in the firm survey allows the identification of firms that have never participated in any program and firms that have participated in different types of SME support programs such as business development services (BDS) or programs to support research and development (R&D).1 Second, the availability of multiple years of information on the characteristics and performance of participant firms—both before and after program participation—allows us to estimate the impacts of support programs that address selection biases arising from differences between the treatment and control groups in observable attributes and in unobserved heterogeneity. Our findings suggest that program participation in certain types of BDS and R&D support programs is associated with a higher value added per worker and increases in employment and exports. The positive impact associated with firm participation is strongest and most robust in the R&D support programs, which include the Sector Promotions Program (PROSEC) of the Ministry of Economy and the Fiscal Incentives and Technological Innovation (FITI) program of the National Science and Technology Council. These programs showed positive and statistically significant impacts on firm performance, ranging from increases of 9–13 percent on sales—and, in the case of PROSEC, increases of 12 percent in value added per worker and 10 percent in production per worker. Both R&D programs increased exports between 16 and 18 percent. The results also indicate that some outcomes, such as employment and value added per worker, showed positive effects only after the third or fourth year of program participation, although the effect increases as time goes on. As for BDS programs, we estimate positive effects [End Page 2] on sales, fixed assets, and productivity measures from the National Environmental Audit Program (PNAA). We were unable to identify positive effects of this program on other outcomes or positive effects of the Comprehensive Quality and Modernization Program for Training the Industrial Workforce (CIMO-PAC). An Overview of SME Programs in Mexico The Mexican federal government supports the development and competitiveness of Mexican firms by improving the overall business environment and through interventions to support individual firms or groups of firms, especially SMEs. Between 2001 and 2006, the Mexican government invested the equivalent of US$13 billion in approximately 3.7 million SMEs (World Bank 2010). Several agencies in Mexico provide SME support. The Ministry of Economy is first with regard to coverage...

Multifamily Power Partner Program Impact Evaluation

ABSTRACTPresented in this article is an evaluation of impacts for the multifamily Power Partner (PP) program. This demand response (DR) program is designed to reduce electrical system peak during periods of high grid demand through air conditioner curtailment in apartments. The multifamily sector is evaluated using both logger and automatic meter read (AMR) data. Impacts from the two methods indicate that the multifamily population trends to higher second hour impacts over the first hour of demand response in contrast to single-family and small commercial customers covered in previous internal Austin Energy (AE) studies. Estimated program curtailment values range from 3 MW to 4.5 MW below the current program estimate of 81 MW for the entire single-family and multifamily DR population. Control group method of impact evaluation is recommended as it is noted that extreme days, when curtailments are initiated, may have a different load profile than more moderate hot days used for baseline statistical analysis...

A comparison of missing data procedures for addressing selection bias in HIV sentinel surveillance data

BackgroundSelection bias is common in clinic-based HIV surveillance. Clinics located in HIV hotspots are often the first to be chosen and monitored, while clinics in less prevalent areas are added to the surveillance system later on. Consequently, the estimated HIV prevalence based on clinic data is substantially distorted, with markedly higher HIV prevalence in the earlier periods and trends that reveal much more dramatic declines than actually occur.MethodsUsing simulations, we compare and contrast the performance of the various approaches and models for handling selection bias in clinic-based HIV surveillance. In particular, we compare the application of complete-case analysis and multiple imputation (MI). Several models are considered for each of the approaches. We demonstrate the application of the methods through sentinel surveillance data collected between 2002 and 2008 from India.ResultsSimulations suggested that selection bias, if not handled properly, can lead to biased estimates of HIV prevalence trends and inaccurate evaluation of program impact. Complete-case analysis and MI differed considerably in their ability to handle selection bias. In scenarios where HIV prevalence remained constant over time (i.e. β = 0), the estimated derived from MI tended to be biased downward. Depending on the imputation model used, the estimated bias ranged from −1.883 to −0.048 in logit prevalence. Furthermore, as the level of selection bias intensified, the extent of bias also increased. In contrast, the estimates yielded by complete-case analysis were relatively unbiased and stable across the various scenarios. The estimated bias ranged from −0.002 to 0.002 in logit prevalence.ConclusionsGiven that selection bias is common in clinic-based HIV surveillance, when analyzing data from such sources appropriate adjustment methods need to be applied. The results in this paper suggest that indiscriminant application of imputation models can lead to biased results.

Using GIS to Evaluate a Fire Safety Program in North Carolina

Evaluating program impact is a critical aspect of public health. Utilizing Geographic Information Systems (GIS) is a novel way to evaluate programs which try to reduce residential fire injuries and deaths. The purpose of this study is to demonstrate the application of GIS within the evaluation of a smoke alarm installation program in North Carolina. This approach incorporates national fire incident data which, when linked with program data, provides a clear depiction of the 10years impact of the Get Alarmed, NC! program and estimates the number of potential lives saved. We overlapped Get Alarmed, NC! program installation data with national information on fires using GIS to identify homes that experienced a fire after an alarm was installed and calculated potential lives saved based on program documentation and average housing occupancy. We found that using GIS was an efficient and quick way to match addresses from two distinct sources. From this approach we estimated that between 221 and 384 residents were potentially saved due to alarms installed in their homes by Get Alarmed, NC!. Compared with other program evaluations that require intensive and costly participant telephone surveys and/or in-person interviews, the GIS approach is inexpensive, quick, and can easily analyze large disparate datasets. In addition, it can be used to help target the areas most at risk from the onset. These benefits suggest that by incorporating previously unutilized data, the GIS approach has the potential for broader applications within public health program evaluation.

Are Our Efforts Worthwhile? How to Improve Impact Evaluation of Programs, Policies, Interventions and Strategies Aimed at Promoting Healthy Lifestyles and Preventing Obesity in Canada?

Are Our Efforts Worthwhile? How to Improve Impact Evaluation of Programs, Policies, Interventions and Strategies Aimed at Promoting Healthy Lifestyles and Preventing Obesity in Canada?

The effects of free government health insurance among small children—evidence from the free care for children under six policy in Vietnam

Over the last few years, there have been an increasing number of impact evaluations of health insurance and other demand-side financing programs in developing countries. Yet the literature on insurance impact among small children is limited. This paper evaluates the effects of a Vietnamese government's policy in 2005, which granted free access to health services in public facilities to all children younger than 6 years. In particular, we focus on children among households who are not eligible for a program for the poor, which has been administered concurrently in the country. Using two waves of the Vietnam Household Living Standard Surveys conducted right before and after the policy started and a difference-in-differences method, we found a major increase in both inpatient and outpatient care in the secondary public hospitals. At the same time, there is evidence indicating a reduction in the use of tertiary hospitals. Compared with the policy's non-beneficiaries, beneficiaries in the age group 4-5 years also experienced fewer sick days, incurred less out-of-pocket spending on healthcare, and were less likely to encounter catastrophic expenditure. Evidence thus suggests that insurance provided by the policy has served the function as a safety net and helped improving efficiency of the health system by reducing the use of costly tertiary care.

Use of Nutritional Biomarkers in Program Evaluation in the Context of Developing Countries ,

The Lancet series on maternal and child undernutrition emphasized the need for accurate and reliable biomarkers that reflect nutrient status and measure the impact of interventions. An initiative called Biomarkers of Nutrition for Development (BOND) by the Eunice Kennedy Shriver National Institute of Child Health and Human Development aims to provide guidance for the selection and interpretation of biomarkers that meet a range of interests, including research, clinical policy, and program development. This article summarizes the activities of the program working group of the BOND initiative. The working group specified biomarkers according to program objectives such as assessing the nutritional situation or status of target populations/areas; monitoring progress of intervention; and evaluating program impact. In addition, the biomarkers developed were required to be feasible in the field settings. Based on these considerations, population-based biomarkers for programs are proposed for case examples of vitamin A, folate, vitamin B-12, iron, and zinc. Biomarkers of underlying infection/inflammation, anthropometric measures of growth, and dietary intake are recommended to be included. A program manager guide and future research to develop biomarkers for program context are recommended.

PROTOCOL: Nutrition interventions and programs for reducing mortality and morbidity in pregnant and lactating women and women of reproductive age: a systematic review

PROTOCOL: Nutrition interventions and programs for reducing mortality and morbidity in pregnant and lactating women and women of reproductive age: a systematic review

American Indian Adolescent Girls: Vulnerability to Sex Trafficking, Intervention Strategies

The Minnesota Indian Women's Resource Center offers harm reduction programming to at-risk adolescent American Indian girls, including outreach, case management, advocacy, healthy sexuality education, and support groups. To evaluate program impact, participants are assessed at intake and every 6 months afterward for current vulnerability to commercial sexual exploitation, violence, and addiction. Evaluation results indicate frequent exposure to sex traffickers and suggest that harm reduction methods can help girls reduce risk of commercial sexual exploitation.

Program impact evaluation using a matching method with panel data

Difference-in-differences with matching is a popular method to measure the impact of an intervention in health and social sciences. This method requires baseline data, that is, data before interventions, which are not always available in reality. Instead, panel data with two time periods are often collected after interventions begin. In this paper, a simple matching method is proposed to measure the impact of an intervention using two-period panel data after the intervention. The method is illustrated by the measurement of the effect of health insurance in Vietnam using household panel data.