Impact Evaluation Methods Research Articles

Reassessing the Impact of Financial Services on Urban Manufacturing Firms: An Impact Evaluation Method in Ethiopia

The stock of financial literature reveals emerging and conflicting stands on the effect of finance on the performance of the economy. The previous thoughts considered finance into account as an important driving force to growth through its role in intermediation, reduction of transaction costs and risk, and efficient uses of resources. Nevertheless, the newly emerging thoughts focus on the vanishing effect of finance due to its stiff competition over resources with the rest of the economy. The dialogues also reflected in the process of industrialization and promoting the manufacturing sector. Therefore, the general objective of this study is to reexamine the impact of financial services on the performance of manufacturing firms in Ethiopia with a special focus on firms in Addis Ababa. The study used propensity score matching method. The result shows that those who access finance has increased their operating margin profit by 2.6 on average in comparison with the non-treated groups. The treated groups have 0.42 greater net return on a net asset than non-treated groups on average. Therefore, the study suggests that financial institutions should increase their involvement to expand the accessibility of financial products to manufacturing firms that are the expected engines of sustainable growth and economic transformation.

Irrigation infrastructure and farm productivity in the Philippines: A stochastic Meta-Frontier analysis

Abstract Irrigation is a lynchpin of rural development strategies and a key input to improving productivity and farm incomes, the key source of livelihood for the majority of the world's poor. Limited land and growing water scarcity mean that establishing systems to maximise the benefits from every drop is pivotal. In this paper, we analyse the impact of a canal irrigation project for smallholders in the Philippines, focusing on rice, one of the world's most water intensive crops. We contribute to two strands of literature by combining impact evaluation and efficiency analysis methods. Using a dataset for 714 treatment and 440 control farm parcels, we apply Propensity Score Matching and a selectivity-corrected Stochastic Production Frontier to handle biases from both observable and unobservable variables. We then analyse technical efficiency (TE) and frontier output using a shared Stochastic Meta-Frontier. We find that the project had a statistically significant impact on frontier output but not on TE, suggesting that improved irrigation technology increased beneficiaries' production potential, but it did not improve TE likely due to insufficient training and input access. Thus, beneficiaries were unable to take full advantage of their improved production potential, highlighting the need for suitable complementary support in future projects. Heterogeneity analysis reveals that the main beneficiaries were downstream parcels, smaller parcels, those located in the poorer project district, and farmers with lower education, all implying a pro-poor impact. Finally, we find that female-headed households benefitted less from the project, suggesting the need for additional support in future interventions.

Screening for internal defects of timber members based on impact evaluation method

This paper proposes a screening method and quantitative criteria for screening the internal defects of ancient timber members. Based on the impact evaluation (IE) method, two parameters, i.e. time of impact (TI) and proportion of integral area (PIA), were used to assess the internal defects. Impact tests were also conducted to study the effects of different influencing factors on the parameters (TI and PIA). The results demonstrated the satisfactory screening ability of PIA when the optimal proportion for the boundary value (fb) in the frequency domain was set as 50%. TI had a strong correlation with both the length of internal defects (ld) and the surface hardness (R2 = 0.74 and 0.93); however, PIA only had a significant correlation with ld (R2 = 0.84). The validation test results revealed that the proposed method and quantitative criteria could be used to detect the internal defects of ancient timber when the distance between internal defects and the surface (ha) was less than 35 mm.

Cluster development policies and firms’ performance: evidence from an emerging economy in Latin America

PurposeImpact evaluations of cluster programs at firm level are still scarce in the literature. The available evidence on the effectiveness of such programs based on rigorous quantitative impact evaluations is mixed. The purpose of this paper is to contribute to the body of literature the evaluates quantitatively the impact of cluster programs in emerging economies on firms’ performance. In particular, the authors evaluate the impact of a cluster program in Uruguay on firms’ sales and exports.Design/methodology/approachThe authors use state-of-the-art impact evaluation methods to evaluate the impact of the program. In particular, difference in differences and matching methodsFindingsThere is very strong evidence that the program had a positive impact on exports and the propensity to export of firms. However, the evidence of a positive impact on sales is weak. The evidence suggests that the maximum effect of the program can be found in the fourth or fifth year after the intervention.Originality/valueThe contribution of this paper to the literature is fourfold. First, this paper adds to the scarce body of literature evaluating the effects of cluster development programs with state-of-the-art impact evaluation methods. Second, it adds evidence for Latin America, a region that has implement a number of cluster policies (Maffioli et al., 2016) and where, as far as the authors know, there is only one additional paper evaluating rigorously the impacts of them (Figal-Garone et al., 2015). In addition, the authors provide evidence about the timing of the effects after the implementation of a cluster policy, an important issue that is mostly overlooked in the existent literature. Finally, the paper focus its attention on the impacts on exports and the propensity to export of firms, key elements for small open economies in Latin America that are heavily reliant on foreign currency inflows.

The Impact of Military Downsizing on Two Italian Communities: A Counterfactual Approach Using the Synthetic Control Method

ABSTRACT In this paper, we propose a quasi-experimental approach to assess the effect that downsizing of the largest experimental military base in Europe, located in Sardinia, Italy, had on local economies. The study shows the fruitfulness of the Synthetic Control method for assessment of socioeconomic impacts on a single treated unit when other standard methods for impact evaluation are not feasible. It is shown that the local communities analysed have been characterised by different levels of frailty, which should be attentively considered if further downsizing of the base will be placed on the political agenda.

Economics of environmental effects on health: A methodological review based on epidemiological information

Abstract Economic analysis of environmental effects on health in terms of use and non-use value opens analytical avenues of evaluating criteria, further beyond the results of epidemiological analysis. From the methodological review of the papers, it is inferred that scanty literatures are partially devoted to yield some use value or facts from epidemiology to economic impact evaluation method, basically taking either temperature or humidity and climate change. Based on the methodological issues available in the literatures, this paper critically explored a state-of-the-art-review of methodology with their gaps for the establishment of a new model methodology, incorporating a strong economic valuation procedure for use and non-use values of environmental health effects which creates many rooms for innovative economic evaluation methods beyond the epidemiological results. The demonstration of a new model methodology is able to moisturize the thrust of researchers in quest of a complete methodology for the economic analysis of effects of environmental change on human health.

Endline Report for the Self-Supporting Rural Development Project with Saemaul Undong's Participatory Approach in Myanmar

The Self-Supporting Rural Development Project with Saemaul Undong's Participatory Approach (hereafter SMU) in Myanmar, implemented jointly by Korea International Cooperation Agency (KOICA) and the Ministry of Agriculture, Livestock, and Irrigation (MOALI) from 2014 to 2019, is the largest rural development project in Southeast Asia implemented by KOICA. Excluding the budget for the construction of the Saemaul Undong Training Institute in Myanmar, the project is worth more than 10 billion Korean won (equivalent to USD 9 million), which includes the budget for the implementation of rural community development projects in 100 pilot villages across nine regions in Myanmar over three years. As the scale of KOICA's development projects has expanded dramatically, and projects with large budgets of over 10 billion won, including Myanmar's rural community development projects, have increased significantly, there have been increasing discussions on how to evaluate various projects implemented by KOICA. The evaluation method used by KOICA can be described as Process Evaluation using Project Design Matrix(PDM), which summarizes the logical framework based on the Theory of Change. While the PDM evaluated in the order of inputs, activities, outputs, outcomes, and impacts has the advantage of being particularly useful for process evaluation, it assumes a causal relationship between inputs/activities and outcomes/impacts and thus cannot be scientifically proven. In an effort to address these limitations of process evaluation, impact evaluation method focused on identifying causal relationships has received increasing attention. Abhijit Banerjee (University of MIT), Esther Duflo (University of MIT), and Michael Kremer (University of Harvard) have conducted impact evaluations applying a randomized controlled trial (RCT) to various development projects since the mid-1990s. They have also received the 2019 Nobel Prize in Economics for their contributions to poverty alleviation and development policies based on scientific evidence derived from impact evaluation findings. National aid agencies such as USAID and DFID, and inter-governmental organizations such as the World Bank and the Asian Development Bank have been actively conducting impact evaluations of their development projects. In particular, the World Bank has established the Development Impact Evaluation Initiative (DIME) in order to evaluate their major projects. KOICA also carried out an impact evaluation, along with process evaluation, when implementing the Myanmar rural community development project in line with these international evaluation trends. The Myanmar Rural Development Project Impact Evaluation Study is meaningful in that it is the first large-scale impact evaluation research conducted by KOICA.3) The rest of the report is as follows. Chapter 2 summarizes the background of the impact evaluation of the KOICA Myanmar Rural Development Project, Chapter 3 discusses the impact evaluation methodology, and Chapter 4 analyzes the main results of the impact evaluation. Finally, in Chapter 5, we provide recommendations for policymakers who are planning to implement a large-scale impact evaluation study such as the Myanmar rural development project. The summary of the impact evaluation results of the Myanmar rural development project is as follows. There was no statistically significant impact on social capital, which was set as one of the main outcome variables, but the Myanmar rural development project improved the living environment in the village, increased access to microfinance, and increased income. Since a randomized controlled trial, comparing randomly selected treatment villages to control villages, was not possible for this impact evaluation study, two behavioral experiments were implemented. KOICA's rural development project in Southeast Asia reflects the experience of Saemaul Undong in Korea, and includes a differential incentive structure that pays incentives to villages with better performance through competition among project villages. The behavioral experiments attempted to analyze whether the inter-village competition, a key mechanism of the KOICA rural development projects, affected the increase of social capital in the villages. In Myanmar, where the level of social capital was remarkably high from the baseline, we could not observe any effect of the inter-village competition from the behavioral experiments.

Impact of Tanzania’s Wildlife Management Areas on household wealth

Large-scale area-based conservation measures affect millions of people globally. Understanding their social impacts is necessary to improve effectiveness and minimize negative consequences. However, quantifying the impacts of conservation measures that affect large geographic areas and diverse peoples is expensive and methodologically challenging, particularly because such evaluations should capture locally defined conceptions of well-being while permitting policy-relevant comparisons. Here, we measure the impact of Tanzania’s Wildlife Management Areas (WMAs), a national community-based conservation and poverty reduction initiative. We use a novel, cost-effective impact evaluation method based on participatory wealth ranking and Bayesian multilevel modelling. We find that from 2007 to 2015 the impacts of WMAs on wealth were small and variable, with no clear evidence of widespread poverty reduction. Accompanying qualitative data suggest that apparently positive effects in one WMA cannot be directly attributed to WMA activities. Our results suggest that current WMA policy needs to be revisited if it is to promote positive local development. A novel method to assess the impacts of large-scale conservation interventions on household wealth while capturing local heterogeneity reveals small and variable impacts of a nationwide conservation programme in Tanzania.

The impacts of agroforestry on agricultural productivity, ecosystem services, and human well-being in low-and middle-income countries: An evidence and gap map.

The impacts of agroforestry on agricultural productivity, ecosystem services, and human well-being in low-and middle-income countries: An evidence and gap map.

Inférence causale et évaluation d'impact

[eng] This paper describes, in a non technical way, the main impact evaluation methods, both experimental and quasi experimental, and the statistical model underlying them. In the first part, we provide a brief survey of the papers making use of those methods that have been published by the journal Economie et Statistique / Economics and Statistics over the past fifteen years. In the second part, some of the most important methodological advances to have recently been put forward in this field of research are presented. To finish, we focus not only on the need to pay particular attention to the accuracy of the estimated effects, but also on the requirement to replicate evaluations, carried out by experimentation or quasi experimentation, in order to distinguish false positives from proven effects.

Predictive Counterfactuals for Event Studies with Staggered Adoption: Recovering Heterogeneous Effects from a Residential Energy Efficiency Program

This paper introduces an approach for estimation of treatment effect heterogeneity in event studies with staggered adoption. Traditional impact evaluation methods can be near-term biased in those settings. The proposed approach attenuates biases and recovers heterogeneity more efficiently than traditional methods. It is shown that machine learning algorithms can be used to accurately predict counterfactuals, which can then be used to estimate a distribution of treatment effects. Simulations demonstrate how that approach can be accurate and efficient, even in the presence of dynamic treatment effects. The paper concludes with an application to a large energy efficiency program in the US.

Estimating inter-regional payments for ecosystem services: Taking China’s Beijing-Tianjin-Hebei region as an example

The inter-regional payments for ecosystem services (PES) is an important policy to promote regional ecological and environmental cooperation. However, the existing inter-regional PES standard in China may underestimate the value of ecosystem services and omit the value of transferred pollutants due to inter-regional trade. A reasonable framework of inter-regional PES standards is necessary for the policymakers, especially in Beijing-Tianjin-Hebei (BTH) region with the most serious environmental problem in China. This paper employed the method of environmental impact evaluation and multi-regional input-output model (MRIO) to build a framework of inter-regional PES estimation, which both contained the regional spillover value of ecosystem services and the regional transfer value of pollutants. Finally, we estimate the amount of inter-regional payment for ecosystem services within BTH region. The results indicated that the PES standards within BTH region were 13.8 billion yuan and 19.2 billion yuan from Beijing, Tianjin to Hebei province in 2012, which accounted for 0.77 % and 1.49 % of their GDP in 2012. These PES standards are effective for regional ecological and environmental cooperation within BTH region and a multi-dimensional marketization mechanism should be implemented to reinforce inter-regional payment for ecosystem services, which may pave the way for other regions or countries.

Impact evaluation using Difference-in-Differences

Purpose This paper aims to present the Difference-in-Differences (DiD) method in an accessible language to a broad research audience from a variety of management-related fields. Design/methodology/approach The paper describes the DiD method, starting with an intuitive explanation, goes through the main assumptions and the regression specification and covers the use of several robustness methods. Recurrent examples from the literature are used to illustrate the different concepts. Findings By providing an overview of the method, the authors cover the main issues involved when conducting DiD studies, including the fundamentals as well as some recent developments. Originality/value The paper can hopefully be of value to a broad range of management scholars interested in applying impact evaluation methods.

Nature documentaries and saving nature: Reflections on the new Netflix series Our Planet

Abstract Netflix recently launched its high‐profile nature documentary Our Planet. Voiced by Sir David Attenborough in English (with Salma Hayek, Penelope Cruz and other Hollywood actors voicing versions simultaneously released in 10 other languages), Netflix are making a clear play for core BBC territory. However, they claim that this is a nature documentary with a difference as it puts the threats facing nature front and center to the narrative. We coded the scripts of Our Planet, and those of three recent Attenborough‐voiced BBC documentaries, to explore the extent to which threats (and conservation action and success) are discussed. The only other series which comes close to the frequency with which these issues are discussed is Blue Planet II, but Our Planet is unique in weaving discussion of these issues throughout all episodes rather than keeping them to a dedicated final episode. However, although Our Planet sounds different to other documentaries, the visuals are very similar. Nature is still mostly shown as pristine, and the presence or impacts of people on the natural world very seldom appear. We discuss the potential consequences of nature documentaries erasing humans from the land/seascape. We also discuss the mechanisms by which nature documentaries may have a positive impact on conservation. Despite links between information provision and behaviour change being complex and uncertain, nature documentaries may, at least in theory, elicit change in a number of ways. They may increase willingness amongst viewers to make personal lifestyle changes, increase support for conservation organizations, and generate positive public attitudes and subsequently social norms towards an issue, making policy change more likely. Netflix is certainly bringing biodiversity and the threats it faces into the mainstream, but the mechanisms by which viewing these representations translates to concrete behaviour change are poorly understood. Increasing interest in robust impact evaluation, integrating qualitative and quantitative methods, means the time is right to explore how both showing nature on screens and talking about the threats it faces, affects people in ways which might, ultimately, contribute to saving it. A free Plain Language Summary can be found within the Supporting Information of this article.

Effectiveness of psychodrama for mitigating school fears among senior secondary school students

This study is focused on the effectiveness of psychodrama for mitigating school fears among senior secondary school students. The study involved 224 students from senior forms (ninth- to eleventh-graders). The efficiency of psychodrama was measured by creating two groups consisting of ninth- and eleventh-graders: the students of the target group (participants of the continuous psychodrama sessions, n = 61) and the students of the comparative group (who did not take part in the continuous psychodrama sessions, n = 163). Two instruments were employed for the purposes of this study: impact method and evaluation method. In order to reduce the manifestations of school fears among secondary school students, psychodrama was used as an impact method. The impact consisted of four continuous psychodrama sessions, each lasting around four academic hours (i.e. group exercises). The differentiated school fear descriptor (DSFD) was used (Rost and Schermer 1997) in order to reveal whether school fears of senior secondary school students changed after the psychodrama sessions. Two questionnaires were used for the purposes of the study: manifestations of school fears (MA) and strategies for coping with school fears (CO). The study results showed that the manifestations of emotional and cognitive school fears among the participants of the target group decreased after the psychodrama sessions; for overcoming school fears, the students of the target group started engaging in productive working activity more often; the students of the target group stated that they experienced fewer emotional and cognitive manifestations of school fears, compared to the students of the comparative group; the students of the target group started using productive working activity for coping with school fears more often, as compared to the students of the comparative group.

In-kind conservation payments crowd in environmental values and increase support for government intervention: A randomized trial in Bolivia

There is growing use of economic incentives such as Payments for Ecosystem Services (PES) to encourage sustainable land management. An important critique is that such approaches may unintentionally disrupt environmental and social values, ‘crowding out’ pre-existing motivations to conserve. Some scholars suggest that the use of in-kind payments and norm-based framing, rather than financial transfers and a market framing, can mitigate these risks. There are calls to use more robust methods for impact evaluation in environmental policy. We use one of the only Randomized Controlled Trials of a conservation incentive scheme to evaluate its impact on self-stated environmental and social values and beliefs. Data from before and after the intervention, from households in villages randomly selected to receive the program or not, demonstrate that the program increased prioritization of environmental values (evidence of crowding-in as opposed to crowding out) and altered social beliefs related to inequality and the role of government. The findings demonstrate that this conservation program had a positive impact on environmental values and increased the belief that government involvement is appropriate. The scheme, with its use of in-kind payments and reciprocity framing, offers lessons to those seeking to develop effective schemes to incentivize positive environmental stewardship.

Electromagnetic resilience of critical national infrastructure

With the increase in the system scale and informatization level of the critical national infrastructures, the electromagnetic security in extreme electromagnetic environments such as the high-altitude electromagnetic pulse, the intentional electromagnetic interference and the geomagnetic storm has attracted more attention. Unlike conventional electromagnetic events such as the lightning and the overvoltage in power systems, extreme electromagnetic environments are small-probability but high-risk events whose impact mechanisms and evaluation methods are quite different. And the conventional reliability analysis, which adopts the expected value indexes, is difficult to effectively evaluate and manage the risks related to extreme electromagnetic environments. In this context, this paper presents a triangular pyramid model for the study of electromagnetic security of critical infrastructures. Then taking the power grid as an example, it discusses the implication and significance of electromagnetic resilience to critical infrastructures, and finally makes a proposal for the future study of electromagnetic resilience.

Measuring the Change Towards More Sustainable Mobility: MUV Impact Evaluation Approach

Urban areas can be considered the ground for the challenges related to the UN’s Sustainable Development Goals (SDGs). The objective of shaping cities as human settlement that will see a more inclusive, safe, resilient, and sustainable future is often argued in literature as an issue dependent on behavioral change of inhabitants in urban areas. In this paper, the authors question if experimental applications based on gamification can co-produce more sustainable neighborhoods through an impact evaluation method that departs from individual choices within the complex of urban mobility. This investigation is carried out within MUV (Mobility Urban Values), an EU research and innovation project, which aims to trigger more sustainable urban mobility in six pilot cities. This article describes the critical method of validation, an impact assessment of the MUV experimental gamification in the pilot cities, in order to represent a proof for future urban strategies. This methodological approach is based on an evaluation structured on indicators of both impact and process suitable for urban contexts. As based on six pilot cities, with possibilities for transferability to other contexts and scalability to other cities, the method represents a reference work for the evaluation of similar experimental applications.

Explaining long-term outcome trajectories in social-ecological systems.

Improved knowledge of long-term social and environmental trends and their drivers in coupled human and natural systems is needed to guide nature and society along a more sustainable trajectory. Here we combine common property theory and experimental impact evaluation methods to develop an approach for analyzing long-term outcome trajectories in social–ecological systems (SESs). We constructed robust counterfactual scenarios for observed vegetation outcome trajectories in the Indian Himalaya using synthetic control matching. This approach enabled us to quantify the contribution of a set of biophysical and socioeconomic factors in shaping observed outcomes. Results show the relative importance of baseline vegetation condition, governance, and demographic change in predicting long-term ecological outcomes. More generally, the findings suggest the broad potential utility of our approach to analyze long-term outcome trajectories, target new policy interventions, and assess the impacts of policies on sustainability goals in SESs across the globe.

An intelligence evaluation method of the environmental impact for the cutting process

Environmental impact evaluations in manufacturing fields is a complicated decision-making problem, which involves wastes emission, resource consumption and energy utility, etc. This paper presents an intelligence method for environmental impact evaluation using kernel fuzzy clustering and a back-propagation neural network. The objective of this article is to apply artificial intelligence technology to evaluate environmental influence of cutting process and develop a decision support tool for selecting the optimal environmental solution from various alternative schemes. There are three stages to accomplish this assessment procedure. First, the evaluation index system of the cutting process was analyzed and established. Next, a training sample set which was used for learning the ‘knowledge and experience’ by neural networks, was acquired by kernel fuzzy clustering algorithm. Finally, an evaluation model based on a back-propagation neural network was constructed, and connection weights of the model were determined after training. A case research on the cutting processes of the automobile workpiece was conducted. Results showed that the proposed method is more concise and practicable than existing evaluation methods and provide a feasible and effective decision-making tool for selecting an optimal cutting process while manufacturing.