Framework For Policy Evaluation Research Articles

A Robust Framework for Renewable Energy Policy Evaluation Using MCDA and Compromise Ranking with Stochastic Weight Identification

A Robust Framework for Renewable Energy Policy Evaluation Using MCDA and Compromise Ranking with Stochastic Weight Identification

Evaluating Ecosystem Services Incorporation in River Policy: A Multidimensional Framework Applied to China's River Chief System

ABSTRACTThis study develops a comprehensive evaluation framework to assess the incorporation of river ecosystem services (ESs) into policy content, focusing on China's River Chief System (RCS) policy. The framework introduces a quantitative tool that bridges the gap between policy formulation evaluation and policy effect evaluation by integrating a hierarchical indicator system, reference values, and a method for quantifying ES incorporation. Key findings from 32 province‐level RCS policy documents revealed selective incorporation of certain ES indicators, significant mismatches between policy focus and the typical structure of river ESs, and notable regional disparities. Provisioning services (i.e., water supply) and regulating services (i.e., pollution mitigation) are prioritized, while critical regulating services (e.g., flood mitigation and habitat maintenance) and cultural services (e.g., education and science) are underrepresented. Stakeholder and relationship indicators are limited, primarily focusing on managers and spatial relationships, while the value of river ESs is largely ignored. Policy optimization should emphasize expanding the coverage of ESs, incorporating broader stakeholder involvement, and integrating financial incentives. Drawing on international river policies, policymakers could create a more flexible and adaptive river governance structure. While the study offers a quantitative framework for policy evaluation, it acknowledges limitations, including reliance on textual data and potential biases in the reference values. Future research should include stakeholder perspectives and dynamically adjust reference values as new data become available.

Toward a citizen science framework for public policy evaluation: Lessons from a field experiment on Freedom of Information laws in Belgium

This study pioneers the use of citizen science in evaluating Freedom of Information laws, with a focus on Belgium, where since its 1994 enactment, Freedom of Information’s effectiveness has remained largely unexamined. Utilizing participatory methods, it engages citizens in assessing transparency policies, significantly contributing to public policy evaluation methodology. The research identifies regional differences in Freedom of Information implementation across Belgian municipalities, highlighting that larger municipalities handle requests more effectively, while administrations generally show reluctance to respond to requests from perceived knowledgeable individuals. This phenomenon reflects a broader European caution toward well-informed requesters. By integrating citizen science, this study not only advances our understanding of Freedom of Information law effectiveness in Belgium but also advocates for a more inclusive, collaborative approach to policy evaluation. It addresses the gap in researchers’ experience with citizen science, showcasing its vast potential to enhance participatory governance and policy evaluation.

Comparative analysis of school food policies in Australian jurisdictions: Insights for more effective policy action.

School food policies in Australia have the potential to positively impact healthy eating in schools. While the policies are reviewed regularly, it is unknown if revised school food policies have incorporated the recommendations from a 2020 review of nutrition policies in publicly funded institutions. This review aims to examine the current state of Australian school food policy environment, including national-level guidelines, food safety, allergy, and anaphylaxis management policies. School food policy documents were identified by searching the federal government, and Departments of Health and Education websites of Australian jurisdictions. The L'Abbé framework for monitoring and evaluation of nutrition policies guided the data analysis. Eleven policies/guidelines from Australian national and state/territory governments were reviewed in 2023. Four state/territory policies had major updates since 2020. The consistency of nutrition standards improved across five jurisdictions, although a common nutrient classification system would have allowed better comparison. Implementation guides/tools and supporting resources were provided in all policies although their comprehensiveness varied. Policy monitoring and evaluation guide/tools, and local food procurement were incorporated in two additional policies. School principals were responsible for policy implementation and schools for independent compliance monitoring. Continued improvement in monitoring and evaluation plans, policy review timelines, and support systems for schools is needed to enhance implementation and impact of school food policies. External stakeholder support may help school leadership in more effective policy implementation. SO WHAT?: Limited progress in incorporating recommendations into current school food policies calls for additional support and strategies to enhance policy implementation and monitoring.

Policy mixes for just transitions: A holistic evaluation framework

In this paper, we develop a holistic policy evaluation framework that aims to harness a fuller potential of just transitions. Although appeals for broader understanding of just transitions are becoming louder, applicable frameworks supporting consideration of justice in the planning and evaluation of transition policies are still lacking. The evaluation framework developed integrates the multidimensional understanding of justice more consistently into the consideration of policy intervention points for sustainability transitions. We test and apply the framework to the discussions concerning just food system transitions in Finland. The results show that justice issues deserve attention across all the policy intervention points, but their relative importance differ. A better understanding of the relative importance of justice concerns is critical when designing policy mixes for more active and emancipatory just transitions governance. We close the paper by explicating further avenues for the application and development of the framework.

Sustainable Stormwater Management for Different Types of Water-Scarce Cities: Environmental Policy Effect of Sponge City Projects in China

With high-speed urbanization, ecological space is seriously shrinking, and lagging drainage facilities contradict the ecological needs of citizens. In particular, water-scarce cities are faced with frequent stormwater disasters, such as excessive accumulation of rainwater, peak runoff and water pollution, which threaten the safety of the urban water ecological environment. This paper combined the actual construction content of the sponge city project with a whole process policy evaluation framework to examine whether the projects solve these problems and to find different approaches to the results. Utilizing entropy fuzzy comprehensive evaluation provides a systematic standard for the evaluation system. The research shows that the sponge city project can achieve a good governance effect, including constructing a suitable scheme for urban hydrological characteristics, effectively improving the rainwater treatment level of different types of water-scarce cities, and alleviating the ecological contradiction of urban water environment. The stages of policy formulation, policy implementation and policy results achieve a good degree of completion. On one hand, sponge city projects transform the infrastructure at key locations, aiming at improving the rainwater interception capacity of the streets; on the other hand, restoring original natural waters improves the capacity of water conservation and forms a sustainable ecosystem between the city and nature.

Promoting Sustainable Development in Urban–Rural Areas: A New Approach for Evaluating the Policies of Characteristic Towns in China

The Sustainable Development Goals, adopted by all United Nations Member States, emphasize sustainable cities and communities, aiming to strengthen development planning and foster positive links between urban and rural areas by 2030. As the newly leading platform for urbanization development, characteristic towns (‘Tese Xiaozhen’ in Chinese) contribute to the target of sustainable urbanisation, wherein town policies are especially important for leading the sound development of characteristic towns. However, the effect of these characteristic towns policies remains uncertain. Additionally, most related studies have utilized qualitative methodologies in policy evaluation, which may be inadequate to guide actual practice. Thus, in order to accurately acknowledge the effect of characteristic towns policies, a policy evaluation framework is established in this paper, utilising the multiple streams theory, text mining and the Policy Modelling Consistency Index model (PMC-Index model). Then, taking 225 policy documents promulgated in China from 2015 to 2022 as the research objects, policy evaluation indicators were selected by combining the multiple streams theory to improve the PMC-Index model and a text mining method. Thereafter, an empirical analysis was conducted to evaluate the consistency of 10 characteristic towns policies chosen. The results indicate that the mean value of the PMC indexes of the 10 examined policies is 7.13 in the total of 9 points, which means that the general performance of the characteristic town policies is satisfactory, yet the low scores in terms of effectiveness, content, and guarantee. In the end, some recommendations and suggestions are proposed for clarifying administrative attributes, optimizing the integration of urban and rural resources, and adjusting the policy hierarchy to a bottom-up approach. This study not only provides a policy evaluation framework to comprehensively understand the consistency of global urbanization policies but also offers a beneficial reference for promote urban–rural sustainable development.

Agricultural Worker Perspectives on Climate Hazards and Risk Reduction Strategies

ABSTRACT Objective The purpose of this study was to understand the experiences of agricultural workers during periods of heat and wildfire smoke exposure and to support the development and implementation of protective workplace interventions. Methods Using community-engaged research and the Center for Disease Control (CDC) framework for policy evaluation, a qualitative descriptive study was conducted with current and former agricultural workers in Central Washington (WA). Twelve participants answered semi-structured questions via interviews or by attending a focus group. Interviews and focus groups were conducted in Spanish, recorded, transcribed, and translated into English; one interview was conducted in English. Results Using Braun and Clarke’s Reflexive Thematic Analysis, five themes were identified among workers from various worksites: 1) Extreme weather and working conditions are becoming increasingly hazardous to worker health, 2) Employers and supervisors lack training and education on current labor laws, and health and safety rules, 3) Employers and supervisors use intimidation and retaliation to ensure productivity and to evoke feelings of replaceability among workers, 4) Workers do not trust regulatory agencies to enforce rules or hold employers accountable, 5) Solutions to climate-driven problems in the agricultural industry need to value worker health and safety, not just productivity. Participants reported experiencing adverse health symptoms related to heat and smoke exposure at work. Workers proposed solutions including improving education, training, and communication, and increased enforcement of existing and forthcoming occupational health and safety rules. Conclusion The agricultural workforce is essential for ensuring a robust food supply and is facing extreme weather events due to climate change. Western states impacted by wildfires and heat are working to develop and implement occupational health and safety rules. Developing effective policies and interventions inclusive of worker perspectives is critical to adapt to a changing climate, retain a stable workforce and promote optimal health.

DEVELOPMENT OF SELF-ASSESSMENT TOOLS FOR ORGANIZATIONS: PREVENTION AND DELAY OF DISABILITY PROGRAMS IN TAIWAN

Abstract This study aimed to develop a tool for service organizations to self-assess their characteristics and abilities of providing better disability prevention and function maintenance services. The development of the tool involved two stages. In the first stage, a self-assessment tool with 40 items was developed through literature based on Dunn’s policy evaluation framework. The tool was then subjected to two rounds of Delphi method, and pilot test. Six experts were invited, and the Delphi method results showed high consensus. Pilot test invited 170 service organizations to complete the survey and the results showed good internal consistency and validity with 20 items and six dimensions, including internal efficacy, responsiveness, professional, community collaboration, environmental factors, and sustainability (χ2[df=171, N = 170] = 282.88; p < 0.001; CFI = 0.923; RMSEA = .071; SRMR=.055). The sustainability (mean:4.20, SD: 0.61) and responsiveness (mean:4.20, SD: 0.59) showed the highest self-assess ratings. Combining internal resources and efficacy as internal efficacy showed good validity. and together with a new domain “internal efficacy” showed strong association with other domains. This tool could provide the program agencies to self-assess their ability to provide better disability prevention and function maintenance services. There were two new concepts, internal efficacy and sustainability, identified through the process which may indicate both of these two concepts are important to organization that provide health promotion services and culture-sensitive in Taiwanese society. Promoting these two concepts for future organizations is recommended.

Proposal of policies based on temporal-spatial dynamic characteristics and co-benefits of CO2 and air pollutants from vehicles in Shanghai, China

In megacities, vehicle emissions face urgent challenges related to air pollution and CO2 control. To achieve the refinement of vehicle control policies for the co-control of air pollutants and CO2, this study established a vehicle emission inventory with high spatial and temporal resolution based on the hourly traffic flow in Shanghai and analyzed the spatial and temporal distribution characteristics of the real-time vehicle emissions. Meanwhile, a policy evaluation framework was constructed by combining pollutant emission predictions with quantitative co-control effect assessments. The results indicated that spatio-temporal variations in different air pollutants and CO2 could mainly be attributed to primary contributing vehicle types. The pollutants (CO2, CO and VOCs) primarily contributed by private cars exhibited a bimodal pattern in 24-h time series and their spatial distribution was concentrated in the urban city center. The spatial distribution of NOx and PM primarily contributed by heavy trucks was still obvious on non-urban center areas. Furthermore, the results of synergistic effect analysis revealed that the alternative energy replacement scenario demonstrated the most significant potential for the co-control. Based on temporal-spatial and co-benefit analysis, the precise control policy of vehicle emissions can be established through time-, region-, and model-control. This study provides references and research methods for the formulation of the vehicle refinement control policies in worldwide megacities.

Baseline Ethical Principles and a Framework for Evaluation of Policies: Recommendations From an International Consensus Forum.

This Forum was initiated by Transplant Québec and co-hosted by the Canadian Donation and Transplantation Program partnered with multiple national and international donation and transplantation organizations. The domain working group members included administrative, clinical, and academic experts in deceased and living donation ethics and 2 Patient, Family, and Donor partners. Identification of internationally accepted baseline ethical principles was done after literature reviews performed by working group members, and a framework for consideration of existing or novel policies was completed over a series of virtual meetings from March to September 2021. Consensus on the framework was achieved by applying the nominal group technique. We used the 30 baseline ethical principles described in World Health Organization Guiding Principles, Declaration of Istanbul, and Barcelona Principles to generate an ethical framework-presented graphically as a spiral series of considerations-designed to assist decision makers in incorporating these ethical principles into practice and policy. We did not seek to determine what is ethical but instead described a method of evaluation for policy decisions. The proposed framework could be applied to new or existing OTDT policy decisions to facilitate the transformation of widely accepted ethical principles into practical evaluations. The framework includes adaptation for local contexts and could be applied broadly internationally.

Bridging principal-agent and mechanism design theories: an integrated conceptual framework for policy evaluation

Bridging principal-agent and mechanism design theories: an integrated conceptual framework for policy evaluation

Desirability–doability group judgment framework for the collaborative multicriteria evaluation of public policies

AbstractDesirability–doability framework (2 × D) is a novel framework for the collaborative evaluation of public policies. Fundamental objectives and performance indicators are agreed upon in workshops, policies are characterised, and barriers to implementation identified. MACBETH interactive protocols are then applied in decision conferences to elicit qualitative judgments about the desirability of policies, within and across objectives; and about their doability under the expected graveness of barriers on contrasting scenarios. Elicited judgments allow, respectively, to construct a shared multicriteria model measuring the overall desirability of policies; and, to measure their doability. Desirability–doability graphs enable visual interactive classification of policies, with sensitivity/robustness analyses of uncertainties. 2 × D was successfully tested in a real‐world urban‐health policymaking case to evaluate spatial policies. The main novelty of 2 × D is that it bridges the socio‐technical gap, present in OR, between the support required by a complex social decision‐making process, and that usually offered by analytic techniques – while keeping modeling theoretically sound and simple.

Performance Evaluation of Fee-Charging Policies to Reduce the Carbon Emissions of Urban Transportation in China

As a market-based instrument for transportation demand management, a transport fee-charging policy can not only effectively reduce traffic congestion, but also improve air quality. Considering the urgent need to improve urban transport fee-charging policies and reduce transport carbon emissions, the paper focuses on the role of the performance of fee-charging policies in reducing the carbon emissions of urban transport. In this study, we propose a methodological framework for the performance evaluation of urban traffic carbon emission fee-charging policies. First, we analyze the current situation of the implementation of fee-charging policies and their relationship with urban traffic congestion. Subsequently, we analyze changing trends of carbon emissions associated with transportation travel in Beijing in recent years, to identify the main sources of carbon emissions from transport. Finally, we used the DEA method to evaluate the performance of the fee policies for urban transport, which are meant to reduce carbon emissions, analyze their implementation efficiency, and then discuss the main factors affecting their efficiency. The results show that with the implementation of fee-charging policies, urban traffic congestion has eased. The overall carbon dioxide (CO2) emissions from transportation in Beijing grew rapidly. CO2 emissions generated by car travel are the main source of carbon emissions from transportation in Beijing. The average value of the overall technical efficiency (TE) of Beijing’s fee-charging policies to reduce transportation carbon emissions from 2006 to 2018 is 0.962, indicating that the overall implementation of Beijing’s fee-charging policies has been effective. Adjustments to the fee structure reduce motor vehicle travel to an extent, increase the proportion of green travel, and reduce the intensity of transportation carbon emissions. The technical non-efficiency in Beijing’s fee-charging policy is mainly due to non-efficiency of scale, followed by pure technical non-efficiency. Appropriately adjusting the fee structures imposed by different policies would help to improve the efficiency of policy implementation.

Evaluating traffic emission control policies based on large-scale and real-time data: A case study in central China

The traffic control policies, including “Odd and Even” (OAE) and “One Day Per Week” (ODPW), were adopted in Zhengzhou, China. In this study, we use the bottom-up policy evaluation framework to capture the temporal-spatial characteristics of traffic conditions and vehicle emissions under various traffic restriction scenarios. Moreover, we use the street-scale simulation model to evaluate the effectiveness of improving air quality. Results showed that the improvements in traffic conditions led to the emission decrease by about 28.3 % for carbon monoxide (CO), 16.2 % for nitrogen oxide (NOx), 21.3 % for particulate matter (PM2.5), and 23.2 % for total hydrocarbon (THC) under OAE. During ODPW, total vehicle emissions decreased by 14.1 % for CO, 10.2 % for NOx, 13.7 % for PM2.5, and 12.4 % for THC. However, the spatial analysis indicates traffic restrictions could not significantly reduce those emissions caused by high traffic volume; besides, buses, middle-duty trucks, and heavy-duty trucks have partly offset the reduction benefit from restrictions. The air quality simulation results reveal no significant concentration decrease of CO and nitrogen dioxide (NO2) in most areas. With the update of vehicles, stricter management of high-emission vehicles, and limited coverage for implementation of policies, the traffic control policies were not as effective as before. The limitations of the restriction policies are gradually prominent, and upgrade policies are urgently needed to continuously improve urban air quality in the future.

How can policymakers incorporate uncertainty (as modelled through foresight) into policy evaluation?

Abstract Background Using a robust policy evaluation framework is critical to inform policymakers about the relevance of a policy in a structured way, clarifying its implications and providing arguments and criteria to compare competing options and decide which policy should be prioritised. In the current context of complex societal, environmental and public health challenges, decision-makers demand tools that allow a comparative assessment of the effectiveness of multisectoral policies and their comparison under different scenarios. Methods This presentation will detail a socio-technical “desirability-doability” framework (2xD). In 2xD, an additive value model is constructed to measure the desirability of public policies, and their doability is appraised under two contrasted scenarios. For these purposes, the MACBETH approach is used in developing the three group modelling phases of 2xD: (I) Structuring facilitated workshops, (II) Evaluation decision conference, and (III) Desirability-doability decision conference. Tailor-made interactive protocols or questioning procedures are used to elicit group judgements, based on which objective-specific value scores are assigned to the policies, and the objectives are weighted (in II). Finally, we elicit doability scores for the policies under each scenario (in III). Results We present how the 2xD framework was applied in Lisbon's urban health policymaking setting and discuss case insights regarding the role of scenario analysis in the appraisal and selection of policies. Conclusions The 2xD framework advances knowledge on how to assist policymakers in evaluating and selecting policies and contributes to the literature on overall policy formulation and evaluation. Specifically, the framework enabled a group of policymakers to balance (multicriteria) desirability versus doability of policies under the light of two contrasting scenarios, therefore incorporating uncertainty in their decision-making process.

A generic learning simulation framework to assess security strategies in cyber-physical production systems

Connected systems through computerized networks are at the heart of the Industry of the future. As they merge physical entities with cyber spaces, they fall under the paradigm of cyber-physical production systems. Cybersecurity is a key challenge for such systems, as they are subject to daily attempts of intruders to gain unauthorized access to their internal resources or to compromise their integrity. The fast increase of new attack strategies requires the rapid design and assessment of new defense strategies. It entails a complex, error-prone and time-consuming process, including the clear specification of the attack and defense strategies involved, and the design and implementation of the simulation model allowing to evaluate the performances of the defense strategy. This work intends to make such a process transparent to cybersecurity managers by limiting their workload to the sole specification of the characteristics of the system and the logic of the attack and the defense. It provides a generic hybrid simulation framework for flexible evaluation of cybersecurity policies, which is demonstrated on a SYN flooding application. Therefore, the contribution is twofold: (1) The proposed framework offers a high-level environment allowing various experts to collaborate by graphically modeling a given attack strategy and the envisioned defense strategy, without engaging in heavy implementation efforts. Then the framework's executable infrastructure, which combines simulation with machine learning to understanding the interactions between the attackers & the defender, will allow them assessing the performances of these strategies. The proposed framework differs from state-of-the-art cybersecurity simulation environments in its uniqueness to combining the expressive power of a universal simulation modeling formalism with the user-friendliness of a visual simulation tool. Therefore, it offers at one side, a very high modeling flexibility for easy exploration of various cybersecurity strategies, and at the other side, integrated learning capabilities for allowing self-adaptive user-based cybersecurity strategy design. (2) The application demonstrating the framework focuses on the most encountered and still uncontrolled threats in cybersecurity, i.e. the SYN-Flooding based Denial of Service (DoS) attack. The application targeted is not meant to propose yet another SYN flood detection algorithm or to improve the state-of-the-art in that domain, but to prove the framework operationality. The experimental results obtained showcase the ability of the framework to support learning simulation-based SYN flood defense algorithm design and validation.

Evaluating Australian drought policy from the perspective of good-practice strategic policymaking

ABSTRACT The sustainable management of Australia’s natural resources has increasingly attracted policy interventions by the Australian government. Drought policy represents one sector where the Australian government is a signatory to sustainable use of natural resources, cognisant of the nexus between agricultural production practices and environmental condition. A strategic sustainability policy evaluation framework is applied to drought policy to analyse policy design and process relationships with a view to informing improvements in these policy facets. The analysis finds that at the national level, the nature of the policy problem is inadequately defined, giving rise to generic objectives and overly narrow policy instrument choices that are inconsistent with the complex nature of drought and resilience objectives. Policy instrument choice is focussed on farm financial viability, with simplistic correlations drawn between farm profitability and improved social and environmental wellbeing. The appetite for broader structural reforms to agricultural and rural community development are lacking politically and by industry, but necessary given emerging social, economic and environmental challenges to agriculture. Structural reforms will be inevitable if the complexities between production and sustainable natural resource management are to be reconciled. More systemic and nuanced understandings of resilience and drought risk are needed to deliver on the policy ambition of drought resilience.

AIMING FOR SUCCESS

Despite the popularity of the Evidence-Based Policy Making paradigm, scholarly evidence often fails to have an impact in emotional or value-laden policy debates. Consequently, changes to Canada’s gun control laws in recent years have often failed to incorporate scholarly research. This is problematic given that the forces of path dependence impose costs on policy makers who seek to reverse established policies, even if they are dysfunctional. This article lays the theoretical foundations for a Firearms Policy Evaluation Framework, which can be used by scholars, policy makers, advocates, and the public to conduct preliminary evaluations of proposed firearms policies before they become law. The utility of the framework is then demonstrated with an evaluation of the 2020 assault-style weapons ban in Canada, which includes a systematic scoping review of the literature on the impact of assault-weapons bans.

Evaluating hiring incentives: evidence from Italian firms

PurposeThis paper analyses the effect of a particular Active Labour Market Policy, the hiring incentives, on firms hiring policies. The effects of a programme on firms' behaviour have in fact rarely been evaluated.Design/methodology/approachThe analysis is based on micro-data drawn from Rilevazione su Imprese Lavoro (RIL), conducted by Inapp in 2010, 2015 and 2018 on a representative sample of limited liability and partnership firms. The authors apply a policy evaluation framework to investigate the impact of the use of incentives in the short run. The authors infer the counterfactual policy scenario thanks to a survey question that asks about firms' behaviour in the absence of the incentives. The authors also control for firms' unobserved heterogeneity, including firm's fixed effects, and endogeneity issues, estimating a differences-in-differences model that exploit the longitudinal component of the RIL survey.FindingsThe authors find that the use of at least one incentive scheme in 2017 is associated with an increase in the share of newly hired of about 0.07 percentage point in the short run. Moreover, hiring incentives have a small positive relationship with the share of both young workers and temporary contracts. Finally, these results are robust to endogeneity issues.Originality/valueThe analysis provides an updated evaluation of the effectiveness of hiring incentives in Italy by focusing on firms' behaviour rather than on the evolution of individual employment. It identifies the impact by using a rich set of econometric methods as well as counterfactual analysis.