Tool For Policy Evaluation Research Articles

Linking implementation science and policy: Process and tools for congressionally mandated implementation, evaluation, and reporting.

To describe a process model for assisting partners in addressing requirements of legislation and review policy analysis, planning, and evaluation design processes and tools. Throughout its 25-year history, the United States Department of Veterans Affairs (VA) Quality Enhancement Research Initiative (QUERI) program has been a forerunner in partnering with organizational leaders to improve health care. The Foundations of Evidence-based Policymaking Act of 2018 provided new opportunities for QUERI and other implementation scientists to support federal agency leaders in implementing, evaluating, and reporting on congressionally mandated programs. Although implementation scientists have the skills to support partnered implementation and evaluation, these skills must be adapted for congressionally mandated projects as many scientists have limited experience in policy analysis and the intersection of data informing organizational policy, programs, and practices (i.e., evidence-based policy). During the conduct of four congressionally mandated projects, our national VA QUERI team developed processes and tools to achieve the goals and aims of our VHA partners and to ensure our collective work and reporting met legislative requirements. Our process model, program planning, and analysis tools were informed by an iterative process of refining and adapting the tools over a period of six years, spanning the years 2017 to 2023. Work to support our partners was conducted across three phases: preparation and planning, conducting implementation and evaluation, and developing the congressionally mandated report. The processes and tools we developed within the context of mutually respectful and honest partnerships have been critical to our QUERI center's success in this area. Lessons we learned may help other scientists partnering in VA or other federal agencies to plan, conduct, and report on congressionally mandated projects.

Estimation of the Local Conditional Tail Average Treatment Effect

The conditional tail average treatment effect (CTATE) is defined as a difference between the conditional tail expectations of potential outcomes, which can capture heterogeneity and deliver aggregated local information on treatment effects over different quantile levels and is closely related to the notion of second-order stochastic dominance and the Lorenz curve. These properties render it a valuable tool for policy evaluation. In this article, we study estimation of the CTATE locally for a group of compliers (local CTATE or LCTATE) under the two-sided noncompliance framework. We consider a semiparametric treatment effect framework under endogeneity for the LCTATE estimation using a newly introduced class of consistent loss functions jointly for the CTE and quantile. We establish the asymptotic theory of our proposed LCTATE estimator and provide an efficient algorithm for its implementation. We then apply the method to evaluate the effects of participating in programs under the Job Training Partnership Act in the United States.

The School Suicide Policy Evaluation Tool (SSPET): A proof‐of‐concept for assessing school suicide prevention policies

AbstractBackgroundAs adolescent suicide rates have increased, there has been a nationwide increase in legislation requiring schools to create suicide prevention policies. In Utah, school districts must implement a youth suicide program for grades 7–12. Currently, there have been no systematic analyses of Utah school district suicide prevention policies. Thus, we developed a new evaluation instrument, the School Suicide Policy Evaluation Tool (SSPET).MethodsThe SSPET was derived from the “Model School District Policy” and evaluates criteria on prevention, intervention, re‐entry, and postvention. Four raters used the SSPET in a systematic analysis of suicide prevention policies for all school districts in the state of Utah.ResultsAnalysis revealed a right‐skewed distribution of policy scores, and a median total score was 2.25/36. The mean percentage of inclusion of the four subsections in policies was less than 20% for most criteria. Median scores were totaled for the four subsections as follows: Prevention score was 1/9, intervention score was 1/14, re‐entry score was 0/4, and postvention score was 0/9.ConclusionThis proof‐of‐concept study demonstrates the utility of the SSPET in identifying shortcomings of school district policies on suicide while generating specific recommendations for improvement. With further validity testing, we expect this tool to be useful to school district administrators, education policymakers, and those studying suicide interventions and outcomes in school‐aged children and adolescents.

Validity and Reliability of Child-Friendly School Policy Evaluation Instruments in Primary Schools: Confirmatory Factor Analysis

<p style="text-align:justify">Evaluation of child-friendly school (CFS) policies is essential to determine the achievements of school efforts in reducing violence cases. This research aims to proving the reliability and validity of CFS policy evaluation instruments in elementary schools with different locations. This investigation uses the Context Input Process Product (CIPP) evaluation model to confirm the factor structure or dimensions of several observed variables (items) to evaluate the reliability and validity of the CFS policy evaluation tool in elementary schools. The validity and reliability of previously created instruments are evaluated quantitatively, but this study utilises different subjects and study sites. There were 320 respondents, with 145 school principals and 175 teachers taken randomly. Confirmatory factors analysis (CFA) results show goodness of fit (GOF), and the model is acceptable. The CFS evaluation instrument can be accepted after eliminating several question items and modifying them. All items of the teal instrument meet the goodness of fit criteria in terms of chi-square and root mean square error of approximation (RMSEA). The instrument for evaluating CFS policies in primary schools using the CIPP model has met a valid and reliable psychometric property test so that it can be applied.</p>

7.F. Workshop: Local physical activity policy evaluation: state of the art, tools and methods and future directions

Abstract Insufficient physical activity (PA) is one of the most significant risk factors for chronic diseases and premature mortality worldwide. Public policies are recognized as upstream factors providing system change beyond individual changes. Policies have been recognised as an effective way to increase population PA levels, as they support the development of evidence-based interventions and programs in different domains, that help create a ‘healthy’ PA environment and enhanced PA opportunities. Previous studies have shown an important impact of PA promotion policies at local level (municipalities), but insufficient research and tool development has been realised on PA policies beyond national level. First, the workshop will briefly describe the scientific state of the art on local PA promotion policies evaluation, based on results from a literature review. Second, the panel will present the CAPLA-Santé, a policy evaluation tool designed by a stakeholders’ group in France, based on the Health Enhancing Physical Activity Policy Audit Tool of the World Health Organisation, to comprehensively review local PA policies. The final version of CAPLA-Santé contains 21 items divided into six sections: 1. Overview of HEPA stakeholders in the local government area, 2. Policy documents, 3. Policy contents, 4. Funding and political engagement, 5. Studies and measures relating to physical activity in the local government area, and 6. Progress achieved and future challenges. CAPLA-Santé allows the collection and in-depth analysis of local level policies to assess the progress in promoting PA and intersectoral collaboration as well as identifying successful policy levers and remaining challenges. Third, the panel will illustrate its use in 5 countries worldwide (Finland, France, Germany, Japan and Romania), summarising findings and explaining how such tools allow cross-cultural comparison. Fourth, panellists will discuss the challenges and success in using such tools and provide insights on how these tools can serve as capacity building for policy makers and to draw recommendations, as well as cross-cultural comparisons. Key messages • The workshop will provide key advise on how to monitor local physical activity policies. • The workshop will equip participants with tools to face the challenge of local physical activity policy evaluation. Speakers/Panelists Petru Sandu Cluj School of Public Health, Babes-Bolyai University in Cluj-Napoca, Cluj-Napoca, Romania Anne Vuillemin Université Côte d’Azur, Nice, France François Berdougo Société Française de Santé Publique, Laxou, France Peter Gelius FAU Erlangen-Nürnberg, Erlangen, Germany

S4-1 Learning from national initiatives from across the EU in HEPA policy monitoring and evaluation

Abstract The monitoring and evaluation of HEPA policy are essential steps towards developing the evidence base to understand the process and impact of policy implementation as well as guide future policy design. The policy development and implementation cycle can be strengthened through effective monitoring and evaluation tools, as well as multi-sectoral engagement. The promotion of physical activity often requires multiple sectors such as public health, sport, urban design etc., and working in diverse settings. In recent years, researchers and international organizations, such as WHO, have produced useful resources and guidance for monitoring and evaluating the impact of policy on a national level. The fast-growing interest in physical activity policy by national governments, for the purposes of increasing health and wellbeing can provide new opportunities for capturing the impact of HEPA policy and learning from other countries. The lack of available evidence on the process and effectiveness of HEPA policy remains a challenge when developing new policy. This highlights the need for sharing good practice from diverse policy contexts and HEPA settings. The symposium presentations will discuss policy monitoring and evaluation tools and case studies to investigate the different implementation aspects of national HEPA policy in four EU countries: Germany, Ireland, Slovenia and Wales. The goal of this symposium is to share lessons from the development of instruments used in (physical activity) policy monitoring and evaluation and to present real-world case studies of cross-sector collaboration. The high quality and diversity of the tools and projects presented in the four presentations, makes this symposium beneficial to both novice and experienced researchers, policy-makers and practitioners. The attendees can update their knowledge regarding different methods and instruments used in physical activity policy monitoring and evaluation and learn from engaging multiple sectors in HEPA such as urban design, sport, environment and public health. Reflections from this symposium will inform and enhance HEPA policy development and implementation in real-time and stimulate future research and practice in this field.

Pay-for-practice or Pay-for-performance? A coupled agent-based evaluation tool for assessing sediment management incentive policies

Cost-shared programs have been applied to incentivize the adoption of agricultural best management practices (BMPs) to address the long-standing water quality issue in the Chesapeake Bay watershed, US. However, the business-as-usual (BAU) incentive program (i.e., pay-for-practice, paying cost share for implementing BMPs) is likely to miss the Total Maximum Daily Load target to reduce 20% of the total suspended sediment (TSS) in 2010 by 2025. Some field experiments indicate that pay-by-performance (PFP; paying lower cost share but with additional bonus payment per unit sediment reduction) can better motivate community involvement leading to greater water quality control outcomes. However, the effectiveness of different incentive policies is still unclear at a basin scale. We propose a coupled agent-based modeling tool to quantify the performance of different incentive policies. The tool considers farmers’ (i.e., agents’) BMP adoption dynamics affected by the social norm and the potential bonus payment. Specifically, we compare individual-based PFP (PFPi) and group-based PFP (PFPg) with BAU. Results of our proposed model applied to the selected study area, the Susquehanna River Basin, Chesapeake Bay’s largest tributary watershed, suggest that PFP can achieve higher TSS reduction with less cost. PFPg shows the best basin-wide TSS reduction associated with the least uncertainty among all tested policies. Also, the performance of PFPg is less impacted by the change in the bonus payment compared to PFPi attributed to farmers’ collaboration efforts. Potentially, the proposed policy evaluation tool can better inform an achievable target with policy suggestions in assistance with social studies (e.g., surveys and behavioral experiments).

Studying tourism recovery options under economic constraints: does stakeholder engagement and financial stability matter?

Researchers neglected the relevance of combining tourism with stakeholders' engagement and financial stability for unlocking the economic constraints. For this, current research intended to study the interplay of tourism recovery options under economic constraints with the role of stakeholders' engagement and financial stability. According to our data, China outperforms the other OECD members. The results shown that Russia comes in second with a consistent in tourism recovery, financial stability, and stakeholders' engagement; Indonesia and Turkey, which finished fourth and fifth in the rankings, have good tourism recovery prospects. Mexico and Brazil come in second and third, respectively, with the worst performance and the lowest scores recorded in the survey findings. The overall interplay among the variables was empirically produced based on the findings of this study and is a new inclusive assessment of financial stability and tourism recovery parameters. It is a useful tool for policy development and evaluation. This study also addresses the criticism of load selection at random and accurately depicts the entire scope of fiscal intervention in the economy to prudently enhance tourism recovery at large.

Restoring the incentives for eco-design in extended producer responsibility: The challenges for eco-modulation

Extended producer responsibility (EPR) is an environmental policy strategy that makes producers responsible for the waste management of their products and packages. A key goal of EPR is to incentivize producers to (re)design their products and packages to improve their environmental performance, especially at end of life. However, because of the way in which the financial structure of EPR has evolved, those incentives have largely been muted or undetectable. Eco-modulation has emerged as an additional component in EPR to restore the missing incentives for eco-design. Eco-modulation operates through changes in the fees that producers pay to meet their EPR obligations. Eco-modulation includes both increased differentiation of types of products and associated fees, and additional bonuses and penalties—environmentally targeted discounts and penalties on the fees that each producer pays.Based on primary, secondary, and grey literature, this article characterizes challenges faced by eco-modulation if it is to restore the incentives for eco-design. These include weak linkages to environmental outcomes, fees too low to induce changes in materials or design, lack of adequate data and ex post policy evaluation, and implementation that differs across jurisdictions. Opportunities to address these challenges include use of life cycle assessment (LCA) to inform eco-modulation, increased eco-modulation fees, strategies to increase harmonization of eco-modulation implementation, mandated provision of data, and policy evaluation tools that establish the efficacy of different eco-modulation schemes. Considering the scope of the challenges and the complexity of establishing eco-modulation programs, we suggest treating eco-modulation at this stage as an experiment in promoting eco-design.

Switching from Risks to Opportunities: The Application of a Superbonus Tax Incentive to Heritage Buildings from the 1960s in Fragile Mountain Contexts

This paper introduces actual considerations for the progressive disuse of residential space in the Alpine territory, considering possible actions. Nowadays, the building complexes built around the 1960s and 1970s (a symbol of mass tourism) are suffering and searching for a new identity. The generation of owners who bought them has aged and the propensities of the new generations for holiday in those places has changed, which means fewer opportunities for leisure, particularly in the winter. Due to the great attention (and seeming opportunities) of current incentive policies toward improving the energy use of the existing stock, the authors investigate the private conveniences of transformations through the refurbishment of these buildings. Starting from a study of the territory and the dynamics of the local population, this research analyzes a possible set of energy works, based on a new (2020) incentive measure, the 110% Superbonus, which consists of a series of facilitation mechanisms, deductions, and reimbursements for building interventions. A large part of the insight is focused on a technical and economic feasibility study of the possible actions, following a process based on the evolution of the legislation. This work is based on a specific case study, located in a small municipality in the Piedmont mountain area, consisting of three apartment blocks of mostly second homes. The methodology adopted lends itself on the one hand, as a guide for preliminary economic energy assessments and, on the other hand, as a policy evaluation tool from the public and private perspectives.

Application of an interrupted time series analysis (ITS) to evaluate the effect of universal dialysis policy from 2006 to 2016 in a province of northeastern Thailand

Background: An interrupted time series (ITS) analysis is a powerful tool for policy evaluation. In Thailand, chronic kidney disease (CKD) is a public health problem that requires a long recovery time and has a high treatment cost. The universal coverage policy for renal replacement therapy (universal dialysis policy), is used to treat this disease but policy evaluation using ITS analysis has rarely been conducted. This study applied ITS analysis to test the effect of such a policy between 2006 and 2016. Methods: Data were retrieved from the electronic database of the health data center in Roi Et Province for the period between January 1, 2006 and December 31, 2016. 15,681 CKD stage 5 patients were included. The intervention under assessment was the universal health coverage system, which has been implemented since 2008. Results: Results showed that before implementation of the universal dialysis policy, the overall trend of access to renal replacement therapy (RRT) slightly increased (0.74; 95% confidence interval (CI): 0.58, 0.90). After implementation of the policy, access sharply increased (6.10; 95%CI: 3.67, 8.54), while the linear trend after policy implementation also slightly increased (0.29; 95%CI: 0.05, 0.14). The stratified analysis showed the same linear directional trend before and immediately after implementing the universal dialysis policy. Conclusions: Implementation of the universal dialysis policy positively impacted the rate of renal replacement therapy in CKD stage 5 patients, while access to health care services also increased.

Ensuring inclusive and accessible digital mobility through universal design

The proliferation of smartphones and digital technologies have produced a quickly expanding array of innovative and disruptive digital mobility and logistics services, but with a varying degree of resulting benefits to different sections of the society. This paper presents the novel approach adopted by Inclusive Digital Mobility Solutions (INDIMO), a Horizon 2020 project to digital inclusion in mobility and logistics based on universal design principles adapted to such services and complemented by socioeconomic equity, cybersecurity and private data protection. Challenges and opportunities have been identified. A Universal Design Manual and a Service and Policy Evaluation Tool have been developed through a series of interviews and co-creation workshops with users, non-users and stakeholders as well as real-life testing of the manual on the (re)development of five pilot services.

Can the Sorghum Planting Industry in Less-Favoured Areas Promote the Income Increase of Farmers? An Empirical Study of Survey Data from 901 Samples in Luquan County

Poverty alleviation is the basic requirement of human social development. However, there is still a lack of quantitative research on the poverty alleviation effect of regional, characteristic industries. Few studies have focused on the increase of micro individual income and used more advanced policy evaluation tools for comparative analysis based on a quasi-experimental perspective. In addition, the existing research ignores the critical question: can characteristic industries really achieve sustainable development goals while bringing poverty alleviation results? We studied regional, characteristic industries from a new perspective, taking into account the poverty alleviation effect and regional sustainable development. Based on the survey data of 901 households of representative village committees of Tanglang and Dache in Luquan Yi and Miao Autonomous County, this study quantitatively analyzed the poverty alleviation effect of the sorghum planting industry by using the Propensity Score Matching Difference-in-Differences (PSM-DID) model. The adoption of the industrial alleviation policy has significantly increased the per capita net income of rural households in Tanglang and Dache village committees, by 2171.64 CNY and 1945.06 CNY, respectively. The estimation results of the whole sample show that the effect of the policy to the per capita net income of households is 1726.87 CNY. The development of the sorghum planting poverty alleviation industry in Luquan County has promoted income increase of households significantly, creating economic, social and ecological sustainability, and can provide a reference for less-favoured areas.

Guidelines for the ethical review of mental health research protocols from a culturally-sensitive perspective

In most of Africa, mental illness is considered a silent epidemic due to structural and systemic barriers such as inadequate health care infrastructure, insufficient number of mental health specialists, stigma and discrimination related to mental illness, and lack of access to all levels of care. The continued suffering and disability due to mental illness calls for newer treatments and continued research into the field of mental health treatment and therapy. Research ethics committees are cognizant of the importance of this aspect yet, the road to this innovative future is fraught with ethical dilemmas as well as ethical, legal, and social implications. There is an urgent need to have guidelines that give guidance to research ethics involving mental health participants. Guidelines were developed for the ethical review of mental health research protocols with the aim of increasing health equity and access in Africa and other low-income countries. We systematically analyzed 13 programme, research, and policy evaluation tools in light of mental health and cultural attributes. A 54-item assessment tool was created to guide the process of mental health research protocol evaluation taking into account ethical, gender, and sociocultural factors in Africa. The emerging themes i.e. research governance, background and justification, methodology and ethical impact of the research forming the basis of the assessment tool are interlinked with the bioethical principles of respect for autonomy, beneficence, non-maleficence and justice. When they are applied to the evaluation of a study they will not only underscore these principles, they enhance their application and increase access to equitable health in Africa and other low-resource settings regardless of the varied contexts.

Evaluating the real changes of air quality due to clean air actions using a machine learning technique: Results from 12 Chinese mega-cities during 2013–2020

China has implemented two national clean air actions in 2013–2017 and 2018–2020, respectively, with the aim of reducing primary emissions and hence improving air quality at a national level. It is important to examine the effectiveness of such emission reductions and assess the resulting changes in air quality. However, such evaluation is difficult as meteorological factors can amplify, or obscure the changes of air pollutants, in addition to the emission reduction. In this study, we applied the random forest machine learning technique to decouple meteorological influences from emissions changes, and examined the deweathered trends of air pollutants in 12 Chinese mega-cities during 2013–2020. The observed concentrations of all criteria pollutants except O3 showed significant declines from 2013 to 2020, with PM2.5 annual decline rates of 6–9% in most cities. In contrast, O3 concentrations increased with annual growth rates of 1–9%. Compared with the observed results, all the pollutants showed smoothed but similar variation in trend and annual rate-of-change after weather normalization. The response of O3 to NO2 concentrations indicated significant regional differences in photochemical regimes, and the differences between observed and deweathered results provided implications for volatile organic compound emission reductions in O3 pollution mitigation. We further evaluated the effectiveness of first and second clean air actions by removing the meteorological influence. We found that the meteorology can make negative or positive contribution in reducing pollutant concentrations from emission reduction, depending on type of pollutants, locations, and time period. Among the 12 mega-cities, only Beijing showed a positive meteorological contribution in amplifying reductions in main pollutants except O3 during both clean air action periods. Considering the large and variable impact of meteorological effects in changing air quality, we suggest that similar deweathered analysis is needed as a routine policy evaluation tool on a regional basis.

China’s Fair Competition Review System: A Single Case Study

The case study methodology has proved to be a useful empirical tool for competition policy evaluation. However, as far as China’s Fair Competition Review System (FCRS) is concerned, empirical studies are scarce. This article aims to partly fill this gap by thoroughly studying the first litigation case in light of three questions: (1) does China’s FCRS contribute to a competitive market?; (2) does it face challenges regarding implementation, including judicial proceedings?; and (3) how to tackle these challenges? We find that China’s FCRS promotes a competitive market to some extent, but diverse issues need to be tackled in the coming years. Some policymakers still lack understanding of the system. Public antitrust enforcement also faces understanding and capability problems to fully implement the FCRS. The review standards are not specific enough. Regarding judicial scrutiny of the FCRS, we note that also judges lack knowledge of the FCRS, especially in primary courts. Court jurisdictions for filing administrative monopoly litigation are not of high enough rank. In addition, the nature of the FCRS brings up doubts when entering into litigation as the case has to be connected with the Anti-Monopoly Law (AML). Consequently, we formulate several suggestions for improvement: First, strengthening competition advocacy and FCRS training for policymakers, antitrust enforcement officials, and judges. Second, establishing disciplinary and incentive mechanisms. Third, increasing enforcement capacity. Fourth, specifying industry-specific review standards. In terms of judicial scrutiny, in addition to the training for judges, we also propose to reform the administrative proceeding system, adding corresponding clauses connected to Chapter V of the AML and the FCRS into the Administrative Procedure Law. Abstract administrative actions should also have the possibility to initiate litigation in the near future, and administrative monopoly cases should be filed at least to an intermediate court or intellectual property court, rather than to a primary court. The establishment of a dedicated competition court could also add value. Last, we recommend upgrading the FCRS to proper law. China, Fair Competition Review System, Evaluation, Case study, Administrative monopoly

Implementation Science: A Critical Tool for Research Utilization and Policy Evaluation

Implementation Science: A Critical Tool for Research Utilization and Policy Evaluation

Implementation Science: A Critical Tool for Research Utilization and Policy Evaluation

Implementation Science: A Critical Tool for Research Utilization and Policy Evaluation

Policy and behavioral response to shock events: An agent-based model of the effectiveness and equity of policy design features.

In the aftermath of shock events, policy responses tend to be crafted under significant time constraints and high levels of uncertainty. The extent to which individuals comply with different policy designs can further influence how effective the policy responses are and how equitably their impacts are distributed in the population. Tools which allow policymakers to model different crisis trajectories, policy responses, and behavioral scenarios ex ante can provide crucial timely support in the decision-making process. Set in the context of COVID-19 shelter in place policies, in this paper we present the COVID-19 Policy Evaluation (CoPE) tool, which is an agent-based modeling framework that enables researchers and policymakers to anticipate the relative impacts of policy decisions. Specifically, this framework illuminates the extent to which policy design features and behavioral responsiveness influence the efficacy and equity of policy responses to shock events. We show that while an early policy response can be highly effective, the impact of the timing is moderated by other aspects of policy design such as duration and targeting of the policy, as well as societal aspects such as trust and compliance among the population. More importantly, we show that even policies that are more effective overall can have disproportionate impacts on vulnerable populations. By disaggregating the impact of different policy design elements on different population groups, we provide an additional tool for policymakers to use in the design of targeted strategies for disproportionately affected populations.

World Religions in Religious Education in Northern Ireland: A Policy Implementation Analysis using Strategic Action Field Theory

This article analyses the implementation of a policy that introduced the teaching of world religions in religious education in Northern Ireland. Using a policy implementation evaluation framework informed by strategic action field (SAF) theory we assess the effectiveness of the policy intervention by exploring teachers’ descriptions of their practice. We discover that there is evidence of teacher buy-in to the policy, but very significant variation in the quality of implementation and reasons for this are explored. It is concluded that the policy evaluation tool developed here is likely to have value in assessing the implementation of curriculum policy in other contexts.