Framework For Policy Evaluation Research Articles

Evaluating childhood policy impacts on lifetime health, wellbeing and inequality: Lifecourse distributional economic evaluation

We introduce and illustrate a new framework for distributional economic evaluation of childhood policies that takes a broad and long view of the impacts on health, wellbeing and inequality from a cross-sectoral whole-lifetime perspective. Total lifetime benefits and public cost savings are estimated using lifecourse microsimulation of diverse health, social and economic outcomes for each individual in a general population birth cohort from birth to death. Cost-effectiveness analysis, policy targeting analysis and distributional analysis of inequality impacts are then conducted using an index of lifetime wellbeing that allow comparisons of both value-for-money (efficiency) and distributional impact (equity) from a cross-sectoral lifetime perspective. We illustrate how this framework can be applied in practice by re-evaluating a training programme in England for parents of children at risk of conduct disorder. Our illustration uses a simple index of lifetime wellbeing based on health-related quality of life and consumption, but other indices could be used based on other kinds of outcomes data such as life satisfaction or multidimensional quality of life. We create the detailed underpinning data needed to apply the framework by using a previously published meta-analysis of randomised controlled trials to estimate the short-term effects and a previously published lifecourse microsimulation model to extrapolate the long-term effects.

Framework for Analysis and Evaluation of Energy Storage Policies

Framework for Analysis and Evaluation of Energy Storage Policies

An assessment of the policy and regulatory outcome by the telecom services users: The emerging economy study

Outcome-based policy evaluation is an established practice in the distributive and redistributive public policies. Such practices are not evident for competitive regulatory policies of telecom, especially in India. This study bridges this research gap by carrying out an outcome-based evaluation of telecom policy and highlighting the importance of such evaluation. Using the methodological pluralism model from Schalock (2002), the outcome of India’s telecom policies was evaluated. Outcome measures from the vision statement of telecom policy were appraised by telecom users by responding to a structured questionnaire-based survey. Factor analysis confirmed that our survey instrument measured the identified policy outcomes. Regression analysis confirmed that users’ appraisal was based on their experiences of telecom services. Against five policy outcome measures, the survey respondents agreed on the achievement of affordability of services: 68.9% of the respondents found telecom services not secure; 74.7% of the survey respondents indicated an issue with quality; 55.6% of the respondents did not agree that the services are available anytime, anywhere. Outcome measures like telephone density (teledensity) as adopted by Telecom Regulatory Authority of India (TRAI) and Department of Telecommunications (DoT) are not the true representative of policy outcome. A multistakeholder policy evaluation will reveal the actual policy outcomes. International Telecommunications Union (ITU) should establish a standardized framework for outcome-based policy evaluation to address such issues.

The technical impacts of the carbon tax in China

Despite the significant impacts of technology on the socioeconomic effects of climate policies, many previous researchers neglected the induced technical impacts and thus resulted in biased evaluations of climate policies. Hence, it is important that the induced technology should be endogenized in the policy evaluation framework. The purpose of this paper is the quantification of the technical impacts of the Chinese carbon tax using a Computable General Equilibrium (CGE) model. The technical impacts are denoted by the induced technological change (ITC), which is a function of the energy-use efficiency (EUE), energy-production efficiency (EPE), and nonenergy-production efficiency (ENE). The carbon tax will increase the energy cost share because of the internalisation of the abatement costs. This paper empirically shows that the carbon tax will decrease the energy cost share and production efficiency but increase the energy use and nonenergy production efficiency. Under the carbon tax, the ITC will decrease the energy use and production efficiency but increase the nonenergy production efficiency. The ITC will increase the RGDP, decrease the household welfare, and increase the average social cost of carbon (ASCC). This finding implies that the ITC of the carbon tax is biased towards the technical progress of nonenergy sectors; the emission abatement will become costlier under the ITC impacts. Although the quantification method of the technical impacts was from an existing published paper, the CGE analysis of the ITC impacts of the carbon tax in China is original in this paper.

Appraising the Implementation of Complexity Approaches Within the Public Health Sector in Scotland. An Assessment Framework for Pre-Implementation Policy Evaluation.

Complexity approaches have gained international attention as potentially effective strategies to address population health challenges. In light of this, the Scottish government (Scot. Gov.) set the implementation of these approaches as the recommended practice for its public health sector organizations. This study evaluates the opportunity and feasibility of implementing complexity approaches in public health Scotland employees' everyday routine by employing a qualitative study that involves 20 stakeholders, representative of different organizations and roles. We made use of an assessment framework based on Soft Systems Methodology (SSm) and Normalization Process Theory (NPT) comprised of five phases: Phase One defines the boundaries, aims, and goals of the issue under study; Phase Two consists of data collection, drawing on the e-Health Implementation Toolkit (e-HIT); Phase Three involves short presentations and breakout group activities to provide information on the new policy; Phase Four employs system thinking tasks to structure debate and builds shared understanding among participants; Phase Five applies NPT to appraise the organizational position around complexity based on information from the preceding steps. We found two main obstacles to implementing complexity approaches: (1) a lack of a shared understanding of the key concepts in complexity and their practical implications; (2) stakeholders' fear of significant disruption to work routines and power relationships. We recommend addressing these issues with appropriate training and customization of goals and tools that may enable complexity approaches to succeed within the Scottish public health context. Our assessment framework allows the recognition of key mechanisms to support how Scotland's Public Health body can enhance the implementation of complexity approaches. The appraisal framework could be used to study early-stage policy implementation in other contexts.

Meso Evaluation for SDGs’ Complexity and Ethics

ABSTRACT Sustainability is normatively defined as the interconnectedness of policy goals and actions; the partnership among governments, civil society, and the private sector; and a transformational vision pursuing structural change against marginalization and environmental degradation. This article provides the conceptual basis for a meso policy analysis and evaluation framework to address the normative dimensions of sustainability-centered policies. Drawing on complexity, behavioral, and sustainability sciences, a meso interpretative lens contributes to articulating the ethical and techno-scientific norms underlying SDGs discourses. Through knowledge co-production, and collaborative governance, a meso policy analysis and evaluation approach can help overcome centralized politics and techno-scientific rationalization.

The Urbanisation Impacts on the Policy Effects of the Carbon Tax in China

In the literature, very few studies have focused on how urbanisation will influence the policy effects of a climate policy even though urbanisation does have profound socioeconomic impacts. This paper has explored the interrelations among the urbanisation, carbon emissions, GDP, and energy consumption in China using the autoregressive distributed lag (ARDL) model. Then, the unit urbanisation impacts are inputted into the policy evaluation framework of the Computable General Equilibrium (CGE) model in 2015–2030. The results show that the urbanisation had a positive impact on the GDP but a negative impact on the carbon emissions in 1980–2014. These impacts were statistically significant, but its impact on the energy consumption was not statistically significant. In 2015–2030, the urbanisation will have negative impacts on the carbon emissions and intensity. It will decrease the GDP and the household welfare under the carbon tax. The urbanisation will increase the average social cost of carbon (ASCC). Hence, the urbanisation will reinforce the policy effects of the carbon tax on the emissions and welfare.

Population Ethics for an Imperfect World: Basic Justice, Reasonable Disagreement, and Unavoidable Value Judgements

Our collective impact on the environment results from a combination of population, affluence and technology. Human population, 2.6bn in 1950, is predicted to reach 10.9bn by 2100. So much we know. But beyond this starting point questions of population are presented in several ways in normative debate, often with problematic underlying moral assumptions. This paper clarifies and critiques four interconnected claims: (1) the Current Carrying Capacity Claim; (2) the Basic Justice Carrying Capacity Claim; (3) the Optimum Population Claim; and (4) the Population Variable Claim. A moral framework for population policy evaluation is then sketched which promises to avoid the most troubling of these moral critiques, whilst maintaining the valuable elements of (2) and (4). This prioritises the ‘morally basic’ at two levels: in terms of specific policy implications and the broader-level need to remain within the circumstances of global and intergenerational justice. However, it faces up to unavoidable trade-offs between other morally significant criteria. Three significant challenges to filling out the model are outlined, and the need acknowledged, however, problematically, for some global-level judgment call.

Designing and Implementing a Quality Physical Education Policy: Successes, Setbacks, and Lessons Learned From the Quality Physical Education Policy Project in Mexico.

Quality physical education (QPE) is part of a whole-of-school approach for school-based physical activity promotion. The United Nations Educational, Scientific and Cultural Organization QPE Policy Project supported 4 countries to develop QPE policies. The authors summarize the process, progress, successes, setbacks, and lessons learned during the implementation of the project in Mexico. The project was developed from August 2016 to April 2018 following the methodology proposed by the United Nations Educational, Scientific and Cultural Organization. Adaptations to the methodology were made to meet local needs. The project successfully implemented the United Nations Educational, Scientific and Cultural Organization methodology and prepared a national strategy for the provision of QPE in Mexico. The national strategy progressed despite the change in presidential administration. Successes included the use of a QPE policy evaluation framework, the inclusion of stakeholders representing extreme PE views and from all regions in the country, and the presence of international agencies in the national team. Setbacks included difficulties in engaging key organizations and a weak communication campaign. Lessons learned are discussed. The QPE project in Mexico served as a pilot project to test the feasibility of implementing a QPE policy revision process. The experience and lessons learned in Mexico can be drawn on to inform the work of other stakeholders interested in advocating for a national QPE policy.

A Critical Analysis of Students' Performance within the 3-Year and 4-Year Secondary Education Policies in Ghana

In its quest for an optimal education system, Ghana has implemented a number of education reform policies ending with the 3-year and 4-year secondary education systems. The main focus of this study was to use research findings to determine which system, between the 3-year and 4-year secondary education policies, is optimal for Ghana. In the methodology, comparative analysis of secondary quantitative data in the form of test scores from the database of the West African Examinations Council was undertaken. The test scores were for the four core subjects, namely, English Language, Mathematics (Core), Integrated Science and Social Studies, for the period 2011 to 2015 within which the two policies operated equally. The difference in performance was subjected to Independent Sample T-test for statistical significance at p≤.05. The Cohen's d effect size of the difference was also considered. Government subsidies for the two systems were examined. The study found that the 4-year secondary education system offers a better chance for academic performance, but the 3-year system is economically more viable. Based on the findings, a 6-year primary, 2-year junior high and 4-year senior high (6+2+4) pre-tertiary education system was proposed for the consideration of policymakers, without abolishing the 2-year kindergarten. A new framework for educational policy evaluation, is proposed. The author believes that the findings will contribute immensely to the search for an optimal secondary education policy for Ghana.

A new framework for policy evaluation: Targets, marine litter, Italy and the Marine Strategy Framework Directive

Italy is a major country bordering the Mediterranean Sea. As part of the implementation of the European Marine Strategy Framework Directive, Italy has set a target for reducing marine litter. This study evaluates the effectiveness of current measures for meeting that target. The paper proposes a framework for analysis, which confronts the primary sources of marine litter (i.e. inadequate waste management, wasteful habits, and waste from productive activities) with the adopted measures (classified in: prevention, management, and monitoring and control). In applying this framework, the paper adopts a new approach that focuses on a few salient aspects of governance responses: current legislation, circular economy, specific waste management practices, and monitoring and control systems. In support of the evaluation, the paper provides detailed data on the current legislation, actors involved, waste management and enforcement activities. The study highlights that Italy is engaged in accomplishing marine litter reduction, even if the target lacks precise quantitative/qualitative details. Despite this shortcoming, the framework allows identifying strong (e.g. extensive legislation, increased awareness, improved collection of waste) and weak points (e.g. lack of a national waste strategy, problems of governance integration, monitoring and control activities to be improved). The latter predictably decrease the effectiveness of the actions against marine litter, thus making the target less reachable. This study suggests enforcing these points and developing an overall strategy based on a stronger commitment, better planning, and enforced coordination. In this regard, additional targets and indicators can significantly contribute to the increased effectiveness of governance measures.

Striving for a Societal Perspective: A Framework for Economic Evaluations When Costs and Effects Fall on Multiple Sectors and Decision Makers.

In most societies, resources are distributed by individuals acting in markets and by governments through some form of collective decision-making process. Economic evaluation offers a set of tools to inform collective decisions by examining the resource requirements and outcomes of alternative policies. The ‘societal perspective’ has been advocated, but less consideration has been given to what this should include and its practical implementation. This paper presents a framework for economic evaluation of policies with costs and outcomes falling on different sectors (e.g. health, criminal justice, education) and involving different decision makers. It extends the ‘impact inventory’ developed by the Second Panel on Cost-Effectiveness in Health and Medicine by considering all affected individuals and reflecting how outcomes attributed to an intervention can be compared with outcomes forgone as a result of resources not being available for other purposes. The framework sets out the series of assessments to be made, distinguishing points at which value judgements feed into the evaluation, and the implications of alternative judgements. These assessments reflect the institutional arrangements of public bodies, for example, their funding, the outcomes they consider important and their relative valuations of these outcomes. By avoiding the use of an abstract ‘societal perspective’, the contribution of the framework is to inform multiple decision makers with different objectives and provide practical guidance on overall societal impact.Electronic supplementary materialThe online version of this article (10.1007/s40258-019-00481-8) contains supplementary material, which is available to authorized users.

Community-Based Tourism in Developing Countries: A Framework for Policy Evaluation

Many developing countries have recognized the potential of their natural resources for the development of tourism. However, the policies designed to provide a framework for socially inclusive and ecologically sound tourism turn out to be weak in fostering community-based tourism (CBT). There is a spectrum of conditions that lead to success or failure of CBT initiatives that should be accounted for by national policies to ensure meaningful community engagement. This paper proposes a framework for the evaluation of tourism policy documents based on factors that facilitate and constrain CBT in developing countries. The framework was applied to seven tourism policy documents in Colombia using a content analysis approach. The results showed that the policies are weak in providing a base for community engagement in CBT. International experiences indicate that the barriers to CBT are similar throughout the world; thus, the framework is useful in the evaluation of policy documents in other developing countries.

Understanding and improving discretionary judgment and decision-making in child protection practice: Towards a whole-of-system policy evaluation

Discretionary judgment is a necessary and desirable attribute of child protection practice and decision-making. Increasingly, approaches towards accountability in child protection services act to constrain the use of practitioner discretionary judgement through ever increasing layers of standardisation and technical-rational approaches to practice. This situation is at odds with the need for professionals to adaptively respond to practice environments that are characterised by uncertainty and complexity. At the same time, there are known weakness and problems that are reported in the decision-making literature, begging questions about how to best support and evaluate for effective and accountable discretion and decision-making across a whole system. In this paper discretion is conceptualised as a structural and epistemic phenomena that is constrained and restricted under the weight of standardisation. A five-part conceptual framework for a systems approach to policy evaluation is presented, and it is argued that this framework would support the capacity for effective discretionary judgement and decision-making to emerge as a property of the system overall. This paper is a theoretical and conceptual argument for a systemic policy evaluation framework that is supportive of discretionary judgment and decision-making in child protection systems shifting the emphasis away from technical-rational compliance.

Impact of School Type On Student Academic Achievement

This study aims to investigate the causal effects of school type on the student achievement. Schools type involves two categories: public schools and private schools; whereas student achievement is defined in terms of an overall measure named as Basis for Admission Score (BAS), which is a weighted score reflecting both the grades obtained from courses in each school and the points obtained at nation-wide centralized exams. Factors used as controls in the study include gender of the student, parental attributes (are they alive and live together; their levels of education and occupations), type of the house (own/rent/public housing), separate room of the student, city the family lives in and the level of development of the geographical region. This study utilizes a dataset comprising 3,752,374 secondary school students, which covers all of the student population of Turkish secondary schools within 2014-2016 period. This comprehensive dataset is utilized for the first time in such a study. At the first step, we present the literature on the effects of school type on academic achievement measured by test scores. This literature can be traced back to 1960’s. On the other hand, a second line of literature, which focuses on the evaluation of causal effects of policies and of programs has progressed swiftly starting from 1980’s and promises an important methodological framework for evaluation of policies and programs in education science. The methodology of this study is designed by bringing together the mentioned two lines of research. Methodology involved the application of regression adjustment, inverse probability weighting, and exact matching techniques in a complementary style in order to ensure the robustness of estimations. Under the assumptions discussed in detail in the study, we have found that school type has significant impact on student achievement in Turkey. Being a private school student instead of public school student leads to 87 points increase (29.6%) on average in BAS score. It is found that school type has a comparatively larger effect in Turkey compared to other country examples. Based on the findings of the study, a set of research topics are suggested with the objective of improving equality of opportunity in education and of identifying new policies to improve quality of education in public schools.

Impact of Workforce Flexibility on Customer Satisfaction: Empirical Framework & Evidence from a Cleaning Services Platform

Problem definition: Contrary to classic applications of matching theory, in most contemporary on-demand service platforms, matches can not be enforced because workers are flexible – they choose their tasks. Such flexibility makes it difficult to manage workers while keeping customers satisfied. We build a framework to compare platform matching policies with less flexible and more flexible workers, and empirically quantify by how much worker flexibility hurts customer satisfaction and customer equity. Academic/Practical relevance: In academic literature, there is no established framework that allows for the comparison of matching policies in on-demand platforms. Further, the link between worker flexibility and customer satisfaction is understudied. Methodology: We propose a tripartite framework for empirical evaluation and comparison of the operational policies with different degrees of worker flexibility. Step 1: Predictive modeling of customer satisfaction based on estimation of individual unobservable characteristics: customer difficulty and worker ability (item-response theory model). Step 2: Evaluation of the effect of matching policy (under a given level of flexibility) on customer satisfaction (bipartite matching). Step 3: Quantification of the associated monetary impact (customer lifetime value model). Results: We apply our framework to the dataset of one of the world's largest on-demand platforms for residential cleanings. We find that customer difficulty and cleaner ability are good predictors of customer satisfaction. Granting full flexibility to workers reduces customer satisfaction by 3% and customer lifetime revenue by 0.2%. We propose a family of matching policies that provide sufficient flexibility to workers, while alleviating 75% of the detrimental effect of worker flexibility on customer satisfaction. Managerial implications: Our results suggest that, in platforms with flexible workforce, the presence of worker and customer heterogeneity reduces customer satisfaction through matching inefficiency. Our empirical framework helps practitioners to decide on the right level of worker flexibility and the means for achieving it.

A water footprint management framework for supply chains under green market behaviour

Abstract Freshwater sustainability has been emerging as a critical corporate social responsibility issue for leading businesses worldwide. An increasing number of companies disclose and evaluate their water footprint, embracing an end-to-end supply chain perspective. In this work, we first propose a strategic system dynamics model for capturing the long-term impact of various water management policies on consumptive water footprint and supply chain profitability under diverse scenarios of consumers' demand patterns depending on blue water-related efficiency (i.e. green market behaviours). We apply the proposed simulation model on a specific wine supply chain and demonstrate that, under specific green market behaviour conditions, technological interventions in agriculture (e.g. drip irrigation) are more environmentally and economically sustainable compared to industrial water recycling and reuse, while the combined application of both policies is even more effective. Additionally, the impact of water-efficient policies on supply chain financial performance is even more potent under increased consumers' environmental sensitivity. Based on the managerial insights obtained from our analysis, as well as current corporate drivers and goals, we finally propose a supporting methodological framework for the ex-ante evaluation of eco-efficient supply chain water management policies, considering the impact of the green market behaviour on water-related and economic sustainability.

Comparing third party policy frameworks: Regulating third party electoral finance in Canada and the United Kingdom

Deciding how to regulate money during elections is a critical policy choice faced by every democracy. Over the last two decades, both the United Kingdom and Canada have implemented substantial revisions to their electoral laws, including policy measures designed to regulate third party spending. Despite similar policy objectives, the countries’ approaches to regulation differ, leaving the potential for significant variation in third party spending outcomes. These differences between countries, with otherwise very similar policy goals and systems of government, provide a unique opportunity to build and test a comparative policy evaluation framework for third party campaign spending. By examining these differences and similarities, this article builds a framework for election policy evaluation that can be adapted to serve as a template for future policy evaluation, facilitating comparative research.

Positioning and Priorities of Growth Management in Construction Industrialization: Chinese Firm-Level Empirical Research

The purpose of this research is to quantitatively evaluate the growth phase, position, and priorities of the industrialization policy management of the construction industry at firm level. The goal is to integrate quantitative dynamics into the policy-making process for sustainable policy development in future China. This research proposes an integrated framework, including growth management model and industrial policy evaluation method, to identify the challenges of construction industrialization and policy management. The research applies the mixed system method, which includes entropy method and average score method, to analyze the growth stage and major impact indexes targeting 327 survey samples. The empirical results show that the proposed conceptual framework and policy evaluation method could effectively determine the growth position and directions of the construction industrialization. For verification purpose, the study uses the local industry data from Shaanxi Province, China. The calculation results substantiate that the construction industry is in the middle section of the third growth phase. The comparison of the results from statistical methods shows that the local construction industry still needs substantial effort in policy management to improve its sustainable industrialization level. As countermeasures, the policy priorities should concentrate on: (1) enhancing effective cooperation among universities, research institutions and enterprises; (2) improving actions towards technology transfer into productivity; and (3) encouraging market acceptance of construction industrialization. This research complements the existing literature of policy evaluation of construction industrialization. Moreover, it provides theoretical and operational steps on industry policy evaluation and growth management framework, with accurate and ample data analysis on firm-level survey. Researchers and policy makers can use this research for further extensions of policy management for construction industrialization.

Re-categorizing innovation policy according to broad-based innovation

ABSTRACTThis study lays the foundations for a theory of broad-based innovation and its potential role in innovation policy. The concept of broad-based innovation and its use in policy documents is discussed, followed by a discussion of the components of such innovation. This article proposes a re-categorization of modes of knowledge production in innovation activities to describe the need to more consistently consider and include different types of innovation activities in innovation policy. The study seeks to outline a novel theoretical structure that bridges a number of separate subfields in the study of innovation, providing a comprehensive treatment of the differential innovation puzzle. The study contributes by providing this novel theoretical framework for future planning and evaluation of innovation and innovation policies, systems and strategies.