Policy Parameters Research Articles

Batch Reinforcement Learning With a Nonparametric Off-Policy Policy Gradient.

Off-policy reinforcement learning (RL) holds the promise of better data efficiency as it allows sample reuse and potentially enables safe interaction with the environment. Current off-policy policy gradient methods either suffer from high bias or high variance, delivering often unreliable estimates. The price of inefficiency becomes evident in real-world scenarios such as interaction-driven robot learning, where the success of RL has been rather limited, and a very high sample cost hinders straightforward application. In this paper, we propose a nonparametric Bellman equation, which can be solved in closed form. The solution is differentiable w.r.t the policy parameters and gives access to an estimation of the policy gradient. In this way, we avoid the high variance of importance sampling approaches, and the high bias of semi-gradient methods. We empirically analyze the quality of our gradient estimate against state-of-the-art methods, and show that it outperforms the baselines in terms of sample efficiency on classical control tasks.

Adaptive Event-Triggered Transmission Scheduling in Rate-Limited Multiloop Remote Control

This article studies the dynamic transmission-scheduling problem of rate-limited networked control systems with multiple loops. It is critical to develop an optimal-scheduling policy because, due to limited spectrum resources and energy budgets, only partial subsystems may have access to shared channels at each time step to update the plant states affected by stochastic disturbances. An adaptive event-triggered stochastic scheduling policy is proposed based on the one-step-ahead comparison error of state estimation between the sensor and controller. By formulating the scheduling problem as a constrained multiagent partially observable Markov decision process, a multiagent reinforcement learning algorithm is proposed to search for the optimal parameters of the scheduling policy. And an edge-assisted learning architecture is introduced to facilitate its implementation. An explicit performance index of the optimal scheduling is further obtained, revealing the effects of system disturbances, observation noises, and the communication rate constraint. Moreover, two numerical examples are given to validate the proposed scheduling policy, showing that it outperforms several competitive scheduling schemes. Due to the low computational complexity, the proposed scheduling may have an advantage in large-scale heterogeneous control systems, e.g., the flow and pressure control in fluid transport pipelines.

A Point Prevalence Study of the Use of Antibiotics in Six Tertiary Care Hospitals in the Kathmandu Valley, Nepal

Background Point prevalence survey (PPS) on antibiotic use developed by the WHO has already been used in many hospitals globally.
 Objective To obtain information on antibiotic prescription using point prevalence survey methodology in six private hospitals in the Kathmandu valley.
 Method This descriptive cross-sectional study was completed during 20th July to 28th July 2021 using point prevalence survey methodology. The study was conducted among inpatients admitted at or before 8:00 AM on the day of survey in various wards. Data was presented as frequencies and percentages.
 Result Maximum number of patients were above 60 years [34 (18.7%)]. Number of male and female participants were equal [91 (50%)]. Only one antibiotic was used in 81 patients (44.5%) followed by two antibiotics in 71 (39%) patients. Duration of prophylactic antibiotic use was one day in 66 (63.7%) patients. Blood, urine, sputum, and wound swabs were the common samples for culture. Cultures were positive for 17 (24.7%) samples. The common organisms isolated were E. Coli, Pseudomonas aeruginosa and Klebsiella pneumoniae. Ceftriaxone was the most used antibiotic. Drug and therapeutics, infection control committee and pharmacovigilance activities were present in 3/6 (50%) study sites. Antimicrobial stewardship was present in 3/6 (50%) and microbiological services was present in all hospitals. Antibiotic formulary and antibiotic guideline were present in 4/6 sites and facilities to audit or review surgical antibiotic prophylaxis choice in 2/6 (33.3%) sites, facility to monitor antibiotic use in 4/6 (66.6%) and cumulative antibiotic susceptibility reports in 2/6 (33.3%) study sites.
 Conclusion Ceftriaxone was the most used antibiotic. E. Coli, Pseudomonas aeruginosa and Klebsiella pneumonia were the commonly isolated organisms. Not all parameters for infrastructure, policy and practice and monitoring and feedback were present at the study sites.

Building back better through feminist entrepreneurship policy

PurposeMost feminists policies are aspirational. Deficiencies include vague terms of what constitutes ‘feminist’ within policy, ambiguous investment criteria, lack of consultation and the use of the binary definition of gender negating gender-diverse people (Tiessen, 2019). The purpose of this study is to identify parameters that characterize feminist entrepreneurship policies and to advance recommendations to operationalize these policies.Design/methodology/approachThe COVID-19 pandemic has unveiled fragilities in the socio-economic gains that women entrepreneurs have achieved. Gender-regression is, in part, the product of entrepreneurship policies that fail to recognize the nature and needs of women entrepreneurs. To inform recovery measures, this article considers two research questions: what are the parameters of feminist entrepreneurship policies? and how can parameters of feminist entrepreneurship policy be operationalized in pandemic recovery measures? To inform the questions, the study draws on the academic literature and thematic analysis of three collective feminist action plans to operationalize ten parameters that characterize feminist entrepreneurship policy.FindingsSupplanting ‘feminist’ for women in the construction of entrepreneurship policies, without specifications of how parameters differ dilutes government's efforts to achieve gender quality and women's economic empowerment. To inform policy, recommendations of three feminist recovery policies clustered under seven themes: importance of addressing root causes of inequality; need to invest in social and economic outcomes; economic security; enhancing access to economic resources; investment in infrastructure; inclusive decision-making; and need for gender disaggregated data to inform policy. Differences in policy priorities between collective feminist recovery plans and the academic literature are reported.Research limitations/implicationsThe parameters of feminist entrepreneurial policy require further interpretation and adaptation in different policy, cultural and geo-political contexts. Scholarly attention might focus on advisory processes that inform feminist policies, such as measures to address gender-regressive impacts of the COVID-19 pandemic. Research is also needed to understand the impacts of feminist policies on the lived experiences of diverse women entrepreneurs. Limitations: The study design did not incorporate viewpoints of policymakers or capture bureaucratic boundary patrolling practices that stymie feminist policies. Thematic analysis was limited to three feminist recovery plans from two countries.Practical implicationsRecommendations to operationalize feminist entrepreneurship policies in the context of pandemic recovery are described.Originality/valueTen parameters of feminist entrepreneurship policy are explored. The conceptual study also advances a framework of feminist entrepreneurship policy and considers boundary conditions for when and how the parameters are applicable.

Reinforcement learning‐based control to suppress the transient vibration of semi‐active structures subjected to unknown harmonic excitation

AbstractThe problem of adaptive semi‐active control of transient structural vibration induced by unknown harmonic excitation is studied. The controller adaptation is attained by using a specially designed reinforcement learning algorithm that adjusts the parameters of a switching control policy to guarantee efficient dissipation of the structural energy. This algorithm relies on an efficient gradient‐based sequence that accelerates the learning protocol and results in suboptimal control. The performance of this method is examined through numerical experiments for a span structure that is equipped with a semi‐active device of controlled stiffness and damping parameters. The experiments cover a selection of control learning scenarios and comparisons to optimal open‐loop and heuristic state‐feedback control strategies. This study has confirmed that the developed method has high stabilizing performance, and the relatively low computational burden of the incorporated iterative learning algorithm facilitates its application to multi–degree‐of‐freedom structures.

Declaring income versus declaring taxes in tax compliance experiments: Does the design of laboratory experiments affect the results?

Laboratory experiments are frequently criticised, in part because of the sensitivity of the results to specific features of the design. This paper addresses an important question regarding the key aspect of the experimental environment: How should the dependent variable – participants’ choices – be operationalised? For the specific context of laboratory research on income tax compliance, we compare the effects of the two most common operationalisation types: the declaration of gross income versus the declaration of tax payment. It is found that compliance is higher when participants indicate their tax payment than when they declare their income. It is also discovered that the effects of the three policy parameters of the economic model (the tax rate, audit probability and fine rate) are stronger when participants declare their taxes than when they declare their income. These results are relevant for interpreting prior and future experimental evidence on tax compliance and can explain some contradictory previous findings. More broadly, this study suggests that the results of laboratory experiments may depend on specific features of the experimental design, which proposes a strong need for more systematic methodological research.

The development of renewable energy industry under renewable portfolio standards: From the perspective of provincial resource differences

A low-carbon energy policy is an effective measure for promoting the development of renewable energy(RE). China has introduced renewable portfolio standards(RPS) and voluntary subscription for tradable green certificates(TGCs), but the market response is not significant. China's electricity trading functions at provincial-level regions, and there are differences in resources between the provinces. New policies need to consider the characteristics of each province to reduce market implementation risk for introduction and implementation. Therefore, based on the provincial characteristics of energy resource endowment in China, we construct a system dynamics(SD) model of the coupling of the RE market, consumption above quota(CAQ), and TGC market based on the structure of the sending-end grid(SG) and receiving-end grid(RG). Three types of TGC models were designed based on the scope of the TGC market. Finally, the impact of policy parameters such as the RPS target, inter-provincial transmission capacity, TGC validity period, and technology cost on the development of RE industries in both SG and RG was analyzed. The results show that the TGC market to be developed independently in various provinces in the initial stage and later expanded into a unified market. Increased RPS requirements, increased inter-provincial transmission capacity, and lower RE costs can enhance policy implementation.

Optimizing design and performance assessment of a community-scale hybrid power system with distributed renewable energy and flexible demand response

Modern urban energy system with renewable energy sources has become an effective solution to enhance energy security and promote carbon neutrality. This paper presents a holistic framework for the optimal design and management of such a flexible hybrid power system at the community scale. This framework integrates the potential assessment of rooftop PV and waste biomass, a general optimization approach with multi-perspective performance evaluation, as well as the impacts analysis of energy policy, demand response, and uncertain economic parameters. The proposed framework was applied to a small neighborhood area in Beijing, China. It was found that 100% utilization of available biomass and rooftop PV would satisfy 73% of local electricity demand with the levelized cost of electricity of 0.1030 $/kWh and the embodied carbon emission of 0.5416 kg/kWh. The flexible shiftable load could lead to lower peak load and less electricity supply cost, and help to match the energy demand with the dispatchable rooftop PV output. However, the economic benefits caused by implementing demand response are more obvious than the environmental benefits. Furthermore, a higher electricity tariff and lower capital cost of PV technology would facilitate the investment in rooftop PV and achieve carbon emission reduction targets.

Designing Affirmative Action Policies under Uncertainty

We study university admissions under a centralized system that uses grades and standardized test scores to match applicants to university programs. In the context of this system, we explore affirmative action policies that seek to narrow the gap between the admission rates of different socio-demographic groups while still accepting students with high scores. Since there is uncertainty about the score distribution of the students who will apply to each program, it is unclear what policy would have the desired effect on the admission rates of different groups. We address this challenge by using a predictive model trained on historical data to help optimize the parameters of such policies. We find that a learned predictive model does significantly better than relying on the ideal parameters for the last year. At the same time, we also find that a large pool of historical data yields similar results as our predictive approach for our data. Due to the more complex nature of the predictive approach, we conclude that a simpler approach should be preferred if enough data is available (e.g., long-standing, traditional university programs), but not for newer programs and other cases in which our predictive strategy can prove helpful.

Deep energy retrofits: How effective and robust are policy instruments?

Motivating landlords to deep energy retrofits is challenging, particularly if only tenants benefit from energy savings. We introduce the concept of net-benefit curves to quantify how the reduction in CO2 emissions on the macro level and the economic viability-based retrofit decision on the micro level are connected, and how climate policy instruments influence this connection. Applying this method to a typical multi-unit dwelling, we derive the parameter levels at which different policy instruments start to show an effect on the building owner’s decision to retrofit. The shape of the net-benefit curve suggests a tipping-point effect, where a slight change in policy parameters will lead from no retrofit at all to substantial retrofitting. Further policy tightening will not lead to a significant further effort, but make the retrofit decision more robust, that is, narrow the range of economically attractive retrofits. As policy instruments, we analyze prototypical systems that share energy savings, retrofit costs, or carbon taxes between landlord and tenants.

Crude oil footprint in the rapidly changing world and implications from their income and price elasticities

Oil demand's income and price elasticities are important behavioral parameters for policy making. Despite its popularity in research, few studies investigated the elasticities of consumption-based oil demand, i.e., the amount of crude oil required to meet a nation's final demand regardless of the location of extractions, namely the oil footprint. Here we quantified the oil footprint of 49 countries/regions from 1995 to 2017 and estimated the elasticities of oil footprints using the panel Autoregressive Distributed Lag model. The results reveal the oil connections among countries hidden in the non-oil trade: the United States, China, and Japan imported a large amount of virtual oil embodied in the commodities and services, while Canada and Russia are the dominant suppliers. The elasticity estimations on 30 OECD countries show that oil footprints are more responsive to income increases in the long run than their short-run counterpart, with the elasticities around 0.75 and 0.48, respectively. Tariffs on oil products might not curb the oil footprint as the price elasticities are not robustly significant or negative. Moreover, the divergent elasticities of oil footprint by consumption categories highlight that more attention should be paid to the surge of oil demand embodied in construction, manufactured products, and services.

The design of the intertemporal trading ratio of carbon quotas

Carbon emission trading achieves both economic and environmental benefits. The imbalance between quota supply and demand restricts the development of the carbon market. The key to solving the issue is to set a reasonable intertemporal trading ratio. Currently, the study of carbon trading ignores the influence of environmental externalities, which leads to the deviation of market participants' behavior choices and government policy making. Considering the influence of non-price factors, taking environmental externalities as the cut-in point, use the optimal control method to analyze the issue of carbon quota conversion, and to make clear the driving factors and mechanism of the intertemporal trading ratio. Results show that the condition of choosing an optimal intertemporal trading ratio is when the value path of per-unit carbon emissions is equal to the marginal cost path of emission reduction. The intertemporal trading ratio depends on the firm discount rate, the natural attenuation rate of carbon emissions, the ratio of the social marginal damage value to the social marginal cost of emission reduction, and the combination effect among marginal cost of emission reduction, marginal profit of production from purchasing carbon quota and price of carbon transaction. The effect of different driving factors on the intertemporal trading ratio is heterogeneous in different situations. The results provide a reference for setting reasonable parameters of carbon trading policy.

An efficient dynamic access control and security sharing scheme using blockchain

This study seeks to understand the role of institutions and organizations that have used cloud service providers to store and share data as ensuring third-party access to storage is a major challenge to avoid data theft and unwanted access. Hence, in this paper, Blockchain-Based Data Access and Secure Sharing Method (BDASS) is introduced to enhance security processes related to personal data through data access control and secure sharing method, the proposed method uses blockchain aggregation, file system (IPFS), dynamic access control (DAC), and ciphertext-attribute-based encryption (CP-ABE) to enhance the security of personal data. To keep the owner safe, a blockchain-based DAC is designed. To keep data storage and sharing secure, the blockchain-based CP-ABE is designed. In this proposed methodology, the data owner encrypts the data they have stored in IPFS, thus enhancing data security, which has been improved with the help of CP-ABE regarding detailed access policy and data owner. Policy parameters are managed by the DAC. In the proposed methodology, the data owner uses the blockchain to control security and access to the data. Finally, the paper has come up with a set of findings in order to achieve data security and access control for the data owner through the blockchain-based approach. To evaluate the performance of the proposed method, MATLAB was used. The proposed technology also contrasts with existing technologies, such as the Blockchain-Based Security Sharing Scheme for Personal Data (BSSPD) as well as the Rivest-Shamir-Adleman Algorithm (RSA) and Elliptic Curve Cryptography (ECC).

Toward Large-Scale Hybrid Edge Server Provision: An Online Mean Field Learning Approach

The efficiency of a large-scale edge computing system primarily depends on three aspects: i) edge server provision, ii) task migration, and iii) computing resource configuration. In this paper, we study the dynamic resource configuration for hybrid edge server provision under two decentralized task migration schemes. We formulate the dynamic resource configuration as an online cost minimization problem, aiming to jointly minimize performance degradation and operation expenditure. Due to the stochastic nature, it is an online learning problem with partial feedback. To address it, we derive a deterministic mean field model to approximate the stochastic edge computing system. We show that the mean field model provides the increasingly accurate full feedback as the system scales. We then propose a learning policy based on the mean field model, and show that our proposed policy performs asymptotically as well as the offline optimal configuration. We provide two ways of setting the policy parameters, which achieve a constant competitive ratio (under certain mild conditions) and a sub-linear regret, respectively. Numerical results show that the mean field model significantly improves the convergence speed. Moreover, our proposed policy under the decentralized task migration schemes considerably reduces the operating cost (by 23%) and incurs little communication overhead.

Dynamic model of CO2 reduction policies: Another step towards a cleaner generation expansion

Cap-and-trade mechanisms and emission taxes are becoming increasingly widespread CO2 reduction policies in liberalized electricity markets. To set the design parameters of a CO2 reduction mechanism, analytical optimization frameworks are a commonly used tools by policy makers. However, such frameworks can suffer from issues related to emerging dynamic behaviors in electricity markets. This paper focuses on the relation between the parameter settings of CO2 reduction policies and dynamics of the electricity market. The authors present an analytical framework based on the system dynamics modeling that improves the understanding of the relation between the quantified impacts of CO2 reduction policies and parameter settings. The framework is based on the insight that the quantified impacts of CO2 reduction policies are attributed to the policy design, delays and mutual interactions inherent in the electricity market. The framework addresses both the quantified impacts and the rationale behind dynamic behaviors in the electricity market after implementing CO2 reduction policies. The framework is based on a causal loop diagram, illustrating the causal relation between different components within a system of electricity market. It offers policy makers a tool to assess the interactions between policy parameters and the structure of the electricity market. The framework is simulated under a power system using different scenarios of CO2 reduction policies and illustrates possible quantification of cleaner production, outperforming the base case scenario up to 63% CO2 reduction.

Flexible drinking water pumping to provide multiple grid services

Drinking water distribution networks (WDNs) can be operated as flexible, controllable loads. In this paper, we consider using WDNs to provide local and grid level services simultaneously to the power grid. We formulate a robust water pumping problem to determine the amount of voltage support and frequency regulation that can be provided subject to network constraints while managing power demand uncertainty. We tractably reformulate the problem as a sequential optimization problem and solve for the scheduled water pumping operation, the frequency regulation capacity, and the optimal control policy parameters that update the pump operation based on the frequency regulation signal and power distribution network demand forecast error. We demonstrate our approach through detailed case studies. Additionally, we evaluate the performance of the reformulation and discuss the benefits and trade-offs of WDNs providing multiple services.

A scalable anticipatory policy for the dynamic pickup and delivery problem

Dynamic vehicle dispatching and routing problems can be tackled by using either reactive policies (that optimize the overall inconvenience on the pending requests) or anticipatory policies (that consider the possible future demands). The anticipatory policies reported in the literature are typically unsuitable for the large instances often encountered in the real-world, where the inter-arrival time can be as little as a few seconds. In this article, we present a new scalable anticipatory policy for the Dynamic Pickup and Delivery Problem which amounts to design routes for a fleet of vehicles that must service a set of pickup and delivery requests, characterized by different priority classes, arriving according to an unknown (possibly time-varying) stochastic process. The algorithm utilizes a parametric policy function approximation in which the best parameter setting is chosen on-line on the basis of a mapping between instance features and policy parameters learned off-line by using simulation experiments. Computational results on large-scale randomly-generated instances indicate that our anticipatory procedure outperforms two reactive approaches while keeping the computational burden at a level suitable for real-world usage.

The Impact of Government Dual Policy on the Selection of OEM’s Regulatory Strategy for Third-Party Remanufacturing

Remanufacturing plays an important role in the circular economy and environmental protection. In the development of the remanufacturing closed-loop supply chain (CLSC), the impact of government policy cannot be ignored. However, previous studies have rarely answered the impact of government dual policy (the policy combination of trade-old-for-remanufactured and carbon tax policy) on the optimal decisions of CLSC enterprises. To fill the gap, this work considers the remanufacturing modes and government dual policy simultaneously and investigates a CLSC of waste of electronic and electrical equipment (WEEE) consisting of an original equipment manufacturer (OEM) and a third-party remanufacturer (TPR). Here the OEM has three strategies for regulating third-party remanufacturing activities: outsourcing (Model O), authorization (Model A), and nonregulation (Model D). The results show that: (i) Whether the government implements the dual policy or not, Model O is the best choice for OEM. And compared with nongovernment policy intervention, government dual policy can improve the recovery rate of WEEE products and then effectively promote the recycling of waste resources; (ii) When the government dual policy parameters meet a certain condition, the dual policy could effectively increase supply chain members’ profits and the total profits of CLSC and boost market demand expansion; (iii) Compared with the other two remanufacturing modes, the CLSC under nonregulation strategy is the optimal remanufacturing mode for the whole society; meanwhile, OEM nonregulation strategy is the worst strategy for the environmental protection. In addition, when the government sets appropriate parameters, government dual policy is beneficial to the consumer, supply chain member, and the society simultaneously, but the impact of the dual policy on the environment is also related to the ratio of the two products’ impact on the environment.

Analysis of Loyalty of Individual Customers to Increase the Customer Focus of the Cargo Carrier

The customer focus of cargo carriers plays an extremely important role in the highly competitive market of transport and logistics services, influencing the willingness of companies to be flexible in the pricing policy and other parameters of provision of services when forming a personalised offer to a customer, allbowing to attract a range of new customers, to expand the pool of regular customers based on their brand loyalty. One of the modern tools intended to achieve this goal is the customer loyalty program. The objective of the article is to review and summarise the research and series of recent publications [1–4] by the author and other researchers to substantiate provisions of the customer loyalty program based on analysis and assessment of motives and wishes of customers themselves. To achieve this objective, besides the method of monographic analysis of publications on this topic, the practical part of the research was based on a customer survey. The object of the study was the RZD-Express service for transportation of groupage cargo provided by JSC RZD Logistic company, the subject of the study was the loyalty of private consignors. The task of survey of respondents was to study consumer preferences and motives for choosing such a service, to analyse their willingness to recommend this service to their entourage, the intention to maintain long-term relations with the company, as well as customers’ expectations from interaction with the company. The interpretation of the results of the survey shows the interest of respondents in availability of such a marketing tool as a loyalty program. The study permitted to suggest a system of indicators for the assessment of loyalty program, as well as to put forward proposals for formation of provisions of the loyalty program to improve quality of transport services.

Cluster analysis of tax indicators in Europien countries

Global trends indicate that at the current stage of European countries’ integration, the processes of unification of taxation and tax administration, and the convergence and harmonization of tax systems are taking place. Therefore, for thorough studying of these phenomena, it is important to classify countries according to the parameters of fiscal systems and tax policy. Our analysis of the tax indicators of European countries on the main groups of taxes according to the methodology of ESA 2010 develops the research in this field. The analysis was carried out by clustering methods in order to identify common and different features in the tax systems of European countries. This study covers 30 European countries for the periods 2018 and 2020. This paper focuses on the analysis of three main indicators of tax revenues. According to the ESA 2010 methodology, these are taxes on production and imports (D.2), current taxes on income and property (D.5), and net social contributions (D.61). Cluster analysis of the tax-to-GDP ratio was performed using a hierarchical agglomerative method and the method of k-means using software R Studio and STATISTICA 7.0. As a result, 5 clusters have been obtained. They are characterized by the following average values: 1) the cluster with the lowest total tax-to-GDP ratio, where income taxes predominate; 2) the cluster with the highest total tax-to-GDP ratio, high tax ratio on income and property, and low social contributions; 3) the cluster with an average tax-to-GDP ratio and the largest social contributions; 4) the cluster with an average tax-to-GDP ratio and the predominance of taxes on production and import; 5) countries, where all analyzed tax groups have a roughly equal ratio to GDP. The classification carried out for 2020 data revealed some insignificant changes in the clusters’ composition. These changes can be considered as a result of the tax policy to counteract the effects of the pandemic.