Policy Parameters Research Articles

Adaptive Lagrangian Policies for a Multiwarehouse, Multistore Inventory System with Lost Sales

Adaptive Lagrangian Policies for Inventory Control Consider the problem of managing inventory in a multiwarehouse, multistore setting where each store can be periodically replenished from potentially a set of nearby warehouses. This is a practically relevant problem, and its optimal policy is not easy to compute because of the curse of dimensionality. Existing works in the literature have proposed a policy based on Lagrangian relaxation of the original stochastic problem. This vanilla policy applies the control parameters derived from the Lagrangian model and has been shown to perform well in some cases. In “Adaptive Lagrangian Policies for a Multiwarehouse, Multistore Inventory System with Lost Sales,” we go one step farther and develop adaptive Lagrangian policies that update (at certain update times) the control parameters of the vanilla policy based on the historical demand realizations. We analytically show that our proposed adjustment significantly improves the performance of the vanilla policy.

Constrained minimum variance and covariance steering based on affine disturbance feedback control parameterization

AbstractThis paper deals with finite‐horizon minimum‐variance and covariance steering problems subject to constraints. The goal of the minimum variance problem is to steer the state mean of an uncertain system to a prescribed vector while minimizing the trace of its terminal state covariance whereas the goal in the covariance steering problem is to steer the covariance matrix of the terminal state to a prescribed positive definite matrix. The paper proposes a solution approach that relies on a stochastic version of the affine disturbance feedback control parametrization. In this control policy parametrization, the control input at each stage is expressed as an affine function of the history of disturbances that have acted upon the system. It is shown that this particular parametrization reduces the stochastic optimal control problems considered in this paper into tractable convex programs or difference of convex functions programs with essentially the same decision variables. In addition, the paper proposes a variation of this control parametrization that relies on truncated histories of past disturbances, which allows for sub‐optimal controllers to be designed that strike a balance between performance and computational cost. The suboptimality of the truncated policies is formally analyzed and closed form expressions are provided for the performance loss due to the use of the truncation scheme. Finally, the paper concludes with a comparative analysis of the truncated versions of the proposed policy parametrization and other standard policy parametrizations through numerical simulations.

Distributed entropy-regularized multi-agent reinforcement learning with policy consensus

Sample efficiency is a limiting factor for existing distributed multi-agent reinforcement learning (MARL) algorithms over networked multi-agent systems. In this paper, the sample efficiency problem is tackled by formally incorporating the entropy regularization into the distributed MARL algorithm design. Firstly, a new entropy-regularized MARL problem is formulated under the model of networked multi-agent Markov decision processes with observation-based policies and homogeneous agents, where the policy parameter sharing among the agents provably preserves the optimality. Secondly, an on-policy distributed actor–critic algorithm is proposed, where each agent shares its parameters of both the critic and actor for consensus update. Then, the convergence analysis of the proposed algorithm is provided based on the stochastic approximation theory under the assumption of linear function approximation of the critic. Furthermore, a practical off-policy version of the proposed algorithm is developed which possesses scalability, data efficiency and learning stability. Finally, the proposed distributed algorithm is compared against the solid baselines including two classic centralized training algorithms in the multi-agent particle environment, whose learning performance is empirically demonstrated through extensive simulation experiments.

POEM: Performance Optimization and Endurance Management for Non-volatile Caches

Non-volatile memories (NVMs) with their high storage density and ultra-low leakage power offer promising potential for redesigning the memory hierarchy in next-generation Multi-Processor Systems-on-Chip (MPSoCs). However, the adoption of NVMs in cache designs introduces challenges such as NVM write overheads and limited NVM endurance. The shared NVM cache in an MPSoC experiences requests from different processor cores and responses from the off-chip memory when the requested data is not present in the cache. Besides, upon evictions of dirty data from higher-level caches, the shared NVM cache experiences another source of write operations, known as writebacks . These sources of write operations: writebacks and responses, further exacerbate the contention for the shared bandwidth of the NVM cache, and create significant performance bottlenecks. Uncontrolled write operations can also affect the endurance of the NVM cache, posing a threat to cache lifetime and system reliability. Existing strategies often address either performance or cache endurance individually, leaving a gap for a holistic solution. This study introduces the Performance Optimization and Endurance Management (POEM) methodology, a novel approach that aggressively bypasses cache writebacks and responses to alleviate the NVM cache contention. Contrary to the existing bypass policies which do not pay adequate attention to the shared NVM cache contention, and focus too much on cache data reuse, POEM’s aggressive bypass significantly improves the overall system performance, even at the expense of data reuse. POEM also employs effective wear leveling to enhance the NVM cache endurance by careful redistribution of write operations across different cache lines. Across diverse workloads, POEM yields an average speedup of \(34\% \) over a naïve baseline and \(28.8\% \) over a state-of-the-art NVM cache bypass technique, while enhancing the cache endurance by \(15\% \) over the baseline. POEM also explores diverse design choices by exploiting a key policy parameter that assigns varying priorities to the two system-level objectives.

PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping

Proximal Policy Optimization algorithm employing a clipped surrogate objective (PPO-Clip) is a prominent exemplar of the policy optimization methods. However, despite its remarkable empirical success, PPO-Clip lacks theoretical substantiation to date. In this paper, we contribute to the field by establishing the first global convergence results of a PPO-Clip variant in both tabular and neural function approximation settings. Our findings highlight the O(1/√T ) min-iterate convergence rate specifically in the context of neural function approximation. We tackle the inherent challenges in analyzing PPO-Clip through three central concepts: (i) We introduce a generalized version of the PPO-Clip objective, illuminated by its connection with the hinge loss. (ii) Employing entropic mirror descent, we establish asymptotic convergence for tabular PPO-Clip with direct policy parameterization. (iii) Inspired by the tabular analysis, we streamline convergence analysis by introducing a two-step policy improvement approach. This decouples policy search from complex neural policy parameterization using a regression-based update scheme. Furthermore, we gain deeper insights into the efficacy of PPO-Clip by interpreting these generalized objectives. Our theoretical findings also mark the first characterization of the influence of the clipping mechanism on PPO-Clip convergence. Importantly, the clipping range affects only the pre-constant of the convergence rate.

Impact of three carbon emission reduction policies on carbon verification behavior: An analysis based on evolutionary game theory

Carbon tax and emission trading compound policy (CT&ET), carbon tax policy (CTP), and emission trading policy (ETP) support countries in achieving "dual carbon" goals. Carbon verification is essential to these policies' efficacy. Policy changes will affect system equilibrium and present unclear risks to policy implementation. This paper compared the impact of three carbon emission reduction policies on the behavior of carbon verification subjects under various conditions, three policies' benefits and drawbacks at different stages, and proposed a dynamic penalty mechanism to restrain subject infractions. Finally, the simulations revealed three policies of system evolution path and parameter sensitivity in various scenarios. CT&ET best guarantees carbon verification quality has a high-risk tolerance, and restricts infractions in most scenes. The excessively conservative carbon quota and severe collusion penalty make CTP optimal for limiting ECE false reports. ETP reduces collusion best when carbon quotas are appropriately allocated. Adjusting taxes and carbon prices, increasing verification levels, and reducing false reporting will be easier under CTP and ETP. Dynamic penalties are particularly effective in limiting infractions. Static penalties stabilize compliance behavior effectively. This study proposed a policy parameter framework and optimization mechanism to improve the carbon verification mechanism and accelerate the "dual carbon" goal.

New evidence on US monetary policy activism and the Taylor rule

AbstractWe provide new evidence about US monetary policy using a model that: (i) estimates time-varying monetary policy weights without relying on stylized theoretical assumptions; (ii) allows for endogenous breakdowns in the relationship between interest rates, inflation, and output; and (iii) generates a unique measure of monetary policy activism that accounts for economic instability. The joint incorporation of endogenous time-varying uncertainty about the monetary policy parameters and the stability of the relationship between interest rates, inflation, and output materially reduces the probability of determinate monetary policy. The average probability of determinacy over the period post-1982 to 1997 is below 60% (hence well below seminal estimates of determinacy probabilities that are close to unity). Post-1990, the average probability of determinacy is 75%, falling to approximately 60% when we allow for typical levels of trend inflation.

Burden of alcohol use and inclusion of alcohol use disorder medications in the essential medicine lists across 132 countries: An observational study

Harmful use of alcohol effects the health of the population. The treatment coverage of alcohol use disorders (AUD) varies among countries. The study aimed to determine the inclusion of AUD medicines in various national Essential Medicine Lists (EMLs) and its association with alcohol consumption. It was a secondary data analysis of alcohol consumptions and AUD related medicines in EML. Data were extracted from the WHO Global Essential Medicines database and the WHO Global Status Report on Alcohol and Health 2018. Data were extracted for 194 countries. Only 132 of 194 countries (68.0%) had EML, and among the 132 countries only 27.3% had included AUD medicines in their EML. Only 36 countries had included any of the AUD medicines in their EML. Disulfiram was included by 23 countries, while Acamprosate and Naltrexone was included by only four and 19 countries, respectively. Among the countries, 36.1% were from upper-middle income countries and 16.65 from low-income countries. The inclusion of AUD medicines in national EML was neither associated with alcohol consumption parameters nor the alcohol consumption related policy parameters. Considering the high prevalence of AUD and its complications, there is an urgent need to focus on including AUD medicines in national EML for making AUD treatment available and accessible across the world.

Stock returns and monetary policy stance

This paper investigates the structural relationships between monetary policy and stock returns. We build an asset pricing model incorporating a standard Taylor rule into a consumption-CAPM framework. Our model quantitatively explains the negative risk premium of expansionary monetary policy. As monetary policy plays a role of insurance, (expansionary) monetary policy beta is negatively related to covariances of returns with output gap and inflation theoretically and empirically. In addition, while systematic responses mainly account for the negative risk premium, pure monetary policy shocks have little influence under plausible calibrations. We also provide theoretical predictions on how the slope of the Phillips curve and the response parameters of monetary policy affect the cross-section of returns.

A simple heuristic for computing non-stationary inventory policies based on function approximation

We consider a finite-horizon periodic-review inventory system with fixed replenishment costs that faces non-stationary demands. The structure of the optimal control policy for this system has long been known. However, finding optimal policy parameters requires solving a large-scale stochastic dynamic program. To circumvent this, we devise a recursion-free approximation for the cost function of the problem. This translates into an efficient and effective heuristic to compute policy parameters that significantly outperforms earlier heuristics. Our approach is easy-to-understand and easy-to-use as it follows by elementary methods of shortest paths and convex minimization.

The dividend policy of business entities and the analogy of law

Author discovers uncertainty that generates ambiguous administrative and judicial law enforcement on the issue related to the limits of business entity freedom in the creation and implementation of dividend policy. Importance of such insufficiently clearly regulated by corporate legislation aspects of the dividend policy as the scope of corporation's powers and limitations in determining frequency of making decisions on dividend payments and timing of the actual distribution of profits among the corporation's participants is shown. Special relevance of the doctrinal study and regulatory clarification of the relevant legal parameters of the dividend policy of domestic corporations is associated by the author with the need to overcome the negative consequences of the sanctions impact of unfriendly foreign states. The presented author's approach is aimed at ensuring greater variability of corporate and legal interaction and increasing the efficiency of business entities. Special technical and legal tools were used in the study, including the method of analogy, as well as civilistic means of analysis and synthesis, comparison and generalization. The idea of the possibility and practicability of flexible negotiation of this uncertainty based on the normative construction of the analogy of law, on the basis of the general principles and meaning of civil legislation, as well as through the analogical use of explanations on understanding of the imperativeness and optionality of legal norms contained in the Resolution of the Russian Federation Supreme Arbitration Court Plenum dated March 14, 2014 No. 16 «On freedom of contract and its limits» is argued. As a result, conceptual admissibility of «nonmonotonic» dividend accruals, not linked to calendar quarters has been proved, as well as the possibility of large delays in the actual payment of declared dividends in comparison with the periods pre-established by norms. Main conditions for the implementation of such a measure of freedom in determining dividend policy are identified, namely its focus on achieving a balance of corporate interests and availability of an actually generated net profit for a business entity.

Effect of Platoon Formation Policy and Platoon Size for Connected Autonomous Vehicles on Mixed Traffic Capacity

Recent technological development allows a queue of vehicles to be driven safely close to each other, which increases road utilization and reduces air drag, thus, resulting in significant energy savings. The queue of vehicles that follow one another in close succession is called a vehicle platoon. In the early stages of connected autonomous vehicles (CAVs) platooning, CAVs are expected to form platoons at their departure point according to an appropriate platoon formation policy. However, the effect of platoon formation policy on traffic capacity is rarely considered. This paper analyzes the effect of platoon formation policy and platoon size on the capacity of mixed traffic flow with CAVs, autonomous vehicles, and human-driven vehicles. First, given platoon size distribution, we derive the probability mass function for the position of an arbitrary CAV within a platoon and obtain mixed traffic capacity as a function of CAV penetration rate and average platoon size. The platoon size is heavily influenced by platoon formation policies. So, the platoon size distributions are derived for several platoon formation policies and are applied to obtain the mixed traffic capacities under the platoon formation policies. Some numerical examples are also provided to demonstrate how the key parameters of platoon formation policies affect the mixed traffic capacity.

A data-driven approach for optimal operational and financial commodity hedging

Commodity price risk management has been subject to various modeling and optimization approaches. Recently, data-driven policies focusing on the decision rather than prediction quality have been developed to overcome price model misspecification. Yet, in the context of data-driven commodity purchasing, the existing literature either considers anticipatory inventory management or forward contracting where the decision frequency corresponds to the maturity of the traded contracts. We prove the optimality of a novel procurement policy combining operational and financial instruments with decision granularities independent of the derivative’s maturity. A mixed-integer programming model is developed to train policy parameters efficiently. We study the implications of policy complexity for learning-stability and out-of-sample generalization. Finally, we backtest the data-driven policy on real market data of four major commodities (i.e., copper, nickel, corn, and soybean) over ten years and show that the average savings potential of a combined financial and operational procurement policy compared to single-instrument strategies is up to 6.38% for corn where warehousing can efficiently mitigate price seasonality. The approach hedges corn and soybean commodities more efficiently through inventories while copper and nickel can be hedged efficiently by leveraging available financial instruments. Best model results are identified for a decision granularity with fewer parameters as high-frequent decisions deteriorate learning stability and model generalization.

Disaster preparedness for tourist village organizers in disaster prone area of Merapi Yogyakarta

Background: A number of tourist villages in Sleman Regency are located in Disaster Prone Area (DPA). To minimize the risks of disasters, preparedness efforts are required. Some of the components of preparedness are: knowledge of disaster risks, polices and guidelines in relation to the preparedness to anticipate the occurrence of natural disaster, emergency plan for natural disasters, disaster alert system, and resources mobilization. Tourist Villages of Pentingsari, Nganggring, and Tunggularum are located in Merapi Disaster Prone Area (DPA). These three tourist villages were selected due to their high vulnerability and risk potential of Merapi eruption due to their locations in DPA III and DPA II. Methods: This is a qualitative study with case study design. The subjects of the study were tourist village organizers who had knowledge and understanding of tourist villages with the criteria of their positions in the tourist village organization. The sample taking technique applied to the subjects was the purposive sampling technique. Results: The level disaster preparedness in terms of knowledge parameter among some tourist village organizers was limited since only few things were understood. In terms of policy and guidelines parameter, these components were not available for the tourist village organizers. With regard to the parameter of emergency plan for the organizers, this component was limited since there were some lack of preparation in a number of components. In relation to the parameter of disaster alert system for the tourist village organizers, this aspect was found to be good. With regard to the parameter of resource mobilization for the tourist village organizers, it was found that this aspect was unavailable since there were not enough decent facilities. Conclusions: the level of disaster preparedness in the disaster prone area of Merapi Yogyakarta was still limited in dealing with the threats of Mount Merapi eruption disaster.

Dimensional traps in evasion models and their effects on industrial structure

Size-dependent firms’ monitoring by the state leads to sub-optimal results in terms of combating evasion and efficient allocation of investment by firms. While Coppier, Michetti and Scaccia (2022) describe how such a policy can be a source of dimensional trap in a single-firm evasion model, the present paper moves to a heterogeneous context, considering industrial structures composed of many firms with different sizes. We then propose a discrete-time nondeterministic dynamic model to describe the consequences of size-dependent policies and potential dimensional traps on industrial structure and its evolution over time. We show that an unwise choice of policy parameters may determine a long-run equilibrium industrial structure characterized by a small number of large firms and a plethora of small firms, with the latter being marked by inefficient resource allocation and noncompliant behavior with regard to tax regulations. These results are robust to different choices of the initial industrial structure suggesting alternative policy indications.

Ethanol Fuel Blending Program in India: Analysis of Environmental, Economic, and Policy Aspects Using System Dynamics Approach

India initiated its ethanol fuel blending program (EBP) two decades ago to enhance energy security, reduce crude imports, and promote low‐carbon transportation. However, despite government initiatives and policies, the EBP has made slower progress than anticipated. For the long‐term adoption and success of the EBP, the following critical areas must be analyzed: integrated ethanol production from multiple feedstocks, demand and linkage to industries requiring ethanol, impact on the environment and revenue prospects, and evaluation of the policy measures adopted. This study addresses these topics by analyzing the interaction between various industries (demand) and ethanol production from multiple sources (supply) using system dynamics modeling. Simulation and scenario analysis have been used to evaluate the environmental and economic performance of ethanol blends under the influence of various policy parameters. The findings indicate that, contrary to the conventional belief, the production of ethanol directly from sugarcane juice does not significantly threaten food security. Higher blending ratios yield enhanced environmental benefits and revenues in the short term, but these are outweighed by the long‐term benefits of lower blending ratios. The findings also indicate that encouraging second‐generation ethanol production from rice stalks and increasing the blending ratios will reduce CO2 emissions. However, the goals set for blending cannot be achieved until measures to diversify feedstocks and improve the infrastructure for ethanol production are scaled up.

From People to Public Policy: the Role of Human Capital in Sustainable Post-War Development of Ukraine

The crucial role of human capital in enhancing the effectiveness of public governance systems and the principles of new public policy is investigated in the context of sustainable development and the most relevant United Nations Sustainable Development Goals. The analysis begins with an overview of human capital as a fundamental asset for socio-economic development, highlighting its impact on innovation, productivity, and overall economic growth. The research methodology includes the application of several general scientific methods for literature review and concise analysis of official data, emphasizing the interrelationship between investments in human capital and improved governance outcomes. Key findings indicate that regions with higher levels of education, health, and skills among the population tend to demonstrate more robust governance and sustainable development indicators, as well as resilience to severe shocks (such as war), and this represents a manageable policy parameter for governments. The paper also discusses challenges and opportunities for optimizing human capital, including policy recommendations for education, healthcare, and workforce development to support sustainable development in line with the UN’s rationale. Through its analysis, the article contributes to strategic planning and public policy formulation, which is crucial for more effective utilization of human capital for regional development and better public governance.

TERL: Two-Stage Ensemble Reinforcement Learning Paradigm for Large-Scale Decentralized Decision Making in Transportation Simulation

Transportation simulation is non-trivial due to the co-existence of thousands of heterogeneous decision makers (or vehicles). Such large-scale decision making is intrinsically a complex decentralized problem, the resolution of which is at the forefront of transportation simulation. Despite many physical or mathematical models proposed to date, underlying them is usually a set of universal rules plus some random perturbations to characterize vehicular movements. Inspired by the decision-making mechanism of rational human beings (i.e., learning iteratively from experience), this study proposes a novel two-stage ensemble reinforcement learning (TERL) paradigm for large-scale decentralized decision making in transportation simulation, in order to enhance the computational efficiency and thus scalability of RL to practical applications at scale. After establishing a problem-specific Markov decision process, the first stage utilizes clustering to group heterogeneous vehicles into quasi-homogeneous clusters. A representative RL model (or agent) is employed for each cluster where the included vehicles share and jointly optimize the policy parameters. The second stage develops an ensemble control strategy based on representative RL models for vehicles isolated as noise during clustering. While vehicles are simultaneously simulated, RL models are separately trained with cluster-specific experience replay. Application of TERL to the classical multi-user dynamic route choice problem in a real-world network of the Gusu District in Suzhou, China demonstrates the effectiveness of the proposed approach in deriving desirable simulation results, compared with the classical shortest path model and the dynamic user equilibrium model.

Implementation of Restorative Justice Policy against Perpetrators of Criminal Negligence that Causes Death

Requests for termination of prosecution against perpetrators of criminal negligence causing death can be rejected or granted even though they use the same technical guidelines, namely Circular Letter of the Deputy Attorney General for General Crimes Number 01/E/EJP/02/2022. The research focuses on understanding the policy parameters of restorative justice for perpetrators of criminal negligence causing death and the ideal implementation of restorative justice for perpetrators of criminal negligence. This research is a normative juridical research with statutory approach, case approach, and conceptual approach. Restorative Justice Application is granted because the parameters of negligent acts are not purely in the perpetrator but also carried out by the victim himself, while the application is rejected because culpa only comes from the actions of the perpetrator. The ideal implementation of restorative justice policy must lead to efforts to restore the original state, no negative stigma from the community, and oriented to technical efficiency and legal certainty for the parties. Suggestions from the research results are that case termination based on restorative justice requires certainty in technical regulations, for example through the Attorney General's Regulation. The ideal implementation of restorative justice policy must pay attention to: a unified written policy / lex scripta that can be accessed by the public, can be tested, integrative between judicial mechanisms and pay attention to functional differentiation of criminal justice, as well as harmonization of subjects and objects in prosecution.

Guided sampling-based evolutionary deep neural network for intelligent fault diagnosis

The selection of model architecture and hyperparameters has a significant impact on the diagnostic performance of most deep learning models. Because training and evaluating the various architectures of deep learning models is a time-consuming procedure, manual selection of model architecture becomes infeasible. Therefore, we have proposed a novel framework for evolutionary deep neural networks that uses a policy gradient to guide the evolution of the DNN architecture towards maximum diagnostic accuracy. We have formulated a policy gradient-based controller that generates an action to sample the new model architecture at every generation so that optimality is obtained quickly. The fitness of the best model obtained is used as a reward to update the policy parameters. Also, the best model obtained is transferred to the next generation for quick model evaluation in the NSGA-II evolutionary framework. Thus, the algorithm gets the benefits of fast non-dominated sorting as well as quick model evaluation. The effectiveness of the proposed framework has been validated on three datasets: the Air Compressor dataset, the Case Western Reserve University dataset, and the Paderborn University dataset.