Policy Parameters Research Articles

Real-Time Policy Optimization for UAV Swarms Based on Evolution Strategies

Multi-agent decision-making faces many challenges such as non-stationarity and sparse rewards, while the complexity and randomness of the real environment further complicate policy development. This paper addresses the high-dimensional policy optimization problems of unmanned aerial vehicle (UAV) swarms. By modeling the problem scenario as a Markov decision process, a real-time policy optimization algorithm based on evolution strategy (ES) pre-training is proposed. This approach combines decision-time planning with background planning to evaluate and integrate different sets of policy parameters in a temporal context. In the experimental phase, the policy network is trained using both ES and REINFORCE algorithms on a constructed simulation platform. Comparative experiments demonstrate the effectiveness of using ES for policy pre-training. Finally, the proposed real-time policy optimization algorithm further improves the performance of the swarm by approximately 10% in simulations, offering a feasible solution for adversarial games between swarms and extending the research scope of evolutionary algorithms.

Reassessing ayahuasca regulation in Brazil: strategic framing and cumulative gains

Background In the early twentieth century, ayahuasca gained popularity in northern Brazil, causing political tension due to moral hostility. However, over the course of the century, despite the proscription of DMT and the intensification of the ‘War on Drugs’, three phases of cumulative political gains eventually led to the full regulation of ayahuasca, in 2010. How was this achieved? Methods This paper provides a comprehensive reassessment of the existing literature on ayahuasca in Brazil, utilizing social movement theory. Emphasis is placed on framing methods, or how movements strategically present their aims to particular audiences, in order to effect political change. Results First, ayahuasca churches constructed a religious frame that emphasized their Christian pedigree. Second, when harassment intensified, ayahuasca was reframed as a charitable endeavour. Finally, in the transition to democracy, the religions added a scientific frame, collaborating with the state to review scientific evidence and establish policy parameters for regulation. Conclusion Framing ayahuasca as Christian and focusing on charity and science were neither straightforward nor unproblematic, but did allow ayahuasca groups to successfully avoid persecution, establish relationships with powerful actors, and eventually achieve full regulation. This case study provides unique insights into the role of social movements in successful drug policy reform, especially how movements can strategically adapt to and influence prevailing moral codes, resulting in concrete political victories.

Fiscal and Monetary Policy Interactions in a Model with Low Interest Rates

We provide a new Keynesian model where entrepreneurs face uninsurable idiosyncratic investment risk and credit constraints. Government bonds provide liquidity services. Multiple steady states with positive values of public debt can be supported for a given permanent deficit-to-output ratio. The steady-state interest rates are lower than the economic growth rate, and public debt contains a bubble component. We analyze the determinacy regions of policy parameter space and find that a large set of monetary and fiscal policy parameters can achieve debt and inflation stability given persistent fiscal deficits both away from and at the zero interest rate lower bound. (JEL E12, E23, E31, E43, E52, E62, H63)

Impact analysis of the four-party evolutionary game in renewable portfolio standard under dynamic parameters

Renewable portfolio standard has become the primary policy tool to improve the integration of renewable energy. The motivation of the parties involved and the formulation of the policy parameters are crucial for the rapid development of renewable energy industry. However, the role of large consumers has often been neglected as the main party for renewable energy consumption, and the policy parameters have been dominated by static parameters. This paper innovatively introduces large consumers into the evolutionary game, constructs a four-party revenue matrix and the replicated dynamic equations, analyzes the equilibrium conditions of the evolutionary game, and constructs the functions of the key dynamic parameters over time, then simulates the behavioral strategies of the game players under the dynamic parameters with a system dynamics model. The optimal strategy is regulators waive regulation, while power plants, large consumers and power sales companies fulfill their quotas. Dynamic fines, dynamic quotas, and dynamic consumption prices have a large impact on each player, while dynamic TGC prices have a limited impact on them. Reasonably high fines in the initial stage are effective in facilitating each player to its stable state more quickly than static fines. Even if the fine is reduced in the later stages, it will not affect their strategies. High quotas and low consumption prices will significantly accelerate the exit of regulators from regulation, while power sales companies and large consumers will quickly fulfill their quota tasks. Only power generators prefer low quotas and high consumption prices. These results provide some recommendations for the setting of policy parameters.

SYNTHETIC INDICATORS OF QUALITY OF LIFE: EVALUATION OF THE EFFECTIVENESS OF UKRAINE'S ECONOMIC POLICY

This paper examines the effectiveness of Ukraine's economic policies through the lens of quality of life and argues for the adoption of a minimal set of evaluation criteria, ideally a single composite criterion, to optimise policy evaluation. Despite ongoing efforts to identify such criteria, the study introduces a novel approach by using synthetic indicators of quality of life as the basis for evaluation. This approach is particularly relevant in the context of Ukraine's post-war recovery and its aspirations towards European Union integration. These conditions require the adoption of economic policies that meet European standards of quality of life and facilitate the necessary reforms. Such policies are crucial for improving the living conditions of the Ukrainian population and facilitating the return of refugees after Ukraine's victory over Russia, the aggressor country. This study develops a methodology that uses synthetic indicators of quality of life as a system of criteria to assess the effectiveness of Ukraine's economic policy. The research involves a cross-country analysis of data from seven EU Member States and Ukraine, and the construction of regression models that link the determinants of state economic policy to several indicators: population quality, welfare, social quality, and a comprehensive synthetic indicator – quality of life. The obtained results confirm the hypothesis about the key parameters of the state economic policy as determinants of improving the quality of life of Ukrainian citizens. In addition, the analysis of the dynamics of synthetic indicators of the quality of life in Ukraine and their determinants makes it possible to identify priority areas of economic policy aimed at improving these indicators, which will contribute to the overall efficiency of Ukraine's economic policy.

An Effective Training Method for Counterfactual Multi-Agent Policy Network Based on Differential Evolution Algorithm

Due to the advantages of a centralized critic to estimate the Q-function value and decentralized actors to optimize the agents’ policies, counterfactual multi-agent (COMA) stands out in most multi-agent reinforcement learning (MARL) algorithms. The sharing of policy parameters can improve sampling efficiency and learning effectiveness, but it may lead to a lack of policy diversity. Hence, to balance parameter sharing and diversity among agents in COMA has been a persistent research topic. In this paper, an effective training method for a COMA policy network based on a differential evolution (DE) algorithm is proposed, named DE-COMA. DE-COMA introduces individuals in a population as computational units to construct the policy network with operations such as mutation, crossover, and selection. The average return of DE-COMA is set as the fitness function, and the best individual of policy network will be chosen for the next generation. By maintaining better parameter sharing to enhance parameter diversity, multi-agent strategies will become more exploratory. To validate the effectiveness of DE-COMA, experiments were conducted in the StarCraft II environment with 2s_vs_1sc, 2s3z, 3m, and 8m battle scenarios. Experimental results demonstrate that DE-COMA significantly outperforms the traditional COMA and most other multi-agent reinforcement learning algorithms in terms of win rate and convergence speed.

Green Technology Innovation Guiding Policies under the Synergy of Government and Financial Institutions

Guiding enterprises towards green technology innovation (GTI) is a pivotal strategy for mitigating pollution and energy wastage, thereby facilitating the transition towards a green and low-carbon economy. In order to solve the single irrationality of the existing guidance policy and improve its implementation effect, parameters of environmental regulation and green credit policy are hereby introduced to construct a tripartite evolutionary game model of the government, financial institutions, and enterprises. The evolutionary stability of the government and financial institutions in guiding enterprises' GTI is analyzed. Simulation analysis method is used to visualize the evolution path of the game system under different policy options. The study results show that increasing interest rate subsidies can promote the financial institutions and the government to formulate GTI guidance policies. Besides, there is a significant synergistic effect between interest rate subsidies and green credits, which can efficiently foster GTI within enterprises and decrease their innovation costs. Building on this foundation, further increasing the carbon tax rate can lead to a reduction in government subsidy expenditures, improve the evolution rate of the ideal strategy, and expedite the realization of a green, low-carbon transformation.

Multiagent Trust Region Policy Optimization.

We extend trust region policy optimization (TRPO) to cooperative multiagent reinforcement learning (MARL) for partially observable Markov games (POMGs). We show that the policy update rule in TRPO can be equivalently transformed into a distributed consensus optimization for networked agents when the agents' observation is sufficient. By using a local convexification and trust-region method, we propose a fully decentralized MARL algorithm based on a distributed alternating direction method of multipliers (ADMM). During training, agents only share local policy ratios with neighbors via a peer-to-peer communication network. Compared with traditional centralized training methods in MARL, the proposed algorithm does not need a control center to collect global information, such as global state, collective reward, or shared policy and value network parameters. Experiments on two cooperative environments demonstrate the effectiveness of the proposed method.

A sequential multi-agent reinforcement learning framework for different action spaces

In many real-world decision-making tasks, multi-agent need to learn collaboration in a high-dimensional complex action space, rather than just a single discrete action space. Recently, value decomposition learning methods such as QMIX have emerged as a promising approaches for collaborative multi-agent tasks. However, most of the value decomposition algorithms can only be used in discrete action space, which would limit their practicability. To address the limitation, we propose a novel algorithm called Multi-Agent Sequential Q-Networks (MASQN), which can be applied to the multi-agent domains with continuous, multidiscrete or hybrid action spaces. The proposed algorithm is based on the structure of centralized training with decentralized execution (CTDE). The decentralized actors ensure adaptability to different action spaces by utilizing action space discretization and sequential models, and the centralized critic utilizes the value decomposition architecture to guide effective updates of the policy parameters for each agent. We also give the convergence of joint policy from the perspective of policy iteration, by combining it with the CTDE structure and the constraint of the Individual Global Max (IGM) condition. Finally, we evaluate the MASQN algorithm on two benchmark environments: MAMuJoCo and Hybrid Predator–Prey. The empirical results show that MASQN out performs the state-of-the-art performance on three different action spaces.

The impact of the urban traffic on the CO2 intensity: a navigation study using GNSS application in Jakarta city

ABSTRACT Massive traffic congestion is one of the many challenges faced by Jakarta as the capital city of Indonesia with the largest metropolitan area. The overarching goal of the study is to explore a combination of global navigation satellite system (GNSS) with vehicle-based measuring devices to precisely map the impact of the urban traffic on the CO2 emissions in Jakarta city. The research objectives include to examine and understand the CO2 intensity in major cities using state-of-the-art research strategies to correctly map the impact of the urban traffic patterns on urban areas and overall well-being of people in these locations. Urban traffic was studied in selected major urban transportation routes that were selected using PPK GNSS and the HT 2000 environmental sensor. This study reports that there is indeed a correlation between urban traffic and air pollution. The study noted that the macro and micro parameters of urban development and traffic policies are influencing the slowing down of traffic in city centers. The research outcomes could serve as a framework for designing the conceptual smart measure and tool of transportation carbon pricing in achieving the more sustainable transportation sector of Indonesia.

Optimal drift rate control and two-sided impulse control for a Brownian system with the long-run average criterion

Abstract In this paper, we consider a joint drift rate control and two-sided impulse control problem in which the system manager adjusts the drift rate as well as the instantaneous relocation for a Brownian motion, with the objective of minimizing the total average state-related cost and control cost. The system state can be negative. Assuming that instantaneous upward and downward relocations take a different cost structure, which consists of both a setup cost and a variable cost, we prove that the optimal control policy takes an $\left\{ {\!\left( {{s^{\ast}},{q^{\ast}},{Q^{\ast}},{S^{\ast}}} \right),\!\left\{ {{\mu ^{\ast}}(x)\,:\,x \in [ {{s^{\ast}},{S^{\ast}}}]} \right\}} \right\}$ form. Specifically, the optimal impulse control policy is characterized by a quadruple $\left( {{s^{\ast}},{q^{\ast}},{Q^{\ast}},{S^{\ast}}} \right)$ , under which the system state will be immediately relocated upwardly to ${q^{\ast}}$ once it drops to ${s^{\ast}}$ and be immediately relocated downwardly to ${Q^{\ast}}$ once it rises to ${S^{\ast}}$ ; the optimal drift rate control policy will depend solely on the current system state, which is characterized by a function ${\mu ^{\ast}}\!\left( \cdot \right)$ for the system state staying in $[ {{s^{\ast}},{S^{\ast}}}]$ . By analyzing an associated free boundary problem consisting of an ordinary differential equation and several free boundary conditions, we obtain these optimal policy parameters and show the optimality of the proposed policy using a lower-bound approach. Finally, we investigate the effect of the system parameters on the optimal policy parameters as well as the optimal system’s long-run average cost numerically.

Incorporating multi-perspective information into reinforcement learning to address multi-hop knowledge graph question answering

Knowledge graph question answering (KGQA) aims to answer natural language questions from structured knowledge graphs (KGs). Traditional KQGA methods are usually limited to single-hop queries and cannot handle complex questions involving multi-hop reasoning well. To overcome this issue, multi-hop KGQA based on reinforcement learning (RL) has been proposed. However, multi-hop KGQA based on RL still faces some challenges. Firstly, due to the insufficient availability of latent environmental information during the reasoning process, the agent finds it challenging to make coherent and correct decisions. Secondly, the agent only receives rewards from the environment upon reaching the answer entity during the exploration, leading to slow or even obstructed learning. To address these shortcomings, we construct multi-perspective information based on the state of the environment, and integrate multi-perspective information with RL framework, thereby creating the Multi-Perspective Information Fusion Reasoning Network (MPIFRN). MPIFRN achieves the goal via three steps. (1) We construct three different views of information, i.e., expectation embedding, instruction-guided embedding, and path-aware embedding. These environmental cues provide more reliable support for decision-making. (2) We still adopt the method of mapping entities and relations into the knowledge graph embedding space to answer multi-hop questions. At each step of reasoning, we use a scoring function to measure the plausibility of each “triple” ¡topic entity, question, candidate entity¿ in the embedding space. (3) Furthermore, we employ the asynchronous advantage actor-critic (A3C) algorithm to guide the agent in selecting the most promising entities and to expand the reasoning paths in parallel by updating policy and value network parameters, thereby facilitating multi-hop knowledge graph question answering. We conduct extensive experiments on KGQA benchmark datasets, providing substantial evidence to demonstrate the effectiveness of our approach.

Simple heuristics for the joint inventory and pricing models with fixed replenishment costs

This study considers the joint inventory and pricing problem of a firm selling a single item over a multi-period planning horizon. A fixed replenishment cost is incurred whenever the replenishment decision is made. Period demands are considered to be non-stationary, stochastic, and price-dependent. The literature shows that the optimal policy in such a system is of an (s, S, p)-type. The implementation of this type of policy might be challenging in practice due to the prevailing computational difficulty in determining the optimal policy parameters. This study, therefore, aims to develop computationally efficient heuristic approaches for the joint inventory and pricing problem. The proposed heuristics provide different levels of flexibility in making inventory and pricing decisions. Furthermore, they are essentially built on the concept of the replenishment cycle and do not require solving a stochastic dynamic program. Our numerical study demonstrates the profit efficiency of the proposed heuristic approaches against the optimal policy. The results also indicate that firms lacking the technical ability to employ a dynamic pricing strategy might still gain significant profit improvement by only using a dynamic inventory control policy.

Can China's energy quota trading impact the market performance and policy effects of carbon emissions trading?

With the gradual shift from dual control of energy consumption to dual control of carbon emissions, energy quota trading (EQT) and carbon emissions trading (CET) policies will be implemented in parallel in China. EQT policy will significantly affect the market performance and policy effects of CET, which consequently affects the dual control of carbon emissions. Therefore, constructing a system dynamics model coupling the national CET, EQT, and electricity markets, this paper researches the impact of EQT policy and its parameter design on the market performance and policy effects of CET through scenario analysis. The results show that: (1) Implementing EQT policy will reduce CET price and trading volume, negatively affecting CET market performance. This indicates that there is a certain substitution between EQT and CET. (2) EQT policy can enhance the emission reduction and decarbonization effects of CET policy. By curbing thermal power investment and generation and promoting green power, the EQT policy reduces carbon emissions by about 4.87% and optimizes the power supply structure. (3) To scientifically design the EQT policy parameters, lower energy saving targets and energy quota benchmark prices can alleviate the negative impact on CET market performance, but are not conducive to improving CET policy effects.

Impact of dual-credit policy on diffusion of technology R & D among automakers: Based on an evolutionary game model with technology-spillover in complex network

Dual-credit policy promotes new-energy vehicle (NEV) automakers to invest in technology R & D by granting NEV with better technological performance to higher NEV-credit. Based on the complex network evolutionary game theory, this paper constructs a technology R & D diffusion model of automakers considering the technology spillover effects, and explores the dynamic impact of adjusting different dual-credit policy parameters on technology R & D diffusion among automakers. Our results show that: (1) increasing the annual increment of NEV-credit requirement ratio can encourage more NEV automakers to invest in technology R & D, but also lead to significant increase in NEV-credit price. (2) Reducing the CAFE target coefficient can simultaneously promote the diffusion of NEV technology R & D and GV green-technology R & D, but excessive reduction can hinder GV automakers from entering NEV market. (3) Lowering the upper limit of credit per unit NEV suppresses the diffusion of NEV technology R & D. (4) Reducing technology-spillover is conducive to the incentive of dual-credit policy on the diffusion of technology R & D.

Tax Incentives for Economic Growth in the Russian Far East: Broad vs. Targeted Stimuli

The purpose of this study is to justify the choice of tax incentive policy instruments aimed at the economic development of the Russian Far East, which is facing acute demographic and environmental problems. To model the dynamics of the region’s real economic system, this study employed a mathematical model based on actual data from 2010 to 2021, covering economic, technological, and socio-ecological aspects. Using the versatile AnyLogic 8.0 platform for agent-based and system dynamics modeling, experiments on alternative tax incentive policy options involving both broad tax incentives and targeted economic development measures were conducted. Specifically, a 50% investment tax deduction for residents in special economic zones in the Russian Far East was implemented. The experimental results show that, despite comparable population dynamics, targeted stimulation of growth poles through public-private partnership programs outperforms broad tax incentives for economic entities in the Russian Far East. This is evident in higher economic growth rates in the region, particularly during the experimental period, except for 2040–2050, where adverse demographic trends constrain growth in both scenarios. The theoretical significance of the application of this method has shown that it allows us to obtain new significant results in the subject area of research due to the consideration of the complex interaction of factors of influence both at the micro- and macro-level, primarily behavioral factors that are fundamentally important for understanding the action of taxes. The practical implications of this study lie in defining the parameters of tax policy to target and stimulate growth poles in regions serving as hubs for generating and disseminating new technologies. The planned perspective is to encourage population growth and ensure sustained economic development in Russia’s Far East. It is advisable to explore comprehensive tax and budgetary regulations that simultaneously address economic, socio-demographic, and environmental issues in the region. Doi: 10.28991/ESJ-2024-08-03-021 Full Text: PDF

Analysis on the synergy between markets of electricity, carbon, and tradable green certificates in China

China has integrated tradable green certificates and carbon markets with its electricity sector to promote renewable energy and reduce carbon emissions. However, the impact mechanism of decision-making behaviors of involved entities in these integrated markets on the market equilibrium is still unclear, hence limiting the further development of renewable energy industries and carbon reduction progress to some extent. Therefore, conducting in-depth research to inform policy makers about the impacts of entities' decision-making behaviors on market equilibrium within the context of integrated multiple markets is crucial. This study aims to analyze the characteristics of entities' decision-making behaviors and equilibrium states within the collaborative dynamics of carbon, electricity, and green certificate markets. Specifically, an optimal decision-making model for entities given the multi-market coupling was developed by employing mathematical programming methods to analyze market equilibriums across various market combinations and policy parameters. The conclusions underscore equilibrium price shifts and decision-making characteristics among entities across multiple markets. The result shows that the carbon market could increase power prices and decrease green certificate prices. To ensure sustainable energy development, policymakers should consider the decision-making behavior of entities under multi-market coupling, which can promote the low-carbon transformation of the Chinese energy industry.

On the Number of Customer Classes in a Single-Period Inventory System

A common practice in inventory systems with several customers requiring differentiated service levels is to group them into two or three classes, where a customer class is a group of customers with the same preset service level in terms of product availability. However, there is no evidence that grouping customers into two or three classes is optimal in terms of the ordering policy parameters. This paper studies the effect of the number of customer classes on the inventory level of a single-period inventory system with stochastic demand and individual service-level requirements from multiple customer classes. Using a Sample Average Approximation approach, we formulate computationally tractable multi-class service level models, under responsive and anticipative priority policies in cases of shortage, as mixed integer linear problems (MIPs). The effect of the number of classes on the inventory level is determined using a round-up aggregation scheme; i.e., given a sufficiently large initial number of classes, it is reduced by adding the lower service level classes to the next higher class. We analytically characterize the optimal inventory level under responsive and anticipative priority policies as a function of the initial number of classes and the number of classes grouped based on the round-up aggregation scheme. Under a responsive priority policy, we show that there is an optimal number of classes, while under an anticipative priority policy, the optimal number of classes is equal to the initial number of classes. The effect of free-riders resulting from the round-up aggregation scheme on the optimal inventory level is studied through numerical experiments.

An application of the graph approach to life-cycle optimisation of vehicle electrification

Although durable goods with low energy consumption are being promoted to achieve a decarbonised society, from the perspective of life-cycle assessment, the choice of new durable goods may increase CO2 emissions. To address this problem, research has been conducted on product replacement based on life-cycle optimisation (LCO), a method for identifying a replacement life span that minimises life-cycle CO2 emissions. However, several additional assumptions complicate the analysis of replacement patterns of products and conditional formulas because cumulative emissions do not increase linearly when considering energy mix and technology improvement, and it is difficult to extend the model to optimisation methods in previous LCO studies. This study developed a new LCO approach by applying the shortest path problem to graph theory. Our methodology can contribute to the following: (i) it is computationally inexpensive; (ii) it is intuitively easy to add complex conditions, such as various policy scenarios and parameter changes; and (iii) once the graph of replacement patterns is defined, the optimal solution can be derived using existing solution methods, such as the Dijkstra algorithm. As a case study, we focused on vehicle replacement, which is a major source of CO2 emissions and is being electrified. In particular, we identified vehicle switching paths that minimise life-cycle CO2 emissions by considering changes in Japan’s energy mix and alternative fuel vehicle (AFV) characteristics. We determined that the optimal vehicle replacement path method to reduce CO2 emissions is to switch first to plug-in hybrid electric vehicles (PHEVs) and then to battery electric vehicles (BEVs). Thus, we suggest that the transition to electric vehicles requires a step-by-step process. This methodology is not only conducive to AFV deployment for decarbonisation but can also be applied to other products, such as air conditioners and lighting. Thus, various transition policies could be formulated using our methodology.

(r, Q) stock replenishment in multi-mode distribution systems with uncertain delivery channels and adaptive channel allocation

To mitigate the impact of uncertainty in goods distribution systems, multiple channels and transportation modes may be used. In this paper, a discrete-time model of supplier-depot interaction involving multiple channels of goods relocation is constructed using the concepts of dynamic systems theory. The considered class of systems faces two types of uncertainty: a priori unknown variations of market demand and unpredictable changes of lead-time delay, different for each channel. The variation pattern is arbitrary. As a control mechanism, the nonlinear (r, Q) replenishment policy (provide Q units of goods when the stock falls below r) is considered. As opposed to the previous studies, the policy is demonstrated robust to both sources of uncertainty acting simultaneously using a formal mathematical argument. Moreover, it is shown how to select the policy parameters to obtain full demand realization, thus eliminating backorders, irrespective of the pattern of delay and demand fluctuations. It is also discussed how to assign the depot capacity so that emergency storage will not be required even though deliveries from a few periods arrive concurrently, or out of order. In addition, a new channel allocation strategy, dynamically adapting itself to the channel temporal characteristics, is developed. As evidenced in the tests, the proposed adaptive strategy, by reducing the waiting time for the lot reassembly at the depot, allows for holding cost reduction by several percent with respect to a static strategy.