Optimal Policy Research Articles

Optimal Monetary and Transfer Policy in a Liquidity Trap

AbstractOptimal monetary and fiscal policy are jointly analyzed in a heterogeneous two‐agent New Keynesian environment, where fiscal policy is modeled as lump‐sum transfers. Transfer policy does not substitute for forward guidance—as it entails consumption dispersion costs—nor affect its optimal duration. The stay at the zero lower bound is indeed influenced through two offsetting channels: a shortening channel works through an initial increase in transfers that mitigates the recession (reducing the need for forward guidance), while a lengthening channel works through a later transfer cut that curbs the expansion (making forward guidance desirable for a longer horizon).

Collaborative optimization of multi-energy multi-microgrid system: A hierarchical trust-region multi-agent reinforcement learning approach

In the context of the expanding diversity of energy demands, an increasing number of heterogeneous Multi-energy Microgrids (MEMGs) are engaging in the collaborative framework of the Multi-energy Multi-microgrid System (MEMMG). However, following this trend, the existing centralized Integrated Energy Management System (IEMS) control strategy is unreliable for future energy systems, characterized by more complex optimization control and a flexible system structure. This paper introduces a hierarchical Multi-agent Deep Reinforcement Learning (HMADRL) approach for distributed IEMS in MEMMG. Firstly, by employing a hierarchical approach, this method simplifies control complexity by segmenting the overarching control challenge into tasks of collaborative planning and action control, which are distributed across varied multi-agent scenes. Considering both macro and microeconomic factors, alongside carbon emissions, the optimal operation of MEMMG is realized through a bottom-up edge multi-agent control approach, in contrast to traditional top-down centralized methods. Secondly, in the phase of the inter-MEMG collaborative strategy, the Centralized Training Decentralized Execution (CTDE) framework is adopted to overcome the problems of unstable training environments and large-scale agent training, and each heterogeneous microgrid can develop local strategies independently with the assurance that their internal data will not be overly exposed. Thirdly, within each MEMG, the Trust-Region (TR) model is introduced for multi-agent action control, adeptly addressing the effects of mutual exclusion in decision-making time series. Simultaneously, an initialized Hot Experience Pool (HEP) is implemented, efficiently reducing exploration in complex, high-dimensional spaces. Finally, the off-time agent model is integrated into the HMADRL environment and undergoes secondary training based on real interactions, thereby deriving the optimal energy management policy. The proposed method markedly reduces reliance on exact physical modeling systems. Comparative simulations validate the proposed control scheme’s efficacy.

Multi-Agent Deep Reinforcement Learning Based Dynamic Task Offloading in a Device-to-Device Mobile-Edge Computing Network to Minimize Average Task Delay with Deadline Constraints.

Device-to-device (D2D) is a pivotal technology in the next generation of communication, allowing for direct task offloading between mobile devices (MDs) to improve the efficient utilization of idle resources. This paper proposes a novel algorithm for dynamic task offloading between the active MDs and the idle MDs in a D2D-MEC (mobile edge computing) system by deploying multi-agent deep reinforcement learning (DRL) to minimize the long-term average delay of delay-sensitive tasks under deadline constraints. Our core innovation is a dynamic partitioning scheme for idle and active devices in the D2D-MEC system, accounting for stochastic task arrivals and multi-time-slot task execution, which has been insufficiently explored in the existing literature. We adopt a queue-based system to formulate a dynamic task offloading optimization problem. To address the challenges of large action space and the coupling of actions across time slots, we model the problem as a Markov decision process (MDP) and perform multi-agent DRL through multi-agent proximal policy optimization (MAPPO). We employ a centralized training with decentralized execution (CTDE) framework to enable each MD to make offloading decisions solely based on its local system state. Extensive simulations demonstrate the efficiency and fast convergence of our algorithm. In comparison to the existing sub-optimal results deploying single-agent DRL, our algorithm reduces the average task completion delay by 11.0% and the ratio of dropped tasks by 17.0%. Our proposed algorithm is particularly pertinent to sensor networks, where mobile devices equipped with sensors generate a substantial volume of data that requires timely processing to ensure quality of experience (QoE) and meet the service-level agreements (SLAs) of delay-sensitive applications.

Optimisation of recovery policies in the era of supply chain disruptions: a system dynamics and reinforcement learning approach

Incidents like the COVID-19 pandemic or military conflicts disrupted global supply chains, causing long-lasting shortages in multiple sectors. This so-called ripple effect denotes the propagation of disruptions to further elements of the supply chain. Due to the severity of the impact that the ripple effect has on revenues, service levels, and reputation among supply chain entities, it is essential to understand the related implications. Given the unpredictable nature of disrupting events, this study emphasises the value of a reactive development of effective recovery policies on an operational level. In this article, a system dynamics model for a supply chain is used as framework to investigate the ripple effect. Based on this model, recovery policies are generated using reinforcement learning (RL), which represents a novel approach in this context. As main findings, the experimental results demonstrate the applicability of the proposed approach in mitigating the ripple effect based on secondary data from a major aerospace and defence supply chain and furthermore, the results indicate a broad applicability of the approach without the need for complete information about the disruption characteristics and supply chain entities. With further refinement and real-world implementation, the presented approach provides the potential to enhance supply chain resilience in practice.

Does macroprudential policy have any side effects on the shadow economy?

Purpose This paper aims to examine the impacts of macroprudential policy on the shadow economy worldwide. Design/methodology/approach We compile a panel dataset covering 125 countries from 1990 to 2018. This paper mitigates potential endogeneity issues via two-stage least squares and the two-step generalized method of moments (GMM). Findings The robust results show that the overall tightening of macroprudential policies exerts an expansion impact on the shadow economy. Further examination of the 16 individual macroprudential policy instruments finds that loan restrictions, countercyclical buffers, surcharges for systemically important financial institutions and capital conservation buffers have positive and statistically significant effect on the shadow economy. This relationship is only present during tightening episodes of macroprudential policy as loosening episodes do not exhibit any significant impact. Finally, this paper documents the nonlinear effects of macroprudential policy. Practical implications The results suggest that the supervisory authorities may need to consider another parameter, which is the development of the shadow economy, when devising the optimal macroprudential policy responses. Originality/value To the best of the authors’ knowledge, this paper is likely the first to empirically document the impact of macroprudential policy on the shadow economy. It contributes to the growing literature on the potential side effects of macroprudential policy on the macro-economy.

Financial Hedging of Operational Risk Constraints: A General Framework

We consider the operation of a non-financial corporation and present a novel risk-management framework that supports the joint optimization of the firm’s operational and financial decisions. Specifically, we study the problem of maximizing the firm’s operating profits when these profits depend on movements in the financial markets, and where the decision maker—the firm’s manager—is risk averse. Critically, our framework (i) imposes risk aversion through a generic class of risk constraints, and (ii) considers explicitly the differences that can exist between the shareholders’ and manager’s beliefs about future market risks. Together, these aspects allow us to remove speculation so that financial instruments can only be used for hedging purposes. We analyze our problem in an incomplete financial market setting and consider different informational assumptions. Within this framework, we characterize the optimal financial hedging strategy so that the problem reduces to a constrained maximization over operational policies. Our results show how financial hedging can be effectively used to expand the set of admissible operating policies, and identify conditions under which the risk constraint can be completely ignored when solving for an optimal operating policy. We illustrate our methodology in the context of an exporter firm which faces exchange rate risk in trading.

Dynamic Control of Service Systems with Returns: Application to Design of Postdischarge Hospital Readmission Prevention Programs

Dynamic Control of Service Systems with Returns Postdischarge interventions (e.g., follow-up phone calls, outpatient appointments) are commonly used to reduce unplanned hospital readmissions. Such interventions have been shown to be effective (to varying degrees) in reducing the probability of readmission (or return). The interventions are, however, costly, and hence, their benefits need to be carefully balanced with the costs. In “Dynamic Control of Service Systems with Returns: Application to Design of Postdischarge Hospital Readmission Prevention Programs,” Chan, Huang, and Sarhangian investigate how postservice interventions should be dynamically allocated, accounting for their costs as well as their benefits in terms of reducing returns and congestion costs. To this end, they study a transient queueing control problem and examine the structure of the optimal policy by analyzing associated fluid-control problems. The structural results motivate the design of intuitive surge protocols whereby different intensities of interventions are provided based on the congestion level of the system.

Reinforcement-Learning-Based Multi-UAV Cooperative Search for Moving Targets in 3D Scenarios

Most existing multi-UAV collaborative search methods only consider scenarios of two-dimensional path planning or static target search. To be close to the practical scenario, this paper proposes a path planning method based on an action-mask-based multi-agent proximal policy optimization (AM-MAPPO) algorithm for multiple UAVs searching for moving targets in three-dimensional (3D) environments. In particular, a multi-UAV high–low altitude collaborative search architecture is introduced that not only takes into account the extensive detection range of high-altitude UAVs but also leverages the benefit of the superior detection quality of low-altitude UAVs. The optimization objective of the search task is to minimize the uncertainty of the search area while maximizing the number of captured moving targets. The path planning problem for moving target search in a 3D environment is formulated and addressed using the AM-MAPPO algorithm. The proposed method incorporates a state representation mechanism based on field-of-view encoding to handle dynamic changes in neural network input dimensions and develops a rule-based target capture mechanism and an action-mask-based collision avoidance mechanism to enhance the AM-MAPPO algorithm’s convergence speed. Experimental results demonstrate that the proposed algorithm significantly reduces regional uncertainty and increases the number of captured moving targets compared to other deep reinforcement learning methods. Ablation studies further indicate that the proposed action mask mechanism, target capture mechanism, and collision avoidance mechanism of the AM-MAPPO algorithm can improve the algorithm’s effectiveness, target capture capability, and UAVs’ safety, respectively.

Cooperative MARL-PPO Approach for Automated Highway Platoon Merging

This paper presents a cooperative highway platooning strategy that integrates Multi-Agent Reinforcement Learning (MARL) with Proximal Policy Optimization (PPO) to effectively manage the complex task of merging. In modern transportation systems, platooning—where multiple vehicles travel closely together under coordinated control—promises significant improvements in traffic flow and fuel efficiency. However, the challenge of merging, which involves dynamically adjusting the formation to incorporate new vehicles, remains challenging. Our approach leverages the strengths of MARL to enable individual vehicles within a platoon to learn optimal behaviors through interactions. PPO ensures stable and efficient learning by optimizing policies balancing exploration and exploitation. Simulation results show that our method achieves merging with safety and operational efficiency.

Dynamic Optimization of a Supply Chain Operation Model with Multiple Products

Determination of the optimal operational policy for an automotive supply chain is explored under a centralized management approach using dynamic programming. A deterministic optimal control model is proposed to meet multi-product demand over a period while minimizing a cost performance index for a five-echelon network. The production-inventory levels are the state variables and the raw material acquisition rates are the control variables to be decided in the problem. The novelties include parts mixing operations, assembly requirements, and a push–pull chain operation strategy. The continuous model is solved using Iterative Dynamic Programming, an algorithm with successful applications in chemical engineering problems. Its implementation here is the first in supply chain (SC) management models. The results demonstrate that the proposal is suitable to represent the dynamic behavior of the SC and provides useful information to outline a cooperative decision-making process. Managerial insights are derived to improve the resilience and efficiency of the chain.

High-dimensional private and social optimal policy valuation model for non-renewable natural resource extraction projects for multivariate public policy decisions

We present a model to find the value and sensitivity function associated with the exploitation of a non-renewable natural resource both for the mining company (private optimum) and for the government (social optimum). Improvements over previous studies are achieved by achieving a multivariate model of optimal policy valuation under a Markov decision process with stopping times based on multiple real options, proposing an iterative algorithm of stochastic approximation with supervised dynamic programming decision under Monte Carlo scenarios until obtaining convergence. Applied to a mining project (Tía María) and extended to the total reserves of a country (Peru), it provides additional decision criteria for fiscal public policy, negotiations, concessions, tax levels, community reparations, as well as benefit estimation, discounted from the potential environmental cost in countries abundant in natural resources that wish to achieve sustainability.

Dynamic mean-variance portfolio selection under factor models

Utilizing insights from financial literature and empirical financial data, we introduce a comprehensive system of factor models designed to capture both return and risk dynamics. Our focus extends to addressing the multi-period mean-variance portfolio selection challenge within the framework of these proposed factor models. Through rigorous analysis, we formulate a semi-analytical optimal portfolio policy, characterized by a linear relationship with the current wealth level. The coefficients of this optimal policy are intricately linked to a specific stochastic process known as the future investment opportunity (FIO), reflecting the investor's anticipation of future investment prospects. Furthermore, empirical examination within the U.S. market context underscores the efficacy of our approach. By incorporating the factor models for return and risk, our optimal portfolio policy exhibits superior out-of-sample Sharpe ratio compared to benchmark policies.

Optimal robust online tracking control for space manipulator in task space using off-policy reinforcement learning

This study addresses the demands for adaptability, uncertainty management, and high performance in the control of space manipulators, and the inadequacies in achieving optimal control and handling external uncertainty in task space in previous research. Based on off-policy reinforcement learning, a model-free and time-efficient method for online robust tracking control in task space is devised. To address the complexity of dynamic equations in task space, a mixed-variable approach is adopted to transform the multivariable coupled time-varying problem into a single-variable problem. Subsequently, the optimal control policy is derived with the disturbance convergence, stability, and optimality of the control method being demonstrated. This marks the first instance of achieving robust optimal tracking control in task space for space manipulators. The efficacy and superiority of the presented algorithm are validated through simulation.

A safe reinforcement learning algorithm for supervisory control of power plants

Traditional control theory-based methods require tailored engineering for each system and constant fine-tuning. In power plant control, one often needs to obtain a precise representation of the system dynamics and carefully design the control scheme accordingly. Model-free Reinforcement learning (RL) has emerged as a promising solution for control tasks due to its ability to learn from trial-and-error interactions with the environment. It eliminates the need for explicitly modeling the environment’s dynamics, which is potentially inaccurate. However, the direct imposition of state constraints in power plant control raises challenges for standard RL methods. To address this, we propose a chance-constrained RL algorithm based on Proximal Policy Optimization for supervisory control. Our method employs Lagrangian relaxation to convert the constrained optimization problem into an unconstrained objective, where trainable Lagrange multipliers enforce the state constraints. Our approach achieves the smallest distance of violation and violation rate in a load-follow maneuver for an advanced Nuclear Power Plant design.

Maintenance optimization for multi-component systems with a single sensor

We consider a multi-component system in which a single sensor monitors a condition parameter. Monitoring gives the decision maker partial information about the system state, but it does not reveal the exact state of the components. Each component follows a discrete degradation process, possibly correlated with the degradation of other components. The decision maker infers a belief about each component’s exact state from the current condition signal and the past data, and uses that to decide when to intervene for maintenance. A maintenance intervention consists of a complete and perfect inspection, and may be followed by component replacements. We model this problem as a partially observable Markov decision process. For a suitable stochastic order, we show that the optimal policy partitions in at most three regions on stochastically ordered line segments. Furthermore, we show that in some instances, the optimal policy can be partitioned into two regions on line segments. In two examples, we visualize the optimal policy. To solve the examples, we modify the incremental pruning algorithm, an exact solution algorithm for partially observable Markov decision processes. Our modification has the potential to also speed up the solution of other problems formulated as partially observable Markov decision processes.

The financial accelerator, wages, and optimal monetary policy

I study the effects of labor market outcomes on firms' loan demand and credit intermediation. I first show in partial equilibrium that the presence of frictions in the banking sector lowers the capital factor demand elasticity to changes in real wages. This finding helps to connect the substitutability of labor and capital with credit conditions. Second, I use a new Keynesian banking model with an endogenous financial accelerator mechanism to study the role of lower capital factor demand elasticity in the transmission mechanism of monetary policy. Stabilizing nominal wages is close to the optimal monetary policy because it coincides with stabilizing the credit spread, the net worth gap, and the output gap. Inflation stabilization, in turn, imposes a policy trade-off with high welfare costs.

Efficient opportunistic maintenance strategies via pruning in parallel–series systems with economic dependence

Opportunistic maintenance stands out as an effective strategy for minimizing maintenance costs and enhancing system efficiency. Despite its importance, current opportunistic maintenance models are often tailored to specific system types, particularly those with series structures, or they offer heuristic approaches for multi-component redundant systems. This paper shifts focus to the most common of such systems, the parallel–series system. We approach the optimization of opportunistic maintenance in a parallel–series system through a Markov Decision Process framework, utilizing phase-type approximations of individual component hazard functions. Addressing the challenge of the curse of dimensionality in the action space, this study conducts a thorough structural analysis of the optimal opportunistic maintenance policy, deriving a pruning procedure, which can reduce the extensive combinatorial action space into a more manageable linear complexity without sacrificing the strategy’s effectiveness in parallel–series systems. Building on this more manageable action space, we introduce a more efficient algorithm leveraging the proposed pruning procedure for determining the optimal opportunistic maintenance policy. The effectiveness and practical applicability of this approach are rigorously demonstrated through comprehensive numerical experiments.

Dynamic determinants of optimal global climate policy

We explore the impact of dynamic characteristics of greenhouse-gas emitting systems, such as inertia, induced innovation, and path-dependency, on optimal responses to climate change. Our compact and analytically tractable model, applied with stylized damage assumptions to derive optimal pathways, highlights how simple dynamic parameters affect responses including the optimal current effort and the cost of delay. The conventional cost-benefit result (i.e., an optimal policy with rising marginal costs that reflects discounted climate damages) arises only as a special case in which the dynamic characteristics of emitting systems are assumed to be insignificant. Our analysis highlights and distinguishes from the (often implicit) assumption in many cost-benefit models, which neglect inertia and assume exogenous technology progress. This tends to defer action. More generally, our model yields useful policy insights for the transition to deep decarbonization, showing that enhanced early action may greatly reduce both damages and abatement costs in the long run.

Optimal dividend and risk control strategies for an insurer when there are multiple reinsurers with different risk attitudes

Suppose that insurer can control dividend, refinancing and reinsurance strategies dynamically. Different from the past, there are multiple reinsurers rather than sole reinsurer in the market. The insurer aim at finding the optimal strategies for maximizing the company’s value. It illustrates that refinancing can be considered iff the company has strong profitability; It should reduce reinsurance purchase when the surplus increases. The amount of risk ceded to the reinsurer depends on its risk attitude. The optimal dividend policy is of barrier type when the dividend rate is unbounded and is of threshold type when the dividend rate is bounded.

World-scale allocation of new mineral-related geological heritage resources

The socio-economical importance of geological heritage (geoheritage) makes it a geological resource. New minerals are reported regularly, and their type specimens and localities are ex-situ and in-situ constituents of geoheritage, respectively. The present study focuses on the contemporary dynamics of these specific resources. For this purpose, the information about new minerals reported during the past five years is analyzed. The attention is paid to the distribution of the new mineral specimens (holotypes and cotypes) and their type localities by country; the transboundary replacement of new minerals is also considered. It is established that the new minerals-related geoheritage resources are accumulated in dozens of countries. The ex-situ geoheritage (specimens in museum collections) is represented twice narrower than the in-situ geoheritage (localities as geosites). About a half of new minerals are replaced to the other countries and often to the other parts of the world. The registered patterns are explained by the peculiarities of the contemporary mineralogical research. The outcomes of the present analysis permit to make recommendations for the development of the optimal geoheritage policy.