Optimal Policy Research Articles

Optimal fiscal policy under finite planning horizons

AbstractWe propose a novel framework that revisits the seminal Chamley-Judd zero capital taxation result in light of bounded rationality stemming from a finite policy planning horizon and structural frictions in fiscal institutions. We show a mechanism that generates positive optimal capital taxation in the long run. Our numerical results indicate that the current tax system in the United States could be near-optimal in a constrained environment where policymakers exhibit limited policy planning horizons and imperfect altruism toward household welfare under subsequent governments.

Neural Circuit Policies for Virtual Character Control

The development of high-stakes decision-making neural agents that interact with complex environments, such as video games, is an important aspect of AI research with numerous potential applications. Reinforcement learning combined with deep learning architectures (DRL) has shown remarkable success in various genres of games. The performance of DRL is heavily dependent upon the neural networks resides within them. Although these algorithms perform well in offline testing but the performance deteriorates in noisy and sub-optimal conditions, creating safety and security issues. To address these, we propose a hybrid deep learning architecture that combines a traditional convolutional neural network with worm brain-inspired neural circuit policies. This allows the agent to learn key coherent features from the environment and interpret its dynamics. The obtained DRL agent was not only able to achieve an optimal policy quickly, but it was also the most noise-resilient with the highest success rate. Our research indicates that only 20 control neurons (12 inter-neurons and 8 command neurons) are sufficient to achieve competitive results. We implemented and analyzed the agent in the popular video game Doom, demonstrating its effectiveness in practical applications.

Research on Talent Policy and Optimization Path of Wuhan City based on SWOT Analysis

This study adopts SWOT analysis to analyze Wuhan's talent policy in depth, aiming to explore its advantages, disadvantages, opportunities and challenges. The study finds that Wuhan has significant geographical location and resource advantages in terms of talent policy, as well as outstanding performance in policy innovation and talent cultivation. However, there are also problems such as talent drain and insufficient policy implementation. In the face of fierce competition for talents at home and abroad, Wuhan should seize the opportunity to strengthen policy promotion and implementation, optimize the talent service system, and enhance the attractiveness of talents. This study proposes targeted optimization paths, including improving the talent policy system, strengthening the construction of talent teams, and improving the quality of talent services, with a view to promoting the sustainable development and optimization of Wuhan's talent policy.

Optimal Joint Defense and Monitoring for Networks Security under Uncertainty: A POMDP-Based Approach

The increasing interconnectivity in our infrastructure poses a significant security challenge, with external threats having the potential to penetrate and propagate throughout the network. Bayesian attack graphs have proven to be effective in capturing the propagation of attacks in complex interconnected networks. However, most existing security approaches fail to systematically account for the limitation of resources and uncertainty arising from the complexity of attacks and possible undetected compromises. To address these challenges, this paper proposes a partially observable Markov decision process (POMDP) model for network security under uncertainty. The POMDP model accounts for uncertainty in monitoring and defense processes, as well as the probabilistic attack propagation. This paper develops two security policies based on the optimal stationary defense policy for the underlying POMDP state process (i.e., a network with known compromises): the estimation-based policy that performs the defense actions corresponding to the optimal minimum mean square error state estimation and the distribution-based policy that utilizes the posterior distribution of network compromises to make defense decisions. Optimal monitoring policies are designed to specifically support each of the defense policies, allowing dynamic allocation of monitoring resources to capture network vulnerabilities/compromises. The performance of the proposed policies is examined in terms of robustness, accuracy, and uncertainty using various numerical experiments.

Green subsidies as strategic trade policy tools

AbstractIn a third-country market model in which two export countries adopt environmental policies (taxes and subsidies), this article analyses how an abatement (“green”) subsidy can become a potential strategic trade policy tool. When governments set the optimal policy considering their local environmental damages, a rich set of equilibria arises. In contrast to the standard result, it is shown that subsidising pollution abatement can 1) emerge as the unique Pareto-efficient Nash equilibrium of the policy game, 2) be the only feasible environmental policy when environmental awareness is low, irrespective of the efficiency of the cleaning technology, and 3) emerge as the unique Pareto-inefficient Nash equilibrium of the policy game at the end of the ecological transition. The article also tackles some dynamic issues that the policy game implies.

Decentralized Online Order Fulfillment in Omni-Channel Retailers

We consider an order fulfillment problem of an omni-channel retailer that ships online orders from its distribution center (DC) and brick-and-mortar stores. Stores use their local information, not observed by the retailer, that can lead them to accept or reject fulfillment requests of items in an online order. We investigate the problem of sequencing requests to stores and inventory rationing decisions at the DC to minimize expected costs under uncertain store acceptance behavior and when items are indistinguishable in terms of shipping. First, under the scenario that stores are used only when the DC has insufficient inventory, we propose a Markov Decision Process formulation and analyze the performance of myopic policies that are preferable because of their interpretability. We show that the performance rate of a myopic approach that orders stores by cost only depends on the number of items in an order, which is small in practice. We also determine conditions for the range of acceptance probabilities for the myopic policy to be optimal for small-sized orders. Using optimality conditions for a special case of the problem, we develop an adaptive variant of the myopic policy, and propose a new degree-based strategy that balances shipping costs and acceptance probabilities. Numerical testing suggests that the best-performing sequencing policy is within 1% of optimality on average. Moreover, using two years of data from a large omni-channel retailer in North America, we observe that adaptive policies, albeit more complex, are beneficial in reducing costs and split deliveries if acceptance rates can be estimated accurately. Second, we determine when the retailer should ship from stores or ration the inventory at the DC. We show that for single-item orders, the optimal policy has a threshold structure, where, remarkably, the highest priority region is also subject to rationing. We then consider the novel multi-unit-single-item rationing problem, and leverage the structure of the single-unit model to develop a heuristic. We numerically establish the efficacy of rationing models and our heuristic.

Cost Optimization in Cloud Computing: Capacity Reservation for Intermittent Random Demand Surges

The adoption of cloud computing has been accelerating over the last decade, while enterprise cloud users (“firms”) are struggling to manage their growing cloud expenditures in the face of intermittent digital demand surges caused by planned or random events. To deal with such challenges, a firm can employ reserved instances with the standard contract length (e.g., one year) to meet the stationary base demand, which we refer to as the base contracts, complemented by additional reserved instances with either standard or shorter contract lengths, which we refer to as the supplementary contracts, to cope with the demand surges. We first analyze a model whereby the surge and inter-surge durations are deterministic, demand magnitude is random, and cancellation of the reservations is allowed. We develop a capacity management plan for the firm including not only the optimal capacity levels, which follow a newsvendor-type solution, but also the optimal policy for managing the purchase, renewal, cancellation, or expiration of the supplementary contracts, which can be characterized as a two-threshold policy. Due to the complexity of the structure of the optimal policy, we also construct an effective heuristic policy by excluding the renewal option from the action space, which can be applied to a more general setting where the surge and inter-surge durations are random. We examine two model extensions: (1) when trades of reserved instances are allowed in a secondary marketplace; (2) when the firm does not have exact information about the distributions of the surge magnitude and duration while it can adjust the capacity levels as data unveils. Our analysis shows that the optimal policy for managing the supplementary contracts depends on the relative magnitude of the surge and inter-surge durations in relationship to the cancellation fee. Moreover, our numerical results show that cloud platforms that offer a secondary marketplace are more attractive to firms from a cost standpoint than those that allow cancellation only. The latter, without the secondary marketplace, however, can achieve parity with the former by offering a deeper discount rate for the reserved instances, thereby bypassing the cost of administering the secondary marketplace.

Monetary Policy and Welfare with Heterogeneous Firms and Endogenous Entry

AbstractThis paper studies monetary policy and welfare in a sticky wage New Keynesian model with heterogeneous firms and endogenously variable markups. We show that stabilizing nominal wages is optimal only when product creation is based on an instantaneous zero‐profit condition and when the aggregate markup is constant. A constant markup requires strong selection effects generated by a Pareto firm‐level productivity distribution. When product creation is based on a dynamic zero‐profit condition optimal monetary policy accounts for the distribution of firms and the welfare loss from stabilizing nominal wages is between and 0.2% of steady‐state consumption.

Performance Analysis of Multi-Tote Storage and Retrieval Autonomous Mobile Robot Systems

Multi-tote storage and retrieval (MTSR) autonomous mobile robots can carry multiple product totes, store and retrieve them from different shelf rack tiers, and transport them to a workstation where the products are picked to fulfill customer orders. In each robot trip, totes retrieved during the previous trip must be stored. This leads to a mixed storage and retrieval route. We analyze this mixed storage and retrieval route problem and derive the optimal travel route for a multiblock warehouse by a layered graph algorithm, based on storage first-retrieval second and mixed storage and retrieval policies. We also propose an effective heuristic routing policy, the closest retrieval (CR) sequence policy, based on a local shortest path. Numerical results show that the CR policy leads to shorter travel times than the well-known S-shape policy, whereas the gap with the optimal mixed storage and retrieval policy in practical scenarios is small. Based on the CR policy, we model the stochastic behavior of the system using a semiopen queuing network (SOQN). This model can accurately estimate average tote throughput time and system throughput capacity as a function of the number of robots in the system. We use the SOQN and corresponding closed queuing network models to optimize the total annual cost as a function of the warehouse shape, the number of robots, and tote buffer positions on the robots for a given average tote throughput time and throughput capacity. Compared with robots that retrieve a single tote per trip, an MTSR system with at least five buffer positions can achieve lower operational costs while meeting given average tote throughput time and tote throughput capacity constraints. Funding: This work was supported by National Natural Science Foundation of China [Grant 72372088] and the Shenzhen Science and Technology Program [Grant GJHZ20220913143003006]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/trsc.2023.0397 .

Research on a Personalized Decision Control Algorithm for Autonomous Vehicles Based on the Reinforcement Learning from Human Feedback Strategy

To address the shortcomings of previous autonomous decision models, which often overlook the personalized features of users, this paper proposes a personalized decision control algorithm for autonomous vehicles based on RLHF (reinforcement learning from human feedback). The algorithm combines two reinforcement learning approaches, DDPG (Deep Deterministic Policy Gradient) and PPO (proximal policy optimization), and divides the control scheme into three phases including pre-training, human evaluation, and parameter optimization. During the pre-training phase, an agent is trained using the DDPG algorithm. In the human evaluation phase, different trajectories generated by the DDPG-trained agent are scored by individuals with different styles, and the respective reward models are trained based on the trajectories. In the parameter optimization phase, the network parameters are updated using the PPO algorithm and the reward values given by the reward model to achieve personalized autonomous vehicle control. To validate the control algorithm designed in this paper, a simulation scenario was built using CARLA_0.9.13 software. The results demonstrate that the proposed algorithm can provide personalized decision control solutions for different styles of people, satisfying human needs while ensuring safety.

Gate Assignment Algorithm for Airport Peak Time Based on Reinforcement Learning

In existing airport gate allocation studies, little consideration has been given to situations where gate resources are limited during peak periods. Under such circumstances, some flights may not be able to make regular stops. In this paper, the airport gate assignment problem under peak time is investigated. We propose a gate pre-assignment model to maximize the gate matching degree and the near gate passenger allocation rate. Besides, to minimize the pre-assignment gate change rate, we propose a dynamic reassignment model based on the pre-assignment model. By considering the non-deterministic polynomial hard (NP-hard) property of this problem, a gate assignment algorithm based on proximal policy optimization (GABPPO) is proposed. The simulation results show that the algorithm can effectively solve the gate shortage problem during the airport peak period. Compared with the adaptive parallel genetic, deep Q-network, and policy gradient algorithms, the target value of solutions obtained by the proposed algorithm in the near gate passenger allocation rate is increased by 5.7%, 3.6%, and 7.9%, respectively, and the target value in the gate matching degree is increased by 10.6%, 4.9%, and 11.5% respectively.

Advanced Liquid-Metal-Cooled Fast Reactor Core Design Using Modern Optimization Algorithms

When performing core loading pattern design in fast spectrum reactors, it is often assumed that the larger neutron mean free path in fast reactors makes the core loading pattern less significant than in thermal reactors. Due to this assumption, the literature often includes homogeneous core designs for liquid-metal-cooled fast reactors (LMFRs). In this paper, heterogeneous loading patterns are investigated using modern LMFR multiphysics analysis. It was found that the figures of merit (FOMs) used to design cores are very sensitive to the core loading pattern, and better core designs (measured by the FOM) can be obtained from heterogeneous designs. Based on this investigation, a need for a modern LMFR core loading pattern design methodology is identified and developed. The methodology is demonstrated through the optimization of the core loading pattern of a sodium-cooled fast reactor. The reactor design used in this demonstration is based on the Super Power Reactor Innovative Small Modular (SPRISM) core, and an optimized loading pattern is obtained by searching for the fuel enrichment and locations of the driver and blanket assemblies. The objectives of the search are to reduce the fuel cost and peaking factors, while meeting the design constraints. To calculate the fuel cost, a preliminary cost model is developed and applied for transuranium fuel loading. Upon establishing the methodology, six optimization algorithms are tested for their effectiveness in solving the LMFR core loading pattern problem. Four of the algorithms are population based: Gray Wolf Optimization, Salp Swarm Optimization, Whale Optimization Algorithm, and the Moth Flame Optimization. A reinforcement learning algorithm, called the proximal policy optimization, was also selected. Finally, the differential evolution (DE) algorithm was selected as the choice of an evolutionary-based algorithm. All algorithms showed a competitive converged design. However, the DE showed more favorable performance since it was able to converge to a superior design compared to the rest of the algorithms with a reasonable number of design sample evaluations, in addition to avoiding local minima entrapment.

Markov models for clinical decision-making in radiation oncology: A systematic review.

The intrinsic stochasticity of patients' response to treatment is a major consideration for clinical decision-making in radiation therapy. Markov models are powerful tools to capture this stochasticity and render effective treatment decisions. This paper provides an overview of the Markov models for clinical decision analysis in radiation oncology. A comprehensive literature search was conducted within MEDLINE using PubMed, following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Only studies published from 2000 to 2023 were considered. Selected publications were summarized in two categories: (i) studies that compare two (or more) fixed treatment policies using Monte Carlo simulation and (ii) studies that seek an optimal treatment policy through Markov Decision Processes (MDPs). Relevant to the scope of this study, 61 publications were selected for detailed review. The majority of these publications (n = 56) focused on comparative analysis of two or more fixed treatment policies using Monte Carlo simulation. Classifications based on cancer site, utility measures and the type of sensitivity analysis are presented. Five publications considered MDPs with the aim of computing an optimal treatment policy; a detailed statement of the analysis and results is provided for each work. As an extension of Markov model-based simulation analysis, MDP offers a flexible framework to identify an optimal treatment policy among a possibly large set of treatment policies. However, the applications of MDPs to oncological decision-making have been understudied, and the full capacity of this framework to render complex optimal treatment decisions warrants further consideration.

Comparing Human Behavior to an Optimal Policy for Innovation

Human learning does not stop at solving a single problem. Instead, we seek new challenges, define new goals, and come up with new ideas. Unlike the classic explore-exploit trade-off between known and unknown options, making new tools or generating new ideas is not about collecting data from existing unknown options, but rather about create new options out of what is currently available. We introduce a discovery game designed to study how rational agents make decisions about pursuing innovations, where discovering new ideas is a process of combining existing ideas in an open-ended compositional space. We derive optimal policies of this decision problem formalized as a Markov decision process, and compare people's behaviors to the model predictions in an online behavioral experiment. We found evidence that people both innovate rationally, guided by potential returns in this discovery game, and under- and over-explore systematically in different settings.

The impact of climate change on vehicle engineering and countermeasures

With the increasingly severe climate change, the field of vehicle engineering is facing numerous challenges and opportunities. This article aims to analyze the impact of climate change on vehicle engineering and propose corresponding strategies. Firstly, the research background and significance are introduced, and the current research status both domestically and internationally is analyzed. Following that, a comprehensive analysis is conducted on the impact of climate change on vehicle engineering, including the influence of current climate change on the development of vehicle engineering, the impact of low-carbon transportation on the development of vehicle engineering, and the influence of climate change on vehicle safety. Then, the role of vehicle engineering in responding to climate change is discussed in detail. Finally, development strategies for vehicle engineering are proposed, including the development of new energy vehicles, improvement of autonomous driving technology, and optimization of transportation policies. This study explains the impact of climate change on vehicle engineering and proposes strategies for long-term development.

Construction of copulas for bivariate failure rates

AbstractThis paper aims to develop a method to construct an asymmetric copula, based on which a closed form of the cumulative bivariate failure rate can be obtained. The construction method differs from existing ones. This new method can facilitate the derivation of some results such as the estimation of the expected number of occurrences for a system whose failure process is modelled by a bivariate stochastic process or the expected cost in optimisation of maintenance policies.

Adaptive event‐triggered control and observer design for discrete‐time nonlinear Markov jump systems with DoS attacks using policy iteration‐based adaptive dynamic programming

AbstractThis article addresses adaptive event‐triggered discrete‐time nonlinear Markov jump systems (MJs) with DoS attacks, where the introduced DoS attacks are considered as more general stochastic models with fixed trigger frequency and duration. To solve the optimal control problem, we use an adaptive dynamic programming (ADP) algorithm based on policy iteration (PI). The approximate process is as follows: the performance index function (PIF) is first updated by the iteration policy in advance, and the control policy is obtained from the PIF. Subsequently, an approximate estimation of the optimal PIF and the optimal control policy is made using the actor‐critic structure obtained through neural network techniques. In order to reduce the occupied communication resources required for control policy iteration, we introduce an adaptive event triggering mechanism with an adaptive triggering threshold, which reduces the conservatism of resource occupation by the PIF compared to the fixed‐threshold ETM. In addition, an observer identifying the unknown dynamics part of the system is designed. Finally, using the Lyapunov function, it is shown that the designed control policy ensures the stability and convergence of the MJS, and the designed observer is effective. Simulation examples are given to verify the feasibility of the controller and the observer.

Maintaining ecological connectivity in highly urbanized areas: Optimal strategy, key patches and protection policy

Maintaining and restoring ecological connectivity (EC) are important measures of mitigating habitat isolation caused by urbanization. Considering the prominent contradiction of land use in highly urbanized areas, the EC protection measures proposed in the previous studies, such as expanding the ecological sources and widening the ecological corridors, are unrealistic. By using the stepping-stone and complex network theories, this paper simulated the ecological network (EN), evaluated the centrality of each ecological source and stepping-stone, and analysed the network robustness under different damage scenarios, then identified the optimal strategy and key patches for EC maintaining. Meanwhile, this paper evaluated whether the key patches are protected by China’s new policies, which are, Ecological Protection Red Line (EPRL) and the Three Lines One Permit (TLOP). The results were as follows: (1) The EN is composed of 53 ecological sources and 291 stepping-stones, with 591 links between them. (2) The network stability can be optimally maintained by prioritizing the protection of stepping-stones with high centrality. (3) The 53 ecological sources and 51 stepping-stones with the highest centrality can ensure the stable operation of the EN, that is, they are key patches for EC maintaining. (4) Compared with EPRL, the protection rate for ecological sources and stepping-stones are significantly improved by TLOP; however, stepping-stones with high centrality require stronger protection. This study enriched the research cases of complex network theory applied in landscape ecology, and provided references for the improvement of policies in the future.

An adaptive PID controller for path following of autonomous underwater vehicle based on Soft Actor–Critic

In recent years, autonomous underwater vehicles (AUVs) have witnessed rapid development, and its motion control has garnered increasing attention. Meanwhile, in industries, PID controllers are still wildly employed by most AUVs due to their simplicity, ease of deployment, and a certain level of robustness. However, they are facing significant challenges in parameter tuning, especially when dealing with various control missions and changing external environments. Deep reinforcement learning, as a data-driven approach, has gradually made its impact in AUV control. However, its lack of interpretability has hindered its deployment in relevant experiments. To address these issues, this paper proposed an adaptive PID controller for path following of AUVs based on the Soft Actor–Critic (SAC). This controller combines the interpretability of PID with the intelligence of reinforcement learning. A simulation platform was established and compared with other typical control methods, demonstrating the superiority of the proposed controller. Finally, the feasibility of the proposed SAC-PID controller was validated by lake trials. The results showed that the SAC-PID controller significantly outperformed the PID and Proximal Policy Optimization (PPO) PID controllers in multiple dimensions, such as control precision and convergence speed.

Portfolio management based on a reinforcement learning framework

AbstractPortfolio management is crucial for investors. We propose a dynamic portfolio management framework based on reinforcement learning using the proximal policy optimization algorithm. The two‐part framework includes a feature extraction network and a full connected network. First, the majority of the previous research on portfolio management based on reinforcement learning has been dedicated to discrete action spaces. We propose a potential solution to the problem of a continuous action space with a constraint (i.e., the sum of the portfolio weights is equal to 1). Second, we explore different feature extraction networks (i.e., convolutional neural network [CNN], long short‐term memory [LSTM] network, and convolutional LSTM network) combined with our system, and we conduct extensive experiments on the six kinds of assets, including 16 features. The empirical results show that the CNN performs best in the test set. Last, we discuss the effect of the trading frequency on our trading system and find that the monthly trading frequency has a higher Sharpe ratio in the test set than other trading frequencies.