Optimal Policy Research Articles

Green subsidies as strategic trade policy tools

AbstractIn a third-country market model in which two export countries adopt environmental policies (taxes and subsidies), this article analyses how an abatement (“green”) subsidy can become a potential strategic trade policy tool. When governments set the optimal policy considering their local environmental damages, a rich set of equilibria arises. In contrast to the standard result, it is shown that subsidising pollution abatement can 1) emerge as the unique Pareto-efficient Nash equilibrium of the policy game, 2) be the only feasible environmental policy when environmental awareness is low, irrespective of the efficiency of the cleaning technology, and 3) emerge as the unique Pareto-inefficient Nash equilibrium of the policy game at the end of the ecological transition. The article also tackles some dynamic issues that the policy game implies.

Decentralized Online Order Fulfillment in Omni-Channel Retailers

We consider an order fulfillment problem of an omni-channel retailer that ships online orders from its distribution center (DC) and brick-and-mortar stores. Stores use their local information, not observed by the retailer, that can lead them to accept or reject fulfillment requests of items in an online order. We investigate the problem of sequencing requests to stores and inventory rationing decisions at the DC to minimize expected costs under uncertain store acceptance behavior and when items are indistinguishable in terms of shipping. First, under the scenario that stores are used only when the DC has insufficient inventory, we propose a Markov Decision Process formulation and analyze the performance of myopic policies that are preferable because of their interpretability. We show that the performance rate of a myopic approach that orders stores by cost only depends on the number of items in an order, which is small in practice. We also determine conditions for the range of acceptance probabilities for the myopic policy to be optimal for small-sized orders. Using optimality conditions for a special case of the problem, we develop an adaptive variant of the myopic policy, and propose a new degree-based strategy that balances shipping costs and acceptance probabilities. Numerical testing suggests that the best-performing sequencing policy is within 1% of optimality on average. Moreover, using two years of data from a large omni-channel retailer in North America, we observe that adaptive policies, albeit more complex, are beneficial in reducing costs and split deliveries if acceptance rates can be estimated accurately. Second, we determine when the retailer should ship from stores or ration the inventory at the DC. We show that for single-item orders, the optimal policy has a threshold structure, where, remarkably, the highest priority region is also subject to rationing. We then consider the novel multi-unit-single-item rationing problem, and leverage the structure of the single-unit model to develop a heuristic. We numerically establish the efficacy of rationing models and our heuristic.

Cost Optimization in Cloud Computing: Capacity Reservation for Intermittent Random Demand Surges

The adoption of cloud computing has been accelerating over the last decade, while enterprise cloud users (“firms”) are struggling to manage their growing cloud expenditures in the face of intermittent digital demand surges caused by planned or random events. To deal with such challenges, a firm can employ reserved instances with the standard contract length (e.g., one year) to meet the stationary base demand, which we refer to as the base contracts, complemented by additional reserved instances with either standard or shorter contract lengths, which we refer to as the supplementary contracts, to cope with the demand surges. We first analyze a model whereby the surge and inter-surge durations are deterministic, demand magnitude is random, and cancellation of the reservations is allowed. We develop a capacity management plan for the firm including not only the optimal capacity levels, which follow a newsvendor-type solution, but also the optimal policy for managing the purchase, renewal, cancellation, or expiration of the supplementary contracts, which can be characterized as a two-threshold policy. Due to the complexity of the structure of the optimal policy, we also construct an effective heuristic policy by excluding the renewal option from the action space, which can be applied to a more general setting where the surge and inter-surge durations are random. We examine two model extensions: (1) when trades of reserved instances are allowed in a secondary marketplace; (2) when the firm does not have exact information about the distributions of the surge magnitude and duration while it can adjust the capacity levels as data unveils. Our analysis shows that the optimal policy for managing the supplementary contracts depends on the relative magnitude of the surge and inter-surge durations in relationship to the cancellation fee. Moreover, our numerical results show that cloud platforms that offer a secondary marketplace are more attractive to firms from a cost standpoint than those that allow cancellation only. The latter, without the secondary marketplace, however, can achieve parity with the former by offering a deeper discount rate for the reserved instances, thereby bypassing the cost of administering the secondary marketplace.

Optimal insurance with mean-deviation measures

This paper studies an optimal insurance contracting problem in which the preferences of the decision maker are given by the sum of the expected loss and a convex, increasing function of a deviation measure. As for the deviation measure, our focus is on convex signed Choquet integrals (such as the Gini coefficient and a convex distortion risk measure minus the expected value) and on the standard deviation. We find that if the expected value premium principle is used, then stop-loss indemnities are optimal, and we provide a precise characterization of the corresponding deductible. Moreover, if the premium principle is based on Value-at-Risk or Expected Shortfall, then a particular layer-type indemnity is optimal, in which there is coverage for small losses up to a limit, and additionally for losses beyond another deductible. The structure of these optimal indemnities remains unchanged if there is a limit on the insurance premium budget. If the unconstrained solution is not feasible, then the deductible is increased to make the budget constraint binding. We provide several examples of these results based on the Gini coefficient and the standard deviation.

Monetary Policy and Welfare with Heterogeneous Firms and Endogenous Entry

AbstractThis paper studies monetary policy and welfare in a sticky wage New Keynesian model with heterogeneous firms and endogenously variable markups. We show that stabilizing nominal wages is optimal only when product creation is based on an instantaneous zero‐profit condition and when the aggregate markup is constant. A constant markup requires strong selection effects generated by a Pareto firm‐level productivity distribution. When product creation is based on a dynamic zero‐profit condition optimal monetary policy accounts for the distribution of firms and the welfare loss from stabilizing nominal wages is between and 0.2% of steady‐state consumption.

Performance Analysis of Multi-Tote Storage and Retrieval Autonomous Mobile Robot Systems

Multi-tote storage and retrieval (MTSR) autonomous mobile robots can carry multiple product totes, store and retrieve them from different shelf rack tiers, and transport them to a workstation where the products are picked to fulfill customer orders. In each robot trip, totes retrieved during the previous trip must be stored. This leads to a mixed storage and retrieval route. We analyze this mixed storage and retrieval route problem and derive the optimal travel route for a multiblock warehouse by a layered graph algorithm, based on storage first-retrieval second and mixed storage and retrieval policies. We also propose an effective heuristic routing policy, the closest retrieval (CR) sequence policy, based on a local shortest path. Numerical results show that the CR policy leads to shorter travel times than the well-known S-shape policy, whereas the gap with the optimal mixed storage and retrieval policy in practical scenarios is small. Based on the CR policy, we model the stochastic behavior of the system using a semiopen queuing network (SOQN). This model can accurately estimate average tote throughput time and system throughput capacity as a function of the number of robots in the system. We use the SOQN and corresponding closed queuing network models to optimize the total annual cost as a function of the warehouse shape, the number of robots, and tote buffer positions on the robots for a given average tote throughput time and throughput capacity. Compared with robots that retrieve a single tote per trip, an MTSR system with at least five buffer positions can achieve lower operational costs while meeting given average tote throughput time and tote throughput capacity constraints. Funding: This work was supported by National Natural Science Foundation of China [Grant 72372088] and the Shenzhen Science and Technology Program [Grant GJHZ20220913143003006]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/trsc.2023.0397 .

Sustainable management of predatory fish affected by an Allee effect through marine protected areas and taxation

Ecological balance and stable economic development are crucial for the fishery. This study proposes a predator–prey system for marine communities, where the growth of predators follows the Allee effect and takes into account the rapid fluctuations in resource prices caused by supply and demand. The system predicts the existence of catastrophic equilibrium, which may lead to the extinction of prey, consequently leading to the extinction of predators, but fishing efforts remain high. Marine protected areas are established near fishing areas to avoid such situations. Fish migrate rapidly between these two areas and are only harvested in the nonprotected areas. A three-dimensional simplified model is derived by applying variable aggregation to describe the variation of global variables on a slow time scale. To seek conditions to avoid species extinction and maintain sustainable fishing activities, the existence of positive equilibrium points and their local stability are explored based on the simplified model. Moreover, the long-term impact of establishing marine protected areas and levying taxes based on unit catch on fishery dynamics is studied, and the optimal tax policy is obtained by applying Pontryagin’s maximum principle. The theoretical analysis and numerical examples of this study demonstrate the comprehensive effectiveness of increasing the proportion of marine protected areas and controlling taxes on the sustainable development of fishery.

Research on a Personalized Decision Control Algorithm for Autonomous Vehicles Based on the Reinforcement Learning from Human Feedback Strategy

To address the shortcomings of previous autonomous decision models, which often overlook the personalized features of users, this paper proposes a personalized decision control algorithm for autonomous vehicles based on RLHF (reinforcement learning from human feedback). The algorithm combines two reinforcement learning approaches, DDPG (Deep Deterministic Policy Gradient) and PPO (proximal policy optimization), and divides the control scheme into three phases including pre-training, human evaluation, and parameter optimization. During the pre-training phase, an agent is trained using the DDPG algorithm. In the human evaluation phase, different trajectories generated by the DDPG-trained agent are scored by individuals with different styles, and the respective reward models are trained based on the trajectories. In the parameter optimization phase, the network parameters are updated using the PPO algorithm and the reward values given by the reward model to achieve personalized autonomous vehicle control. To validate the control algorithm designed in this paper, a simulation scenario was built using CARLA_0.9.13 software. The results demonstrate that the proposed algorithm can provide personalized decision control solutions for different styles of people, satisfying human needs while ensuring safety.

Joint learning of reward machines and policies in environments with partially known semantics

We study the problem of reinforcement learning for a task encoded by a reward machine. The task is defined over a set of properties in the environment, called atomic propositions, and represented by Boolean variables. One unrealistic assumption commonly used in the literature is that the truth values of these propositions are accurately known. In real situations, however, these truth values are uncertain since they come from sensors that suffer from imperfections. At the same time, reward machines can be difficult to model explicitly, especially when they encode complicated tasks. We develop a reinforcement-learning algorithm that infers a reward machine that encodes the underlying task while learning how to execute it, despite the uncertainties of the propositions' truth values. In order to address such uncertainties, the algorithm maintains a probabilistic estimate about the truth value of the atomic propositions; it updates this estimate according to new sensory measurements that arrive from exploration of the environment. Additionally, the algorithm maintains a hypothesis reward machine, which acts as an estimate of the reward machine that encodes the task to be learned. As the agent explores the environment, the algorithm updates the hypothesis reward machine according to the obtained rewards and the estimate of the atomic propositions' truth value. Finally, the algorithm uses a Q-learning procedure for the states of the hypothesis reward machine to determine an optimal policy that accomplishes the task. We prove that the algorithm successfully infers the reward machine and asymptotically learns a policy that accomplishes the respective task.

Gate Assignment Algorithm for Airport Peak Time Based on Reinforcement Learning

In existing airport gate allocation studies, little consideration has been given to situations where gate resources are limited during peak periods. Under such circumstances, some flights may not be able to make regular stops. In this paper, the airport gate assignment problem under peak time is investigated. We propose a gate pre-assignment model to maximize the gate matching degree and the near gate passenger allocation rate. Besides, to minimize the pre-assignment gate change rate, we propose a dynamic reassignment model based on the pre-assignment model. By considering the non-deterministic polynomial hard (NP-hard) property of this problem, a gate assignment algorithm based on proximal policy optimization (GABPPO) is proposed. The simulation results show that the algorithm can effectively solve the gate shortage problem during the airport peak period. Compared with the adaptive parallel genetic, deep Q-network, and policy gradient algorithms, the target value of solutions obtained by the proposed algorithm in the near gate passenger allocation rate is increased by 5.7%, 3.6%, and 7.9%, respectively, and the target value in the gate matching degree is increased by 10.6%, 4.9%, and 11.5% respectively.

Raising America’s future: Search for optimal child-related transfers

The US differs from other OECD countries in terms of family policy size and composition. This study examines the welfare and macroeconomic effects of family policy reforms. I explore three policy instruments: child-related tax credits, childcare subsidies, and child allowances. The children are merit good due to pay-as-you-go social security structure. I show that expanding family policy, like the American Rescue Plan, improves welfare. I also define the optimal family policy for the United States. It accounts for about 3% of GDP, three times larger than the existing policy, and primarily focused on childcare subsidies. Family policy structure is critical for welfare evaluation because similar expenditure levels can result in contrasting welfare outcomes depending on policy composition. This study emphasizes the importance of carefully designed family policies, as well as the need for ongoing research and policy innovation to maximize societal benefits and promote equitable economic growth.

Optimal scheduling imperfect maintenance policy for a system with multiple random works

Optimal scheduling imperfect maintenance policy for a system with multiple random works

Advanced Liquid-Metal-Cooled Fast Reactor Core Design Using Modern Optimization Algorithms

When performing core loading pattern design in fast spectrum reactors, it is often assumed that the larger neutron mean free path in fast reactors makes the core loading pattern less significant than in thermal reactors. Due to this assumption, the literature often includes homogeneous core designs for liquid-metal-cooled fast reactors (LMFRs). In this paper, heterogeneous loading patterns are investigated using modern LMFR multiphysics analysis. It was found that the figures of merit (FOMs) used to design cores are very sensitive to the core loading pattern, and better core designs (measured by the FOM) can be obtained from heterogeneous designs. Based on this investigation, a need for a modern LMFR core loading pattern design methodology is identified and developed. The methodology is demonstrated through the optimization of the core loading pattern of a sodium-cooled fast reactor. The reactor design used in this demonstration is based on the Super Power Reactor Innovative Small Modular (SPRISM) core, and an optimized loading pattern is obtained by searching for the fuel enrichment and locations of the driver and blanket assemblies. The objectives of the search are to reduce the fuel cost and peaking factors, while meeting the design constraints. To calculate the fuel cost, a preliminary cost model is developed and applied for transuranium fuel loading. Upon establishing the methodology, six optimization algorithms are tested for their effectiveness in solving the LMFR core loading pattern problem. Four of the algorithms are population based: Gray Wolf Optimization, Salp Swarm Optimization, Whale Optimization Algorithm, and the Moth Flame Optimization. A reinforcement learning algorithm, called the proximal policy optimization, was also selected. Finally, the differential evolution (DE) algorithm was selected as the choice of an evolutionary-based algorithm. All algorithms showed a competitive converged design. However, the DE showed more favorable performance since it was able to converge to a superior design compared to the rest of the algorithms with a reasonable number of design sample evaluations, in addition to avoiding local minima entrapment.

Enhancing NOMA Network Security with RIS-UAV Integration: Exploring PPO Technique

In order to improve the security aspects of a nonorthogonal multiple access (NOMA) downlink network, we investigate the use of the proximal policy optimization (PPO) technique in this study. This network is uniquely augmented with reconfi- gurable intelligent surfaces (RIS) mounted on unmanned aerial vehicles (UAVs). The main objective of this article is to prevent possible eavesdroppers from accessing the network while still assuring continuous and secure connectivity for legitimate users. The network configuration includes a UAV outfitted with a RIS, which plays an important role in signal propagation. This work aims to increase the secrecy rates of all user communications in eavesdropping prone locations. By adaptively modifying the phase shifts and power distributions through the RIS the proposed PPO algorithm not only shows adaptability and effectiveness in securing wireless communications, but also highlights the significant advances made in communication system security through this technology.

Markov models for clinical decision-making in radiation oncology: A systematic review.

The intrinsic stochasticity of patients' response to treatment is a major consideration for clinical decision-making in radiation therapy. Markov models are powerful tools to capture this stochasticity and render effective treatment decisions. This paper provides an overview of the Markov models for clinical decision analysis in radiation oncology. A comprehensive literature search was conducted within MEDLINE using PubMed, following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Only studies published from 2000 to 2023 were considered. Selected publications were summarized in two categories: (i) studies that compare two (or more) fixed treatment policies using Monte Carlo simulation and (ii) studies that seek an optimal treatment policy through Markov Decision Processes (MDPs). Relevant to the scope of this study, 61 publications were selected for detailed review. The majority of these publications (n = 56) focused on comparative analysis of two or more fixed treatment policies using Monte Carlo simulation. Classifications based on cancer site, utility measures and the type of sensitivity analysis are presented. Five publications considered MDPs with the aim of computing an optimal treatment policy; a detailed statement of the analysis and results is provided for each work. As an extension of Markov model-based simulation analysis, MDP offers a flexible framework to identify an optimal treatment policy among a possibly large set of treatment policies. However, the applications of MDPs to oncological decision-making have been understudied, and the full capacity of this framework to render complex optimal treatment decisions warrants further consideration.

Comparing Human Behavior to an Optimal Policy for Innovation

Human learning does not stop at solving a single problem. Instead, we seek new challenges, define new goals, and come up with new ideas. Unlike the classic explore-exploit trade-off between known and unknown options, making new tools or generating new ideas is not about collecting data from existing unknown options, but rather about create new options out of what is currently available. We introduce a discovery game designed to study how rational agents make decisions about pursuing innovations, where discovering new ideas is a process of combining existing ideas in an open-ended compositional space. We derive optimal policies of this decision problem formalized as a Markov decision process, and compare people's behaviors to the model predictions in an online behavioral experiment. We found evidence that people both innovate rationally, guided by potential returns in this discovery game, and under- and over-explore systematically in different settings.

The impact of climate change on vehicle engineering and countermeasures

With the increasingly severe climate change, the field of vehicle engineering is facing numerous challenges and opportunities. This article aims to analyze the impact of climate change on vehicle engineering and propose corresponding strategies. Firstly, the research background and significance are introduced, and the current research status both domestically and internationally is analyzed. Following that, a comprehensive analysis is conducted on the impact of climate change on vehicle engineering, including the influence of current climate change on the development of vehicle engineering, the impact of low-carbon transportation on the development of vehicle engineering, and the influence of climate change on vehicle safety. Then, the role of vehicle engineering in responding to climate change is discussed in detail. Finally, development strategies for vehicle engineering are proposed, including the development of new energy vehicles, improvement of autonomous driving technology, and optimization of transportation policies. This study explains the impact of climate change on vehicle engineering and proposes strategies for long-term development.

Construction of copulas for bivariate failure rates

AbstractThis paper aims to develop a method to construct an asymmetric copula, based on which a closed form of the cumulative bivariate failure rate can be obtained. The construction method differs from existing ones. This new method can facilitate the derivation of some results such as the estimation of the expected number of occurrences for a system whose failure process is modelled by a bivariate stochastic process or the expected cost in optimisation of maintenance policies.

Adaptive event‐triggered control and observer design for discrete‐time nonlinear Markov jump systems with DoS attacks using policy iteration‐based adaptive dynamic programming

AbstractThis article addresses adaptive event‐triggered discrete‐time nonlinear Markov jump systems (MJs) with DoS attacks, where the introduced DoS attacks are considered as more general stochastic models with fixed trigger frequency and duration. To solve the optimal control problem, we use an adaptive dynamic programming (ADP) algorithm based on policy iteration (PI). The approximate process is as follows: the performance index function (PIF) is first updated by the iteration policy in advance, and the control policy is obtained from the PIF. Subsequently, an approximate estimation of the optimal PIF and the optimal control policy is made using the actor‐critic structure obtained through neural network techniques. In order to reduce the occupied communication resources required for control policy iteration, we introduce an adaptive event triggering mechanism with an adaptive triggering threshold, which reduces the conservatism of resource occupation by the PIF compared to the fixed‐threshold ETM. In addition, an observer identifying the unknown dynamics part of the system is designed. Finally, using the Lyapunov function, it is shown that the designed control policy ensures the stability and convergence of the MJS, and the designed observer is effective. Simulation examples are given to verify the feasibility of the controller and the observer.

Maintaining ecological connectivity in highly urbanized areas: Optimal strategy, key patches and protection policy

Maintaining and restoring ecological connectivity (EC) are important measures of mitigating habitat isolation caused by urbanization. Considering the prominent contradiction of land use in highly urbanized areas, the EC protection measures proposed in the previous studies, such as expanding the ecological sources and widening the ecological corridors, are unrealistic. By using the stepping-stone and complex network theories, this paper simulated the ecological network (EN), evaluated the centrality of each ecological source and stepping-stone, and analysed the network robustness under different damage scenarios, then identified the optimal strategy and key patches for EC maintaining. Meanwhile, this paper evaluated whether the key patches are protected by China’s new policies, which are, Ecological Protection Red Line (EPRL) and the Three Lines One Permit (TLOP). The results were as follows: (1) The EN is composed of 53 ecological sources and 291 stepping-stones, with 591 links between them. (2) The network stability can be optimally maintained by prioritizing the protection of stepping-stones with high centrality. (3) The 53 ecological sources and 51 stepping-stones with the highest centrality can ensure the stable operation of the EN, that is, they are key patches for EC maintaining. (4) Compared with EPRL, the protection rate for ecological sources and stepping-stones are significantly improved by TLOP; however, stepping-stones with high centrality require stronger protection. This study enriched the research cases of complex network theory applied in landscape ecology, and provided references for the improvement of policies in the future.