Optimal Stationary Policy Research Articles

Optimal control of production time of perishable inventory system with postponed demands

We consider a continuous review production inventory system in which the demands arrive according to a Poisson process. We assume that the demands that occur during stock out periods enter a pool which has finite capacity. Any demand that arrives when the pool is full and the inventory level is zero, is also assumed to be lost. The demands in the pool are selected one by one, if the stock is above the prefixed value, with interval time between any two successive selections is distributed as exponential. In order to replenish stock, a production facility is switched on when the inventory level drops to prefixed value s and items are produced one by one until the stock level reaches a maximum capacity S(> s). The successive production time is assumed to follow an exponential distribution whose parameter is chosen from a given set of positive values. The objective of this paper is to investigate the optimal set of production rates that minimises the total expected cost per unit time. We formulate this problem as a semi-Markov decision problem. The stationary optimal policy is computed using linear programming algorithm and the results are illustrated numerically.

Stochastic Dynamic Programming with Non-linear Discounting

In this paper, we study a Markov decision process with a non-linear discount function and with a Borel state space. We define a recursive discounted utility, which resembles non-additive utility functions considered in a number of models in economics. Non-additivity here follows from non-linearity of the discount function. Our study is complementary to the work of Jaśkiewicz et al. (Math Oper Res 38:108–121, 2013), where also non-linear discounting is used in the stochastic setting, but the expectation of utilities aggregated on the space of all histories of the process is applied leading to a non-stationary dynamic programming model. Our aim is to prove that in the recursive discounted utility case the Bellman equation has a solution and there exists an optimal stationary policy for the problem in the infinite time horizon. Our approach includes two cases: (a) when the one-stage utility is bounded on both sides by a weight function multiplied by some positive and negative constants, and (b) when the one-stage utility is unbounded from below.

Dynamic pricing and fleet management for electric autonomous mobility on demand systems

The proliferation of ride sharing systems is a major drive in the advancement of autonomous and electric vehicle technologies. This paper considers the joint routing, battery charging, and pricing problem faced by a profit-maximizing transportation service provider that operates a fleet of autonomous electric vehicles. We first establish the static planning problem by considering time-invariant system parameters and determine the optimal static policy. While the static policy provides stability of customer queues waiting for rides even if consider the system dynamics, we see that it is inefficient to utilize a static policy as it can lead to long wait times for customers and low profits. To accommodate for the stochastic nature of trip demands, renewable energy availability, and electricity prices and to further optimally manage the autonomous fleet given the need to generate integer allocations, a real-time policy is required. The optimal real-time policy that executes actions based on full state information of the system is the solution of a complex dynamic program. However, we argue that it is intractable to exactly solve for the optimal policy using exact dynamic programming methods and therefore apply deep reinforcement learning to develop a near-optimal control policy. The two case studies we conducted in Manhattan and San Francisco demonstrate the efficacy of our real-time policy in terms of network stability and profits, while keeping the queue lengths up to 200 times less than the static policy.

Discounted Markov decision processes with fuzzy costs

Fuzzy theory is a discipline that has recently appeared in the mathematical literature. It generalizes classic situations. Therefore, its success continues to increase and to keep going up from time to time. In this work, we consider the model of Markov decision processes where the information on the costs includes imprecision. The fuzzy cost is represented by the fuzzy number set and the infinite horizon discounted cost is minimized from any stationary policy. This paper presents in the first part the notion of fuzzy sets and some axiomatic basis and relevant concepts with fuzzy theory in short. In second part, we propose a new definition of total discounted fuzzy cost in infinite planning horizon. We will compute an optimal stationary policy that minimizes the total fuzzy discounted cost by a new approach based on some standard algorithms of the dynamic programming using the ranking function concept. The last adapted criterion has many applications in several areas such that the forest management, the management of energy consumption, the finance, the communication system (mobile networks).

Minimizing the Age of Information of Cognitive Radio-Based IoT Systems Under a Collision Constraint

This article considers a cognitive radio-based IoT monitoring system, consisting of an IoT device that aims to update its measurement to a destination using cognitive radio technique. Specifically, the IoT device as a secondary user (SIoT), seeks and exploits the spectrum opportunities of the licensed band vacated by its primary user (PU) to deliver status updates without causing visible effects to the licensed operation. In this context, the SIoT should carefully make use of the licensed band and schedule when to transmit to maintain the timeliness of the status update. The timeliness of the status update characterizes how the destination knows the latest information of the SIoT. We adopt a recent metric, Age of Information (AoI), to characterize the timeliness of the status update of the SIoT. We aim to minimize the long-term average AoI of the SIoT while satisfying the collision constraint imposed by the PU by formulating a constrained Markov decision process (CMDP) problem. We first prove the existence of optimal stationary policy of the CMDP problem. The optimal stationary policy (termed age-optimal policy) is shown to be a randomized simple policy that randomizes between two deterministic policies with a fixed probability. We prove that the two deterministic policies have a threshold structure and further derive the closed-form expression of average AoI and collision probability for the deterministic threshold-structured policy by conducting Markov Chain analysis. The analytical expression offers an efficient way to calculate the threshold and randomization probability to form the age-optimal policy. For comparison, we also consider the throughput maximization policy (termed throughput-optimal policy) and analyze the average AoI performance under the throughput-optimal policy in the considered system. Numerical simulations show the superiority of the derived age-optimal policy over the throughput-optimal policy. We also unveil the impacts of various system parameters on the corresponding optimal policy and the resultant average AoI.

Distributionally Robust Hub Location

We study the adaptive distributionally robust hub location problem with multiple commodities under demand and cost uncertainty in both uncapacitated and capacitated cases. The hub location decision anticipates the worst-case expected cost over an ambiguity set of possible distributions of the uncertain demand and cost, and the routing policy, being adaptive to the uncertainty realization, ships commodities through selected hubs. We investigate the adaptivity and tractability of the distributionally robust model under different distributional information about uncertainty. In the uncapacitated case in which demand and cost are independent and costs of different commodities are also mutually independent, the adaptive distributionally robust model is equivalent to a nonadaptive classical robust model and the second-stage routing decision follows an optimal static policy. We then relax the independence assumption and show that the second-stage routing decision follows an optimal scenario-wise policy if either the demand or the cost is supported on a convex hull of given scenarios. We extend our analysis to the capacitated case and show that the second-stage routing decision still follows an optimal scenario-wise policy if the demand is supported on the convex hull of given scenarios. In terms of tractability, for all mentioned cases, we reformulate the distributionally robust model as a moderate-sized mixed-integer linear program, and we recover the associated worst-case distribution by solving a collection of linear programs. Through numerical studies using the Civil Aeronautics Board data set, we demonstrate the advantages of the distributionally robust model by examining its superior out-of-sample performance against the classical robust model and the stochastic model.

Optimal pricing policies for tandem queues: Asymptotic optimality

We study the optimal pricing problem for a tandem queueing system with an arbitrary number of stations, finite buffers, and blocking. The problem is formulated using a Markov decision process model with the objective to maximize the long-run expected time-average revenue or gain of the service provider. Our interest lies in comparing the performances of static and dynamic pricing policies in maximizing the gain. We show that the optimal static pricing policies perform as well as the optimal dynamic pricing policies when the buffer size at station 1 becomes large and the arrival rate is either small or large. More importantly, we propose two specific static pricing policies for systems with small and large arrival rates, respectively, and show that each proposed policy produces a gain converging to the optimal gain with an approximately exponential rate as the buffer size before station 1 becomes large. We learn from numerical results that the proposed static policies perform as well as optimal dynamic policies even for a moderate-sized buffer at station 1. We also learn that there exist cases where optimal static pricing policies are, however, neither optimal nor near-optimal.

Electric Vehicle Routing with Public Charging Stations

We introduce the electric vehicle routing problem with public-private recharging strategy in which vehicles may recharge en route at public charging infrastructure as well as at a privately-owned depot. To hedge against uncertain demand at public charging stations, we design routing policies that anticipate station queue dynamics. We leverage a decomposition to identify good routing policies, including the optimal static policy and fixed-route-based rollout policies that dynamically respond to observed queues. The decomposition also enables us to establish dual bounds, providing a measure of goodness for our routing policies. In computational experiments using real instances from industry, we show the value of our policies to be within 10% of a dual bound. Furthermore, we demonstrate that our policies significantly outperform the industry-standard routing strategy in which vehicle recharging generally occurs at a central depot. Our methods stand to reduce the operating costs associated with electric vehicles, facilitating the transition from internal-combustion engine vehicles.

On continuous-time constrained stochastic linear–quadratic control

This paper studies a class of continuous-time scalar-state stochastic Linear–Quadratic (LQ) optimal control problems with the linear control constraints. Using the state separation theorem induced from its special structure, we derive the analytical solution for this class of problems. The revealed optimal control policy is a piece-wise affine function of system state. This control policy can be computed efficiently by solving two Riccati equations off-line. Under some mild conditions, the stationary optimal control policy can be also achieved for this class of problems over an infinite horizon. Examples are presented to shed light on the theoretical results established.

Optimal Scheduling of Multiple Sensors Over Lossy and Bandwidth Limited Channels

This work considers the sensor scheduling for multiple dynamic processes. We consider $n$ linear dynamic processes, the state of each process is measured by a sensor, which transmits their local state estimates over wireless channels to a remote estimator with certain communication costs. In each time step, only a portion of the sensors is allowed to transmit data to the remote estimator and the packet might be lost due to unreliability of the wireless channels. Our goal is to find a scheduling policy which coordinates the sensors in a centralized manner to minimize the total expected estimation error of the remote estimator and the communication costs. We formulate the problem as a Markov decision process. We develop an algorithm to check whether there exists a deterministic stationary optimal policy. We show the optimality of monotone policies, which saves computational effort of finding an optimal policy and facilitates practical implementation. Nevertheless, obtaining an exact optimal policy still suffers from curse of dimensionality when the number of processes are large. We further provide an index-based heuristics to avoid brute force computation. Numerical examples are presented to illustrate our theoretical results.

Optimal Control of Production Time of Perishable Inventory System with Postponed Demands

We consider a continuous review production inventory system in which the demands arrive according to a Poisson process. We assume that the demands that occur during stock out periods enter a pool which has finite capacity. Any demand that arrives when the pool is full and the inventory level is zero, is also assumed to be lost. The demands in the pool are selected one by one, if the stock is above the prefixed value, with interval time between any two successive selections is distributed as exponential. In order to replenish stock, a production facility is switched on when the inventory level drops to prefixed value s and items are produced one by one until the stock level reaches a maximum capacity S(> s). The successive production time is assumed to follow an exponential distribution whose parameter is chosen from a given set of positive values. The objective of this paper is to investigate the optimal set of production rates that minimises the total expected cost per unit time. We formulate this problem as a semi-Markov decision problem. The stationary optimal policy is computed using linear programming algorithm and the results are illustrated numerically.

Optimal service rates of a perishable inventory system with service facility

This paper studies a continuous-review perishable inventory system with service facility. We consider a finite capacity inventory system with Markovian arrivals and exponentially distributed service times. The inventory is replenished according to an (s, S) policy and the lead times follow phase type distribution. The life time of each item is assumed to be exponential. By controlling the service rate at each instance of time, we can minimise the total expected cost rate. We formulate this model as a semi Markov decision problem. The stationary optimal policy is computed using linear programming algorithm and the results are illustrated numerically.

Allocation of COVID-19 Vaccines Under Limited Supply

Problem definition: This paper considers how to allocate COVID-19 vaccines to different age groups when limited vaccines are available over time. Academic/practical relevance: Vaccine is one of the most effective interventions to contain the ongoing COVID-19 pandemic. However, the initial supply of the COVID-19 vaccine will be limited. An urgent problem for the government is to determine who to get the first dose of the future COVID-19 vaccine. Methodology: We use epidemic data from New York City to calibrate an age-structured SAPHIRE model that captures the disease dynamics within and across various age groups. The model and data allow us to derive effective static and dynamic vaccine allocation policies minimizing the number of confirmed cases or the numbers of deaths. Results: The optimal static policies achieve a much smaller number of confirmed cases and deaths compared to other static benchmark policies including the pro rata policy. Dynamic allocation policies, including various versions of the myopic policy, significantly improve on static policies. Managerial implications: For static policies, our numerical study shows that prioritizing the older groups is beneficial to reduce deaths while prioritizing younger groups is beneficial to avert infections. For dynamic policies, the older groups should be vaccinated at early days and then switch to younger groups. Our analysis provides insights on how to allocate vaccines to the various age groups, which is tightly connected to the decision-maker’s objective.

Delay-Optimal Cooperation Policy in a Slotted Aloha Full-Duplex Wireless Network: Static Approach

We consider a cooperative wireless communication network comprising two full-duplex (FD) nodes transmitting to a common destination based on slotted Aloha protocol. Each node has exogenous arrivals and also may relay some of the unsuccessfully transmitted packets of the other node. In this article, we find the optimal static policies of nodes in order to minimize the sum of the average transmission delays, while the average transmission delay of each node is constrained. The static policy of each node specifies the probability of accepting an unsuccessfully transmitted packet of the other node and how the node prioritizes its transmissions. We show that in the optimal policies, just the node with the better link to destination may relay the packets of the other node. We derive optimal acceptance probability of the relaying node in closed form, and show that it should transmit the packets of the other node first. Moreover, we prove that the noncooperation policy is optimal if the ratio of successful transmission rates of nodes does not exceed the Golden ratio. We also show that FD capability of nodes is helpful only when full-acceptance is the optimal policy. Efficiency of the attained optimal policy is illustrated by numerical results.

Dynamic Pricing of Wireless Internet Based on Usage and Stochastically Changing Capacity

Problem definition: Inspired by new developments in dynamic spectrum access, we study the dynamic pricing of wireless Internet access when demand and capacity (bandwidth) are stochastic. Academic/practical relevance: The demand for wireless Internet access has increased enormously. However, the spectrum available to wireless service providers is limited. The industry has, thus, altered conventional license-based spectrum access policies through unlicensed spectrum operations. The additional spectrum obtained through these operations has stochastic capacity. Thus, the pricing of this service by the service provider has novel challenges. The problem considered in this paper is, therefore, of high practical relevance and new to the academic literature. Methodology: We study this pricing problem using a Markov decision process model in which customers are posted dynamic prices based on their bandwidth requirement and the available capacity. Results: We characterize the structure of the optimal pricing policy as a function of the system state and of the input parameters. Because it is impossible to solve this problem for practically large state spaces, we propose a heuristic dynamic pricing policy that performs very well, particularly when the ratio of capacity to demand rate is low. Managerial implications: We demonstrate the value of using a dynamic heuristic pricing policy compared with the myopic and optimal static policies. The previous literature has studied similar systems with fixed capacity and has characterized conditions under which myopic policies perform well. In contrast, our setting has dynamic (stochastic) capacity, and we find that identifying good state-dependent heuristic pricing policies is of greater importance. Our heuristic policy is computationally more tractable and easier to implement than the optimal dynamic and static pricing policies. It also provides a significant performance improvement relative to the myopic and optimal static policies when capacity is scarce, a condition that holds for the practical setting that motivated this research.

Risk-sensitive control of branching processes

This article solves the risk-sensitive control problem for branching processes where the one-period progeny of an individual can take values from a finite set. The decision maker is assumed to maximize the expected risk-averse exponential utility (or to minimize the expected risk-averse exponential disutility) of the rewards earned in an infinite horizon. Individuals are assumed to produce progeny independently, and with the same probability mass function if they take the same action. This article characterizes the expected disutility of stationary policies, identifies necessary and sufficient conditions for the existence of a stationary optimal policy that assigns the same action to all individuals in all periods, and discusses computational methods to obtain such a policy. Supplementary materials are available for this article. See the publisher’s online edition of IIE Transactions, datasets, additional tables, detailed proofs, etc.

Optimal Pricing for Tandem Queues with Finite Buffers

We consider optimal pricing for a two-station tandem queueing system with finite buffers and price-sensitive customers. The service provider quotes prices to customers using either static or dynamic pricing. The objective is to maximize either the infinite-horizon discounted profit or the long-run average profit of the provider. We show that there exists an optimal dynamic policy that exhibits an interesting monotone structure, in which the quoted prices have greater dependency on the queue length at station 1 than at station 2, for both the discounted and long-run average problems. We then study the long-run average problem and show that the optimal static policy performs as well as the optimal dynamic policy when the buffer size at station 1 becomes large, there are no holding costs, and the arrival rate is either small or large. We learn from numerical results that for systems with small arrival rates, the optimal static policy is near optimal even when the buffer at station 1 is small and there are positive holding costs. On the other hand, for systems with arrival rates that are not small, there are cases where the optimal dynamic policy performs much better than the optimal static policy.

Optimality of admission control in a repairable queue

We study a repairable M∕M∕1∕N queueing system with multiple types of customers. The difference between customers’ types is defined by the offered rewards. Once customers are admitted to the system, they have the same service rate. We show that under the average reward criterion, all optimal (deterministic) stationary policies have the trunk reservation form. We then prove the uniqueness of the bias optimal policy.

Optimal Service Rates of a perishable Inventory System with Service Facility

This paper studies a continuous-review perishable inventory system with service facility. We consider a finite capacity inventory system with Markovian arrivals and exponentially distributed service times. The inventory is replenished according to an (s, S) policy and the lead times follow phase type distribution. The life time of each item is assumed to be exponential. By controlling the service rate at each instance of time, we can minimise the total expected cost rate. We formulate this model as a semi Markov decision problem. The stationary optimal policy is computed using linear programming algorithm and the results are illustrated numerically.

Hamilton-Jacobi-Bellman inequality for the average control of piecewise deterministic Markov processes

ABSTRACTThe main goal of this paper is to study the infinite-horizon long run average continuous-time optimal control problem of piecewise deterministic Markov processes (PDMPs) with the control acting continuously on the jump intensity λ and on the transition measure Q of the process. We provide conditions for the existence of a solution to an integro-differential optimality inequality, the so called Hamilton-Jacobi-Bellman (HJB) equation, and for the existence of a deterministic stationary optimal policy. These results are obtained by using the so-called vanishing discount approach, under some continuity and compactness assumptions on the parameters of the problem, as well as some non-explosive conditions for the process.