Infinite Horizon Average Cost Research Articles

Convex analytic method revisited: Further optimality results and performance of deterministic policies in average cost stochastic control

The convex analytic method has proved to be a very versatile method for the study of infinite horizon average cost optimal stochastic control problems. In this paper, we revisit the convex analytic method and make three primary contributions: (i) We present an existence result for controlled Markov models that lack weak continuity of the transition kernel but are strongly continuous in the action variable for every fixed state variable. (ii) For average cost stochastic control problems in standard Borel spaces, while existing results establish the optimality of stationary (possibly randomized) policies, few results are available on the optimality of deterministic policies. We review existing results and present further conditions under which an average cost optimal stochastic control problem admits optimal solutions that are deterministic stationary. (iii) We establish conditions under which the performance under stationary deterministic (and also quantized) policies is dense in the set of performance values under randomized stationary policies.

When to Preprocess? Keeping Information Fresh for Computing-Enable Internet of Things

Age of information (AoI), a notion that measures the information freshness, is an essential performance measure for time-critical applications in Internet of Things (IoT). With the surge of computing resources at the IoT devices, it is possible to preprocess the information packets that contain the status update before sending them to the destination so as to alleviate the transmission burden. However, the additional time and energy expenditure induced by computing also make the optimal updating a non-trivial problem. In this paper, we consider a time-critical IoT system, where the IoT device is capable of preprocessing the status update before the transmission. Particularly, we aim to jointly design the preprocessing and transmission so that the weighted sum of the average AoI of the destination and the energy consumption of the IoT device is minimized. Due to the heterogeneity in transmission and computation capacities, the durations of distinct actions of the IoT device are non-uniform. Therefore, we formulate the status updating problem as an infinite horizon average cost semi-Markov decision process (SMDP) and then transform it into a discrete-time Markov decision process. We demonstrate that the optimal policy is of threshold type with respect to the AoI. Equipped with this, a structure-aware relative policy iteration algorithm is proposed to obtain the optimal policy of the SMDP. Our analysis shows that preprocessing is more beneficial in regimes of high AoIs, given it can reduce the time required for updates. We further prove the switching structure of the optimal policy in a special scenario, where the status updates are transmitted over a reliable channel, and derive the optimal threshold. Finally, simulation results demonstrate the efficacy of preprocessing and show that the proposed policy outperforms two baseline policies.

Sleep, Sense or Transmit: Energy-Age Tradeoff for Status Update With Two-Threshold Optimal Policy

Age-of-Information (AoI), or simply age, which measures the data freshness, is essential for real-time Internet-of-Things (IoT) applications. On the other hand, energy saving is urgently required by many energy-constrained IoT devices. This paper studies the energy-age tradeoff for status update from a sensor to a monitor over an error-prone channel. The sensor can sleep, sense and transmit a new update, or retransmit by considering both sensing energy and transmit energy. An infinite-horizon average cost problem is formulated as a Markov decision process (MDP) with the objective of minimizing the weighted sum of average AoI and average energy consumption. By solving the associated discounted cost problem and analyzing the Markov chain under the optimal policy, we prove that there exists a threshold optimal stationary policy with only two thresholds, i.e., one threshold on the AoI at the transmitter (AoIT) and the other on the AoI at the receiver (AoIR). Moreover, the two thresholds can be efficiently found by a line search. Numerical results show the performance of the optimal policies and the tradeoff curves with different parameters. Comparisons with the conventional policies show that considering sensing energy is of significant impact on the policy design, and introducing sleep mode greatly expands the tradeoff range.

Age of Changed Information: Content-Aware Status Updating in the Internet of Things

In Internet of Things (IoT), the freshness of status updates is crucial for mission-critical applications. In this regard, it is suggested to quantify the freshness of updates by using Age of Information (AoI) from the receiver’s perspective. Specifically, the AoI measures the freshness over time. However, the freshness in the content is neglected. In this paper, we introduce an age-based utility, named as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Age of Changed Information</i> (AoCI), which captures both the passage of time and the change of information content. By modeling the underlying physical process as a discrete time Markov chain, we investigate the AoCI in a time-slotted status update system, where a sensor samples the physical process and transmits the update packets to the destination. With the aim of minimizing the weighted sum of the AoCI and the update cost, we formulate an infinite horizon average cost Markov Decision Process. We show that the optimal updating policy has a special structure with respect to the AoCI and identify the condition under which the special structure exists. By exploiting the special structure, we provide a low complexity relative policy iteration algorithm that finds the optimal updating policy. We further investigate the optimal policy for two special cases. In the first case where the state of the physical process transits with equiprobability, we show that optimal policy is of threshold type and derive the closed-form of the optimal threshold. We then study a more generalized periodic Markov model of the physical process in the second case. Lastly, simulation results are laid out to exhibit the performance of the optimal updating policy and its superiority over the zero-wait baseline policy.

Online Optimal State Feedback Control of Linear Systems Over Wireless MIMO Fading Channels

We consider the optimal control of linear systems over wireless MIMO fading channels, where the MIMO wireless fading and random access of the remote controller may cause intermittent controllability or uncontrollability of the closed-loop control system. We formulate the optimal control design over random access MIMO fading channels as an infinite horizon average cost Markov decision process (MDP), and we propose a novel state reduction technique such that the optimality condition is transformed into a time-invariant reduced-state Bellman optimality equation. We provide the closed-form characterizations on the existence and uniqueness of the optimal control solution via analyzing the reduced-state Bellman optimality equation. Specifically, in the case that the closed-loop system is almost surely controllable, we show that the optimal control solution always exists and is unique. In the case that MIMO fading channels and the random access of the remote controller destroy the closed-loop controllability, we propose a novel controllable and uncontrollable positive semidefinite (PSD) cone decomposition induced by the singular value decomposition (SVD) of the MIMO fading channel contaminated control input matrix. Based on the decomposed fine-grained reduced-state Bellman optimality equation, we further propose a closed-form sufficient condition for both the existence and the uniqueness of the optimal control solution. The closed-form sufficient condition reveals the fact that the optimal control action may still exist even if the closed-loop system suffers from intermittent controllability or almost sure uncontrollability.

Adaptive Algorithms for Multi-Warehouse Multi-Store Inventory System with Lost Sales and Fixed Replenishment Cost

We consider a two-echelon inventory control problem for a Multi-Warehouse Multi-Store (MWMS) system with lost sales and fixed ordering cost at each store. We focus on a time horizon with no external replenishment, or the problem of “what to do until your (external) shipment comes in. The warehouses are stocked with inventories at the beginning of the horizon, and the stores dynamically replenish from the warehouses in each period. The delivery lead time is zero from a warehouse to any of the store it serves (e.g., each store is within 24 hrs driving distance from its servicing warehouses). This is a prevalent problem in many retail settings. The optimal policy for this problem is complex and state-dependent, and due to the well-known curse of dimensionality, computing the optimal policy using dynamic program is numerically intractable. In this paper, we develop provably near-optimal algorithms by integrating and extending three key ideas/approaches: (i) Lagrangian relaxation of the original complex dynamic inventory control problem, (ii) approximating the finite-horizon cost of an inventory problem with its infinite-horizon average cost, and (iii) dynamic re-adjustments of both replenishment and allocation policies. We first present a Lagrangian-based algorithm that is open-loop and non-adaptive, and discuss its performance in terms of loss rate (optimality gap) between its expected total cost and that of a true optimal policy. Then, we develop several easy-to-compute adaptive algorithms that dynamically re-adjust the policy parameters after observing sales at each store. We highlight the importance of proper policy adjustment to realize the benefit of pooling effect, which typically exists in the setting where one warehouse replenishes multiple stores. Our results show that adaptive readjustments significantly improve the performance of the base algorithm both theoretically and numerically.

Policy Iteration Approach to the Infinite Horizon Average Optimal Control of Probabilistic Boolean Networks

This article studies the optimal control of probabilistic Boolean control networks (PBCNs) with the infinite horizon average cost criterion. By resorting to the semitensor product (STP) of matrices, a nested optimality equation for the optimal control problem of PBCNs is proposed. The Laurent series expression technique and the Jordan decomposition method derive a novel policy iteration-type algorithm, where finite iteration steps can provide the optimal state feedback law, which is presented. Finally, the intervention problem of the probabilistic Ara operon in E. coil, as a biological application, is solved to demonstrate the effectiveness and feasibility of the proposed theoretical approach and algorithms.

Minimum Age of Information in the Internet of Things With Non-Uniform Status Packet Sizes

In this paper, a real-time Internet of Things (IoT) monitoring system is considered in which the IoT devices are scheduled to sample associated underlying physical processes and send the status updates to a common destination. In a real-world IoT, due to the possibly different dynamics of each physical process, the sizes of the status updates for different devices are often different and each status update typically requires multiple transmission slots. By taking into account such multi-time slot transmissions with non-uniform sizes of the status updates under noisy channels, the problem of joint device scheduling and status sampling is studied in order to minimize the average age of information (AoI) at the destination. This stochastic problem is formulated as an infinite horizon average cost Markov decision process (MDP). The monotonicity of the value function of the MDP is characterized and then used to show that the optimal scheduling and sampling policy is threshold-based with respect to the AoI at each device. To overcome the curse of dimensionality, a low-complexity suboptimal policy is proposed through a semi-randomized base policy and linear approximated value functions. The proposed suboptimal policy is shown to exhibit a similar structure to the optimal policy, which provides a structural base for its effective performance. A structure-aware algorithm is then developed to obtain the suboptimal policy. The analytical results are further extended to the IoT monitoring system with random status update arrivals, for which, the optimal scheduling and sampling policy is also shown to be threshold-based with the AoI at each device. Simulation results illustrate the structures of the optimal policy and show a near-optimal AoI performance resulting from the proposed suboptimal solution approach.

Canonical Dynamic Programming Equations subject to Ambiguity

This paper studies the infinite horizon average cost Markov control model subject to ambiguity on the controlled process conditional distribution. The stochastic control problem is formulated as a minimax optimization in which, (i) the existence of optimal policies is established through a pair of canonical dynamic programming equations derived for Borel state and action spaces, and (ii) the controlled process maximizing conditional distribution is characterized through a water-filling solution derived for finite state and action spaces. To obtain average cost optimal policies numerically a policy iteration algorithm is also developed. Finally, as an application of the proposed canonical dynamic programming equations, an example is provided.

Joint Status Sampling and Updating for Minimizing Age of Information in the Internet of Things

The effective operation of time-critical Internet of things (IoT) applications requires real-time reporting of fresh status information of underlying physical processes. In this paper, a real-time IoT monitoring system is considered, in which the IoT devices sample a physical process with a sampling cost and send the status packet to a given destination with an updating cost. This joint status sampling and updating process is designed to minimize the average age of information (AoI) at the destination node under an average energy cost constraint at each device. This is formulated as an infinite horizon average cost constrained Markov decision process (CMDP) and transformed into an unconstrained MDP using a Lagrangian method. For the single IoT device case, the optimal policy for the CMDP is shown to be a randomized mixture of two deterministic policies for the unconstrained MDP, which is of threshold type. Then, a structure-aware optimal algorithm to obtain the optimal policy of the CMDP is proposed and the impact of the wireless channel dynamics is studied while demonstrating that channels having a larger mean channel gain and less scattering can achieve better AoI performance. For the case of multiple IoT devices, a low-complexity distributed suboptimal policy is proposed with the updating control at the destination and the sampling control at each device. Then, an online learning algorithm is developed to obtain this policy, which can be implemented at each IoT device and requires only the local knowledge and small signaling from the destination. The proposed learning algorithm is shown to converge almost surely to the suboptimal policy. Simulation results show the structural properties of the optimal policy for the single IoT device case; and show that the proposed policy for multiple IoT devices outperforms a zero-wait baseline policy, with average AoI reductions reaching up to 33%.

Closed-Form Delay-Optimal Computation Offloading in Mobile Edge Computing Systems

Mobile edge computing (MEC) has recently emerged as a promising technology to release the tension between computation-intensive applications and resource-limited mobile terminals (MTs). In this paper, we study the delay-optimal computation offloading in computation-constrained MEC systems. We consider the computation task queue at the MEC server due to its constrained computation capability. In this case, the task queue at the MT and that at the MEC server are strongly coupled in a cascade manner, which creates complex interdependences and brings new technical challenges. We model the computation offloading problem as an infinite horizon average cost Markov decision process (MDP) and approximate it to a virtual continuous time system (VCTS) with reflections. Different from most of the existing works, we develop the dynamic instantaneous rate estimation for deriving the closed-form approximate priority functions in different scenarios. Based on the approximate priority functions, we propose a closed-form multi-level water-filling computation offloading solution to characterize the influence of not only the local queue state information (LQSI) but also the remote queue state information (RQSI). Furthermore, we discuss the extension of our proposed scheme to multi-MT multi-server scenarios. Finally, the simulation results show that the proposed scheme outperforms the conventional schemes.

Infinite Horizon Average Cost Dynamic Programming Subject to Total Variation Distance Ambiguity

We analyze the per unit-time infinite horizon average cost Markov control model, subject to a total variation distance ambiguity on the controlled process conditional distribution. This stochastic optimal control problem is formulated as a minimax optimization problem in which the minimization is over the admissible set of control strategies, while the maximization is over the set of conditional distributions which are in a ball, with respect to the total variation distance, centered at a nominal distribution. We derive two new equivalent dynamic programming equations, and a new policy iteration algorithm. The main feature of the new dynamic programming equations is that the optimal control strategies are insensitive to inaccuracies or ambiguities in the controlled process conditional distribution. The main feature of the new policy iteration algorithm is that the policy evaluation and policy improvement steps are performed using the maximizing conditional distribution, which is obtained via a water filling solution of aggregating states together to form new states. Throughout the paper, we illustrate the new dynamic programming equations and the corresponding policy iteration algorithm to various examples.

Joint Pushing and Caching for Bandwidth Utilization Maximization in Wireless Networks

Joint pushing and caching is recognized as an efficient remedy to the problem of spectrum scarcity incurred by tremendous mobile data traffic. In this paper, we design the optimal joint pushing and caching policy to maximize bandwidth utilization, which is of fundamental importance to mobile telecom carriers. In particular, we consider a multiuser wireless network with multicast opportunities where each user is equipped with a cache of limited size. First, we formulate the stochastic optimization problem as an infinite horizon average cost Markov decision process. By the structural analysis, we show that how the optimal policy achieves a balance between the current transmission cost and the future average transmission cost. We also show that the optimal average transmission cost decreases with the cache sizes, revealing a tradeoff between storage and bandwidth. Then, due to the fact that obtaining a numerical optimal solution suffers the curse of dimensionality and implementing it requires a centralized controller and global system information, we develop a low-complexity decentralized policy (LDP) by using a linear approximation of the value function and transforming challenging discrete optimization problems into difference of convex (DC) problems, which can be efficiently solved by using DC algorithms. We also obtain an upper bound on the performance gap between the average cost of LDP and the minimum average cost, which can be easily evaluated. Next, we propose an online decentralized algorithm to implement the proposed LDP, when priori knowledge of user demand processes is not available. Finally, using numerical results, we demonstrate the advantage of the proposed solutions over some existing designs. The results in this paper offer useful guidelines for designing practical cache-enabled multiuser wireless networks.

Delay-Aware Uplink Fronthaul Allocation in Cloud Radio Access Networks

In cloud radio access networks (C-RANs), the baseband units and radio units of base stations are separated, which requires high-capacity fronthaul links connecting both parts. In this paper, we consider the delay-aware fronthaul allocation problem for C-RANs. The stochastic optimization problem is formulated as an infinite horizon average cost Markov decision process. To deal with the curse of dimensionality, we derive a closed-form approximate priority function and the associated error bound using perturbation analysis. Based on the closed-form approximate priority function, we propose a low-complexity delay-aware fronthaul allocation algorithm solving the per-stage optimization problem. The proposed solution is further shown to be asymptotically optimal for sufficiently small cross link path gains. Finally, the proposed fronthaul allocation algorithm is compared with various baselines through simulations, and it is shown that significant performance gain can be achieved.

Optimal Dynamic Multicast Scheduling for Cache-Enabled Content-Centric Wireless Networks

Caching and multicasting at base stations are two promising approaches to support massive content delivery over wireless networks. However, existing scheduling designs do not make full use of the advantages of the two approaches. In this paper, we consider the optimal dynamic multicast scheduling to jointly minimize the average delay, power, and fetching costs for cache-enabled content-centric wireless networks. We formulate this stochastic optimization problem as an infinite horizon average cost Markov decision process (MDP). It is well-known to be a difficult problem due to the curse of dimensionality, and there generally only exist numerical solutions. By using relative value iteration algorithm and the special structures of the request queue dynamics, we analyze the properties of the value function and the state-action cost function of the MDP for both the uniform and nonuniform channel cases. Based on these properties, we show that the optimal policy, which is adaptive to the request queue state, has a switch structure in the uniform case and a partial switch structure in the nonuniform case. Moreover, in the uniform case with two contents, we show that the switch curve is monotonically non-decreasing. Then, by exploiting these structural properties of the optimal policy, we propose two low-complexity optimal algorithms. Motivated by the switch structures of the optimal policy, to further reduce the complexity, we also propose a low-complexity suboptimal policy, which possesses similar structural properties to the optimal policy, and develop a low-complexity algorithm to compute this policy.

Managing Patient Admissions in a Neurology Ward

We study patient admission policies in a neurology ward where there are multiple types of patients with different medical characteristics. Patients receive specialized care inside the neurology ward and delays in admission to the ward will have negative impact on their health status. The level of this impact varies among patient types and depends on the severity of patients. Patients are also different in terms of arrival rate and length of stay at the ward. The patients normally wait in the emergency department until a ward bed is assigned to them. We formulate this problem as an infinite-horizon average cost dynamic program and propose an efficient approximation scheme to solve large-scale problem instances. The computational results from applying our model to a neurology ward show that dynamic policies generated by our approach can reduce the overall deterioration in patients’ health status compared to several alternative policies. The online appendix is available at https://doi.org/10.1287/opre.2016.1574 .

Inpatient Overflow: An Approximate Dynamic Programming Approach

Problem definition: Inpatient beds are usually grouped into several wards, and each ward is assigned to serve patients from certain “primary” specialties. However, when a patient waits excessively long before a primary bed becomes available, hospital managers have the option to assign her to a nonprimary bed. although it is undesirable. Deciding when to use such “overflow” is difficult in real time and under uncertainty. Relevance: To aid the decision making, we model hospital inpatient flow as a multiclass, multipool parallel-server queueing system and formulate the overflow decision problem as a discrete-time, infinite-horizon average cost Markov decision process (MDP). The MDP incorporates many realistic and important features such as patient arrival and discharge patterns depending on time of day. Methodology: To overcome the curse-of-dimensionality of this formulated MDP, we resort to approximate dynamic programming (ADP). A critical part in designing an ADP algorithm is to choose appropriate basis functions to approximate the relative value function. Using a novel combination of fluid control and single-pool approximation, we develop analytical forms to approximate the relative value functions at midnight, which then guides the choice of the basis functions for all other times of day. Results: We demonstrate, via numerical experiments in realistic hospital settings, that our proposed ADP algorithm is remarkably effective in finding good overflow policies. These ADP policies can significantly improve system performance over some commonly used overflow strategies—for example, in a baseline scenario, the ADP policy achieves a congestion level similar to that achieved by a complete bed sharing policy, while reduces the overflow proportion by 20%. Managerial implications: We quantify the trade-off between the overflow proportion and congestion from implementing ADP policies under a variety of system conditions and generate useful insights. The plotted efficient frontiers allow managers to observe various performance measures in different parameter regimes, and the ADP policies provide managers with operational strategies to achieve the desired performance.

Inpatient Bed Overflow: An Approximate Dynamic Programming Approach

Problem definition: Inpatient beds are usually grouped into several wards, and each ward is assigned to serve patients from certain specialties. However, when a patient waits excessively long before a primary bed becomes available, hospital managers have the option to assign her to a non-primary bed though it is undesirable. Deciding when to use such is difficult in real time and under uncertainty. Relevance: To aid the decision making, we model hospital inpatient flow as a multi-class, multi-pool parallel-server queueing system and formulate the overflow decision problem as a discrete-time, infinite-horizon average cost Markov decision process (MDP). The MDP incorporates many realistic and important features such as patient arrival and discharge patterns depending on time of day. Methodology: To overcome the curse-of-dimensionality of this formulated MDP, we resort to approximate dynamic programming (ADP). A critical part in designing an ADP algorithm is to choose appropriate basis functions to approximate the relative value function. Using a novel combination of fluid control and single-pool approximation, we develop analytical forms to approximate the relative value functions at midnight, which then guides the choice of the basis functions for all other times of day. Results: We demonstrate, via numerical experiments in realistic hospital settings, that our proposed ADP algorithm is remarkably effective in finding good overflow policies. These ADP policies can significantly improve system performance over some commonly used overflow strategies, e.g., in a baseline scenario, the ADP policy achieves a congestion level similar to that achieved by a complete bed sharing policy, while reduces the overflow proportion by 20%. Managerial Implications: We quantify the trade-off between the overflow proportion and congestion from implementing ADP policies under a variety of system conditions and generate useful insights. The plotted efficient frontiers allow managers to observe various performance measures in different parameter regimes, and the ADP policies provide managers with operational strategies to achieve the desired performance.

Stochastic Content-Centric Multicast Scheduling for Cache-Enabled Heterogeneous Cellular Networks

Caching at small base stations (SBSs) has demonstrated significant benefits in alleviating the backhaul requirement in heterogeneous cellular networks (HetNets). While many existing works focus on what contents to cache at each SBS, an equally important problem is what contents to deliver so as to satisfy dynamic user demands given the cache status. In this paper, we study optimal content delivery in cache-enabled HetNets by taking into account the inherent multicast capability of wireless medium. We consider stochastic content multicast scheduling to jointly minimize the average network delay and power costs under a multiple access constraint. We establish a content-centric request queue model and formulate this stochastic optimization problem as an infinite horizon average cost Markov decision process (MDP). By using \emph{relative value iteration} and special properties of the request queue dynamics, we characterize some properties of the value function of the MDP. Based on these properties, we show that the optimal multicast scheduling policy is of threshold type. Then, we propose a structure-aware optimal algorithm to obtain the optimal policy. We also propose a low-complexity suboptimal policy, which possesses similar structural properties to the optimal policy, and develop a low-complexity algorithm to obtain this policy.

Stochastic Throughput Optimization for Two-Hop Systems With Finite Relay Buffers

Optimal queueing control of multi-hop networks remains a challenging problem even in the simplest scenarios. In this paper, we consider a two-hop half-duplex relaying system with random channel connectivity. The relay is equipped with a finite buffer. We focus on stochastic link selection and transmission rate control to maximize the average system throughput subject to a half-duplex constraint. We formulate this stochastic optimization problem as an infinite horizon average cost Markov decision process (MDP), which is well-known to be a difficult problem. By using sample-path analysis and exploiting the specific problem structure, we first obtain an \emph{equivalent Bellman equation} with reduced state and action spaces. By using \emph{relative value iteration algorithm}, we analyze the properties of the value function of the MDP. Then, we show that the optimal policy has a threshold-based structure by characterizing the \emph{supermodularity} in the optimal control. Based on the threshold-based structure and Markov chain theory, we further simplify the original complex stochastic optimization problem to a static optimization problem over a small discrete feasible set and propose a low-complexity algorithm to solve the simplified static optimization problem by making use of its special structure. Furthermore, we obtain the closed-form optimal threshold for the symmetric case. The analytical results obtained in this paper also provide design insights for two-hop relaying systems with multiple relays equipped with finite relay buffers.