Value Iteration Algorithm Research Articles

Strong convergence and dynamic economic models

Morton and Wecker (1977) stated that the value iteration algorithm solves a dynamic program's policy function faster than its value function when the limiting Markov chain is ergodic. I show that their proof is incomplete, and provide a new proof of this classic result. I use this result to accelerate the estimation of Markov decision processes and the solution of Markov perfect equilibria. Markov decision process Markov perfect equilibrium strong convergence relative value iteration dynamic discrete choice nested fixed point nested pseudo‐likelihood C01 C13 C15 C61 C63 C65

A novel use of value iteration for deriving bounds for threshold and switching curve optimal policies

AbstractIn this article, we develop a novel role for the initial function v0 in the value iteration algorithm. In case the optimal policy of a countable state Markovian queueing control problem has a threshold or switching curve structure, we conjecture, that one can tune the choice of v0 to generate monotonic sequences of n‐stage threshold or switching curve optimal policies. We will show this for three queueing control models, the M/M/1 queue with admission and with service control, and the two‐competing queues model with quadratic holding cost. As a consequence, we obtain increasingly tighter upper and lower bounds. After a finite number of iterations, either the optimal threshold, or the optimal switching curve values in a finite number of states is available. This procedure can be used to increase numerical efficiency.

Proper Policies in Infinite-State Stochastic Shortest Path Problems

We consider stochastic shortest path problems with infinite state and control spaces, a nonnegative cost per stage, and a termination state. We extend the notion of a proper policy, a policy that terminates within a finite expected number of steps, from the context of finite state space to the context of infinite state space. We consider the optimal cost function $J^*$ , and the optimal cost function $\hat{J}$ over just the proper policies. We show that $J^*$ and $\hat{J}$ are the smallest and largest solutions of Bellman's equation, respectively, within a suitable class of Lyapounov-like functions. If the cost per stage is bounded, these functions are those that are bounded over the effective domain of $\hat{J}$ . The standard value iteration algorithm may be attracted to either $J^*$ or $\hat{J}$ , depending on the initial condition.

Lexicographic refinements in stationary possibilistic Markov Decision Processes

Possibilistic Markov Decision Processes offer a compact and tractable way to represent and solve problems of sequential decision under qualitative uncertainty. Even though appealing for its ability to handle qualitative problems, this model suffers from the drowning effect that is inherent to possibilistic decision theory. The present1 paper proposes to escape the drowning effect by extending to stationary possibilistic MDPs the lexicographic preference relations defined by Fargier and Sabbadin [13] for non-sequential decision problems. We propose a value iteration algorithm and a policy iteration algorithm to compute policies that are optimal for these new criteria. The practical feasibility of these algorithms is then experimented on different samples of possibilistic MDPs.

Check in or Not? A Stochastic Game for Privacy Preserving in Point-of-Interest Recommendation System

With the growing popularity of mobile social networks, point-of-interest (POI) recommendation, which utilizes users’ check-in data to suggest interesting places for users, has attracted much attention in recent years. The check-in data, containing time and location information, are closely related to the user’s personal life. Due to privacy concerns, users are reluctant to share check-in data with the service provider (SP), which causes a negative effect on recommendations. It is important for the user to find a balance between privacy and recommendation quality. In this paper, we consider a POI recommendation scenario where an adversary can access the data that a user reports to the SP. The user sequentially decides whether to check in for the POI he has visited. A stochastic game model is proposed to analyze the interaction between the user and the adversary. To find a good policy for the user, two value iteration algorithms are applied. The proposed game has a large state set, which makes it difficult for policy learning. To deal with this problem, we use some tricks when implementing the minimax Q-learning algorithm, and a set of neural networks are trained to approximate the Q-functions. To evaluate the performance of the learning algorithms, we conduct a series of simulations by using real-world check-in data. Simulation results show that the proposed learning algorithms can help the user to make good decisions, in the sense that the user can get a high long-term return.

Finite horizon continuous-time Markov decision processes with mean and variance criteria

This paper studies mean maximization and variance minimization problems in finite horizon continuous-time Markov decision processes. The state and action spaces are assumed to be Borel spaces, while reward functions and transition rates are allowed to be unbounded. For the mean problem, we design a method called successive approximation, which enables us to prove the existence of a solution to the Hamilton-Jacobi-Bellman (HJB) equation, and then the existence of a mean-optimal policy under some growth and compact-continuity conditions. For the variance problem, using the first-jump analysis, we succeed in converting the second moment of the finite horizon reward to a mean of a finite horizon reward with new reward functions under suitable conditions, based on which the associated HJB equation for the variance problem and the existence of variance-optimal policies are established. Value iteration algorithms for computing mean- and variance-optimal policies are proposed.

Optimizing Daily Service Scheduling for Medical Diagnostic Equipment Considering Patient Satisfaction and Hospital Revenue

Under the background of the unbalanced supply and demand of medical diagnostic equipment and rising health care costs, this study aims to optimize the service scheduling for medical diagnostic equipment so as to improve patient satisfaction by ensuring the equipment utilization rate and hospital revenue. The finite horizon Markov Decision Process (MDP) was adopted to solve this problem. On the basis of field research, we divided patients into four categories: emergency patients, inpatients, appointed outpatients, and the randomly arrived outpatients according to the severity of illness and appointment situations. In the construction of the MDP model, we considered the possibility of cancellation (no-show patients) in scheduling optimization. Combined with the benefits and costs related to patient satisfaction, based on the value iteration algorithm, we took patient satisfaction and hospital revenue as the objective functions. Results indicated that, compared with the current scheduling strategy, the integrated strategy proposed in this study has a better performance, which could maintain the sustainable usage rate of large medical resources and patient satisfaction.

Energy-efficient proactive edge caching with sleep scheduling for green networks

Proactive content caching in small cells (SCs) (i.e., proactive edge caching) can significantly reduce energy consumption of networks. However, the sleep mode of SCs can make the cached contents in SCs unavailable. Therefore, a joint optimization of edge caching strategy and sleep scheduling should be conducted to maximize the effectiveness of edge caching. In this paper, we propose an energy-efficient proactive edge caching with sleep scheduling (E3CS) where the controller jointly adjusts the caching strategy and the sleep scheduling of SCs by considering the content popularity dynamics. To optimize the performance of E3CS (i.e., minimize the overall energy consumption), we formulate a Markov decision process (MDP) and the joint optimal policy on the caching strategy and sleep scheduling is obtained by a value iteration algorithm. Evaluation results demonstrate that E3CS with the optimal policy outperforms the comparison schemes in terms of the overall energy consumption.

The Benefits of State Aggregation with Extreme-Point Weighting for Assemble-to-Order Systems

We provide a new method for solving a very general model of an assemble-to-order system: multiple products, multiple components that may be demanded in different quantities by different products, batch production, random lead times, and lost sales, modeled as a Markov decision process under the discounted cost criterion. A control policy specifies when a batch of components should be produced and whether an arriving demand for each product should be satisfied. As optimal solutions for our model are computationally intractable for even moderately sized systems, we approximate the optimal cost function by reformulating it on an aggregate state space and restricting each aggregate state to be represented by its extreme original states. Our aggregation drastically reduces the value iteration computational burden. We derive an upper bound on the distance between aggregate and optimal solutions. This guarantees that the value iteration algorithm for the original problem initialized with the aggregate solution converges to the optimal solution. We also establish the optimality of a lattice-dependent base-stock and rationing policy in the aggregate problem when certain product and component characteristics are incorporated into the aggregation/disaggregation schemes. This enables us to further alleviate the value iteration computational burden in the aggregate problem by eliminating suboptimal actions. Leveraging all of our results, we can solve the aggregate problem for systems of up to 22 components, with an average distance of 11.09% from the optimal cost in systems of up to 4 components (for which we could solve the original problem to optimality). The e-companion is available at https://doi.org/10.1287/opre.2017.1710 .

Mobility-Aware Vehicle-to-Grid Control Algorithm in Microgrids

In a vehicle-to-grid (V2G) system, electric vehicles (EVs) can be efficiently used as power consumers and suppliers to achieve microgrid (MG) autonomy. Since EVs can act as energy transporters among different regions (i.e., MGs), it is an important issue to decide where and when EVs are charged or discharged to achieve the optimal performance in a V2G system. In this paper, we propose a mobility-aware V2G control algorithm (MACA) that considers the mobility of EVs, states of charge of EVs, and the estimated/actual demands of MGs and then determines charging and discharging schedules for EVs. To optimize the performance of MACA, the Markov decision process problem is formulated and the optimal policy on charging and discharging is obtained by a value iteration algorithm. Since the mobility of EVs and the estimated/actual demand profiles of MGs may not be easily obtained, a reinforcement learning approach is also introduced. Evaluation results demonstrate that MACA with the optimal and learning-based policies can effectively achieve MG autonomy and provide higher satisfaction on the charging.

Linear quadratic tracking control of unknown discrete-time systems using value iteration algorithm

Abstract In this paper, an optimal tracking control scheme is proposed to solve the infinite-horizon linear quadratic tracking (LQT) problem using iterative adaptive dynamic programming (ADP) algorithm. The reference trajectory is assumed to be produced by a linear command generator. First, via system transformation, an augmented system composed of controlled system and command generator is constructed. Then we derive the Bellman equation in terms of the transformed system with discount factor in cost function. In order to avoid requirement for knowledge of system dynamics, the iterative ADP algorithm is introduced to solve the Bellman equation with convergence analysis. A novel approach based on controllability and observability analysis is presented to show the stability of tracking error. For facilitating the implementation of this iterative approach, three neural networks (NNs) are employed as parametric structures to identify the unknown system dynamics, approximate performance function and search control policy, respectively. Finally, a simulation example is included to verify the effectiveness of the proposed scheme.

SMDP-Based Coordinated Virtual Machine Allocations in Cloud-Fog Computing Systems

Heterogeneous computing powered by remote clouds and local fogs is a promising technology to improve the performance of user terminals in the Internet of Things. In this paper, two semi-Markov decision process (SMDP)-based coordinated virtual machine (VM) allocation methods are proposed to balance the tradeoff between the high cost of providing services by the remote cloud and the limited computing capacity of the local fog. We first present a model-based planning method in which it is necessary to train the state transition probabilities and the expected time intervals between adjacent decision epochs. To facilitate training them, the SMDP is degraded into a continuous-time Markov decision process (CTMDP) in which the service requests and ongoing service completions follow a continuous-time Markov chain. The relative value iterative algorithm for the CTMDP is used to find an asymptotically optimal VM allocation policy. In addition, we also propose a model-free reinforcement learning (RL) method, where an optimal coordinated VM allocation policy is approximated by learning from the states and rewards of feedback. The simulation results show that the performance of the model-free RL method can converge to a level similar to that of the model-based planning method and outperform the greedy VM allocation method.

POMDP and MOMDP solutions for structural life-cycle cost minimization under partial and mixed observability

Scheduling of inspection and maintenance policies during the life-cycle of operating infrastructure necessitates optimization of long-term objectives in stochastic environments. Modern answers to the problem should focus on quantitative decision-making techniques, taking advantage of informative but uncertain data that become available in time. As such, the problem is efficiently addressed within the framework of stochastic dynamic programming by means of Partially Observable Markov Decision Processes (POMDPs) and Mixed Observability Markov Decision Processes (MOMDPs). Although these methodologies can provide very sophisticated solutions with optimality guarantees, important computational challenges often emerge, mainly due to the continuity of the multidimensional belief space on the probability simplex. In response, recent value iteration algorithms based on point-based approaches have been suggested, focusing on reachable belief points that can support an accurate value function. In this work, several POMDP and MOMDP point-based algorithms, with various characteristics regarding the exploration of the belief space and the value function update procedures, are rigorously analyzed. The algorithms are compared and evaluated in terms of accuracy and performance in stationary and nonstationary problems of structural inspection and maintenance for life-cycle cost minimization. Results are thoroughly discussed and several insights along with practical suggestions for similar problems are provided.

Game-Theoretic Cross Social Media Analytic: How Yelp Ratings Affect Deal Selection on Groupon?

Deal selection on Groupon is a typical social learning and decision making process, where the quality of a deal is usually unknown to the customers. The customers must acquire this knowledge through social learning from other social medias such as reviews on Yelp. Additionally, the quality of a deal depends on both the state of the vendor and decisions of other customers on Groupon. How social learning and network externality affect the decisions of customers in deal selection on Groupon is our main interest. We develop a data-driven game-theoretic framework to understand the rational deal selection behaviors cross social medias. The sufficient condition of the Nash equilibrium is identified. A value-iteration algorithm is proposed to find the optimal deal selection strategy. We conduct a year-long experiment to trace the competitions among deals on Groupon and the corresponding Yelp ratings. We utilize the dataset to analyze the deal selection game with realistic settings. Finally, the performance of the proposed social learning framework is evaluated with real data. The results suggest that customers do make decisions in a rational way instead of following naive strategies, and there is still room to improve their decisions with assistance from the proposed framework.

Generalized Value Iteration Networks:Life Beyond Lattices

In this paper, we introduce a generalized value iteration network (GVIN), which is an end-to-end neural network planning module. GVIN emulates the value iteration algorithm by using a novel graph convolution operator, which enables GVIN to learn and plan on irregular spatial graphs. We propose three novel differentiable kernels as graph convolution operators and show that the embedding-based kernel achieves the best performance. Furthermore, we present episodic Q-learning, an improvement upon traditional n-step Q-learning that stabilizes training for VIN and GVIN. Lastly, we evaluate GVIN on planning problems in 2D mazes, irregular graphs, and real-world street networks, showing that GVIN generalizes well for both arbitrary graphs and unseen graphs of larger scaleand outperforms a naive generalization of VIN (discretizing a spatial graph into a 2D image).

Joint control of dynamic maintenance and production in a failure-prone manufacturing system subjected to deterioration

Abstract This paper addresses the integrated control of dynamic maintenance and production in a deteriorating manufacturing system. The previous literature on the integrated control of production and maintenance usually ignores maintenance opportunities that may arise during machines’ failures. Therefore, this paper proposes a dynamic maintenance policy which comprises of corrective, preventive and opportunistic maintenance. Opportunistic maintenance uses the downtime of machines as potential opportunities to perform maintenance on other machines. In this paper, control of dynamic maintenance is integrated with production to achieve the minimal total production cost, which includes costs of inventory, backlog, repair, preventive and opportunistic maintenance. By applying an approximation technique and the value iteration algorithm, the authors obtain a near optimal control policy for the deteriorating manufacturing system. In addition, the authors compare the performance of the obtained control policy with that of a previous control policy reported in the literature. The sensitivity analysis is also conducted to examine the effect of some system parameters on the configuration of the control policy. With the proposed control policy, industry practitioners are able to avoid unnecessary preventive maintenance and fully utilize the opportunistic time window to improve system reliability while achieving minimal total production cost.

Parallel Hierarchical Pre-Gauss-Seidel Value Iteration Algorithm

The standard Value Iteration (VI) algorithm, referred to as Value Iteration Pre-Jacobi (PJ-VI) algorithm, is the simplest Value Iteration scheme, and the well-known algorithm for solving Markov Decision Processes (MDPs). In the literature, several versions of VI algorithm were developed in order to reduce the number of iterations: the VI Jacobi (VI-J) algorithm, the Value Iteration Pre-Gauss-Seidel (VI-PGS) algorithm and the VI Gauss-Seidel (VI-GS) algorithm. In this article, the authors combine the advantages of VI Pre Gauss-Seidel algorithm, the decomposition technique and the parallelism in order to propose a new Parallel Hierarchical VI Pre-Gauss-Seidel algorithm. Experimental results show that their approach performs better than the traditional VI schemes in the case where the global problem can be decomposed into smaller problems.

A hybrid stochastic game for secure control of cyber-physical systems

In this paper, we establish a zero-sum, hybrid state stochastic game model for designing defense policies for cyber-physical systems against different types of attacks. With the increasingly integrated properties of cyber-physical systems (CPS) today, security is a challenge for critical infrastructures. Though resilient control and detecting techniques for a specific model of attack have been proposed, to analyze and design detection and defense mechanisms against multiple types of attacks for CPSs requires new system frameworks. Besides security, other requirements such as optimal control cost also need to be considered. The hybrid game model we propose contains physical states that are described by the system dynamics, and a cyber state that represents the detection mode of the system composed by a set of subsystems. A strategy means selecting a subsystem by combining one controller, one estimator and one detector among a finite set of candidate components at each state. Based on the game model, we propose a suboptimal value iteration algorithm for a finite horizon game, and prove that the algorithm results an upper bound for the value of the finite horizon game. A moving-horizon approach is also developed in order to provide a scalable and real-time computation of the switching strategies. Both algorithms aim at obtaining a saddle-point equilibrium policy for balancing the system’s security overhead and control cost. The paper illustrates these concepts using numerical examples, and we compare the results with previously system designs that only equipped with one type of controller.

Data-driven optimization of railway maintenance for track geometry

Railway big data technologies are transforming the existing track inspection and maintenance policy deployed for railroads in North America. This paper develops a data-driven condition-based policy for the inspection and maintenance of track geometry. Both preventive maintenance and spot corrective maintenance are taken into account in the investigation of a 33-month inspection dataset that contains a variety of geometry measurements for every foot of track. First, this study separates the data based on the time interval of the inspection run, calculates the aggregate track quality index (TQI) for each track section, and predicts the track spot geo-defect occurrence probability using random forests. Then, a Markov chain is built to model aggregated track deterioration, and the spot geo-defects are modeled by a Bernoulli process. Finally, a Markov decision process (MDP) is developed for track maintenance decision making, and it is optimized by using a value iteration algorithm. Compared with the existing maintenance policy using Markov chain Monte Carlo (MCMC) simulation, the maintenance policy developed in this paper results in an approximately 10% savings in the total maintenance costs for every 1 mile of track.

Optimal Inspection and Replacement Policies for Multi-Unit Systems Subject to Degradation

Condition-based maintenance (CBM) is proved to be effective in reducing the long-run operational cost for a system subject to degradation failure. Most existing research on CBM focuses on single-unit systems where the whole system is treated as a black box. However, a system usually consists of a number of components and each component has its failure behavior. When degradation of the components is observable, CBM can be applied to the component level to improve the maintenance efficiency. This paper aims to study the optimal inspection/replacement CBM strategy for a multi-unit system. Degradation of each component is assumed to follow a Wiener process and periodic inspection is considered. We cast the problem into a Markov decision framework and derive the optimal maintenance decisions that minimize the maintenance cost. To better illustrate the optimal maintenance strategy, we start from a 1-out-of-2: G system and show that the optimal maintenance policy is a two-dimensional control limit policy. The argument used in the 1-out-of-2: G system can be readily extended to general cases in a similar way. The value iteration algorithm is used to find the optimal control limits, and the optimal inspection interval is subsequently determined through a one-dimensional search. A numerical study and a comprehensive sensitivity analysis are provided to illustrate the optimal maintenance strategy.