Value Iteration Algorithm Research Articles

The optimal probability of the risk for finite horizon partially observable Markov decision processes

<abstract><p>This paper investigates the optimality of the risk probability for finite horizon partially observable discrete-time Markov decision processes (POMDPs). The probability of the risk is optimized based on the criterion of total rewards not exceeding the preset goal value, which is different from the optimal problem of expected rewards. Based on the Bayes operator and the filter equations, the optimization problem of risk probability can be equivalently reformulated as filtered Markov decision processes. As an advantage of developing the value iteration technique, the optimality equation satisfied by the value function is established and the existence of the risk probability optimal policy is proven. Finally, an example is given to illustrate the effectiveness of using the value iteration algorithm to compute the value function and optimal policy.</p></abstract>

Solving zero-sum one-sided partially observable stochastic games

Many real-world situations are dynamic, with long-term interactions between multiple agents with uncertainty and limited observations. The agents must reason about which actions to take while also predicting and learning about what actions the other agents will take and how their choices will interact. In the most general setting, there is no limitation on the length of the sequence of actions the agent can perform — that is, there is no fixed horizon that can be used as an endpoint for analysis. These settings can be modeled as partially observable stochastic games (POSGs). Many adversarial domains (e.g., security settings) can be modeled as strictly competitive (or zero-sum) variants of these games.While these models are capable of modeling a wide variety of realistic problems, solving general POSGs is computationally intractable, so we focus on a broad subclass of POSGs called one-sided POSGs. In these games, only one agent has imperfect information while their opponent has full knowledge of the current situation. We provide a complete approach for solving zero-sum, one-sided POSGs: we (1) give a theoretical analysis of one-sided POSGs and their value functions, (2) show that a variant of a value-iteration algorithm converges in this setting, (3) adapt the heuristic search value-iteration algorithm for solving one-sided POSGs, (4) describe how to use approximate value functions to derive strategies in the game, and (5) experimentally demonstrate that our algorithm can solve one-sided POSGs of non-trivial sizes and analyze the scalability of our algorithm in three different domains: pursuit-evasion, patrolling, and search games.

Reinforcement learning and cooperative [formula omitted] output regulation of linear continuous-time multi-agent systems

This paper proposes a novel control approach to solve the cooperative H∞ output regulation problem for linear continuous-time multi-agent systems (MASs). Different from existing solutions to cooperative output regulation problems, a distributed feedforward-feedback controller is developed to achieve asymptotic tracking and reject both modeled and unmodeled disturbances. The feedforward control policy is computed via solving regulator equations, and the optimal feedback control policy is obtained through handling a zero-sum game. Instead of relying on the knowledge of system matrices in the state equations of the followers’ dynamics and initial stabilizing feedback control gains, a value iteration (VI) algorithm is proposed to learn the optimal feedback control gain and feedforward control gain using online data. To the best of our knowledge, this paper is the first to show that the proposed VI algorithm can approximate the solution to continuous-time game algebraic Riccati equations with guaranteed convergence. Finally, the numerical analysis is provided to show the effectiveness of the proposed approach.

State of the Art of Adaptive Dynamic Programming and Reinforcement Learning

This article introduces the state-of-the-art development of adaptive dynamic programming and reinforcement learning (ADPRL). First, algorithms in reinforcement learning (RL) are introduced and their roots in dynamic programming are illustrated. Adaptive dynamic programming (ADP) is then introduced following a brief discussion of dynamic programming. Researchers in ADP and RL have enjoyed the fast developments of the past decade from algorithms, to convergence and optimality analyses, and to stability results. Several key steps in the recent theoretical developments of ADPRL are mentioned with some future perspectives. In particular, convergence and optimality results of value iteration and policy iteration are reviewed, followed by an introduction to the most recent results on stability analysis of value iteration algorithms.

Replacement and Repair Optimization for Production Systems Under Random Production Waits

This study considers a repairable production system operated under an age-based preventive replacement policy that is subject to independent random production waits and failures. In addition to the age-based replacement policy, we propose a maintenance model that uses production waits to schedule preventive replacement. That is, a decision maker can preventively replace the system either during a production wait or at the age threshold, whereas if a failure occurs during production, the decision maker must decide whether to perform a minimal repair or a corrective replacement to restore the system. Under the above setting, we develop a semi-Markov decision process (SMDP) to obtain the optimal maintenance policy that minimizes the long-run average maintenance cost rate. We establish the existence of the optimal maintenance policy and provide an algorithm to numerically obtain the optimal action for each state. We further generalize the model to incorporate imperfect maintenance and the nonhomogeneous arrival of production waits. In the latter case, it is computationally intractable to optimize the SMDP using a value iteration algorithm due to the curse of dimensionality. To address this challenge, we further develop an approximate dynamic programming framework to generate high-quality solutions. An attractive feature of our model is its generality, such that the model includes many existing maintenance models as special cases. A real-world example from a steel factory is used to demonstrate the proposed model.

Reinforcement learning approach to the control of heavy material handling manipulators for agricultural robots

In this paper, we consider the optimal control problem of heavy material handling manipulators for agricultural robots. Unlike the existing results on agricultural robots, the robot parameters may be unknown for the designer in this paper. To learn the linear quadratic control gain under unknown robot parameters, two reinforcement learning algorithms, i.e., policy iteration (PI) algorithm and value iteration (VI) algorithm, are proposed. Then, through combining the advantages of PI algorithm and VI algorithm, i.e., satisfactory convergence rate and without the restriction on feasibility initial control policy, respectively, a hybrid iteration (HI) algorithm is proposed, which can both achieve a satisfactory convergence rate and remove restrictions on feasibility initial control policy. It is shown that the convergence of the proposed HI algorithm can be achieved in theory. Finally, a simulation example is given to show that our designed HI algorithm can achieve a satisfactory simulation time.

DROP: Deep relocating option policy for optimal ride-hailing vehicle repositioning

In a ride-hailing system, an optimal relocation of vacant vehicles can significantly reduce fleet idling time and balance the supply–demand distribution, enhancing system efficiency and promoting driver satisfaction and retention. Model-free deep reinforcement learning (DRL) has been shown to dynamically learn the relocating policy by actively interacting with the intrinsic dynamics in large-scale ride-hailing systems. However, the issues of sparse reward signals and unbalanced demand and supply distribution place critical barriers in developing effective DRL models. Conventional exploration strategy (e.g., the ϵ-greedy) may barely work under such an environment because of dithering in low-demand regions distant from high-revenue regions. This study proposes the deep relocating option policy (DROP) that supervises vehicle agents to escape from oversupply areas and effectively relocate to potentially underserved areas. We propose to learn the Laplacian embedding of a time-expanded relocation graph, as an approximation representation of the system relocation policy. The embedding generates task-agnostic signals, which in combination with task-dependent signals, constitute the pseudo-reward function for generating DROPs. We present a hierarchical learning framework that trains a high-level relocation policy and a set of low-level DROPs. We note that DROP is a general method and can be incorporated into existing model-free RL advances for further improvements in ride-hailing applications. The effectiveness of our approach is demonstrated using a custom-built high-fidelity simulator with real-world trip record data. We report that DROP significantly improves baseline value and policy iteration algorithms and can effectively resolve the dithering issue in low-demand areas.

Optimal decision strategy for discrete-time Markovian jump linear systems

This paper investigates the discrete-time Markovian jump linear systems (MJLSs) whose mode transition probability matrix (MTPM) can be adjusted by decisions. Motivated by switching law design in switched systems, the optimal decision strategy is proposed for stabilisation and optimisation of such MJLSs where decision cost is taken into account. First, aiming at system stability, the feasible domain of decision is given for stable and unstable MJLSs with initial MTPM. Second, a generalised performance index is put forward which contains both state cost and decision cost, and we obtain the quantitative relationship between the index and decision via stochastic dynamic programming. Finally, for the optimisation of the performance index, a value iteration algorithm is proposed with its convergence proof. This algorithm, searching for the optimal decision within the feasible domain, achieves superior performance on the basis of ensuring stability. Simulation results illustrate the effectiveness of the proposed optimal decision strategy.

Optimal consensus of a class of discrete-time linear multi-agent systems via value iteration with guaranteed admissibility

This paper investigates the optimal consensus problem for heterogeneous discrete-time(DT) linear multi-agent systems. The optimal consensus problem is formulated as finding a global Nash equilibrium solution subjected to the defined local performance index. A reinforcement learning(RL) value iteration(VI) algorithm is introduced to obtain the optimal policies in the sense of Nash equilibrium. To ensure the effectiveness of the VI algorithm, the admissibility of the iterative control policies for multi-agent systems is considered. With theoretical analysis, a new termination criterion is established to guarantee the admissibility of the iterative control policies. Furthermore, an online learning framework is designed with an actor-critic neural network(NN) to implement the VI algorithm. Finally, two simulation examples are presented respectively for leader–follower and leaderless multi-agent systems to verify the effectiveness of the proposed method.

Adaptive optimal tracking control with novel event‐triggered formulation for a type of nonlinear systems

SummaryIn this article, an event‐based intelligent critic algorithm is developed to address the optimal tracking control problem for a type of discrete‐time nonlinear systems. The nonlinear optimal tracking control design is replaced by solving the optimal regulation problem of the error system. Then, the generalized value iteration algorithm is employed to obtain the admissible tracking control law with off‐line learning. Next, a novel triggering condition is designed to reduce the update times of the controller and improve the resource utilization. It is emphasized that this triggering condition is established based on the iteration of the time‐triggered mechanism. Moreover, in order to realize the cost guarantee of the error system, the real cost function is proved to possess a predetermined upper bound. By analysis, it is shown that the error system is asymptotically stable while the tracking error and the sampling signal are uniformly ultimately bounded during the process of training neural networks. Finally, two examples are conducted to demonstrate the effectiveness of the proposed algorithm.

Online Value Iteration for Intelligent Discounted Tracking Design of Constrained Systems

In this brief, in order to solve the discounted optimal tracking control problem for discrete-time systems with control constraints, an advanced online value iteration (VI) algorithm is developed. First, we revisit discounted general value iteration (GVI) for optimal tracking control design. Second, the stability condition of GVI is established, which can be applied to evaluate the current iterative tracking control policy. Third, based on the concept of attraction domain, the evolving tracking control policies generated by online VI can be obtained through judging the location of the current tracking error. Finally, we prove system stability based on online VI for the present tracking control problem. A simulation example is shown to illustrate the effectiveness of the developed algorithm.

A Generalized Minimax Q-Learning Algorithm for Two-Player Zero-Sum Stochastic Games

We consider the problem of two-player zero-sum games. This problem is formulated as a min-max Markov game in the literature. The solution of this game, which is the min-max payoff, starting from a given state is called the min-max value of the state. In this work, we compute the solution of the two-player zero-sum game utilizing the technique of successive relaxation that has been successfully applied in the literature to compute a faster value iteration algorithm in the context of Markov Decision Processes. We extend the concept of successive relaxation to the setting of two-player zero-sum games. We show that, under a special structure on the game, this technique facilitates faster computation of the min-max value of the states. We then derive a generalized minimax Q-learning algorithm that computes the optimal policy when the model information is not known. Finally, we prove the convergence of the proposed generalized minimax Q-learning algorithm utilizing stochastic approximation techniques, under an assumption on the boundedness of iterates. Through experiments, we demonstrate the effectiveness of our proposed algorithm.

Opportunistic offloading scheme for content delivery service using electro‐mobility networks

AbstractVehicular caching (VC) in electro‐mobility networks has become promising for supporting the needs of low end‐to‐end service delays and reducing the load of networks (i.e. edge caching [EC]) for content delivery service. However, since VCs are available only when they are in the vicinity of the user, intermittent connectivity should be considered. In this paper, the opportunistic offloading scheme for content delivery service is proposed, and a decision is made regarding whether to use VC or EC for the content delivery service. In the proposed scheme, when VC is not available, the user determines whether to choose VC or EC for the content download. If VC is chosen, the user delays the content download to increase the VC utilization according to the expectation of VC contacts. This results in the delayed service completion but can reduce the EC load (i.e. offloading gain). If EC is chosen, the user can download the content without any delay or service interruption. To find the optimal policy, this paper formulates a Markov decision process (MDP) problem and uses the value iteration algorithm to solve the problem. The evaluation results show that the proposed scheme outperforms the comparison schemes based on the expected reward.

Maintenance analysis of a partial observable K-out-of-N system with load sharing units

In this paper, we consider the inspection and maintenance optimization of a K-out-of- N load-sharing system that operates in a deteriorating working condition. The failure rate of each component depends on its load-sharing and the system working condition. During the operation, the system working condition can be deteriorated from the healthy state to the abnormal state. Both the states of the components and the system working environment are hidden. To ensure the system safety, periodical inspection is implemeted, upon which, two-folds of information can be obtained: the state of each component and the partial revealed information corresponding to the state of the working condition. A maintenance policy is proposed based on the observations. The policy is assessed by the total expected discounted maintenance cost in the long-run horizon. We cast the problem into a partially observable Markov decision process framework. We utilize the value iteration algorithm to solve the inspection and maintenance optimization problem. Sensitivity analyses through numerical examples are carried out. A case study of a parallel system with electric motors is examined to show the applicability of the proposed model.

Learning on the Job: Long-Term Behavioural Adaptation in Human-Robot Interactions

In this work, we propose a framework for allowing autonomous robots deployed for extended periods of time in public spaces to adapt their own behaviour online from user interactions. The robot behaviour planning is embedded in a Reinforcement Learning (RL) framework, where the objective is maximising the level of overall user engagement during the interactions. We use the Upper-Confidence-Bound Value-Iteration (UCBVI) algorithm, which gives a helpful way of managing the exploration-exploitation trade-off for real-time interactions. An engagement model trained end-to-end generates the reward function in real-time during policy execution. We test this approach in a public museum in Lincoln (U.K.), where the robot is deployed as a tour guide for the visitors. Results show that after a couple of months of exploration, the robot policy learned to maintain the engagement of users for longer, with an increase of 22.8% over the initial static policy in the number of items visited during the tour and a 30% increase in the probability of completing the tour. This work is a promising step toward behavioural adaptation in long-term scenarios for robotics applications in social settings.

Adaptive task offloading of rechargeable UAV edge computing network based on double decision value iteration

With the development of Internet of Things technology, unmanned aerial vehicles (UAVs) have attracted tremendous attentions due to its broad applications in the conditions of high rate and low delay communication. In this paper, we consider a wireless-powered and UAV-enabled MEC system. Because of the limitation of UAV’s battery capacity, we deploy microwave power stations next to the base stations to provide microwave energy to the UAV from ground microwave antenna array. The UAV receives tasks from mobile users and processes the task locally or offloads task to edge servers for processing, or charges its battery through microwave stations when its battery is low. Due to the uncertainty of the UAV’s trajectory and the working status of the microwave power stations, we build the offloading decision process as a finite Markov Decision Process(MDP) model. In the MDP model, we solve the optimization problem of minimizing the total consumption of time and energy through an optimal offloading algorithm. The algorithm we proposed is based on the fast-converging value iteration algorithm. Furthermore, extensive numerical results shows that the proposed algorithm outperforms other four conventional baseline schemes.

Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium

We develop provably efficient reinforcement learning algorithms for two-player zero-sum finite-horizon Markov games with simultaneous moves. To incorporate function approximation, we consider a family of Markov games where the reward function and transition kernel possess a linear structure. Both the offline and online settings of the problems are considered. In the offline setting, we control both players and aim to find the Nash equilibrium by minimizing the duality gap. In the online setting, we control a single player playing against an arbitrary opponent and aim to minimize the regret. For both settings, we propose an optimistic variant of the least-squares minimax value iteration algorithm. We show that our algorithm is computationally efficient and provably achieves an [Formula: see text] upper bound on the duality gap and regret, where d is the linear dimension, H the horizon and T the total number of timesteps. Our results do not require additional assumptions on the sampling model. Our setting requires overcoming several new challenges that are absent in Markov decision processes or turn-based Markov games. In particular, to achieve optimism with simultaneous moves, we construct both upper and lower confidence bounds of the value function, and then compute the optimistic policy by solving a general-sum matrix game with these bounds as the payoff matrices. As finding the Nash equilibrium of a general-sum game is computationally hard, our algorithm instead solves for a coarse correlated equilibrium (CCE), which can be obtained efficiently. To our best knowledge, such a CCE-based scheme for optimism has not appeared in the literature and might be of interest in its own right. Funding: Q. Xie is partially supported by the National Science Foundation [Grant CNS-1955997] and by J.P. Morgan. Y. Chen is partially supported by the National Science Foundation [Grants CCF-1657420, CCF-1704828, and CCF-2047910]. Z. Wang acknowledges the National Science Foundation [Grants 2048075, 2008827, 2015568, and 1934931], the Simons Institute (Theory of Reinforcement Learning), Amazon, J.P. Morgan, and Two Sigma for their support.

Innovative Construction of Reinforcement Learning Model for Information Fusion in Music Education

In order to improve the effect of modern music education, this paper applies the digital information technology of music resources to the construction of the music teaching system and derives two new types of semi-decision-making process reinforcement learning algorithms based on the Bellman optimality equation base on discrete time. Moreover, this paper uses the comparative research methods to obtain the Q-value learning curve of the incremental value iterative reinforcement learning algorithm based on the semi-Markov decision process and the incremental value iterative reinforcement learning algorithm based on the dichotomy to improve the fusion effect of music teaching resources. Finally, this paper combines the actual needs of modern music education to construct an intelligent music teaching model.

Optimal output tracking control of linear discrete-time systems with unknown dynamics by adaptive dynamic programming and output feedback

This paper discusses the output-feedback-based model-free optimal output tracking control problem of discrete-time systems with completely unknown system models under mild assumptions. The only information that allows utilisation is the system output and the reference output. To overcome these challenges, the paper aims to solve a model-free optimal output regulation problem for achieving optimal output tracking control; solving an optimal output regulation problem is equivalent to solving a linear quadratic regulation (LQR) problem and a constrained static optimisation problem. The state reconstruction is first given to represent the system state in terms of the input and output sequences. A data-driven value iteration (VI) algorithm is then proposed to iteratively approximate the solution to the discrete-time algebraic Riccati equation (ARE) of the corresponding LQR problem on the basis of input and output data. Next, based on the iterative solutions with respect to the ARE, a model-free solution is provided for the corresponding constrained static optimisation problem. Finally, an alternative reference system equivalent to the original reference system is established to avoid the requirement of having the knowledge of the reference system dynamics and the reference state. A numerical example is employed to demonstrate the effectiveness of the proposed control scheme.

Maintenance Policies for Two-Unit Balanced Systems Subject to Degradation

In this article, we present a novel maintenance policy optimization method for systems with two balanced components. The components in the system are assumed to degrade over time according to a bivariate Wiener process. The maintenance actions aim at eliminating the differences of degradation levels of system components at the cost of aggravating the degradation. Utilizing the Markov decision process, the maintenance model is put forward under both the finite and the infinite planning horizons, from which we find the structural properties of the optimal policies. Backwards dynamic programming and value iteration algorithms are employed to optimize the maintenance decisions. Examples along with sensitivity analysis are presented to facilitate the illustration and insight attainment. We find that the maintenance policies are to a great extent regulated by the absolute degradation difference between the two components.