Factored Markov Decision Process Research Articles

Online reinforcement learning for condition-based group maintenance using factored Markov decision processes

We investigate a condition-based group maintenance problem for multi-component systems, where the degradation process of a specific component is affected only by its neighbouring ones, leading to a special type of stochastic dependence among components. We formulate the maintenance problem into a factored Markov decision process taking advantage of this dependence property, and develop a factored value iteration algorithm to efficiently approximate the optimal policy. Through both theoretical analyses and numerical experiments, we show that the algorithm can significantly reduce computational burden and improve efficiency in solving the optimization problem. Moreover, since model parameters are unknown a priori in most practical scenarios, we further develop an online reinforcement learning algorithm to simultaneously learn the model parameters and determine an optimal maintenance action upon each inspection. A novel feature of this online learning algorithm is that it is capable of learning both transition probabilities and system structure indicating the stochastic dependence among components. We discuss the error bound and sample complexity of the developed learning algorithm theoretically, and test its performance through numerical experiments. The results reveal that our algorithm can effectively learn the model parameters and approximate the optimal maintenance policy.

Maintenance optimisation of a series production system with intermediate buffers using a multi-agent FMDP

Maintenance decision making of a series production system with intermediate buffers is a research topic of practical significance. The Markov decision process (MDP) is an effective tool to undertake maintenance optimisation of a series production system with buffers due to its capacity in dealing with complex structure of maintenance strategies for such systems. However, the MDP has only been employed to analyse small systems due to the “curse of dimensionality”. A multi-agent factored Markov decision process (FMDP) is adopted in this study to remit the “curse of dimensionality”. The series production system considered in the study consists of several subsystems, and each subsystem is managed by an agent. A new method is developed to select maintenance actions in cooperation of different agents. An approximate linear programming algorithm is used to solve the FMDP model efficiently. The numerical study shows that the developed methods can deal with medium-scale series production systems with an insignificant small error in maintenance decision making.

Efficient incremental planning and learning with multi-valued decision diagrams

In the domain of decision theoretic planning, the factored framework (Factored Markov Decision Process, fmdp) has produced optimized algorithms using structured representations such as Decision Trees (Structured Value Iteration (svi), Structured Policy Iteration (spi)) or Algebraic Decision Diagrams (Stochastic Planning Using Decision Diagrams (spudd)). Since it may be difficult to elaborate the factored models used by these algorithms, the architecture sdyna, which combines learning and planning algorithms using structured representations, was introduced. However, the state-of-the-art algorithms for incremental learning, for structured decision theoretic planning or for reinforcement learning require the problem to be specified only with binary variables and/or use data structures that can be improved in term of compactness. In this paper, we propose to use Multi-Valued Decision Diagrams (mdds) as a more efficient data structure for the sdyna architecture and describe a planning algorithm and an incremental learning algorithm dedicated to this new structured representation. For both planning and learning algorithms, we experimentally show that they allow significant improvements in time, in compactness of the computed policy and of the learned model. We then analyzed the combination of these two algorithms in an efficient sdyna instance for simultaneous learning and planning using mdds.

Maintenance optimisation of a parallel-series system with stochastic and economic dependence under limited maintenance capacity

Maintenance optimisation of a parallel-series system considering both stochastic and economic dependence among components as well as limited maintenance capacity is studied in this paper. The maintenance strategies of the components are jointly optimised, and the degradation process of the system is modelled to address the stochastic dependence and limited maintenance capacity issues. To overcome the “curse of dimensionality” problem where the state space of a parallel-series system increases rapidly with the increased number of components in the system, the factored Markov decision process (FMDP) is employed for maintenance optimisation in this work. An improved approximate linear programming (ALP) algorithm is then developed. The selection of the basis functions and the state relevance weights for ALP is also investigated to enhance the performance of the ALP algorithm. Results from the numerical study show that the current approach can handle the decision optimisation problem for multi-component systems of moderate size, and the error of maintenance decision-making induced by the improved ALP is negligible. The outcome from this research provides a useful reference to overcome the “curse of dimensionality” problem during the maintenance optimisation of multi-component systems.

Model-based Bayesian Reinforcement Learning in Factored Markov Decision Process

Learning the enormous number of parameters is a challenging problem in model-based Bayesian reinforcement learning. In order to solve the problem, we propose a model-based factored Bayesian reinforcement learning (F-BRL) approach. F-BRL exploits a factored representation to describe states to reduce the number of parameters. Representing the conditional independence relationships between state features using dynamic Bayesian networks, F-BRL adopts Bayesian inference method to learn the unknown structure and parameters of the Bayesian networks simultaneously. A point-based online value iteration approach is then used for planning and learning online. The experimental and simulation results show that the proposed approach can effectively reduce the number of learning parameters, and enable online learning for dynamic systems with thousands of states.

Using Equilibrium Policy Gradients for Spatiotemporal Planning in Forest Ecosystem Management

Spatiotemporal planning involves making choices at multiple locations in space over some planning horizon to maximize utility and satisfy various constraints. In Forest Ecosystem Management, the problem is to choose actions for thousands of locations each year including harvesting, treating trees for fire or pests, or doing nothing. The utility models could place value on sale of lumber, ecosystem sustainability or employment levels and incorporate legal and logistical constraints on actions such as avoiding large contiguous areas of clearcutting. Simulators developed by forestry researchers provide detailed dynamics but are generally inaccesible black boxes. We model spatiotemporal planning as a factored Markov decision process and present a policy gradient planning algorithm to optimize a stochastic spatial policy using simulated dynamics. It is common in environmental and resource planning to have actions at different locations be spatially interelated; this makes representation and planning challenging. We define a global spatial policy in terms of interacting local policies defining distributions over actions at each location conditioned on actions at nearby locations. Markov chain Monte Carlo simulation is used to sample landscape policies and estimate their gradients. Evaluation is carried out on a forestry planning problem with 1,880 locations using a variety of value models and constraints.

Solving Factored MDPs with Hybrid State and Action Variables

Efficient representations and solutions for large decision problems with continuous and discrete variables are among the most important challenges faced by the designers of automated decision support systems. In this paper, we describe a novel hybrid factored Markov decision process (MDP) model that allows for a compact representation of these problems, and a new hybrid approximate linear programming (HALP) framework that permits their efficient solutions. The central idea of HALP is to approximate the optimal value function by a linear combination of basis functions and optimize its weights by linear programming. We analyze both theoretical and computational aspects of this approach, and demonstrate its scale-up potential on several hybrid optimization problems.

Apprentissage par renforcement dans le cadre des processus décisionnels de Markov factorisés observables dans le désordre. Etude expérimentale du Q-Learning parallèle appliqué aux problèmes du labyrinthe et du New York Driving

This paper presents experimental results obtained with an original architecture that can do generic learning for randomly observable factored Markov decision process (ROFMDP). First, the paper describes the theoretical framework of ROFMDP and the working of this algorithm, in particular the parallelization principle and the dynamic reward allocation process. Then, the architecture is applied to two navigation problems (gridworld and New York Driving). The tests show that the architecture allows to learn a good and generic policy in spite of the large dimensions of the state spaces of both systems.

An Efficient Solution Algorithm for Factored MDP Using Feature Vector Extraction

In factored Markov decision process (FMDP) such as Robocup system, the effect to value evaluation of various states is different from each other within state attributes. There are some important state attributes that can determine the whole state value either uniquely, or at least, approximately. Instead of using the relevance among states to reduce the state space, this paper addresses the problem of curse of dimensionality in large FMDP by approximating state value function through feature vector extraction. A key contribution of this paper is that it reduces the computation complexity by constraints reduction in linear programming, speeds up the production of joint strategy by transplanting the value function to the more complex game in reinforcement learning. Experimental results are provided on Robocup free kick, demonstrating a promising indication of the efficiency of the approach and its’ ability of transplanting the learning result. Comparing this algorithm to an existing state-of-the-art approach indicates that it can not only improve the learning speed, but also can transplant state value function to the Robocup with more players instead of learning again.