Markov Chains and Decision Processes for Engineers and Managers

Abstract

Markov Chain Structure and Models Historical Note States and Transitions Model of the Weather Random Walks Estimating Transition Probabilities Multiple-Step Transition Probabilities State Probabilities after Multiple Steps Classification of States Markov Chain Structure Markov Chain Models Problems References Regular Markov Chains Steady State Probabilities First Passage to a Target State Problems References Reducible Markov Chains Canonical Form of the Transition Matrix The Fundamental Matrix Passage to a Target State Eventual Passage to a Closed Set Within a Reducible Multichain Limiting Transition Probability Matrix Problems References A Markov Chain with Rewards (MCR) Rewards Undiscounted Rewards Discounted Rewards Problems References A Markov Decision Process (MDP) An Undiscounted MDP A Discounted MDP Problems References Special Topics: State Reduction and Hidden Markov Chains State Reduction An Introduction to Hidden Markov Problems References Index