Imperfect State Observation Research Articles

Stagewise Newton Method for Dynamic Game Control With Imperfect State Observation

In this letter, we study dynamic game optimal control with imperfect state observations and introduce an iterative method to find a local Nash equilibrium. The algorithm consists of an iterative procedure combining a backward recursion similar to minimax differential dynamic programming and a forward recursion resembling a risksensitive Kalman smoother. A coupling equation renders the resulting control dependent on the estimation. In the end, the algorithm is equivalent to a Newton step but has linear complexity in the time horizon length. Furthermore, a merit function and a line search procedure are introduced to guarantee convergence of the iterative scheme. The resulting controller reasons about uncertainty by planning for the worst case disturbances. Lastly, the low computational cost of the proposed algorithm makes it a promising method to do output-feedback model predictive control on complex systems at high frequency. Numerical simulations on realistic robotic problems illustrate the risk-sensitive behavior of the resulting controller.

Optimally Myopic Scheduling Policy for Downlink Channels With Imperfect State Observation

We consider a downlink channel scheduling problem in a wireless communication system where a base station serves a user by a number $N$ of multistate Markov channels. In each time slot one channel is allocated to the user according to imperfect state observation information, and some reward is accrued depending on the state of the allocated channel. The objective is to design a scheduling policy that maximizes the expected accumulated discounted reward over an infinite time horizon. Mathematically, the decision-making problem can be formulated into a restless multiarmed bandit with exponential memory and time complexity. One of the feasible policies is the myopic (or greedy) one, which only maximizes the immediate reward with linear complexity. Specifically, we carry out a theoretical analysis on the performance of the myopic policy with imperfect observation, introduce the monotone likelihood ratio order to characterize the evolving structure of information states, and establish a set of closed-form conditions to guarantee the optimality of the myopic policy in downlink channel allocation.

Robust Partially Observable Markov Decision Processes

In a variety of applications, decisions needs to be made dynamically after receiving imperfect observations about the state of an underlying system. Partially Observable Markov Decision Processes (POMDPs) are widely used in such applications. To use a POMDP, however, a decision-maker must have access to reliable estimations of core state and observation transition probabilities under each possible state and action pair. This is often challenging mainly due to lack of ample data, especially when some actions are not taken frequently enough in practice. This significantly limits the application of POMDPs in real-world settings. In healthcare, for example, medical tests are typically subject to false-positive and false-negative errors, and hence, the decision-maker has imperfect information about the health state of a patient. Furthermore, since some treatment options have not been recommended or explored in the past, data cannot be used to reliably estimate all the required transition probabilities regarding the health state of the patient. We introduce an extension of POMDPs, termed Robust POMDPs (RPOMDPs), which allows dynamic decision-making when there is ambiguity regarding transition probabilities. This extension enables making robust decisions by reducing the reliance on a single probabilistic model of transitions, while still allowing for imperfect state observations. We develop dynamic programming equations for solving RPOMDPs, provide a sufficient statistic and an information state, discuss ways in which their computational complexity can be reduced, and connect them to stochastic zero-sum games with imperfect private monitoring.

Pessimistic Optimal Choice for Risk-Averse Agents: The Continuous-Time Limit

We extend Hansen and Sargent's (Discounted linear exponential quadratic gaussian control, 1994, IEEE Trans Autom Control 40:968---971 1995, 2013) analysis of dynamic optimization with risk-averse agents in two directions. Firstly, following Whittle (Risk-sensitive optimal control, 1990), we show that the optimal risk-averse policy is identified via a pessimistic choice mechanism and described by simple recursive formulae. Secondly, we investigate the continuous-time limit and show that sufficient conditions for the existence of optimal solutions coincide with those which apply under risk-neutrality. Our analysis is conducted both under perfect and imperfect state observation. As an illustrative example, we analyze the optimal production policy of an entrepreneur running a monopolistic firm which faces a demand schedule subject to stochastic shocks, showing that risk-aversion induces her to act more aggressively.

Robust Control, Informational Frictions, and International Consumption Correlations

In this paper we examine the effects of model misspecification (robustness or RB) on international consumption correlations in two otherwise standard small open economy models: one with perfect state observation and the other with imperfect state observation. We show that in the presence of capital mobility in financial markets, RB lowers the international consumption correlations by generating heterogeneous responses of consumption to income shocks across countries facing different macroeconomic uncertainty. However, the calibrated RB model with perfect state observation cannot explain the observed consumption correlations quantitatively. We then show that the RB model with imperfect state observation is capable of matching the behavior of international consumption quantitatively via two channels: (i) the gradual response to income shocks that increases the correlations and (ii) the presence of the common noise shocks that reduce the correlations.

A least squares temporal difference actor–critic algorithm with applications to warehouse management

AbstractThis article develops a new approximate dynamic programming (DP) algorithm for Markov decision problems and applies it to a vehicle dispatching problem arising in warehouse management. The algorithm is of the actor‐critic type and uses a least squares temporal difference learning method. It operates on a sample‐path of the system and optimizes the policy within a prespecified class parameterized by a parsimonious set of parameters. The method is applicable to a partially observable Markov decision process setting where the measurements of state variables are potentially corrupted, and the cost is only observed through the imperfect state observations. We show that under reasonable assumptions, the algorithm converges to a locally optimal parameter set. We also show that the imperfect cost observations do not affect the policy and the algorithm minimizes the true expected cost. In the warehouse application, the problem is to dispatch sensor‐equipped forklifts in order to minimize operating costs involving product movement delays and forklift maintenance. We consider instances where standard DP is computationally intractable. Simulation results confirm the theoretical claims of the article and show that our algorithm converges more smoothly than earlier actor–critic algorithms while substantially outperforming heuristics used in practice. © 2012 Wiley Periodicals, Inc. Naval Research Logistics, 2012

Multistate capture–recapture analysis under imperfect state observation: an application to disease models

Summary 1 Multistate capture–recapture models are frequently used to estimate the survival and state transition parameters needed to parameterize stage-structured population models, tools that are important for conservation and management. Typically, such models assume that all encountered individuals can be assigned to a particular state without error or ambiguity, a requirement which is difficult to meet in practice. Model extensions to relax this assumption would increase the richness of ecological data sets available for estimating life-history and stage-transition parameters with multistate models. 2 One relatively common analytical approach when confronted with ambiguity in state determination is to censor all encounters where the state of an animal cannot be ascertained. Here, we present an alternative approach, which uses a hidden Markov (or multievent) modelling framework that can incorporate data from encounters of unknown state. Using simulation, we show that our approach leads to estimators of state-specific survival and transition probabilities that are more precise, and sometimes considerably so, than methods based on censoring. 3 We demonstrate our approach using field data from a study of the dynamics of conjunctivitis in the house finch Carpodacus mexicanus Muller. A fundamental challenge in modelling disease dynamics involves the estimation of the rates of entry and exit from one or more disease states, which can be complicated when disease state is uncertain. We show that incorporating data from unknown states made substantial improvements to parameter precision. 4 Synthesis and applications. Missing or incomplete records are an unfortunate but common feature of many ecological field studies, often diminishing the quality and quantity of data. Our approach of treating state as a hidden Markov process allows such records to be used, increasing the precision of survival and state transition parameters in multistate mark–recapture studies. Our approach is more general than other approaches in the literature, and does not require specialized sampling designs or ancillary information to inform state assignment. We suggest that ecologists consider using this modelling approach instead of censoring records whenever state information is missing.

Discussion of the Paper by Whittle

Journal of the Royal Statistical Society: Series B (Methodological)Volume 53, Issue 3 p. 529-538 DiscussionFree Access Discussion of the Paper by Whittle First published: 1991 https://doi.org/10.1111/j.2517-6161.1991.tb01843.xAboutPDF ToolsExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat REFERENCES IN THE DISCUSSION Akaike, H. (1980) Seasonal adjustment by a Bayesian modeling. J. Time Ser. Anal., 1, 1– 13. Bartlett, M. S. (1949) Some evolutionary stochastic processes. J. R. Statist. Soc. B, 11, 211– 229. Bartlett, M. S. (1961) Equations for stochastic path integrals. Proc. Camb. Phil. Soc., 57, 568– 573. Bartlett, M. S. (1981) Population and community structure and interaction. In The Mathematical Theory of the Dynamics of Biological Populations II (eds R. W. Hiorns and D. Cook), pp. 119– 137. New York: Academic Press. van den Bol, M. E. (1987) The analysis of a mixed model. Genstat Newslett., no. 20, 7– 12. Brotherton, T. and Gersch, W. (1981) A data analytic approach to the smoothing problem and some of its variations. In Proc. IEEE Conf. Decision and Control. San Diego: IEEE. Duncan, D. B. and Horn, S. D. (1972) Linear dynamic recursive estimation from the point of view of regression analysis. J. Am. Statist. Ass., 67, 815– 821. Hannan, E. J. and Deistler, M. (1988) The Statistical Theory of Linear Systems. New York: Wiley. Harvey, A. C. (1984) A unified view of statistical forecasting procedures. J. Forecast., 3, 245– 275. Harvey, A. C. (1989) Forecasting, Structural Time Series Models and the Kalman Filter. Cambridge: Cambridge University Press. Henderson, C. R. (1953) Estimation of variance and covariance components. Biometrics, 9, 226– 252. Jakeman, A. J. and Young, P. C. (1984) Recursive filtering and the inversion of ill-posed causal problems. Util. Math., 25, 351– 376. de Jong, P. (1988) The likelihood for a state space model. Biometrika, 75, 165– 169. Kitagawa, G. (1981) Anon-stationary time-series model and its fitting by a recursive filter. J. Time Ser., 2, 103– 116. Kohn, R. and Ansley, C. F. (1986) Estimation, prediction and interpolation for ARIMA models with missing data. J. Am. Statist. Ass., 81, 751– 761. Kydland, F. E. and Prescott, E. C. (1990) Business cycles: real facts and a monetary myth. Q. Rev. Fed. Reserve Bank, Minneap., 14, 3– 18. Lukas, M. A. (1980) Regularisation. In The Application and Numerical Solution of Integral Equations (eds R. S. Anderrsen et al.). Sijthoff and Noordhoff. Ng, C. N. and Young, P. C. (1990) Recursive estimation and forecasting of nonstationary time-series. J. Forecast., 9, 173– 204. Reinsch, C. H. (1967) Smoothing by spline functions. Numer. Math., 10, 177– 183. Schiller, R. (1973) A distributed lag estimator derived from smoothness priors. Econometrica, 41, 775– 778. Simon, H. A. (1956) Dynamic programming under uncertainty with a quadratic criterion function. Econometrica, 24, 74– 81. Stigler, S. M. (1978) Mathematical statistics in the early states. Ann. Statist., 6, 239– 265. Theil, H. (1957) A note on certainty equivalence in economic planning. Econometrica, 25, 346– 349. West, M., Harrison, P. J. and Migon, H. S. (1985) Dynamic generalised linear models and Bayesian forecasting (with discussion). J. Am. Statist. Ass., 80, 73– 97. Whittaker, E. T. (1923) On a new method of graduation. Proc. Edinb. Math. Soc., 63– 75. Whittle, P. (1957) Curve and periodogram smoothing. J. R. Statist. Soc. B, 19, 38– 47. Whittle, P. (1963) Prediction and Regulation. London: English Universities Press. Whittle, P. (1983) Prediction and Regulation, 2nd edn. Oxford: Blackwell. Whittle, P. (1990) A risk-sensitive maximum principle. Syst. Control Lett., 15, 183– 192. Whittle, P. (1991) A risk-sensitive maximum principle; the case of imperfect state observation. IEEE Trans. Autotn. Control, to be published. Young, P. C. (1988) Recursive extrapolation, interpolation and smoothing of nonstationary time-series. In Identification and System Parameter Estimation (ed. H. F. Chen), pp. 33– 44. Oxford: Pergamon. Young, P. C. (1990) State space forecasting and seasonal adjustment. J. Forecast., 9, 2. Young, P. C. and Ng, C. N. (1989) Variance intervention. J. Forecast., 8, 399– 416. Young, P. C., Ng, C. N. and Armitage, P. (1989) A systems approach to economic forecasting and seasonal adjustment. Int. J. Comput. Math. Applic., 18, 481– 501. Volume53, Issue31991Pages 529-538 This article also appears in:Discussion Papers ReferencesRelatedInformation

A risk-sensitive maximum principle: the case of imperfect state observation

The risk-sensitive maximum principle for optimal stochastic control derived by the author in an earlier work (System Control Letters, vol.15, 1990) is restated. This is an immediate generalization of the classic Pontryagin principle, to which it reduces in the deterministic case, and is expressed immediately in terms of observables. It is derived on the assumption that the criterion function is the exponential of an additive cost function, and is exact under linear-quadratic Gaussian assumptions, but is otherwise valid as a large deviation approximation. The principle is extended to the case of imperfect state observation after preliminary establishment of a certainty-equivalence principle. The derivation yields as byproduct a large-deviation version of the updating equation for nonlinear filtering. The development is heuristic. It is believed that the mathematical arguments given are the essential ones, and provide a self-contained treatment at this level. >

Optimal stabilization policy with control lags and imperfect state observations

Optimal stabilization policy with control lags and imperfect state observations

Optimal Infinite-Horizon Undiscounted Control of Finite Probabilistic Systems

A finite-input, finite-state, finite-output stochastic control problem with imperfect state observation and classical information pattern is shown to be meaningful as the horizon increases without bound and the discount rate approaches unity. The plant model, a finite probabilistic system, includes the Markov decision and partially-observed Markov decision problems as special cases. Under conditions resembling controllability and observability in linear systems we show that: an optimal strategy exists, it may be realized by a stationary policy on the state estimate, its performance does not depend on the initial state distribution, and convergence rates for its finite-horizon and discounted performances are readily established.