Use Of Dynamic Programming Techniques Research Articles

Graph Theory-Based Approach to Accomplish Complete Coverage Path Planning Tasks for Reconfigurable Robots

Extensive studies regarding complete coverage problems have been conducted, but a few tackle scenarios where the mobile robot is equipped with reconfigurable modules. The reconfigurability of these robots creates opportunities to develop new navigation strategies with higher dexterity; however, it also simultaneously adds in constraints to the direction of movements. This paper aims to develop a valid navigation strategy that allows tetromino-based self-reconfigurable robots to perform complete coverage tasks. To this end, a novel graph theory-based model to simulate the workspace coverage and make use of dynamic programming technique for optimal path searching and adaptive robot morphology shifting algorithms is proposed. Moreover, the influence of algorithms starting variables on workspace coverage outcome is analyzed thoughtfully in this paper. The simulation results showed that the proposed method is capable of generating navigation paths throughout the workspace, which ensures complete workspace coverage while minimizing the total number of actions performed by the robot.

Convergence and numerical stability of action-dependent heuristic dynamic programming algorithms based on RLS learning for online DLQR optimal control

The development and the numerical stability analysis of a novel algorithm of approximate dynamic programming (ADP) based on RLS learning for approximating the optimal control solution online in real-time are the main issues of this paper. The approximate dynamic programming is a method developed to make possible the use of dynamic programming techniques in real-time, but this method has a reasonable mathematical complexity due to the size of the internal matrices of the algorithm and the need for inversion of some of them. Thus, focusing on improving numerical stability and computational cost of ADP algorithms, more specifically in the action-dependent heuristic dynamic programming and optimal control context, UDUT-type unitary transformations are integrated in actor-critic architectures, which produce algorithms with better specifications for implementation in real-world optimal control systems. The control and stabilisation of the inverted pendulum system on a motor driven cart is established as a study platform to evaluate the convergence and numerical stability for the estimated parameters of the proposed algorithm.

Convergence and numerical stability of action-dependent heuristic dynamic programming algorithms based on RLS learning for online DLQR optimal control

The development and the numerical stability analysis of a novel algorithm of approximate dynamic programming (ADP) based on RLS learning for approximating the optimal control solution online in real-time are the main issues of this paper. The approximate dynamic programming is a method developed to make possible the use of dynamic programming techniques in real-time, but this method has a reasonable mathematical complexity due to the size of the internal matrices of the algorithm and the need for inversion of some of them. Thus, focusing on improving numerical stability and computational cost of ADP algorithms, more specifically in the action-dependent heuristic dynamic programming and optimal control context, UDUT-type unitary transformations are integrated in actor-critic architectures, which produce algorithms with better specifications for implementation in real-world optimal control systems. The control and stabilisation of the inverted pendulum system on a motor driven cart is established as a study platform to evaluate the convergence and numerical stability for the estimated parameters of the proposed algorithm.

Distortion Minimization in Multi-Sensor Estimation With Energy Harvesting

This paper presents a design methodology for optimal energy allocation to estimate a random source using multiple wireless sensors equipped with energy harvesting technology. In this framework, multiple sensors observe a random process and then transmit an amplified uncoded analog version of the observed signal through Markovian fading wireless channels to a remote station. The sensors have access to an energy harvesting source, which is an everlasting but unreliable random energy source compared to conventional batteries with fixed energy storage. The remote station or so-called fusion centre estimates the realization of the random process by using a best linear unbiased estimator. The objective is to design optimal energy allocation policies at the sensor transmitters for minimizing total distortion over a finite-time horizon or a long term average distortion over an infinite-time horizon subject to energy harvesting constraints. This problem is formulated as a Markov decision process (MDP) based stochastic control problem and the optimal energy allocation policies are obtained by the use of dynamic programming techniques. Using the concept of submodularity, the structure of the optimal energy allocation policies is studied, which leads to an optimal threshold policy for binary energy allocation levels. Motivated by the excessive communication burden for the optimal control solutions where each sensor needs to know the channel gains and harvested energies of all other sensors, suboptimal decentralized strategies are developed where only statistical information about all other sensors' channel gains and harvested energies is required. Numerical simulation results are presented illustrating the performance of the optimal and suboptimal algorithms.

Decision assistance agent in real-time simulation

Urban society relies heavily on critical infrastructure (CI) such as power and water systems. The anticipated prosperity and the national security of society depend on the ability to understand, measure and analyse the vulnerabilities and interdependencies of this system of infrastructures. Only then can emergency responders (ER) react quickly and effectively to any major disruption that the system might face. In this paper, we propose a model to train a reinforcement learning (RL) agent that is able to optimise resource usage following an infrastructure disruption. The novelty of our approach is the use of dynamic programming techniques to build an agent that is able to learn from experience, where the experience is generated by a simulator. The goal of the agent is to maximise an output, which in our case is the number of discharged patients (DP) from hospitals or on-site emergency units. We show that by exposing such an intelligent agent to a large sequence of simulated disaster scenarios, we can capture enough experience to enable the agent to make informed decisions.

An Optimal Transmission Strategy for Kalman Filtering Over Packet Dropping Links With Imperfect Acknowledgements

This paper presents a novel design methodology for optimal transmission policies at a smart sensor to remotely estimate the state of a stable linear stochastic dynamical system. The sensor makes measurements of the process and forms estimates of the state using a local Kalman filter. The sensor transmits quantized information over a packet dropping link to the remote receiver. The receiver sends packet receipt acknowledgments back to the sensor via an erroneous feedback communication channel which is itself packet dropping. The key novelty of this formulation is that the smart sensor decides, at each discrete time instant, whether to transmit a quantized version of either its local state estimate or its local innovation. The objective is to design optimal transmission policies in order to minimize a long term average cost function as a convex combination of the receiver's expected estimation error covariance and the energy needed to transmit the packets. The optimal transmission policy is obtained by the use of dynamic programming techniques. Using the concept of submodularity, the optimality of a threshold policy in the case of scalar systems with perfect packet receipt acknowledgments is proved. Suboptimal solutions and their structural results are also discussed. Numerical results are presented illustrating the performance of the optimal and suboptimal transmission policies.

Optimal Energy Allocation for Kalman Filtering Over Packet Dropping Links With Imperfect Acknowledgments and Energy Harvesting Constraints

This paper presents a design methodology for optimal transmission energy allocation at a sensor equipped with energy harvesting technology for remote state estimation of linear stochastic dynamical systems. In this framework, the sensor measurements as noisy versions of the system states are sent to the receiver over a packet dropping communication channel. The packet dropout probabilities of the channel depend on both the sensor's transmission energies and time varying wireless fading channel gains. The sensor has access to an energy harvesting source which is an everlasting but unreliable energy source compared to conventional batteries with fixed energy storages. The receiver performs optimal state estimation with random packet dropouts to minimize the estimation error covariances based on received measurements. The receiver also sends packet receipt acknowledgments to the sensor via an erroneous feedback communication channel which is itself packet dropping. The objective is to design optimal transmission energy allocation at the energy harvesting sensor to minimize either a finite-time horizon sum or a long term average (infinite-time horizon) of the trace of the expected estimation error covariance of the receiver's Kalman filter. These problems are formulated as Markov decision processes with imperfect state information. The optimal transmission energy allocation policies are obtained by the use of dynamic programming techniques. Using the concept of submodularity, the structure of the optimal transmission energy policies are studied. Suboptimal solutions are also discussed which are far less computationally intensive than optimal solutions. Numerical simulation results are presented illustrating the performance of the energy allocation algorithms.

Optimal control laws for lot-sizing and timing of jobs on a single production facility

The optimal control of a single machine processing a certain number of jobs and modeled as a discrete-event dynamic system is considered. The number of jobs and their sequence are fixed, whereas their timing and sizes represent the control variables of the system. The objective function to be optimized is a weighted sum of the quadratic earliness and tardiness of each job, and of the quadratic deviations of job lot sizes and actual machine service speeds from those specified by the production demand and by the regular machine speeds. An optimization problem with quadratic cost function and nonlinear constraints is stated and formalized as a multistage optimal control problem. Necessary conditions to be satisfied by an optimal control sequence are derived. A simpler model is also considered in which the machine speeds are fixed; in this case, the control problem is solved by a procedure making use of dynamic programming techniques. The optimal control laws at each stage are thus obtained.

Private pensions, mortality risk, and the decision to annuitize

This paper examines household decisions about whether or not to annuitize retirement resources. A life-cycle model of consumption, implemented with the use of dynamic programming techniques, is used to construct a utility-based measure of annuity value for individuals and couples in the Health and Retirement Survey. Variation in the calculated ‘annuity equivalent wealth’ arises from differences in mortality risk, marital status, risk aversion, and the presence of pre-existing annuities such as Social Security. I find that a one-percentage point increase in the annuity equivalent wealth leads to nearly a one-percentage point increase in the ex ante probability of annuitizing balances in defined contribution pension plans. However, because much of the variation in the expected annuity decision is left unexplained by the life-cycle model, other factors are also analyzed. Health status and an individual’s time horizon for financial decision making are significant determinants of the decision. There is no evidence that bequest motives are an important factor in making marginal annuity decisions.

A Dynamic Programming Algorithm for the Distance Between Two Finite Areas

The problems of speech recognition and orthographic word correction have been greatly mitigated by the use of dynamic programming techniques for finding the distance between two finite sequences. This paper extends the technique into two dimensions, and presents an algorithm for finding the distance between two finite areas. Applications of the algorithm are suggested.

A Simple Approach to the Valuation of Risky Streams

The optimal investment decision or the capital budgeting problem in a competitive environment are equivalent to the problem of valuing a stream of returns. If the returns from a project can be valued, then the criterion for acceptance is simply whether value exceeds cost. More generally, if returns are net of costs then the criterion for acceptance is whether the value of the net stream is positive.' There are a number of methodologies currently available for determining the value of a stream of returns. If the returns are certain, then there is widespread agreement that value is determined as present discounted value. If returns are uncertain, matters become more complex. The classical method of simply substituting expected returns-or, in its refined version, certainty equivalents-and discounting at some riskadjusted rate has passed from fashion, at least in academic circles if not in business practice. The current view is that the valuation problem calls for the use of dynamic programming techniques coupled with capital asset pricing models. At each stage a capital asset pricing model is In an asset market where there are no unexploited arbitrage opportunities, there will exist a linear valuation operator that can unambiguously price return streams with perfect market substitutes or bound values for streams bounded by market combinations. This is possible, without further assumptions, only if the project retums can be duplicated (or bounded) by a deterministic intertemporal program of purchasing a portfolio of marketed assets. These results are proven and used to simplify and unify a number of topics in financial economics, including project valuation, Modigliani-Miller theory, forward pricing, the closed-end mutual fund paradox, and efficient market theories. * The author is grateful to NSF grant # SOC77-22301 for financial support and to the participants in the seminar at Stanford University and the June 1977 ESSEC Conference in Cergy, France, for helpful comments. 1. This is true under capital budgeting constraints as well; value then includes the shadow prices of rationed resources, but then the statement is simply true as a tautology at this level of generality.

The use of dynamic programming techniques for determining resource allocations among R/D projects: An example

An example of the use of dynamic programming for R/D resource allocation in the aerospace industry is described. Procedures for developing the required input data are described, and a quantitative comparison is made of the effectiveness of resource allocation by dynamic programming vis-a-vis a conventional method. It is shown that dynamic programming can result in significant improvements in program effectiveness.