Value Iteration Method Research Articles

Resilient strategy design for cyber-physical system under DoS attack over a multi-channel framework

This paper is concerned with the design of resilient strategy for a class of cyber-physical system (CPS) under denial-of-service (DoS) attack. The DoS attack occurring in wireless network between sensor and remote estimation is considered. For the addressed CPS, a minimax control strategy is designed in the presence of DoS attack in delta domain under imperfect state measurements (IPSM). Then, a novel multi-channel transmission framework is constructed to reduce the probability of being attacked, where transmitter chooses a single channel to send data packet and meanwhile the DoS attacker decides a target channel to attack. Taking account of the control system performance, a two-player Markov stochastic game is built to model the interactive decision-making of both sides under energy budget constraints. Dynamic programming and value iteration methods are applied for the control strategy and power transmission strategy design, respectively. Numerical simulations are presented to illustrate the validity of the scheme.

An adaptive overload threshold selection process using Markov decision processes of virtual machine in cloud data center

In Cloud Computing (CC), the cost for computation and energy is less by current cloud data centers because it exploits virtualization for an effective resource management. The Virtual Machine (VM) migration authorizes virtualization because it mitigates the difficulties of dynamic workload by repositioning VMs within cloud data centers. Through VM migration many goals of resource management are attained like load balancing, power management, fault tolerance, and system maintenance. The overload threshold is one of the key criterions to determine whether a host is overloaded or not. Achieving desired balance in guaranteeing quality of service, improving resource utilization and degrading energy consumption in data centers is the expected results of any overload threshold selection strategies. But, it is difficult due to the stochastic resource demands of VMs. In this paper, to address this problem, the overload threshold selection is modelled as a Markov decision process. With the solution of the improved Bellman optimality equation by the value iteration method, the optimization model is resolved, and the optimum overload threshold is adaptively selected. The hybrid processes are summarized as the Markov decision processes based adaptive overload threshold selection algorithm. Validations and comparisons are performed to illustrate its effectiveness and efficiency.

Towards a common implementation of reinforcement learning for multiple robotic tasks

Mobile robots are increasingly being employed for performing complex tasks in dynamic environments. Those tasks can be either explicitly programmed by an engineer or learned by means of some automatic learning method, which improves the adaptability of the robot and reduces the effort of setting it up. In this sense, reinforcement learning (RL) methods are recognized as a promising tool for a machine to learn autonomously how to do tasks that are specified in a relatively simple manner. However, the dependency between these methods and the particular task to learn is a well-known problem that has strongly restricted practical implementations in robotics so far. Breaking this barrier would have a significant impact on these and other intelligent systems; in particular, having a core method that requires little tuning effort for being applicable to diverse tasks would boost their autonomy in learning and self-adaptation capabilities. In this paper we present such a practical core implementation of RL, which enables the learning process for multiple robotic tasks with minimal per-task tuning or none. Based on value iteration methods, we introduce a novel approach for action selection, called Q-biased softmax regression (QBIASSR), that takes advantage of the structure of the state space by attending the physical variables involved (e.g., distances to obstacles, robot pose, etc.), thus experienced sets of states accelerate the decision-making process of unexplored or rarely-explored states. Intensive experiments with both real and simulated robots, carried out with the software framework also introduced here, show that our implementation is able to learn different robotic tasks without tuning the learning method. They also suggest that the combination of true online SARSA(λ) (TOSL) with QBIASSR can outperform the existing RL core algorithms in low-dimensional robotic tasks. All of these are promising results towards the possibility of learning much more complex tasks autonomously by a robotic agent.

Large-Rotation-Angle Photogrammetric Resection Based on Least-Squares Homotopy Iteration Method

Inclined imaging is an advanced sensing technique, and has been extensively used. Therefore, large-rotation angle photogrammetric resection has become an important topic. However, traditional iterative methods are limited by requiring good initial values. By contrast, noniterative methods do not require an initial value, although they exhibit relatively low accuracy and robustness. To obtain results with superior precision and universality, this letter proposes an improved approach by modifying the initial value acquisition and iterative methods. This algorithm uses nonlinear iteration to reduce the model error, thereby possibly achieving an exceptional convergence for large-rotation-angle photogrammetric resection. Experimental results on the real data indicate that the proposed algorithm outperforms the previous methods.

Equity-constrained dispatching models for emergency medical services

PurposeThe purpose of this paper is to apply agent-based modeling and simulation concepts in evaluating different approaches to solve ambulance-dispatching decision problems under bounded rationality. The paper investigates the effect of over-responding, i.e. dispatching ambulances even for doubtful high-risk patients, on the performance of equity constrained emergency medical services.Design/methodology/approachAgent-based modeling and simulation was used to evaluate two different dispatching policies: first, a policy based on maximum reward, and second, a policy based on the Markov decision process formulation. Four equity constraints were used: two from the patients’ side and two from the providers’ side.FindingsThe Markov decision process formulation, solved using value iteration method, performed better than the maximum reward method in terms of number of patients served. As the equity constraints conflict with each other, at most three equity constraints could be enforced at a time. The study revealed that it is safe to over-respond if there is uncertainty in the risk level of the patients.Research limitations/implicationsFurther research is required to understand the implications of under-responding, where doubtful high-risk patients are denied an ambulance service.Practical implicationsThe need for good triage system is apparent as over-responding badly affects the operational budget. The model can be used for evaluating various dispatching policy decisions.Social implicationsEmergency medical services have to ensure efficient and equitable provision of services, from the perception of both patients and service providers.Originality/valueThe paper applies agent-based modeling to equity constrained emergency medical services and highlights findings that are not reported in the existing literature.

Integral temporal difference learning for continuous-time linear quadratic regulations

In this paper, we propose a temporal difference (TD) learning method, called integral TD learning that efficiently finds solutions to continuous-time (CT) linear quadratic regulation (LQR) problems in an online fashion where system matrix A is unknown. The idea originates from a computational reinforcement learning method known as TD(0), which is the simplest TD method in a finite Markov decision process. For the proposed integral TD method, we mathematically analyze the positive definiteness of the updated value functions, monotone convergence conditions, and stability properties concerning the locations of the closed-loop poles in terms of the learning rate and the discount factor. The proposed method includes the existing value iteration method for CT LQR problems as a special case. Finally, numerical simulations are carried out to verify the effectiveness of the proposed method and further investigate the aforementioned mathematical properties.

Adaptive Critic Design with Local Gaussian Process Models

In this paper, local Gaussian process (GP) approximation is introduced to build the critic network of adaptive dynamic programming (ADP). The sample data are partitioned into local regions, and for each region, an individual GP model is utilized. The nearest local model is used to predict a given state-action point. With the two-phase value iteration method for a Gaussian-kernel (GK)-based critic network which realizes the update of the hyper-parameters and value functions simultaneously, fast value function approximation can be achieved. Combining this critic network with an actor network, we present a local GK-based ADP approach. Simulations were carried out to demonstrate the feasibility of the proposed approach.

Output-feedback adaptive optimal control of interconnected systems based on robust adaptive dynamic programming

This paper studies the adaptive and optimal output-feedback problem for continuous-time uncertain systems with nonlinear dynamic uncertainties. Data-driven output-feedback control policies are developed by approximate/adaptive dynamic programming (ADP) based on both policy iteration and value iteration methods. The obtained adaptive and optimal output-feedback controllers differ from the existing literature on the ADP in that they are derived from sampled-data systems theory and are guaranteed to be robust to dynamic uncertainties. A small-gain condition is given under which the overall system is globally asymptotically stable at the origin. An application to power systems is given to test the effectiveness of the proposed approaches.

Joint Routing and Admission Control in Wireless Mesh Network

Wireless Mesh Network is a promising technology with many challenges yet to be addressed. Novel and efficient algorithms need to be developed for routing and admission control with the objective to increase the acceptance ratio of new calls without affecting the Quality of Service (QoS) of the existing calls and to maintain the QoS level provided for the mobile calls. In this paper, a novel Markov Decision-based Admission Control and Routing (MDACR) algorithm is proposed. The MDACR algorithm finds a near optimal solution using the value iteration method. To increase the admission rate for both types of calls, a multi-homing admission and routing algorithm for handoff and new calls is proposed. This algorithm associates the user with two different access points which is beneficial in a highly congested network and proposes a new routing metric to assure seamless handoff in the network. Our proposed algorithm outperforms other algorithms in the literature in terms of handoff delay, blocking probability, and number of hard handoff.

Evaluation on ecological restoration capability of revetment in inland restricted channel

Jiangsu Province has the longest inland restricted waterway among all the provinces in China. Ecological revetment is the hot topic of current research on the river or channel regulation engineering in China. Evaluation of ecological restoration capability of revetment is an important part of revetment design as well as an effective basis of the revetment selection. Liudaxian channel of Jiangsu province is chosen for this study, an index system and method for evaluating of ecological restoration capability of revetment is established. By analyzing the mechanism and process of revetment impact on river ecosystems, indicators of ecological restoration capability and quantitative calculation method are proposed in revetment structure level. Based on the degree of influence of the indicator to river ecological function, instead of expert opinion, the set value iteration method was used to analyze the degree of influence of the indicator to river ecosystem, which made weight assignments objectively. Combining with gray relational analysis (GRA) method, the set value iteration based on ecological theory - gray correlation evaluation model was established, which achieved quantitative evaluation of the ecological restoration of revetment. The evaluating result of the projects shows that revetment of wood pile with ladder-sand tube + three-dimensional vegetation net is chosen as the recommended slope protection measure. The revetment ecological restoration evaluation model proposed in this paper with the advantages of objective, standardized and quantitative, offers an effective and feasible tool for designers and managers to evaluate the ecological restoration capability of revetment.

A Mixed Value and Policy Iteration Method for Stochastic Control with Universally Measurable Policies

We consider stochastic optimal control models with Borel spaces and universally measurable policies. For such models the standard policy iteration is known to have difficult measurability issues and cannot be carried out in general. We present a mixed value and policy iteration method that circumvents this difficulty. The method allows the use of stationary policies in computing the optimal cost function in a manner that resembles policy iteration. It can also be used to address similar difficulties of policy iteration in the context of upper and lower semicontinuous models. We analyze the convergence of the method in infinite horizon total cost problems for the discounted case where the one-stage costs are bounded and for the undiscounted case where the one-stage costs are nonpositive or nonnegative. For undiscounted total cost problems with nonnegative one-stage costs, we also give a new convergence theorem for value iteration that shows that value iteration converges whenever it is initialized with a function that is above the optimal cost function and yet bounded by a multiple of the optimal cost function. This condition resembles Whittle’s bridging condition and is partly motivated by it. The theorem is also partly motivated by a result of Maitra and Sudderth that showed that value iteration, when initialized with the constant function zero, could require a transfinite number of iterations to converge. We use the new convergence theorem for value iteration to establish the convergence of our mixed value and policy iteration method for the nonnegative cost case.

Massively parallel motion planning algorithms under uncertainty using POMDP

We present new parallel algorithms that solve continuous-state partially observable Markov decision process (POMDP) problems using the GPU (gPOMDP) and a hybrid of the GPU and CPU (hPOMDP). We choose the Monte Carlo value iteration (MCVI) method as our base algorithm and parallelize this algorithm using the multi-level parallel formulation of MCVI. For each parallel level, we propose efficient algorithms to utilize the massive data parallelism available on modern GPUs. Our GPU-based method uses the two workload distribution techniques, compute/data interleaving and workload balancing, in order to obtain the maximum parallel performance at the highest level. Here we also present a CPU–GPU hybrid method that takes advantage of both CPU and GPU parallelism in order to solve highly complex POMDP planning problems. The CPU is responsible for data preparation, while the GPU performs Monte Cacrlo simulations; these operations are performed concurrently using the compute/data overlap technique between the CPU and GPU. To the best of the authors’ knowledge, our algorithms are the first parallel algorithms that efficiently execute POMDP in a massively parallel fashion utilizing the GPU or a hybrid of the GPU and CPU. Our algorithms outperform the existing CPU-based algorithm by a factor of 75–99 based on the chosen benchmark.

Replenishment and Fulfillment Based Aggregation for General Assemble-to-Order Systems

We consider an assemble-to-order system with multiple products, multiple components which may be demanded in different quantities by different products, batch ordering of components, random lead times, and lost sales. We model the system as an infinite-horizon Markov decision process under the discounted cost criterion. A control policy specifies when a batch of components should be produced (i.e., inventory replenishment) and whether an arriving demand for each product should be satisfied (i.e., inventory allocation). As optimal solutions for such problems are computationally intractable for even moderate sized systems, we approximate the optimal cost function by reducing the state space of the original problem via a novel aggregation technique that uses knowledge of products' component requirements and components' replenishment batch sizes.We establish that a lattice-dependent base-stock and lattice-dependent rationing policy is the optimal inventory replenishment and allocation policy for the aggregate problem under a disaggregation rule that disaggregates each aggregate state into its two extreme original states. This rule drastically reduces the per iteration computational complexity of the value iteration algorithm for the aggregate problem (without sacrificing much accuracy, according to our numerical experiments). We further alleviate the value iteration computational burden by eliminating suboptimal actions based on our optimal policy structure.For systems in which there is a product that has fulfillment priority over all other products at optimality, we are able to derive finite error bound for the cost function of the aggregate problem. With these bounds we show that the value iteration algorithm in the original problem that starts with the aggregate solution converges to the optimal cost function. Numerical experiments indicate that such an algorithm has distinct computational advantage over the standard value iteration method in the original problem.

Approximate modified policy iteration and its application to the game of Tetris

Modified policy iteration (MPI) is a dynamic programming (DP) algorithm that contains the two celebrated policy and value iteration methods. Despite its generality, MPI has not been thoroughly stud...

An Efficient Policy Iteration Algorithm for Dynamic Programming Equations

We present an accelerated algorithm for the solution of static Hamilton--Jacobi--Bellman equations related to optimal control problems. Our scheme is based on a classic policy iteration procedure, which is known to have superlinear convergence in many relevant cases provided the initial guess is sufficiently close to the solution. This limitation often degenerates into a behavior similar to a value iteration method, with an increased computation time. The new scheme circumvents this problem by combining the advantages of both algorithms with an efficient coupling. The method starts with a coarse-mesh value iteration phase and then switches to a fine-mesh policy iteration procedure when a certain error threshold is reached. A delicate point is to determine this threshold in order to avoid cumbersome computations with the value iteration and at the same time to ensure the convergence of the policy iteration method to the optimal solution. We analyze the methods and efficient coupling in a number of examples in different dimensions, illustrating their properties.

A New Method for Computing Eigenpairs of a Finite Element Structure

Based on the weighted inverse topological change method and by introducing a new concept of mass submembers, a dynamical system can be transformed into a static one. Using the properties of the weighted D value, i.e. the weighted D value decreases monotonously with parameter λ increasing; a new method called the weighted D value iteration method is presented for computing the eigenpairs (eigenvalues and eigenvectors). Using this method a series of eigenpairs of a finite element structure can be obtained. It has a merit of simpler algorithm and less computation efforts. Not as the power method, its stability and convergence rate does not depend on the distribution of eigenvalues, and convergent quickly. An example is given to demonstrate the valid of this method.

Distributed Unit Commitment Scheduling in the Future Smart Grid with Intermittent Renewable Energy Resources and Stochastic Power Demands

There is growing interest in renewable energy resources and the smart grid. Since most renewable sources are highly intermittent, they can induce significant fluctuations on the supply side of the power grid. On the other hand, the use of smart meters and smart appliances in the smart grid can cause significant uncertainties on the demand side as well. Therefore, unit commitment scheduling of power generation systems is an important issue in the smart grid to coordinate energy demand and power generation. In this paper, a distributed stochastic scheduling scheme is proposed in the smart grid with intermittent renewable energy resources and stochastic power demand loads. Due to meteorological instability and complex system dynamics, hidden Markov models are used in modeling renewable energy resources. A Markov-modulated Poisson process (MMPP) is used to model the stochastic power demand loads. The scheduling problem is formulated as a distributed stochastic control problem. A value iteration method and a structural results method are presented as possible solutions to this problem. The structural results method is easy to implement, which makes the solution practical for the future smart grid with intermittent renewable energy resources and stochastic power demands. Simulation results are presented to show the effectiveness of the proposed scheme.

An Optimal Policy for a Manufacturer’s Sales Problem Using Markov Decision Model

The success rate of any firm in the industry depends greatly on its sales output. This is achievable only when certain strategies are employed to overcome competition pressures from allied firms. We utilize the tools of Markov decision process in designing our model that improves on the quality of advertisement, research and manpower development to yield an optimal choice leading to profit maximization and cost effectiveness. Finally, we solve our model equations by the value iteration method

Adaptive Channel Recommendation for Opportunistic Spectrum Access

We propose a dynamic spectrum access scheme where secondary users cooperatively recommend "goodâ channels to each other and access accordingly. We formulate the problem as an average reward-based Markov decision process. We show the existence of the optimal stationary spectrum access policy and explore its structure properties in two asymptotic cases. Since the action space of the Markov decision process is continuous, it is difficult to find the optimal policy by simply discretizing the action space and use the policy iteration, value iteration, or Q-learning methods. Instead, we propose a new algorithm based on the model reference adaptive search method and prove its convergence to the optimal policy. Numerical results show that the proposed algorithms achieve up to 18 and 100 percent performance improvement than the static channel recommendation scheme in homogeneous and heterogeneous channel environments, respectively, and is more robust to channel dynamics.

Optimal Stopping Under Partial Observation: Near-Value Iteration

We propose a new approximate value iteration method, namely near-value iteration (NVI), to solve continuous-state optimal stopping problems under partial observation, which in general cannot be solved analytically and also pose a great challenge to numerical solutions. NVI is motivated by the expression of the value function as the supremum over an uncountable set of linear functions in the belief state. After a smart manipulation of the operations in the updating equation for the value function, we reduce the set to only two functions at every time step, so as to achieve significant computational savings. NVI yields a value function approximation bounded by the tightest lower and upper bounds that can be achieved by existing algorithms in the same class, so the NVI approximation is closer to the true value function than at least one of these bounds. We demonstrate the effectiveness of our approach on an example of pricing American options under stochastic volatility.