Hierarchical Markov Decision Process Research Articles

Hamlet: Hierarchical Multi-Objective Reinforcement Learning Method for Charging Power Control of Electric Vehicles With Dynamic Quantity

Electric vehicle (EV) is emerging as an effective choice to reduce carbon emissions in the modern transportation system. However, the large access of EVs brings tremendous passive influence on the stability and performance of power grid operation. Thus, it is valuable to investigate the charging power management method to improve charging experience of EV drivers while maintaining normal bus voltage of the power system. This paper builds the charging power control problem of multiple EVs as a hierarchical Markov decision process (MDP) model and propose a hierarchical multi-objective reinforcement learning method (Hamlet) to obtain the real-time charging power control decisions. Specifically, by treating EVs with the same remaining charging time as the same agent, this paper addresses the dynamic state and action spaces issue and significantly reduce the input dimension of both state and action spaces. Besides, this paper quantifies drivers’ anxiety about the battery volume and distributes the comfort reward over the entire charging interval, which overcomes the delayed comfort reward issue. In order to alleviate the influence of multiple objectives on training stability, this paper develops a multi-objective learning method to dynamically adjust the optimization direction. Simulation results on IEEE 33 bus and 69 bus test feeders prove the validity of the proposed method in mitigating voltage fluctuation, minimizing bill payment and maximizing battery comfort.

Two-Timescale Learning-Based Task Offloading for Remote IoT in Integrated Satellite–Terrestrial Networks

In this paper, we propose an integrated satellite-terrestrial network (ISTN) architecture to support delay-sensitive task offloading for remote Internet of Things (IoT), in which satellite networks serve as a complement to terrestrial networks by providing additional communication resources, backhaul capacities, and seamless coverage. Under this architecture, we investigate how to jointly make offloading link selection and bandwidth allocation decisions for BSs and IoT users. Considering the differentiated decision-making time granularities, we formulate a two-timescale stochastic optimization problem to minimize the overall task offloading delay. To accommodate the two-timescale network dynamics and characterize state-action relations, we establish a hierarchical Markov decision process (H-MDP) framework with two separate agents tackling two-timescale network management decisions, and two evolved MDP-based subproblems are formulated accordingly. To efficiently solve the subproblems, we further develop a hybrid proximal policy optimization (H-PPO)-based algorithm. Specifically, a hybrid actor-critic architecture is designed to deal with the mixed discrete and continuous actions. In addition, an action mask layer and an action shaping function are designed to sample feasible task offloading decisions from the time-variant action set. Extensive simulation results have validated the superiority of the proposed ISTN architecture and the H-PPO-based algorithm, especially in scenarios with scarce spectrum resources and heavy traffic loads.

Adaptive Multi-scale Prognostics and Health Management for Smart Manufacturing Systems

The Adaptive Multi-scale Prognostics and Health Management (AM-PHM) is a methodology designed to enable PHM in smart manufacturing systems. In application, PHM information is not yet fully utilized in higher-level decision-making in manufacturing systems. AM-PHM leverages and integrates lower-level PHM information such as from a machine or component with hierarchical relationships across the component, machine, work cell, and assembly line levels in a manufacturing system. The AM-PHM methodology enables the creation of actionable prognostic and diagnostic intelligence up and down the manufacturing process hierarchy. Decisions are then made with the knowledge of the current and projected health state of the system at decision points along the nodes of the hierarchical structure. To overcome the issue of exponential explosion of complexity associated with describing a large manufacturing system, the AM-PHM methodology takes a hierarchical Markov Decision Process (MDP) approach into describing the system and solving for an optimized policy. A description of the AM-PHM methodology is followed by a simulated industry-inspired example to demonstrate the effectiveness of AM-PHM.

Optimizing pig marketing decisions under price fluctuations

In the manufacturing of fattening pigs, pig marketing refers to a sequence of culling decisions until the production unit is empty. The profit of a production unit is highly dependent on the price of pork, the cost of feeding and the cost of buying piglets. Price fluctuations in the market consequently influence the profit, and the optimal marketing decisions may change under different price conditions. Most studies have considered pig marketing under constant price conditions. However, because price fluctuations have an influence on profit and optimal marketing decisions it is relevant to consider pig marketing under price fluctuations. In this paper we formulate a hierarchical Markov decision process with two levels which model sequential marketing decisions under price fluctuations in a pig pen. The state of the system is based on information about pork, piglet and feed prices. Moreover, the information is updated using a Bayesian approach and embedded into the hierarchical Markov decision process. The optimal policy is analyzed under different patterns of price fluctuations. We also assess the value of including price information into the model.

A hierarchical Markov decision process modeling feeding and marketing decisions of growing pigs

Feeding is the most important cost in the production of growing pigs and has a direct impact on the marketing decisions, growth and the final quality of the meat. In this paper, we address the sequential decision problem of when to change the feed-mix within a finisher pig pen and when to pick pigs for marketing. We formulate a hierarchical Markov decision process with three levels representing the decision process. The model considers decisions related to feeding and marketing and finds the optimal decision given the current state of the pen. The state of the system is based on information from on-line repeated measurements of pig weights and feeding and is updated using a Bayesian approach. Numerical examples are given to illustrate the features of the proposed optimization model.

Hierarchical Markov Decision Based Path Planning for Palletizing Robot

On account of the complex application environment and the large number of uncertain conditions for the palletizing robot, we do path-planning for the multiple joints robot by the algorithm based on Hierarchical Markov Decision Process. First, according to the actual working environment, we set the range of the robot’s motion and select the conventional movement combination as the basic set of the robot’s behaviors. We can get the possible reward of various situations. We divide the state space in accordance with the location information of the obstacle space into a small number of state clusters, sub-level step by step to determine the precise trajectory of palletizing robots. We simulate 3D robot motion trajectory, including barrier-free and spherical obstacle conditions. Finally, experimental verification is carried out, the algorithm has been proved to control the compatible movements of each joint effectively and keep the error within the allowed range. The experiment results meet the requirement well. DOI : http://dx.doi.org/10.11591/telkomnika.v12i3.3851 Full Text: PDF

A scalable method for DCLC problem using hierarchical MDP model

It is well known that the delay-constrained least-cost (DCLC) routing problem is NP-complete, hence various heuristic methods have been proposed for this problem. However, these heuristic methods have poor scalability as the network scale increases. In this paper we propose a new method based on the Markov Decision Process (MDP) theory and the hierarchical routing scheme to address the scalability issue of the DCLC routing problem. We construct a new two-level hierarchy MDP model and apply an infinite-horizon discounted cost model to the upper level for the end-to-end inter-domain link selection. Since the infinite-horizon discounted cost model is independent of network scale, the scalability problem is resolved. With the proposed model, we further give the algorithm of solving the optimal policy to obtain the DCLC routing. Simulation results show that the proposed method improves the scalability significantly.

Unifying temporal and organizational scales in multiscale decision-making

In enterprise systems, making decisions is a complex task for agents at all levels of the organizational hierarchy. To calculate an optimal course of action, an agent has to include uncertainties and the anticipated decisions of other agents, recognizing that they also engage in a stochastic, game-theoretic reasoning process. Furthermore, higher-level agents seek to align the interests of their subordinates by providing incentives. Incentive-giving and receiving agents need to include the effect of the incentive on their payoffs in the optimal strategy calculations. In this paper, we present a multiscale decision-making model that accounts for uncertainties and organizational interdependencies over time. Multiscale decision-making combines stochastic games with hierarchical Markov decision processes to model and solve multi-organizational-scale and multi-time-scale problems. This is the first model that unifies the organizational and temporal scales and can solve a 3-agent, 3-period problem. Solutions can be derived as analytic equations with low computational effort. We apply the model to a service enterprise challenge that illustrates the applicability and relevance of the model. This paper makes an important contribution to the foundation of multiscale decision theory and represents a key step towards solving the general X-agent, T-period problem.

Optimal replacement policies and economic value of clinical observations in sow herds

Even though several sow replacement models have been published, integrating information about the health status of sows has not yet been handled satisfactorily. This paper presents a framework for integrating a Weak Sow Index (WSI) into an existing sow replacement model. The WSI, which is developed as part of this study, quantifies various clinical signs into one numerical value representing the risk of a sow to be involuntarily culled. The objective of the study is to investigate the effect of observing clinical signs of sows on the optimal replacement policy. A second objective is to estimate the economic value of observing clinical signs of individual sows. Bayesian networks are used to develop the WSI models for lactating and pregnant sows, respectively. The optimization of the replacement policy is done in a multi-level hierarchical Markov decision process. To illustrate the behaviour of the model, the effect of the WSI on the replacement policy, and the economic benefit of observing clinical signs of individual sows are investigated in two fictitious herds with a high and a low risk of involuntary culling of sows. In general, the value of the WSI has a high influence on the optimal replacement policy, allowing for a better economic classification of sows when taking information about the health status into account. It is shown that the economic value of the WSI is higher in a high risk herd compared to a low risk herd. Among the individual clinical signs, “unwillingness to stand” made the lowest contribution to the economic value of the WSI. The highest contribution was made by the clinical sign “vulva bite” in pregnant sows.

Optimal replacement policies for dairy cows based on daily yield measurements

Markov decision processes (MDP) with finite state and action space have often been used to model sequential decision making over time in dairy herds. However, the length of each stage has been at least 1 mo, resulting in models that do not support decisions on a daily basis. The present paper describes the first step of developing an MDP model that can be integrated into a modern herd management system. A hierarchical MDP was formulated for the dairy cow replacement problem with stage lengths of 1 d. It can be used to assist the farmer in replacement decisions on a daily basis and is based on daily milk yield measurements that are available in modern milking systems. Bayesian updating was used to predict the performance of each cow in the herd and economic decisions were based on the prediction. Moreover, parameters in the model were estimated using data records of the specific herd under consideration. This includes herd-specific lactation curves.

Energy Efficient Multiple Target Tracking in Wireless Sensor Networks

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> Energy awareness is a crucial component in the design of wireless sensor networks at all layers. This paper looks into efficient energy utilization of a target-tracking sensor network by predicting a target's trajectory through experience. While this is not new, the chief novelty comes in conserving energy through both dynamic spatial and temporal management of sensors while assuming minimal locality information. We adapted our target trajectory model from the Gauss–Markov mobility model, formulated the tracking problem as a hierarchical Markov decision process (HMDP), and solved it through neurodynamic programming. Our HMDP for target-tracking (HMTT) algorithm conserves energy by reducing the rate of sensing (temporal management) but maintains an acceptable tracking accuracy through trajectory prediction (spatial management) of multiple targets. We derived some theoretical bounds on accuracy and energy utilization of HMTT. Simulation results demonstrated the effectiveness of HMTT in energy conservation and tracking accuracy against two other predictive tracking algorithms, with accuracy of up to 47% higher and energy savings of up to 200%. </para>

Actor-critic algorithms for hierarchical Markov decision processes

We consider the problem of control of hierarchical Markov decision processes and develop a simulation based two-timescale actor-critic algorithm in a general framework. We also develop certain approximation algorithms that require less computation and satisfy a performance bound. One of the approximation algorithms is a three-timescale actor-critic algorithm while the other is a two-timescale algorithm, however, which operates in two separate stages. All our algorithms recursively update randomized policies using the simultaneous perturbation stochastic approximation (SPSA) methodology. We briefly present the convergence analysis of our algorithms. We then present numerical experiments on a problem of production planning in semiconductor fabs on which we compare the performance of all algorithms together with policy iteration. Algorithms based on certain Hadamard matrix based deterministic perturbations are found to show the best results.