Dynamic Markov Decision Research Articles

Energy and comfort aware operation of multi-zone HVAC system through preference-inspired deep reinforcement learning

This paper proposes a novel optimal control method for multi-zone HVAC systems to enhance energy efficiency and improve occupant comfort. To address the disturbances in outdoor weather and indoor loads, the proposed method formulates the HVAC control problem as a dynamic Markov decision process, and employs deep reinforcement learning (DRL) techniques to obtain the optimal control policy. To manage multiple goals among thermal comfort, indoor air quality (IAQ) and system energy efficiency, a preference-inspired (P-ins) mechanism is designed to achieve the optimal balance among different objectives. The P-ins mechanism effectively guides the agent towards the optimal control policy with high convergence rate. The proposed method has been validated through EnergyPlus-Python co-simulation testbed with real-world data traces, and assessed by an overall evaluation indicator. Results demonstrate that the proposed method can reduce energy consumption without compromising thermal comfort and IAQ. Specifically, the occurrence of temperature violations is reduced below 0.8 %, and a maximum energy saving of 9.41 % can be achieved, compared with traditional methods.

Deep clustering of reinforcement learning based on the bang-bang principle to optimize the energy in multi-boiler for intelligent buildings

The bang-bang relays of the multiple-boiler system (MBS) control, are characterized by complex limiter saturation functions and classified as fixed parameters. Their action signals cannot precisely control the nonlinear dynamic building heating demand over their entire range of operation. Moreover, in a mono-boiler system, the bang-bang controller endures increasing short cycling over partial load time due to the heating system being considered to have an oversized boiler at most times of running, thus promoting high energy consumption and fluctuating indoor thermal comfort. So, it is difficult to cope with uncertainties in outdoor environments and indoor heating load. Hence, this study formulates the MBS control problem as a dynamic Markov decision process and applies a deep clustering of reinforcement learning approach to obtain the optimal control policy through interaction with the environment based on multi-agent learning according to bang-bang action. With such an approach, adopting a new boiler sequencing control (BSC) strategy using deep clustering of reinforcement learning based on a bang-bang (DCRLBB) manner. The deep clustering is configured to break Lagrangian trajectory curves into piecewise segments to represent the RL agent's action policy. The agent's action policy signals are configured from the bang-bang reward formula based on trade-off implications to be more adjustable than traditional fixed parameters such as fuzzy bang-bang controller (FBBC). The agent of BSC significantly affects the energy performance of the MBS, whereas the other agent resizes boiler capacity by acting to adjust the boiler solenoid fuel valve. The comparison of results between the proposed strategy and conventional FBBC shows distinct differences in the superior response of DCRLBB under dynamic indoor/outdoor actual conditions and energy saving by more than 32% while maintaining the indoor thermal in the comfortable range.

Smart Elderly Care: An Intelligent e-Procurement System for Elderly Supplier Selecting

(1) Objective: to accelerate the digitalization of the elderly care service industry and the construction of the smart elderly care industry, this paper designs an intelligent e-procurement system for elderly suppliers selecting from the perspective of smart elderly care, which can enhance the efficiency of elderly care supply chains and assist manufacturers of elderly products in choosing a reliable, high-quality supplier during trades. (2) Methods: the e-procurement system, including six modules, is built with an improved dynamic Markov Decision Process selection model combined with an Analytic Network Process, bringing dynamic evolution of both inventory cost and purchasing cost into long-term reward calculation, and taking into account 15 common indexes and 7 specific indexes when evaluating suppliers’ competitiveness. (3) Results: a real sample shows that when facing 50 suppliers with 50 different quotations, the e-procurement system selects a stable and reliable supplier that brings the best long-term profits for demand enterprises in ten purchase periods, and it makes the selecting process more efficient and more prompt. (4) Conclusions: the model can be used in the circumstance where an elderly product producer is forced to decide on a long-term strategy or reselect a new stable supplier since it is focused on choosing long-term and high-quality suppliers over numerous periods.

AoI-aware energy control and computation offloading for industrial IoT

In Industrial Internet of Things (IIoT), a large volume of data is collected periodically by IoT devices, and timely data routing and processing are important requirements. Age of Information (AoI), which is a metric to evaluate the freshness of status information in data processing, has become one of the most important objectives in IIoT. In this paper, considering limited communication, computation and energy resources on IoT devices, we jointly study the optimal AoI-aware energy control and computation offloading problem within a dynamic IIoT scenario with multiple IoT devices and multiple edge servers. Based on extensive analysis of real-life IoT dataset, Markovian queueing models are constructed to capture the dynamics of IoT devices and edge servers, and their corresponding analyses are provided. With the quantitative analytical results, we formulate a dynamic Markov decision problem with the objective of minimizing the long-term energy consumption while satisfying AoI constraints for real-time data processing. To solve the problem, we apply Deep Reinforcement Learning (DRL) techniques for adapting to large-scale dynamic IIoT environments, and design an intelligent Energy Control and Computation Offloading (ECCO) algorithm. Extensive simulation experiments are conducted based on real-world dataset, and the comparison results illustrate the superiority of our ECCO algorithm over both existing DRL and non-DRL algorithms.

Stochastic resource scheduling via bilayer dynamic Markov decision process in mobile cloud networks

In mobile cloud networks, the efficiency and the quality of service are mainly affected by the service delay, the drop rate of service requests and CPU utilization. For these reasons, we propose a real-time stochastic resource scheduling framework (R-SRSF) to realize pre-allocation of resources by bilayer dynamic markov decision processes (DMDPs). In R-SRSF, we divide the resource scheduling into two parts, the virtual machine (VM) basic configuration resources management and VM switching management. To this end, we firstly design a strategy transformation management algorithm (STMA) to control the VM basic configuration resources. Secondly, we design a scheme-changed drift-plus-penalty algorithm (SDPPA) to execute VM switching management. We also prove the convergence of SDPPA based on Lyapunov optimization theory. For better VM scheduling, we propose a stochastic dynamic policy adjustment algorithm to cooperate with SDPPA. The experimental results show that R-SRSF can reduce the service delay, expenditure and the drop rate of service requests. Moreover, R-SRSF also has a high CPU utilization and stability.

Assigning multiple job types to parallel specialized servers

In this paper methods of mixing decision rules are investigated and applied to the so-called multiple job type assignment problem with specialized servers. This problem is modeled as continuous time Markov decision process. For this assignment problem performance optimization is in general considered to be difficult. Moreover, for optimal dynamic Markov decision policies the corresponding decision rules have in general a complicated structure not facilitating a smooth implementation. On the other hand optimization over the subclass of so-called static policies is known to be tractable. In the current paper a suitable static decision rule is mixed with dynamic decision rules which are selected such that these rules are relatively easy to describe and implement. Some mixing methods are discussed and optimization is performed over corresponding classes of so-called mixing policies. These mixing policies maintain the property that they are easy to describe and implement compared to overall optimal dynamic Markov decision policies. Besides for all investigated instances the optimized mixing policies perform substantially better than optimal static policies.

A Framework for Dynamic Context-Centric Commander Decision Support

This paper addresses the fundamental research question: “How can we determine the sequential decision- making process inside a decision maker’s mind?” We construct a dynamic Markov Decision Process using a Double Transition Model (DTM). The DTM is a cognitive model decomposing the decision-making process into episodic tasks that are extracted from a stream of incoming information. In a DTM, each state reflects a stage en route to a decision, and each action reflects a possible move from collecting data to hypothesizing and inferencing. The reward reflects how close a stage is to the final decision. We demonstrate this process through a proof-of-concept DTM using a hypothetical scenario for Typhoon Haiyan in the Philippines (2013). The DTM constructed from this scenario enables a Commander to reason about damaged areas, death tolls, and assistance methods while allowing his actions to be captured and used to explain why and how each decision is made.

Dynamic Markov Decision Policies for Delay Constrained Wireless Scheduling

We consider a one-hop wireless system with a small number of delay constrained users and a larger number of users without delay constraints. We develop a scheduling algorithm that reacts to time varying channels and maximizes throughput utility (to within a desired proximity), stabilizes all queues, and satisfies the delay constraints. The problem is solved by reducing the constrained optimization to a set of weighted stochastic shortest path problems, which act as natural generalizations of max-weight policies to Markov decision networks. We also present approximation results for the corresponding shortest path problems, and discuss the additional complexity and delay incurred as compared to systems without delay constraints. The solution technique is general and applies to other constrained stochastic decision problems.

Modeling prawn production management system: A dynamic Markov decision approach

This paper describes an attempt to develop a computer-operational bio-economic model for assessing the economics of alternative pond management and marketing strategies for a freshwater prawn production system in Hawaii. The model structure incorporates a biological component, which is represented by a simple Markov process, into a dynamic program for economic analysis. The Markov transition probability matrices are estimated based on 4 months of production data from a commercial facility on Oahu. Price and cost data from the same 4-month period are used as a baseline in the dynamic program. Different price and cost assumptions are used to test the sensitivity of the baseline results. An assessment of the present model structure, its associated data requirements, and results are discussed.