semi-Markov Decision Process Model Research Articles

A smart inventory management system with medication demand dependencies in a hospital supply chain: A multi-agent reinforcement learning approach

In light of the intense need for quality health and well-being, the healthcare industry must improve its operations by strengthening its service delivery and reducing overall costs. A smart inventory management system for managing medicines in a hospital supply chain (HSC) is one of the best solutions as its advantages ensure availability of affordable medicines as well as inventory cost reduction. Significant effort is made on inventory management from literature; however, smart inventory systems for HSC considering the various complexities and uniqueness of healthcare systems are limited. This paper attempts to fill this gap by developing a stochastic semi-Markov decision process model which is solved by a multi-agent reinforcement learning method. Data collected from a multispecialty hospital in India’s eastern region serves to validate the proposed model. The results present the optimal order quantities and optimal inventory control policy for the HSC considering the uncertain medication demand at the hospital pharmacy and multiple point-of-care units, and demand dependencies among medicines prescribed to the patients admitted in the hospital. The study also has significant managerial implications for enhancing HSC’s inventory performance, improving the healthcare service delivery, reducing healthcare cost, thereby ensuring the affordable, accessible, and quality healthcare to the society.

A heuristically accelerated reinforcement learning method for maintenance policy of an assembly line

<p style='text-indent:20px;'>This paper aims to investigate the maintenance policy for a two-machine one-buffer (2M1B) assembly line system. We assume that the observed quality states of the deteriorating machines in the system are characterized by multiple decreasing yield stages. A semi-Markov decision process (SMDP) model is used for describing the deteriorating process of the system. A heuristically accelerated multi-agent reinforcement learning (HAMRL) method is conducted to solve the problem model. The asynchronous updating rules are introduced in the HAMRL method, and the production time, preventive maintenance (PM) time and corrective repair (CR) time are random, and the deterioration mode of the device is not fixed. Meanwhile, a comparison with a simulated annealing search (SAS) based exploration algorithm and a neighborhood search (NS) based exploration algorithm in reinforcement learning (RL) is presented. The empirical results indicate that the proposed HAMRL algorithm can speed up the learning process, and has a certain advantage for the larger space and the more practical problem. And the maintenance strategy for the 2M1B assembly line system is obtained under the condition of convergent system average cost rate. This paper provides new and practical insights into the application and selection of techniques for maintenance policy of the 2M1B assembly line system.</p>

Predictive Maintenance and Sensitivity Analysis for Equipment with Multiple Quality States

This paper discusses the predictive maintenance (PM) problem of a single equipment system. It is assumed that the equipment has deteriorating quality states as it operates, resulting in multiple yield levels represented as system observation states. We cast the equipment deterioration as discrete-state and continuous-time semi-Markov decision process (SMDP) model and solve the SMDP problem in reinforcement learning (RL) framework using the strategy-based method. In doing so, the goal is to maximize the system average reward rate (SARR) and generate the optimal maintenance strategy for given observation states. Further, the PM time is capable of being produced by a simulation method. In order to prove the advantage of our proposed method, we introduce the standard sequential preventive maintenance algorithm with unequal time interval. Our proposed method is compared with the sequential preventive maintenance algorithm in a test objective of SARR, and the results tell us that our proposed method can outperform the sequential preventive maintenance algorithm. In the end, the sensitivity analysis of some parameters on the PM time is given.

Efficient Task Offloading for 802.11p-Based Cloud-Aware Mobile Fog Computing System in Vehicular Networks

Various emerging vehicular applications such as autonomous driving and safety early warning are used to improve the traffic safety and ensure passenger comfort. The completion of these applications necessitates significant computational resources to perform enormous latency-sensitive/nonlatency-sensitive and computation-intensive tasks. It is hard for vehicles to satisfy the computation requirements of these applications due to the limit computational capability of the on-board computer. To solve the problem, many works have proposed some efficient task offloading schemes in computing paradigms such as mobile fog computing (MFC) for the vehicular network. In the MFC, vehicles adopt the IEEE 802.11p protocol to transmit tasks. According to the IEEE 802.11p, tasks can be divided into high priority and low priority according to the delay requirements. However, no existing task offloading work takes into account the different priorities of tasks transmitted by different access categories (ACs) of IEEE 802.11p. In this paper, we propose an efficient task offloading strategy to maximize the long-term expected system reward in terms of reducing the executing time of tasks. Specifically, we jointly consider the impact of priorities of tasks transmitted by different ACs, mobility of vehicles, and the arrival/departure of computing tasks, and then transform the offloading problem into a semi-Markov decision process (SMDP) model. Afterwards, we adopt the relative value iterative algorithm to solve the SMDP model to find the optimal task offloading strategy. Finally, we evaluate the performance of the proposed scheme by extensive experiments. Numerical results indicate that the proposed offloading strategy performs well compared to the greedy algorithm.

SMDP-based sleep policy for base stations in heterogeneous cellular networks

A dense heterogeneous cellular network can effectively increase the system capacity and enhance the network coverage. It is a key technology for the new generation of the mobile communication system. The dense deployment of small base stations not only improves the quality of network service, but also brings about a significant increase in network energy consumption. This paper mainly studies the energy efficiency optimization of the Macro-Femto heterogeneous cellular network. Considering the dynamic random changes of the access users in the network, the sleep process of the Femto Base Stations (FBSs) is modeled as a Semi-Markov Decision Process (SMDP) model in order to save the network energy consumption. And further, this paper gives the dynamic sleep algorithm of the FBS based on the value iteration. The simulation results show that the proposed SMDP-based adaptive sleep strategy of the FBS can effectively reduce the network energy consumption.

Collision avoidance for an unmanned surface vehicle using deep reinforcement learning

In this paper, a deep reinforcement learning (DRL)-based collision avoidance method is proposed for an unmanned surface vehicle (USV). This approach is applicable to the decision-making stage of collision avoidance, which determines whether the avoidance is necessary, and if so, determines the direction of the avoidance maneuver. To utilize the visual recognition capability of deep neural networks as a tool for analyzing the complex and ambiguous situations that are typically encountered, a grid map representation of the ship encounter situation was suggested. For the composition of the DRL network, we proposed a neural network architecture and semi-Markov decision process model that was specially designed for the USV collision avoidance problem. The proposed DRL network was trained through repeated simulations of collision avoidance. After the training process, the DRL network was implemented in collision avoidance experiments and simulations to evaluate its situation recognition and collision avoidance capability.

HDec-POSMDPs MRS Exploration and Fire Searching Based on IoT Cloud Robotics

The multi-robot systems (MRS) exploration and fire searching problem is an important application of mobile robots which require massive computation capability that exceeds the ability of traditional MRS’s. This paper propose a cloud-based hybrid decentralized partially observable semi-Markov decision process (HDec-POSMDPs) model. The proposed model is implemented for MRS exploration and fire searching application based on the Internet of things (IoT) cloud robotics framework. In this implementation the heavy and expensive computational tasks are offloaded to the cloud servers. The proposed model achieves a significant improvement in the computation burden of the whole task relative to a traditional MRS. The proposed model is applied to explore and search for fire objects in an unknown environment; using different sets of robots sizes. The preliminary evaluation of this implementation demonstrates that as the parallelism of computational instances increase the delay of new actuation commands which will be decreased, the mean time of task completion is decreased, the number of turns in the path from the start pose cells to the target cells is minimized and the energy consumption for each robot is reduced.

Automatic EEG classification: a path to smart and connected sleep interventions

We develop a random forest classifier to automatically classify brain waves into sleep stages by using the publicly available data from PhysioBank. More specifically, we use the EEG signals from a single pair of electrodes (FPz–Cz) recorded from 20 patients and evaluate the impact of data balancing and incorporating signal history on classification results. The accuracy of the model is objectively evaluated using leave-one-out cross-validation. The developed model achieves the mean accuracy of 0.74, with that of the individual sleep stages ranging from 0.65 to 0.91. Next, we leverage this online sleep scoring scheme to introduce dynamic interventions as sleep process evolves over night. We develop a semi-Markov decision process model to determine optimal intervention policies to minimize the gap between the amount of sleep experienced in different stages and predetermined targets and provide computational results.

Dynamic Abandon/Extract Decisions for Failed Cardiac Leads

When a cardiac lead fails, physicians implant a new lead and may opt to extract the failed lead and/or any previously abandoned leads. Because the risk of extraction increases in lead age, physicians may extract leads to reduce the future risk of mandatory extraction, due to either infection or limited space in the vein. We develop discrete-time semi-Markov decision process models for various types of cardiac devices to determine patient-specific, lifetime-maximizing extraction policies as a function of patient age and the age of every implanted lead. We use clinical data to calibrate these models and present insightful numerical results, including comparisons to policies commonly used in practice. Our numerical experiments suggest that extracting failed leads only when forced to because of space limitations is usually a good rule of thumb, but that following the optimal policy, as opposed to the commonly used heuristic policies, can extend an average patient’s expected lifetime by up to 1.2 years and decrease the likelihood of device-related death by up to 94% in some cases. The online appendix is available at https://doi.org/10.1287/mnsc.2016.2621 . This paper was accepted by Noah Gans, stochastic models and simulation.

Theoretical and empirical studies on essence-based adaptive software engineering

The OMG Essence standard was published as the kernel for software engineering methods in 2014. We show that the Essence view of software engineering is reminiscent of a nondeterministic, multidimensional finite state machine, and that the Essence lends support to a semi-Markov decision process model of software engineering which, in practice, facilitates a goal-driven adaptive software engineering. We develop an activity-state mapping algorithm and a goal-activity cover algorithm based on the Essence, which can help automate the health monitoring of project states and the adaptive planning of project activities in a software engineering project. Practical applications of the model and algorithms are illustrated. Benefits to software practitioners of the proposed approach are measured through statistical experiments.

QoS-Aware Energy-Efficient Joint Radio Resource Management in Multi-RAT Heterogeneous Networks

A heterogeneous wireless network (HetNet), which combines multiple cooperating radio access technologies in an overlapping structure, is a communication system that has been recognized as an efficient way to meet the increasing traffic demand in broadband wireless networks. In this paper, we exploit the network cooperation in HetNets to propose two joint radio resource management (JRRM) schemes that improve energy savings while satisfying the system quality-of-service (QoS) performance requirements. First, we present an optimal QoS-aware energy-efficient JRRM scheme, which is formulated as a semi-Markov decision process model, and provide the optimal control policy for the HetNet under analysis. Second, we present an implementation-friendly QoS-aware energy-efficient JRRM scheme that utilizes a threshold on the macrocell radio resource occupancy to trigger the switching-on/off procedure of the base transceiver station resources, as well as a load-balancing procedure to minimize the service disruptions that may occur because of radio resource shortage and to reduce power consumption during the HetNet operation. This JRRM scheme is analyzed by means of a multidimensional continuous-time Markov chain model. Third, we devise an algorithm to determine the threshold setting in the implementation-friendly JRRM scheme according to a desirable power saving level that is prespecified by the mobile network operator. Numerical results show that the proposed schemes achieve substantial energy savings while keeping satisfactory performance levels.

A stochastic model of acute-care decisions based on patient and provider heterogeneity

The primary cause of preventable death in many hospitals is the failure to recognize and/or rescue patients from acute physiologic deterioration (APD). APD affects all hospitalized patients, potentially causing cardiac arrest and death. Identifying APD is difficult, and response timing is critical - delays in response represent a significant and modifiable patient safety issue. Hospitals have instituted rapid response systems or teams (RRT) to provide timely critical care for APD, with thresholds that trigger the involvement of critical care expertise. The National Early Warning Score (NEWS) was developed to define these thresholds. However, current triggers are inconsistent and ignore patient-specific factors. Further, acute care is delivered by providers with different clinical experience, resulting in quality-of-care variation. This article documents a semi-Markov decision process model of APD that incorporates patient and provider heterogeneity. The model allows for stochastically changing health states, while determining patient subpopulation-specific RRT-activation thresholds. The objective function minimizes the total time associated with patient deterioration and stabilization; and the relative values of nursing and RRT times can be modified. A case study from January 2011 to December 2012 identified six subpopulations. RRT activation was optimal for patients in “slightly concerning” health states (NEWS > 0) for all subpopulations, except surgical patients with low risk of deterioration for whom RRT was activated in “concerning” states (NEWS > 4). Clustering methods identified provider clusters considering RRT-activation preferences and estimation of stabilization-related resource needs. Providers with conservative resource estimates preferred waiting over activating RRT. This study provides simple practical rules for personalized acute care delivery.

A reinforcement-learning approach for admission control in distributed network service systems

In the distributed network service systems such as streaming-media systems and resource-sharing systems with multiple service nodes, admission control (AC) technology is an essential way to enhance performance. Model-based optimization approaches are good ways to be applied to analyze and solve the optimal AC policy. However, due to "the curse of dimensionality", computing such policy for practical systems is a rather difficult task. In this paper, we consider a general model of the distributed network service systems, and address the problem of designing an optimal AC policy. An analytical model is presented for the system with fixed parameters based on semi-Markov decision process (SMDP). We design an event-driven AC policy, and the stationary randomized policy is taken as the policy structure. To solve the SMDP, both the state aggregation approach and the reinforcement-learning (RL) method with online policy optimization algorithm are applied. Then, we extend the problem by considering the system with time-varying parameters, where the arrival rates of requests at each service node may change over time. In view of this situation, an AC policy switching mechanism is presented. This mechanism allows the system to decide whether to adjust its AC policy according to the policy switching rule. And in order to maximize the gain of system, that is, to obtain the optimal AC policy switching rule, another RL-based algorithm is applied. To assess the effectiveness of SMDP-based AC policy and policy switching mechanism for the system, numerical experiments are presented. We compare the performance of optimal policies obtained by the solutions of proposed methods with other classical AC policies. The simulation results illustrate that higher performance and computational efficiency could be achieved by using the SMDP model and RL-based algorithms proposed in this paper.

An SMDP model for a multiclass multi-server queueing control problem considering conversion times

We address a queueing control problem considering service times and conversion times following normal distributions. We formulate the multi-server queueing control problem by constructing a semi-Markov decision process (SMDP) model. The mechanism of state transitions is developed through mathematical derivation of the transition probabilities and transition times. We also study the property of the queueing control system and show that optimizing the objective function of the addressed queueing control problem is equivalent to maximizing the time-average reward.

Trajectory Optimization of Packet Ferries in Opportunistic Social-Based Machine-to-Machine Networks

Machine-to-machine (M2M) communication is an emerging paradigm to connect a large number of devices through wireless technologies. In recent application scenarios, devices are normally carried by people in social contexts. Like the behavior of people, the moving activity of devices always happens in a specific area, which corresponds to some specific communities. How to guarantee the higher packet delivery ratio while reducing forwarding delay in such M2M networks is a challenging issue and has not been well addressed yet. In this paper, we put some special mobile devices (called postmen) in the opportunistic social-based M2M networks and their main responsibility is to carry packets for normal devices which belong to specific communities. Our work focuses on the optimization of the moving trajectory by considering the minimum transmission delay. We formulate the optimal issue into semi-Markov Decision Process model. The decision process includes two parts: packets-choosing strategy and trajectory of packet-ferrying determination strategy. By maximizing the individual reward of a postman, its optimal trajectory will be found. Furthermore, the proposed solution guarantees the packet delivery ratio and delay for isolated communities. Simulation results show that the proposed packet-ferrying solution outperforms the two existing ferrying solutions in terms of packet delivery ratio.

Continuous-time MAXQ Algorithm for Web Service Composition

Web services composition present a technology to compose complex service applications from individual (atomic) services, that is, through web services composition, distributed applications and enterprise business processes can be integrated by individual service components developed independently. In this paper, we concentrate on the optimization problems of dynamic web service composition, and our goal is to find an optimal composite policy. Different from many traditional composite methods that do not scale to large continuous-time processes, we introduce a hierarchical reinforcement learning technique, i.e., a continuous-time unified MAXQ algorithm, to solve large-scale web service composition problems in the context of continuous-time semi-Markov decision process (SMDP) model under either average- or discounted-cost criteria. The proposed algorithm can avoid the “curse of modeling” and the “curse of dimensionality” existing in the optimization process. Finally, we use a travel reservation as an example to illustrate the high effectiveness of the proposed algorithm, and the simulation results show that, it has better optimization performance and faster learning speed than the flat Q-learning.

Construction of Semi-Markov Decision Process Models of Continuous State Space Environments Using Growing Cell Structures and Multiagentk-Certainty Exploration Method

k-certainty exploration method, an efficient reinforcement learning algorithm, is not applied to environments whose state space is continuous because continuous state space must be changed to discrete state space. Our purpose is to construct discrete semi-Markov decision process (SMDP) models of such environments using growing cell structures to autonomously divide continuous state space then usingk-certainty exploration method to construct SMDP models. Multiagentk-certainty exploration method is then used to improve exploration efficiency. Mobile robot simulation demonstrated our proposal's usefulness and efficiency.

Look-ahead control of conveyor-serviced production station by using potential-based online policy iteration

We consider the look-ahead control of a conveyor-serviced production station (CSPS) in the context of semi-Markov decision process (SMDP) model, and our goal is to find an optimal control policy under either average- or discounted-cost criteria. Policy iteration (PI), combined with the concept of performance potential, can be applied to provide a unified optimisation framework for both criteria. However, a major difficulty arises in the exact solution scheme, that is, it requires not only the full knowledge of model parameters, but also a considerable amount of work to obtain and process the necessary system and performance matrices. To overcome this difficulty, we propose a potential-based online PI algorithm in this article. During implementation, by analysing and utilising the historic information of all the past operation of a practical CSPS system, the potentials and state-action values are learned on line through an effective exploration scheme. We finally illustrate the successful application of this learning-based technique in CSPS systems by an example.

Evaluation of a hierarchical reinforcement learning spoken dialogue system

We describe an evaluation of spoken dialogue strategies designed using hierarchical reinforcement learning agents. The dialogue strategies were learnt in a simulated environment and tested in a laboratory setting with 32 users. These dialogues were used to evaluate three types of machine dialogue behaviour: hand-coded, fully-learnt and semi-learnt. These experiments also served to evaluate the realism of simulated dialogues using two proposed metrics contrasted with ‘Precision-Recall’. The learnt dialogue behaviours used the Semi-Markov Decision Process (SMDP) model, and we report the first evaluation of this model in a realistic conversational environment. Experimental results in the travel planning domain provide evidence to support the following claims: (a) hierarchical semi-learnt dialogue agents are a better alternative (with higher overall performance) than deterministic or fully-learnt behaviour; (b) spoken dialogue strategies learnt with highly coherent user behaviour and conservative recognition error rates (keyword error rate of 20%) can outperform a reasonable hand-coded strategy; and (c) hierarchical reinforcement learning dialogue agents are feasible and promising for the (semi) automatic design of optimized dialogue behaviours in larger-scale systems.

Determination of optimal call admission control policy in wireless networks

This paper investigates the optimal call admission control (CAC) policy for non-priority scheme (NPS) and reserved channel scheme (RCS) in wireless networks, respectively. Both new call and handoff call arrival processes are assumed to be Poisson processes, and the call holding times are exponentially distributed with different rate for new call and handoff calls. Admitting each call would bring a reward to the network provider but holding each call in the system would also incur some expenses (or cost) to the provider. We concentrate on the optimization problems of when to admit or reject a call in order to achieve the maximum total expected discounted reward. By establishing a discounted semi Markov decision process (SMDP) model, we verify that the optimal policies are state-related control limit policies for both NPS and RCS. Our numerical results explained in both tables and diagrams are consistent with our theoretic results.