semi-Markov Decision Process Research Articles

A condition-based maintenance policy for a two-component balanced system with dependent degradation processes

Preventive maintenance of balanced systems has garnered increasing research attention due to its significance in production and servicing processes. This paper investigates the optimization problem of a condition-based maintenance policy for a two-component balanced system with dependent degradation processes. The system’s degradation follows a bivariate gamma process. Once the degradation level of one component exceeds a critical value, the system fails and triggers an immediate corrective replacement. Moreover, if the degradation difference between two components exceeds a threshold, the system is in an unbalanced state, which incurs a penalty cost. Periodic inspections are implemented to reveal the degradation of the system. At each inspection epoch, the decision-maker chooses an action among do-nothing, repair, and preventive replacement. Different from previous research, a repair action does not reduce the degradation levels of either component but increases the degradation level of the lower one to rebalance the system. The objective is to determine the optimal condition-based maintenance policy that minimizes the long-run average cost rate. The optimization problem is analyzed in a semi-Markov decision process framework. A policy-iteration dynamic programming algorithm is developed to derive the optimal policy. A numerical example is presented to illustrate the effectiveness of the proposed approach.

Integrated energy hub dispatch with a multi-mode CAES–BESS hybrid system: An option-based hierarchical reinforcement learning approach

The high penetration of renewable energy sources (RES) in power generation has driven demand for advanced integrated energy management systems (IEMS). In this study, to address the challenges of insufficient adaptability to dynamic supply–demand, a multi-type energy IEMS combining compressed air energy storage (CAES) and a battery energy storage system (BESS) is proposed, which operates under a multi-mode energy storage (MES) mechanism with rapid response, long-term balance, and synergic adjustment modes. To address the complexity of sequential decisions, an option-critic based twin delayed deep deterministic policy gradient (OCTD3) algorithm is firstly proposed within the hierarchical reinforcement learning (HRL) framework, enhancing efficiency through encapsulation of subtasks within ”options”. Additionally, model precision is refined by fitting the electricity–gas–heat conversion dynamics of CAES under off-design conditions. Dispatch tasks are modeled as an option-based Semi-Markov Decision Process (SMDP) and optimized by the OCTD3 to improve the power fluctuations index (PFI), comprehensive costs index (CCI), and system response synergy index (SRSI). Comparative simulations reveal that the MES mechanism boosts SRSI by 91.8%, showcasing high adaptability to varied supply–demand scenarios. The OCTD3 algorithm develops five hybrid strategies for CAES–BESS across three modes, effectively cutting costs by reducing electricity purchases and fluctuations expenses, and lowering PFI by 42.2% through balancing peak–valley loads and swiftly responding to transient shifts.

Optimal condition-based opportunistic maintenance policy for two-component systems considering common cause failure

This paper introduces a maintenance optimization model designed for a system comprising two distinct components that are arranged in series and susceptible to common cause failure. Both the components undergo dependent degradation and shock processes whose performance levels are periodically inspected. A condition-based opportunistic maintenance policy is considered, indicating that either preventive or corrective maintenance may be conducted depending on the deterioration state observed during inspections. Meanwhile, when maintenance is performed on one component, the other can be opportunistically maintained. An optimization problem is then formulated under the framework of semi-Markov decision processes, in order to find the optimal preventive and opportunistic maintenance thresholds for the two components. Three algorithms are introduced to address this problem, namely the improved genetic algorithm, the cyclic coordinate search algorithm, and the enumeration algorithm. Their effectiveness is demonstrated through a case study focusing on offshore wind turbines. The result shows that the efficiency of the improved genetic algorithm is significantly superior to the other two algorithms.

A smart inventory management system with medication demand dependencies in a hospital supply chain: A multi-agent reinforcement learning approach

In light of the intense need for quality health and well-being, the healthcare industry must improve its operations by strengthening its service delivery and reducing overall costs. A smart inventory management system for managing medicines in a hospital supply chain (HSC) is one of the best solutions as its advantages ensure availability of affordable medicines as well as inventory cost reduction. Significant effort is made on inventory management from literature; however, smart inventory systems for HSC considering the various complexities and uniqueness of healthcare systems are limited. This paper attempts to fill this gap by developing a stochastic semi-Markov decision process model which is solved by a multi-agent reinforcement learning method. Data collected from a multispecialty hospital in India’s eastern region serves to validate the proposed model. The results present the optimal order quantities and optimal inventory control policy for the HSC considering the uncertain medication demand at the hospital pharmacy and multiple point-of-care units, and demand dependencies among medicines prescribed to the patients admitted in the hospital. The study also has significant managerial implications for enhancing HSC’s inventory performance, improving the healthcare service delivery, reducing healthcare cost, thereby ensuring the affordable, accessible, and quality healthcare to the society.

Discounted cost exponential semi-Markov decision processes with unbounded transition rates: a service rate control problem with impatient customers

Discounted cost exponential semi-Markov decision processes with unbounded transition rates: a service rate control problem with impatient customers

Joint optimization of maintenance and spares inventory policy for a series-parallel system considering dependent failure processes

This paper investigates the joint optimization of condition-based opportunistic maintenance and spare inventory policies for a series-parallel system consisting of software and components. Software failure follows a homogeneous Poisson process. Components are subject to mutually dependent competing failure processes. When the software failure occurs, an opportunistic inspection will be implemented for the components. Besides the maintenance policy, the (s, S) inventory policy is applied for components. The maintenance and order decisions will be made according to the conditions of the components and inventory level at each inspection. The state transition probabilities and expected sojourn times can be obtained by the formulated semi-Markov decision process. The optimal inspection interval, the preventive replacement threshold of the component, and the order point can be derived by minimizing the expected average cost rate. Finally, the effectiveness and superiority of the proposed joint optimization model are verified by an illustrative example.

Optimal maintenance of deteriorating equipment using semi-Markov decision processes and linear programming

Optimal maintenance of deteriorating equipment using semi-Markov decision processes and linear programming

CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments

Audio-visual navigation of an agent towards locating an audio goal is a challenging task especially when the audio is sporadic or the environment is noisy. In this paper, we present CAVEN, a Conversation-based Audio-Visual Embodied Navigation framework in which the agent may interact with a human/oracle for solving the task of navigating to an audio goal. Specifically, CAVEN is modeled as a budget-aware partially observable semi-Markov decision process that implicitly learns the uncertainty in the audio-based navigation policy to decide when and how the agent may interact with the oracle. Our CAVEN agent can engage in fully-bidirectional natural language conversations by producing relevant questions and interpret free-form, potentially noisy responses from the oracle based on the audio-visual context. To enable such a capability, CAVEN is equipped with: i) a trajectory forecasting network that is grounded in audio-visual cues to produce a potential trajectory to the estimated goal, and (ii) a natural language based question generation and reasoning network to pose an interactive question to the oracle or interpret the oracle's response to produce navigation instructions. To train the interactive modules, we present a large scale dataset: AVN-Instruct, based on the Landmark-RxR dataset. To substantiate the usefulness of conversations, we present experiments on the benchmark audio-goal task using the SoundSpaces simulator under various noisy settings. Our results reveal that our fully-conversational approach leads to nearly an order-of-magnitude improvement in success rate, especially in localizing new sound sources and against methods that use only uni-directional interaction.

Traffic pattern-aware elevator dispatching via deep reinforcement learning

This study addresses the elevator dispatching problem using deep reinforcement learning, with a specific emphasis on traffic pattern awareness. Previous studies on reinforcement learning-based elevator dispatching have largely focused on training separate models for single traffic patterns, such as up-peak, down-peak, lunch-peak, and inter-floor. This separate training approach not only introduces practical complexities by requiring an auxiliary model to predict traffic patterns for guiding dispatching decisions but is also computationally burdensome. In contrast, our goal is to develop a unified, traffic pattern-aware dispatching model. We formulate the elevator dispatching problem as a Semi-Markov Decision Process (SMDP) with novel state representation, action space, and reward function designs. To solve the formulated SMDP, we propose a Dueling Double Deep Q-Network (D3QN) architecture associated with the training algorithm. To ensure traffic pattern awareness, we train our model in a unified ‘All in One’ traffic scenario, employing two practical techniques to enhance the training process: (1) temporal grouping with gradient surgery and (2) incorporation of passenger arrival information. Empirical evaluations confirm the superiority of our model over multiple benchmarks, including those relying on separate, pattern-specific models. Remarkably, our unified model demonstrates robust performance across unseen traffic scenarios and performs exceptionally well in single traffic patterns despite being trained solely on the unified ‘All in One’ scenario. The short inference time for decision-making further solidifies the model’s practical viability. Additionally, the incremental benefits contributed by each of our introduced techniques are also investigated. Our code is available at https://github.com/jswan95/RL-based-traffic-pattern-aware-elevator-dispatching

Bi-level corrected residual life-based maintenance for deteriorating systems under competing risks

The accurate prediction of residual life in deteriorating systems is a challenging task due to uncertainties and idealized assumptions during data acquisition and model construction. To address this challenge, the paper proposes a bi-level maintenance model that incorporates a corrected residual life (CRL) approach for a deteriorating system under competing risks. Specifically, the degradation failure is modeled using a Gamma process, while the competing sudden failure is described by a proportional hazards (PH) model. To reduce inconsistencies in residual life prediction, a CRL model is presented and continuously updated with a dynamic corrective factor at each monitoring epoch. Based on the updated CRL, a bi-level maintenance model is developed, which considers both system availability and maintenance cost objectives, enabling dynamic monitoring of the system's degradation state and residual life. The optimal decision variables in the maintenance model are determined through a multi-objective optimization algorithm formulated within a semi-Markov decision process (SMDP) framework. The unique aspect of this work lies in the consideration of prediction errors in residual life and the incorporation of multi-attribute optimization for maintenance design. The proposed method is validated through a case study on light-emitting diodes, confirming its effectiveness in improving residual life prediction and optimizing maintenance decisions.

Optimizing warfarin dosing for patients with atrial fibrillation using machine learning

While novel oral anticoagulants are increasingly used to reduce risk of stroke in patients with atrial fibrillation, vitamin K antagonists such as warfarin continue to be used extensively for stroke prevention across the world. While effective in reducing the risk of strokes, the complex pharmacodynamics of warfarin make it difficult to use clinically, with many patients experiencing under- and/or over- anticoagulation. In this study we employed a novel implementation of deep reinforcement learning to provide clinical decision support to optimize time in therapeutic International Normalized Ratio (INR) range. We used a novel semi-Markov decision process formulation of the Batch-Constrained deep Q-learning algorithm to develop a reinforcement learning model to dynamically recommend optimal warfarin dosing to achieve INR of 2.0–3.0 for patients with atrial fibrillation. The model was developed using data from 22,502 patients in the warfarin treated groups of the pivotal randomized clinical trials of edoxaban (ENGAGE AF-TIMI 48), apixaban (ARISTOTLE) and rivaroxaban (ROCKET AF). The model was externally validated on data from 5730 warfarin-treated patients in a fourth trial of dabigatran (RE-LY) using multilevel regression models to estimate the relationship between center-level algorithm consistent dosing, time in therapeutic INR range (TTR), and a composite clinical outcome of stroke, systemic embolism or major hemorrhage. External validation showed a positive association between center-level algorithm-consistent dosing and TTR (R2 = 0.56). Each 10% increase in algorithm-consistent dosing at the center level independently predicted a 6.78% improvement in TTR (95% CI 6.29, 7.28; p < 0.001) and a 11% decrease in the composite clinical outcome (HR 0.89; 95% CI 0.81, 1.00; p = 0.015). These results were comparable to those of a rules-based clinical algorithm used for benchmarking, for which each 10% increase in algorithm-consistent dosing independently predicted a 6.10% increase in TTR (95% CI 5.67, 6.54, p < 0.001) and a 10% decrease in the composite outcome (HR 0.90; 95% CI 0.83, 0.98, p = 0.018). Our findings suggest that a deep reinforcement learning algorithm can optimize time in therapeutic range for patients taking warfarin. A digital clinical decision support system to promote algorithm-consistent warfarin dosing could optimize time in therapeutic range and improve clinical outcomes in atrial fibrillation globally.

Deep reinforcement learning for demand fulfillment in online retail

A distinctive feature of online retail is the flexibility to ship items to customers from different distribution centers (DCs). This creates interdependence between DCs and poses new challenges in demand fulfillment to decide from which DC to satisfy each customer demand. This paper addresses a demand fulfillment problem in a multi-DC online retail environment where demand and replenishment lead time are stochastic. The objective of the problem is to minimize the long-term operational costs by determining the source DC for each customer demand. We formulate the problem as a semi-Markov decision process and develop a deep reinforcement learning (DRL) algorithm to solve the problem. To evaluate the performance of the DRL algorithm, we compare it with a set of heuristic rules and exact solutions obtained by linear programming. Numerical results show that the DRL policy performs equally well with the most competitive heuristic on complete pooling DC networks and outperforms all the heuristics on partial pooling DC networks. Additionally, by analyzing the transshipment ratio of the best-observed policies, we provide managerial insights regarding the circumstances in which transshipment is more favorable.

Agent-based Decision Making and Control of Manufacturing System Considering the Joint Production, Maintenance, and Quality by Reinforcement Learning

Taking an integrated approach towards production, maintenance, and control in manufacturing systems is crucial due to the profound impact of their interconnections. Investigating these aspects in isolation may lead to infeasible solutions. This research focuses on the real-time and autonomous decision-making process concerning joint production planning, maintenance, and quality problems in a stochastic deteriorating production system with limited maintenance activities. Formulating the problem as a continuous semi-Markov decision process accounts for the complexities of the real production system and the occurrence of events over an uneven and continuous period. While dynamic programming is a common tool for addressing joint optimization problems, it has limitations, such as the curse of dimensionality. In this study, the optimal policy of the decision-maker agent is obtained by the goal-directed machine learning method called (R-SMART) and agent-based modeling. To the author's knowledge, the proposed approach is novel, and there is little research on such an implementation of the joint optimization problem. The quality of the optimal policy is evaluated through heuristic and simulation-optimization methods in various scenarios. The results demonstrate that the proposed RL-based method outperforms others in most scenarios, achieving a stable, integrated optimal policy.

Structural properties of optimal fidelity selection policies for human-in-the-loop queues

We study optimal fidelity selection for a human operator servicing a queue of homogeneous tasks. The agent can service a task with a normal or high fidelity level, where fidelity refers to the degree of exactness and precision while servicing the task. Therefore, high-fidelity servicing results in higher-quality service but leads to larger service times and increased operator tiredness. We treat the human cognitive state as a lumped parameter that captures psychological factors such as workload and fatigue. The operator’s service time distribution depends on her cognitive dynamics and the fidelity level selected for servicing the task. Her cognitive dynamics evolve as a Markov chain in which the cognitive state increases with high probability whenever she is busy and decreases while resting. The tasks arrive according to a Poisson process and the operator is penalized at a fixed rate for each task waiting in the queue. We address the trade-off between high-quality service of the task and consequent penalty due to a subsequent increase in queue length using a discrete-time Semi-Markov Decision Process framework. We numerically determine an optimal policy and the corresponding optimal value function. Finally, we establish the structural properties of an optimal fidelity policy and provide conditions under which the optimal policy is a threshold-based policy.

Service Control in a Retrial Service Facility System with Two-type of Customer-Semi MDP

In the literature on service facility system, it is common to assume fixed service rate, such as flexible retrial arrival, this assumption is not realistic. In contrast retrial demands model rarely define the mechanism through which arrival of customers is classified into two types say priority customers and ordinary customers. For the given values of maximum inventory, maximum waiting space, reorder level and lead times, we determine the optimal ordering policy at various instants of time. The system is formulated as a Semi-Markov Decision Process and the optimum policy to be employed is found using linear programming method. Numerical examples are provided to illustrate the model.

A Hierarchical Deep Learning Approach for Optimizing CCA Threshold and Transmit Power in Wi-Fi Networks

The nodes, e.g., access points and clients, in current WiFi networks rely on carrier sense multiple access (CSMA) for channel access. This means they rely on a clear channel assessment (CCA) threshold to determine when it is safe to transmit a packet. In practice, this threshold is fixed, and nodes do not adapt their CCA threshold given varying channel conditions. To this end, this paper proposes and studies the use of deep reinforcement learning (DRL) to optimize the CCA threshold and transmit power of an AP. It allows an AP to learn the optimal policy for setting its CCA threshold and transmit power given its queue length and interference level. We outline a semi-Markov decision process (Semi-MDP) for the problem at hand and solve it using a decentralized deep hierarchical reinforcement learning (HRL) approach. We then evaluate its performance using traffic generated via a Poisson distribution and also from a trace file. The results show that when compared to two competing solutions, an AP with HRL reduces its average queue length by 24.4% and 52.4%, respectively.

An Optimal Opportunistic Maintenance Planning Integrating Discrete- and Continuous-State Information

Information-driven group maintenance is crucial to enhance the operational availability and profitability of diverse industrial systems. Existing group maintenance models have primarily concentrated on a single health criterion upon maintenance implementation, where the fusion of multiple health criteria is rarely reported. However, this is not aligned with actual maintenance planning of multi-component systems on many occasions, where multi-source health information can be integrated to support robust decision making. Additionally, how to improve maintenance effectiveness through a scientific union of both scheduled and unscheduled maintenance remains a challenge in group maintenance. This study addresses these research gaps by devising an innovative multiple-information-driven group replacement policy for serial systems. In contrast to existing studies, both discrete-state information (hidden defect) and continuous degradation information are employed for group maintenance planning, and scheduled postponed maintenance and unscheduled opportunistic maintenance are dynamically integrated for the first time to mitigate downtime loss. To be specific, inspections are equally spaced to reveal system health states, followed by the multi-level replacement implemented when either (a) the degradation of the continuously degrading unit reaches a specified threshold, or (b) the age of the multi-state unit since the defect’s identification reaches a pre-set age (delayed replacement). Such scheduling further enables the implementation of multi-source opportunistic replacement to alleviate downtime. The Semi-Markov Decision Process (SMDP) is utilized for the collaborative optimization of continuous- and discrete-state thresholds, so as to minimize the operational costs. Numerical experiments conducted on the critical structure of circulating pumps verify the model’s applicability.

Optimal rearrangement and preventive maintenance policies for heterogeneous balanced systems with three failure modes

This paper studies a heterogeneous balanced system composed of multiple interchangeable components. The degradation process of components is described by a gamma process, and deterioration rates in different positions are different due to the effect of loading stress or temperature. Three competing failures may occur: a) shock failure caused by environment shocks, b) soft failure when the deterioration level of any component exceeds a critical value, and c) out of balance when the difference value among components reaches the failure threshold. To avoid system failure, a rearrangement action is adopted to change the position of components. Besides, preventive maintenance is considered when a component deteriorates severely. A semi-Markov decision process (SMDP) is developed to obtain the optimal policy by minimizing the average maintenance cost. To facilitate calculation, the data transmission method is used to convert the semi-Markov decision model to the Markov decision model, and a value iteration algorithm is established to obtain the optimal maintenance action at each state. Considering practical implications, an imperfect preventive and opportunistic maintenance model is formulated under the SMDP framework. Finally, a typical tire rotation problem in motorcycles proves that the imperfect maintenance policy outperforms the other policies when replacement fees are more expensive.

MAESTRO-X: Distributed Orchestration of Rotary-Wing UAV-Relay Swarms

This work details a scalable framework to orchestrate a swarm of rotary-wing UAVs serving as cellular relays to facilitate beyond line-of-sight connectivity and traffic offloading for ground users. First, a Multiscale Adaptive Energyconscious Scheduling and TRajectory Optimization (MAESTRO) framework is developed for a single UAV. Aiming to minimize the time-averaged latency to serve user requests, subject to an average UAV power constraint, it is shown that the optimization problem can be cast as a semi-Markov decision process, and exhibits a multiscale structure: outer actions on radial wait velocities and terminal service positions minimize the longterm delay-power trade-off, optimized via value iteration; given these outer actions, inner actions on angular wait velocities and service trajectories minimize a short-term delay-energy cost; finally, rate adaptation is embedded along the trajectory to leverage air-to-ground channel propagation conditions. A novel hierarchical competitive swarm optimization scheme is developed in the inner optimization, to devise high-resolution trajectories via iterative pair-wise updates. Next, MAESTRO is eXtended to UAV swarms (MAESTRO-X) via scalable policy replication, enabled by a decentralized command-and-control network augmented with: (1) spread maximization to proactively position UAVs to serve future requests; (2) consensus-driven conflict resolution to orchestrate scheduling decisions based on delay-energy costs including queuing dynamics; (3) adaptive frequency reuse to improve spectrum utilization across the network; and (4) a piggybacking mechanism allowing UAVs to serve multiple ground users simultaneously. Numerical evaluations show that, for user requests of 10 Mbits, generated according to a Poisson arrival process with rate 0.2 req/min/UAV, single-agent MAESTRO offers 3.8× faster service than a high-altitude platform and 29% faster than a static UAV deployment; moreover, for a swarm of 3 UAV-relays, MAESTRO-X delivers data payloads 4.7× faster than a successive convex approximation scheme; and remarkably, a single UAV optimized via MAESTRO outclasses 3 UAVs optimized via a deep-Q network by 38%.

An adaptive reliability-based maintenance policy for mechanical systems under variable environments

The performance of mechanical systems is greatly influenced by varying operational environments such as load, temperature, and humidity, which can result in accelerated degradation and failure. To address this issue, this article considers an adaptive maintenance policy based on updated conditional reliability for mechanical systems under variable environments. A model of deterioration-integrated failure process is presented, which integrates degradation process, failure events, and environmental variables of the system. The degradation process is modeled by a random effect Wiener process, with its drift parameter following a general distribution to represent the environmental random effects. The failure events are modeled by a proportional hazards (PH) model with environment-varying covariate. The health of the system is evaluated using a matrix-based approximation method that provides estimates of the system's conditional reliability and mean residual life. Using the prognostic information, an adaptive maintenance model with a switchable control chart is then developed to monitor and maintain system based on the status of system reliability. The decision variables of the adaptive reliability-based maintenance are determined by a computational algorithm formulated in a semi-Markov decision process (SMDP) framework, with the objective of minimizing the long-run expected average cost per unit time. Vibrational data of bearings under varying operational environments are presented to illustrate the effectiveness of the proposed approach.