semi-Markov Decision Process Research Articles

An integrated optimization of lot-sizing and condition-based maintenance policy of a two-stage deteriorating system

In this study, a two-stage deteriorating production system is considered. The initiation time of the deterioration follows an exponential distribution. A homogeneous Gamma process is used to model the deterioration propagation process. The equipment produces a single product with a constant production rate and demand rate. To ensure the efficiency of the production, inspection is carried out at the end of each production run. The lot-sizing and the condition-based maintenance threshold are optimized by minimizing the long-run average cost rate, which is derived in the framework of a semi-Markov decision process. The traversal algorithm is used to obtain optimal results. A numerical example is presented to illustrate the applicability of the proposed model. Some sensitivity analysis are carried out in order to highlight the interest and properties of the model. It may provide a theoretical reference for production and maintenance schedules.

Adaptive Maintenance Scheme for Degrading Devices With Dynamic Conditions and Random Failures

The estimations of degradation and remaining useful life are important for realizing the maintenance of modern devices, which generally operate under varying conditions. Although some prognostic approaches have been presented, there is still a lack of an effective maintenance scheme that utilizes prognostic information to maintain systems under varying operational conditions. Therefore, a new degradation-integrated failure model is presented in this article for deteriorating systems that are subject to dynamic conditions and random failures. An adaptive maintenance scheme is then developed on the basis of the failure model. The proposed scheme is used to adaptively monitor the system conditions and switch the monitoring frequency according to the system hazard level. The monitoring frequencies in the maintenance scheme are optimized by a computational algorithm formulated in a semi-Markov decision process framework using estimated conditional reliability under varying conditions. The particularity of this work is the consideration of a changeable monitoring scheme under dynamic conditions, which is effective in reducing false alarms of maintenance incurred by environmental shocks. The proposed approach is demonstrated by a case study of power devices, and comparisons with other advanced approaches are also presented.

Optimal production lot-sizing and condition-based maintenance policy considering imperfect manufacturing process and inspection errors

In this paper, we propose an integrated Economic manufacturing quantity (EMQ) model combining both the concepts of condition-based maintenance (CBM) and imperfect manufacturing process. The manufacturing process is modelled by two indicators. One possesses binary state, indicating whether the manufacturing process is in-control or not. The other one is modelled by a homogeneous Gamma process, representing the degradation of the manufacturing equipment. The system is inspected at the end of each production run, upon which, the deterioration level can be perfectly observed, while two types of errors may occur in revealing the state of the manufacturing process. Defective products can be fabricated when the manufacturing state degrades. An integrated production and CBM policy is proposed. The objective is to develop the optimal production lot-sizing and preventive maintenance threshold in order to minimize the expected cost rate in the long-time horizon. We model the problem in the framework of a semi-Markov decision process. The successive-approximations method is applied to solve the problem numerically. The applicability of the proposed model and some sensitivity analysis are presented in a numerical illustration. It can provide theoretical reference to the decision-maker in production and maintenance planning.

A heuristically accelerated reinforcement learning method for maintenance policy of an assembly line

<p style='text-indent:20px;'>This paper aims to investigate the maintenance policy for a two-machine one-buffer (2M1B) assembly line system. We assume that the observed quality states of the deteriorating machines in the system are characterized by multiple decreasing yield stages. A semi-Markov decision process (SMDP) model is used for describing the deteriorating process of the system. A heuristically accelerated multi-agent reinforcement learning (HAMRL) method is conducted to solve the problem model. The asynchronous updating rules are introduced in the HAMRL method, and the production time, preventive maintenance (PM) time and corrective repair (CR) time are random, and the deterioration mode of the device is not fixed. Meanwhile, a comparison with a simulated annealing search (SAS) based exploration algorithm and a neighborhood search (NS) based exploration algorithm in reinforcement learning (RL) is presented. The empirical results indicate that the proposed HAMRL algorithm can speed up the learning process, and has a certain advantage for the larger space and the more practical problem. And the maintenance strategy for the 2M1B assembly line system is obtained under the condition of convergent system average cost rate. This paper provides new and practical insights into the application and selection of techniques for maintenance policy of the 2M1B assembly line system.</p>

Cooperative Learning for Smart Charging of Shared Autonomous Vehicle Fleets

We study the operational problem of shared autonomous electric vehicles that cooperate in providing on-demand mobility services while maximizing fleet profit and service quality. Therefore, we model the fleet operator and vehicles as interactive agents enriched with advanced decision-making aids. Our focus is on learning smart charging policies (when and where to charge vehicles) in anticipation of uncertain future demands to accommodate long charging times, restricted charging infrastructure, and time-varying electricity prices. We propose a distributed approach and formulate the problem as a semi-Markov decision process to capture its stochastic and dynamic nature. We use cooperative multiagent reinforcement learning with reshaped reward functions. The effectiveness and scalability of the proposed model are upgraded through deep learning. A mean-field approximation deals with environment instabilities, and hierarchical learning distinguishes high-level and low-level decisions. We evaluate our model using various numerical examples based on real data from ShareNow in Berlin, Germany. We show that the policies learned using our decentralized and dynamic approach outperform central static charging strategies. Finally, we conduct a sensitivity analysis for different fleet characteristics to demonstrate the proposed model’s robustness and provide managerial insights into the impacts of strategic decisions on fleet performance and derived charging policies. Supplemental Material: The online appendix is available at https://doi.org/10.1287/trsc.2022.1187 .

Distributed Real-Time Scheduling in Cloud Manufacturing by Deep Reinforcement Learning

With the extensive application of automated guided vehicles, real-time production scheduling considering logistics services in cloud manufacturing (CM) becomes an urgent problem. Thus, this study focuses on the distributed real-time scheduling (DRTS) of multiple services to respond to dynamic and customized orders. First, a DRTS framework with cloud–edge collaboration is proposed to improve performance and satisfy responsiveness, where distributed actors and one centralized learner are deployed in the edge and cloud layer, respectively. And, the DRTS problem is modeled as a semi-Markov decision process, where the processing services sequencing and logistics services assignment are considered simultaneously. Then, we developed a distributed dueling deep Q network (D3QN) with cloud–edge collaboration to optimize the weighted tardiness of jobs. The experimental results show that the proposed D3QN obtains lower weighted tardiness and shorter flow-time than other state-of-the-art algorithms. It indicates the proposed DRTS method has significant potential to provide efficient real-time decision-making in CM.

Semi-Markovian Maintenance Optimization for Reinforced Concrete Enabled by a Synthesized Deterioration Model

Aging bridge infrastructure may jeopardize safety, and its economic impact has raised concerns worldwide. Reinforced concrete is a major component of bridge infrastructure, whose condition is directly related to reliability and safety. However, evaluating and maintaining reinforced concrete remains a complex issue, as it is often exposed to a dynamic and stochastic environment. In this article, we provide an integrated approach to capturing the characteristics of reinforced concrete deterioration by synthesizing knowledge from the physical model and a stochastic model. Then, a semi-Markov decision process is constructed for optimizing the maintenance of reinforced concrete. To improve the computation efficiency of the approach, we design the policy iteration algorithm with finite evaluations to reduce computation cost. The designed approach can further calibrate the temporal and complex dynamic information of the reinforced concrete deterioration and has a better performance than that of the Markov decision process. Finally, we explore the sensitivity of the maintenance policy under the changing of cost rates and durations of the maintenance activities.

Replacement and Repair Optimization for Production Systems Under Random Production Waits

This study considers a repairable production system operated under an age-based preventive replacement policy that is subject to independent random production waits and failures. In addition to the age-based replacement policy, we propose a maintenance model that uses production waits to schedule preventive replacement. That is, a decision maker can preventively replace the system either during a production wait or at the age threshold, whereas if a failure occurs during production, the decision maker must decide whether to perform a minimal repair or a corrective replacement to restore the system. Under the above setting, we develop a semi-Markov decision process (SMDP) to obtain the optimal maintenance policy that minimizes the long-run average maintenance cost rate. We establish the existence of the optimal maintenance policy and provide an algorithm to numerically obtain the optimal action for each state. We further generalize the model to incorporate imperfect maintenance and the nonhomogeneous arrival of production waits. In the latter case, it is computationally intractable to optimize the SMDP using a value iteration algorithm due to the curse of dimensionality. To address this challenge, we further develop an approximate dynamic programming framework to generate high-quality solutions. An attractive feature of our model is its generality, such that the model includes many existing maintenance models as special cases. A real-world example from a steel factory is used to demonstrate the proposed model.

Joint optimization of production lot-sizing and condition-based maintenance in an imperfect production process with dependent indicators

ABSTRACT This paper addresses the integrated optimization of the economic manufacturing quantity and the condition-based maintenance policy of a deteriorating manufacturing system. The considered facility produces a single type of product and is inspected at the end of each production run. Upon inspection, two dependent indicators are revealed: one implies whether the production process is in control or not and the other one represents the deterioration level of the facility. The dependence between the two indicators is that whenever the facility deterioration exceeds a pre-determined level, the system becomes more vulnerable such that the production process may switch to the out-of-control state. In the out-of-control state, a proportion of defective items is fabricated. Considering the inter-dependencies between production process and machine degradation, this paper develops an integrated production and maintenance model to minimize the overall cost. The expected cost rate in the long run is taken as the objective function to assess the proposed model, where the joint optimization of the lot-sizing and the maintenance policy is developed. The problem is formulated in the context of a semi-Markov decision process and solved with the successive approximation method. A numerical example is given to illustrate the applicability of the proposed model.

Event-Driven Computation Offloading in IoT With Edge Computing

Edge computing, which provides computation services at the edge of networks, has become a promising method to meet the massive computation demands of Internet of Things (IoT). To make full use of resources, a computation offloading scheme is needed in edge computing system. In this work, we propose an event-driven computation offloading scheme for the first time. Compared with the existing time-driven schemes, the proposed scheme has a smaller implementation complexity in some scenarios with computation-intensive task computing. In the proposed scheme, the priority of different tasks is jointly considered. To decide the optimal offloading action of the scheme, we formulate the offloading problem as a semi-Markov decision process (SMDP). Then, a model-based method is proposed to derive the optimal offloading policy under fully explored system by addressing the challengs of modeling. On the other hand, considering partially explored system, we propose an online double deep Q-network algorithm, which can deal with the poor scalability of the standard Q-learning algorithm, to derive the optimal offloading policy. In addition, we also introduce some tricks to accelerate the learning procedure. The simulation results show the superior performance of our proposed scheme.

Data-driven optimization of repair schemes and inspection intervals for highway bridges

Routine bridge inspections enable timely maintenance plans based on potential risks. These inspections are performed at fixed time intervals, which may be unnecessarily short for bridges in excellent conditions and precariously long for those in poor conditions. Also, deterioration models used in current bridge management approaches fail to include the effect of age, causing inaccurate predictions. This paper develops a hazard-based bridge management approach where inspection frequencies and repair suggestions are optimized based on hazard levels. A two-dimensional Markov model describes the age-dependent bridge deterioration, and a semi-Markov decision process optimizes inspection intervals and repair decisions. The objective is to minimize the expected long-term total annual costs, including expenses associated with maintenance and traffic detours. The modeling process and maintenance management framework was demonstrated on an example bridge superstructure in New York. The results showed lower annual and total costs than the conventional management method. As expected, sensitivity analyses indicated that the pre-specified choices of inspection intervals for low and high hazard levels have a significant effect on the calculated hazard threshold levels. The proposed maintenance management method is adaptable and practical; bridge authorities can identify appropriate repair actions and make inspection decisions based on simple threshold comparisons.

The exponential cost optimality for finite horizon semi-Markov decision processes

The exponential cost optimality for finite horizon semi-Markov decision processes

RAT selection for IoT devices in HetNets: Reinforcement learning with hybrid SMDP algorithm

Due to the increasing deployment of heterogeneous networks (HetNets), the selection of which radio access technologies (RATs) for Internet of Things (IoT) devices such as user equipments (UEs) has recently received extensive attention in mobility management research. Most of existing RAT selection methods only optimize the selection strategies from the UE side or network side, which results in heavy network congestion, poor user experience and system utility degradation. In this paper the UE side and the network side are considered comprehensively, based on the game theory (GT) model we propose a reinforcement learning with assisted network information algorithm to overcome the crucial points. The assisted information is formulated as a semi-Markov decision process (SMDP) provided for UEs to make accurate decisions, and we adopt the iteration approach to reach the optimal policy. Moreover, we investigate the impacts of different parameters on the system utility and handover performance. Numerical results validate that our proposed algorithm can mitigate unnecessary handovers and improve system throughputs.

Potential Impacts of Smart Homes on Human Behavior: A Reinforcement Learning Approach

We aim to investigate the potential impacts of smart homes on human behavior. To this end, we simulate a series of human models capable of performing various activities inside a reinforcement learning-based smart home. We then investigate the possibility of human behavior being altered as a result of the smart home and the human model adapting to one-another. We design a semi-Markov decision process human task interleaving model based on hierarchical reinforcement learning that learns to make decisions to either pursue or leave an activity. We then integrate our human model in the smart home which is based on Q-learning. We show that a smart home trained on a generic human model is able to anticipate and learn the thermal preferences of human models with intrinsic rewards similar to the generic model. The hierarchical human model learns to complete each activity and set optimal thermal settings for maximum comfort. With the smart home, the number of time steps required to change the thermal settings are reduced for the human models. Interestingly, we observe that small variations in the human model reward structures can lead to the opposite behavior in the form of unexpected switching between activities which signals changes in human behavior due to the presence of the smart home.

A Sojourn-Based Approach to Semi-Markov Reinforcement Learning

In this paper we introduce a new approach to discrete-time semi-Markov decision processes based on the sojourn time process. Different characterizations of discrete-time semi-Markov processes are exploited and decision processes are constructed by their means. With this new approach, the agent is allowed to consider different actions depending also on the sojourn time of the process in the current state. A numerical method based on Q-learning algorithms for finite horizon reinforcement learning and stochastic recursive relations is investigated. Finally, we consider two toy examples: one in which the reward depends on the sojourn-time, according to the gambler’s fallacy; the other in which the environment is semi-Markov even if the reward function does not depend on the sojourn time. These are used to carry on some numerical evaluations on the previously presented Q-learning algorithm and on a different naive method based on deep reinforcement learning.

Innovative Construction of Reinforcement Learning Model for Information Fusion in Music Education

In order to improve the effect of modern music education, this paper applies the digital information technology of music resources to the construction of the music teaching system and derives two new types of semi-decision-making process reinforcement learning algorithms based on the Bellman optimality equation base on discrete time. Moreover, this paper uses the comparative research methods to obtain the Q-value learning curve of the incremental value iterative reinforcement learning algorithm based on the semi-Markov decision process and the incremental value iterative reinforcement learning algorithm based on the dichotomy to improve the fusion effect of music teaching resources. Finally, this paper combines the actual needs of modern music education to construct an intelligent music teaching model.

A Dynamic Resource Allocation Model Based on SMDP and DRL Algorithm for Truck Platoon in Vehicle Network

The rapid development of self-driving cars and breakthroughs in key technologies have made the truck platoon possible. In addition to reducing truck fuel consumption and air pollution by reducing air resistance, effective platoon strategies can also maximize highway throughput while improving driving safety. However, the truck platoon strategy’s current resource allocation model is still in the preliminary research stage. Therefore, inspired by the successful experience of deep reinforcement learning (DRL) in solving resource allocation problems, this article proposes a dynamic resource allocation model for the truck platoon based on the semi-Markov decision process (SMDP) and DRL, which is used to maximize system revenue when considering the resource cost and income balance of the transportation system. Precisely, the proposed method first models the process of controlling the dynamic in and out of the truck platoon as SMDP. The action value in a specific state obtained by the planning algorithm is used as a DRL sample for model training. Finally, the SMDP is optimized through the trained model to obtain a truck platoon resource that approximates the optimal strategy distribution plan. The experimental results show that compared with the traditional greedy algorithm, value iteration, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> -learning scheme concerning solving the dynamic resource allocation model of the truck platoon, the Deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> -Network (DQN) used in this article can reduce the probability of request processing delay while causing the system to obtain higher rewards.

AI-Enabled Energy-Efficient Fog Computing for Internet of Vehicles

Future autonomous electric vehicles (EVs) are equipped with several IoT sensors, smart devices, and wireless adapters, thus forming an Internet of Vehicles (IoVs). These intelligent EVs are envisioned to be a promising solution for improving transportation efficiency, road safety, and driving experience. Vehicular fog computing (VFC) is an evolving technology that allows vehicular application-related tasks to be offloaded to nearby computing nodes and process them quickly. A major challenge in the VFC system is to design energy-efficient task offloading algorithms. In this paper, we propose an optimal energy-efficient algorithm for task offloading in a VFC system that maximizes the expected reward function which is derived using the total energy and time delay of the system for the computation of the task. We use parallel computing and formulate the optimization problem as semi-Markov decision process (SMDP). Bellman optimal equation is used in value iteration algorithm (VIA) to get an optimal scheme by selecting the best action for the current state that maximizes the energy-based reward function. Numerical results show that the proposed scheme outperforms the greedy algorithm in terms of energy consumption.

A hybrid repair-replacement policy in the proportional hazards model

Cox's proportional hazards model is widely used to describe the hazard rate of a system deteriorating with age and diagnostic covariates. Existing maintenance in the proportional hazards model was primarily concerned with replacement, resulting in excessive maintenance for many repairable systems. This paper develops a novel hybrid repair-replacement model in the proportional hazards model with a stochastically increasing Markovian covariate process. Preventive repair reduces both age and covariate, and the reduction rate decreases as the number of repairs increases. At an inspection epoch where the system age, covariate state, and repair number are available, the decision-maker considers three possible actions, i.e., no maintenance, preventive repair, and preventive replacement. The objective is to derive the optimal policy that minimizes the long-run average maintenance cost rate. The optimization problem is formulated in the semi-Markov decision process (SMDP) framework. The structural properties of the optimal policy are examined to reduce the policy space. Then a policy-iteration algorithm with a backward policy-improvement step is developed for efficiently finding the optimization results. A practical numerical example with sensitivity analysis is conducted to illustrate the effectiveness of the proposed approach. A comparison with two heuristic policies confirms the superiority of the proposed policy in reducing maintenance costs.

Bi-level bayesian control scheme for fault detection under partial observations

Due to limitations of sensor techniques and system structures, the internal states of most complex devices are unobservable. Only partial observations can be monitored at discrete time epochs, which poses great challenges for fault detection and prediction. To facilitate the development of advanced approaches, this paper proposes a bi-level Bayesian control scheme for detecting faults of complex devices under partial observations. The deterioration of an operational system is modeled as a hidden three-state multivariate Markov process, in which the parameter estimates are obtained by an expectation maximization (EM) algorithm. Based on the multivariate Markov process, a bi-level Bayesian control scheme is presented to monitor the Bayesian posterior probability of the system in a warning state and uses two levels of sampling frequency to dynamically detect impending failures. The decision variables of the bi-level fault detection scheme are optimized and solved in a semi-Markov decision process (SMDP) framework. The particularity of this work is that two phases of system deterioration are considered into detection scheme design to avoid unnecessary false alarms. Using partially observed multivariate data from discrete monitoring, the proposed approach is validated to be capable of detecting the early fault occurrences. By comparing with other advanced methods, the superiority of the proposed model is demonstrated.