Optimal Control Limit Policy Research Articles

Optimal replacement in a proportional hazards model with cumulative and dependent risks

This paper deals with a new maintenance problem with two competing dependent risks: minor failures and major failures. The cumulative number of minor failures is incorporated into the proportional hazards model as a covariant process of the hazard function of major failures. We introduce the “Time Discrete Markovian Approximation” (TDMA) technique to solve the “curse of dimensionality” in Markov Decision Processes (MDP) and simplify the high-dimensional integration when computing the average cost by renewal theory. We develop a new optimal control limit policy with a “mixed hazards function” as the threshold and reveal its agreement with the solution from MDP. Additionally, a corresponding iterative algorithm is developed to produce a sequence converging to the optimal solution faster than the policy iteration.

A reinforcement learning approach to distribution-free capacity allocation for sea cargo revenue management

In this paper, we propose learning-based adaptive control based on reinforcement learning for the booking policy in sea cargo revenue management. The problem setting is that the demand distribution is unknown while the historical data is available, and the problem is formulated as a stochastic dynamic programming model. We demonstrate the existence of an optimal control limit policy and investigate the important properties and optimal policy structures of the model. We then propose a reinforcement learning approach for the data-driven approximation of the optimal booking policy to maximize shipping line revenue. The performance of the proposed approach is very close to that of the optimal policy and superior to that of the EMSR-b algorithm.

Optimal Maintenance of a Production-inventory System With Continuous Repair Times and Idle Periods

In this paper two similar models for the maintenance of a production-inventory system are considered. In both models, an input generating installation supplies a buffer with a raw material and a production unit pulls the raw material from the buffer. The installation in the first model and the production unit in the second model deteriorate stochastically over time and the problem of their optimal preventive maintenance is considered. In the first model, it is assumed that the installation, after the completion of its maintenance, remains idle until the buffer is evacuated, while in the second model, it is assumed that the production unit, after the completion of its maintenance, remains idle until the buffer is filled up. The preventive and corrective repair times of the installation in the first model and the preventive and corrective repair times of the production unit in the second model are continuous random variables with known probability density functions. Under a suitable cost structure, semi-Markov decision processes are considered for both models in order to find a policy that minimizes the long-run expected average cost per unit time. A great number of numerical examples provide strong evidence that, for each fixed buffer content, the average-cost optimal policy is of control-limit type in both models, i.e. it prescribes a preventive maintenance of the installation in the first model and a preventive maintenance of the production unit in the second model if and only if their degree of deterioration is greater than or equal to a critical level. Using the usual regenerative argument, the average cost of the optimal control-limit policy is computed exactly in both models. Four numerical examples are also presented in which the preventive and corrective repair times follow the Exponential, the Weibull, the Gamma and the Log-Normal distribution, respectively.

Optimal decision policy for deteriorating systems with on-line monitoring in a variable environment with selectable operations

ABSTRACTWe investigate the optimal decision-making problem for deteriorating systems operating in a variable environment with selectable operations. We assume that information about system deterioration and the surrounding environment are obtained by on-line monitoring. Using this information, the decision-maker selects the operation that will minimise the total expected cost over an infinite horizon. Since there are strong dynamic interactions among a deteriorating system, its surrounding environment, and the operation being executed, we formulate the optimal decision-making problem for this system as a non-stationary Markov decision process with both environment and operation as parameters. The properties of the resulting optimal expected cost function are examined, and the optimal decision policy for operation selection is found to be given by a two-dimenional control limit policy under certain conditions. The optimal control limit policy reduces the total operating costs and enhances the dynamic decision-making process for deteriorating systems operating in a variable environment.

Optimal control limit policy for age-dependent deteriorating systems under incomplete observations

The focus of this study is the optimal decision making problem for an age-dependent deteriorating system in which the deterioration unfolds as a non-stationary Markov process. The true deterioration state of the system cannot be known directly and is assumed to be observed incompletely by a monitor that provides information related to the true deterioration state stochastically. The optimal decision making problem is formulated as a partially observable Markov decision process. The optimal maintenance policy is investigated and the structural properties of the resulting optimal expected cost function are obtained. These structural properties establish the existence of an optimal control limit policy with respect to both the deterioration information vector and the age of the system under intuitively meaningful assumptions. The monotonic property of the control limits is also clarified.

Efficient solutions for a class of optimal stopping problems

We show that a practical class of optimal stopping problems can be solved in at most N iterations of policy iteration. Under a lower triangularity assumption, one iteration of Gauss–Seidel value iteration yields the solution, while a subadditivity assumption implies an optimal control limit policy and an O(logN) algorithm.

Control Limit Policy for Partially Observable Markov Decision Process Based on Stochastic Increasing Ordering

We investigated an optimal maintenance problem for a condition monitoring system that is formulated as a partially observable Markov decision process. In almost all previous research, the optimal control limit policy with respect to totally positive of order 2 (TP2) ordering of the state probability vectors was derived using the property of posterior probability. We call this approach “Approach PP.” Attempts to achieve an optimal control limit policy based on stochastic increasing (SI) ordering, which is weaker than TP2 ordering, ran into a difficulty. That is, the SI property of the posterior probability vectors could not be obtained, and this property is needed to derive results expected using Approach PP. We investigated the conditions for the SI property of posterior probability vectors and derived a necessary and sufficient condition that cannot be further relaxed when we try to derive an optimal control limit policy based on SI ordering using Approach PP. This condition is that the conditional probability matrix of the monitor observation given the state of the system is given as Type 1), the probabilities of the same monitoring output are the same whatever the true state, or Type 2), an identity matrix. Type 1) means that the monitoring output is independent of the true state.

Optimal maintenance of a production-inventory system with idle periods

In this paper we consider a production-inventory system in which an input generating installation supplies a buffer with a raw material and a production unit pulls the raw material from the buffer with constant rate. The installation deteriorates in time and the problem of its optimal preventive maintenance is considered. It is assumed that the installation after the completion of its maintenance remains idle until the buffer is evacuated. Under a suitable cost structure it is shown that the average-cost optimal policy for fixed buffer content is of control-limit type, i.e. it prescribes a preventive maintenance of the installation if and only if its degree of deterioration is greater than or equal to a critical level. Using the usual regenerative argument, the average cost of a control-limit policy is computed exactly and then, the optimal control-limit policy is determined. Furthermore, the stationary probabilities of the system under the optimal policy are computed.

Structural properties for a two-state partially observable Markov decision process with an average cost criterion

This paper investigates an optimal replacement problem for a discrete-time Markovian deterioration system. We consider partially observable Markov decision processes with finite core state and observation spaces and finite action set. The problem is to obtain an optimal replacement control-limit policy. Sufficient conditions, are then derived for a bounded solution to the average cost optimality equation to exist. Furthermore, under some reasonable conditions reflecting the practical meaning of the deterioration, structural properties for average costs policies are obtained for a two state replacement problem, and we develop a solution procedure utilizing these structural properties. We have that control permits working directly making it suitable for dealing with real-world processes and applications.

Optimal replacement of a system according toa semi-Markov decision process in a semi-Markov environment

This paper investigates an optimal replacement problem of a system in a semi-Markov environment. The system itself deteriorates according to a semi-Markov process, and is further influenced by its environment, which changes according to a semi-Markov process. Each change of the environment's state will change the parameters modelling the system and also cause damage on the system. For minimizing the discounted total costs with finite and infinite horizons, we show the existence of optimal control limit policies. A special case of Markov environment is discussed, and the state space can be simplified equivalently to be finite, so the real computation of the problem is feasible. Finally, a numerical example is given to prove the correctness and validity of the analysis.

A random walk model for a multi-component deteriorating system

We consider a Markovian l-unit system which is subject to shocks causing it to deteriorate in each of its stochastically dependent components. The net reward produced by the system is assumed to be an l-dimensional function of the amounts of deterioration of the components. After every shock the controller has the option to replace the system by a new one. The objective is to maximize the long-run average reward. Under natural conditions we prove the existence of an optimal control-limit policy and the unimodality of the long-term reward as a function of the threshold value.

The Optimal Control-Limit Policies for Buffer Systems with Different Recovery Functions

The Optimal Control-Limit Policies for Buffer Systems with Different Recovery Functions

A study on the optimal control limit policy for inspections of hard disk sectors

Presently, hard disks have become indispensable elements in most computer systems. If any malfunction occurs in the hard disk, valuable data can be lost; malfunctions such as sector failures can be caused by defects of the magnetic materials on the recording medium. When a sector failure occurs, the data in the failed sector are lost. However, the entire function can be restored to a normal condition by providing a process of replacement of the subject sector into a spare sector in advance. Since there is an upper limit to the number of spare sectors, a general failure can occur if the total number of sector failures exceeds the upper limit. Alternatively, an unstable state occurs in which the restoration of the information is not possible even though the entire structure appears to be working. It is rare that sector failure is discovered immediately. Instead, it is discovered when the hard disk is inspected. It is therefore possible to determine the total number of sector failures by periodic inspection and to avoid general failures and the extremely unstable states due to sector failure by preventive replacement. For the above situations, this paper proposes a control limit policy for the inspection of sector failure and verifies its effectiveness. © 1999 Scripta Technica, Electron Comm Jpn Pt 1, 82(10): 18–24, 1999

Bias Optimality in Controlled Queueing Systems

This paper studies an admission control M/M/1 queueing system. It shows that the only gain (average) optimal stationary policies with gain and bias which satisfy the optimality equation are of control limit type, that there are at most two and, if there are two, they occur consecutively. Conditions are provided which ensure the existence of two gain optimal control limit policies and are illustrated with an example. The main result is that bias optimality distinguishes these two gain optimal policies and that the larger of the two control limits is the unique bias optimal stationary policy. Consequently it is also Blackwell optimal. This result is established by appealing to the third optimality equation of the Markov decision process and some observations concerning the structure of solutions of the second optimality equation.

Bias Optimality in Controlled Queueing Systems

This paper studies an admission control M/M/1 queueing system. It shows that the only gain (average) optimal stationary policies with gain and bias which satisfy the optimality equation are of control limit type, that there are at most two and, if there are two, they occur consecutively. Conditions are provided which ensure the existence of two gain optimal control limit policies and are illustrated with an example. The main result is that bias optimality distinguishes these two gain optimal policies and that the larger of the two control limits is the unique bias optimal stationary policy. Consequently it is also Blackwell optimal. This result is established by appealing to the third optimality equation of the Markov decision process and some observations concerning the structure of solutions of the second optimality equation.

Determining Optimal Advertising Strategies: A Markov Decision Model Approach

This study falls in the class of models in which advertising wearout and the differences between the learning and forgetting of advertisements are explicitly included. A discrete time Markov decision modeling approach is used to obtain optimal control limit policies, and an algorithm is provided to identify these policies. A control limit policy specifies whether or not to advertise in a specific time period on the basis of the level of awareness in that time period. Thus, the duration for which advertising is not done is determined endogenously, and the algorithm helps determine this duration for a given set of parameters. This is a particularly desirable feature, since advertising practitioners are interested in determining the optimal duration of advertising pulses. Computational experience suggests that the algorithm is very fast and easy to implement. Also, conditions on model parameters indicating the relative efficacy of pulsing versus uniform advertising are provided.

A dynamic multi-item two-activity problem

We consider the following “symmetric” generalization of the well-known asset-selling problem. Assume that there are two activities, each of which must be carried out a given number of times. An example is the disposal ofi andj containers of toxic waste of two different kinds. Exactly one of the activities is performed per time period. The random rewards (possibly negative) for performing the different tasks are i.i.d. and have a known two-dimensional joint distribution. In addition, there are fixed costsc, c1 andc2 (possibly negative) per period as long as activities of both kinds, or only of the first or only of the second kind must be performed. Based on a realization of the two random rewards, each period a choice must be made which of the two activities to perform such that the expected total reward becomes maximal. We show for arbitrary discount factor the existence of an optimal control limit policy with recursively computable critical numbersd ij . Some explicitly solvable cases are presented. Under appropriate assumptions thed ij 's are monotone ini and/orj which allows, despite a 4-dimensional state space, to represent the optimal policy by a family of 2-dimensional monotone decision regions. And then, the limits of thed ij 's can be used to identify “large” sets of states where the optimal decision can be found without numerical computations. The basic tool is the integrated distribution function of the difference of the two rewards. The paper generalizes many of the known results on asset-selling problems without recall.

Opportunity-based age replacement: Exponentially distributed times between opportunities

textabstractThis article gives a full analysis of a component-replacement model in which preventive replacements are only possible at maintenance opportunities. These oppertunities arise according to a Poisson process, independently of failures of the component. Conditions for the existence of a unique average optimal control limit policy are established and an equation characterizing the optimal policy and minimal average costs is derived. An important result is that the optimal policy can be described as a so-called one-opportunity-look-ahead policy. Such policies play an important role as heuristic in more gernal models. It is shown that there is a correspondence with the will-known age-replacement model, which can be considered as an extreme case of the model. Finally, some numerical results are given.

The Optimal Look-Ahead Policy for Admission to a Single Server System

This paper considers a service system with a single server, finite waiting room, and a renewal arrival process. Customers who arrive while the server is busy are lost. Upon completing service, the server chooses between two actions: either he immediately starts a new service, provided a customer is present, or he admits the newly arrived customer to the system, but delays service pending the next arrival, whereupon he again chooses between these two actions. This process continues until either the system is full or a new service is started. Once a service has been started, all customers who arrive while the server is busy are lost. We assume that at each decision epoch the server knows the arrival epoch of the first arriving customer. We show that there exists an optimal control-limit policy that minimizes the average expected idle time per customer served (equivalently, maximizes the average number of customers served per unit of time). The special case of Poisson arrivals leads to an explicit expression for this delay that generalizes exisiting results.

On an optimal harvesting problem

Kaplan [1972] treated a harvesting problem as a discrete time stochastic control model with independent disturbances and with decreasing mean value increase depending either on the value or on the age of the asset. We consider the more general model where the mean value increase depends on the value and also on the age. As main result we obtain the existence of optimal control-limit policies with respect to three different natural orders in state space. A basic role play additional convexity and boundedness assumptions. Our findings extend and correct the main result of Kaplan. The paper contains further detailed information about the solution.