Infinite Planning Horizon Research Articles

Drift Control of High-Dimensional Reflected Brownian Motion: A Computational Method Based on Neural Networks

Motivated by applications in queueing theory, we consider a stochastic control problem whose state space is the d-dimensional positive orthant. The controlled process Z evolves as a reflected Brownian motion whose covariance matrix is exogenously specified, as are its directions of reflection from the orthant’s boundary surfaces. A system manager chooses a drift vector [Formula: see text] at each time t based on the history of Z, and the cost rate at time t depends on both [Formula: see text] and [Formula: see text]. In our initial problem formulation, the objective is to minimize expected discounted cost over an infinite planning horizon, after which we treat the corresponding ergodic control problem. Extending the earlier work by Han et al. [Han J, Jentzen A, Weinan E (2018) Solving high-dimensional partial differential equations using deep learning. Proc. Natl. Acad. Sci. USA 115(34):8505–8510], we develop and illustrate a simulation-based computational method that relies heavily on deep neural network technology. For the test problems studied thus far, our method is accurate to within a fraction of 1% and is computationally feasible in dimensions up to at least [Formula: see text].

Optimal Control of Harvesting of a Distributed Renewable Resource on the Earth’s Surface

This paper is devoted to the optimal control of mixed (stationary and periodic impulse) harvesting of a renewable resource distributed on the Earth’s surface. Examples of such a resource are biological populations, including viruses, chemical contaminants, dust particles, and the like. It is proved that on an infinite planning horizon, there exists an admissible control ensuring the maximum of time-averaged harvesting.

Condition-based maintenance policy for systems under dynamic environment

This paper addresses the challenge of optimizing maintenance for systems with varying degradation characteristics in dynamic environment. The focus is on repairable systems, where degradation is characterized by a Wiener process that incorporates the dynamic influence of the environment. We investigate a condition-based maintenance model aiming at minimizing the expected cost over a finite and infinite horizon. Firstly, the model considers the evolution of environment as a Markov process, which influences the drift parameter in the Wiener process. The choice between a corrective and preventive replacement rests on the periodic inspections on the system and environment. Subsequently, the maintenance model is formulated in the framework of Markov decision process. Meanwhile, the backward dynamic programming algorithm and value iteration algorithm are employed to gain the optimal maintenance policies over the finite and the infinite planning horizons, respectively. Finally, the proposed model is further explained with concrete examples and a comprehensive sensitivity analysis. Numerical results clearly demonstrate notable variations in the optimal maintenance strategies in different periods and underscore the importance of taking proactive maintenance measures in harsh environmental conditions, even if the degradation level of the system is relatively low.

Carbon Emissions Effect on Vendor-Managed Inventory System Considering Displaced Re-Start-Up Production Time

Background: The classical mathematical formulation of the vendor-managed inventory (VMI) model assumes an infinite planning horizon, and consequently, the solution derived ignored the impact of the first cycle. The classical formulation is associated with another implicit assumption that input parameters remain static indefinitely. Methods: This paper develops two mathematical models for VMI for a joint economic lot-sizing (JELS) policy. Each model considers investment in green production, energy used for keeping items in storage, and carbon emissions from production, storage, and transportation activities under the carbon cap-and-trade policy. The first model underlies the first cycle, while the second underlies subsequent cycles. Results: The re-start-up production time for subsequent cycles commences only at the time required to produce and replenish the first lot, which implies further cost reduction. Mathematical formulations are perceived as important both for academics and practitioners. For example, the base model of the first cycle (subsequent cycles) generates an optimal produced quantity with 18.42% (4.35%) less total system cost when compared with the pest scenario in favor of the existing literature. Moreover, such a percentage of total system cost reduction increases as the production rate increases. Further, the proposed models not only produce better results but also offer the opportunity to adjust the input parameters for subsequent cycles, where each cycle is independent from the previous one. Conclusions: The emissions generated by the system are very much related to the demand rate and the amount of investment in green production. Illustrative examples, special cases, model overview, and managerial insights are given. The discussion related to the contribution of the proposed model, the concluding remarks, and further research are also provided. The proposed model rectifies the base model adopted by the existing literature, which can be further extended to be implemented in several interesting further inquiries related to JELS inventory mathematical modeling.

Data Analytics of an Information System Based on a Markov Decision Process and a Partially Observable Markov Decision Process

Data analytics of an information system is conducted based on a Markov decision process (MDP) and a partially observable Markov decision process (POMDP) in this paper. Data analytics over a finite planning horizon and an infinite planning horizon for a discounted MDP is performed, respectively. Value iteration (VI), policy iteration (PI), and Q-learning are utilized in the data analytics for a discounted MDP over an infinite planning horizon to evaluate the validity of the MDP model. The optimal policy to minimize the total expected cost of states of the information system is obtained based on the MDP. In the analytics for a discounted POMDP over an infinite planning horizon of the information system, the effects of various parameters on the total expected cost of the information system are studied.

Analysis of a growth model with a production CES-function

The paper investigates a growth model with a production function of constant elasticity of substitution, which generalizes such functions as Cobb-Douglas or Leontief, when one of its parameter tends to zero or infinity respectively. The investment indicators of the model are considered as control parameters that are chosen in order to maximize the utility functional. An optimal control problem with an infinite planning horizon is formulated. Using the Pontryagin maximum principle, the paper studies a Hamiltonian function and a Hamiltonian system and provides its qualitative analysis. Next, the existence and uniqueness of a steady state are proven, and an algorithm for its search is given by solving a nonlinear equation of one special variable. The following section performs the stabilization of the Hamiltonian system in the vicinity of the steady state using a controller that can be constructed due to saddle character of the steady state. Finally, a numerical example is given that illustrates obtained analytical results.

Predictive Modelling of a Honeypot System Based on a Markov Decision Process and a Partially Observable Markov Decision Process

A honeypot is used to attract and monitor attacker activities and capture valuable information that can be used to help practice good cybersecurity. Predictive modelling of a honeypot system based on a Markov decision process (MDP) and a partially observable Markov decision process (POMDP) is performed in this paper. Analyses over a finite planning horizon and an infinite planning horizon for a discounted MDP are respectively conducted. Four methods, including value iteration (VI), policy iteration (PI), linear programming (LP), and Q-learning, are used in the analyses over an infinite planning horizon for the discounted MDP. The results of the various methods are compared to evaluate the validity of the created MDP model and the parameters in the model. The optimal policy to maximise the total expected reward of the states of the honeypot system is achieved, based on the MDP model employed. In the modelling over an infinite planning horizon for the discounted POMDP of the honeypot system, the effects of the observation probability of receiving commands, the probability of attacking the honeypot, the probability of the honeypot being disclosed, and transition rewards on the total expected reward of the honeypot system are studied.

A joint replenishment problem with the (T,ki) policy under obsolescence

Companies are frequently confronted with the need to order different types of items from a single supplier or to manufacture the items in a production line. Indeed, coordinated ordering of multiple items may lead to important savings whenever a family of items can be ordered from a common supplier, produced in a common facility, or use a common mode of transportation. The Joint Replenishment Problem (JRP) tackles the coordinated replenishment of multiple items by minimizing the total cost, composed of ordering (or setup) costs and holding costs, while satisfying the demand. On the other hand, when items are subject to obsolescence, they may face an abrupt decline in demand as they are no longer needed. This decline can be caused by reasons such as rapid advancements in technology, going out of fashion, or ceasing to be economically viable. The present article develops an extension of the JRP where the items may suddenly become obsolete during an infinite planning horizon. The point at which an item becomes obsolete is uncertain. The lifetimes of the items are assumed to follow independent negative exponential distributions. A model is proposed by using the total expected discounted cost as the minimization criterion. The time value of money is considered through an appropriate discount rate. Extensive tests were performed to assess the impact of obsolescence rates and discount rates on the ordering policies. The progressive increase of the obsolescence rates determines smaller periods between successive replenishments, while the progressive increase of the discount rate determines smaller lot sizes.

A mathematical model for optimizing pricing-inventory, and advertising frequency decisions with a multivariate demand function and a time-dependent holding-cost function

ABSTRACT This article investigates inventory replenishment, advertising, and pricing decisions for perishable products with fixed expiration dates. Specifically, the problem is modelled using mixed-integer nonlinear programming to maximize the retailer’s total profit on an infinite planning horizon. Several assumptions are made to formulate the problem: i) the demand is considered a multivariate function of freshness, selling price, displayed inventory level, and advertisement frequency; ii) the holding cost of inventory increases linearly as the product ages; iii) the ending inventory is not necessarily zero. A search algorithm is proposed to solve this problem. The impact of each parameter on the optimal solution is investigated using sensitivity analysis. The advertising frequency determined via this model is an optimal compromise between advertising costs and yield. This model can be applied to reduce perishable food waste and meet the rising demand for fresh food.

A Lot-Sizing Model for a Multi-State System with Deteriorating Items, Variable Production Rate, and Imperfect Quality

Conventional production systems assume that during the manufacturing processes, machines operate without breakdown over an infinite planning horizon and manufacture only products of good quality. Imperfect production processes as a result of machine degradation are common in manufacturing. This paper deals with a problem that concerns the modelling and evaluation of the performance of a multi-state production system that is subject to degradation and its effect on lot sizing. Here, we consider that the cycle starts with a particular production rate until a point when the inventory reaches a certain level after which the failure mode is activated due to the deterioration of certain components, leading to a reduction in the production rate in order to ensure the continuity of supply until the maximum inventory level is reached. Production then stops to restore the machine and the cycle starts again. We have assumed that the rate at which inventory deteriorates is exponential and that demand is constant. A numerical example is used to illustrate the model application, followed by sensitivity analysis. This paper contributes to lot sizing in the area of machine reliability by considering a production system in a degraded state with a non-increasing production rate for deteriorating items with imperfect quality and partial backlogging.

Maintenance Policies for Two-Unit Balanced Systems Subject to Degradation

In this article, we present a novel maintenance policy optimization method for systems with two balanced components. The components in the system are assumed to degrade over time according to a bivariate Wiener process. The maintenance actions aim at eliminating the differences of degradation levels of system components at the cost of aggravating the degradation. Utilizing the Markov decision process, the maintenance model is put forward under both the finite and the infinite planning horizons, from which we find the structural properties of the optimal policies. Backwards dynamic programming and value iteration algorithms are employed to optimize the maintenance decisions. Examples along with sensitivity analysis are presented to facilitate the illustration and insight attainment. We find that the maintenance policies are to a great extent regulated by the absolute degradation difference between the two components.

An EOQ model for non-instantaneous deteriorating items with time-dependent demand under partial backlogging

This article discusses an economic order quantity model for non-instantaneous deteriorating items in which the demand is assumed to be a linear function of time over an infinite planning horizon. In addition, the salvage value associated with the deteriorated units is also considered. The shortages are allowed and partially backlogged. A mathematical model is framed to obtain the replenishment policy which aids the retailer to minimize the total inventory cost. The objective of this work is to minimize the total inventory cost and to find the optimal length of replenishment and the optimal order quantity. The theory developed in this article is illustrated using numerical examples. A computational algorithm is designed to find the optimal solution. Sensitivity analysis is carried out to study the changes in the effect on the optimal solutions and some managerial insights are obtained.

Optimal policies for a deterministic continuous-time inventory model with several suppliers: when a supplier incurs no set-up cost

The subject is a deterministic continuous-time continuous-state inventory control model. Stock is replenished by ordering from one of a number of suppliers incurring a different cost per item and a different set-up cost. Taking the cost of procurement into account, the objective is to minimize the total discounted cost over an infinite planning horizon. The size of the order that is to be placed and the supplier with which it is to be placed are to be decided. Earlier studies of the problem have relied substantially on the assumption that the set-up cost of every supplier is strictly positive. Removing this restriction calls for a significant modification of the adopted approach. This is realized in the present study. It is shown that there is a stable unique optimal policy of a type that encompasses (s, S) and generalized (s, S) policies. Conditions that are necessary and sufficient for it to reduce to each of these types are established. The case of two suppliers is studied in detail, properties of the solution are investigated, numerical examples illustrating various aspects are included, and the connection with antecedent results is assessed.

Economic Production Quantity Model for Deteriorating Items under the Constant and Non Linear GHG Emission

Industry is becoming the largest source of GHG emissions, requiring action from industrialists. The model provided here is developed to address two different scenarios of GHG emission. This paper examines a model for producing inventory to meet the demand rate intended to operate on an infinite planning horizon where the demand rate for deteriorating items is determined by a function based on the on-hand inventory, given that the production rate is linearly related to the demand rate. In the forward manufacturing system, linear and nonlinear GHG emission costs are compared to observe the effects of nonlinearity in emission costs. At last numerical verification is used to illustrate the mathematical expressions, and the sensitivity of the different parameters is examined to draw some managerial conclusions.

A general approach for the derivation of optimal solutions without derivatives

In recent years, there have been a number of approaches in the literature that determine closed form optimal solutions to some classical inventory problems and their various extensions without using derivatives. Earliest of such approaches are based on algebraic manipulations with some a priori assumptions about the optimal solutions or based on properties of the quadratic function, whereas more recent approaches are based on problem specific assumptions such as an infinite planning horizon and integer decision variables, and analysing limiting behaviour of the decision variables and the objective function. None of the approaches is extendable to other similar optimisation problems. In this paper, we propose a general approach without any a priori assumptions of the optimal solution and without any problem specific assumptions. We demonstrate the proposed approach by applying it to some classical problems such as the EOQ and EPQ, with and without planned backorders.

Dynamic reinsurance in discrete time minimizing the insurer's cost of capital

In the classical static optimal reinsurance problem, the cost of capital for the insurer's risk exposure determined by a monetary risk measure is minimized over the class of reinsurance treaties represented by increasing Lipschitz retained loss functions. In this paper, we consider a dynamic extension of this reinsurance problem in discrete time which can be viewed as a risk-sensitive Markov Decision Process. The model allows for both insurance claims and premium income to be stochastic and operates with general risk measures and premium principles. We derive the Bellman equation and show the existence of a Markovian optimal reinsurance policy. Under an infinite planning horizon, the model is shown to be contractive and the optimal reinsurance policy to be stationary. The results are illustrated with examples where the optimal policy can be determined explicitly.

Minimizing spectral risk measures applied to Markov decision processes

We study the minimization of a spectral risk measure of the total discounted cost generated by a Markov Decision Process (MDP) over a finite or infinite planning horizon. The MDP is assumed to have Borel state and action spaces and the cost function may be unbounded above. The optimization problem is split into two minimization problems using an infimum representation for spectral risk measures. We show that the inner minimization problem can be solved as an ordinary MDP on an extended state space and give sufficient conditions under which an optimal policy exists. Regarding the infinite dimensional outer minimization problem, we prove the existence of a solution and derive an algorithm for its numerical approximation. Our results include the findings in Bäuerle and Ott (Math Methods Oper Res 74(3):361–379, 2011) in the special case that the risk measure is Expected Shortfall. As an application, we present a dynamic extension of the classical static optimal reinsurance problem, where an insurance company minimizes its cost of capital.

Optimal pricing policy for non-instantaneous deteriorating items with multivariate demand under infinite planning horizon

Optimal pricing policy for non-instantaneous deteriorating items with multivariate demand under infinite planning horizon

Optimal policies for a deterministic continuous-time inventory model with several suppliers: a hyper-generalized (s,S) policy

A deterministic continuous-time continuous-state inventory model is studied. In the absence of intervention, the level of stock evolves by a process governed by a differential equation. The inventory level is monitored continuously, and can be adjusted upwards at any time. The decision maker can order from several suppliers, each of which charges a different ordering and purchasing cost. The problem of selecting the supplier and the size of the order to minimize the total inventory cost over an infinite planning horizon is formulated as the solution of a quasi-variational inequality (QVI). It is shown that the QVI has a unique solution. This corresponds to a generalized (s,S) policy under amenable conditions, which have been characterized in an earlier work by the present authors. Under the complementary conditions a new type of optimal control policy emerges. This leads to the concept of a hyper-generalized (s,S) policy. The theory behind a policy of this type is exposed.

Markov decision processes with recursive risk measures

In this paper, we consider risk-sensitive Markov Decision Processes (MDPs) with Borel state and action spaces and unbounded cost. We treat both finite and infinite planning horizons. Our optimality criterion is based on the recursive application of static risk measures. This is motivated by recursive utilities in the economic literature. It has been studied before for the entropic risk measure and is extended here to general static risk measures. Under direct assumptions on the model data we derive a Bellman equation and prove the existence of optimal Markov policies. For an infinite planning horizon, the model is shown to be contractive and the optimal policy to be stationary. Our approach unifies results for a number of well-known risk measures. Moreover, we establish a connection to distributionally robust MDPs, which provides a global interpretation of the recursively defined objective function. Monotone models are studied in particular.