Stochastic Dynamic Programming Approach Research Articles

Robust optimal reinsurance and investment strategy for an insurer and a reinsurer with default risks and jumps

In this article, we analyze a robust optimal reinsurance and investment problem in a model with default risks and jumps for a general company which holds shares of an insurer and a reinsurer. The insurer’s surplus is characterized by a diffusion model and the insurer has the opportunity to buy proportional reinsurance from the reinsurer. Moreover, the insurer can invest in a risk-free asset (bank account), a risky asset (stock), and a defaultable bond. Meanwhile, the reinsurer can also invest in a risk-free asset and another risky asset. The price dynamics of the risky asset is assumed to follow a jump-diffusion model. What is more, the general company is concerned about the ambiguity and aims to find a robust optimal reinsurance and investment strategy. The objective of the general company is to maximize the minimal expected exponential utility of the weighted sums of the insurer’s and the reinsurer’s terminal wealth. Applying stochastic dynamic programming approach, we first establish the robust HJB (Hamilton-Jacobi-Bellman) equations for the post-default case and the pre-default case, respectively. Furthermore, we obtain the robust optimal strategy explicitly and present the corresponding verification theorem. Finally, we give some numerical examples to demonstrate the influences of model parameters on the robust optimal strategy.

Adaptive Double-Spending Attacks on PoW-Based Blockchains

Previous state-of-the-art studies have proposed various analytical models to understand the double-spending attacks (DSA) occurred in Proof-of-Work (PoW) based Blockchains. Although many insights behind the double-spending attacks have been disclosed, we still believe that advanced versions of DSA can be developed to create new threats for the PoW-based blockchains such as the Bitcoin blockchain. In this paper, we present two new types of double-spending attacks in the context of the PoW-based blockchain and discuss the insights behind them. By considering more practical network parameters, such as the number of confirmation blocks, the hashpower of the double-spending attacker, the amount of coins in the target transaction, and the network-status parameter, we first analyze the success probability of the conventional double-spending attack, named Naive DSA. Based on Naive DSA, we create two new adaptive DSA, i.e., the Adaptive DSA and the Reinforcement Adaptive DSA (RA-DSA). In our analytical models, a double-spending attack is converted into a Markov Decision Process. We then exploit the Stochastic Dynamic Programming (SDP) approach to obtain the optimal attack strategies under Adaptive DSA and RA-DSA. Numerical simulation results demonstrate the correlations between each critical network parameter and the expected attacker's reward. Through the proposed analytical models, we aim to alert the PoW-based blockchain ecosystem that the threat of double-spending attacks is still at a dangerous level. For example, our findings show that the attacker can launch a successful attack with a small hashpower proportion much lower than 51% under RA-DSA.

Robust time-consistent strategies of DC pension plans with the return of premiums clauses under inflation

. This article investigates the robust time-consistent investment strategy for a DC pension plan with model uncertainty under the mean-variance optimization objective. For the avoidance of reductions in investment returns due to inflation risk and premature death of members, an inflation-indexed bond is introduced into the financial market and a return of premium clause is incorporated into the model. By establishing extended Hamilton-Jacobi-Bellman (HJB) equations, the explicit solutions of both the robust precommitment strategy and the robust time-consistent strategy are presented by virtue of the robust optimal control theory and the stochastic dynamic programming approach. Furthermore, two degradation cases are derived in detail. Finally, a numerical example demonstrates the analysis of the obtained results.

A stochastic programming approach to the antibiotics time machine problem

Antibiotics Time Machine is an important problem to understand antibiotic resistance and how it can be reversed. Mathematically, it can be modeled as follows: Consider a set of genotypes, each of which contain a set of mutated and unmutated genes. Suppose that a set of growth rate measurements of each genotype under a set of antibiotics is given. The transition probabilities of a ‘realization’ of a Markov chain associated with each arc under each antibiotic are computable via a predefined function given the growth rate realizations. The aim is to maximize the expected probability of reaching to the genotype with all unmutated genes given the initial genotype in a predetermined number of transitions, considering the following two sources of uncertainties: (i) the randomness in growth rates, (ii) the randomness in transition probabilities, which are functions of growth rates. We develop stochastic mixed-integer linear programming and dynamic programming approaches to solve static and dynamic versions of the Antibiotics Time Machine Problem under the aforementioned uncertainties. We adapt a Sample Average Approximation approach that exploits the special structure of the problem and provide accurate solutions that perform very well in an out-of-sample analysis.

On Distributed Detection in EH-WSNs With Finite-State Markov Channel and Limited Feedback

We consider a network of N sensors, tasked with solving binary distributed detection, a fusion center (FC), and a feedback channel from the FC to sensors. Each sensor is capable of harvesting energy and is equipped with a finite-size battery to store randomly arrived energy. Sensors process their observations and transmit their symbols to the FC over orthogonal and Markovian time correlated fading channels. The FC fuses the received symbols and makes a global binary decision. We aim at developing adaptive channel-dependent transmit power control policies such that J-divergence based detection metric is maximized at the FC, subject to total transmit power constraint. Modeling quantized fading channel, energy arrival, and battery dynamics as time-homogeneous finite-state Markov chains, and the network lifetime as a geometric random variable, we formulate our power control optimization problem as a discounted infinite-horizon constrained Markov decision process (MDP) problem, where sensors’ transmit powers are functions of the battery states, quantized channel gains, and the arrived energies. We utilize stochastic dynamic programming and Lagrangian approach to find the optimal and sub-optimal power control policies. We demonstrate that our sub-optimal policy provides a close-to-optimal performance with a reduced computational complexity and without imposing signaling overhead on sensors.

Medium-Term Hydrothermal Scheduling of the Infiernillo Reservoir Using Stochastic Dual Dynamic Programming (SDDP): A Case Study in Mexico

This article aims to obtain and evaluate medium-term operating policies for the hydrothermal scheduling problem by using the stochastic dual dynamic programming (SDDP) approach. To this end, to feed the mathematical model and build the probability distribution functions that best fit each month of the actual inflow volume, monthly inflow data recorded from 1938 to 2018 for the Infiernillo reservoir located in Mexico were employed. Moreover, we simulated inflow volume scenarios using the Monte Carlo method for each month of a one-year planning period. The SDDP approach to solving the optimization problem consisted of the simulation of one forward scenario per iteration and the stabilization of the total operating cost as a convergence criterion, which results in an operating policy. We then assessed its quality by estimating the one-sided optimality gap. It is worth mentioning that the best operation policy required scenario trees of up to 17,000 inflow realizations per stage. Additionally, to study the evolution of the expected value along the planning horizon of the main variables involved in the medium-term hydrothermal scheduling problem, we simulated the best operation policy over 10,000 inflow scenarios. Finally, to show the practical value of the proposed approach, we report its computational complexity.

Valuation of an unexplored oilfield under uncertain oil price and reservoir condition: A stochastic dynamic programming approach with simulation-based reward function

We develop an unexplored oilfield valuation model under uncertain exploration outcomes, reservoir conditions, and oil prices. The exploration outcomes follow a binomial process, while oil prices are represented using the Schwartz-Smith model. The reservoir conditions are characterized by the joint probability distribution of post-discovery parameters estimated using a machine learning approach. The corresponding valuation model is a typical stochastic dynamic programming problem with a simulation-based reward function. The net cash flow lattice takes the form of a recombining quadrinomial derived from the discrete representation of the Schwartz-Smith oil price model. We apply backward induction with embedded real-coded genetic algorithms to the net cash flow lattice to calculate the oilfield value. The model allows for field abandonment before lease expiration if the remaining reserve is uneconomical. To improve computational efficiency, we combine the Latin hypercube sampling and antithetic variates to reduce the variances. The model is implemented in an unexplored oilfield under two scenarios of oil presence probability: 100% and 75%. In the first scenario, we obtained a mean oilfield value of US$5.5 million with a CVaR of US$0.79 million, while in the second, we came up with a mean of -US$0.77 million and a CVaR of US$31.74 million.

Dynamic surgery management under uncertainty

Real-time surgery management involves a complex and dynamic decision-making process. The duration of surgeries in many cases cannot be known until the surgery has actually been completed. Furthermore, disruptions such as equipment failure or the arrival of a non-elective surgery can occur simultaneously. Thus, the assignment of surgeries needs to be updated, as and when disruptions occur, to minimize their effects. In this paper, we present a stochastic dynamic programming approach to the surgery allocation problem with multiple operating rooms under uncertainty. Given an elective list for the day, the dynamic optimization model minimizes the number of surgeries not carried out by the end of the shift and the total waiting times of patients during the day weighted according to their urgency level. Due to the curse of dimensionality, we apply an approximate dynamic programming algorithm to solve the stochastic dynamic surgery management model. Computational experiments are designed to demonstrate the performance of the proposed algorithm and its applicability to practical settings. The results show that the approximate dynamic programming algorithm provides a good approximation to the optimum policy and leads to some managerial insights.

Robust optimal reinsurance strategy with correlated claims and competition

<abstract><p>This paper investigates the robust optimal reinsurance strategy, which simultaneously takes into account the ambiguity aversion, the correlated claims and the joint interests of an insurer and a reinsurer. The correlated claims mean that future claims are correlated with historical claims, which are measured by an extrapolative bias. The joint interests of the insurer and the reinsurer are reflected by the competition between them. To better reflect competition, we assume that the insurer and the reinsurer are engaged in related insurance business. The insurer is allowed to purchase proportional reinsurance or acquire a new business. Under ambiguity aversion and the criterion of maximizing the expected utility of terminal wealth, we obtain explicit solutions for the robust optimal reinsurance strategy and the corresponding value function by using the stochastic dynamic programming approach. Furthermore, we obtain the optimal reinsurance strategy under four typical cases. A series of numerical experiments were carried out to illustrate how the robust optimal reinsurance strategy varies with model parameters, and the result analyses reveal some interesting phenomena and provide useful guidance for reinsurance in reality.</p></abstract>

Robust optimal reinsurance-investment problem for $ n $ competitive and cooperative insurers under ambiguity aversion

<abstract><p>We investigate a robust optimal reinsurance-investment problem for $ n $ insurers under multiple interactions, which arise from the insurance market, the financial market, the competition mechanism and the cooperation mechanism. Each insurer's surplus process is assumed to follow a diffusion model, which is an approximation of the classical Cramér-Lundberg model. Each insurer is allowed to purchase proportional reinsurance to reduce their claim risk. To reflect the first moment and second moment information on claims, we use the variance premium principle to calculate reinsurance premiums. To increase wealth, each insurer can invest in a financial market, which includes one risk-free asset and $ n $ correlated stocks. Each insurer wants to obtain the robust optimal reinsurance and investment strategy under the mean-variance criterion. By applying a stochastic control technique and dynamic programming approach, the extended Hamilton-Jacobi-Bellman (HJB) equation is established. Furthermore, we derive both the robust optimal reinsurance-investment strategy and the corresponding value function by solving the extended HJB equation. Finally, we present numerical experiments, which yield that competition and cooperation have an important influence on the insurer's decision-making.</p></abstract>

A stochastic dynamic programming approach for the machine replacement problem

This paper addresses both the modeling and the resolution of the replacement problem for a population of machines. The main objective is the computation of a minimum cost replacement policy, which, based on the status of each machine, determines whether one or more machines have to be replaced over a given finite time horizon.The replacement problem of a set of machines can be regarded as a sequential decision-making problem under uncertainty. Thanks to this, we propose a novel formulation for such problems consisting of a composition of discrete-time multi-state Markov Decision Processes (MDPs), one for each specific machine. The underlying optimization problem is formulated as a stochastic Dynamic Programming (DP), and then solved by using the principles of the backward DP algorithm. Moreover, to deal with the curse of dimensionality due to the high-cardinality state–space of real-world/industrial applications, a new generalized multi-trajectory Least-Squares Temporal Difference (LSTD) based method is introduced. The resulting algorithm computes an approximate optimal cost function by: (i) running Monte Carlo simulations over different trajectories of a given length; (ii) embedding the policy improvement step within the recursive LSTD iterations; (iii) enforcing an off-policy mechanism to improve the LSTD exploration capabilities. A study on the convergence properties of the proposed approach is also provided. Several numerical examples are given to illustrate its effectiveness in terms of parametric sensitivity, computational burden, and performance of the computed policies compared with some heuristics defined in the literature.

Cooperative Output Regulation Quadratic Control for Discrete-Time Heterogeneous Multiagent Markov Jump Systems.

This article investigates the cooperative output regulation problem for discrete-time heterogeneous multiagent Markov jump systems. Two cases are studied: 1) output regulation quadratic control in the case where the exosystem is accessible to all agents and 2) cooperative output regulation quadratic control in the case where only a part of agents can directly communicate with the exosystem. The hidden Markov models are employed to describe the asynchronous modes of the agents and their corresponding controllers. Via the jumping regulator equation, asynchronous control laws are constructed and the algorithms to obtain control parameters are presented in terms of linear matrix inequalities. For the first case, the optimal synchronous/mode-dependent control law, which is a special case of the asynchronous control protocol, is also given via the stochastic dynamic programming approach. Finally, an example is given to illustrate the effectiveness of the proposed approaches.

Online learning for distributed optimal control of an electric vehicle fleet

The management of electrical power systems requires the resolution of large-scale problems whose agents are linked by coupling constraints. Nevertheless, decomposition methods cannot provide an exact solution while dealing with temporal dynamics in a stochastic environment. Indeed, each agent would have to solve a local minimisation in which future quantities intervene. However, these quantities depend on other agents’ future decisions which are still unknown. In order to enhance the existing approximate approaches to this challenge, the proposed method involves Alternating Direction Method of Multipliers to overcome the large dimension by an iterative resolution of local coordinated minimisations. Uncertain temporal dynamics are handled by a stochastic dynamic programming approach. In order to make local problems solvable, the online learning of a Markov process is added. The agents can then anticipate future global variations in a local probabilistic way. The optimal charging of an electric vehicle fleet paired with a wind power plant is considered as a case study. The expected benefits are highlighted, both at the outset and after training the anticipative models. The discussion addresses the learning parameters allowing the fastest convergence.

Optimization of a domestic microgrid equipped with solar panel and battery: Model Predictive Control and Stochastic Dual Dynamic Programming approaches

In this study, a microgrid with storage (battery, hot water tank) and solar panel is considered. We benchmark two algorithms, MPC and SDDP, that yield online policies to manage the microgrid, and compare them with a rule based policy. Model Predictive Control (MPC) is a well-known algorithm which models the future uncertainties with a deterministic forecast. By contrast, Stochastic Dual Dynamic Programming (SDDP) models the future uncertainties as stagewise independent random variables with known probability distributions. We present a scheme, based on out-of-sample validation, to fairly compare the two online policies yielded by MPC and SDDP. Our numerical studies put to light that MPC and SDDP achieve significant gains compared to the rule based policy, and that SDDP overperforms MPC not only on average but on most of the out-of-sample assessment scenarios.

Designing Watersheds for Integrated Development (DWID): A stochastic dynamic optimization approach for understanding expected land use changes to meet potential water quality regulations

This study investigates the tradeoff between agricultural economic losses and potential water quality regulations at the watershed level over time. Specifically, we applied a stochastic dynamic mathematical programming approach to evaluate the impact of land use changes on agricultural economic losses under uncertain water quality regulations and crop yields. We selected the Little River Experimental Watershed in South Georgia as a case study. Seven potential water pollution reduction trajectories for total phosphorus (TP) were developed. The proposed model applied a chance-constraint technique to allow some degree of uncertainty in meeting the water quality standards. The result shows that agricultural profits decrease when uncertainties are considered at the watershed level. The findings further show that uncertainties of meeting the water quality standard expedited the land use transition from croplands to forestlands and magnified the share of forestlands, highlighting the crucial role of forests in managing water pollution in the selected watershed. The results also indicate that under a moderate pollution reduction scenario, an economic loss of $232/ha/yr is expected to achieve a 16 % reduction in TP concentration. In a more ambitious pollution mitigation scenario, agricultural economic losses are expected to be $778/ha/yr and may result in a 45 % decrease in TP concentration. The modeling approach developed in Designing Watersheds for Integrated Development (DWID) could easily feed into those policy deliberations which operate at the interface of economics and water-related policies for ensuring sustainable management of watersheds worldwide.

Optimal joint production, maintenance and product quality control policies for a continuously deteriorating manufacturing system

ABSTRACT This article presents a study with the aim of optimizing a production system, consisting of a machine that produces a single type of parts, that degrades over time. This degradation affects the availability of the deployed machine and increases the defective rate of the manufactured products. The machine is subject to random failures and repairs. It has a defective rate which increases with its degradation. An overhaul of the machine allows to reduce this defective rate and bring it back to its initial condition. The objective of this study is to find a joint policy for production, maintenance and quality control, in order to increase the availability of the machine, improve the quality of the manufactured products and minimize the total cost of production. To achieve this goal, we formulated the research problem and used a stochastic dynamic programming approach to develop the Hamilton-Jacobi-Bellman (HJB) type optimum conditions. Then, we simulated a practical hybrid application to optimize the production of a Router class R CNC cutting machine that transforms polypropylene sheets in a Quebec (Canada) company specialized in the production of fire pumps. The obtained results allowed us to propose a critical threshold production policy, corrective and preventive maintenance strategies, and a sampling type of quality control to the company. Finally, we performed a sensitivity analysis to ensure the validation of the proposed policies.

No Longer in the Dark: Utilizing Imperfect Advance Load Information for Single-Truck Operators

This study investigates how imperfect advance load information (IALI) can improve profits and other operational indicators, such as empty movements, for a single-truck company. To analyze the value of IALI, we first develop a deterministic mathematical model. Then, we propose a stochastic dynamic programming approach that can utilize IALI. After designing a comprehensive set of experiments, we employ both models using a dynamic implementation mechanism to assess the benefits of using IALI. Our statistical analysis reveals that (1) utilizing IALI can significantly improve a single-truck company’s profits, by as much as almost 30% on average, and (2) the impact of using IALI can be affected by other factors (e.g., network size). In another set of experiments, we examine the benefits of IALI in a new environment where there are two classes of shippers, high risk and low risk. The results suggest that the potential benefits can be even larger with two classes of shippers. Last, we collect data over two three-week periods for a single-truck company that operates in Ontario, Canada, and we apply our methods for evaluating the benefits of IALI.

Robust optimal asset-liability management with penalization on ambiguity

<p style='text-indent:20px;'>In this paper, we study the robust optimal asset- problems for an ambiguity-averse investor, who does not have perfect information in the drift terms of the risky asset and liability processes. Two different kinds of objectives are considered: <inline-formula><tex-math id="M1">\begin{document}$ (i) $\end{document}</tex-math></inline-formula> Maximizing the minimal expected utility of the terminal wealth; <inline-formula><tex-math id="M2">\begin{document}$ (ii) $\end{document}</tex-math></inline-formula> Minimizing the maximal cumulative deviation. The ambiguity in both problems is described by a set of equivalent measures to the reference model. By the stochastic dynamic programming approach and Hamilton-Jacobi-Bellman (HJB) equation, we derive closed-form expressions for the value function and corresponding robust optimal investment strategy in each problem. Furthermore, some special cases are provided to investigate the effect of model uncertainty on the optimal investment strategy. Finally, the economic implication and parameter sensitivity are analyzed by some numerical examples. We also compare the robust optimal investment strategies in two different problems.</p>

Valuation of an Unexplored Oilfield Under Uncertain Oil Price and Reservoir Condition: A Stochastic Dynamic Programming Approach with Simulation-Based Reward Function

Valuation of an Unexplored Oilfield Under Uncertain Oil Price and Reservoir Condition: A Stochastic Dynamic Programming Approach with Simulation-Based Reward Function

The impact of water conservation regulations on mining firms: A stochastic control approach

Large water demands by the mining industry are of increasing concern around the world. The cost of a specific water management regulation is studied for an oil sands mining operation in Canada, where restrictions on water withdrawals vary with fluctuations in the river. A stochastic optimal control problem is formulated for a firm choosing production, water use, and the timing to build a water storage facility, under conditions of uncertain oil prices and uncertain water withdrawal limits. As no closed form solution is available, a stochastic dynamic programming approach is implemented to determine the difference in value and optimal controls for the oil-producing asset, with and without water restrictions. The cost of the restrictions is estimated to be quite small given historical river flow conditions, while cost is shown to increase under drier conditions. A long run marginal cost curve is developed showing the cost of increasing restrictions given expectations about future river conditions and oil prices.