Discrete-time Problem Research Articles

Optimal equilibrium barrier strategies for time-inconsistent dividend problems in discrete time

This paper studies a time-inconsistent dividend problem in discrete time with nonexponential discounting. Motivated by the decreasing impatience in behaviour economics, a general discount function is used and assumed to be log sub-additive. Using a game-theoretic approach equilibrium barrier strategies are considered. It is shown that in the case of multiple equilibria, there exists an optimal one that pointwisely dominates all the other equilibria. Case studies are conducted where there is no equilibrium, multiple equilibria, and a unique equilibrium.

Multi-consensus of second-order agents in discrete-time directed networks

This paper deals with the multi-consensus problem in discrete-time directed networks with second-order agent dynamics. Necessary and sufficient conditions to ensure multi-equilibria consensus are obtained by utilising some concepts related to graph structures. It is shown that while the choice of control parameters directly affects the stability of the consensus problem, the equilibria states do not depend on them. Furthermore, the initial values of the states of the secondary layer subgraphs have no effect on the consensus equilibria of any agent in the network. A systematic method for choosing the control parameters is proposed and it is proved that this choice ensures multi-consensus of the network. Finally, illustrative examples are provided to better highlight the theoretical results.

Approximations for the Boundary Crossing Probabilities of Moving Sums of Random Variables

In this paper we study approximations for the boundary crossing probabilities of moving sums of i.i.d. normal random variables. We approximate a discrete time problem with a continuous time problem allowing us to apply established theory for stationary Gaussian processes. By then subsequently correcting approximations for discrete time, we show that the developed approximations are very accurate even for a small window length. Also, they have high accuracy when the original r.v. are not exactly normal and when the weights in the moving window are not all equal. We then provide accurate and simple approximations for ARL, the average run length until crossing the boundary.

Discrete-time Deployment of Agents on a Line Segment: Delays and Switches Do Not Matter

This paper considers the discrete-time problem of deploying agents on a line segment under delay in communication channels and switching and also under incomplete information about the value of the delay and the law of switching. It is shown that neither delay nor switching affect the convergence of the agents' states to the uniform allocation on the line segment. The theoretical results are illustrated by numerical simulation. The evidence is based on the well-known and new approaches to the stability analysis of positive systems.

Optimal Equilibrium Barrier Strategies for Time-Inconsistent Dividend Problems in Discrete Time

This paper studies a time-inconsistent dividend problem in discrete time with nonex- ponential discounting. Motivated by the decreasing impatience in behaviour economics, a general discount function is used and assumed to be log sub-additive. Using a game-theoretic approach equilibrium barrier strategies are considered. It is shown that in the case of multiple equilibria, there exists an optimal one that pointwisely dominates all the other equilibria. Case studies are conducted where there is no equilibrium, multiple equilibria, and a unique equilibrium.

Fast value iteration: an application of Legendre-Fenchel duality to a class of deterministic dynamic programming problems in discrete time

ABSTRACTWe propose an algorithm, which we call ‘Fast Value Iteration’ (FVI), to compute the value function of a deterministic infinite-horizon dynamic programming problem in discrete time. FVI is an efficient algorithm applicable to a class of multidimensional dynamic programming problems with concave return (or convex cost) functions and linear constraints. In this algorithm, a sequence of functions is generated starting from the zero function by repeatedly applying a simple algebraic rule involving the Legendre-Fenchel transform of the return function. The resulting sequence is guaranteed to converge, and the Legendre-Fenchel transform of the limiting function coincides with the value function.

Time-Consistent Investment and Reinsurance Strategies for Insurers Under Multi-Period Mean-Variance Formulation with Generalized Correlated Returns

The existing literature on investment and reinsurance is limited to the study of continuous-time problems, while discrete-time problems are always ignored by researchers. In this study, we first discuss a multi-period investment and reinsurance optimization problem under the classical mean-variance framework. When the asset returns with a serially correlated structure, the time-consistent investment and reinsurance strategies are acquired via backward induction. In addition, we propose an alternative time-consistent mean-variance optimization model that contrasts with the classical mean-variance model, and the corresponding optimal strategy and value function are also derived. We find that the investment and reinsurance strategies are both independent of the current wealth for the above two optimization problems, which coincides with the conclusion presented in the continuous-time problems. Most importantly, the above investment strategies with serially correlated structures are both conditional mean-based strategies, rather than unconditional ones. Finally, we compare the investment and reinsurance strategies suggested above based on the simulation approach, to shed light on which investment-reinsurance strategies are more suitable for insurers.

A New Convergent Pseudospectral Method for Delay Differential Equations

Delay differential equations have a wide range of applications in science and engineering. When these equations are nonlinear, we cannot usually obtain an exact solution. Hence, we must utilize an efficient numerical method with high convergence rate and low error to approximate the solution. In this paper, we propose a new shifted pseudospectral method to solve nonlinear delay differential equations. First, we convert the problem into an equivalent continuous-time optimization problem and then use a pseudospectral method to discretize the problem. By solving the obtained discrete-time problem, we achieve an approximate solution for the main delay differential equation. Here, we analyze the convergence of the method and solve some practical delay differential equations to show the efficiency of the method.

Optimal Investment-Consumption Decisions With Partially Observed Inflation: A Discrete-Time Formulation

We consider a discrete-time optimal consumption and investment problem of an investor who is interested in maximizing his utility from consumption and terminal wealth subject to a random inflation in the consumption basket price over time. We consider two cases: (i) when the investor observes the basket price and (ii) when he receives only noisy signals on the basket price. We derive the optimal policies and show that a modified Mutual Fund Theorem consisting of three funds holds in both cases, as it does in the continuous-time setting. The compositions of the funds in the two cases are the same but in general the investor’s allocations of his wealth into these funds differ.

Approximations for the boundary crossing probabilities of moving sums of normal random variables

In this article we study approximations for boundary crossing probabilities for the moving sums of i.i.d. normal random variables. We propose approximating a discrete time problem with a continuous time problem allowing us to apply developed theory for stationary Gaussian processes and to consider a number of approximations (some well known and some not). We bring particular attention to the strong performance of a newly developed approximation that corrects the use of continuous time results in a discrete time setting. Results of extensive numerical comparisons are reported. These results show that the developed approximation is very accurate even for small window length.

Time-consistent investment and reinsurance strategies for insurers under multi-period mean-variance formulation with generalized correlated returns

The existing literature on investment and reinsurance is limited to the study of continuous-time problems, while discrete-time problems are always ignored by researchers. In this study, we first discuss a multi-period investment and reinsurance optimization problem under the classical mean-variance framework. When the asset returns with a serially correlated structure, the time-consistent investment and reinsurance strategies are acquired via backward induction. In addition, we propose an alternative time-consistent mean-variance optimization model that contrasts with the classical mean-variance model, and the corresponding optimal strategy and value function are also derived. We find that the investment and reinsurance strategies are both independent of the current wealth for the above two optimization problems, which coincides with the conclusion presented in the continuous-time problems. Most importantly, the above investment strategies with serially correlated structures are both conditional mean-based strategies, rather than unconditional ones. Finally, we compare the investment and reinsurance strategies suggested above based on the simulation approach, to shed light on which investment-reinsurance strategies are more suitable for insurers.

AFMBC for a Class of Nonlinear Discrete-Time Systems with Dead Zone

This paper is fretful about an adaptive fuzzy model-based controller (AFMBC), which is studied and implemented for class of nonlinear discrete-time system with dead zone. Due to immeasurable states and the presence of symmetric/non-symmetric dead zones, design of controller becomes more challenging. AFMBC is design for approximation of such nonlinear system to a relative degree of accuracy, which can be used for adaptation of nonlinear discrete-time systems with or without the presence of symmetric/non-symmetric dead zones. AFMBC employs as a reference model which is useful to closed-loop pure feedback form of fuzzy controller. AFMBC provides approximation of immeasurable states and minimizes effects of unknown bounded disturbances in the system. Based on Lyapunov method, it is proved that proposed scheme for discrete-time nonlinear systems is asymptotically stable. Hence, not only stability of proposed system is assured, but it is also shows that tracking error of model lies in closed neighbourhood of zero after sufficient number of iterations, i.e. tracking error $$ (e(t) \to 0\;{\text{as}}\;t \to \infty ) $$ . The feasibility of the AFMBC is demonstrated by well-known direct current (DC) motor example and other nonlinear discrete-time problem through simulation.

A geometric approach for the optimal control of difference inclusions

Difference inclusions provide a discrete-time analogue of differential inclusions, which in turn play an important role in the theories of optimal control, implicit differential equations, and invariance and viability, to name a few. In this paper we: (i) introduce a framework suitable for the study of difference inclusions for which the state evolves on a manifold; (ii) use this framework to develop necessary conditions for optimality for a broad class of discrete-time problems of dynamic optimization in which the state evolves on a manifold M. The necessary conditions for optimality we derive include the case for which the state $$q_i$$ is subject to constraints $$q_i \in S_i \subseteq M$$ , for $$S_i$$ a closed set. The resulting necessary conditions for optimality appear as discrete-time versions of the Euler–Lagrange inclusion studied by Ioffe (in Trans Am Math Soc 349(7):2871–2900, 1997), Ioffe and Rockafellar (in Calc Var Partial Differ Equ 4(1):59–87, 1996), Mordukhovich (in SIAM J Control Optim 33(3):882–915, 1995), Mordukhovich (in Variational analysis and generalized differentiation II: applications. Springer, Berlin, 2006), and Vinter and Zheng (in SIAM J Control Optim 35(1):56–77, 1997) generalized in a natural way to the case in which the state is evolving on a manifold.

Exponential utility maximization under model uncertainty for unbounded endowments

We consider the robust exponential utility maximization problem in discrete time: An investor maximizes the worst case expected exponential utility with respect to a family of nondominated probabilistic models of her endowment by dynamically investing in a financial market, and statically in available options. We show that, for any measurable random endowment (regardless of whether the problem is finite or not) an optimal strategy exists, a dual representation in terms of (calibrated) martingale measures holds true, and that the problem satisfies the dynamic programming principle (in case of no options). Further it is shown that the value of the utility maximization problem converges to the robust superhedging price as the risk aversion parameter gets large, and examples of nondominated probabilistic models are discussed.

Management of a hydropower system via convex duality

We consider a stochastic hydroelectric power plant management problem in discrete time with arbitrary scenario space. The inflow to the system is some stochastic process, representing the precipitation to each dam. The manager can control how much water to turbine from each dam at each time. She would like to choose this in a way which maximizes the total profit from the initial time 0 to some terminal time T. The total profit of the hydropower dam system depends on the price of electricity, which is also a stochastic process. The manager must take this price process into account when controlling the draining process. However, we assume that the manager only has partial information of how the price process is formed. She can observe the price, but not the underlying processes determining it. By using the conjugate duality framework, we derive a dual problem to the management problem. This dual problem turns out to be simple to solve in the case where the profit rate process is a martingale or submartingale with respect to the filtration modeling the information of the dam manager. In the case where we only consider a finite number of scenarios, solving the dual problem is computationally more efficient than the primal problem.

Adaptive Robust Control under Model Uncertainty

In this paper we propose a new methodology for solving an uncertain stochastic Markovian control problem in discrete time. We call the proposed methodology the adaptive robust control. We demonstra...

The discrete-time tracking problem with H∞ model matching approach plus integral control

In this study, the discrete-time H∞ model matching problem with integral control by using 2 DOF static output feedback is presented. First, the motivation and the problem is stated. After presenting the notation, the two lemmas toward the discrete-time H∞ model matching problem with integral control are proven. The controller synthesis theorem and the controller design algorithm is elaborated in order to minimize the H∞ norm of the closed-loop transfer function and to maximize the closed-loop performance by introducing the model transfer matrix. In following, the discrete-time H∞ MMP via LMI approach is derived as the main result. The controller construction procedure is implemented by using a well-known toolbox to improve the usability of the presented results. Finally, some conclusions are given.

On the time discretization of stochastic optimal control problems: The dynamic programming approach

In this work, we consider the time discretization of stochastic optimal control problems. Under general assumptions on the data, we prove the convergence of the value functions associated with the discrete time problems to the value function of the original problem. Moreover, we prove that any sequence of optimal solutions of discrete problems is minimizing for the continuous one. As a consequence of the Dynamic Programming Principle for the discrete problems, the minimizing sequence can be taken in discrete time feedback form.

A general method for finding the optimal threshold in discrete time

ABSTRACTWe develop an approach for solving one-sided optimal stopping problems in discrete time for general underlying Markov processes on the real line. The main idea is to transform the problem into an auxiliary problem for the ladder height variables. In case that the original problem has a one-sided solution and the auxiliary problem has a monotone structure, the corresponding myopic stopping time is optimal for the original problem as well. This elementary line of argument directly leads to a characterization of the optimal boundary in the original problem. The optimal threshold is given by the threshold of the myopic stopping time in the auxiliary problem. Supplying also a sufficient condition for our approach to work, we obtain solutions for many prominent examples in the literature, among others the problems of Novikov-Shiryaev, Shepp-Shiryaev, and the American put in option pricing under general conditions. As a further application we show that for underlying random walks (and Lévy processes in continuous time), general monotone and log-concave reward functions g lead to one-sided stopping problems.

Off-Policy Interleaved Q -Learning: Optimal Control for Affine Nonlinear Discrete-Time Systems.

In this paper, a novel off-policy interleaved Q-learning algorithm is presented for solving optimal control problem of affine nonlinear discrete-time (DT) systems, using only the measured data along the system trajectories. Affine nonlinear feature of systems, unknown dynamics, and off-policy learning approach pose tremendous challenges on approximating optimal controllers. To this end, on-policy Q-learning method for optimal control of affine nonlinear DT systems is reviewed first, and its convergence is rigorously proven. The bias of solution to Q-function-based Bellman equation caused by adding probing noises to systems for satisfying persistent excitation is also analyzed when using on-policy Q-learning approach. Then, a behavior control policy is introduced followed by proposing an off-policy Q-learning algorithm. Meanwhile, the convergence of algorithm and no bias of solution to optimal control problem when adding probing noise to systems are investigated. Third, three neural networks run by the interleaved Q-learning approach in the actor-critic framework. Thus, a novel off-policy interleaved Q-learning algorithm is derived, and its convergence is proven. Simulation results are given to verify the effectiveness of the proposed method.