Continuous Transition Probabilities Research Articles

On Exit and Ergodicity of the Spectrally One-Sided Lévy Process Reflected at Its Infimum

Consider a spectrally one-sided Levy process X and reflect it at its past infimum I. Call this process Y. For spectrally positive X, Avram et al.(2) found an explicit expression for the law of the first time that Y=X−I crosses a finite positive level a. Here we determine the Laplace transform of this crossing time for Y, if X is spectrally negative. Subsequently, we find an expression for the resolvent measure for Y killed upon leaving [0,a]. We determine the exponential decay parameter ϱ for the transition probabilities of Y killed upon leaving [0,a], prove that this killed process is ϱ-positive and specify the ϱ-invariant function and measure. Restricting ourselves to the case where X has absolutely continuous transition probabilities, we also find the quasi-stationary distribution of this killed process. We construct then the process Y confined in [0,a] and prove some properties of this process.

Policy iteration type algorithms for recurrent state Markov decision processes

We introduce and analyze several new policy iteration type algorithms for average cost Markov decision processes (MDPs). We limit attention to “recurrent state” processes where there exists a state which is recurrent under all stationary policies, and our analysis applies to finite-state problems with compact constraint sets, continuous transition probability functions, and lower-semicontinuous cost functions. The analysis makes use of an underlying relationship between recurrent state MDPs and the so-called stochastic shortest path problems of Bertsekas and Tsitsiklis (Math. Oper. Res. 16(3) (1991) 580). After extending this relationship, we establish the convergence of the new policy iteration type algorithms either to optimality or to within ε>0 of the optimal average cost.

Symmetric Stable Laws and Stable-Like Jump-Diffusions

Asymptotic expansions are obtained for finite-dimensional symmetric stable distributions. They are used to prove the existence of continuous transition probability densities of stable and stable-like jump-diffusions, and to construct local multiplicative asymptotics and global two-sided estimates for these densities. As a consequence, the distribution of the first passage times for stable jump-diffusions is estimated and the integral test for the limsup behaviour of their sample paths as t → 0 is provided. 1991 Mathematics Subject Classification: 60E07, 60G17, 60J35, 47D07.

Completely asymmetric Lévy processes confined in a finite interval

Consider a completely asymmetric Lévy process which has absolutely continuous transition probabilities. By harmonic transform, we establish the existence of the Lévy process conditioned to stay in a finite interval, called the confined process (the confined Brownian motion is F.B. Knight's Brownian taboo process). We show that the confined process is positive-recurrent and specify some useful identities concerning its excursion measure away from a point. We investigate the rate of convergence of the supremum process to the right-end point of the interval.

Exponential decay and ergodicity of completely asymmetric Lévy processes in a finite interval

Consider a completely asymmetric Levy process which has absolutely continuous transition probabilities. We determine the exponential decay parameter $\rho$ and the quasistationary distribution for the transition probabilities of the Levy process killed as it exits from a finite interval, prove that the killed process is $\rho$-positive and specify the $\rho$-invariant function and measure.

Constrained Discounted Dynamic Programming

This paper deals with constrained optimization of Markov Decision Processes with a countable state space, compact action sets, continuous transition probabilities, and upper semicontinuous reward functions. The objective is to maximize the expected total discounted reward for one reward function, under several inequality constraints on similar criteria with other reward functions. Suppose a feasible policy exists for a problem with M constraints. We prove two results on the existence and structure of optimal policies. First, we show that there exists a randomized stationary optimal policy which requires at most M actions more than a nonrandomized stationary one. This result is known for several particular cases. Second, we prove that there exists an optimal policy which is (i) stationary (nonrandomized) from some step onward, (ii) randomized Markov before this step, but the total number of actions which are added by randomization is at most M, (iii) the total number of actions that are added by nonstationarity is at most M. We also establish Pareto optimality of policies from the two classes described above for multi-criteria problems. We describe an algorithm to compute optimal policies with properties (i)–(iii) for constrained problems. The policies that satisfy properties (i)–(iii) have the pleasing aesthetic property that the amount of randomization they require over any trajectory is restricted by the number of constraints. In contrast, a randomized stationary policy may require an infinite number of randomizations over time.

Kolmogorov's differential equations for non-stationary, countable state Markov processes with uniformly continuous transition probabilities

1. SummaryWe shall consider a non-stationary Markov chain on a countable state space E. The transition probabilities {P(s, t), 0 ≤ s ≤ t <t0 ≤ ∞} are assumed to be continuous in (s, t) uniformly in the state i ε E.

Atomic Absorption Coefficients and Transition Probabilities

THREE recent letters raise the question as to whether there is any experimental basis for applying Kramers' equation for continuous absorption to astrophysical problems1. Ditchburn cites the anomalous phenomena shown by alkalis at the absorption series limits as the only direct experimental evidence. A series of papers2 by me on the emission spectra of caesium vapour gives direct evidence as to the transition probabilities for the continuous spectra and line spectra of the levels 6 P and 5 D. The first paper indicated that the probability of recombination depended on the pressure but subsequent work showed this to be erroneous, and the last paper (published after the appearance of Ditch-burn's letter) shows that the continuous transition probabilities remain nearly constant with electron concentrations ranging from 1012 to 1016 per cm.3 and with pressures from 0.001 mm. to 17 mm. of mercury.