Finite-state Markov Process Research Articles

Opportunistic File Transfer Over a Fading Channel Under Energy and Delay Constraints

We consider transmission control (rate and power) strategies for transferring a fixed-size file (finite number of bits) over fading channels under constraints on both transmit energy and transmission delay. The goal is to maximize the probability of successfully transferring the entire file over a time-varying wireless channel modeled as a finite-state Markov process. We study two implementations regarding the delay constraints: an average delay constraint and a strict delay constraint. We also investigate the performance degradation caused by the imperfect (delayed or erroneous) channel knowledge. The resulting optimal policies are shown to be a function of the channel-state information (CSI), the residual battery energy, and the number of residual information bits in the transmit buffer. It is observed that the probability of successful file transfer increases significantly when the CSI is exploited opportunistically. When the perfect instantaneous CSI is available at the transmitter, the faster channel variations increase the success probability under delay constraints. In addition, when considering the power expenditure in the pilot for channel estimation, the optimal policy shows that the transmitter should use the pilot only if there is sufficient energy left for packet transfer; otherwise, a channel-independent policy should be used.

Discrete denoising for channels with memory

We consider the problem of estimating a discrete signal $X^n = (X_1, \ldots, X_n)$ based on its noise-corrupted observation signal $Z^n = (Z_1, \ldots, Z_n)$. The noise-free, noisy, and reconstruction signals are all assumed to have components taking values in the same finite $M$-ary alphabet $\{0, \ldots, M-1 \}$. For concreteness we focus on the additive noise channel $Z_i = X_i + N_i$, where addition is modulo-$M$, and $\{N_i\}$ is the noise process. The cumulative loss is measured by a given loss function. The distribution of the noise is assumed known, and may have memory restricted only to stationarity and a mild mixing condition. We develop a sequence of denoisers (indexed by the block length $n$) which we show to be asymptotically universal in both a semi-stochastic setting (where the noiseless signal is an individual sequence) and in a fully stochastic setting (where the noiseless signal is emitted from a stationary source). It is detailed how the problem formulation, denoising schemes, and performance guarantees carry over to non-additive channels, as well as to higher-dimensional data arrays. The proposed schemes are shown to be computationally implementable. We also discuss a variation on these schemes that is likely to do well on data of moderate size. We conclude with a report of experimental results for the binary burst noise channel, where the noise is a finite-state hidden Markov process (FS-HMP), and a finite-state hidden Markov random field (FS-HMRF), in the respective cases of one- and two-dimensional data. These support the theoretical predictions and show that, in practice, there is much to be gained by taking the channel memory into account.

Peeling phylogenetic ‘oranges’

We investigate the combinatorics of a topological space that is generated by the set of edge-weighted finite trees. This space arises by multiplying the weights of edges on paths in trees and is closely connected to tree reconstruction problems involving finite state Markov processes. We show that this space is a contractible finite CW-complex whose face poset can be described via a partial order on semilabelled forests. We then describe some combinatorial properties of this poset, showing that, for example, it is pure, thin and contractible.

Analysis and Estimation of the States of Special Jump Markov Processes. II. Optimal Filtration in Wiener Noise

The second part was devoted to the rms-optimal filtration of the states of the Markov jump processes in continuous time which generalize the finite-state Markov processes. Equations for the conditional expectations and the probability density function were obtained. The Zakai equations for the corresponding unnormalized characteristics also were obtained. The proposed best nonlinear estimates were compared by way of a numerical example with the best linear estimates of the Kalman–Bucy filtration.

Analysis and Estimation of the States of Special Jump Markov Processes. I. Martingale Representation

The first part of this paper was devoted to a class of continuous-time jump processes generalizing the finite-state Markov processes. Main characteristics of this process such as the transition probabilities, infinitesimal generator, and so on were established. Processes of this class were proved to be solutions of linear differential equations with a martingale in the right-hand side. Stochastic analysis of a hidden Markov model of evolution of risky assets was presented as an example.

Optimal investment for investors with state dependent income, and for insurers

Abstract. An optimal control problem is considered where a risky asset is used for investment, and this investment is financed by initial wealth as well as by a state dependent income. The objective function is accumulated discounted expected utility of wealth, where the utility function is nondecreasing and bounded. This problem is investigated for constant as well as for stochastic discount rate, where the stochastic model is a time homogeneous finite state Markov process. We prove that the Bellman equation to this optimization problem has a classical solution and give a verification argument. Based on this we deal with the problem of optimal investment for an insurer with an insurance business modelled by a compound Poisson or a compound Cox process, under the presence of constant as well as (finite state space Markov) stochastic interest rate.

Stable Noisy K-state Markov Sunspots

We consider a linear stochastic univariate rational expectations model, with a predetermined variable, and consider solutions driven by an extraneous finite state Markov process as well as by the fundamental noise. We obtain conditions for existence of noisy k-state sunspot equilibria (noisy k-SSEs) and, for the case k=2, of noisy k-state dependent sunspot equilibria (noisy k-*SDSs). k-*SDSs are driven by a finite state sunspot but have an infinite range of values even in the nonstochastic model. Stability under econometric learning is analyzed using representations that nest both types of solution. For the case k=2, we find that noisy 2-SSEs and noisy 2-*SDSs are learnable for parameters in proper subsets of the regions of their existence.

Robust kalman filtering for continuous time-lag systems with markovian jump parameters

The problem of continuous-time Kalman filtering for a class of linear, uncertain time-lag systems with randomly jumping parameters is considered. The parameter uncertainties are norm bounded and the transitions of the jumping parameters are governed by a finite-state Markov process. We establish LMI-based sufficient conditions for stochastic stability. The conditions under which a linear delay-less state estimator guarantees that the estimation error covariance lies within a prescribed bound for all admissible uncertainties are investigated. It is established that a robust Kalman filter algorithm can be determined in terms of two Riccati equations involving scalar parameters. The developed theory is illustrated by a numerical example.

Maximum Likelihood Blind Channel Estimation for Space-Time Coding Systems

Sophisticated signal processing techniques have to be developed for capacity enhancement of future wireless communication systems. In recent years, space-time coding is proposed to provide significant capacity gains over the traditional communication systems in fading wireless channels. Space-time codes are obtained by combining channel coding, modulation, transmit diversity, and optional receive diversity in order to provide diversity at the receiver and coding gain without sacrificing the bandwidth. In this paper, we consider the problem of blind estimation of space-time coded signals along with the channel parameters. Both conditional and unconditional maximum likelihood approaches are developed and iterative solutions are proposed. The conditional maximum likelihood algorithm is based on iterative least squares with projection whereas the unconditional maximum likelihood approach is developed by means of finite state Markov process modelling. The performance analysis issues of the proposed methods are studied. Finally, some simulation results are presented.

A mixed MAP/MLSE receiver for convolutional coded signals transmitted over a fading channel

This paper addresses the problem of estimating a rapidly fading convolutionally coded signal such as might be found in a wireless telephony or data network. We model both the channel gain and the convolutionally coded signal as Markov processes and, thus, the noisy received signal as a hidden Markov process (HMP). Two now-classical methods for estimating finite-state hidden Markov processes are the Viterbi (1967) algorithm and the a posteriori probability (APP) filter. A hybrid recursive estimation procedure is derived whereby one hidden process (the encoder state in our application) is estimated using a Viterbi-type (i.e., sequence based) cost and the other (the fading process) using an APP-based cost such as maximum a posteriori probability. The paper presents the new algorithm as applied specifically to this problem but also formulates the problem in a more general setting. The algorithm is derived in this general setting using reference probability methods. Using simulations, performance of the optimal scheme is compared with a number of suboptimal techniques-decision-directed Kalman and HMP predictors and Kalman filter and HMP filter per-survivor processing techniques.

Robust one-step receding horizon control of discrete-time Markovian jump uncertain systems

This paper proposes a receding horizon control scheme for a set of uncertain discrete-time linear systems with randomly jumping parameters described by a finite-state Markov process whose jumping transition probabilities are assumed to belong to some convex sets. The control scheme for the underlying systems is based on the minimization of the worst-case one-step finite horizon cost with a finite terminal weighting matrix at each time instant. This robust receding horizon control scheme has a more general structure than the existing robust receding horizon control for the underlying systems under the same design parameters. The proposed controller is obtained using semidefinite programming.

CREDIT CONTAGION: PRICING CROSS-COUNTRY RISK IN BRADY DEBT MARKETS

Credit contagion means that the credit deterioration of an entity causes the credit deterioration of other entities. In this paper, we build and test a continuous-time model for defaultable securities using a diffusive process for risk-free interest rate, and a finite-state continuous-time Markov process for the correlation of credit. The credit contagion, in particular, is established by relating transition rates of various credit states. Examples of derivative pricing with calibrated credit contagion model are provided. Initial empirical results with the benchmarks of Brady bonds show that our model is a viable new technique for the pricing and risk-managing of credit derivatives.

Effects of decision interval on optimal intertemporal portfolios with serially correlated returns

This paper considers portfolio choice when decisions are made for several future time periods all at once. The risky asset share sequence must be precommitted for the entire decision interval, either constrained (as in Samuelson (1991)) or not constrained (as in Balvers and Mitchell (2000)) to be constant across time periods within the interval. For a broad, plausible class of dynamic returns processes, contrary to Samuelson, under log utility the decisions for the more distant future are more conservative. This class is exemplified by autocorrelated ARMA(p,q) processes and finite-state Markov processes. The source of Samuelson’s contrary result is elucidated.

Perturbation of multivariable linear quadratic systems with jump parameters

Considers the problem of the perturbation of a class of linear quadratic systems where the change from one structure (for the dynamics and costs) to another is governed by a finite-state Markov process. The problem above leads to the analysis of some perturbed linearly coupled sets of Riccati equations. We show that the matrix obtained as the solution of the equations, which determines the optimal value and control, has a Taylor expansion in the perturbation parameter. We compute explicitly the terms of this expansion.

Simulation-based optimization of Markov reward processes

This paper proposes a simulation-based algorithm for optimizing the average reward in a finite-state Markov reward process that depends on a set of parameters. As a special case, the method applies to Markov decision processes where optimization takes place within a parametrized set of policies. The algorithm relies on the regenerative structure of finite-state Markov processes, involves the simulation of a single sample path, and can be implemented online. A convergence result (with probability 1) is provided.

Convergence to Equilibrium for Spin Glasses

We estimate the distance in total variation between the law of a finite state Markov process at time t, starting from a given initial measure, and its unique invariant measure. We derive upper bounds for the time to reach the equilibrium. As an example of application we consider a special case of finite state Markov process in random environment: the Metropolis dynamics of the Random Energy Model. We also study the process of the environment as seen from the process.

Filtering of Finite-State Time-Nonhomogeneous Markov Processes, a Direct Approach

A filtering equation is derived for P(x{sub t}=x vertical bar y{sub s},s element of [0,t]) for a continuous-time finite-state two-component time-nonhomogeneous cadlag Markov process z{sub t}=(x{sub t},y{sub t}) . The derivation is based on some new ideas in the filtering theory and does not require any knowledge of stochastic integration.

Perturbation of linear quadratic systems with jump parameters and hybrid controls

We consider the problem of the perturbation of a class of linear-quadratic differential games with piecewise deterministic dynamics, where the changes from one structure (for the dynamics) to another are governed by a finite-state Markov process. Player 1 controls the continuous dynamics, whereas Player 2 controls the rate of transition for the finite-state Markov process; both have access to the states of both processes. Player 1 wishes to minimize a given quadratic performance index, while player 2 wishes to maximize or minimize the same quantity. The problem above leads to the analysis of some linearly coupled set of quadratic equations (Riccati equations). We obtain a Taylor expansion in the perturbation for the solution of the equation for a fixed stationary policy of the player 2. This allows us to solve the game or team problem as a function of the perturbation.

Robust H/sub ∞/ control of uncertain Markovian jump systems with time-delay

This paper is concerned with the robust stochastic stabilizability and robust H/sub /spl infin// disturbance attenuation for a class of uncertain linear systems with time delay and randomly jumping parameters. The transition of the jumping parameters is governed by a finite-state Markov process. Sufficient conditions on the existence of a robust stochastic stabilizing and /spl gamma/-suboptimal H/sub /spl infin// state-feedback controller are presented using the Lyapunov functional approach. It is shown that a robust stochastically stabilizing H/sub /spl infin// state-feedback controller can be constructed through the numerical solution of a set of coupled linear matrix inequalities.

Occupation times in markov processes

In a homogeneous finite-state Markov process, we consider the occupation times, that is, the times spent by the process in given subsets of the state space during a finite interval of time. We first derive the distribution of the occupation time of one subset and then we generalize that result to the joint distribution of occupation times of different subsets of the state space by the use of order statistics from the uniform distribution. Next, we consider the distribution of weighted sums of occupation times. We obtain forward and backward equations describing the behavior of these weighted sums and we show how these lead to simple expressions for that distribution