Some dynamical properties of sequentially acquired information

In this paper we obtain two closely related theorems that essentially say that no matter what information metric is used, on the average the value of the accumulated information at stopping time is bounded by a multiple of the expected stopping time. These results are also independent of the particular stopping strategy employed although they do require that the expected stopping time be finite. These results, along with a general type of stopping strategy based on incremental information, are given. Later we apply our general theorem to a specific stopping strategy associated with the GIS model. Although we concentrate on the problem of stopping, the information function on which this stopping decision is based can also be used to choose the COA for the next cycle of the feedback loop. We apply our results to an estimation problem involving the well known Shannon-Weiner measure of information. Since our theorems require that the expected stopping times be finite, sometime is devoted to a discussion of necessary and sufficient conditions for finite expected stopping times.

Graph planning gives rise to fundamental algorithmic questions such as shortest path, traveling salesman problem, etc. A classical problem in discrete planning is to consider a weighted graph and construct a path that maximizes the sum of weights for a given time horizon T. However, in many scenarios, the time horizon is not fixed, but the stopping time is chosen according to some distribution such that the expected stopping time is T. If the stopping time distribution is not known, then to ensure robustness, the distribution is chosen by an adversary, to represent the worst-case scenario. A stationary plan for every vertex always chooses the same outgoing edge. For fixed horizon or fixed stopping-time distribution, stationary plans are not sufficient for optimality. Quite surprisingly we show that when an adversary chooses the stopping-time distribution with expected stopping time T, then stationary plans are sufficient. While computing optimal stationary plans for fixed horizon is NP-complete, we show that computing optimal stationary plans under adversarial stopping-time distribution can be achieved in polynomial time. Consequently, our polynomial-time algorithm for adversarial stopping time also computes an optimal plan among all possible plans.

Markov chains are the de facto finite-state model for stochastic dynamical systems, and Markov decision processes (MDPs) extend Markov chains by incorporating non-deterministic behaviors. Given an MDP and rewards on states, a classical optimization criterion is the maximal expected total reward where the MDP stops after T steps, which can be computed by a simple dynamic programming algorithm. We consider a natural generalization of the problem where the stopping times can be chosen according to a probability distribution, such that the expected stopping time is T, to optimize the expected total reward. Quite surprisingly we establish inter-reducibility of the expected stopping-time problem for Markov chains with the Positivity problem (which is related to the well-known Skolem problem), for which establishing either decidability or undecidability would be a major breakthrough. Given the hardness of the exact problem, we consider the approximate version of the problem: we show that it can be solved in exponential time for Markov chains and in exponential space for MDPs.

Markov chains are the de facto finite-state model for stochastic dynamical systems, and Markov decision processes (MDPs) extend Markov chains by incorporating non-deterministic behaviors. Given an MDP and rewards on states, a classical optimization criterion is the maximal expected total reward where the MDP stops after T steps, which can be computed by a simple dynamic programming algorithm. We consider a natural generalization of the problem where the stopping times can be chosen according to a probability distribution, such that the expected stopping time is T, to optimize the expected total reward. Quite surprisingly we establish inter-reducibility of the expected stopping-time problem for Markov chains with the Positivity problem (which is related to the well-known Skolem problem), for which establishing either decidability or undecidability would be a major breakthrough. Given the hardness of the exact problem, we consider the approximate version of the problem: we show that it can be solved in exponential time for Markov chains and in exponential space for MDPs.

Summary Generalized sequential probability-ratio procedures are defined for dependent and non-identically distributed random variables. For these procedures, conditions are found implying that the stopping time is finite with probability 1 and the expected stopping time is finite. These results are applied to rank order problems.

This paper considers the problem of sequential point estimation, under an appropriate loss function, of the location parameter when the errors form an autoregressive process with unknown scale and autoregressive parameters, A sequential procedure is developed and an asymptotic second order expansion is provided for the difference between expected stopping time and the optimal fixed sample size procedure. Also, the asymptotic normality of the stopping time is proved. Though the procedure Is asymptotically risk efficient, it. Is not clear whether it has bounded regret.

Graph planning with expected finite horizon

A sequential sampling procedure is developed to estimate the scale parameter θ of the Pareto distribution when the shape parameterθ is unknown using the cost function , where A is a positive constant. The probability distribution of the stopping time for this procedure is tabulated and the expected stopping time and cost under the sequential sampling procedure are computed and compared with the optimum sample size and minimum cost under the fixed sample size procedure (for the case when the shape parameter is known).

We introduce a regime-switching Ornstein–Uhlenbeck (O–U) model to address an optimal investment problem. Our study gives a closed-form expression for a regime-switching pairs trading value function consisting of probability and expectation of the double boundary stopping time of the Markov-modulated O–U process. We derive analytic solutions for the homogenous and non-homogenous ODE systems with initial value conditions for probability and expectation of the double boundary stopping time, and translate the solutions with boundary value conditions into solutions with initial value conditions. Based on the smoothness and continuity of the value function, we can obtain the optimum of the value function with thresholds and guarantee the existence of optimal thresholds in a finite closed interval. Our numerical analysis illustrates the rationality of theoretical model and the shape of transition probability and expected stopping time, as well as discusses sensitivity analysis in both one-state and two-state regime-switching models. We find that the optimal expected return per unit time in the two-state regime-switching model is higher than that of one-state regime-switching model. Likewise, the regime-switching model’s optimal thresholds are closer and more symmetric to the long-term mean.

This paper studies the problem of sequential Gaussian shift-in-mean hypothesis testing in a distributed multi-agent network. A sequential probability ratio test (SPRT) type algorithm in a distributed framework of the consensus+innovations form is proposed, in which the agents update their decision statistics by simultaneously processing latest observations (innovations) sensed sequentially over time and information obtained from neighboring agents (consensus). For each pre-specified set of type I and type II error probabilities, local decision parameters are derived which ensure that the algorithm achieves the desired error performance and terminates in finite time almost surely (a.s.) at each network agent. Large deviation exponents for the tail probabilities of the agent stopping time distributions are obtained and it is shown that asymptotically (in the number of agents or in the high signal-to-noise-ratio regime) these exponents associated with the distributed algorithm approach that of the optimal centralized detector. The expected stopping time for the proposed algorithm at each network agent is evaluated and is benchmarked with respect to the optimal centralized algorithm. The efficiency of the proposed algorithm in the sense of the expected stopping times is characterized in terms of network connectivity. Finally, simulation studies are presented which illustrate and verify the analytical findings.

The sequential testing of more than two hypotheses has important applications in direct-sequence spread spectrum signal acquisition, multiple-resolution-element radar, and other areas. A useful sequential test which we term the MSPRT is studied in this paper. The test is shown to be a generalization of the sequential probability ratio test. Under Bayesian assumptions, it is argued that the MSPRT approximates the much more complicated optimal test when error probabilities are small and expected stopping times are large. Bounds on error probabilities are derived, and asymptotic expressions for the stopping time and error probabilities are given. A design procedure is presented for determining the parameters of the MSPRT. Two examples involving Gaussian densities are included, and comparisons are made between simulation results and asymptotic expressions. Comparisons with Bayesian fixed sample size tests are also made, and it is found that the MSPRT requires two to three times fewer samples on average.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Let $T$ be the first time that a perturbed random walk crosses a nonlinear boundary. This paper concerns the approximations of the distribution of the excess over the boundary, the expected stopping time $ET$ and the variance of the stopping time $\operatorname{Var}(T)$. Expansions are obtained by using linear renewal theorems with varying drift.

We study best arm identification in a variant of the multi-armed bandit problem where the learner has limited precision in arm selection. The learner can only sample arms via certain exploration bundles, which we refer to as boxes. In particular, at each sampling epoch, the learner selects a box, which in turn causes an arm to get pulled as per a box-specific probability distribution. The pulled arm and its instantaneous reward are revealed to the learner, whose goal is to find the best arm by minimising the expected stopping time, subject to an upper bound on the error probability. We present an asymptotic lower bound on the expected stopping time, which holds as the error probability vanishes. We show that the optimal allocation suggested by the lower bound is, in general, non-unique and therefore challenging to track. We propose a modified tracking-based algorithm to handle non-unique optimal allocations, and demonstrate that it is asymptotically optimal. We also present non-asymptotic lower and upper bounds on the stopping time in the simpler setting when the arms accessible from one box do not overlap with those of others.

In this article accurate approximations and inequalities are derived for the distribution, expected stopping time and variance of the stopping time associated with moving sums of independent and identically distributed continuous random variables. Numerical results for a scan statistic based on a sequence of moving sums are presented for a normal distribution model, for both known and unknown mean and variance. The new R algorithms for the multivariate normal and t distributions established by Genz et al. (2010) provide readily available numerical values of the bounds and approximations.

We formulate a notion of doubly reflected BSDE in the case where the barriers $\xi$ and $\zeta$ do not satisfy any regularity assumption. Under a technical assumption (a Mokobodzki-type condition), we show existence and uniqueness of the solution. In the case where $\xi$ is right upper-semicontinuous and $\zeta$ is right lower-semicontinuous, the solution is characterized in terms of the value of a corresponding $\mathcal{E}^f$-Dynkin game, i.e. a game problem over stopping times with (non-linear) $f$-expectation, where $f$ is the driver of the doubly reflected BSDE. In the general case where the barriers do not satisfy any regularity assumptions, the solution of the doubly reflected BSDE is related to the value of "an extension" of the previous non-linear game problem over a larger set of "stopping strategies" than the set of stopping times. This characterization is then used to establish a comparison result and \textit{a priori} estimates with universal constants.