Arbitrary Stopping Research Articles

Anytime-valid off-policy Inference for Contextual Bandits

Contextual bandit algorithms are ubiquitous tools for active sequential experimentation in healthcare and the tech industry. They involve online learning algorithms that adaptively learn policies over time to map observed contexts X t to actions A t in an attempt to maximize stochastic rewards R t . This adaptivity raises interesting but hard statistical inference questions, especially counterfactual ones: for example, it is often of interest to estimate the properties of a hypothetical policy that is different from the logging policy that was used to collect the data—a problem known as “off-policy evaluation” (OPE). Using modern martingale techniques, we present a comprehensive framework for OPE inference that relaxes unnecessary conditions made in some past works (such as performing inference at prespecified sample sizes, uniformly bounded importance weights, constant logging policies, and constant policy values, among others), significantly improving on them both theoretically and empirically. Importantly, our methods can be employed while the original experiment is still running (that is, not necessarily post hoc), when the logging policy may be itself changing (due to learning), and even if the context distributions are a highly dependent time series (such as if they are drifting over time). More concretely, we derive confidence sequences for various functionals of interest in OPE. These include doubly robust ones for time-varying off-policy mean reward values, but also confidence bands for the entire cumulative distribution function of the off-policy reward distribution. All of our methods (a) are valid at arbitrary stopping times; (b) only make nonparametric assumptions; (c) do not require importance weights to be uniformly bounded, and if they are, we do not need to know these bounds; and (d) adapt to the empirical variance of our estimators. In summary, our methods enable anytime-valid off-policy inference using adaptively collected contextual bandit data.

Fluctuation theorems and thermodynamic inequalities for nonequilibrium processes stopped at stochastic times.

We investigate the thermodynamics of general nonequilibrium processes stopped at stochastic times. We propose a systematic strategy for constructing fluctuation-theorem-like martingales for each thermodynamic functional, yielding a family of stopping-time fluctuation theorems. We derive second-law-like thermodynamic inequalities for the mean thermodynamic functional at stochastic stopping times, the bounds of which are even stronger than the thermodynamic inequalities resulting from the traditional fluctuation theorems when the stopping time is reduced to a deterministic one. Numerical verification is carried out for three well-known thermodynamic functionals, namely, entropy production, free energy dissipation, and dissipative work. These universal equalities and inequalities are valid for arbitrary stopping strategies, and thus provide a comprehensive framework with insights into the fundamental principles governing nonequilibrium systems.

Selecting a significance level in sequential testing procedures for community detection

While there have been numerous sequential algorithms developed to estimate community structure in networks, there is little available guidance and study of what significance level or stopping parameter to use in these sequential testing procedures. Most algorithms rely on prespecifiying the number of communities or use an arbitrary stopping rule. We provide a principled approach to selecting a nominal significance level for sequential community detection procedures by controlling the tolerance ratio, defined as the ratio of underfitting and overfitting probability of estimating the number of clusters in fitting a network. We introduce an algorithm for specifying this significance level from a user-specified tolerance ratio, and demonstrate its utility with a sequential modularity maximization approach in a stochastic block model framework. We evaluate the performance of the proposed algorithm through extensive simulations and demonstrate its utility in controlling the tolerance ratio in single-cell RNA sequencing clustering by cell type and by clustering a congressional voting network.

Martingale Methods for Sequential Estimation of Convex Functionals and Divergences

We present a unified technique for sequential estimation of convex divergences between distributions, including integral probability metrics like the kernel maximum mean discrepancy, $\varphi$-divergences like the Kullback-Leibler divergence, and optimal transport costs, such as powers of Wasserstein distances. This is achieved by observing that empirical convex divergences are (partially ordered) reverse submartingales with respect to the exchangeable filtration, coupled with maximal inequalities for such processes. These techniques appear to be complementary and powerful additions to the existing literature on both confidence sequences and convex divergences. We construct an offline-to-sequential device that converts a wide array of existing offline concentration inequalities into time-uniform confidence sequences that can be continuously monitored, providing valid tests or confidence intervals at arbitrary stopping times. The resulting sequential bounds pay only an iterated logarithmic price over the corresponding fixed-time bounds, retaining the same dependence on problem parameters (like dimension or alphabet size if applicable). These results are also applicable to more general convex functionals -- like the negative differential entropy, suprema of empirical processes, and V-Statistics -- and to more general processes satisfying a key leave-one-out property.

Anytime-Valid Tests of Conditional Independence Under Model-X

ABSTRACT We propose a sequential, anytime-valid method to test the conditional independence of a response Y and a predictor X given a random vector Z. The proposed test is based on e-statistics and test martingales, which generalize likelihood ratios and allow valid inference at arbitrary stopping times. In accordance with the recently introduced model-X setting, our test depends on the availability of the conditional distribution of X given Z, or at least a sufficiently sharp approximation thereof. Within this setting, we derive a general method for constructing e-statistics for testing conditional independence, show that it leads to growth-rate optimal e-statistics for simple alternatives, and prove that our method yields tests with asymptotic power one in the special case of a logistic regression model. A simulation study is done to demonstrate that the approach is competitive in terms of power when compared to established sequential and nonsequential testing methods, and robust with respect to violations of the model-X assumption. Supplementary materials for this article are available online.

The Probability of Reaching a Receding Boundary by a Branching Random Walk with Fading Branching and Heavy-Tailed Jump Distribution

Foss and Zachary (2003) and Foss, Palmowski and Zachary (2005) studied the probability of achieving a receding boundary on a time interval of random length by a random walk with a heavy-tailed jump distribution. They have proposed and developed a new approach that allows to generalise results of Asmussen (1998) onto the case of arbitrary stopping times and a wide class of nonlinear boundaries, and to obtain uniform results over all stopping times. In this paper, we consider a class of branching random walks with fading branching and obtain results on the tail asymptotics for the maximum of a branching random walk on a time interval of random (possibly unlimited) length, as well as uniform results within a class of bounded random time intervals.

Binomial early stopping times

Sequential analysis for the purposes of possibly stopping a trial early is important whenever a result must be obtained as quickly as possible for public health, economic, or other reasons. A dominant research stream since the middle of the 20th century has been the challenge of generalizing Wald’s sequential probability ratio test (SPRT) to include composite alternative hypotheses. This article offers an alternative for a binomial distribution by constructing a single-parameter family of triangular rejection regions for the null hypothesis using a generation function argument in the two-dimensional space of successes versus failures. The result is algebraically equivalent to one line of the SPRT with unit power and with the second value of the binomial parameter as the undetermined parameter. Rounding then discretizes the line to the grid of two-dimensional integers and classical results for arbitrary stopping conditions are used to give expressions for the estimator, of the binomial parameter and its variance The choice can be made by the practitioner in terms of their appetite for risk or more formally via a power analysis. An example is given of an ecological study where a quick binary decision was required and this desire had to be weighed against robustness of the results.

Conglomerability, disintegrability and the comparative principle

Abstract Our aim here is to present a result that connects some approaches to justifying countable additivity. This result allows us to better understand the force of a recent argument for countable additivity due to Easwaran. We have two main points. First, Easwaran’s argument in favour of countable additivity should have little persuasive force on those permissive probabilists who have already made their peace with violations of conglomerability. As our result shows, Easwaran’s main premiss – the comparative principle – is strictly stronger than conglomerability (and equivalent to disintegrability). Second, with the connections between the comparative principle and other probabilistic concepts clearly in view, we point out that opponents of countable additivity can still make a case that countable additivity is an arbitrary stopping point between finite and full additivity.

Reflected BSDEs with two optional barriers and monotone coefficient on general filtered space

We consider reflected backward stochastic differential equations with two optional barriers of class (D) satisfying Mokobodzki’s separation condition, and coefficient which is only continuous and non-increasing. We assume that data are merely integrable and the terminal time is an arbitrary (possibly infinite) stopping time. We study the problem of the existence and uniqueness of solutions to the mentioned equations, and their connections with the value process in nonlinear Dynkin games.

Performance of Test Supermartingale Confidence Intervals for the Success Probability of Bernoulli Trials.

Given a composite null hypothesis , test supermartingales are non-negative supermartingales with respect to with an initial value of . Large values of test supermartingales provide evidence against . As a result, test supermartingales are an effective tool for rejecting , particularly when the -values obtained are very small and serve as certificates against the null hypothesis. Examples include the rejection of local realism as an explanation of Bell test experiments in the foundations of physics and the certification of entanglement in quantum information science. Test supermartingales have the advantage of being adaptable during an experiment and allowing for arbitrary stopping rules. By inversion of acceptance regions, they can also be used to determine confidence sets. We used an example to compare the performance of test supermartingales for computing -values and confidence intervals to Chernoff-Hoeffding bounds and the "exact" -value. The example is the problem of inferring the probability of success in a sequence of Bernoulli trials. There is a cost in using a technique that has no restriction on stopping rules, and, for a particular test supermartingale, our study quantifies this cost.

Guaranteed -optimal solutions with the linear optimizer ART3+O

The linear optimization algorithm ART3+O introduced by Chen et al (2010 Med. Phys. 37 4938–45) can efficiently solve large scale inverse planning problems encountered in radiation therapy by iterative projection. Its major weakness is that it cannot guarantee -optimality of the final solution due to an arbitrary stopping criterion. We propose an improvement to ART3+O where the stopping criterion is based on Farkas’ lemma. The same theory can be used to detect inconsistency in other projection methods as well. The proposed algorithm guarantees to find an -optimal solution in finite time. The algorithm is demonstrated on numerical examples in radiation therapy.

On the dynamic programming principle for uniformly nondegenerate stochastic differential games in domains

We prove the dynamic programming principle for uniformly nondegenerate stochastic differential games in the framework of time-homogeneous diffusion processes considered up to the first exit time from a domain. The zeroth-order “coefficient” and the “free” term are only assumed to be measurable. In contrast with previous results established for constant stopping times we allow arbitrary stopping times and randomized ones as well. The main assumption, which will be removed in a subsequent article, is that there exists a sufficiently regular solution of the Isaacs equation.

On the dynamic programming principle for uniformly nondegenerate stochastic differential games in domains and the Isaacs equations

We prove the dynamic programming principle for uniformly nondegenerate stochastic differential games in the framework of time-homogeneous diffusion processes considered up to the first exit time from a domain. In contrast with previous results established for constant stopping times we allow arbitrary stopping times and randomized ones as well. There is no assumption about solvability of the the Isaacs equation in any sense (classical or viscosity). The zeroth-order “coefficient” and the “free” term are only assumed to be measurable in the space variable. We also prove that value functions are uniquely determined by the functions defining the corresponding Isaacs equations and thus stochastic games with the same Isaacs equation have the same value functions.

Conformal loop ensembles: the Markovian characterization and the loop-soup construction

For random collections of self-avoiding loops in two-dimensional domains, we dene a simple and natural conformal restriction property that is conjecturally satised by the scaling limits of interfaces in models from statistical physics. This property is basically the combination of conformal invariance and the locality of the interaction in the model. Unlike the Markov property that Schramm used to characterize SLE curves (which involves conditioning on partially generated interfaces up to arbitrary stopping times), this property only involves conditioning on entire loops and thus appears at rst glance to be weaker. Our rst main result is that there exists exactly a one-dimensional family of random loop collections with this property | one for each 2 (8=3; 4] | and that the loops are forms of SLE . The proof proceeds in two steps. First, uniqueness is established by showing that every such loop ensemble can be generated by an \exploration process based on SLE. Second, existence is obtained using the two-dimensional Brownian loopsoup, which is a Poissonian random collection of loops in a planar domain. When the intensity parameterc of the loop-soup is less than 1, we show that the outer boundaries of the loop clusters are disjoint simple loops (when c > 1 there is almost surely only one cluster) that satisfy the conformal restriction axioms. We prove various results about loop-soups, cluster sizes, and the c = 1 phase transition. Taken together, our results imply that the following families are equiv

Nearly Minimax One-Sided Mixture-Based Sequential Tests

We focus on one-sided, mixture-based stopping rules for the problem of sequential testing a simple null hypothesis against a composite alternative. For the latter, we consider two cases—either a discrete alternative or a continuous alternative that can be embedded into an exponential family. For each case, we find a mixture-based stopping rule that is nearly minimax in the sense of minimizing the maximal Kullback–Leibler information. The proof of this result is based on finding an almost Bayes rule for an appropriate sequential decision problem and on high-order asymptotic approximations for the performance characteristics of arbitrary mixture-based stopping times. We also evaluate the asymptotic performance loss of certain intuitive mixture rules and verify the accuracy of our asymptotic approximations with simulation experiments.

The Effect of Arbitrary Stopping of Public Vehicles on Flow of Traffic In One–Line Streets

Cellular automaton method was used to simulate the flow of cars in typical streets in Indonesia. In this streets, arbitrary stopping of vehicles at anytime along the streets might happen and violating the traffic regulation by the drivers frequently occurs. This study was restricted to a one-line street passed by two types of vehicles: private car and public vehicles, and a car/vehicle cannot advance the preceding ones (car following model). Arbitrary stopping of public vehicles leads to occurrence of vehicle clustering along the line. The number of main clusters equals to the number of the public vehicles. The total distance traveled by all vehicles during iteration approximates a periodical function, with a period is equal to the stopping duration of the public vehicle. The power spectrum of this movement scales as f -ɑ , with ɑ = 2 and seems to be universal. The distance traveled in unit volume of fuel consumption was also investigated and showed that this distance decreases monotonically when the ratio of public vehicle to the total number of vehicles increases. Key words: Cellular automata; traffic flow; public vehicle; one–line street; fuel consumption

Characterization of arbitrage-free markets

The present paper deals with the characterization of no-arbitrage properties of a continuous semimartingale. The first main result, Theorem 2.1, extends the no-arbitrage criterion by Levental and Skorohod [Ann. Appl. Probab. 5 (1995) 906–925] from diffusion processes to arbitrary continuous semimartingales. The second main result, Theorem 2.4, is a characterization of a weaker notion of no-arbitrage in terms of the existence of supermartingale densities. The pertaining weaker notion of no-arbitrage is equivalent to the absence of immediate arbitrage opportunities, a concept introduced by Delbaen and Schachermayer [Ann. Appl. Probab. 5 (1995) 926–945]. Both results are stated in terms of conditions for any semimartingales starting at arbitrary stopping times σ. The necessity parts of both results are known for the stopping time σ=0 from Delbaen and Schachermayer [Ann. Appl. Probab. 5 (1995) 926–945]. The contribution of the present paper is the proofs of the corresponding sufficiency parts.

The Power Configuration Sequence of the Central World System, 1500-700 BC

This article is the fifth in a series in which the political careers of civilizations/world systems receive snapshot codings of their overall power structures at feasible intervals. The narratives are produced by collating histories with large frames of reference. The codings are done using a nominal variable, polarity, with seven available values. Previous articles in the series have examined the Indic system 550 bc–ad 1800, the Far Eastern 1025 bc–ad 1850, the Southwest Asian c. 2700–1500 bc, and the Northeast African c. 2625–1500 bc. The Northeast African and Southwest Asian systems and sequences merge c. 1500 bc to form the Central system. In the current article, the power structure of the Central Civilization/ World System is appraised over its first 800 years at 10–year intervals, from 1500 bc to an arbitrary stopping point of 700 bc. The systemic power structure is evaluated in terms of its predominant forms andtheir stability. During this 8-century period, the Central world system showed a distinct individuality, or “character”: multipolar and unipolar structures predominated; there was limited variety in structure, with extreme forms excluded; there was substantial structural stability. Over time, the Central system “aged”: its already limited structural variety further diminished, while its structural stability increased. The sequence of power configurations in the Central system is compared to the expectations of several theories. Toynbee‘s revised civilizational model fares best, but leaves dynamical issues unaddressed; the classical European balance of power model matches the kinematics (the sequence of forms), but not the dynamics, of behavior of the Central system. Alternative future directions of inquiry are discussed.

Bernoulli Sequential Estimation of the Size of a Finite Population

The problem considered is that of unbiased estimation of the size (N) of a finite closed population under Capture-mark-Release-Recapture (CMRR) sequential sampling procedure. Borrowing ideas from the Bernoulli sequential estimation and using the notions of “closed” and “pushed-up” sampling plans, we provide here an unified approach to the problem of unbiased estimation of N and, in particular, give a necessary and sufficient condition for unbiased estimability under an arbitrary stopping rule. The ideas are illustrated with several examples.

Decoupling and domination inequalities with application to Wald's identity for martingales

The aim of this paper is to show how decoupling techniques can be useful in proving domination results involving dependent random variables and arbitrary stopping times. Let { X i } be a sequence of random variables adapted to { F i} , an increasing sequence of σ-fields and set G=σ(X 1,X 2,…) . Assume that for all t>0, (0.1) P(|X i|>t| F i−1)⩽P(|Y i|>t| G), where { Y i } is a sequence of conditionally independent given G . A case of particular importance is the one when the { Y i } is independent of G in which case the conditioning drops from (0.1). Then, for all stopping times T adapted to G , E max j⩽T |∑ i=1 jX i| p⩽C pE|∑ i=1 TY i| p whenever { X i } is a mean-zero martingale difference sequence and p⩾1 or it is a sequence of non-negative or conditionally symmetric random variables and p>0. As a consequence of this, we obtain a sharp extension of Wald's equation for randomly stopped sums of i.i.d. random variables to the case of sums of the form ∑ i= k T X i− k+1 … X i where the mean-zero martingale sequence { X i } satisfies (0.1).