Theory Of Algorithmic Randomness Research Articles

From Wald to Schnorr: von Mises’ definition of randomness in the aftermath of Ville’s Theorem

The first formal definition of randomness, seen as a property of sequences of events or experimental outcomes, dates back to Richard von Mises’ work in the foundations of probability and statistics. The randomness notion introduced by von Mises is nowadays widely regarded as being too weak. This is, to a large extent, due to the work of Jean Ville, which is often described as having dealt the death blow to von Mises’ approach, and which was integral to the development of algorithmic randomness—the now-standard theory of randomness for elements of a probability space. The main goal of this article is to trace the history and provide an in-depth appraisal of two lesser-known, yet historically and methodologically notable proposals for how to modify von Mises’ definition so as to avoid Ville’s objection. The first proposal is due to Abraham Wald, while the second one is due to Claus-Peter Schnorr. We show that, once made precise in a natural way using computability theory, Wald’s proposal constitutes a much more radical departure from von Mises’ framework than intended. Schnorr’s proposal, on the other hand, does provide a partial vindication of von Mises’ approach: it demonstrates that it is possible to obtain a satisfactory randomness notion—indeed, a canonical algorithmic randomness notion—by characterizing randomness in terms of the invariance of limiting relative frequencies. More generally, we argue that Schnorr’s proposal, together with a number of little-known related results, reveals that there is more continuity than typically acknowledged between von Mises’ approach and algorithmic randomness. Even though von Mises’ exclusive focus on limiting relative frequencies did not survive the passage to the theory of algorithmic randomness, another crucial aspect of his conception of randomness did endure; namely, the idea that randomness amounts to a certain type of stability or invariance under an appropriate class of transformations.

Algorithmically random series

Rademacher (Mathematische Annalen 87 (1922) 112–138), Steinhaus (Mathematische Zeitschrift 31 (1930) 408–416) and Paley and Zygmund (Mathematical Proceedings of the Cambridge Philosophical Society 26 (1930) 337–257, Mathematical Proceedings of the Cambridge Philosophical Society 26 (1930) 458–474, Mathematical Proceedings of the Cambridge Philosophical Society 28 (1932) 190–205) initiated the extensive study of random series. Using the theory of algorithmic randomness, which is a mix of computability theory and probability theory, we investigate the effective content of some classical theorems. We discuss how this is related to an old question of Kahane and Bollobás. We also discuss how considerations of such algorithmic questions about random series seem to lead to new notions of algorithmic randomness.

A LEARNING-THEORETIC CHARACTERISATION OF MARTIN-LÖF RANDOMNESS AND SCHNORR RANDOMNESS

AbstractNumerous learning tasks can be described as the process of extrapolating patterns from observed data. One of the driving intuitions behind the theory of algorithmic randomness is that randomness amounts to the absence of any effectively detectable patterns: it is thus natural to regard randomness as antithetical to inductive learning. Osherson and Weinstein [11] draw upon the identification of randomness with unlearnability to introduce a learning-theoretic framework (in the spirit of formal learning theory) for modelling algorithmic randomness. They define two success criteria—specifying under what conditions a pattern may be said to have been detected by a computable learning function—and prove that the collections of data sequences on which these criteria cannot be satisfied correspond to the set of weak 1-randoms and the set of weak 2-randoms, respectively. This learning-theoretic approach affords an intuitive perspective on algorithmic randomness, and it invites the question of whether restricting attention to learning-theoretic success criteria comes at an expressivity cost. In other words, is the framework expressive enough to capture most core algorithmic randomness notions and, in particular, Martin-Löf randomness—arguably, the most prominent algorithmic randomness notion in the literature? In this article, we answer the latter question in the affirmative by providing a learning-theoretic characterisation of Martin-Löf randomness. We then show that Schnorr randomness, another central algorithmic randomness notion, also admits a learning-theoretic characterisation in this setting.

Microscopic Reversibility and Macroscopic Irreversibility: From the Viewpoint of Algorithmic Randomness

Emergence of deterministic and irreversible macroscopic behavior from deterministic and reversible microscopic dynamics is understood as a result of the law of large numbers. In this paper, we prove on the basis of the theory of algorithmic randomness that Martin-L\"of random initial microstates satisfy an irreversible macroscopic law in the Kac infinite chain model. We find that the time-reversed state of a random state is not random as well as violates the macroscopic law.

Extracting randomness within a subset is hard

The tree forcing method of Liu enables the cone avoiding of bounded enumeration of a given tree, within subsets or co-subsets of an arbitrary given set, provided the given tree does not admit computable bounded enumeration. Using this result, he settled and reproduced a series of problems and results in reverse mathematics and the theory of algorithmic randomness, including showing that every 1-random set has an infinite subset or co-subset which computes no 1-random set. In this paper, we show that for any given 1-random set A, there exists an infinite subset G of A such that G does not compute any set with positive effective Hausdorff dimension. In particular, we answer in the affirmative Kjos-Hanssen’s 2006 question whether each 1-random set has an infinite subset which computes no 1-random set. The result is surprising in that the tree forcing technique seems to heavily rely on subset co-subset combinatorics, whereas this result does not.

Layerwise Computability and Image Randomness

Algorithmic randomness theory starts with a notion of an individual random object. To be reasonable, this notion should have some natural properties; in particular, an object should be random with respect to the image distribution F(P) (for some distribution P and some mapping F) if and only if it has a P-random F-preimage. This result (for computable distributions and mappings, and Martin-Lof randomness) was known for a long time (folklore); for layerwise computable mappings it was mentioned in Hoyrup and Rojas (2009, Proposition 5) (even for more general case of computable metric spaces). In this paper we provide a proof and discuss the related quantitative results and applications.

Coherence of Reducibilities with Randomness Notions

Loosely speaking, when A is “more random” than B and B is “random”, then A should be random. The theory of algorithmic randomness has some formulations of “random” sets and “more random” sets. In this paper, we study which pairs (R, r) of randomness notions R and reducibilities r have the follwing property: if A is r-reducible to B and A is R-random, then B should be R-random. The answer depends on the notions R and r. The implications hold for most pairs, but not for some. We also give characterizations of n-randomness via complexity.

How Powerful Are Integer-Valued Martingales?

In the theory of algorithmic randomness, one of the central notions is that of computable randomness. An infinite binary sequence X is computably random if no recursive martingale (strategy) can win an infinite amount of money by betting on the values of the bits of X. In the classical model, the martingales considered are real-valued, that is, the bets made by the martingale can be arbitrary real numbers. In this paper, we investigate a more restricted model, where only integer-valued martingales are considered, and we study the class of random sequences induced by this model.

Separations of non-monotonic randomness notions

In the theory of algorithmic randomness, several notions of random sequence are defined via a game-theoretic approach, and the notions that received the most attention are perhaps Martin-Lof (ML) randomness and computable randomness. The latter notion was introduced by Schnorr and is rather natural: an infinite binary sequence is computably random if no total computable strategy succeeds on it by betting on bits in order. However, computably random sequences can have properties that one may consider to be incompatible with being random, in particular, there are computably random sequences that are highly compressible. The concept of ML randomness is much better behaved in this and other respects, on the other hand its definition in terms of martingales is considerably less natural. Muchnik, elaborating on ideas of Kolmogorov and Loveland, refined Schnorr’s model by also allowing non-monotonic strategies, i.e. strategies that do not bet on bits in order. The subsequent ‘non-monotonic’ notion of randomness, now called Kolmogorov–Loveland randomness, has been shown to be quite close to ML randomness, but whether these two classes coincide remains a fundamental open question. In order to get a better understanding of non-monotonic randomness notions, Miller and Nies introduced some interesting intermediate concepts, where one only allows non-adaptive strategies, i.e. strategies that can still bet non-monotonically, but such that the sequence of betting positions is known in advance (and computable). Recently, these notions were shown by Kastermans and Lempp to differ from ML randomness. We continue the study of the non-monotonic randomness notions introduced by Miller and Nies and obtain results about the Kolmogorov complexities of initial segments that may and may not occur for such sequences, where these results then imply a complete classification of these randomness notions by order of strength.

Prequential randomness and probability

This paper studies Dawid’s prequential framework from the point of view of the algorithmic theory of randomness. Our first main result is that two natural notions of randomness coincide. One notion is the prequential version of the measure-theoretic definition due to Martin-Löf, and the other is the prequential version of the game-theoretic definition due to Schnorr and Levin. This is another manifestation of the close relation between the two main paradigms of randomness. The algorithmic theory of randomness can be stripped of its algorithmic aspect and still give meaningful results; the measure-theoretic paradigm then corresponds to Kolmogorov’s measure-theoretic probability and the game-theoretic paradigm corresponds to game-theoretic probability. Our second main result is that measure-theoretic probability coincides with game-theoretic probability on all analytic (in particular, Borel) sets.

Merging of opinions in game-theoretic probability

This paper gives game-theoretic versions of several results on “merging of opinions” obtained in measure-theoretic probability and algorithmic randomness theory. An advantage of the game-theoretic versions over the measure-theoretic results is that they are pointwise, their advantage over the algorithmic randomness results is that they are non-asymptotic, but the most important advantage over both is that they are very constructive, giving explicit and efficient strategies for players in a game of prediction.

Quality assessment of individual classifications in machine learning and data mining

Although in the past machine learning algorithms have been successfully used in many problems, their serious practical use is affected by the fact that often they cannot produce reliable and unbiased assessments of their predictions' quality. In last few years, several approaches for estimating reliability or confidence of individual classifiers have emerged, many of them building upon the algorithmic theory of randomness, such as (historically ordered) transduction-based confidence estimation, typicalness-based confidence estimation, and transductive reliability estimation. Unfortunately, they all have weaknesses: either they are tightly bound with particular learning algorithms, or the interpretation of reliability estimations is not always consistent with statistical confidence levels. In the paper we describe typicalness and transductive reliability estimation frameworks and propose a joint approach that compensates the above-mentioned weaknesses by integrating typicalness-based confidence estimation and transductive reliability estimation into a joint confidence machine. The resulting confidence machine produces confidence values in the statistical sense. We perform series of tests with several different machine learning algorithms in several problem domains. We compare our results with that of a proprietary method as well as with kernel density estimation. We show that the proposed method performs as well as proprietary methods and significantly outperforms density estimation methods.

Uniform test of algorithmic randomness over a general space

The algorithmic theory of randomness is well developed when the underlying space is the set of finite or infinite sequences and the underlying probability distribution is the uniform distribution or a computable distribution. These restrictions seem artificial. Some progress has been made to extend the theory to arbitrary Bernoulli distributions (by Martin-Löf) and to arbitrary distributions (by Levin). We recall the main ideas and problems of Levin's theory, and report further progress in the same framework. The issues are the following: • Allow non-compact spaces (like the space of continuous functions, underlying the Brownian motion). • The uniform test (deficiency of randomness) d P ( x ) (depending both on the outcome x and the measure P) should be defined in a general and natural way. • See which of the old results survive: existence of universal tests, conservation of randomness, expression of tests in terms of description complexity, existence of a universal measure, expression of mutual information as “deficiency of independence”. • The negative of the new randomness test is shown to be a generalization of complexity in continuous spaces; we show that the addition theorem survives. The paper's main contribution is introducing an appropriate framework for studying these questions and related ones (like statistics for a general family of distributions).

Kolmogorov's Contributions to the Foundations of Probability

Andrei Nikolaevich Kolmogorov was the foremost contributor to the mathematical and philosophical foundations of probability in the twentieth century, and his thinking on the topic is still potent today. In this article we first review the three stages of Kolmogorov's work on the foundations of probability: (1) his formulation of measure-theoretic probability, 1933; (2) his frequentist theory of probability, 1963; and (3) his algorithmic theory of randomness, 1965–1987. We also discuss another approach to the foundations of probability, based on martingales, which Kolmogorov did not consider.

Prediction algorithms and confidence measures based on algorithmic randomness theory

This paper reviews some theoretical and experimental developments in building computable approximations of Kolmogorov's algorithmic notion of randomness. Based on these approximations a new set of machine learning algorithms have been developed that can be used not just to make predictions but also to estimate the confidence under the usual iid assumption.