Kolmogorov Complexity Theory Research Articles

Web Similarity in Sets of Search Terms Using Database Queries

Normalized web distance (NWD) is a similarity or normalized semantic distance based on the World Wide Web or another large electronic database, for instance Wikipedia, and a search engine that returns reliable aggregate page counts. For sets of search terms the NWD gives a common similarity (common semantics) on a scale from 0 (identical) to 1 (completely different). The NWD approximates the similarity of members of a set according to all (upper semi)computable properties. We develop the theory and give applications of classifying using Amazon, Wikipedia, and the NCBI website from the National Institutes of Health. The last gives new correlations between health hazards. A restriction of the NWD to a set of two yields the earlier normalized Google distance (NGD), but no combination of the NGD’s of pairs in a set can extract the information the NWD extracts from the set. The NWD enables a new contextual (different databases) learning approach based on Kolmogorov complexity theory that incorporates knowledge from these databases.

Review of Kolmogorov Complexity and Algorithmic Randomness by A. Shen, V. A. Uspensky, and N. Vereshchagin

This is a book about Kolmogorov complexity and algorithmic randomness. The main idea in Kolmogorov complexity, introduced by A. N. Kolmogorov and others in the 1960's, is to use an algorithmic approach to measure the amount of information in finite objects [1]. Kolmogorov wanted to create an information theory that, unlike Shannon information theory, is founded independently from probability theory. This approach results in a notion of randomness that is independent from the notion of probability and is closer to the intuition that "randomness is the absence of regularities" [2, 1]. As authors stated in the introduction, this book is not intended to be comprehensive, especially with respect to the more recent results; However, the authors cover a wide range of topics and results in Kolmogorov complexity theory and whenever a result is not rigorously investigated, sufficient references are provided. The book starts from the basic concepts and builds its way up toward the more advanced concepts and techniques. The progress from the introductory to the advanced topics is seamless which makes the reading an enjoyable experience. Concepts and techniques are explained very well and proofs are rigorous, detailed, and clear. Important and complicated proofs are usually followed by excellent discussions about the main ideas and techniques used in them. The book also contains many interesting and insightful philosophical discussions.

Composition as pattern

I argue for patternism, a new answer to the question of when some objects compose a whole. None of the standard principles of composition comfortably capture our natural judgments, such as that my cat exists and my table exists, but there is nothing wholly composed of them. Patternism holds, very roughly, that some things compose a whole whenever together they form a “real pattern”. Plausibly we are inclined to acknowledge the existence of my cat and my table but not of their fusion, because the first two have a kind of internal organizational coherence that their putative fusion lacks. Kolmogorov complexity theory supplies the needed rigorous sense of “internal organizational coherence”.

Universal probability-free prediction

We construct universal prediction systems in the spirit of Popper’s falsifiability and Kolmogorov complexity and randomness. These prediction systems do not depend on any statistical assumptions (but under the IID assumption they dominate, to within the usual accuracy, conformal prediction). Our constructions give rise to a theory of algorithmic complexity and randomness of time containing analogues of several notions and results of the classical theory of Kolmogorov complexity and randomness.

An algorithmic information theory of consciousness.

Providing objective metrics of conscious state is of great interest across multiple research and clinical fields—from neurology to artificial intelligence. Here we approach this challenge by proposing plausible mechanisms for the phenomenon of structured experience. In earlier work, we argued that the experience we call reality is a mental construct derived from information compression. Here we show that algorithmic information theory provides a natural framework to study and quantify consciousness from neurophysiological or neuroimaging data, given the premise that the primary role of the brain is information processing. We take as an axiom that “there is consciousness” and focus on the requirements for structured experience: we hypothesize that the existence and use of compressive models by cognitive systems, e.g. in biological recurrent neural networks, enables and provides the structure to phenomenal experience. Self-awareness is seen to arise naturally (as part of a better model) in cognitive systems interacting bidirectionally with the external world. Furthermore, we argue that by running such models to track data, brains can give rise to apparently complex (entropic but hierarchically organized) data. We compare this theory, named KT for its basis on the mathematical theory of Kolmogorov complexity, to other information-centric theories of consciousness. We then describe methods to study the complexity of the brain’s output streams or of brain state as correlates of conscious state: we review methods such as (i) probing the brain through its input streams (e.g. event-related potentials in oddball paradigms or mutual algorithmic information between world and brain), (ii) analyzing spontaneous brain state, (iii) perturbing the brain by non-invasive transcranial stimulation, and (iv) quantifying behavior (e.g. eye movements or body sway).

Identification of probabilities

Within psychology, neuroscience and artificial intelligence, there has been increasing interest in the proposal that the brain builds probabilistic models of sensory and linguistic input: that is, to infer a probabilistic model from a sample. The practical problems of such inference are substantial: the brain has limited data and restricted computational resources. But there is a more fundamental question: is the problem of inferring a probabilistic model from a sample possible even in principle? We explore this question and find some surprisingly positive and general results. First, for a broad class of probability distributions characterized by computability restrictions, we specify a learning algorithm that will almost surely identify a probability distribution in the limit given a finite i.i.d. sample of sufficient but unknown length. This is similarly shown to hold for sequences generated by a broad class of Markov chains, subject to computability assumptions. The technical tool is the strong law of large numbers. Second, for a large class of dependent sequences, we specify an algorithm which identifies in the limit a computable measure for which the sequence is typical, in the sense of Martin-Löf (there may be more than one such measure). The technical tool is the theory of Kolmogorov complexity. We analyze the associated predictions in both cases. We also briefly consider special cases, including language learning, and wider theoretical implications for psychology.

Some Properties of Antistochastic Strings

Algorithmic statistics is a part of algorithmic information theory (Kolmogorov complexity theory) that studies the following task: given a finite object x (say, a binary string), find an ‘explanation’ for it, i.e., a simple finite set that contains x and where x is a ‘typical element’. Both notions (‘simple’ and ‘typical’) are defined in terms of Kolmogorov complexity. It is known that this cannot be achieved for some objects: there are some “non-stochastic” objects that do not have good explanations. In this paper we study the properties of maximally non-stochastic objects; we call them “antistochastic”. In this paper, we demonstrate that the antistochastic strings have the following property (Theorem 6): if an antistochastic string x has complexity k, then any k bit of information about x are enough to reconstruct x (with logarithmic advice). In particular, if we erase all but k bits of this antistochastic string, the erased bits can be restored from the remaining ones (with logarithmic advice). As a corollary we get the existence of good list-decoding codes with erasures (or other ways of deleting part of the information). Antistochastic strings can also be used as a source of counterexamples in algorithmic information theory. We show that the symmetry of information property fails for total conditional complexity for antistochastic strings. An extended abstract of this paper was presented at the 10th International Computer Science Symposium in Russia (Milovanov, 2015).

Lower bounds of shortest vector lengths in random NTRU lattices

Finding the shortest vector of a lattice is one of the most important problems in computational lattice theory. For a random lattice, one can estimate the length of the shortest vector using the Gaussian heuristic. However, no rigorous proof can be provided for some classes of lattices, as the Gaussian heuristic may not hold for them. In this paper, we propose a general method to estimate lower bounds of the shortest vector lengths for random integral lattices in certain classes, which is based on the incompressibility method from the theory of Kolmogorov complexity. As an application, we can prove that for a random NTRU lattice, with an overwhelming probability, the ratio between the length of the shortest vector and the length of the target vector, which corresponds to the secret key, is at least a constant, independent of the rank of the lattice.

Image distortion analysis based on normalized perceptual information distance

Image distortion analysis is a fundamental issue in many image processing problems, including compression, restoration, recognition, classification, and retrieval. Traditional image distortion evaluation approaches tend to be heuristic and are often limited to specific application environment. In this work, we investigate the problem of image distortion measurement based on the theory of Kolmogorov complexity, which has rarely been studied in the context of image processing. This work is motivated by the normalized information distance (NID) measure that has been shown to be a valid and universal distance metric applicable to similarity measurement of any two objects (Li et al. in IEEE Trans Inf Theory 50:3250–3264, 2004). Similar to Kolmogorov complexity, NID is non-computable. A useful practical solution is to approximate it using normalized compression distance (NCD) (Li et al. in IEEE Trans Inf Theory 50:3250–3264, 2004), which has led to impressive results in many applications such as construction of phylogeny trees using DNA sequences (Li et al. in IEEE Trans Inf Theory 50:3250–3264, 2004). In our earlier work, we showed that direct use of NCD on image processing problems is difficult and proposed a normalized conditional compression distance (NCCD) measure (Nikvand and Wang, 2010), which has significantly wider applicability than existing image similarity/distortion measures. To assess the distortions between two images, we first transform them into the wavelet transform domain. Assuming stationarity and good decorrelation of wavelet coefficients beyond local regions and across wavelet subbands, the Kolmogorov complexity may be approximated using Shannon entropy (Cover et al. in Elements of information theory. Wiley-Interscience, New York, 1991). Inspired by Sheikh and Bovik (IEEE Trans Image Process 15(2):430–444, 2006), we adopt a Gaussian scale mixture model for clusters of neighboring wavelet coefficients and a Gaussian channel model for the noise distortions in the human visual system. Combining these assumptions with the NID framework, we derive a novel normalized perceptual information distance measure, where maximal likelihood estimation and least square regression are employed for parameter fitting. We validate the proposed distortion measure using three large-scale, publicly available, and subject-rated image databases, which include a wide range of practical image distortion types and levels. Our results demonstrate the good prediction power of the proposed method for perceptual image distortions.

$LT^2C^2$ : A language of thought with Turing-computable Kolmogorov complexity

In this paper, we present a theoretical effort to connect the theory of program size to psychology by implementing a concrete language of thought with Turing-computable Kolmogorov complexity ($LT^2C^2$ ) satisfying the following requirements: 1) to be simple enough so that the complexity of any given finite binary sequence can be computed, 2) to be based on tangible operations of human reasoning (printing, repeating,. . . ), 3) to be sufficiently powerful to generate all possible sequences but not too powerful as to identify regularities which would be invisible to humans. We first formalize $LT^2C^2$ , giving its syntax and semantics, and defining an adequate notion of program size. Our setting leads to a Kolmogorov complexity function relative to $LT^2C^2$ which is computable in polynomial time, and it also induces a prediction algorithm in the spirit of Solomonoff’s inductive inference theory. We then prove the efficacy of this language by investigating regularities in strings produced by participants attempting to generate random strings. Participants had a profound understanding of randomness and hence avoided typical misconceptions such as exaggerating the number of alternations. We reasoned that remaining regularities would express the algorithmic nature of human thoughts, revealed in the form of specific patterns. Kolmogorov complexity relative to $LT^2C^2$ passed three expected tests examined here: 1) human sequences were less complex than control PRNG sequences, 2) human sequences were not stationary showing decreasing values of complexity resulting from fatigue 3) each individual showed traces of algorithmic stability since fitting of partial data was more effective to predict subsequent data than average fits. This work extends on previous efforts to combine notions of Kolmogorov complexity theory and algorithmic information theory to psychology, by explicitly proposing a language which may describe the patterns of human thoughts.Received: 12 December 2012, Accepted: 3 February 2013; Edited by: G. Mindlin; DOI: http://dx.doi.org/10.4279/PIP.050001Cite as: S Romano, M Sigman, S Figueira, Papers in Physics 5, 050001 (2013)

On unbounded probability theory

The value of the empirical expectation coincides with that of the mean energy of an ideal Bose gas for one particle. The exact mathematical identity for these quantities makes it possible to carry over the concept of temperature corresponding to the mean energy to an unboundedly increasing sequence of random values for a new unbounded probability theory and for a generalization of Kolmogorov complexity theory. The notion of spectral gap, which was introduced in superconductivity theory, is carried over to unbounded probability theory.

A FORMULATION OF THE PRELIMINARY DESING PHASE USING COMPLEXITY-BASED RULES

This paper focuses on the application of complexity theory and entropy concepts in the design process. Broadly speaking, the design process involves three phases: problem definition, conceptual design and embodiment. In the conceptual design phase, concepts that satisfy the functional requirements of the desired product are identified and compared. It is said that approximately 75% of the total product life-cycle cost is committed in this phase. The conceptual design phase has two essential sub-phases, namely, obtaining a solution set and selecting the most suitable solutions. Our work focuses on the selection sub-phase. The aim within this sub-phase is to minimize the number of selected concept variants and to reduce their chances of rejection in later stages. However, the solution to this problem is quite elusive, mostly because information about concept variants is scarce and rather qualitative at this stage. A common method is to perform a cost-benefit analysis. However, the analysis relies heavily on expert intuition and is thus subjected to high uncertainties. Recently, axiomatic design is gaining popularity. This is a framework that incorporates two axioms, namely, the Independence Axiom and the Minimum Information Axiom, accompanied by several corollaries. However, criticism on the integrity of the Independence Axiom has appeared recently in the literature. Further, the formulation of axiomatic design appears to have logical flaws. Finally, the conceptual design phase, a distinct phase in the design process, cannot be distinguished clearly in axiomatic design. In this paper we try to improve the selection phase of the conceptual design by improving the existing cost-benefit approach. In this vein, performance features against which concepts would be evaluated are established. We propose the use of Kolmogorov complexity theory and entropy concepts from information theory and physics to evaluate the complexity of the performance features. The design concepts are then improved based on the rule to reduce complexity of each performance feature. Weights are finally assigned to each performance feature and an overall complexity index is obtained which is suitable to compare designs. The ideas are further elaborated on by examples.

Depth as Randomness Deficiency

Depth of an object concerns a tradeoff between computation time and excess of program length over the shortest program length required to obtain the object. It gives an unconditional lower bound on the computation time from a given program in absence of auxiliary information. Variants known as logical depth and computational depth are expressed in Kolmogorov complexity theory. We derive quantitative relation between logical depth and computational depth and unify the different depth notions by relating them to A. Kolmogorov and L. Levin’s fruitful notion of randomness deficiency. Subsequently, we revisit the computational depth of infinite strings, study the notion of super deep sequences and relate it with other approaches.

Nonlinear Dimensionality Reduction by Locally Linear Inlaying

High-dimensional data is involved in many fields of information processing. However, sometimes, the intrinsic structures of these data can be described by a few degrees of freedom. To discover these degrees of freedom or the low-dimensional nonlinear manifold underlying a high-dimensional space, many manifold learning algorithms have been proposed. Here we describe a novel algorithm, locally linear inlaying (LLI), which combines simple geometric intuitions and rigorously established optimality to compute the global embedding of a nonlinear manifold. Using a divide-and-conquer strategy, LLI gains some advantages in itself. First, its time complexity is linear in the number of data points, and hence LLI can be implemented efficiently. Second, LLI overcomes problems caused by the nonuniform sample distribution. Third, unlike existing algorithms such as isometric feature mapping (Isomap), local tangent space alignment (LTSA), and locally linear coordination (LLC), LLI is robust to noise. In addition, to evaluate the embedding results quantitatively, two criteria based on information theory and Kolmogorov complexity theory, respectively, are proposed. Furthermore, we demonstrated the efficiency and effectiveness of our proposal by synthetic and real-world data sets.

Modeling human-like intelligent image processing: An information processing perspective and approach

We live in the Information Age, and information has become a critically important component of our life. The success of the Internet made huge amounts of it easily available and accessible to everyone. To keep the flow of this information manageable, means for its faultless circulation and effective handling have become urgently required. Considerable research efforts are dedicated today to address this necessity, but they are seriously hampered by the lack of a common agreement about “What is information?” In particular, what is “visual information”—human's primary input from the surrounding world. The problem is further aggravated by a long-lasting stance borrowed from the biological vision research that assumes human-like information processing as an enigmatic mix of perceptual and cognitive vision faculties. I am trying to find a remedy for this bizarre situation. Relying on a new definition of “information”, which can be derived from Kolmogorov's complexity theory and Chaitin's notion of algorithmic information, I propose a unifying framework for visual information processing, which explicitly accounts for the perceptual and cognitive image processing peculiarities. I believe that this framework will be useful to overcome the difficulties that are impeding our attempts to develop the right model of human-like intelligent image processing.

Compression-based data mining of sequential data

The vast majority of data mining algorithms require the setting of many input parameters. The dangers of working with parameter-laden algorithms are twofold. First, incorrect settings may cause an algorithm to fail in finding the true patterns. Second, a perhaps more insidious problem is that the algorithm may report spurious patterns that do not really exist, or greatly overestimate the significance of the reported patterns. This is especially likely when the user fails to understand the role of parameters in the data mining process. Data mining algorithms should have as few parameters as possible. A parameter-light algorithm would limit our ability to impose our prejudices, expectations, and presumptions on the problem at hand, and would let the data itself speak to us. In this work, we show that recent results in bioinformatics, learning, and computational theory hold great promise for a parameter-light data-mining paradigm. The results are strongly connected to Kolmogorov complexity theory. However, as a practical matter, they can be implemented using any off-the-shelf compression algorithm with the addition of just a dozen lines of code. We will show that this approach is competitive or superior to many of the state-of-the-art approaches in anomaly/interestingness detection, classification, and clustering with empirical tests on time series/DNA/text/XML/video datasets. As a further evidence of the advantages of our method, we will demonstrate its effectiveness to solve a real world classification problem in recommending printing services and products.

A GENERAL FRAMEWORK FOR BICLUSTERING GENE EXPRESSION DATA

A large number of biclustering methods have been proposed to detect patterns in gene expression data. All these methods try to find some type of biclusters but no one can discover all the types of patterns in the data. Furthermore, researchers have to design new algorithms in order to find new types of biclusters/patterns that interest biologists. In this paper, we propose a novel approach for biclustering that, in general, can be used to discover all computable patterns in gene expression data. The method is based on the theory of Kolmogorov complexity. More precisely, we use Kolmogorov complexity to measure the randomness of submatrices as the merit of biclusters because randomness naturally consists in a lack of regularity, which is a common property of all types of patterns. On the basis of algorithmic probability measure, we develop a Markov Chain Monte Carlo algorithm to search for biclusters. Our method can also be easily extended to solve the problems of conventional clustering and checkerboard type biclustering. The preliminary experiments on simulated as well as real data show that our approach is very versatile and promising.

100 years of Zermelo's axiom of choice: what was the problem with it?

In February 2005 the Third Annual University of London Kolmogorov Lecture was held at Royal Holloway. These annual lectures were established in 2003. The lecture was given by Professor Per Martin-Löf of Stockholm University—one of the world's foremost researchers in the field of constructive mathematics. Professor Martin-Löf was presented with the Kolmogorov Lecture medal in recognition of his contribution to mathematical logic. Professor Alex Gammerman, Director of Royal Holloway's Computer Learning Research Centre, said: ‘The annual lectures are devoted to Andrei Nikolaevich Kolmogorov (1903–1987) who was one of the foremost Soviet mathematicians of the Twentieth Century. He was active in many mathematical fields, with interests in both theory and application. The work of Kolmogorov is the basis of Complexity Science: he made one of the most beautiful discoveries of the 20th century—what is now known as Kolmogorov Complexity or Algorithmic Information Theory. This discovery is crucial to the great advances currently being made in machine learning, and particularly here at Royal Holloway's world-renowned Computer Learning Research Centre. We have established the annual Kolmogorov Lecture to acknowledge the profound importance of his research. For many years A. N. Kolmogorov was head of the mathematical logic department at Moscow State University, and Per Martin-Löf studied under his supervision in 1964–65’.

How many strings are easy to predict?

It is well known in the theory of Kolmogorov complexity that most strings cannot be compressed; more precisely, only exponentially few (Θ (2 n − m )) binary strings of length n can be compressed by m bits. This paper extends the ‘incompressibility’ property of Kolmogorov complexity to the ‘unpredictability’ property of predictive complexity. The ‘unpredictability’ property states that predictive complexity (defined as the loss suffered by a universal prediction algorithm working infinitely long) of most strings is close to a trivial upper bound (the loss suffered by a trivial minimax constant prediction strategy). We show that only exponentially few strings can be successfully predicted and find the base of the exponent.

A computable version of the random signs problem and Kolmogorov complexity

An effective solution to a computable version of the random signs problem is formulated using a coefficient from Kolmogorov complexity theory.