Kolmogorov Complexity Research Articles

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models.

In deep learning, different kinds of deep networks typically need different optimizers, which have to be chosen after multiple trials, making the training process inefficient. To relieve this issue and consistently improve the model training speed across deep networks, we propose the ADAptive Nesterov momentum algorithm, Adan for short. Adan first reformulates the vanilla Nesterov acceleration to develop a new Nesterov momentum estimation (NME) method, which avoids the extra overhead of computing gradient at the extrapolation point. Then Adan adopts NME to estimate the gradient's first- and second-order moments in adaptive gradient algorithms for convergence acceleration. Besides, we prove that Adan finds an ϵ-approximate first-order stationary point within O(ϵ-3.5) stochastic gradient complexity on the non-convex stochastic problems (e.g., deep learning problems), matching the best-known lower bound. Extensive experimental results show that Adan consistently surpasses the corresponding SoTA optimizers on vision, language, and RL tasks and sets new SoTAs for many popular networks and frameworks, e.g., ResNet, ConvNext, ViT, Swin, MAE, DETR, GPT-2, Transformer-XL, and BERT. More surprisingly, Adan can use half of the training cost (epochs) of SoTA optimizers to achieve higher or comparable performance on ViT, GPT-2, MAE, etc, and also shows great tolerance to a large range of minibatch size, e.g., from 1 k to 32 k.

An accelerated decentralized stochastic optimization algorithm with inexact model

This paper considers the decentralized stochastic optimization problems where each node of network has only access to the local large data samples and local functions, which are distributed to the computational nodes. We extend the centralized fast adaptive gradient method with inexact model to deal with the large scale problem in the decentralized manner. Moreover, we propose an accelerated decentralized stochastic optimization algorithm with reconstructing parameter equations and defining new approximate local functions. Further, we provide the convergence analysis of the proposed algorithm and illustrate that our algorithm can achieve both the optimal stochastic oracle complexity and communication complexity that depend on the global condition number. Finally, the numerical experiments validate the convergence results of the proposed algorithm.

An Additively Optimal Interpreter for Approximating Kolmogorov Prefix Complexity.

We study practical approximations of Kolmogorov prefix complexity (K) using IMP2, a high-level programming language. Our focus is on investigating the optimality of the interpreter for this language as the reference machine for the Coding Theorem Method (CTM). This method is designed to address applications of algorithmic complexity that differ from the popular traditional lossless compression approach based on the principles of algorithmic probability. The chosen model of computation is proven to be suitable for this task, and a comparison to other models and methods is conducted. Our findings show that CTM approximations using our model do not always correlate with the results from lower-level models of computation. This suggests that some models may require a larger program space to converge to Levin's universal distribution. Furthermore, we compare the CTM with an upper bound on Kolmogorov complexity and find a strong correlation, supporting the CTM's validity as an approximation method with finer-grade resolution of K.

Communication Complexity of the Secret Key Agreement in Algorithmic Information Theory

It is known that the mutual information, in the sense of Kolmogorov complexity, of any pair of strings x and y is equal to the length of the longest shared secret key that two parties can establish via a probabilistic protocol with interaction on a public channel, assuming that the parties hold as their inputs x and y , respectively. We determine the worst-case communication complexity of this problem for the setting where the parties can use private sources of random bits. We show that for some x , y the communication complexity of the secret key agreement does not decrease even if the parties have to agree on a secret key, the size of which is much smaller than the mutual information between x and y . However, we discuss a natural class of x , y such that the communication complexity of the protocol declines gradually with the size of the derived secret key. The proof of the main result uses the spectral properties of appropriate graphs and the expander mixing lemma, as well as information-theoretic techniques, including constraint information inequalities and Muchnik’s conditional descriptions. A preliminary version of this article was published in the proceedings of MFCS 2020. In the present version, we give full proofs of all theorems and get rid of the assumption that the number of random bits used in the communication protocols is polynomial.

Towards Understanding Convergence and Generalization of AdamW.

AdamW modifies Adam by adding a decoupled weight decay to decay network weights per training iteration. For adaptive algorithms, this decoupled weight decay does not affect specific optimization steps, and differs from the widely used l2-regularizer which changes optimization steps via changing the first- and second-order gradient moments. Despite its great practical success, for AdamW, its convergence behavior and generalization improvement over Adam and l2-regularized Adam ( l2-Adam) remain absent yet. To solve this issue, we prove the convergence of AdamW and justify its generalization advantages over Adam and l2-Adam. Specifically, AdamW provably converges but minimizes a dynamically regularized loss that combines vanilla loss and a dynamical regularization induced by decoupled weight decay, thus yielding different behaviors with Adam and l2-Adam. Moreover, on both general nonconvex problems and PŁ-conditioned problems, we establish stochastic gradient complexity of AdamW to find a stationary point. Such complexity is also applicable to Adam and l2-Adam, and improves their previously known complexity, especially for over-parametrized networks. Besides, we prove that AdamW enjoys smaller generalization errors than Adam and l2-Adam from the Bayesian posterior aspect. This result, for the first time, explicitly reveals the benefits of decoupled weight decay in AdamW. Experimental results validate our theory.

Complexity Scaling of Liquid Dynamics.

According to excess-entropy scaling, dynamic properties of liquids like viscosity and diffusion coefficient are determined by the entropy. This link between dynamics and thermodynamics is increasingly studied and of interest also for industrial applications, but hampered by the challenge of calculating entropy efficiently. Utilizing the fact that entropy is basically the Kolmogorov complexity, which can be estimated from optimal compression algorithms [Avinery etal., Phys. Rev. Lett. 123, 178102 (2019)PRLTAO0031-900710.1103/PhysRevLett.123.178102; Martiniani etal., Phys. Rev. X 9, 011031 (2019)PRXHAE2160-330810.1103/PhysRevX.9.011031], we here demonstrate that the diffusion coefficients of four simple liquids follow a quasiuniversal exponential function of the optimal compression length of a single equilibrium configuration. We conclude that "complexity scaling" has the potential to become a useful tool for estimating dynamic properties of any liquid from a single configuration.

A Novel Hybrid Model for Financial Forecasting Based on CEEMDAN-SE and ARIMA-CNN-LSTM

Financial time series data are characterized by non-linearity, non-stationarity, and stochastic complexity, so predicting such data presents a significant challenge. This paper proposes a novel hybrid model for financial forecasting based on CEEMDAN-SE and ARIMA- CNN-LSTM. With the help of the CEEMDAN-SE method, the original data are decomposed into several IMFs and reconstructed via sample entropy into a lower-complexity stationary high-frequency component and a low-frequency component. The high-frequency component is predicted by the ARIMA statistical forecasting model, while the low-frequency component is predicted by a neural network model combining CNN and LSTM. Compared to some classical prediction models, our algorithm exhibits superior performance in terms of three evaluation indexes, namely, RMSE, MAE, and MAPE, effectively enhancing model accuracy while reducing computational overhead.

Measuring linguistic complexity in Chinese: An information-theoretic approach

The measurement of linguistic complexity has been a heated topic in linguistics. While previous studies have mainly centered around English and other European languages, Chinese seems neglected to some extent, and the existing Chinese linguistic complexity metrics have room for improvement. Against these backdrops, the present study applied an information-theoretic approach - the Kolmogorov complexity system - to measure Chinese linguistic complexity and examined its reliability and validity. To this end, a corpus of c.a. 60 million characters was compiled, based on which the morphological, syntactical and overall Kolmogorov complexity metrics and 18 other complexity metrics used in previous studies were calculated and subjected to a series of correlation analyses. The results show that the three Kolmogorov complexity metrics are significantly correlated with each other and with most of the previously-used metrics at the character, lexical, syntactic and collocation tiers, indicating that the Kolmogorov complexity metrics are reliable in measuring Chinese. In terms of validity, a comparison between the Kolmogorov complexity of Chinese and nine European languages and a comparison between the Kolmogorov complexity of Chinese L1 and L2 of different proficiencies show that the Kolmogorov complexity precisely and parsimoniously captured the linguistic features of Chinese. The morphological Kolmogorov complexity reflects the richness of Chinese morphemes and the flexibility of word formation, and the syntactic Kolmogorov complexity supports the view that Chinese is a topic-prominent language. This study both extends the generalizability of Kolmogorov complexity and refines the measurement of the Chinese language.

Kolmogorov complexity and nondeterminism versus determinism for polynomial time computations

We call any consistent and sufficiently powerful formal theory that enables to algorithmically verify whether a text is a proof algorithmically verifiable mathematics (av-mathematics). We study the fundamental question whether nondeterminism is more powerful than determinism for polynomial time computations in the framework of av-mathematics.Our goal is to show strong indications that nondeterminism is more powerful than determinism for polynomial time computations. To do that, we do not consider decision problems only, but also compression algorithms. We show that at least one of the following three claims must be true:(i)▪(ii)non-determinism is more powerful than determinism for polynomial-time compression(iii)for each polynomial-time compression algorithm there exists another one of the same asymptotic time complexity that compresses infinitely many strings logarithmically strongerAnother surprising consequence of P = NP would be that time-bounded Kolmogorov complexity for any polynomial bound can be computed by deterministic algorithms in polynomial time.

Construction of pan-cancer regulatory networks based on causal inference

The pan-cancer initiative aims to study the origin patterns of cancer cell, the processes of carcinogenesis, and the signaling pathways from a perspective that spans across different types of cancer. The construction of the pan-cancer related gene regulatory network is helpful to excavate the commonalities in regulatory relationships among different types of cancers. It also aids in understanding the mechanisms behind cancer occurrence and development, which is of great scientific significance for cancer prevention and treatment. In light of the high dimension and large sample size of pan-cancer omics data, a causal pan-cancer gene regulation network inference algorithm based on stochastic complexity is proposed. With the network construction strategy of local first and then global, the stochastic complexity is used in the conditional independence test and causal direction inference for the candidate adjacent node set of the target nodes. This approach aims to decrease the time complexity and error rate of causal network learning. By applying this algorithm to the sample data of seven types of cancers in the TCGA database, including breast cancer, lung adenocarcinoma, and so on, the pan-cancer related causal regulatory networks are constructed, and their biological significance is verified. The experimental results show that this algorithm can eliminate the redundant regulatory relationships effectively and infer the pan-cancer regulatory network more accurately (https://github.com/LindeEugen/CNI-SC).

An Information-Theoretic Approach to Morphosyntactic Complexity in English, Dutch and German

ABSTRACT Though equi-complexity of languages has long been an assumption in the study of language, recent research has argued that languages differ with regard to their morphological complexity. Some languages rely more on bound morphology to express grammatical meanings, whereas other languages rely more on lexical or word-order-based strategies. It is a moot point to what extent these differences correlate with demographic factors such as population size and language contact, and how complexity should be measured. In this study, we use information-theory, more specifically Kolmogorov Complexity, to assess morphological and word-order-based strategies and apply the procedure to three West Germanic languages, English, Dutch and German, which have been argued to form a continuum along the morphological complexity cline, plausibly due to different rates of demographic upheaval and concomitant language contact. Tracing the morphological and word-order-based complexity through time in parallel Bible translations, our results show that English is consistently less morphologically complex than its sisters, continuing on its path of morphological simplification. We see a statistically robust trade-off between morphology and word order complexity. We find support for earlier findings in the literature, with the exception of the difference between Dutch and German, which does not transpire in our results.

DeltaGzip: Computing Biopolymer-Ligand Binding Affinity via Kolmogorov Complexity and Lossless Compression.

The design of biosequences for biosensing and therapeutics is a challenging multistep search and optimization task. In principle, computational modeling may speed up the design process by virtual screening of sequences based on their binding affinities to target molecules. However, in practice, existing machine-learned models trained to predict binding affinities lack the flexibility with respect to reaction conditions, and molecular dynamics simulations that can incorporate reaction conditions suffer from high computational costs. Here, we describe a computational approach called DeltaGzip that evaluates the free energy of binding in biopolymer-ligand complexes from ultrashort equilibrium molecular dynamics simulations. The entropy of binding is evaluated using the Kolmogorov complexity definition of entropy and approximated using a lossless compression algorithm, Gzip. We benchmark the method on a well-studied data set of protein-ligand complexes comparing the predictions of DeltaGzip to the free energies of binding obtained using Jarzynski equality and experimental measurements.

LIMIT COMPLEXITIES, MINIMAL DESCRIPTIONS, AND n-RANDOMNESS

Abstract Let K denote prefix-free Kolmogorov complexity, and let $K^A$ denote it relative to an oracle A. We show that for any n, $K^{\emptyset ^{(n)}}$ is definable purely in terms of the unrelativized notion K. It was already known that 2-randomness is definable in terms of K (and plain complexity C) as those reals which infinitely often have maximal complexity. We can use our characterization to show that n-randomness is definable purely in terms of K. To do this we extend a certain “limsup” formula from the literature, and apply Symmetry of Information. This extension entails a novel use of semilow sets, and a more precise analysis of the complexity of $\Delta _2^0$ sets of minimal descriptions.

Resting-State EEG Signature of Early Consciousness Recovery in Comatose Patients with Traumatic Brain Injury.

Resting-state electroencephalography (rsEEG) is usually obtained to assess seizures in comatose patients with traumatic brain injury (TBI). We aim to investigate rsEEG measures and their prediction of early recovery of consciousness in patients with TBI. This is a retrospective study of comatose patients with TBI who were admitted to a trauma center (October 2013 to January 2022). Demographics, basic clinical data, imaging characteristics, and EEGs were collected. We calculated the following using 10-min rsEEGs: power spectral density, permutation entropy (complexity measure), weighted symbolic mutual information (wSMI, global information sharing measure), Kolmogorov complexity (Kolcom, complexity measure), and heart-evoked potentials (the averaged EEG signal relative to the corresponding QRS complex on electrocardiography). We evaluated the prediction of consciousness recovery before hospital discharge using clinical, imaging, and rsEEG data via a support vector machine. We studied 113 of 134 (84%) patients with rsEEGs. A total of 73 (65%) patients recovered consciousness before discharge. Patients who recovered consciousness were younger (40 vs. 50years, p = 0.01). Patients who recovered also had higher Kolcom (U = 1688, p = 0.01), increased beta power (U = 1,652 p = 0.003) with higher variability across channels (U = 1534, p = 0.034) and epochs (U = 1711, p = 0.004), lower delta power (U = 981, p = 0.04), and higher connectivity across time and channels as measured by wSMI in the theta band (U = 1636, p = 0.026; U = 1639, p = 0.024) than those who did not recover. The area under the receiver operating characteristic curve for rsEEG was higher than that for clinical data (using age, motor response, pupil reactivity) and higher than that for the Marshall computed tomography classification (0.69 vs. 0.66 vs. 0.56, respectively; p < 0.001). We describe the rsEEG signature in recovery of consciousness prior to discharge in comatose patients with TBI. rsEEG measures performed modestly better than the clinical and imaging data in predicting recovery.

Order estimation for autoregressive models using criteria based on stochastic complexity

In this paper, we are interested in the order estimation of an autoregressive model using the information criterion developed by El Matouat and Hallin (1996), which is based on stochastic complexity. This criterion is a generalization of the Hannan and Quinn criterion and provides a convergence of the model order estimator, but it depends on a parameter that is sensitive to the sample size. In order to select the exact order of the candidate model, we propose a method for identifying the values of this parameter from the sample using the information contained in sub-samples of increasing size. To study the performance of the proposed method in comparison with the usual criteria, we simulated samples from autoregressive models on which we applied our procedure. Simulation results support the relevance of our procedure when compared to the Akaike criterion, the Hannan and Quinn criterion, and the Schwarz criterion.

Wave-Function Network Description and Kolmogorov Complexity of Quantum Many-Body Systems

Programmable quantum devices are now able to probe wave functions at unprecedented levels. This is based on the ability to project the many-body state of atom and qubit arrays onto a measurement basis which produces snapshots of the system wave function. Extracting and processing information from such observations remains, however, an open quest. One often resorts to analyzing low-order correlation functions—that is, discarding most of the available information content. Here, we introduce wave-function networks—a mathematical framework to describe wave-function snapshots based on network theory. For many-body systems, these networks can become scale-free—a mathematical structure that has found tremendous success and applications in a broad set of fields, ranging from biology to epidemics to Internet science. We demonstrate the potential of applying these techniques to quantum science by introducing protocols to extract the Kolmogorov complexity corresponding to the output of a quantum simulator and implementing tools for fully scalable cross-platform certification based on similarity tests between networks. We demonstrate the emergence of scale-free networks analyzing experimental data obtained with a Rydberg quantum simulator manipulating up to 100 atoms. Our approach illustrates how, upon crossing a phase transition, the simulator complexity decreases while correlation length increases—a direct signature of buildup of universal behavior in data space. Comparing experiments with numerical simulations, we achieve cross-certification at the wave-function level up to timescales of 4 μs with a confidence level of 90% and determine experimental calibration intervals with unprecedented accuracy. Our framework is generically applicable to the output of quantum computers and simulators with access to the system wave function and requires probing accuracy and repetition rates accessible to most currently available platforms. Published by the American Physical Society 2024

Natural language syntax complies with the free-energy principle

Natural language syntax yields an unbounded array of hierarchically structured expressions. We claim that these are used in the service of active inference in accord with the free-energy principle (FEP). While conceptual advances alongside modelling and simulation work have attempted to connect speech segmentation and linguistic communication with the FEP, we extend this program to the underlying computations responsible for generating syntactic objects. We argue that recently proposed principles of economy in language design—such as “minimal search” criteria from theoretical syntax—adhere to the FEP. This affords a greater degree of explanatory power to the FEP—with respect to higher language functions—and offers linguistics a grounding in first principles with respect to computability. While we mostly focus on building new principled conceptual relations between syntax and the FEP, we also show through a sample of preliminary examples how both tree-geometric depth and a Kolmogorov complexity estimate (recruiting a Lempel–Ziv compression algorithm) can be used to accurately predict legal operations on syntactic workspaces, directly in line with formulations of variational free energy minimization. This is used to motivate a general principle of language design that we term Turing–Chomsky Compression (TCC). We use TCC to align concerns of linguists with the normative account of self-organization furnished by the FEP, by marshalling evidence from theoretical linguistics and psycholinguistics to ground core principles of efficient syntactic computation within active inference.

MEPSI: An MDL-Based Ensemble Pruning Approach with Structural Information

Ensemble pruning that combines a subset of individual learners generated in parallel to make predictions is an important topic in ensemble learning. Past decades have developed a lot of pruning algorithms that focus on the external behavior of learners on samples, which may lead to over-fitting. In this paper, we conjecture that the generalization performance of an ensemble is not only related to its external behavior on samples but also dependent on the internal structure of individual learners. We propose the general MEPSI approach based on Kolmogorov complexity and the Minimum Description Length (MDL) principle, which formulates the ensemble pruning task as the two-objective optimization problem that comprises the empirical error and structural information among individual learners. We also provide a concrete implementation of MEPSI on decision trees. The theoretical results provide generalization bounds for both the general MEPSI approach and tree-based implementation. The comparative experiments conducted on multiple real-world data sets demonstrate the effectiveness of our proposed method.

Predictability of monthly streamflow by considering complexity measures

Natural fluid flow systems exhibit turbulent and chaotic behavior that determines their high-level complexity. Chaos has an accurate mathematical definition, while turbulence is a property of fluid flow without an accurate mathematical definition. Using the Kolmogorov complexity (KC) and its derivative (KC spectrum), permutation entropy (PE), and Lyapunov exponent (LE), we considered how chaos affected the predictability of natural fluid flow systems. This paper applied these measures to investigate the turbulent, complex and chaotic behaviors of monthly streamflow of rivers from Bosnia and Herzegovina, the United States, and the Mendoza Basin (Argentina) and evaluated their time horizons using the Lyapunov time (LT). Based on the measures applied for river streamflow, we derived four modes of the interrelationship between turbulence, complexity, and chaos. Finally, using the measures, we clustered rivers with similar time horizons representing their predictability.

Prioritizing Causation in Decision Trees: A Framework for Interpretable Modeling

As a popular machine learning model, decision trees classify and generalize well, but face challenges in engineering applications: 1) Sensitivity to perturbations and lack of interpretability due to correlation reliance. 2) Manual setting of stopping criterion which is unrelated to correlation strength and easily leads to over-partitioning. To address these two challenges, we first theoretically analyze what leads to sub-optimal decision trees. By incorporating causal discovery, this limitation can be attributed to the fact that trees grown with spurious correlations often fall into sub-optimal that lead to overfitting and unfair behaviors. Neglecting causality motivates us to develop a ‘better’ tree with low Kolmogorov complexity and high generalization capability. Then we propose a causality decision tree framework, CausalDT, based on our theoretical expectation, where Hilbert-Schmidt independence criterion serves as a baseline. Unlike previous approaches that prioritize relevance, our framework determines branch nodes based on causation between features, with the significance level determining whether the tree should be expanded further. Experimental results demonstrate that our model maintains performance while reducing average tree depth by 35% on various datasets. Furthermore, our model enhances decision fairness and interpretability.