Information-theoretic Measure Of Complexity Research Articles

Spatial Structure of Places in Singapore from a Complexity Perspective of the Public Transport Network

As cities become increasingly developed, inclusive and equal provision of public services and affordable housing is essential in achieving sustainable growth and development. In Singapore, with much of the population living in public housing and taking public transport, there is a great need to understand their landscape across the country. In this study, using tools from Complexity Science, we elucidated the spatial organization of places in Singapore through public bus transport connectivity, and investigated the association between such pattern to the distribution of the population age groups and property price. From the spatial network of the public transport services, we probed the places’ scale using the continuum percolation approach to uncover their spatial pattern. After that, we employed an information-theoretic measure of complexity to quantify the spatial organization of the system. The results show that there is a critical distance embedded within the public transport system at which the structure of the places in Singapore emerges. In many areas, the connectivity between places map well, and are greatest within the official planning boundary, demonstrating the influence of upstream spatial planning in determining bus networks. Yet in mature towns (particularly those in the central region), bus connectivity spans beyond planning boundaries, suggesting a more organic pattern on the ground in places with a longer history of development. The complexity of the bus network maps remarkably well to residential areas, and was associated with the distribution of resident age groups, and to property prices. In particular, the well-connected areas in and around the central region of Singapore are associated with higher property price and an older population. In contrast, the peripheral less-connected areas tend to see a lower property price and a younger population. The findings can contribute to understanding the existing pattern of physical and social form within an urban system, highlighting the role of complexity science methods in uncovering ‘emergent’ patterns in bus transport networks, beyond initial spatial plans, as areas mature over time. It can also guide future planning by highlighting the areas where more attention is needed to enhance island wide public bus connectivity, and by doing so, enable the public transport system to contribute to reducing potential socio-economic polarization within the system.

Multivariate information theory uncovers synergistic subsystems of the human cerebral cortex

One of the most well-established tools for modeling the brain is the functional connectivity network, which is constructed from pairs of interacting brain regions. While powerful, the network model is limited by the restriction that only pairwise dependencies are considered and potentially higher-order structures are missed. Here, we explore how multivariate information theory reveals higher-order dependencies in the human brain. We begin with a mathematical analysis of the O-information, showing analytically and numerically how it is related to previously established information theoretic measures of complexity. We then apply the O-information to brain data, showing that synergistic subsystems are widespread in the human brain. Highly synergistic subsystems typically sit between canonical functional networks, and may serve an integrative role. We then use simulated annealing to find maximally synergistic subsystems, finding that such systems typically comprise ≈10 brain regions, recruited from multiple canonical brain systems. Though ubiquitous, highly synergistic subsystems are invisible when considering pairwise functional connectivity, suggesting that higher-order dependencies form a kind of shadow structure that has been unrecognized by established network-based analyses. We assert that higher-order interactions in the brain represent an under-explored space that, accessible with tools of multivariate information theory, may offer novel scientific insights.

Fast characterization of plasma states in W7-X with permutation entropy

Detecting changes in plasmas is compulsory for control and the detection of novelties. Moreover, automated novelty detection allows one to investigate large data sets to substantially enhance the efficiency of data mining approaches. To this end we introduce permutation entropy (PE) for the detection of changes in plasmas. PE is an information-theoretic complexity measure based in fluctuation analysis that quantifies the degree of randomness (resp. disorder, unpredictability) of the ordering of time series data. This method is computationally fast and robust against noise, which allows the evaluation of large data sets in an automated procedure. PE is applied on electron cyclotron emission and soft x-ray measurements in different Wendelstein 7-X low-iota configuration plasmas. A spontaneous transition to high core-electron temperature () was detected, as well as a localized low-coherent intermittent oscillation which ceased when increased in the transition. The results are validated with spectrogram analysis and provide evidence that a complexity measure such as PE is a method to support in-situ monitoring of plasma parameters and for novelty detection in plasma data. Moreover, the acceleration in processing time offers implementations of plasma-state-detection that provides results fast enough to induce control actions even during the experiment.

Surveying Structural Complexity in Quantum Many-Body Systems

Quantum many-body systems exhibit a rich and diverse range of exotic behaviours, owing to their underlying non-classical structure. These systems present a deep structure beyond those that can be captured by measures of correlation and entanglement alone. Using tools from complexity science, we characterise such structure. We investigate the structural complexities that can be found within the patterns that manifest from the observational data of these systems. In particular, using two prototypical quantum many-body systems as test cases - the one-dimensional quantum Ising and Bose-Hubbard models - we explore how different information-theoretic measures of complexity are able to identify different features of such patterns. This work furthers the understanding of fully-quantum notions of structure and complexity in quantum systems and dynamics.

Differences in Performance of ASD and ADHD Subjects Facing Cognitive Loads in an Innovative Reasoning Experiment

We aim to investigate whether EEG dynamics differ in adults with ASD (Autism Spectrum Disorders) and ADHD (attention-deficit/hyperactivity disorder) compared with healthy subjects during the performance of an innovative cognitive task, Aristotle’s valid and invalid syllogisms, and how these differences correlate with brain regions and behavioral data for each subject. We recorded EEGs from 14 scalp electrodes (channels) in 21 adults with ADHD, 21 with ASD, and 21 healthy, normal subjects. The subjects were exposed in a set of innovative cognitive tasks (inducing varying cognitive loads), Aristotle’s two types of syllogism mentioned above. A set of 39 questions were given to participants related to valid–invalid syllogisms as well as a separate set of questionnaires, in order to collect a number of demographic and behavioral data, with the aim of detecting shared information with values of a feature extracted from EEG, the multiscale entropy (MSE), in the 14 channels (‘brain regions’). MSE, a nonlinear information-theoretic measure of complexity, was computed to extract a feature that quantifies the complexity of the EEG. Behavior-Partial Least Squares Correlation, PLSC, is the method to detect the correlation between two sets of data, brain, and behavioral measures. -PLSC, a variant of PLSC, was applied to build a functional connectivity of the brain regions involved in the reasoning tasks. Graph-theoretic measures were used to quantify the complexity of the functional networks. Based on the results of the analysis described in this work, a mixed 14 × 2 × 3 ANOVA showed significant main effects of group factor and brain region* syllogism factor, as well as a significant brain region* group interaction. There are significant differences between the means of MSE (complexity) values at the 14 channels of the members of the ‘pathological’ groups of participants, i.e., between ASD and ADHD, while the difference in means of MSE between both ASD and ADHD and that of the control group is not significant. In conclusion, the valid–invalid type of syllogism generates significantly different complexity values, MSE, between ASD and ADHD. The complexity of activated brain regions of ASD participants increased significantly when switching from a valid to an invalid syllogism, indicating the need for more resources to ‘face’ the task escalating difficulty in ASD subjects. This increase is not so evident in both ADHD and control. Statistically significant differences were found also in the behavioral response of ASD and ADHD, compared with those of control subjects, based on the principal brain and behavior saliences extracted by PLSC. Specifically, two behavioral measures, the emotional state and the degree of confidence of participants in answering questions in Aristotle’s valid–invalid syllogisms, and one demographic variable, age, statistically and significantly discriminate the three groups’ ASD. The seed-PLC generated functional connectivity networks for ASD, ADHD, and control, were ‘projected’ on the regions of the Default Mode Network (DMN), the ‘reference’ connectivity, of which the structural changes were found significant in distinguishing the three groups. The contribution of this work lies in the examination of the relationship between brain activity and behavioral responses of healthy and ‘pathological’ participants in the case of cognitive reasoning of the type of Aristotle’s valid and invalid syllogisms, using PLSC, a machine learning approach combined with MSE, a nonlinear method of extracting a feature based on EEGs that captures a broad spectrum of EEGs linear and nonlinear characteristics. The results seem promising in adopting this type of reasoning, in the future, after further enhancements and experimental tests, as a supplementary instrument towards examining the differences in brain activity and behavioral responses of ASD and ADHD patients. The application of the combination of these two methods, after further elaboration and testing as new and complementary to the existing ones, may be considered as a tool of analysis in helping detecting more effectively such types of disorders.

Robust model selection in linear regression models using information complexity

In recent years, in the literature of linear regression models, robust model selection methods have received increasing attention when the datasets contain even a small fraction of outliers. Outliers can have a serious impact on statistical inference and the choice of models using model selection criteria. Most of the existing robust information-based model selection methods are confined to robust AIC and robust AIC-type criteria. However, the penalty function used in AIC-type criteria happens to be insufficient to measure the overall model complexity in the presence of outliers and when the model is misspecified. Furthermore, in the literature, there is not much attention paid to the overall model complexity despite its importance. To overcome these problems, this paper proposes a robust version of the information-theoretic measure of complexity (ICOMP) criterion due to Bozdogan based on robust alternatives to the maximum likelihood (ML) method. Unlike the AIC criterion, ICOMP penalizes not only the number of free parameters but also the profusion of the model complexity to reduce the effect of outliers. The proposed robust ICOMP criteria are based on robust M, S, and MM estimation methods. Numerical examples are shown using a large-scale Monte Carlo simulation to study the performance of the proposed criteria. Also, a real numerical example is provided to detect outliers and to choose the best subset of predictors. Our results show the flexibility and the versatility of the proposed new approach.

Information-theoretic Modeling of Perceived Musical Complexity

What makes a piece of music appear complex to a listener? This research extends previous work by Eerola (2016), examining information content generated by a computational model of auditory expectation (IDyOM) based on statistical learning and probabilistic prediction as an empirical definition of perceived musical complexity. We systematically manipulated the melody, rhythm, and harmony of short polyphonic musical excerpts using the model to ensure that these manipulations systematically varied information content in the intended direction. Complexity ratings collected from 28 participants were found to positively correlate most strongly with melodic and harmonic information content, which corresponded to descriptive musical features such as the proportion of out-of-key notes and tonal ambiguity. When individual differences were considered, these explained more variance than the manipulated predictors. Musical background was not a significant predictor of complexity ratings. The results support information content, as implemented by IDyOM, as an information-theoretic measure of complexity as well as extending IDyOM's range of applications to perceived complexity.

Partially ordered permutation complexity of coupled time series

A new information-theoretic complexity measure of coupling among multiple time series is proposed based on ordinal patterns. We regard ordinal patterns extracted from time series as total orders and exploit their algebraic properties to introduce a multiplication among them. The result of the multiplication is in general a partial order. A Kullback–Leibler divergence for the partial orders gives rise to a complexity measure of coupling in the sense that it vanishes in two extreme cases: when time series are uncoupled and when they are identical. The performance of the proposed complexity measure is tested on two model multivariate dynamical systems and its advantage over the existing complexity measures is discussed.

Hybridized Support Vector Machine and Recursive Feature Elimination with Information Complexity

In statistical data mining research, datasets often have nonlinearity and at the same time high-dimensionality. It has become difficult to analyze such datasets in a comprehensive manner using traditional statistical methodologies. In this paper, a novel wrapper method called SVM-ICOMP-RFE based on a hybridized support vector machine (SVM) and recursive feature elimination (RFE) with information-theoretic measure of complexity (ICOMP) is introduced and developed to classify high-dimensional data sets and to carry out subset selection of the features in the original data space for finding the best subset of features which are discriminating between the groups. Recursive feature elimination (RFE) ranks features based on information complexity (ICOMP) criterion. ICOMP plays an important role not only in choosing an optimal kernel function from a portfolio of many other kernel functions, but also in selecting important subset(s) of features. The potential and the flexibility of our approach are illustrated on two real benchmark data sets, one is ionosphere data which includes radar returns from the ionosphere, and another is aorta data which is used for the early detection of atheroma most commonly resulting heart attack. Also, the proposed method is compared with other RFE based methods using different measures (i.e., weight and gradient) for feature rankings.

Comparing Information-Theoretic Measures of Complexity in Boltzmann Machines

In the past three decades, many theoretical measures of complexity have been proposed to help understand complex systems. In this work, for the first time, we place these measures on a level playing field, to explore the qualitative similarities and differences between them, and their shortcomings. Specifically, using the Boltzmann machine architecture (a fully connected recurrent neural network) with uniformly distributed weights as our model of study, we numerically measure how complexity changes as a function of network dynamics and network parameters. We apply an extension of one such information-theoretic measure of complexity to understand incremental Hebbian learning in Hopfield networks, a fully recurrent architecture model of autoassociative memory. In the course of Hebbian learning, the total information flow reflects a natural upward trend in complexity as the network attempts to learn more and more patterns.

Compressing learner language: An information-theoretic measure of complexity in SLA production data

We present a proof-of-concept study that sketches the use of compression algorithms to assess Kolmogorov complexity, which is a text-based, quantitative, holistic, and global measure of structural surface redundancy. Kolmogorov complexity has been used to explore cross-linguistic complexity variation in linguistic typology research, but we are the first to apply it to naturalistic second language acquisition (SLA) data. We specifically investigate the relationship between the complexity of second language (L2) English essays and the amount of instruction the essay writers have received. Analysis shows that increased L2 instructional exposure predicts increased overall complexity and increased morphological complexity, but decreased syntactic complexity (defined here as less rigid word order). While the relationship between L2 instructional exposure and complexity is robust across a number of first language (L1) backgrounds, L1 background does predict overall complexity levels.

Hybrid kernel density estimation for discriminant analysis with information complexity and genetic algorithm

A new hybrid approach is proposed which is computationally effective and easy to use in selecting the best subset of predictor variables in discriminant analysis (DA) under the assumption that data sets do not follow the normal distribution. The proposed approach integrates kernel density estimation for discriminant analysis (KDE-DA) and the information theoretic measure of complexity (ICOMP) with the genetic algorithm (GA). The ICOMP plays an important role in finding both the best bandwidth matrix for KDE-DA and the best subset of predictor variables which discriminate between the groups. The genetic algorithm (GA) is introduced and used within KDE-DA as a clever stochastic search algorithm. To show the working of this new and novel approach, six benchmark real data sets are considered and the results are compared with results of linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), and k-nearest neighbor discriminant analysis (k-NNDA) to choose the best fitting model. The experimental results show that the proposed hybrid kernel density estimation approach outperforms LDA, QDA, and k-NNDA.

A novel hybrid dimension reduction technique for undersized high dimensional gene expression data sets using information complexity criterion for cancer classification.

Gene expression data typically are large, complex, and highly noisy. Their dimension is high with several thousand genes (i.e., features) but with only a limited number of observations (i.e., samples). Although the classical principal component analysis (PCA) method is widely used as a first standard step in dimension reduction and in supervised and unsupervised classification, it suffers from several shortcomings in the case of data sets involving undersized samples, since the sample covariance matrix degenerates and becomes singular. In this paper we address these limitations within the context of probabilistic PCA (PPCA) by introducing and developing a new and novel approach using maximum entropy covariance matrix and its hybridized smoothed covariance estimators. To reduce the dimensionality of the data and to choose the number of probabilistic PCs (PPCs) to be retained, we further introduce and develop celebrated Akaike's information criterion (AIC), consistent Akaike's information criterion (CAIC), and the information theoretic measure of complexity (ICOMP) criterion of Bozdogan. Six publicly available undersized benchmark data sets were analyzed to show the utility, flexibility, and versatility of our approach with hybridized smoothed covariance matrix estimators, which do not degenerate to perform the PPCA to reduce the dimension and to carry out supervised classification of cancer groups in high dimensions.

Model selection in multivariate adaptive regression splines (MARS) using information complexity as the fitness function

This paper introduces information-theoretic measure of complexity (ICOMP) criterion for model selection in multivariate adaptive regression splines (MARS) to tradeoff efficiently between how well the model fits the data and the model complexity. As is well known, MARS is a popular nonparametric regression technique used to study the nonlinear relationship between a response variable and the set of predictors with the help of piecewise linear or cubic splines as basis functions. A critical aspect in determining the form of the nonparametric regression model during the MARS strategy is the evaluation of portfolio of submodels to select the best submodel with the appropriate number of knots over subset of predictors. In the usual regression modeling, when a large number of predictor variables are present in the model, and there is no precise information about the exact functional relationships among the variables, many model selection criteria still overfit the model. In this paper, to find the simplest model that balances the overfitting and underfitting for the model, ICOMP is proposed as a powerful model selection criterion for MARS modeling. Here, the model complexity is treated with respect to the interdependency of parameter estimates, as well as the number of free parameters in the model. We develop and study the performance of ICOMP along with several most popular model selection criteria such as Akaike's information criterion, Schwarz's Bayesian information criterion and generalized cross-validation in MARS modeling to select the best subset models. We provide two Monte Carlo simulation examples and a real benchmark example to demonstrate the utility and versatility of the proposed model selection approach to determine best functional form of the predictive model. Our numerical examples show that ICOMP provides a general model selection criterion with an insight to the interdependencies and/or correlational structure between parameter estimates in the selected model. This new approach can also be applicable to many complex statistical modeling problems.

Information-theoretic lower bound on energy cost of stochastic computation

Physical systems are often simulated using a stochastic computation where different final states result from identical initial states. Here, we derive the minimum energy cost of simulating a data sequence of a general physical system by stochastic computation. We show that the cost is proportional to the difference between two information-theoretic measures of complexity of the data—the statistical complexity and the predictive information . We derive the difference as the amount of information erased during the computation. Finally, we illustrate the physics of information by implementing the stochastic computation as a Gedanken experiment with a Szilard-type engine. The results create a new link between thermodynamics, information theory and complexity.

A measure of statistical complexity based on predictive information with application to finite spin systems

We propose the binding information as an information theoretic measure of complexity between multiple random variables, such as those found in the Ising or Potts models of interacting spins, and compare it with several previously proposed measures of statistical complexity, including excess entropy, Bialek et al.ʼs predictive information, and the multi-information. We discuss and prove some of the properties of binding information, particularly in relation to multi-information and entropy, and show that, in the case of binary random variables, the processes which maximise binding information are the ‘parity’ processes. The computation of binding information is demonstrated on Ising models of finite spin systems, showing that various upper and lower bounds are respected and also that there is a strong relationship between the introduction of high-order interactions and an increase of binding-information. Finally we discuss some of the implications this has for the use of the binding information as a measure of complexity.

Symbolic time series analysis and dynamic regimes

The mathematical theory of symbolic trajectories, whose use in multi-regime dynamics we are exploring, has also generated a set of qualitative methods of symbolic time series analysis (STSA), to be reviewed hereafter. They are illustrated by instances of multi-regime dynamics characterized by a progressively broadening regime menu, offering thus the possibility of increasingly rich regime-histories. Information-theoretic measures of complexity in dynamics are introduced, allowing for instance for cross-country or cross-sector comparisons as well as for a qualitative appraisal of structural change. The key application is to the six-regime dynamics in the Framework Space of the Italian macro-regions, whose sectorally disaggregated growth performances are taken to reflect differentiated patterns of structural change induced by capital accumulation and innovation.

What makes a categorization task difficult?

To understand why some categorization tasks are more difficult than others, we consider five factors that may affect human performance--namely, covariance complexity, optimal accuracy level with and without internal noise, orientation of the optimal categorization rule, and class separability. We argue that covariance complexity, an information-theoretic measure of complexity, is an excellent predictor of task difficulty. We present an experiment that consists of five conditions using a simulated medical decision-making task In the task human observers view hundreds of hypothetical patient profiles and classify each profile into Disease Category A or B. Each profile is a continuous-valued, three-dimensional stimulus consisting of three vertical bars, where each bar height represents the result of a medical test. Across the five conditions, covariance complexity was systematically manipulated. Results indicate that variation in performance is largely a function of covariance complexity and partly a function of internal noise. The remaining three factors do not explain performance results. We present a challenge to categorization theorists to design models that account for human performance as predicted by covariance complexity.

On the information-based measure of covariance complexity and its application to the evaluation of multivariate linear models

This paper introduces a new information-theoretic measure of complexity called ICOMP as a decision rule for model selection and evaluation for multivariate linear models. The development of ICOMP is based on the generalization and utilization of the covariance complexity index of van Emden (1971) in estimation of the multivariate linear model. ICOMP is motivated by Akaike's (1973) Information Criterion (AIC), but it is a different procedure than AIC. In linear or nonlinear statistical models ICOMP uses an information-based characterization of: (i) the covariance matrix properties of the parameter estimates of a model starting from their finite sampling distributions, and (ii) the complexity of the inverse-Fisher information matrix (i-FIM) as a new criterion of achievable accuracy of the model As a result, it provides a trade-off between the accuracy of the parameter estimates and the interaction of the residuals of a model via the measure of complexity of their respective covariances. It controls the risk...