Information Loss Measures Research Articles

The hypergeometric connectivity hypothesis: Divergent performance of brain circuits with different synaptic connectivity distributions

The development of connectivity among brain networks (e.g., thalamocortical, cortico-thalamic, cortico-cortical) proceeds via a combination of axon and dendrite growth followed by a later process of synaptic pruning [Purves, D., Lichtman, J.W., 1980. Elimination of synapses in the developing nervous system. Science, 210, 153–157; Oppenheim, R.W., 1991. Cell death during development of the nervous system. Annual Review of Neuroscience, 14(1), 453–501.; Oppenheim, R., Qin-Wei Y., Prevette D., Yan Q., 1992. Brain-derived neurotrophic factor rescues developing avian motoneurons from cell death. Nature, 360, 755–757]. Sparse synaptic distribution (i.e., the low probability (< 0.1) of contact among neurons; [Braitenberg, V., Schüz, A., 1998. Cortex: Statistics and geometry of neuronal connectivity: Springer Berlin.] can conform to any of a range of connectivity patterns with different distributional characteristics; and different distribution patterns can yield networks with very different functional properties. We rigorously investigate a range of different connectivity characteristics, and show that different synaptic distributions can substantially affect the functional capabilities of the resulting networks. In particular, we provide formal measures of information loss in transmission from one set of neurons to another as a function of synaptic distribution, and show a set of empirical cases with different information-theoretic utility. We characterize the trade-offs among utility and costs, and their dependence on different classes of developmental strategies by which axons from one cell group are “assigned” to synapses on dendrites from a target cell group. It is shown that hypergeometric distributions minimize a range of measured costs, compared to competing synaptic distributions. It is also found that the divergent performance among differently organized brain circuits expands with brain size, rendering the effects increasingly consequential for big brains. In summary, we propose that the characteristics of hypergeometric connectivity provide a coherent explanatory hypothesis of a range of developmental and anatomical data.

Preserving edits when perturbing microdata for statistical disclosure control

To protect individuals in microdata from the risk of re-identification, a general perturbative method called PRAM (the Post-Randomization Method) is sometimes used for masking records. This method adds “noise” to categorical variables by changing values of categories for a small number of records according to a prescribed probability matrix and a stochastic process based on the outcome of a random multinomial draw. Changing values of categorical variables, however, will cause fully edited and clean records in microdata to start failing edit constraints resulting in data of low utility. In addition, an inconsistent record pinpoints to a potential attacker that the record was perturbed and attempts can be made to unmask the data. Therefore, the perturbation process must take into account micro edit constraints which will ensure that perturbed microdata satisfy all edits. Macro edit constraints which take the form of information loss measures also need to be defined in order to ensure that the overall utility of the data will not be badly compromised given an acceptable level of disclosure risk. This paper will discuss methods for perturbing microdata using PRAM while minimizing micro and macro edit failures. (Updated 10th August 2005)

Probabilistic Information Loss Measures in Confidentiality Protection of Continuous Microdata

Inference control for protecting the privacy of microdata (individual data) should try to optimize the tradeoff between data utility (low information loss) and protection against disclosure (low disclosure risk). Whereas risk measures are bounded between 0 and 1, information loss measures proposed in the literature for continuous data are unbounded, which makes it awkward to trade off information loss for disclosure risk. We propose in this paper to use probabilities to define bounded information loss measures for continuous microdata.

Privacy in Data Mining

In this chapter we describe the main tools for privacy in data mining. We present an overview of the tools for protecting data, and then we focus on protection procedures. Information loss and disclosure risk measures are also described.

Minimum spanning tree partitioning algorithm for microaggregation

This paper presents a clustering algorithm for partitioning a minimum spanning tree with a constraint on minimum group size. The problem is motivated by microaggregation, a disclosure limitation technique in which similar records are aggregated into groups containing a minimum of k records. Heuristic clustering methods are needed since the minimum information loss microaggregation problem is NP-hard. Our MST partitioning algorithm for microaggregation is sufficiently efficient to be practical for large data sets and yields results that are comparable to the best available heuristic methods for microaggregation. For data that contain pronounced clustering effects, our method results in significantly lower information loss. Our algorithm is general enough to accommodate different measures of information loss and can be used for other clustering applications that have a constraint on minimum group size.

An Integral approach to dynamic data rectification

Because industrial process is often under changes, it is necessary to reconcile the measurement of the dynamic process. There are many problems in the present dynamics data reconciliation methods, such as low calculation efficiency, difficulty of treatment for the reconciliation of input variables. Based on the analysis of the characteristic of dynamic data reconciliation, an integral approach is presented which integrates finite element collocation method, filtering technique and robust method. The finite element collocation method can reduce the amount of discrete model constrained equations to decrease the problem complexity without any loss of measurement information. The filtering technique can eliminate random errors in input variables effectively and no lag or signal distortion will be introduced. Monte Carlo method is used to test the efficiency of the robust method for gross error detection. The calculation results show that the integral approach can improve the calculation efficiency greatly and can deal with the gross error of abnormal type properly.

Direct Machine Translation Systems as Dynamical Systems: The Iterative Semantic Processing (ISP) Paradigm

Direct machine translation systems are usually implemented as uni-directional mappings between a source and target sentence, where the accuracy of the translation is evaluated with respect to the consistency of meaning of the target with respect to the source. However, this approach does not allow us to understand how comparable the semantic representations in target and source really are, since both are described by words in different (and possibly semantically incompatible) languages. In this paper, we generalise the common method of inverting translations to generate a mapping between target and source, by proposing a new, iterative translation methodology which is based on dynamical systems theory, the Iterative Semantic Processing (ISP) Paradigm. The basic premise of this paradigm is that the preservation (or loss) of semantic representations during successive iterative translations between source and target, target and source, source and target, and so on, can be best described as one of three possible dynamical system states: point attractors (i.e., invariant semantic representations), limit cycles (i.e., steady-state variant but predictable cross-language mappings) and chaotic cycles (i.e., variant mappings with rapid short-term information loss). The mathematical properties of each of these system states as a measure of semantic stability are described in this study, and quantitative measures of semantic information loss are derived, and applied to semantic-mis-translations from a freely-available direct translation system.

Information theoretical performance measure for associative memories and its application to neural networks.

We present a general performance measure (information loss) for associative memories based on information theoretical concepts. This performance measure can be estimated, provided that mean values of observables have been determined for the associative memory. Then the estimation guarantees a minimal association quality. The formalism allows the application of the performance measure to complex systems where the relation between input and output of the associative memory is not explicitly known. Here we apply our formalism to the Hopfield model and estimate the storage capacity alpha(c) from the numerically determined information loss. In contrast to other numerical methods the whole overlap distribution is taken into account. Our numerical value alpha(c)=0.1379(4) for the storage capacity in the Hopfield model is below numerical values obtained previously. This indicates that the consideration of small remnant overlaps lowers the storage capacity of the Hopfield model.

Measuring information loss due to inconsistencies in duration data from longitudinal surveys

In this paper, we take a functional errors-in-variables approach to modeling longitudinal duration data containing observed inconsistencies. This approach separates the problem into two parts: an errors-in-variables problem and an estimation problem. This approach also enables us to develop measures of the effect of the data inconsistencies on the parameter estimates from a duration model. One measure is the loss in precision in the parameter estimates that results from the inconsistencies. We refer to the proportion of parameter variances that is due to the inconsistencies in the data as information loss. The second is a distance measure which is used to provide information about how close we are to a consistent estimator. An example is provided using a four-month panel of Current Population Survey data.

Measuring information loss due to inconsistencies in duration data from longitudinal surveys

In this paper, we take a functional errors-in-variables approach to modeling longitudinal duration data containing observed inconsistencies. This approach separates the problem into two parts: an errors-in-variables problem and an estimation problem. This approach also enables us to develop measures of the effect of the data inconsistencies on the parameter estimates from a duration model. One measure is the loss in precision in the parameter estimates that results from the inconsistencies. We refer to the proportion of parameter variances that is due to the inconsistencies in the data as information loss. The second is a distance measure which is used to provide information about how close we are to a consistent estimator. An example is provided using a four-month panel of Current Population Survey data.

A randomisation method for discriminating between signal and noise in recordings of rhythmic electromyographic activity

A method is described which enables an amplitude threshold level to be derived from previously recorded rhythmic electromyographic activity, or from other rhythmic physiological data, so that the signal and noise components can be best separated. The method utilises randomisation of the original data and a non-parametric measure of the resultant information loss: this avoids any assumptions about the amplitude density function. The probability that the division into signal and noise is no better than chance may also be calculated.

INFLUENCE OF MATING TYPE AND SELECTION ON THE GENETIC INFORMATION FLOW

In this work a quantitative evaluation of the influence of mating type and selection on the genetic information flow in ecosystems is considered. Assuming the Schmalhausen’s model of information dynamics, a measure of information that accounts both, for the amount of genetic information in the population and the transformations induced by selection and mating type on the amount of information is employed. Furthermore, some measures of information transfer, channel noises and information loss are employed in the evaluation of two sexual reproductive types very relevant in plant populations: random mating and selfing. The measures employed arise from the Shannon’s General Theory of Communication. They are conditional entropies and the average mutual information. Reproduction takes place in the past generation after transformation of the genetic structure of the population through selection. In case of clastic selection an expression for the measure of fitness entropy is computed. Basic Mendelian rules of inheritance are assumed in the derivation of the one-generation transition probabilities. Finally, a numerical application is analyzed.

Information, ancillarity, and sufficiency in the presence of nuisance parameters

AbstractThe Fisher information about parameters of interest (P‐information) is invariant with respect to nuisance parameters, and induces an information inequality associated with likelihood factorization. This information inequality provides a natural basis for measuring information loss due to using only a sublikelihood function for inference. In contrast with the global reparametrization of some previous concepts in the literature, the concepts of P‐ancillarity and P‐sufficiency proposed in this article are characterized by the notion of no pointwise information loss with respect to the parameters of interest. A conditional version of P‐sufficiency is also proposed. The asymptotic efficiency of likelihood inference under P‐ancillarity or P‐sufficiency is outlined.

Asymptotic Likelihood-Based Prediction Functions

This paper develops asymptotic prediction functions that approximate the shape of the density of future observations and correct for parameter uncertainty. The functions are based on extensions to a definition of predictive likelihood originally suggested by Lauritzen and Hinkley. The prediction function is shown to possess efficiency properties based on the Kullback-Leibler measure of information loss. Examples of the application of the prediction function and the derivation of relative efficiency are shown for linearnormal models, nonnormal models, and ARCH models.

ASSESSMENT OF INFORMATION LOSS IN SCORING MONOTONE ITEMS

For monotone, dichotomous items, two interpretations of the usual scoring procedure may be made. One interpretation (A) is that failure of the items to form a perfect Guttman Scale leads to loss of discriminative infor- mation when the usual procedure of scoring is applied. Another interpretation (B) is that nonunidimensionality of the item-set leads to information confusion under usual scoring procedures. Measures of information loss for each interpretation are developed and tested using Monte Carlo methods. It is concluded that interpretation (B) is the most appropriate.