Relative Information Loss Research Articles

How to Select an Objective Function Using Information Theory

AbstractIn machine learning or scientific computing, model performance is measured with an objective function. But why choose one objective over another? According to the information‐theoretic paradigm, the “best” objective function is whichever minimizes information loss. To evaluate different objectives, transform them into likelihoods. The ratios of these likelihoods represent how strongly we should prefer one objective versus another, and the log of that ratio represents the relative information loss (or gain) from one objective to another. In plain terms, minimizing information loss is equivalent to minimizing uncertainty, as well as maximizing probability and general utility. We argue that this paradigm is well‐suited to models that have many uses and no definite utility like the complex Earth system models used to understand the effects of climate change. Furthermore, the benefits of “maximizing information and general utility” extend beyond model accuracy to other important considerations including how efficiently the model calibrates, how well it generalizes, and how well it compresses data.

Information Recovery in Shuffled Graphs via Graph Matching

While many multiple graph inference methodologies operate under the implicit assumption that an explicit vertex correspondence is known across the vertex sets of the graphs, in practice these correspondences may only be partially or errorfully known. Herein, we provide an information theoretic foundation for understanding the practical impact that errorfully observed vertex correspondences can have on subsequent inference, and the capacity of graph matching methods to recover the lost vertex alignment and inferential performance. Working in the correlated stochastic blockmodel setting, we establish a duality between the loss of mutual information due to an errorfully observed vertex correspondence and the ability of graph matching algorithms to recover the true correspondence across graphs. In the process, we establish a phase transition for graph matchability in terms of the correlation across graphs, and we conjecture the analogous phase transition for the relative information loss due to shuffling vertex labels. We demonstrate the practical effect that graph shuffling—and matching—can have on subsequent inference, with examples from two sample graph hypothesis testing and joint spectral graph clustering.

Model Selection in Seismic Hazard Analysis: An Information-Theoretic Perspective

Abstract Although the methodological framework of probabilistic seismic hazard analysis is well established, the selection of models to predict the ground motion at the sites of interest remains a major challenge. Information theory provides a powerful theoretical framework that can guide this selection process in a consistent way. From an information-theoretic perspective, the appropriateness of models can be expressed in terms of their relative information loss (Kullback–Leibler distance) and hence in physically meaningful units (bits). In contrast to hypothesis testing, information-theoretic model selection does not require ad hoc decisions regarding significance levels nor does it require the models to be mutually exclusive and collectively exhaustive. The key ingredient, the Kullback–Leibler distance, can be estimated from the statistical expectation of log-likelihoods of observations for the models under consideration. In the present study, data-driven ground-motion model selection based on Kullback–Leibler-distance differences is illustrated for a set of simulated observations of response spectra and macroseismic intensities. Information theory allows for a unified treatment of both quantities. The application of Kullback–Leibler-distance based model selection to real data using the model generating data set for the Abrahamson and Silva (1997) ground-motion model demonstrates the superior performance of the information-theoretic perspective in comparison to earlier attempts at data-driven model selection (e.g., Scherbaum et al. , 2004).

XML-Based DICOM Data Format

To enhance the readability, improve the structure, and facilitate the sharing of digital imaging and communications in medicine (DICOM) files, this research proposed one kind of XML-based DICOM data format. Because XML Schema offers great flexibility for expressing constraints on the content model of elements, we used it to describe the new format, thus making it consistent with the one originally defined by DICOM. Meanwhile, such schemas can be used in the creation and validation of the XML-encoded DICOM files, acting as a standard for data transmission and sharing on the Web. Upon defining the new data format, we started with representing a single data element and further improved the whole data structure with the method of modularization. In contrast to the original format, the new one possesses better structure without loss of related information. In addition, we demonstrated the application of XSLT and XQuery. All of the advantages mentioned above resulted from this new data format.

A Molecular Cytogenetics Profile of Chromosome 9p in CMPD.

Chronic Myeloproiferative Disorders (CMPDs) form a continuum of haematological malignancies of which the molecular pathogenesis remains obscure. Chronic Myeloid Leukaemia (CML) being the only disorder with a recognisable pathogenomic abnormality, the Ph chromosome. Tyrosine kinases are clearly implicated in the disease pathogenesis, in particular the role of JAK2. It has been shown that over 80% of Polycythaemia Vera (PV) patients and 30% of Essential Thrombocythaemia (ET) patients demonstrate a clonal and recurrent mutation V617F in the JH2 pseudo-kinase domain of JAK2 located on 9p24.1. Array CGH investigations of cell lines, revealed a complex pattern of imbalances affecting the short arm of chromosome 9: on the background of 2–3-fold amplification, a loss of the 9p24.1 region, which harbours the JAK 2 along with a number of other kinases, was repeatedly observed. The aim of this study was to clarify the aCGH observations using a series of BAC DNA probes mapping from the telomere of 9p toward the centromere. Results indicate the presence of imbalances (amplifications & deletions) and rearrangements, involving the region encompassing JAK2. In all CML-BC cell lines the FISH data supports the observation of a relative loss of the 9p24.1 region on the background of amplification of chromosome 9. An on-going study of CMPD and MDS patients with both normal and abnormal karyotypes has revealed imbalances affecting the short arm of chromosome 9. In 23% (7/30) of patients there were imbalances; the majority of these (56%) demonstrated a relative loss of information in this region. The pathegenomic significance of imbalances involving 9p remains unclear. It may however represent an acquired homozygousity as a consequence of mitotic recombination, with a ‘second hit' removing a wild type allele, leading on to further, as yet un-clarified changes.

Cardiac gated equilibrium radionuclide angiography and multiharmonic Fourier phase analysis: Optimal acquisition parameters in arrhythmogenic right ventricular cardiomyopathy

Background. Multiharmonic Fourier phase analysis of radionuclide angiography is a well-established method for the diagnosis of arrhythmogenic right ventricular cardiomyopathy. We sought to determine the optimal acquisition parameters: number of frames per cycle and number of counts per frame, with all other acquisition and processing parameters being fixed. Methods and Results. Radionuclide angiography with list mode acquisition was performed in 10 normal subjects (pilot group) and 11 patients with arrhythmogenic right ventricular cardiomyopathy (validation group), allowing the reconstruction of electrocardiography-gated constant phase studies with different parameters: 16, 24, and 32 frames per cycle and 200, 400, 600, and 800 kcounts per frame. Three harmonics Fourier phase analysis was applied, and optimal acquisition parameters (defined as those providing best homogeneous phase distribution histogram in the pilot group) were defined as judged by the H3 right ventricular phase SD and Δ 95%. These were 16 frames per cycle and 600 kcounts per frame. Then we verified in the validation group that these optimal acquisition parameters did not induce any significant relative loss of information compared with other acquisition parameters with more temporal resolution (24 and 32 frames per cycle) or more statistics (800 kcounts per frame). This result was realized by the calculation of normalized H3 right ventricular SD, right ventricular Δ 95%, and (SD [left ventricle] - SD [right ventricle]). Conclusions. In practice, 16 frames per cycle and 600 kcounts per frame are optimal for multiharmonic Fourier phase analysis, with all other acquisition and processing variables being fixed as specified.

Relative information loss under Type II censored exponential data

SUMMARY This paper uses information theory to quantify information loss in Type II censored samples drawn from an exponential distribution. Indexes of information loss for the maximum likelihood estimation and for Bayesian analysis are defined. Properties of the proposed information measures as functions of the sample size, the exponential parameter, and the parameters of the prior distribution are studied. It is shown that the relative loss of information in Type II censoring may be compensated by increasing the prior precision. Tables of the information loss indexes are provided.

A Note on Block Designs with Nested Rows and Columns

AbstractBlock designs with nested rows and columns have been introduced by Singh and Dey (1979). This note gives some results regarding the total relative loss of information and patterns of efficiency balanced block designs with nested rows and columns in variable replications of treatments.

On the Optimum Choice of Categories for the Classification of Biomedical Data Patterns

The optimum choice of categories for pattern recognition problems is on the one hand determined by the requirement of a low rate of misclassification. On the other hand, the classification of a pattern should result in an information gain as high as possible. A criterion for an optimum choice of categories which is the best compromise between the demands mentioned above is worked out. The Bayes rule is used as a decision function. The alteration of the Bayes risk as indicator for the rate of malrecognition is examined for different choice of categories concerning the very same classification problem. As the calculation of the Bayes risk is commonly difficult, an estimation using the Bhattacharyya coefficient is given. The information content of a choice of categories is defined using Shannon’s information measure. The alteration of the information contents is analyzed by putting together certain categories, i.e. a coarser choice of categories. With the aid of the relative information loss and the relative reduction of the Bayes risk coefficient, a criterion on the goodness of a choice of categories can be given. The criterion also serves as an optimum choice of classes. The extension of the latter criterion to a generalized decision rule is possible.

On the Optimum Choice of Categories for the Classification of Biomedical Data Patterns

The optimum choice of categories for pattern recognition problems is on the one hand determined by the requirement of a low rate of misclassification. On the other hand, the classification of a pattern should result in an information gain as high as possible. A criterion for an optimum choice of categories which is the best compromise between the demands mentioned above is worked out. The Bayes rule is used as a decision function. The alteration of the Bayes risk as indicator for the rate of malrecognition is examined for different choice of categories concerning the very same classification problem. As the calculation of the Bayes risk is commonly difficult, an estimation using the Bhattacharyya coefficient is given. The information content of a choice of categories is defined using Shannon’s information measure. The alteration of the information contents is analyzed by putting together certain categories, i.e. a coarser choice of categories. With the aid of the relative information loss and the relative reduction of the Bayes risk coefficient, a criterion on the goodness of a choice of categories can be given. The criterion also serves as an optimum choice of classes. The extension of the latter criterion to a generalized decision rule is possible.

A Note on the Total Relative Loss of Information in any Design

A Note on the Total Relative Loss of Information in any Design