Information Theoretic Criteria Research Articles

Is Cryocide an Ethically Feasible Alternative to Euthanasia?

While some countries are moving toward legalization, euthanasia is still criticized on various fronts. Most importantly, it is considered a violation of the medical ethics principle of non-maleficence, because it actively seeks a patient's death. But, medical ethicists should consider an ethical alternative to euthanasia. In this article, we defend cryocide as one such alternative. Under this procedure, with the consent of terminally-ill patients, their clinical death is induced, in order to prevent the further advance of their brain's deterioration. Their body is then cryogenically preserved, in the hope that in the future, there will be a technology to reanimate it. This prospect is ethically distinct from euthanasia if a different criterion of death is assumed. In the information-theoretic criterion of death, a person is not considered dead when brain and cardiopulmonary functions cease, but rather, when information constituting psychology and memory is lost.

Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data.

Standard machine learning is unable to accommodate inputs which do not belong to the training distribution. The resulting models often give rise to confident incorrect predictions which may lead to devastating consequences. This problem is especially demanding in the context of dense prediction since input images may be only partially anomalous. Previous work has addressed dense out-of-distribution detection by discriminative training with respect to off-the-shelf negative datasets. However, real negative data may lead to over-optimistic evaluation due to possible overlap with test anomalies. To this end, we extend this approach by generating synthetic negative patches along the border of the inlier manifold. We leverage a jointly trained normalizing flow due to a coverage-oriented learning objective and the capability to generate samples at different resolutions. We detect anomalies according to a principled information-theoretic criterion which can be consistently applied through training and inference. The resulting models set the new state of the art on benchmarks for out-of-distribution detection in road-driving scenes and remote sensing imagery despite minimal computational overhead.

Conformal Field Theory Ground States as Critical Points of an Entropy Function.

We derive an entropy formula satisfied by the ground states of 1+1D conformal field theories. The formula implies that the ground state is the critical point of an entropy function. We conjecture that this formula may serve as an information-theoretic criterion for conformal field theories, which differs from the conventional algebraic definition. In addition to these findings, we use the same proof method to extract the six global conformal generators of the conformal field theory from its ground state. We validate our results by testing them on different critical lattice models, with excellent agreement.

Step‐stress life‐testing under tampered random variable modeling for Weibull distribution in presence of competing risk data

AbstractIn this paper, we have considered the classical and Bayesian inference of the unknown parameters of the lifetime distribution based on the observations obtained from a simple step‐stress life‐testing (SSLT) experiment and when more than one cause of failures are observed. We have used the Tampered Random Variable (TRV) approach. The main advantage of the TRV approach is that it can be easily extended to a multiple step‐stress model as well as for different lifetime distributions. In this paper, it is assumed that the lifetime of the experimental units at each stress level follows Weibull distribution with the same shape parameter and different scale parameters. Further, we have introduced different tempering co‐efficient for different causes of failures. The maximum likelihood estimators and the associated asymptotic confidence intervals are obtained based on Type‐II censored observations. Further, we have considered the Bayesian inference of the unknown model parameters based on a fairly general class prior distributions. An extensive simulation study is performed to examine the performances of the proposed method, and the analysis of a real data set has been provided to show how the method can be used in practice. We have compared the TRV model with some of the other existing models, and the TRV model provides a better fit in terms of information theoretic criteria. We have also provided some optimality criteria, to determine the optimal stress change time and some sensitivity analyses have been performed. Most of the methods can be extended for other lifetime distributions also.

Fault detection and isolation in uncertain dynamic systems using composite optimization and inferential sensing

Inferential sensing is a cost-effective and reliable approach to replace expensive and impractical hardware sensing, while yielding robust fault detection and isolation (FDI) during system maintenance or operation. Yet, their high computational footprint necessitates the employment of advanced and efficient estimation techniques, along with symbolic mathematics and automatic differentiation. Herein, we attempt to combine model-driven sensor selection methods with inferential sensing to accurately identify faults in the presence of system noise and uncertainty. During sensor selection, we employ criteria from information theory to maximize the estimability of system faults from available system measurements. The most informative sensor set stems from a comparison of all possible sensor suites via information theoretic criteria expressed as functions of the Fisher Information Matrix. Optimal inferential sensors are calculated through a novel fusion of symbolic regression and information theory. For built-in test deployment, we utilize k-Nearest Neighbors classification to assess the capability of each sensor suite for all plausible instantiations of uncertainty and system noise, and evaluate FDI performance benefits from the inclusion of inferential sensor(s). The proposed computational framework is deployed with steady-state and dynamic models of a cross-flow plate-fin heat exchanger, at varying levels of measurement noise and uncertainty. As shown, the augmented system of composite sensors (i.e., inferential and hardware) provides accurate information on system fault(s) and reduces the evidence of uncertainty. Compared to traditional sensor approaches, the proposed techniques are shown to be more precise while simultaneously granting increased insight into details otherwise hidden by noise. Thus, they form a robust and reliable tool when conducting fault diagnosis and possibly toward system safety.

Direction‐of‐arrival estimation via coarray‐domain RELAX algorithm with source number estimation

AbstractA modified RELAX algorithm based on iterative coarray‐domain beamforming for fast source direction‐of‐arrival (DOA) estimation with fully augmentable sparse arrays is proposed. The authors exploit the deterministic centralised nature of the noise in the coarray domain and the Hermitian symmetry of the spatial autocorrelation function to efficiently incorporate source number estimation within the iterative framework. In doing so, the proposed algorithm allows low‐complexity, fast DOA estimation of more sources than sensors, without resorting to computationally expensive implementations of source number estimation using information theoretic criteria. Three variants of the proposed algorithm are presented, each differing in terms of the specific method employed for source number estimation. Extensive simulations are performed with a minimum redundancy array to compare and contrast the performance of the three variants in terms of their accuracy in estimating the number of sources in the field‐of‐view of the array. The results demonstrate that the modified RELAX algorithm can provide accurate estimates of the number and directions of sources, especially when the number of uncorrelated sources is equal to or higher than the number of sensors.

A practical utility-based but objective approach to model selection for regression in scientific applications

In many fields of science, various types of models are available to describe phenomena, observations and the results of experiments. In the last decades, given the enormous advances of information gathering technologies, also machine learning techniques have been systematically deployed to extract models from the large available databases. However, regardless of their origins, no universal criterion has been found so far to select the most appropriate model given the data. A unique solution is probably a chimera, particularly in applications involving complex systems. Consequently, in this work a utility-based approach is advocated. However, the solutions proposed are not purely subjective but all based on “objective” criteria, rooted in the properties of the data, to preserve generality and to allow comparative assessments of the results. Several methods have been developed and tested, to improve the discrimination capability of basic Bayesian and information theoretic criteria, with particular attention to the BIC (Bayesian Information Criterion) and AIC (Akaike Information Criterion) indicators. Both the quality of the fits and the evaluation of model complexity are aspects addressed by the advances proposed. The competitive advantages of the individual alternatives, for both cross sectional data and time series, are clearly identified, together with their most appropriate fields of application. The proposed improvements of the criteria allow selecting the right models more reliably, more efficiently in terms of data requirements and can be adjusted to very different circumstances and applications. Particular attention has been paid to ensure that the developed versions of the indicators are easy to implement in practice, in both confirmatory and exploratory settings. Extensive numerical tests have been performed to support the conceptual and theoretical considerations.

On the selection of predictors by using greedy algorithms and information theoretic criteria

SummaryWe discuss the use of the following greedy algorithms in the prediction of multivariate time series: Matching Pursuit Algorithm (MPA), Orthogonal Matching Pursuit (OMP), Relaxed Matching Pursuit (RMP), Frank–Wolfe Algorithm (FWA) and Constrained Matching Pursuit (CMP). The last two are known to be solvers for the lasso problem. Some of the algorithms are well‐known (e.g. OMP), while others are less popular (e.g. RMP). We provide a unified presentation of all the algorithms, and evaluate their computational complexity for the high‐dimensional case and for the big data case. We show how 12 information theoretic (IT) criteria can be used jointly with the greedy algorithms. As part of this effort, we derive new theoretical results that allow modification of the IT criteria such that to be compatible with RMP. The prediction capabilities are tested in experiments with two data sets. The first one involves air pollution data measured in Auckland (New Zealand) and the second one concerns the House Price Index in England (the United Kingdom).

Optimum calibration of a corrosion rate instrument using information gain criterion within a Bayesian framework

Structural health monitoring (SHM) and non-destructive testing (NDT) provide necessary information in assessing the current condition of an ageing structure. However, measurements obtained from SHM and NDT can be erroneous and it is important that the uncertainties associated with these measurement errors are quantified through instrument calibration. Considering that the process of calibration is expensive and time consuming, a novel optimum instrument calibration methodology is proposed in this work, which uses an information theoretic criterion to decide on the optimum number of calibration data points required. The proposed methodology is demonstrated through the calibration of a corrosion rate instrument with measurement uncertainty. The calibrated model is validated using Bayes’ factor as the validation metric. Case studies demonstrate 42.5% and more reduction in the required number of calibration data points. The proposed method also suggests an optimum sequence of data collection for calibration.

Codesign for Hybrid MU-MIMO Communication and MIMO Radar Systems Based on Mutual Information

To enhance the coexistence performance of multiple-input multiple-output (MIMO) radar system and multiuser MIMO communication system with hybrid beamforming architecture, this work proposes a codesign problem regarding the digital and analog beamformers of communication base station, the receive beamformers of downlink users, and the transmit waveform of radar. An information-theoretic criterion is adopted to improve the radar detection performance, while the signal-to-interference-plus-noise ratio requirement is imposed on each downlink user to ensure communication quality. In addition, similarity restriction is also involved to manage some significant radar waveform characteristics. An iterative algorithm is introduced to tackle the resulted nonconvex and NP-hard problem. In each iteration, several subproblems are developed to decompose the original problem and are solved by the lower bound maximization and the alternating direction method of multiplier algorithm. Numerical simulations show that the proposed algorithm allows the objective function to increase monotonically and finally converge to a finite value. It also shows that the codesign scheme can effectively optimize the radar detection performance while ensuring the achievable sum rate and the bit error rate of the communication system.

Measuring the relative contribution to predictive power of modern nucleotide substitution modeling approaches.

Traditional approaches to probabilistic phylogenetic inference have relied on information-theoretic criteria to select among a relatively small set of substitution models. These model selection criteria have recently been called into question when applied to richer models, including models that invoke mixtures of nucleotide frequency profiles. At the nucleotide level, we are therefore left without a clear picture of mixture models' contribution to overall predictive power relative to other modeling approaches. Here, we utilize a Bayesian cross-validation method to directly measure the predictive performance of a wide range of nucleotide substitution models. We compare the relative contributions of free nucleotide exchangeability parameters, gamma-distributed rates across sites, and mixtures of nucleotide frequencies with both finite and infinite mixture frameworks. We find that the most important contributor to a model's predictive power is the use of a sufficiently rich mixture of nucleotide frequencies. These results suggest that mixture models should be given greater consideration in nucleotide-level phylogenetic inference.

Signal Detection in MIMO Communications System with Non-Gaussian Noises based on Deep Learning and Maximum Correntropy Criterion

In this paper, we study signal detection in multi-input-multi output (MIMO) communications system with non-Gaussian noises such as Middleton Class A noise, Gaussian mixtures and alpha stable distributions, using several deep neural network-based detector models such as FULLYCONNECTED and DETNET detector. By applying information theoretic criterion of Maximum Correntropy , SVD analysis on the channel matrix and reducing network complexity, the suggested deep neural network detector performs well in environments with non-Gaussian noises and, compared to the deep neural network-based detector with MSE loss function, achieves better performance.

Complexity: Frontiers in Data-Driven Methods for Understanding, Prediction, and Control of Complex Systems 2022 on the Development of Information Theoretic Model Selection Criteria for the Analysis of Experimental Data

It can be argued that the identification of sound mathematical models is the ultimate goal of any scientific endeavour. On the other hand, particularly in the investigation of complex systems and nonlinear phenomena, discriminating between alternative models can be a very challenging task. Quite sophisticated model selection criteria are available but their deployment in practice can be problematic. In this work, the Akaike Information Criterion is reformulated with the help of purely information theoretic quantities, namely, the Gibbs-Shannon entropy and the Mutual Information. Systematic numerical tests have proven the improved performances of the proposed upgrades, including increased robustness against noise and the presence of outliers. The same modifications can be implemented to rewrite also Bayesian statistical criteria, such as the Schwartz indicator, in terms of information-theoretic quantities, proving the generality of the approach and the validity of the underlying assumptions.

Multivariate Time Series Imputation: An Approach Based on Dictionary Learning.

The problem addressed by dictionary learning (DL) is the representation of data as a sparse linear combination of columns of a matrix called dictionary. Both the dictionary and the sparse representations are learned from the data. We show how DL can be employed in the imputation of multivariate time series. We use a structured dictionary, which is comprised of one block for each time series and a common block for all the time series. The size of each block and the sparsity level of the representation are selected by using information theoretic criteria. The objective function used in learning is designed to minimize either the sum of the squared errors or the sum of the magnitudes of the errors. We propose dimensionality reduction techniques for the case of high-dimensional time series. For demonstrating how the new algorithms can be used in practical applications, we conduct a large set of experiments on five real-life data sets. The missing data (MD) are simulated according to various scenarios where both the percentage of MD and the length of the sequences of MD are considered. This allows us to identify the situations in which the novel DL-based methods are superior to the existing methods.

Operational reality of quantum nonlocality

Does the remote measurement-disturbance of the quantum state of a system $B$ by measurement on system $A$ entangled with $B$, constitute a real disturbance -- i.e., an objective alteration -- of $B$ in an operational sense? Employing information theoretic criteria motivated by operational considerations alone, we argue that the disturbance in question is real for a subset of steerable correlations. This result highlights the distinction between quantum no-signaling and relativistic signal-locality. It furthermore suggests a natural reason why a convex operational theory should be non-signaling: namely, to ensure the consistency between the properties of reduced systems and those of single systems.

Generalized maximum entropy based identification of graphical ARMA models

This paper focuses on the joint estimation of parameters and topologies of multivariate graphical autoregressive moving-average (ARMA) processes. Since the graphical structure of a model can be characterized by the sparsity pattern of its inverse spectral density matrix, a generalized maximum entropy optimization model is derived by imposing a sparse regularization on the inverse spectrum from which the parameters of the AR part and the graph topology can be simultaneously estimated. Then, the whole graphical ARMA model is identified by alternatingly estimating the graphical AR part by solving the regularized maximum entropy problem and the MA part by applying the moment estimation method. The weight to the sparsity regularization term is chosen according to the information theoretic model selection criterion, and simulation examples are provided to illustrate the effectiveness of the proposed identification approach.

Dictionary learning for signals in additive noise with generalized Gaussian distribution

We propose a dictionary learning (DL) algorithm for signals in additive noise with generalized Gaussian distribution (GGD) by redesigning three key components used in DL for Gaussian signals: (i) the orthogonal matching pursuit algorithm, (ii) the approximate K-SVD algorithm and (iii) the information theoretic criteria. In experiments with simulated data, we show that the performance of the new algorithm is higher or equal to the performance of the DL algorithms for signals in Laplacian noise. We also discuss how the shape parameter of the GGD noise can be estimated. For image data, we examine the relationship between the complexity of the DL model and the errors obtained on the test set. This provides guidance on the values of the shape parameter that should be employed in image modeling.

Sparsity Regularized Deep Subspace Clustering for Multicriterion-Based Hyperspectral Band Selection

Hyperspectral images provide rich spectral information corresponding to visible and Near-infrared imaging (NIR) regions, facilitating accurate classication, object identication, and target detection. However, the high volume of data creates a computational challenge in processing. The band selection process identies specic informative and discriminative spectral bands from the data speed up the processing without impeding the performance. This paper presents an application-independent band selection framework that utilizes improved sparse deep subspace clustering and introduces an efcient multicriteria-based representative band selection (BS). The proposed sparse deep subspace clustering approach efciently identies the underlying non-linear subspace structure of the data and organizes the data accordingly. The work introduces a novel, robust sparsity measure to obtain a powerful self-representation and ameliorated performance compared to the prevalent subspace clustering methods. The work subsequently selects the representative bands from each cluster by combining structural information of the band images with the statistical similarity measure. We evaluate the BS performance on standard real images using information-theoretic criterion, classication, and unmixing performance. The comparative performance assessment demonstrates that our proposed method identies the informative bands and out.

Development of Methods for Identifying Nonlinear Stochastic Systems According to Information-theoretic Criteria based on Symmetric Measures of Divergence and Taking into Account Quantitative Characteristics of Identifiability

The paper considers the problems of identifying non-linear stochastic systems and its application to a specific task of the nuclear power plant control. The main attention is paid to the application of consistent (what means vanishing if and only if two random (in general) vectors are stochastically independent) measures of dependence. There are presented particular important cases concerned with applying Tsallis divergence as a basic tool to construct a corresponding measure of dependence leading, finally, to obtaining a total input/output system model by the use of local ones.

Developmental gene regulatory network connections predicted by machine learning from gene expression data alone.

Gene regulatory network (GRN) inference can now take advantage of powerful machine learning algorithms to complement traditional experimental methods in building gene networks. However, the dynamical nature of embryonic development–representing the time-dependent interactions between thousands of transcription factors, signaling molecules, and effector genes–is one of the most challenging arenas for GRN prediction. In this work, we show that successful GRN predictions for a developmental network from gene expression data alone can be obtained with the Priors Enriched Absent Knowledge (PEAK) network inference algorithm. PEAK is a noise-robust method that models gene expression dynamics via ordinary differential equations and selects the best network based on information-theoretic criteria coupled with the machine learning algorithm Elastic Net. We test our GRN prediction methodology using two gene expression datasets for the purple sea urchin, Stronglyocentrotus purpuratus, and cross-check our results against existing GRN models that have been constructed and validated by over 30 years of experimental results. Our results find a remarkably high degree of sensitivity in identifying known gene interactions in the network (maximum 81.58%). We also generate novel predictions for interactions that have not yet been described, which provide a resource for researchers to use to further complete the sea urchin GRN. Published ChIPseq data and spatial co-expression analysis further support a subset of the top novel predictions. We conclude that GRN predictions that match known gene interactions can be produced using gene expression data alone from developmental time series experiments.