Kullback-Leibler Information Criterion Research Articles

Functional Mixed Effects Clustering with Application to Longitudinal Urologic Chronic Pelvic Pain Syndrome Symptom Data

By clustering patients with the urologic chronic pelvic pain syndromes (UCPPS) into homogeneous subgroups and associating these subgroups with baseline covariates and other clinical outcomes, we provide opportunities to investigate different potential elements of pathogenesis, which may also guide us in selection of appropriate therapeutic targets. Motivated by the longitudinal urologic symptom data with extensive subject heterogeneity and differential variability of trajectories, we propose a functional clustering procedure where each subgroup is modeled by a functional mixed effects model, and the posterior probability is used to iteratively classify each subject into different subgroups. The classification takes into account both group-average trajectories and between-subject variabilities. We develop an equivalent state-space model for efficient computation. We also propose a cross-validation based Kullback–Leibler information criterion to choose the optimal number of subgroups. The performance of the proposed method is assessed through a simulation study. We apply our methods to longitudinal bi-weekly measures of a primary urological urinary symptoms score from a UCPPS longitudinal cohort study, and identify four subgroups ranging from moderate decline, mild decline, stable and mild increasing. The resulting clusters are also associated with the one-year changes in several clinically important outcomes, and are also related to several clinically relevant baseline predictors, such as sleep disturbance score, physical quality of life and painful urgency. Supplementary materials for this article are available online.

Adaptive Radar Detection in the Presence of Multiple Alternative Hypotheses Using Kullback-Leibler Information Criterion-Part II: Applications

This paper deals with adaptive radar detection problems where several alternative hypotheses may be plausible. This kind of problems naturally extends the conventional binary tests that often occur in radar (as well as in other application fields) by including a further uncertainty degree related to the number of unknown signal parameters (model order). Such a modification consequently leads to multiple composite alternative hypotheses. In the companion paper (Addabbo et al., 2021), we have defined a new design framework which allows us to come up with decision schemes for these hypothesis testing problems by exploiting the Kullback-Leibler Information Criterion and without resorting to heuristic design criteria. The architectures devised within the proposed framework consist of the sum between the compressed log-likelihood ratio and a penalty term inherited from model order selection rules. Such a penalty term accounts for the number of unknown parameters to overcome the limitation of the generalized likelihood ratio test in the presence of nested hypotheses. In the present paper, we apply the new design framework to different detection problems related to both real aperture and (polarimetric) synthetic aperture radar. The analysis is carried out in comparison with suitable competitors (possibly based upon heuristic design criteria) and shows that the architectures devised within the proposed theoretically-founded design framework represent an effective means to deal with detection problems where the uncertainty on some parameters leads to multiple alternative hypotheses.

Parameter Selection Criteria for Tomo-SAR Focusing

The synthetic aperture radar (SAR) tomography (TomoSAR) inverse problem is commonly tackled in the context of the direction-of-arrival estimation theory. The latter allows achieving super resolution, along with ambiguity levels reduction, thanks to the use of parametric focusing methods, as multiple signal classification (MUSIC) and statistical regularization techniques, like the maximum-likelihood-inspired adaptive robust iterative approach (MARIA). Nevertheless, in order to correctly suit the considered signal model, MUSIC and most regularization approaches require an appropriate setting of the involved parameters. In both cases, the accuracy of the retrieved solutions depends on the right selection of the assigned values. Thus, with the aim of dealing with such an issue, this article addresses several parameter selection strategies, adapted specifically to the TomoSAR scenario. Parametric techniques as MUSIC solve the TomoSAR problem in a different manner as the regularization methods do, hence, each approach demands different methodologies for the proper estimation of their parameters. Consequently, we refer to the Kullback-Leibler information criterion for the model order selection of parametric techniques as MUSIC, whereas we rather explore the Morozov's discrepancy principle, the L-Curve, the Stein's unbiased risk estimate, and the generalized cross-validation to choose the regularization parameters. After the incorporation of these criteria to MUSIC and MARIA, respectively, their capabilities are first analyzed through simulations, and later on, utilizing real data acquired from an urban area.

Robust estimation with exponentially tilted Hellinger distance

This paper is concerned with estimation of parameters defined by moment equalities. We introduce the exponentially tilted Hellinger distance (ETHD) estimator which is efficient under correct specification, and robust to both global and local misspecification. In the spirit of Schennach (2007), ETHD combines the Hellinger distance and the Kullback–Leibler information criterion. We show that it achieves optimal minimax robust properties under local deviations from the model and remains well-behaved under global misspecification.

Academic English Proficiency Assessment Using a Computerized Adaptive Test

This paper describes the steps to convert a paper-and-pencil English proficiency test for academic purposes, consisting of multiple choice items administered following the Admissible Probability Measurement Procedure [24], adopted by the graduate program at the Institute of Mathematics and Computer Sciences at the University of São Paulo (ICMC-USP), Brazil, to a computerized adaptive test (CAT) based on an Item Response Theory Model (IRT). Despite the fact that the program accepts various internationally recognized tests that attest non-native speakers English proficiency, such as the Test of English as a Foreign Language (TOEFL), the International English Language Testing System (IELTS) and the Cambridge English: Proficiency (CPE), for instance, its requirement is incoherent in public universities in Brazil due to the cost, which ranges from US$ 200.00 to US$ 300.00 per exam. The TAI-PI software (Computerized Adaptive Test for English Proficiency), which was developed in Java language and SQLite, started to be used to assess the English pro?ciency of students on the program from October, 2013. The statistical methodology used was defined considering the history and aims of the test and adopted Samejima's Graded Response Model [21], the Kullback-Leibler information criterion for item selection, the a posteriori estimation method for latent trait [2] and the Shadow Test approach [29] to impose restrictions (content and test length) on the test composition of each individual. A description of the test design, the statistical methods used, and the results of a real application of TAI-PI for graduate students are presented in this paper, as well as the validation studies of the new methodology for pass or fail classification, showing the good quality of the new evaluation system and examination of improvement using the IRT and CAT methods.

Tensor clustering on outer-product of coefficient and component matrices of independent component analysis for reliable functional magnetic resonance imaging data decomposition

BackgroundStability of spatial components is frequently used as a post-hoc selection criteria for choosing the dimensionality of an independent component analysis (ICA) of functional magnetic resonance imaging (fMRI) data. Although the stability of the ICA temporal courses differs from that of spatial components, temporal stability has not been considered during dimensionality decisions. New methodThe current study aims to (1) develop an algorithm to incorporate temporal course stability into dimensionality selection and (2) test the impact of temporal course on the stability of the ICA decomposition of fMRI data via tensor clustering. Resting state fMRI data were analyzed with two popular ICA algorithms, InfomaxICA and FastICA, using our new method and results were compared with model order selection based on spatial or temporal criteria alone. ResultsHierarchical clustering indicated that the stability of the ICA decomposition incorporating spatiotemporal tensor information performed similarly when compared to current best practice. However, we found that component spatiotemporal stability and convergence of the model varied significantly with model order. Considering both may lead to methodological improvements for determining ICA model order. Selected components were also significantly associated with relevant behavioral variables.Comparison with Existing Method: The Kullback–Leibler information criterion algorithm suggests the optimal model order for group ICA is 40, compared to the proposed method with an optimal model order of 20. ConclusionThe current study sheds new light on the importance of temporal course variability in ICA of fMRI data.

Regularization parameter selection for penalized empirical likelihood estimator

Penalized estimation is a useful technique for variable selection when the number of candidate variables is large. A crucial issue in penalized estimation is the selection of the regularization parameter because the performance of the estimator largely depends on an appropriate choice. However, no theoretically sound selection method currently exists for the penalized estimation of moment restriction models. To address this important issue, we develop a novel information criterion, which we call the empirical likelihood information criterion, to select the regularization parameter of the penalized empirical likelihood estimator. The information criterion is derived as an estimator of the expected value of the Kullback–Leibler information criterion from an estimated model to the true data generating process.

Regularization Parameter Selection for Penalized Empirical Likelihood Estimator

Penalized estimation is a useful technique for variable selection when the number of candidate variables is large. A crucial issue in penalized estimation is the selection of the regularization parameter because the performance of the estimator largely depends on an appropriate choice. However, no theoretically sound selection method currently exists for the penalized estimation of moment restriction models. To address this important issue, we develop a novel information criterion, which we call the empirical likelihood information criterion, to select the regularization parameter of the penalized empirical likelihood estimator. The information criterion is derived as an estimator of the expected value of the Kullback–Leibler information criterion from an estimated model to the true data generating process. We present a Monte Carlo simulation that demonstrates the efficacy of the proposed method.

Components of the item selection algorithm in computerized adaptive testing.

Computerized adaptive testing (CAT) greatly improves measurement efficiency in high-stakes testing operations through the selection and administration of test items with the difficulty level that is most relevant to each individual test taker. This paper explains the 3 components of a conventional CAT item selection algorithm: test content balancing, the item selection criterion, and item exposure control. Several noteworthy methodologies underlie each component. The test script method and constrained CAT method are used for test content balancing. Item selection criteria include the maximized Fisher information criterion, the b-matching method, the a-stratification method, the weighted likelihood information criterion, the efficiency balanced information criterion, and the Kullback-Leibler information criterion. The randomesque method, the Sympson-Hetter method, the unconditional and conditional multinomial methods, and the fade-away method are used for item exposure control. Several holistic approaches to CAT use automated test assembly methods, such as the shadow test approach and the weighted deviation model. Item usage and exposure count vary depending on the item selection criterion and exposure control method. Finally, other important factors to consider when determining an appropriate CAT design are the computer resources requirement, the size of item pools, and the test length. The logic of CAT is now being adopted in the field of adaptive learning, which integrates the learning aspect and the (formative) assessment aspect of education into a continuous, individualized learning experience. Therefore, the algorithms and technologies described in this review may be able to help medical health educators and high-stakes test developers to adopt CAT more actively and efficiently.

Spatial Bayesian surveillance for small area case event data.

There has been little development of surveillance procedures for epidemiological data with fine spatial resolution such as case events at residential address locations. This is often due to difficulties of access when confidentiality of medical records is an issue. However, when such data are available, it is important to be able to affect an appropriate analysis strategy. We propose a model for point events in the context of prospective surveillance based on conditional logistic modeling. A weighted conditional autoregressive model is developed for irregular lattices to account for distance effects, and a Dirichlet tessellation is adopted to define the neighborhood structure. Localized clustering diagnostics are compared including the proposed local Kullback-Leibler information criterion. A simulation study is conducted to examine the surveillance and detection methods, and a data example is provided of non-Hodgkin's lymphoma data in South Carolina.

Expert Information and Nonparametric Bayesian Inference of Rare Events

Prior distributions are important in Bayesian inference of rare events because historical data information is scarce, and experts are an important source of information for elicitation of a prior distribution. I propose a method to incorporate expert information into nonparametric Bayesian inference on rare events when expert knowledge is elicited as moment conditions on a finite dimensional parameter θ only. I generalize the Dirichlet process mixture model to merge expert information into the Dirichlet process (DP) prior to satisfy expert’s moment conditions. Among all the priors that comply with expert knowledge, we use the one that is closest to the original DP prior in the Kullback–Leibler information criterion. The resulting prior distribution is given by exponentially tilting the DP prior along θ. I provide a Metropolis–Hastings algorithm to implement this approach to sample from posterior distributions with exponentially tilted DP priors. The proposed method combines prior information from a statistician and an expert by finding the least-informative prior given expert information.

Model selection in the presence of incidental parameters

This paper considers model selection in panels where incidental parameters are present. Primary interest centers on selecting a model that best approximates the underlying structure involving parameters that are common within the panel. It is well known that conventional model selection procedures are often inconsistent in panel models and this can be so even without nuisance parameters. Modifications are then needed to achieve consistency. New model selection information criteria are developed here that use either the Kullback–Leibler information criterion based on the profile likelihood or the Bayes factor based on the integrated likelihood with a bias-reducing prior. These model selection criteria impose heavier penalties than those associated with standard information criteria such as AIC and BIC. The additional penalty, which is data-dependent, properly reflects the model complexity arising from the presence of incidental parameters. A particular example is studied in detail involving lag order selection in dynamic panel models with fixed effects. The new criteria are shown to control for over/under-selection probabilities in these models and lead to consistent order selection criteria.

Transformation-based nonparametric estimation of multivariate densities

We present a probability-integral-transformation-based estimator of multivariate densities. Given a sample of random vectors, we first transform the data into their corresponding marginal distributions. The marginal densities and the joint density of the transformed data are estimated nonparametrically. The joint density of the original data is constructed as the product of the density of the transformed data and marginal densities, which coincides with the copula representation of multivariate densities. We show that the Kullback–Leibler Information Criterion (KLIC) between the true density and its estimate can be decomposed into the KLIC of the marginal densities and that of the copula density. We derive the convergence rate of the proposed estimator in terms of the KLIC and propose a supervised hierarchical procedure of model selection. Monte Carlo simulations demonstrate the good performance of the estimator. An empirical example on the US and UK stock markets is presented. The estimated conditional copula density provides useful insight into the joint movements of the US and UK markets under extreme Asian markets. • We propose a transformation based nonparametric multivariate density estimator. • We establish convergence in terms of the Kullback–Leibler Information Criterion. • We propose a supervised hierarchical procedure of model specification. • Our simulations demonstrate good performance of the proposed estimator. • Our analysis reveals interesting features of global financial markets.

Comparing the accuracy of multivariate density forecasts in selected regions of the copula support

This paper develops a testing framework for comparing the predictive accuracy of competing multivariate density forecasts with different predictive copulas, focusing on specific parts of the copula support. The tests are framed in the context of the Kullback–Leibler Information Criterion, using (out-of-sample) conditional likelihood and censored likelihood in order to focus the evaluation on the region of interest. Monte Carlo simulations document that the resulting test statistics have satisfactory size and power properties for realistic sample sizes. In an empirical application to daily changes of yields on government bonds of the G7 countries we obtain insights into why the Student-t and Clayton mixture copula outperforms the other copulas considered; mixing in the Clayton copula with the t-copula is of particular importance to obtain high forecast accuracy in periods of jointly falling yields.

Comparing the Accuracy of Copula-Based Multivariate Density Forecasts in Selected Regions of Support

This paper develops a testing framework for comparing the predictive accuracy of copula-based multivariate density forecasts, focusing on a specific part of the joint distribution. The test is framed in the context of the Kullback-Leibler Information Criterion, but using (out-of-sample) conditional likelihood and censored likelihood in order to focus the evaluation on the region of interest. Monte Carlo simulations document that the resulting test statistics have satisfactory size and power properties in small samples. In an empirical application to daily exchange rate returns we find evidence that the dependence structure varies with the sign and magnitude of returns, such that different parametric copula models achieve superior forecasting performance in different regions of the support. Our analysis highlights the importance of allowing for lower and upper tail dependence for accurate forecasting of common extreme appreciation and depreciation of different currencies.

Model Selection in the Presence of Incidental Parameters

This paper considers model selection in nonlinear panel data models where incidental parameters or large-dimensional nuisance parameters are present. Primary interest typically centres on selecting a model that best approximates the underlying structure involving parameters that are common within the panel after concentrating out the incidental parameters. It is well known that conventional model selection procedures are often inconsistent in panel models and this can be so even without nuisance parameters (Han et al, 2012). Modifications are then needed to achieve consistency. New model selection information criteria are developed here that use either the Kullback-Leibler information criterion based on the profile likelihood or the Bayes factor based on the integrated likelihood with the robust prior of Arellano and Bonhomme (2009). These model selection criteria impose heavier penalties than those associated with standard information criteria such as AIC and BIC. The additional penalty, which is data-dependent, properly reflects the model complexity arising from the presence of incidental parameters. A particular example is studied in detail involving lag order selection in dynamic panel models with fixed individual effects. The new criteria are shown to control for over/under-selection probabilities in these models and lead to consistent order selection criteria.

Copula-GARCH versus dynamic conditional correlation: an empirical study on VaR and ES forecasting accuracy

In this paper, we analyze the accuracy of the copula-GARCH and Dynamic Conditional Correlation (DCC) models for forecasting the value-at-risk (VaR) and expected shortfall (ES) of bivariate portfolios. We then try to answer two questions: First, does the correlation-based DCC model outperform the copula models? Second, how can the optimal model for forecasting portfolio risk be identified via in-sample analysis? We address these questions using an extensive empirical study of 1,500 bivariate portfolios containing data on stocks, commodities and foreign exchange futures. Furthermore, we propose to use linear discriminant analysis estimated from descriptive statistics on bivariate data samples as independent variables to identify a parametric model yielding optimal portfolio VaR and ES estimates. In particular, we try to answer the question whether the quality of a parametric model’s VaR and ES estimates is driven by common data characteristics. The results show that the proposed use of linear discriminant analysis is superior to both the Kullback-Leibler Information Criterion and several copula goodness-of-fit tests in terms of overall classification accuracy. Furthermore, the results show that the quality of the DCC model’s VaR and ES estimates is positively correlated with the portfolio marginals’ volatility, while the opposite is true for the elliptical copulas. For the Archimedean copulas in particular, the excess kurtosis of the marginals has a significant positive influence on quality of the VaR and ES estimates.

A GEL-based AIC for model selection

This paper proposes a model selection criterion for models defined by a set of moment conditions that are possibly nonsmooth and discontinuous. The model selection criterion is based on an asymptotically unbiased approximation of the Kullback–Leibler Information Criterion (KLIC), using the generalised empirical likelihood (GEL) criterion function as a semiparametric likelihood function.

Distinguishing between a true period and its alias, and other tasks of model discrimination

We consider the task of distinguishing between two different alternative models that can roughly equally explain observed time series data, mainly focusing on the period ambiguity case (aliasing). We propose a test for checking whether the rival models are observationally equivalent for now or they are already distinguishable. It is the Vuong closeness test, which is based on the Kullback-Leibler Information Criterion. It is asymptotically normal and can work (in certain sense) even in the misspecified case, when the both proposed alternatives are actually wrong. This test is also very simple for practical use. We apply it to several known extrasolar planetary systems and find that our method often helps to resolve various model ambiguities emerging in astronomical practice, but preventing us from hasty conclusions in other cases.

A Transformation-Based Nonparametric Estimator of Multivariate Densities with an Application to Global Financial Markets

We propose a probability-integral-transformation-based estimator of multivariate densities. Given a sample of random vectors, we rst transform the data into their corresponding marginal distributions. We then estimate the density of the transformed data via the Exponential Series Estimator in Wu (2010). The density of the original data is then estimated as the product of the density of the transformed data and marginal densities of the original data. This construction coincides with the copula decomposition of multivariate densities. We decompose the Kullback-Leibler Information Criterion (KLIC) between the true density and our estimate into the KLIC of the marginal densities and that between the true copula density and a variant of the estimated copula density. This result is of independent interest in itself, and provides a framework for our asymptotic analysis. We derive the large sample properties of the proposed estimator, and further propose a stepwise hierarchical method of basis function selection that features a preliminary subset selection within each candidate set. Monte Carlo simulations demonstrate the superior performance of the proposed method. We employ the proposed method to model the joint densities of the US and UK stock market returns under dierent Asian market conditions. The estimated copula density function, a by-product of our estimation, provides useful insight into the conditional dependence structure between the US and UK markets, and suggests a certain resilience against nancial contagions originated from the Asian market.