Structural Zeros Research Articles

Detecting over- and under-dispersion in zero inflated data with the hyper-Poisson regression model

The zero inflated hyper-Poisson regression model permits count data to be analysed with covariates that determine different levels of dispersion and that present structural zeros due to the existence of a non-users group. A simulation study demonstrates the capability of the model to detect over- and under-dispersion of the potential users group of the dataset in relation to the value of covariates, and to estimate the proportion of structural zeros with great accuracy. An application of the model to fit the number of children per family in relation to several covariates confirms the presence of structural zeros in fertility data at the same time as it detects under-dispersion in most of the levels determined by the covariates.

Modeling repeated count measures with excess zeros in an epidemiological study

PurposeHighly skewed count data with excess zeros challenge the application of conventional statistical methods. Additional problems arise from repeated zero-inflated measures. Longitudinal zero-inflated Poisson (ZIP-mixed) models are mixtures of logistic and Poisson models that accommodate excess zeros and repeated counts. We compared a ZIP-mixed model with traditional Poisson and negative binomial models using data on problems with female condom use reported by women at high risk of sexually transmitted diseases. MethodsThe follow-up experience of this cohort represents a mixture of “perfect use” (no opportunity to report problems), represented by the structural zeros, and use experience that bears the risk of condom use problems, represented by a Poisson distribution. ResultsThe ZIP-mixed model provided better fit and richer results than other models. The odds of being in the zero problem category increased with age (odds ratio [OR] = 1.1 per additional year, 95% confidence interval [CI]: 1.0–1.3) and with follow-up (OR = 3.0 per additional month, 95% CI: 1.4–6.0).The nonzero problem rate was lower among women who believed in the benefits of condom use (rate ratio [RR] = 0.9, 95% CI: 0.7–1.0) and had no sexually transmitted diseases at baseline (RR = 0.7, 95% CI: 0.6–0.9), and it decreased during follow-up (RR = 0.8 per additional month, 95% CI: 0.7–0.9). ConclusionsUsing ZIP-mixed model provided further insights into the determinants of condom failure.

Distribution-free inference of zero-inflated binomial data for longitudinal studies

Count responses with structural zeros are very common in medical and psychosocial research, especially in alcohol and HIV research, and the zero-inflated Poisson (ZIP) and zero-inflated negative binomial models are widely used for modeling such outcomes. However, as alcohol drinking outcomes such as days of drinkings are counts within a given period, their distributions are bounded above by an upper limit (total days in the period) and thus inherently follow a binomial or zero-inflated binomial (ZIB) distribution, rather than a Poisson or ZIP distribution, in the presence of structural zeros. In this paper, we develop a new semiparametric approach for modeling ZIB-like count responses for cross-sectional as well as longitudinal data. We illustrate this approach with both simulated and real study data.

PRM237 - Bayesian Models for Cost-Effectiveness Analysis in the Presence of Structural Zero Costs

Bayesian modelling for cost-effectiveness data has received much attention in both the health economics and the statistical literature in recent years. Cost-effectiveness data are characterised by a relatively complex structure of relationships linking the suitable measure of clinical benefit (\eg QALYs) and the associated costs. Simplifying assumptions, such as (bivariate) normality of the underlying distributions are usually not granted, particularly for the cost variable, which is characterised by markedly skewed distributions. In addition, individual-level datasets are often characterised by the presence of structural zeros in the cost variable. Hurdle models can be used to account for the presence of excess zeros in a distribution and have been applied in the context of cost data. We extend their application to cost-effectiveness data, defining a full Bayesian model which consists of a selection model for the subjects with null costs, a marginal model for the costs and a conditional model for the measure of effectiveness (conditionally on the observed costs). The model is presented using a working example to describe its main features.

Bayesian Estimation of Discrete Multivariate Latent Structure Models With Structural Zeros

In multivariate categorical data, models based on conditional independence assumptions, such as latent class models, offer efficient estimation of complex dependencies. However, Bayesian versions of latent structure models for categorical data typically do not appropriately handle impossible combinations of variables, also known as structural zeros. Allowing nonzero probability for impossible combinations results in inaccurate estimates of joint and conditional probabilities, even for feasible combinations. We present an approach for estimating posterior distributions in Bayesian latent structure models with potentially many structural zeros. The basic idea is to treat the observed data as a truncated sample from an augmented dataset, thereby allowing us to exploit the conditional independence assumptions for computational expediency. As part of the approach, we develop an algorithm for collapsing a large set of structural zero combinations into a much smaller set of disjoint marginal conditions, which speeds up computation. We apply the approach to sample from a semiparametric version of the latent class model with structural zeros in the context of a key issue faced by national statistical agencies seeking to disseminate confidential data to the public: estimating the number of records in a sample that are unique in the population on a set of publicly available categorical variables. The latent class model offers remarkably accurate estimates of population uniqueness, even in the presence of a large number of structural zeros.

Bayesian models for cost‐effectiveness analysis in the presence of structural zero costs

Bayesian modelling for cost-effectiveness data has received much attention in both the health economics and the statistical literature, in recent years. Cost-effectiveness data are characterised by a relatively complex structure of relationships linking a suitable measure of clinical benefit (e.g. quality-adjusted life years) and the associated costs. Simplifying assumptions, such as (bivariate) normality of the underlying distributions, are usually not granted, particularly for the cost variable, which is characterised by markedly skewed distributions. In addition, individual-level data sets are often characterised by the presence of structural zeros in the cost variable. Hurdle models can be used to account for the presence of excess zeros in a distribution and have been applied in the context of cost data. We extend their application to cost-effectiveness data, defining a full Bayesian specification, which consists of a model for the individual probability of null costs, a marginal model for the costs and a conditional model for the measure of effectiveness (given the observed costs). We presented the model using a working example to describe its main features. © 2013 The Authors. Statistics in Medicine published by John Wiley & Sons, Ltd.

A likelihood ratio test of quasi-independence for sparse two-way contingency tables

We consider a likelihood ratio test of independence for large two-way contingency tables having both structural (non-random) and sampling (random) zeros in many cells. The solution of this problem is not available using standard likelihood ratio tests. One way to bypass this problem is to remove the structural zeroes from the table and implement a test on the remaining cells which incorporate the randomness in the sampling zeros; the resulting test is a test of quasi-independence of the two categorical variables. This test is based only on the positive counts in the contingency table and is valid when there is at least one sampling (random) zero. The proposed (likelihood ratio) test is an alternative to the commonly used ad hoc procedures of converting the zero cells to positive ones by adding a small constant. One practical advantage of our procedure is that there is no need to know if a zero cell is structural zero or a sampling zero. We model the positive counts using a truncated multinomial distribution. In fact, we have two truncated multinomial distributions; one for the null hypothesis of independence and the other for the unrestricted parameter space. We use Monte Carlo methods to obtain the maximum likelihood estimators of the parameters and also the p-value of our proposed test. To obtain the sampling distribution of the likelihood ratio test statistic, we use bootstrap methods. We discuss many examples, and also empirically compare the power function of the likelihood ratio test relative to those of some well-known test statistics.

Minimal primes of ideals arising from conditional independence statements

We study ideals whose primary decomposition specifies the relevant structural zeros of certain conditional independence models. The ideals we study generalize the class of ideals considered by Fink (2011) [5] in a way distinct from the generalizations of Herzog, Hibi, Hreinsdottir, Kahle, and Rauh (2010) [10] and Ay and Rauh (2011) [1]. We introduce switchable sets to give a combinatorial description of the minimal prime ideals, and for some classes we describe the minimal components. We discuss possible interpretations of the ideals we study, including as 2×2 minors of generic hypermatrices. We also introduce a definition of diagonal monomial orders on generic hypermatrices to compute some Gröbner bases.

A nonparametric prior for simultaneous covariance estimation

In the modeling of longitudinal data from several groups, appropriate handling of the dependence structure is of central importance. Standard methods include specifying a single covariance matrix for all groups or independently estimating the covariance matrix for each group without regard to the others, but when these model assumptions are incorrect, these techniques can lead to biased mean effects or loss of efficiency, respectively. Thus, it is desirable to develop methods to simultaneously estimate the covariance matrix for each group that will borrow strength across groups in a way that is ultimately informed by the data. In addition, for several groups with covariance matrices of even medium dimension, it is difficult to manually select a single best parametric model among the huge number of possibilities given by incorporating structural zeros and/or commonality of individual parameters across groups. In this paper we develop a family of nonparametric priors using the matrix stick-breaking process of Dunson et al. (2008) that seeks to accomplish this task by parameterizing the covariance matrices in terms of the parameters of their modified Cholesky decomposition (Pourahmadi, 1999). We establish some theoretic properties of these priors, examine their effectiveness via a simulation study, and illustrate the priors using data from a longitudinal clinical trial.

Distribution‐free models for longitudinal count responses with overdispersion and structural zeros

Overdispersion and structural zeros are two major manifestations of departure from the Poisson assumption when modeling count responses using Poisson log-linear regression. As noted in a large body of literature, ignoring such departures could yield bias and lead to wrong conclusions. Different approaches have been developed to tackle these two major problems. In this paper, we review available methods for dealing with overdispersion and structural zeros within a longitudinal data setting and propose a distribution-free modeling approach to address the limitations of these methods by utilizing a new class of functional response models. We illustrate our approach with both simulated and real study data.

Testing homogeneity of variances with unequal sample sizes

When sample sizes are unequal, problems of heteroscedasticity of the variables given by the absolute deviation from the median arise. This paper studies how the best known heteroscedastic alternatives to the ANOVA F test perform when they are applied to these variables. This procedure leads to testing homoscedasticity in a similar manner to Levene’s (1960) test. The difference is that the ANOVA method used by Levene’s test is non-robust against unequal variances of the parent populations and Levene’s variables may be heteroscedastic. The adjustment proposed by O’Neil and Mathews (Aust Nz J Stat 42:81–100, 2000) is approximated by the Keyes and Levy (J Educ Behav Stat 22:227–236, 1997) adjustment and used to ensure the correct null hypothesis of homoscedasticity. Structural zeros, as defined by Hines and O’Hara Hines (Biometrics 56:451–454, 2000), are eliminated. To reduce the error introduced by the approximate distribution of test statistics, estimated critical values are used. Simulation results show that after applying the Keyes–Levy adjustment, including estimated critical values and removing structural zeros the heteroscedastic tests perform better than Levene’s test. In particular, Brown–Forsythe’s test controls the Type I error rate in all situations considered, although it is slightly less powerful than Welch’s, James’s, and Alexander and Govern’s tests, which perform well, except in highly asymmetric distributions where they are moderately liberal.

Pollen-based climate reconstruction: Calibration of the vegetation–pollen processes

Palaeoclimate reconstructions are based on the relationship between climate and sediment pollen assemblages. This model is called the transfer function (TF). Process-based TF emerge as an opportunity to better quantify past climate changes. For example, when a process-based model of vegetation dynamics is part of the TF it allows to include atmospheric CO2 concentration and plant–plant interactions as factors affecting the reconstruction. We propose the missing piece for a fully process-based TF: the model linking, at a continental scale, vegetation model outputs and pollen sampled in sediments. We perform its calibration and we explore the quality of fit.The model represents the error of the vegetation model LPJ-GUESS and four main processes: pollen production, dispersal, accumulation and sampling. Accumulation and sampling processes are either modelled using a multinomial-Poisson (MP) or a multinomial-negative binomial (MNB) model, both models allowing for overdispersion and structural zeros in the sense of null multinomial probabilities. We perform inference for a European pollen dataset by parallelising a Monte Carlo Markov Chain algorithm.Model fitness diagnostics indicate that MP model is not supported by the European dataset. The MNB model is also detected inconsistent, but with a p-value of 0.014 and without stationarity nor overdispersion problems. At this stage, the MNB model is considered as a robust alternative to more complex models. We finally discuss the challenge of the TF inversion for palaeoclimate reconstruction and vegetation model re-calibration.

Dynamic Markov Bases

This article presents a computational approach for generating Markov bases for multiway contingency tables whose cell counts might be constrained by fixed marginals and by lower and upper bounds. Our framework includes tables with structural zeros as a particular case. Instead of computing the entire Markov bases in an initial step, our framework finds sets of local moves that connect each table in the reference set with a set of neighbor tables. We construct a Markov chain on the reference set of tables that requires only a set of local moves at each iteration. The union of these sets of local moves forms a dynamic Markov basis. We illustrate the practicality of our algorithms in the estimation of exact p-values for a three-way table with structural zeros and a sparse eight-way table. This article has online supplementary materials.

Covariance structure approximation via gLasso in high-dimensional supervised classification

Recent work has shown that the Lasso-based regularization is very useful for estimating the high-dimensional inverse covariance matrix. A particularly useful scheme is based on penalizing the ℓ1 norm of the off-diagonal elements to encourage sparsity. We embed this type of regularization into high-dimensional classification. A two-stage estimation procedure is proposed which first recovers structural zeros of the inverse covariance matrix and then enforces block sparsity by moving non-zeros closer to the main diagonal. We show that the block-diagonal approximation of the inverse covariance matrix leads to an additive classifier, and demonstrate that accounting for the structure can yield better performance accuracy. Effect of the block size on classification is explored, and a class of asymptotically equivalent structure approximations in a high-dimensional setting is specified. We suggest a variable selection at the block level and investigate properties of this procedure in growing dimension asymptotics. We present a consistency result on the feature selection procedure, establish asymptotic lower an upper bounds for the fraction of separative blocks and specify constraints under which the reliable classification with block-wise feature selection can be performed. The relevance and benefits of the proposed approach are illustrated on both simulated and real data.

Log‐linear modeling

AbstractThis article describes log‐linear models as special cases of generalized linear models. Specifically, log‐linear models use a logarithmic link function. Log‐linear models are used to examine joint distributions of categorical variables, dependency relations, and association patterns. Three types of log‐linear models are discussed, hierarchical models, nonhierarchical models, and nonstandard models. Emphasis is placed on parameter interpretation. It is demonstrated that parameters are best interpretable when they represent the effects specified in the design matrix of the model. Parameter interpretation is illustrated first for a standard hierarchical model, and then for a nonstandard model that includes structural zeros. In a data example, the relationships among race of defendant, race of victim, and death penalty sentence are examined using a log‐linear model with all three two‐way interactions. Recent developments in log‐linear modeling are discussed. WIREs Comput Stat 2012, 4:218–223. doi: 10.1002/wics.203This article is categorized under: Statistical Models > Generalized Linear Models

Constructing priors based on model size for nondecomposable Gaussian graphical models: A simulation based approach

A method for constructing priors is proposed that allows the off-diagonal elements of the concentration matrix of Gaussian data to be zero. The priors have the property that the marginal prior distribution of the number of nonzero off-diagonal elements of the concentration matrix (referred to below as model size) can be specified flexibly. The priors have normalizing constants for each model size, rather than for each model, giving a tractable number of normalizing constants that need to be estimated. The article shows how to estimate the normalizing constants using Markov chain Monte Carlo simulation and supersedes the method of Wong et al. (2003) [24] because it is more accurate and more general. The method is applied to two examples. The first is a mixture of constrained Wisharts. The second is from Wong et al. (2003) [24] and decomposes the concentration matrix into a function of partial correlations and conditional variances using a mixture distribution on the matrix of partial correlations. The approach detects structural zeros in the concentration matrix and estimates the covariance matrix parsimoniously if the concentration matrix is sparse.

A Bayesian Approach for Zero-Inflated Count Regression Models by Using the Reversible Jump Markov Chain Monte Carlo Method and an Application

In this study, estimation of the parameters of the zero-inflated count regression models and computations of posterior model probabilities of the log-linear models defined for each zero-inflated count regression models are investigated from the Bayesian point of view. In addition, determinations of the most suitable log-linear and regression models are investigated. It is known that zero-inflated count regression models cover zero-inflated Poisson, zero-inflated negative binomial, and zero-inflated generalized Poisson regression models. The classical approach has some problematic points but the Bayesian approach does not have similar flaws. This work points out the reasons for using the Bayesian approach. It also lists advantages and disadvantages of the classical and Bayesian approaches. As an application, a zoological data set, including structural and sampling zeros, is used in the presence of extra zeros. In this work, it is observed that fitting a zero-inflated negative binomial regression model creates no problems at all, even though it is known that fitting a zero-inflated negative binomial regression model is the most problematic procedure in the classical approach. Additionally, it is found that the best fitting model is the log-linear model under the negative binomial regression model, which does not include three-way interactions of factors.

Markov bases and subbases for bounded contingency tables

In this paper we study the computation of Markov bases for contingency tables whose cell entries have an upper bound. It is known that in this case one has to compute universal Gröbner bases, and this is often infeasible also in small- and medium-sized problems. Here we focus on bounded two-way contingency tables under independence model. We show that when these bounds on cells are positive the set of basic moves of all 2 × 2 minors connects all tables with given margins. We also give some results about bounded incomplete table and we conclude with an open problem on the necessary and sufficient condition on the set of structural zeros so that the set of basic moves of all 2 × 2 minors connects all incomplete contingency tables with given positive margins.

Compatibility of discrete conditional distributions with structural zeros

A general algorithm is provided for determining the compatibility among full conditionals of discrete random variables with structural zeros. The algorithm is scalable and it can be implemented in a fairly straightforward manner. A MATLAB program is included in the Appendix and therefore, it is now feasible to check the compatibility of multi-dimensional conditional distributions with constrained supports. Rather than the linear equations in the restricted domain of Arnold et al. (2002) [11] Tian et al. (2009) [16], the approach is odds-oriented and it is a discrete adaptation of the compatibility check of Besag (1994) [17]. The method naturally leads to the calculation of a compatible joint distribution or, in the absence of compatibility, a nearly compatible joint distribution. Besag’s [5] factorization of a joint density in terms of conditional densities is used to justify the algorithm.

Tests for zero-inflation and overdispersion: A new approach based on the stochastic convex order

A new methodology to detect zero-inflation and overdispersion is proposed, based on a comparison of the expected sample extremes among convexly ordered distributions. The method is very flexible and includes tests for the proportion of structural zeros in zero-inflated models, tests to distinguish between two ordered parametric families and a new general test to detect overdispersion. The performance of the proposed tests is evaluated via some simulation studies. For the well-known fetal lamb data, the conclusion is that the zero-inflated Poisson model should be rejected against other more disperse models, but the negative binomial model cannot be rejected.