Excess Zeros Research Articles

Modelling auto insurance claims in Singapore

Claim frequency data in general insurance may not follow the traditional Poisson distribution when there are many zeros. When the number of observed zeros exceeds the number of expected zeros under the Poisson distribution, extra dispersion appears. This paper summarizes several dispersed and zero-inflated count data models, which are used to handle dispersion and excess zeros. We model the insurance claim count data with excess zeros with these models. We use chi-square goodness-of-fit, to test the validity of the assumption of the count data distribution and fit count data regression model with predictors. We compare the fits through AIC and BIC. The generalized Poisson model and Negative binomial model provide a good fit to the data.

Comments on “Marginalized multilevel hurdle and zero‐inflated models for overdispersed and correlated count data with excess zeros”

Comments on “Marginalized multilevel hurdle and zero‐inflated models for overdispersed and correlated count data with excess zeros”

A quasi-score statistic for homogeneity testing against covariate-varying heterogeneity.

In statistical modeling, it is often of interest to evaluate non-negative quantities that capture heterogeneity in the population such as variances, mixing proportions and dispersion parameters. In instances of covariate-dependent heterogeneity, the implied homogeneity hypotheses are nonstandard and existing inferential techniques are not applicable. In this paper, we develop a quasi-score test statistic to evaluate homogeneity against heterogeneity that varies with a covariate profile through a regression model. We establish the limiting null distribution of the proposed test as a functional of mixtures of chi-square processes. The methodology does not require the full distribution of the data to be entirely specified. Instead, a general estimating function for a finite dimensional component of the model that is of interest is assumed but other characteristics of the population are left completely unspecified. We apply the methodology to evaluate the excess zero proportion in zero-inflated models for count data. Our numerical simulations show that the proposed test can greatly improve efficiency over tests of homogeneity that neglect covariate information under the alternative hypothesis. An empirical application to dental caries indices demonstrates the importance and practical utility of the methodology in detecting excess zeros in the data.

A random-effects hurdle model for predicting bycatch of endangered marine species

Understanding and reducing the incidence of accidental bycatch, particularly for vulnerable species such as sharks, is a major challenge for contemporary fisheries management worldwide. Bycatch data, most often collected by at-sea observers during fishing trips, are clustered by trip and/or vessel and typically involve a large number of zero counts and very few positive counts. Though hurdle models are very popular for count data with excess zeros, models for clustered forms have received far less attention. Here we present a novel random-effects hurdle model for bycatch data that makes available accurate estimates of bycatch probabilities as well as other cluster-specific targets. These are essential for informing conservation and management decisions as well as for identifying bycatch hotspots, often considered the first step in attempting to protect endangered marine species. We validate our methodology through simulation and use it to analyze bycatch data on critically endangered hammerhead sharks from the U.S. National Marine Fisheries Service Pelagic Observer Program.

Tweedie distributions for fitting semicontinuous health care utilization cost data

BackgroundThe statistical analysis of health care cost data is often problematic because these data are usually non-negative, right-skewed and have excess zeros for non-users. This prevents the use of linear models based on the Gaussian or Gamma distribution. A common way to counter this is the use of Two-part or Tobit models, which makes interpretation of the results more difficult. In this study, I explore a statistical distribution from the Tweedie family of distributions that can simultaneously model the probability of zero outcome, i.e. of being a non-user of health care utilization and continuous costs for users.MethodsI assess the usefulness of the Tweedie model in a Monte Carlo simulation study that addresses two common situations of low and high correlation of the users and the non-users of health care utilization. Furthermore, I compare the Tweedie model with several other models using a real data set from the RAND health insurance experiment.ResultsI show that the Tweedie distribution fits cost data very well and provides better fit, especially when the number of non-users is low and the correlation between users and non-users is high.ConclusionThe Tweedie distribution provides an interesting solution to many statistical problems in health economic analyses.

Classifying next-generation sequencing data using a zero-inflated Poisson model.

With the development of high-throughput techniques, RNA-sequencing (RNA-seq) is becoming increasingly popular as an alternative for gene expression analysis, such as RNAs profiling and classification. Identifying which type of diseases a new patient belongs to with RNA-seq data has been recognized as a vital problem in medical research. As RNA-seq data are discrete, statistical methods developed for classifying microarray data cannot be readily applied for RNA-seq data classification. Witten proposed a Poisson linear discriminant analysis (PLDA) to classify the RNA-seq data in 2011. Note, however, that the count datasets are frequently characterized by excess zeros in real RNA-seq or microRNA sequence data (i.e. when the sequence depth is not enough or small RNAs with the length of 18-30 nucleotides). Therefore, it is desired to develop a new model to analyze RNA-seq data with an excess of zeros. In this paper, we propose a Zero-Inflated Poisson Logistic Discriminant Analysis (ZIPLDA) for RNA-seq data with an excess of zeros. The new method assumes that the data are from a mixture of two distributions: one is a point mass at zero, and the other follows a Poisson distribution. We then consider a logistic relation between the probability of observing zeros and the mean of the genes and the sequencing depth in the model. Simulation studies show that the proposed method performs better than, or at least as well as, the existing methods in a wide range of settings. Two real datasets including a breast cancer RNA-seq dataset and a microRNA-seq dataset are also analyzed, and they coincide with the simulation results that our proposed method outperforms the existing competitors. The software is available at http://www.math.hkbu.edu.hk/∼tongt. xwan@comp.hkbu.edu.hk or tongt@hkbu.edu.hk. Supplementary data are available at Bioinformatics online.

Analysis of Microbiome Data in the Presence of Excess Zeros.

Motivation: An important feature of microbiome count data is the presence of a large number of zeros. A common strategy to handle these excess zeros is to add a small number called pseudo-count (e.g., 1). Other strategies include using various probability models to model the excess zero counts. Although adding a pseudo-count is simple and widely used, as demonstrated in this paper, it is not ideal. On the other hand, methods that model excess zeros using a probability model often make an implicit assumption that all zeros can be explained by a common probability models. As described in this article, this is not always recommended as there are potentially three types/sources of zeros in a microbiome data. The purpose of this paper is to develop a simple methodology to identify and accomodate three different types of zeros and to test hypotheses regarding the relative abundance of taxa in two or more experimental groups. Another major contribution of this paper is to perform constrained (directional or ordered) inference when there are more than two ordered experimental groups (e.g., subjects ordered by diet or age groups or environmental exposure groups). As far as we know this is the first paper that addresses such problems in the analysis of microbiome data.Results: Using extensive simulation studies, we demonstrate that the proposed methodology not only controls the false discovery rate at a desired level of significance while competing well in terms of power with DESeq2, a popular procedure derived from RNASeq literature. As expected, the method using pseudo-counts tends to be very conservative and the classical t-test that ignores the underlying simplex structure in the data has an inflated FDR.

Network analysis for count data with excess zeros

BackgroundUndirected graphical models or Markov random fields have been a popular class of models for representing conditional dependence relationships between nodes. In particular, Markov networks help us to understand complex interactions between genes in biological processes of a cell. Local Poisson models seem to be promising in modeling positive as well as negative dependencies for count data. Furthermore, when zero counts are more frequent than are expected, excess zeros should be considered in the model.MethodsWe present a penalized Poisson graphical model for zero inflated count data and derive an expectation-maximization (EM) algorithm built on coordinate descent. Our method is shown to be effective through simulated and real data analysis.ResultsResults from the simulated data indicate that our method outperforms the local Poisson graphical model in the presence of excess zeros. In an application to a RNA sequencing data, we also investigate the gender effect by comparing the estimated networks according to different genders. Our method may help us in identifying biological pathways linked to sex hormone regulation and thus understanding underlying mechanisms of the gender differences.ConclusionsWe have presented a penalized version of zero inflated spatial Poisson regression and derive an efficient EM algorithm built on coordinate descent. We discuss possible improvements of our method as well as potential research directions associated with our findings from the RNA sequencing data.

A Bayesian approach for analyzing zero-inflated clustered count data with dispersion.

In practice, count data may exhibit varying dispersion patterns and excessive zero values; additionally, they may appear in groups or clusters sharing a common source of variation. We present a novel Bayesian approach for analyzing such data. To model these features, we combine the Conway-Maxwell-Poisson distribution, which allows both overdispersion and underdispersion, with a hurdle component for the zeros and random effects for clustering. We propose an efficient Markov chain Monte Carlo sampling scheme to obtain posterior inference from our model. Through simulation studies, we compare our hurdle Conway-Maxwell-Poisson model with a hurdle Poisson model to demonstrate the effectiveness of our Conway-Maxwell-Poisson approach. Furthermore, we apply our model to analyze an illustrative dataset containing information on the number and types of carious lesions on each tooth in a population of 9-year-olds from the Iowa Fluoride Study, which is an ongoing longitudinal study on a cohort of Iowa children that began in 1991.

Modeling the Infant’s Age at Hospital Admission in Neonatal with Jaundice at the Ghaem Mashhad Hospital Using Count Models with Excess Zeros

Modeling the Infant’s Age at Hospital Admission in Neonatal with Jaundice at the Ghaem Mashhad Hospital Using Count Models with Excess Zeros

Three-part joint modeling methods for complex functional data mixed with zero-and-one-inflated proportions and zero-inflated continuous outcomes with skewness.

We take a functional data approach to longitudinal studies with complex bivariate outcomes. This work is motivated by data from a physical activity study that measured 2 responses over time in 5-minute intervals. One response is the proportion of time active in each interval, a continuous proportions with excess zeros and ones. The other response, energy expenditure rate in the interval, is a continuous variable with excess zeros and skewness. This outcome is complex because there are 3 possible activity patterns in each interval (inactive, partially active, and completely active), and those patterns, which are observed, induce both nonrandom and random associations between the responses. More specifically, the inactive pattern requires a zero value in both the proportion for active behavior and the energy expenditure rate; a partially active pattern means that the proportion of activity is strictly between zero and one and that the energy expenditure rate is greater than zero and likely to be moderate, and the completely active pattern means that the proportion of activity is exactly one, and the energy expenditure rate is greater than zero and likely to be higher. To address these challenges, we propose a 3-part functional data joint modeling approach. The first part is a continuation-ratio model to reorder the ordinal valued 3 activity patterns. The second part models the proportions when they are in interval (0,1). The last component specifies the skewed continuous energy expenditure rate with Box-Cox transformations when they are greater than zero. In this 3-part model, the regression structures are specified as smooth curves measured at various time points with random effects that have a correlation structure. The smoothed random curves for each variable are summarized using a few important principal components, and the association of the 3 longitudinal components is modeled through the association of the principal component scores. The difficulties in handling the ordinal and proportional variables are addressed using a quasi-likelihood type approximation. We develop an efficient algorithm to fit the model that also involves the selection of the number of principal components. The method is applied to physical activity data and is evaluated empirically by a simulation study.

Time series analysis based on two-part models for excessive zero count data to detect farm-level outbreaks of swine echinococcosis during meat inspections

Echinococcus multilocularis is a parasite that causes highly pathogenic zoonoses and is maintained in foxes and rodents on Hokkaido Island, Japan. Detection of E. multilocularis infections in swine is epidemiologically important. In Hokkaido, administrative information is provided to swine producers based on the results of meat inspections. However, as the current criteria for providing administrative information often results in delays in providing information to producers, novel criteria are needed. Time series models were developed to monitor autocorrelations between data and lags using data collected from 84 producers at the Higashi-Mokoto Meat Inspection Center between April 2003 and November 2015. The two criteria were quantitatively compared using the sign test for the ability to rapidly detect farm-level outbreaks. Overall, the time series models based on an autoexponentially regressed zero-inflated negative binomial distribution with 60th percentile cumulative distribution function of the model detected outbreaks earlier more frequently than the current criteria (90.5%, 276/305, p<0.001). Our results show that a two-part model with autoexponential regression can adequately deal with data involving an excessive number of zeros and that the novel criteria overcome disadvantages of the current criteria to provide an earlier indication of increases in the rate of echinococcosis.

Zero-inflated ordered probit approach to modeling mushroom consumption in the United States

This paper investigates the determinants of fresh and processed mushroom consumption in the United States by employing the zero-inflated ordered probit (ZIOP) model. The ZIOP model accounts for excessive zero observations and allows us to differentiate between genuine non-consumers and individuals who did not consume during the given period but might under different circumstances. The results indicate that the market for fresh mushrooms is larger than that for processed mushrooms. However, the market for processed mushrooms has a larger portion of potential consumers which might indicate more potential if appropriate marketing strategies are applied. The results also suggest that the decisions to participate in the market or not and the consumption frequency are driven by structurally different factors. A comparison of the ZIOP to other models is included to show the advantages of allowing for non-consumers and potential consumers to be analyzed separately.

Zero-inflated multiscale models for aggregated small area health data.

It is our primary focus to study the spatial distribution of disease incidence at different geographical levels. Often, spatial data are available in the form of aggregation at multiple scale levels such as census tract, county, state, and so on. When data are aggregated from a fine (e.g. county) to a coarse (e.g. state) geographical level, there will be loss of information. The problem is more challenging when excessive zeros are available at the fine level. After data aggregation, the excessive zeros at the fine level will be reduced at the coarse level. If we ignore the zero inflation and the aggregation effect, we could get inconsistent risk estimates at the fine and coarse levels. Hence, in this paper, we address those problems using zero inflated multiscale models that jointly describe the risk variations at different geographical levels. For the excessive zeros at the fine level, we use a zero inflated convolution model, whereas we consider a regular convolution model for the smoothed data at the coarse level. These methods provide a consistent risk estimate at the fine and coarse levels when high percentages of structural zeros are present in the data.

Score tests for zero-inflated double poisson regression models

Count data with excess zeros encountered in many applications often exhibit extra variation. Therefore, zero-inflated Poisson (ZIP) model may fail to fit such data. In this paper, a zero-inflated double Poisson model (ZIDP), which is generalization of the ZIP model, is studied and the score tests for the significance of dispersion and zero-inflation in ZIDP model are developed. Meanwhile, this work also develops homogeneous tests for dispersion and/or zero-inflation parameter, and corresponding score test statistics are obtained. One numerical example is given to illustrate our methodology and the properties of score test statistics are investigated through Monte Carlo simulations.

Analysis of multinomial counts with joint zero-inflation, with an application to health economics

Zero-inflated regression models for count data are often used in health economics to analyze demand for medical care. Indeed, excess of zeros often affects health-care utilization data. Much of the recent econometric literature on the topic has focused on univariate health-care utilization measures, such as the number of doctor visits. However, health service utilization is usually measured by a number of different counts (e.g., numbers of visits to different health-care providers). In this case, zero-inflation may jointly affect several of the utilization measures. In this paper, a zero-inflated regression model for multinomial counts with joint zero-inflation is proposed. Maximum likelihood estimators in this model are constructed and their properties are investigated, both theoretically and numerically. We apply the proposed model to an analysis of health-care utilization.

Empirical null estimation using zero-inflated discrete mixture distributions and its application to protein domain data.

In recent mutation studies, analyses based on protein domain positions are gaining popularity over gene-centric approaches since the latter have limitations in considering the functional context that the position of the mutation provides. This presents a large-scale simultaneous inference problem, with hundreds of hypothesis tests to consider at the same time. This article aims to select significant mutation counts while controlling a given level of Type I error via False Discovery Rate (FDR) procedures. One main assumption is that the mutation counts follow a zero-inflated model in order to account for the true zeros in the count model and the excess zeros. The class of models considered is the Zero-inflated Generalized Poisson (ZIGP) distribution. Furthermore, we assumed that there exists a cut-off value such that smaller counts than this value are generated from the null distribution. We present several data-dependent methods to determine the cut-off value. We also consider a two-stage procedure based on screening process so that the number of mutations exceeding a certain value should be considered as significant mutations. Simulated and protein domain data sets are used to illustrate this procedure in estimation of the empirical null using a mixture of discrete distributions. Overall, while maintaining control of the FDR, the proposed two-stage testing procedure has superior empirical power.

Extended Poisson–Tweedie: Properties and regression models for count data

We propose a new class of discrete generalized linear models based on the class of Poisson–Tweedie factorial dispersion models with variance of the form [Formula: see text], where [Formula: see text] is the mean and [Formula: see text] and [Formula: see text] are the dispersion and Tweedie power parameters, respectively. The models are fitted by using an estimating function approach obtained by combining the quasi-score and Pearson estimating functions for the estimation of the regression and dispersion parameters, respectively. This provides a flexible and efficient regression methodology for a comprehensive family of count models including Hermite, Neyman Type A, Pólya–Aeppli, negative binomial and Poisson-inverse Gaussian. The estimating function approach allows us to extend the Poisson–Tweedie distributions to deal with underdispersed count data by allowing negative values for the dispersion parameter [Formula: see text]. Furthermore, the Poisson–Tweedie family can automatically adapt to highly skewed count data with excessive zeros, without the need to introduce zero-inflated or hurdle components, by the simple estimation of the power parameter. Thus, the proposed models offer a unified framework to deal with under-, equi-, overdispersed, zero-inflated and heavy-tailed count data. The computational implementation of the proposed models is fast, relying only on a simple Newton scoring algorithm. Simulation studies showed that the estimating function approach provides unbiased and consistent estimators for both regression and dispersion parameters. We highlight the ability of the Poisson–Tweedie distributions to deal with count data through a consideration of dispersion, zero-inflated and heavy tail indices, and illustrate its application with four data analyses. We provide an R implementation and the datasets as supplementary materials.

Correcting for Non-Sum to 1 Estimated Probabilities in Applications of Discrete Probability Models to Count Data

In this paper, we examine some often ignored or assumed problems relating with fitting probability models to count data either exhibiting over, equi, or under dispersion. Of particular concern are last category truncated data, where most often, expected values in this last category are collapsed together so that the sum of the expected values sum to the sample size in the data. That is, so that $\displaystyle \sum_{i=0}^{k} \hat{m}_i=n$, the sample size. We shall for illustrative purposes in this paper, consider the following distributions: the negative binomial (NB), the Inverse trinomial (IT), the hyper-Poisson (HP), the Quasi-negative binomial (QNBD), the extended com-Poisson distribution (ECOMP) as well as the negative binomial-exponential distribution (NBGE).Though, we have restricted our discussion to these six distributions, other distributions may also be employed but the patterns are always the same, that is, the sum of the estimated probabilities does not equal 1.00 and consequently, the sum of the expected values is always less or equal (Poisson case only) the sample size in the observed data. We propose a common procedure to rectify this problem for both right truncated or non-truncated frequency count data exhibiting either excess zeros or regular frequency data.

Local institutions in Peru and the presence and number of fast-food outlets

PurposeThe paper aims to study the role of local institutions in the establishment of fast-food outlets in urban districts of Peru. In most urban districts, there are no fast-food outlets. The authors, therefore, study the effect of institutional quality on the presence or absence of these outlets and the number of outlets if these are present.Design/methodology/approachThe theoretical framework in which this paper is based on is the theory of agglomeration, which establishes that firms benefit from being close to each other. In particular, the paper builds on a model of market entry and competition in geographically independent local markets. An explicit expression was found for the equilibrium number of outlets (including zero) as a function of exogenous determinants of the demand for fast-food in each market, available infrastructure and institutional quality of the district’s government. Principal component analysis was used to construct measures of institutional quality based on administrative and organizational characteristics of district’s municipalities. These measures were incorporated as explanatory variables in a zero-inflated Poisson model, which is appropriate to handle count data and to accommodate excess zeros and which also allows the specification of different models for the zero part and the positive part.FindingsInstitutional quality mainly affects the presence of fast-food outlets in a district. The quality of urban development management and use of information systems are relevant. An institutional variable particularly relevant in explaining the number of outlets is the presence of an investment programming office in the municipality. The authors confirm the general hypothesis of the paper: institutions have a role in explaining both the presence and number of fast-food outlets in a district. Overall, the results of this paper suggest that institutional quality of a municipal district is related to better infrastructure, which lowers the costs of establishing outlets.Research limitations/implicationsLimitations in the availability of data at the regional and urban district level did not allow the authors to analyze other factors that affect entry decisions in the fast-food industry in Peru, such as controls to prevent corruption, legal uncertainty or crime. Another limitation was the lack of data on entry costs for each franchisee in each urban district. This forced the authors to use public infrastructure characteristics of the district as (imperfect) proxies of the entry costs.Practical implicationsThe instruments of urban development management and information systems can be effective at attracting investment to a district. These tools operate partly through an indirect effect, namely, the improvement of district infrastructure, which is necessary to reduce the costs of establishing companies. There is also synergy between national government’s programs to attract investment and the good institutional quality in local governments. On the contrary, poor local institutions can be an obstacle to the successful implementation of those national programs.Social implicationsForeign direct investment has a positive impact on the economic development of a country through knowledge spillovers. Therefore, any administrative reform to make local government practices more efficient can have an indirect impact on development.Originality/valuePrincipal component analysis is a statistical tool that can be important in building good measures of institutional quality by allowing the combination of different observable characteristics into one component that can be interpreted as an operational restriction. The count model allows the use of the primary, easily observable, dependent variable, namely, the number of outlets. Finally, the two-part model makes it possible to discern the effect of institutional quality on the presence or absence of outlets and the number of outlets if these are present.