Overdispersed Count Data Research Articles

On the Discretization of Continuous Probability Distributions Using a Probabilistic Rounding Mechanism

Most existing flexible count distributions allow only approximate inference when used in a regression context. This work proposes a new framework to provide an exact and flexible alternative for modeling and simulating count data with various types of dispersion (equi-, under-, and over-dispersion). The new method, referred to as “balanced discretization”, consists of discretizing continuous probability distributions while preserving expectations. It is easy to generate pseudo random variates from the resulting balanced discrete distribution since it has a simple stochastic representation (probabilistic rounding) in terms of the continuous distribution. For illustrative purposes, we develop the family of balanced discrete gamma distributions that can model equi-, under-, and over-dispersed count data. This family of count distributions is appropriate for building flexible count regression models because the expectation of the distribution has a simple expression in terms of the parameters of the distribution. Using the Jensen–Shannon divergence measure, we show that under the equidispersion restriction, the family of balanced discrete gamma distributions is similar to the Poisson distribution. Based on this, we conjecture that while covering all types of dispersions, a count regression model based on the balanced discrete gamma distribution will allow recovering a near Poisson distribution model fit when the data are Poisson distributed.

Exponentiated Weibull-geometric distribution and its application to count data

An exponentiated Weibull-geometric distribution is defined and studied. A new count data regression model, based on the exponentiated Weibull-geometric distribution, is also defined. The regression model can be applied to fit an underdispersed or an over-dispersed count data. The exponentiated Weibull-geometric regression model is fitted to two numerical data sets. The new model provided a better fit than the fit from its competitors.

Flexible models for overdispersed and underdispersed count data

Within the framework of probability models for overdispersed count data, we propose the generalized fractional Poisson distribution (gfPd), which is a natural generalization of the fractional Poisson distribution (fPd), and the standard Poisson distribution. We derive some properties of gfPd and more specifically we study moments, limiting behavior and other features of fPd. The skewness suggests that fPd can be left-skewed, right-skewed or symmetric; this makes the model flexible and appealing in practice. We apply the model to real big count data and estimate the model parameters using maximum likelihood. Then, we turn to the very general class of weighted Poisson distributions (WPD’s) to allow both overdispersion and underdispersion. Similarly to Kemp’s generalized hypergeometric probability distribution, which is based on hypergeometric functions, we analyze a class of WPD’s related to a generalization of Mittag–Leffler functions. The proposed class of distributions includes the well-known COM-Poisson and the hyper-Poisson models. We characterize conditions on the parameters allowing for overdispersion and underdispersion, and analyze two special cases of interest which have not yet appeared in the literature.

Inference Procedures on the Generalized Poisson Distribution from Multiple Samples: Comparisons with Nonparametric Models for Analysis of Covariance (ANCOVA) of Count Data

Count data that exhibit over dispersion (variance of counts is larger than its mean) are commonly analyzed using discrete distributions such as negative binomial, Poisson inverse Gaussian and other models. The Poisson is characterized by the equality of mean and variance whereas the Negative Binomial and the Poisson inverse Gaussian have variance larger than the mean and therefore are more appropriate to model over-dispersed count data. As an alternative to these two models, we shall use the generalized Poisson distribution for group comparisons in the presence of multiple covariates. This problem is known as the ANCOVA and is solved for continuous data. Our objectives were to develop ANCOVA using the generalized Poisson distribution, and compare its goodness of fit to that of the nonparametric Generalized Additive Models. We used real life data to show that the model performs quite satisfactorily when compared to the nonparametric Generalized Additive Models.

Blinded Sample Size Re-Estimation in Comparative Clinical Trials With Overdispersed Count Data: Incorporation of Misspecification of the Variance Function

In randomized clinical trials to compare overdispersed count data between treatment groups, blinded sample size re-estimation (BSSR) is an effective approach to ensure power control even under possible misspecifications of the variance function on overdispersion specified at the design stage. Based on interim clinical trial data, the existing BSSR methods try to find a more appropriate value of the dispersion parameter for a “fixed” variance function specified at the design stage. In this article, we develop a new BSSR method that allows for modification of the variance function itself. Simulation studies demonstrated a stable power control of the proposed method. An application to a phase III clinical trial in chronic obstructive pulmonary disease also demonstrated effectiveness of the proposed method.

NBZIMM: negative binomial and zero-inflated mixed models, with application to microbiome/metagenomics data analysis

BackgroundMicrobiome/metagenomic data have specific characteristics, including varying total sequence reads, over-dispersion, and zero-inflation, which require tailored analytic tools. Many microbiome/metagenomic studies follow a longitudinal design to collect samples, which further complicates the analysis methods needed. A flexible and efficient R package is needed for analyzing processed multilevel or longitudinal microbiome/metagenomic data.ResultsNBZIMM is a freely available R package that provides functions for setting up and fitting negative binomial mixed models, zero-inflated negative binomial mixed models, and zero-inflated Gaussian mixed models. It also provides functions to summarize the results from fitted models, both numerically and graphically. The main functions are built on top of the commonly used R packages nlme and MASS, allowing us to incorporate the well-developed analytic procedures into the framework for analyzing over-dispersed and zero-inflated count or proportion data with multilevel structures (e.g., longitudinal studies). The statistical methods and their implementations in NBZIMM particularly address the data characteristics and the complex designs in microbiome/metagenomic studies. The package is freely available from the public GitHub repository https://github.com/nyiuab/NBZIMM.ConclusionThe NBZIMM package provides useful tools for complex microbiome/metagenomics data analysis.

A two-part modelling approach reveals a positive effect of pollinator biodiversity in boosting the pollination of apple flowers

Pollinator biodiversity may benefit crop pollination. Yet benefits in agro-ecosystems may be context-dependent and offset by agronomic or other limiting orchard-specific or tree-specific factors that obscure biodiversity-ecosystem service relationships. To test if crop pollination benefitted from pollinator biodiversity, we sampled local wild bee communities in five organic and five Integrated Pest Management (IPM) apple orchards in Germany and experimentally measured the pollination success of apple flowers, quantified as the number of pollen tubes reaching the base of styles. Using standard statistical modelling approaches for over-dispersed count data, we found little or no effect of wild bee biodiversity on pollination success, irrespectively of farm management (organic versus IPM). There was, however, a positive relationship between the number of pollen tubes in insect-pollinated and pollen-supplemented flowers across trees and orchards, suggesting confounding effects of local tree- or orchard-specific factors limiting pollination success. Using a statistical two-part hurdle model which allowed us to separate (i) the probability of pollination and (ii) the quantity of pollination, we were able to demonstrate that the collective effects of local tree/orchard factors acted as a primary limiting threshold for pollination success. Once the threshold was crossed, the hurdle model demonstrated that increasing wild bee abundance enhanced pollination success of apple trees. We advocate the use of statistical two-part models as a more powerful approach to identifying limiting factors of, and the role of biodiversity in, crop pollination and potentially other ecosystem services in agro-ecosystems.

Modeling socio-demographic and clinical factors influencing psychiatric inpatient service use: a comparison of models for zero-Inflated and overdispersed count data

BackgroundPsychiatric disorders may occur as a single episode or be persistent and relapsing, sometimes leading to suicidal behaviours. The exact causes of psychiatric disorders are hard to determine but easy access to health care services can help to reduce their severity. The aim of this study was to investigate the factors associated with repeated hospitalizations among the patients with psychiatric illness, which may help the policy makers to target the high-risk groups in a more focused manner.MethodsA large linked administrative database consisting of 200,537 patients with psychiatric diagnosis in the years of 2008-2012 was used in this analysis. Various counts regression models including zero-inflated and hurdle models were considered for analyzing the hospitalization rate among patients with psychiatric disorders within three months follow-up since their index visit dates. The covariates for this study consisted of socio-demographic and clinical characteristics of the patients.ResultsThe results show that the odds of hospitalization are significantly higher among registered Indians, male patients and younger patients. Hospitalization rate depends on the patients’ disease types. Having previously visited a general physician served a protective role for psychiatric hospitalization during the study period. Patients who had seen an outpatient psychiatrist were more likely to have a higher number of psychiatric hospitalizations. This may indicate that psychiatrists tend to see patients with more severe illnesses, who require hospital-based care for managing their illness.ConclusionsProviding easier access to registered Indian people and youth may reduce the need for hospital-based care. Patients with mental health conditions may benefit from greater and more timely access to primary care.

Poisson Weighted Ishita Distribution: Model for Analysis of Over-Dispersed Medical Count Data

Abstract A new over-dispersed discrete probability model is introduced, by compounding the Poisson distribution with the weighted Ishita distribution. The statistical properties of the newly introduced distribution have been derived and discussed. Parameter estimation has been done with the application of the maximum likelihood method of estimation, followed by the Monte Carlo simulation procedure to examine the suitability of the ML estimators. In order to verify the applicability of the proposed distribution, a real-life set of data from the medical field has been analysed for modeling a count dataset representing epileptic seizure counts.

Bayesian gamma-negative binomial modeling of single-cell RNA sequencing data

BackgroundSingle-cell RNA sequencing (scRNA-seq) is a powerful profiling technique at the single-cell resolution. Appropriate analysis of scRNA-seq data can characterize molecular heterogeneity and shed light into the underlying cellular process to better understand development and disease mechanisms. The unique analytic challenge is to appropriately model highly over-dispersed scRNA-seq count data with prevalent dropouts (zero counts), making zero-inflated dimensionality reduction techniques popular for scRNA-seq data analyses. Employing zero-inflated distributions, however, may place extra emphasis on zero counts, leading to potential bias when identifying the latent structure of the data.ResultsIn this paper, we propose a fully generative hierarchical gamma-negative binomial (hGNB) model of scRNA-seq data, obviating the need for explicitly modeling zero inflation. At the same time, hGNB can naturally account for covariate effects at both the gene and cell levels to identify complex latent representations of scRNA-seq data, without the need for commonly adopted pre-processing steps such as normalization. Efficient Bayesian model inference is derived by exploiting conditional conjugacy via novel data augmentation techniques.ConclusionExperimental results on both simulated data and several real-world scRNA-seq datasets suggest that hGNB is a powerful tool for cell cluster discovery as well as cell lineage inference.

Blinded continuous information monitoring of recurrent event endpoints with time trends in clinical trials.

Blinded sample size re-estimation and information monitoring based on blinded data has been suggested to mitigate risks due to planning uncertainties regarding nuisance parameters. Motivated by a randomized controlled trial in pediatric multiple sclerosis (MS), a continuous monitoring procedure for overdispersed count data was proposed recently. However, this procedure assumed constant event rates, an assumption often not met in practice. Here we extend the procedure to accommodate time trends in the event rates considering two blinded approaches: (a) the mixture approach modeling the number of events by a mixture of two negative binomial distributions and (b) the lumping approach approximating the marginal distribution of the event counts by a negative binomial distribution. Through simulations the operating characteristics of the proposed procedures are investigated under decreasing event rates. We find that the type I error rate is not inflated relevantly by either of the monitoring procedures, with the exception of strong time dependencies where the procedure assuming constant rates exhibits some inflation. Furthermore, the procedure accommodating time trends has generally favorable power properties compared with the procedure based on constant rates which stops often too late. The proposed method is illustrated by the clinical trial in pediatric MS.

A new one-parameter discrete distribution with associated regression and integer-valued autoregressive models

Abstract This study introduces the Poisson-Bilal distribution and its associated two models for modeling the over-dispersed count data sets. The Poisson-Bilal distribution has tractable properties and explicit forms for its statistical properties. A new over-dispersed count regression model and integer-valued autoregressive process with flexible innovation distribution are defined and studied comprehensively. Two real data sets are analyzed to prove empirically the importance of proposed models. Empirical findings show that the Poisson-Bilal distribution has important application fields in time series and regression modeling.

Inference Procedures on the Ratio of Modified Generalized Poisson Distribution Means: Applications to RNA_SEQ Data

The Poisson and the Negative Binomial distributions are commonly used as analytic tools to model count data. The Poisson is characterized by the equality of mean and variance whereas the Negative Binomial has a variance larger than the mean and therefore is appropriate to model over-dispersed count data. The Generalized Poisson Distribution is becoming a popular alternative to the Negative Binomial. We have considered inference procedures on a modified form of this distribution when two samples are available from two independent populations and the target effect size of interest is the ratio of the two population means. The statistical objective is to construct confidence limits on the ratio. We first test the presence of over dispersion and derive several estimators in the single sample situation. When two samples are available, our interest is focused on the estimation of an effect size measured by the ratio of the respective population means. We have compared two methods; namely the Fieller’s and the delta methods in terms of coverage probabilities. We have illustrated the methodologies on published genomic datasets.

A New Discrete Analog of the Continuous Lindley Distribution, with Reliability Applications.

In this paper, we propose and study a new probability mass function by creating a natural discrete analog to the continuous Lindley distribution as a mixture of geometric and negative binomial distributions. The new distribution has many interesting properties that make it superior to many other discrete distributions, particularly in analyzing over-dispersed count data. Several statistical properties of the introduced distribution have been established including moments and moment generating function, residual moments, characterization, entropy, estimation of the parameter by the maximum likelihood method. A bias reduction method is applied to the derived estimator; its existence and uniqueness are discussed. Applications of the goodness of fit of the proposed distribution have been examined and compared with other discrete distributions using three real data sets from biological sciences.

A Tree-Based Semi-Varying Coefficient Model for the COM-Poisson Distribution

We propose a tree-based semi-varying coefficient model for the Conway–Maxwell–Poisson (CMP or COM-Poisson) distribution which is a two-parameter generalization of the Poisson distribution and is flexible enough to capture both under-dispersion and over-dispersion in count data. The advantage of tree-based methods is their scalability to high-dimensional data. We develop CMPMOB, an estimation procedure for a semi-varying coefficient model, using model-based recursive partitioning (MOB). The proposed framework is broader than the existing MOB framework as it allows node-invariant effects to be included in the model. To simplify the computational burden of the exhaustive search employed in the original MOB algorithm, a new split point estimation procedure is proposed by borrowing tools from change point estimation methodology. The proposed method uses only the estimated score functions without fitting models for each split point and, therefore, is computationally simpler. Since the tree-based methods only provide a piece-wise constant approximation to the underlying smooth function, we further propose the CMPBoost semi-varying coefficient model which uses the gradient boosting procedure for estimation. The usefulness of the proposed methods are illustrated using simulation studies and a real example from a bike sharing system in Washington, DC. Supplementary files for this article are available online.

Time Series Regression for Zero-Inflated and Overdispersed Count Data: A Functional Response Model Approach

Count time series data feature prominently in epidemiology, business, and environmental sciences. Often, such data exhibit zero-inflation and overdispersion in addition to serial dependence. Parametric models such as the negative binomial distribution are employed to account for overdispersion. In practice, the conditional variance structure may be unknown or may not be negative binomial. In this paper, a distribution-free approach for estimation of regression parameters of conditionally overdispersed and zero-inflated time series models is developed. Parameter estimates are optimal in the Godambe-information sense. Simulation studies indicate that our method is robust to model misspecification with small relative bias and nearly the same efficiency as that of the MLE for some observation-driven count time series processes. A case study comparing our method with fully parametric methods using weekly syphilis counts from 2007–2010 in Virginia, USA, illustrates the benefit of our method.

Efficient monitoring of overdispersed counts with time-varying population sizes

The monitoring of count data is a major issue in many industrial applications and research activities. Count data often exhibit overdispersion with variance greater than mean, which makes the commonly used Poisson distribution problematic because of its underlying assumption of equal mean and variance. In this article, we consider the surveillance strategy for overdispersed count data, and we also take into account the effect of time-varying population sizes. In particular, to model the data we propose to adapt the generalized Poisson distribution to incorporate the incidence rate, the overdispersion factor, and the non-constant population sizes. The weighted likelihood ratio test is employed for online monitoring. Simulations show that the proposed method is efficient at detecting changes simultaneously in both the incidence rate and the overdispersion factor, and it is robust under different time-varying patterns of the population sizes.

An empirical comparison of methods for analyzing over-dispersed zero-inflated count data from stratified cluster randomized trials

BackgroundThe assessment of methods for analyzing over-dispersed zero inflated count outcome has received very little or no attention in stratified cluster randomized trials. In this study, we performed sensitivity analyses to empirically compare eight methods for analyzing zero inflated over-dispersed count outcome from the Vitamin D and Osteoporosis Study (ViDOS) – originally designed to assess the feasibility of a knowledge translation intervention in long-term care home setting. MethodForty long-term care (LTC) homes were stratified and then randomized into knowledge translation (KT) intervention (19 homes) and control (21 homes) groups. The homes/clusters were stratified by home size (<250/> = 250) and profit status (profit/non-profit). The outcome of this study was number of falls measured at 6-month post-intervention. The following methods were used to assess the effect of KT intervention on number of falls: i) standard Poisson and negative binomial regression; ii) mixed-effects method with Poisson and negative binomial distribution; iii) generalized estimating equation (GEE) with Poisson and negative binomial; iv) zero inflated Poisson and negative binomial — with the latter used as a primary approach. All these methods were compared with or without adjusting for stratification. ResultsA total of 5,478 older people from 40 LTC homes were included in this study. The mean (=1) of the number of falls was smaller than the variance (=6). Also 72% and 46% of the number of falls were zero in the control and intervention groups, respectively. The direction of the estimated incidence rate ratios (IRRs) was similar for all methods. The zero inflated negative binomial yielded the lowest IRRs and narrowest 95% confidence intervals when adjusted for stratification compared to GEE and mixed-effect methods. Further, the widths of the 95% confidence intervals were narrower when the methods adjusted for stratification compared to the same method not adjusted for stratification. ConclusionThe overall conclusion from the GEE, mixed-effect and zero inflated methods were similar. However, these methods differ in terms of effect estimate and widths of the confidence interval. Trial registrationClinicalTrials.gov: NCT01398527. Registered: 19 July 2011.

Discrete Weibull and Artificial Neural Network Models in Modelling Over-dispersed Count Data

In modelling count data, the use of least square regression models suffers several methodological limitations and statistical properties in instances of discrete, non-negative integer count of a dependent variable. Unlike the classical regression model, count data models are non-linear with many properties of the response variable relating to discreteness, non-linearity and deal with non-negative values only. A good starting point for modelling count data is the Poisson regression model since it lends itself well with the nature properties of count data. However, the limitation of equi-dispersion renders it inappropriate for modelling over-dispersed data. Negative Binomial regression model has been widely used and considered as the default regression model for over-dispersed count data. This model is a modification of Poisson regression model and though widely used, it might not be the best model for over-dispersion and other models have been found to perform better. Over-dispersion in this study was defined relative to the Poisson model. This study models over-dispersed count data using discrete Weibull regression model and artificial neural network model with a median neuron in the hidden layer. After fitting the two models on simulated data and real data, the artificial neural network model outperformed the discrete Weibull regression model. Application on data set from German health survey gave RMSE of DW regression model as 69.0668 and 35.5652 for the artificial neural network.

Negative Binomial Matrix Factorization

We introduce negative binomial matrix factorization (NBMF), a matrix factorization technique specially designed for analyzing over-dispersed count data. It can be viewed as an extension of Poisson factorization (PF) perturbed by a multiplicative term which models exposure. This term brings a degree of freedom for controlling the dispersion, making NBMF more robust to outliers. We describe a majorization-minimization (MM) algorithm for a maximum likelihood estimation of the parameters. We provide results on a recommendation task and demonstrate the ability of NBMF to efficiently exploit raw data.