Overdispersed Count Data Research Articles

Efficient Analysis of Overdispersed Data Using an Accurate Computation of the Dirichlet Multinomial Distribution.

Modeling count data using suitable statistical distributions has been instrumental for analyzing the patterns it conveys. However, failing to address critical aspects, like overdispersion, jeopardizes the effectiveness of such an analysis. In this paper, overdispersed count data is modeled using the Dirichlet Multinomial (DM) distribution by maximizing its likelihood using a fixed-point iteration algorithm. This is achieved by estimating the DM distribution parameters while comparing the recent Languasco-Migliardi (LM), and the Yu-Shaw (YS) procedures, which address the well-known computational difficulties of evaluating its log-likelihood. Experiments were conducted using multiple datasets from different domains spanning polls, images, and IoT network traffic. They all showed the superiority of the LM procedure as it succeeded at estimating the DM parameters at the designated level of accuracy in all experiments, while the YS procedure failed to produce sufficiently accurate results (or any results at all) in several experiments. Moreover, the LM procedure achieved a speedup that ranged from 2-fold to 20-fold over YS.

Prediction Intervals for Overdispersed Poisson Data and Their Application in Medical and Pre-Clinical Quality Control.

In pre-clinical and medical quality control, it is of interest to assess the stability of the process under monitoring or to validate a current observation using historical control data. Classically, this is done by the application of historical control limits (HCL) graphically displayed in control charts. In many applications, HCL are applied to count data, for example, the number of revertant colonies (Ames assay) or the number of relapses per multiple sclerosis patient. Count data may be overdispersed, can be heavily right-skewed and clusters may differ in cluster size or other baseline quantities (e.g., number of petri dishes per control group or different length of monitoring times per patient). Based on the quasi-Poisson assumption or the negative-binomial distribution, we propose prediction intervals for overdispersed count data to be used as HCL. Variable baseline quantities are accounted for by offsets. Furthermore, we provide a bootstrap calibration algorithm that accounts for the skewed distribution and achieves equal tail probabilities. Comprehensive Monte-Carlo simulations assessing the coverage probabilities of eight different methods for HCL calculation reveal, that the bootstrap calibrated prediction intervals control the type-1-error best. Heuristics traditionally used in control charts (e.g., the limits in Shewhart c- or u-charts or the mean ± 2 SD) fail to control a pre-specified coverage probability. The application of HCL is demonstrated based on data from the Ames assay and for numbers of relapses of multiple sclerosis patients. The proposed prediction intervals and the algorithm for bootstrap calibration are publicly available via the R package predint.

Estimation of mean using under-reported and overdispersed count data

Count data arising in various fields of applications are often under-reported. Ignoring undercount naturally leads to biased estimators and inaccurate confidence intervals. Further, overdispersion in count data may arise due to inherent heterogeneity of the data. Negative Binomial distribution is a viable candidate for modeling overdispersed count data. However, in presence of undercount the negative binomial model needs to be adjusted. In this paper, we shall develop likelihood-based methodologies for estimation of mean using validation data after accounting for underreporting and overdispersion. The impact of ignoring undercount on the coverage and length of the confidence intervals is investigated using extensive numerical studies. The study is supplemented with a real-life data analysis.

Modified ridge estimator in the Bell regression model

Abstract The Bell regression model (BRM), a member of the generalized linear models (GLMs), can be used when the dependent variable consists of overdispersed count data. The maximum likelihood estimator (MLE) is generally used to estimate unknown regression coefficients. The major drawback of the MLE is an inflated variance when multicollinearity problems occur. In this study, we proposed a new biased estimator to cope with the multicollinearity in the BRM. The simulation study is conducted to illustrate the performance of the proposed estimator over the MLE and Bell ridge estimator (BRE). Also, we give real data examples to approve the applicability of the proposed estimator in real data problems.

Regression to the mean for overdispersed count data

In repeated measurements, regression to the mean (RTM) is a tendency of subjects with observed extreme values to move closer to the mean when measured a second time. Not accounting for RTM could lead to incorrect decisions such as when observed natural variation is incorrectly attributed to the effect of a treatment/intervention. A strategy for addressing RTM is to decompose the total effect, the expected difference in paired random variables conditional on the first being in the tail of its distribution, into regression to the mean and unbiased treatment effects. The unbiased treatment effect can then be estimated by subtraction. Formulae are available in the literature to quantify RTM for Poisson distributed data which are constrained by mean–variance equivalence, although there are many real life examples of overdispersed count data that are not well approximated by the Poisson. The negative binomial can be considered an explicit overdispersed Poisson process where the Poisson intensity is chosen from a gamma distribution. In this study, the truncated bivariate negative binomial distribution is used to decompose the total effect formulae into RTM and treatment effects. Maximum likelihood estimators (MLE) and method of moments estimators are developed for the total, RTM, and treatment effects. A simulation study is carried out to investigate the properties of the estimators and compare them with those developed under the assumption of the Poisson process. Data on the incidence of dengue cases reported from 2007 to 2017 are used to estimate the total, RTM, and treatment effects.

Semi-parametric approach for modelling overdispersed count data with application to Industry 4.0

The paper deals with a test for the goodness-of-fit of a model for count data, in the framework of Generalized Linear Models. The motivating example concerns the study on the effectiveness of policy incentives for the adoption of 4.0 technologies by Small and Medium Enterprises. According to the literature, openness to Industry 4.0 should be measured in terms of the number of 4.0 technologies adopted, represented by a count variable. To investigate the effectiveness of public policy interventions to encourage the adoption of 4.0 technologies, we propose the application of a model for count data with a permutation ANOVA to test the goodness-of-fit and for the model selection. When the distribution of the response is neither Poisson nor Negative Binomial, and in the quite common situation in which the response variance is much greater than the mean, the classic Poisson regression and Negative Binomial regression are not valid. The proposed testing method is based on the combination of permutation tests on the significance of the regression coefficient estimates. The power behaviour of the proposed semi-parametric solution is investigated through a comparative Monte Carlo simulation study. The performance of such a method is compared to those of the two main parametric competitors. The application of the permutation test to an interesting case study is presented. The dataset is original, and related to a sample survey carried out in Italy, about the adoption of Industry 4.0 technologies by Italian enterprises.

Modified Bivariate Poisson-Lindley Model: Properties and Applications in Soccer

Abstract This paper presents the bivariate Poisson-new XLindley distribution (BPNXLD), which may be used to represent dependent and over-dispersed countdata. Among the characteristics considered are the correlation coefficient, mean, and variance of the distribution. Acomparison with several Bivariate distributions is included. The goodness of fit of this novel model iscompared with the bivariate Poisson, bivariate negative binomial and bivariate Poisson-Lindley distributions using two data sets from a German Bundesliga season.

One-misrecorded Poisson INAR(1) model via two random operators with application to crime and economics data

In this paper, we propose a first-order non-negative integer-valued autoregressive (INAR(1)) model with one-misrecorded Poisson (OMP) marginal via a combination of the generalized binomial thinning and mixture Pegram operators for zero-inflated and one deflated count time series. The suggested model is suitable for multimodal, equi- and over-dispersed data modeling. It contains two particular cases: the mixture of Pegram and thinning of a first-order integer-valued autoregressive (MPT(1)) with Poisson and one-misrecorded Poisson. The distribution of the innovation term is derived as a mixture of degenerate distributions at 0 and 1, and two Poisson distributions with certain parameters. Regression and several statistical properties of the proposed model are discussed. We investigate the distribution of the runs (the lengths of zeros and ones). The parameters of the model are estimated using the maximum-likelihood, modified Yule–Walker, and modified conditional least squares methods. The estimation of the parameters, their behavior, and their performance are presented through a simulation study. Two practical data sets on the monthly cases of criminal records and weekly sales are applied to check the proposed process’s performance against other relevant INAR(1) models, showing its capabilities in the challenging case of over-dispersed count data. Furthermore, the proposed model discusses data forecasting.

New ridge parameter estimators for the quasi-Poisson ridge regression model

The quasi-Poisson regression model is used for count data and is preferred over the Poisson regression model in the case of over-dispersed count data. The quasi-likelihood estimator is used to estimate the regression coefficients of the quasi-Poisson regression model. The quasi-likelihood estimator gives sub-optimal estimates if regressors are highly correlated—multicollinearity issue. Biased estimation methods are often used to overcome the multicollinearity issue in the regression model. In this study, we explore the ridge estimator for the quasi-Poisson regression model to mitigate the multicollinearity issue. Furthermore, we propose various ridge parameter estimators for this model. We derive the theoretical properties of the ridge estimator and compare its performance with the quasi-likelihood estimator in terms of matrix and scalar mean squared error. We further compared the proposed estimator numerically through a Monte Carlo simulation study and a real-life application. We found that both the simulation and application results show the superiority of the ridge estimator, particularly with the best ridge parameter estimator, over the quasi-likelihood estimator in the presence of multicollinearity issue.

Poisson-New Quadratic-Exponential Distribution

This proposed distribution is a discrete compound probability distribution with only one parameter. To get this distribution, Poisson distribution has been mixed with the New Quadratic-Exponential distribution of Sah (2022). Hence, it is named as “Poisson-New Quadratic-Exponential Exponential Distribution (PNLED)”. The important statistical characteristics needed to check the validity of this distribution have been derived and clearly explained. To check the validity of the theoretical works of this distribution, while using goodness of fit on some over-dispersed count data, what we have been found that this distribution seems a better alternative of Poisson-Lindley distribution (PLD) of Sankaran (1970), Poisson Mishra distribution (PMD) of Sah (2017) and Poisson-Modified Mishra distribution (PMMD) of Sah and Sahani (2023).

Bootstrapping generalized linear models to accommodate overdispersed count data

Bootstrapping generalized linear models to accommodate overdispersed count data

High-dimensional covariate-augmented overdispersed poisson factor model.

The current Poisson factor models often assume that the factors are unknown, which overlooks the explanatory potential of certain observable covariates. This study focuses on high dimensional settings, where the number of the count response variables and/or covariates can diverge as the sample size increases. A covariate-augmented overdispersed Poisson factor model is proposed to jointly perform a high-dimensional Poisson factor analysis and estimate a large coefficient matrix for overdispersed count data. A group of identifiability conditions is provided to theoretically guarantee computational identifiability. We incorporate the interdependence of both response variables and covariates by imposing a low-rank constraint on the large coefficient matrix. To address the computation challenges posed by nonlinearity, two high-dimensional latent matrices, and the low-rank constraint, we propose a novel variational estimation scheme that combines Laplace and Taylor approximations. We also develop a criterion based on a singular value ratio to determine the number of factors and the rank of the coefficient matrix. Comprehensive simulation studies demonstrate that the proposed method outperforms the state-of-the-art methods in estimation accuracy and computational efficiency. The practical merit of our method is demonstrated by an application to the CITE-seq dataset. A flexible implementation of our proposed method is available in the R package COAP.

A new over-dispersed count model based on Poisson-Geometric convolution

A new two-parameter discrete distribution, namely the PoiG distribution, is derived by the convolution of a Poisson variate and an independently distributed geometric random variable. This distribution generalizes both the Poisson and geometric distributions and can be used for modeling over-dispersed as well as equi-dispersed count data. A number of important statistical properties of the proposed count model, such as the probability generating function, the moment generating function, the moments, the survival function and the hazard rate function have been studied in detail. Monotonic properties such as the log concavity are also studied. Stochastic ordering for the proposed model is also investigated in detail. Method of moments and the maximum-likelihood estimators of the parameters of the proposed model are found to work well in simulation experiments as well as in the real-life data analysis. Asymptotic confidence intervals and the coverage probabilities of the parameters are also studied through extensive simulation experiments. Two real life modelling applications demonstrate that the proposed distribution may prove to be useful for the practitioners for modelling over-dispersed count data compared to its closest competitors.

Premium Linear-Exponential Mixture of Poisson Distribution

The proposed distribution is a discrete probability distribution with only one parameter and it has been derived by compounding the Poisson distribution with the Premium Linear-exponential distribution (PLED) of Sah (2022). So, it is named as Premium Linear-Exponential Mixture of Poisson Distribution (PLEMPD). The required statistical characteristics have been derived and explained to give the theoretical concept of this distribution. The main purpose of proposing this distribution is to provide a better goodness of fit for over-dispersed count data than the previously derived one parameter continuous mixtures of Poisson distribution of same nature by others. Taking some secondary over-dispersed count data and using the theoretical concept of this distribution, it has been observed that this distribution is more suitable for statistical modeling of over-dispersed count data than Poisson-Lindley distribution (PLD) of Sankaran (1970) and Poisson-Mishra distribution (PMD) of Sah (2017).

Multilevel modeling in single-case studies with zero-inflated and overdispersed count data.

Count outcomes are frequently encountered in single-case experimental designs (SCEDs). Generalized linear mixed models (GLMMs) have shown promise in handling overdispersed count data. However, the presence of excessive zeros in the baseline phase of SCEDs introduces a more complex issue known as zero-inflation, often overlooked by researchers. This study aimed to deal with zero-inflated and overdispersed count data within a multiple-baseline design (MBD) in single-case studies. It examined the performance of various GLMMs (Poisson, negative binomial [NB], zero-inflated Poisson [ZIP], and zero-inflated negative binomial [ZINB] models) in estimating treatment effects and generating inferential statistics. Additionally, a real example was used to demonstrate the analysis of zero-inflated and overdispersed count data. The simulation results indicated that the ZINB model provided accurate estimates for treatment effects, while the other three models yielded biased estimates. The inferential statistics obtained from the ZINB model were reliable when the baseline rate was low. However, when the data were overdispersed but not zero-inflated, both the ZINB and ZIP models exhibited poor performance in accurately estimating treatment effects. These findings contribute to our understanding of using GLMMs to handle zero-inflated and overdispersed count data in SCEDs. The implications, limitations, and future research directions are also discussed.

A variable clustering approach for overdispersed high-dimensional count data using a copula-based mixture model

In this paper, we propose a latent variable model for the analysis and clustering of high-dimensional correlated and overdispersed count data. We use a set of random effects to capture within-group correlations and a copula to address between-group correlations. Since maximum likelihood (ML) estimation is computationally intensive for high-dimensional data, we propose a fast moment-based estimation procedure. Additionally, we implement the proposed estimation procedure into a clustering algorithm, borrowing ideas from the K-means algorithm. Based on a simulation study, we demonstrate that the estimation procedure shows good stability and precision. Furthermore, it is computationally fast, unlike the ML approach which shows serious convergence issues. In addition, the clustering algorithm with our moment-based estimation procedure can identify the simulated grouped data structure. The approach is further illustrated with a high-dimensional RNA-Seq dataset of commercial potatoes.

Poisson XRani Distribution: An Alternative Discrete Distribution for Overdispersed Count Data

In this article, a one‐parameter probability mass function called Poisson XRani is proposed by combining the Poisson and the XRani distributions. A wide range of distributional properties such as the shape of the probability mass function, probability generating function, factorial moments, moment generating function, characteristic function, raw moments, dispersion index, mean, variance, coefficient of skewness, and coefficient of kurtosis are investigated. It is proven that the proposed probability mass function can easily handle overdispersed and right‐skewed count observations with heavy tails. Both maximum likelihood and Bayesian estimation techniques are used to estimate the unknown parameter of the proposed distribution and a simulation study was conducted to examine and analyze the performance of the maximum likelihood estimation procedure based on sample size, absolute bias, relative absolute bias, and root mean square error. The usefulness of the proposed distribution is assessed using two distinctive real datasets. These applications reveal that the new distribution provides an adequate fit compared to the other eight discrete distributions.

A New Effective Jackknifing Estimator in the Negative Binomial Regression Model

The negative binomial regression model is a widely adopted approach when dealing with dependent variables that consist of non-negative integers or counts. This model serves as an alternative regression technique for addressing issues related to overdispersion in count data. Typically, the maximum likelihood estimator is employed to estimate the parameters of the negative binomial regression model. However, the maximum likelihood estimator can be highly sensitive to multicollinearity, leading to unreliable results. To eliminate the adverse effects of multicollinearity in the negative binomial regression model, we propose the use of a jackknife version of the Kibria–Lukman estimator. In this study, we conducted a theoretical comparison between the proposed jackknife Kibria–Lukman negative binomial regression estimator and several existing estimators documented in the literature. To assess the performance of the proposed estimator, we conducted two simulation studies and performed a real data application. The results from both the simulation studies and the real data application consistently demonstrated that the proposed jackknife Kibria–Lukman negative binomial regression estimator outperforms other estimators.

A truncated mean-parameterized Conway-Maxwell-Poisson model for the analysis of Test match bowlers

A truncated, mean-parameterized Conway-Maxwell-Poisson model is developed to handle under- and overdispersed count data owing to individual heterogeneity. The truncated nature of the data allows for a more direct implementation of the model than is utilized in previous work without too much computational burden. The model is applied to a large dataset of Test match cricket bowlers, where the data are in the form of small counts and range in time from 1877 to the modern day, leading to the inclusion of temporal effects to account for fundamental changes to the sport and society. Rankings of sportsmen and women based on a statistical model are often handicapped by the popularity of inappropriate traditional metrics, which are found to be flawed measures in this instance. Inferences are made using a Bayesian approach by deploying a Markov Chain Monte Carlo algorithm to obtain parameter estimates and to extract the innate ability of individual players. The model offers a good fit and indicates that there is merit in a more sophisticated measure for ranking and assessing Test match bowlers.

A Family of Finite Mixture Distributions for Modelling Dispersion in Count Data

This paper considers the construction of a family of discrete distributions with the flexibility to cater for under-, equi- and over-dispersion in count data using a finite mixture model based on standard distributions. We are motivated to introduce this family because its simple finite mixture structure adds flexibility and facilitates application and use in analysis. The family of distributions is exemplified using a mixture of negative binomial and shifted negative binomial distributions. Some basic and probabilistic properties are derived. We perform hypothesis testing for equi-dispersion and simulation studies of their power and consider parameter estimation via maximum likelihood and probability-generating-function-based methods. The utility of the distributions is illustrated via their application to real biological data sets exhibiting under-, equi- and over-dispersion. It is shown that the distribution fits better than the well-known generalized Poisson and COM–Poisson distributions for handling under-, equi- and over-dispersion in count data.