Source Of Overdispersion Research Articles

Outliers as a Source of Overdispersion in Poisson Regression Modelling: Evidence from Simulation and Real Data

The Poisson regression model is a well-known technique for modelling count data. However, it is necessary to satisfy the overdispersion assumption in order to fit the Poisson regression model. Due to the overdispersion problem in the Poisson regression model, standard errors might be underestimated, which may lead to a highly misleading inference. There are several tests in the literature to check the presence of overdispersion in the Poisson model. In this study, we apply a regression-based t test to identify the overdispersion. The simulation study and real data example clearly show that the overdispersion in the Poisson model is caused by the existence of outliers. International Journal of Statistical Sciences, Vol. 23(2), November, 2023, pp 31-37

Zero-inflated poisson quasi-Lindley regression for modeling number of doctor visit data

The Poisson regression is a popular approach in modeling count data. However, in many situations often the variance of data is greater than the mean (over-dispersed data) and the generalized Poisson or mixed Poisson models such as the Poisson gamma (negative binomial), Poisson inverse Gaussian, Poisson lognormal, and Poisson Lindley have been proposed as the alternatives to the Poisson for describing over-dispersed count data. In some situations, the source of over-dispersion is the large percentage of zeros in the dataset. In the other words, the dataset involves an excessive number of zeros than are expected in the common discrete distributions which are known as the zero-inflated events. In order to analyze these data, zero-inflated models such as the zero-inflated Poisson, zero-inflated generalized Poisson, and zero-inflated negative binomial have been applied. This work proposes the functional form and the regression model of the zero-inflated Poisson quasi-Lindley (ZIPQL) and then, beside the alternative models, it was fitted and compared to US National Medical Expenditure Survey data.

Modeling zero inflation is not necessary for spatial transcriptomics

BackgroundSpatial transcriptomics are a set of new technologies that profile gene expression on tissues with spatial localization information. With technological advances, recent spatial transcriptomics data are often in the form of sparse counts with an excessive amount of zero values.ResultsWe perform a comprehensive analysis on 20 spatial transcriptomics datasets collected from 11 distinct technologies to characterize the distributional properties of the expression count data and understand the statistical nature of the zero values. Across datasets, we show that a substantial fraction of genes displays overdispersion and/or zero inflation that cannot be accounted for by a Poisson model, with genes displaying overdispersion substantially overlapped with genes displaying zero inflation. In addition, we find that either the Poisson or the negative binomial model is sufficient for modeling the majority of genes across most spatial transcriptomics technologies. We further show major sources of overdispersion and zero inflation in spatial transcriptomics including gene expression heterogeneity across tissue locations and spatial distribution of cell types. In particular, when we focus on a relatively homogeneous set of tissue locations or control for cell type compositions, the number of detected overdispersed and/or zero-inflated genes is substantially reduced, and a simple Poisson model is often sufficient to fit the gene expression data there.ConclusionsOur study provides the first comprehensive evidence that excessive zeros in spatial transcriptomics are not due to zero inflation, supporting the use of count models without a zero inflation component for modeling spatial transcriptomics.

Beta dose heterogeneity in sediment samples measured using a Timepix pixelated detector and its implications for optical dating of individual mineral grains

Single-grain optical dating has been widely used to estimate depositional ages for Quaternary sediments. An understanding of the sources influencing the dispersion of equivalent dose (D e ) estimates from individual grain measurements is essential for accurate age determination. Beta microdosimetry (i.e., beta dose deposition at the sub-millimetre scale) is a known cause of spread in single-grain D e values, so the detection and assessment of beta dose rate ( D ˙ β ) variation is important to properly interpret these data. Here we demonstrate the application of Timepix, a pixelated detector that directly measures in situ beta emissions based on cluster analysis, to determine the beta microdosimetry of natural sediment samples from a cave deposit in Russia and a sand dune in Australia. We describe a Timepix measurement and data processing procedure for natural sediments, and establish sample-specific calibration curves using associated sample radioelement concentrations to convert Timepix count rates into estimates of D ˙ β . On the basis of the Timepix analysis, a 2D ‘heat map’ of D ˙ β at sub-millimetre resolution was obtained for each sample. Our results show that the D ˙ β estimates are heterogeneous and their non-uniformity gives rise to 9 ± 4 to 26 ± 5% overdispersion in the single-grain D e distributions for the samples examined here. We discuss the likely sources of D ˙ β heterogeneity, based on micromorphological investigations of these sediment samples, which include a variety of materials present in natural deposits associated with D ˙ β ‘hot’ and ‘cold’ spots. A comparison of the D ˙ β dispersion with that of the corresponding single-grain D e values shows that the scatter among the latter can be fully or partly explained by beta microdosimetry and other known sources of overdispersion. • Timepix pixelated detector used to measure beta dose rates of natural sediments. • Measurement and data analysis procedure proposed and tested on four samples. • Spatially-resolved ‘heat map’ of beta dose rates obtained at sub-millimetre resolution. • Beta heterogeneity gives rise to ∼9–26% overdispersion of single-grain D e values. • Beta heterogeneity can fully or partly explain overdispersion of single-grain D e values.

A new regression model for overdispersed binomial data accounting for outliers and an excess of zeros.

Binary outcomes are extremely common in biomedical research. Despite its popularity, binomial regression often fails to model this kind of data accurately due to the overdispersion problem. Many alternatives can be found in the literature, the beta‐binomial (BB) regression model being one of the most popular. The additional parameter of this model enables a better fit to overdispersed data. It also exhibits an attractive interpretation in terms of the intraclass correlation coefficient. Nonetheless, in many real data applications, a single additional parameter cannot handle the entire excess of variability. In this study, we propose a new finite mixture distribution with BB components, namely, the flexible beta‐binomial (FBB), which is characterized by a richer parameterization. This allows us to enhance the variance structure to account for multiple causes of overdispersion while also preserving the intraclass correlation interpretation. The novel regression model, based on the FBB distribution, exploits the flexibility and large variety of the distribution's possible shapes (which includes bimodality and various tail behaviors). Thus, it succeeds in accounting for several (possibly concomitant) sources of overdispersion stemming from the presence of latent groups in the population, outliers, and excessive zero observations. Adopting a Bayesian approach to inference, we perform an intensive simulation study that shows the superiority of the new regression model over that of the existing ones. Its better performance is also confirmed by three applications to real datasets extensively studied in the biomedical literature, namely, bacteria data, atomic bomb radiation data, and control mice data.

The beta‐binomial: A preliminary comparison of smaller samples having many replications versus larger samples having fewer replications

AbstractThe experiment investigated whether a body of difference test data, arranged into small consumer samples with many replications, would elicit more or less overdispersion than an equal number of tests from same body of data, rearranged into a series of larger consumer samples, with fewer replications. Four separate experiments were performed, each with a separate set of 200 consumers. In two of the experiments, consumers used 2‐AFC tests while in the remaining two, consumers used triangle tests. Data were then assigned to samples of different size ranging from 100 to 28. The number of replicate tests assigned for each sample size was arranged so that the total number of tests performed was always close to 200. For the two experiments involving 2‐AFC tests, smaller physical differences were assessed in one experiment and larger physical differences in the other. Same was true for the two experiments using triangle tests. For the two experiments where smaller physical differences were assessed, overdispersion tended to increase as sample sizes became larger. For the two experiments where larger physical differences were assessed, overdispersion tended to decrease as sample sizes became larger. These apparently contradictory results were explained by the sensitivity variation hypothesis.Practical applicationsBeta‐binomial statistics are used for significance testing for difference tests, when the experimenter combines the number of subjects with the number of replicate tastings, to increase test power. This is to circumvent the problems of overdispersion, the extra variance involved when this combination is made. Nearly all the work on overdispersion is theoretical. Very little experimental work has been published. From prior experimentation, the sensitivity variation hypothesis was developed, which indicates that overdispersion simply depends on the performance of the particular consumers in the sample and is not governed by the type of test used. This experiment successfully applied the sensitivity variation hypothesis to a set of seemingly contradictory results, regarding the overdispersion elicited in a series of four experiments. It provides further much needed experimental insight, into the effects of sample size and numbers of replications as well as the sources of overdispersion per se.

On performance of parametric and distribution-free models for zero-inflated and over-dispersed count responses.

Zero-inflated Poisson (ZIP) and negative binomial (ZINB) models are widely used to model zero-inflated count responses. These models extend the Poisson and negative binomial (NB) to address excessive zeros in the count response. By adding a degenerate distribution centered at 0 and interpreting it as describing a non-risk group in the population, the ZIP (ZINB) models a two-component population mixture. As in applications of Poisson and NB, the key difference between ZIP and ZINB is the allowance for overdispersion by the ZINB in its NB component in modeling the count response for the at-risk group. Overdispersion arising in practice too often does not follow the NB, and applications of ZINB to such data yield invalid inference. If sources of overdispersion are known, other parametric models may be used to directly model the overdispersion. Such models too are subject to assumed distributions. Further, this approach may not be applicable if information about the sources of overdispersion is unavailable. In this paper, we propose a distribution-free alternative and compare its performance with these popular parametric models as well as a moment-based approach proposed by Yu et al. [Statistics in Medicine 2013; 32: 2390-2405]. Like the generalized estimating equations, the proposed approach requires no elaborate distribution assumptions. Compared with the approach of Yu et al., it is more robust to overdispersed zero-inflated responses. We illustrate our approach with both simulated and real study data.

Approaches for dealing with various sources of overdispersion in modeling count data: Scale adjustment versus modeling.

Overdispersion is a common problem in count data. It can occur due to extra population-heterogeneity, omission of key predictors, and outliers. Unless properly handled, this can lead to invalid inference. Our goal is to assess the differential performance of methods for dealing with overdispersion from several sources. We considered six different approaches: unadjusted Poisson regression (Poisson), deviance-scale-adjusted Poisson regression (DS-Poisson), Pearson-scale-adjusted Poisson regression (PS-Poisson), negative-binomial regression (NB), and two generalized linear mixed models (GLMM) with random intercept, log-link and Poisson (Poisson-GLMM) and negative-binomial (NB-GLMM) distributions. To rank order the preference of the models, we used Akaike's information criteria/Bayesian information criteria values, standard error, and 95% confidence-interval coverage of the parameter values. To compare these methods, we used simulated count data with overdispersion of different magnitude from three different sources. Mean of the count response was associated with three predictors. Data from two real-case studies are also analyzed. The simulation results showed that NB and NB-GLMM were preferred for dealing with overdispersion resulting from any of the sources we considered. Poisson and DS-Poisson often produced smaller standard-error estimates than expected, while PS-Poisson conversely produced larger standard-error estimates. Thus, it is good practice to compare several model options to determine the best method of modeling count data.

Evidence for overdispersion in the distribution of malaria parasites and leukocytes in thick blood smears

BackgroundMicroscopic examination of stained thick blood smears (TBS) is the gold standard for routine malaria diagnosis. Parasites and leukocytes are counted in a predetermined number of high power fields (HPFs). Data on parasite and leukocyte counts per HPF are of broad scientific value. However, in published studies, most of the information on parasite density (PD) is presented as summary statistics (e.g. PD per microlitre, prevalence, absolute/assumed white blood cell counts), but original data sets are not readily available. Besides, the number of parasites and the number of leukocytes per HPF are assumed to be Poisson-distributed. However, count data rarely fit the restrictive assumptions of the Poisson distribution. The violation of these assumptions commonly results in overdispersion. The objectives of this paper are to investigate and handle overdispersion in field-collected data.MethodsThe data comprise the records of three TBSs of 12-month-old children from a field study of Plasmodium falciparum malaria in Tori Bossito, Benin. All HPFs were examined systemically by visually scanning the film horizontally from edge to edge. The numbers of parasites and leukocytes per HPF were recorded and formed the first dataset on parasite and leukocyte counts per HPF. The full dataset is published in this study. Two sources of overdispersion in data are investigated: latent heterogeneity and spatial dependence. Unobserved heterogeneity in data is accounted for by considering more flexible models that allow for overdispersion. Of particular interest were the negative binomial model (NB) and mixture models. The dependent structure in data was modelled with hidden Markov models (HMMs).ResultsThe Poisson assumptions are inconsistent with parasite and leukocyte distributions per HPF. Among simple parametric models, the NB model is the closest to the unknown distribution that generates the data. On the basis of model selection criteria AIC and BIC, HMMs provided a better fit to data than mixtures. Ordinary pseudo-residuals confirmed the validity of HMMs.ConclusionFailure to take overdispersion into account in parasite and leukocyte counts may entail important misleading inferences when these data are related to other explanatory variables (malariometric or environmental). Its detection is therefore essential. In addition, an alternative PD estimation method that accounts for heterogeneity and spatial dependence should be seriously considered in epidemiological studies with field-collected parasite and leukocyte data.

Modeling heterogeneity for count data: A study of maternal mortality in health facilities in Mozambique

Count data are very common in health services research, and very commonly the basic Poisson regression model has to be extended in several ways to accommodate several sources of heterogeneity: (i) an excess number of zeros relative to a Poisson distribution, (ii) hierarchical structures, and correlated data, (iii) remaining "unexplained" sources of overdispersion. In this paper, we propose hierarchical zero-inflated and overdispersed models with independent, correlated, and shared random effects for both components of the mixture model. We show that all different extensions of the Poisson model can be based on the concept of mixture models, and that they can be combined to account for all different sources of heterogeneity. Expressions for the first two moments are derived and discussed. The models are applied to data on maternal deaths and related risk factors within health facilities in Mozambique. The final model shows that the maternal mortality rate mainly depends on the geographical location of the health facility, the percentage of women admitted with HIV and the percentage of referrals from the health facility.

Industrial Location and Spatial Dependence: An Empirical Application

Liviano D. and Arauzo-Carod J.-M. Industrial location and spatial dependence: an empirical application, Regional Studies. This paper tries to resolve some of the main shortcomings in the empirical literature on location decisions for new plants, that is, spatial effects and over-dispersion. Spatial effects are omnipresent, being a source of over-dispersion in the data as well as a factor shaping the functional relationship between the variables that explain a firm's location decisions. Using count data models, empirical researchers have dealt with over-dispersion and excess zeros by developments of the Poisson regression model. This study aims to take this a step further by adopting Bayesian methods and models in order to tackle the excess of zeros, spatial and non-spatial over-dispersion, and spatial dependence simultaneously. Data for Catalonia (Spain) are used and location determinants are analysed to that end. The results show that spatial effects are determinant. Additionally, over-dispersion is decomposed into an unstructured independently and identically distributed (i.i.d.) effect and a spatially structured effect.

Sources of overdispersion in a K-rich feldspar sample from north-central India: Insights from De, K content and IRSL age distributions for individual grains

Abstract Luminescence dating of individual sand-sized grains of quartz is a well-established technique in Quaternary geochronology, but the most ubiquitous mineral on the surface of the Earth—feldspar—has received much less attention at the single-grain level. In this study, we estimated single-grain equivalent dose values and infrared stimulated luminescence (IRSL) ages for K-rich feldspar (KF) grains from a fluvial sample underlying Youngest Toba Tuff (YTT) deposits in north-central India, and compared these ages (corrected for anomalous fading) with those obtained from individual grains of quartz from the same sample. Both minerals have broadly similar single-grain age distributions, but both are greatly overdispersed and most grains have ages substantially younger than the expected age of the YTT deposit (∼74 ka). Almost half (45%) of KF grains used for age calculation have fading rates statistically consistent with zero, but the age distribution of these grains is as dispersed as that of the entire population. We obtained a similar distribution of ages calculated for 51 grains using their individually measured internal K contents, which exhibited only minor grain-to-grain variation. Given the lack of dependency of single-grain ages on the measured fading rates and internal K contents, and the overall adequacy of bleaching of grains collected from a sandbar in the modern river channel, we consider the spread in ages is most likely due to mixing, at the time of deposition and after the YTT event, of potentially well-bleached fluvially-transported sediments with older grains derived from slumping of riverbank deposits. Some spread may also be due to natural variations in the IRSL properties of individual KF grains.

The dynamic influence of genetic variation on the susceptibility of sheep to gastrointestinal nematode infection.

The interaction between sheep and the nematode Teladorsagia circumcincta is one of the best understood of all host-parasite interactions. Following infection, there is considerable variation among lambs in the number of nematode eggs produced, the number of early fourth-stage larvae and the number of adult worms in the mucosa. These traits have a high variance to mean ratio (i.e. they are overdispersed or aggregated among hosts), they are skewed and approximately negative binomially distributed. The sources of overdispersion are differences among lambs in the ingestion of infective larvae and the immune response. Both forces can produce aggregation but their relative importance is unknown. The key components of variation can be identified by variance analysis. The sum of the average effects of polymorphic genes is known as additive genetic variation and this increases essentially from zero at one month of age to quite high values at six months of age. The major mechanism underlying genetic variation appears to be the differences among individuals in immune responses. Two of the major sources of variation in immune responses are differences in antigen recognition and differences in the type of cytokines produced. Genes that influence both these sources of variation are associated with differences in resistance to nematode infection. Therefore, much of the heterogeneity among animals in parasite transmission appears to be due to genetic variation in immune responsiveness.

Adult Survival Estimates From Two Atlantic Puffin (Fratercula Arctica) Colonies in The Gulf of Maine

AbstractWe report survival probabilities for 148 breeding adult Atlantic Puffins (Fratercula arctica) monitored through capture-mark-resight at two colonies for 11 years (1992–2003). The colonies, Eastern Egg Rock and Seal Island, are ∼42 km apart in the Gulf of Maine. Support for competing models in the program MARK suggests constant survival of 0.95 ± 0.01 (SE) that is independent of colony. Our high survival probability is consistent with published estimates for Atlantic Puffins and other long-lived seabirds. No time-variance contrasts with many long-term seabird studies, which often report high survival in most years, broken occasionally by low-survival events. However, a post-hoc observation of survival estimates from the time-dependent model suggests that there may have been at least two low-survival events in our time-series; sparse data may have precluded detection by our models. In this study, each bird received an individually engraved, plastic, field-readable leg band, as well as the standard metal band. Using an index of band readability, we show that plastic bands wore rapidly, resulting in accumulating losses of engraved characters through time. Degradation and loss of marks is a common source of overdispersion in capture-mark-re-encounter data and results in underestimated sampling variances. In the presence of a 70% reduction in band readability over eight years, an estimate of the adjusted overdispersion factor (ĉ = 1.14) identified very little overdispersion in our data. Overdispersion was avoided by double banding and intensively resighting metal bands.Estimation de la Survie Adulte de Deux Colonies de Fratercula arctica dans le Golfe du Maine

Overdispersion Phenomenon in Stochastic Modeling of Precipitation

Simple stochastic models fit to time series of daily precipitation amount have a marked tendency to underestimate the observed (or interannual) variance of monthly (or seasonal) total precipitation. By considering extensions of one particular class of stochastic model known as a chain-dependent process, the extent to which this “overdispersion” phenomenon is attributable to an inadequate model for high-frequency variation of precipitation is examined. For daily precipitation amount in January at Chico, California, fitting more complex stochastic models greatly reduces the underestimation of the variance of monthly total precipitation. One source of overdispersion, the number of wet days, can be completely eliminated through the use of a higher-order Markov chain for daily precipitation occurrence. Nevertheless, some of the observed variance remains unexplained and could possibly be attributed to low-frequency variation (sometimes termed “potential predictability”). Of special interest is the fact that these more complex stochastic models still underestimate the monthly variance, more so than does an alternative approach, in which the simplest form of chain-dependent process is conditioned on an index of large-scale atmospheric circulation.

HETEROGENEITY, EXCESS ZEROS, AND THE STRUCTURE OF COUNT DATA MODELS

This paper demonstrates that the unobserved heterogeneity commonly assumed to be the source of overdispersion in count data models has predictable implications for the probability structure of such mixture models. In particular, the common observation of excess zeros is a strict implication of unobserved heterogeneity. This result has important implications for using count model estimates for predicting certain interesting parameters. Test statistics to detect such heterogeneity-related departures from the null model are proposed and applied in a health-care utilization example, suggesting that a null Poisson model should be rejected in favour of a mixed alternative. © 1997 John Wiley & Sons, Ltd.