Observed Count Data Research Articles

Scalability challenges of machine learning models for estimating walking and cycling volumes in large networks

This study explores the scalability of machine learning models for estimating walking and cycling volumes across the extensive New South Wales (NSW) Six Cities Region in Australia using mobile phone and crowdsourced data. Previous research has focused on localized applications, missing the complexities of larger networks. The research addresses this gap by identifying unique challenges such as the scarcity and representativeness of observed count data, gaps in the crowdsourced and mobile phone data, and inconsistencies in link-level volume estimates. We propose and demonstrate the application of strategies like enhancing geographical diversity of observed count data and employing an extensive cross-validation approach in model training and testing. By leveraging various auxiliary datasets, the study demonstrates the effectiveness of these strategies in improving model performance. These findings provide valuable insights for transportation modelers, policymakers, and urban planners, offering a robust framework for supporting sustainable transportation infrastructure and policies with advanced data-driven methodologies.

Variation of the electron flux spectrum along a solar flare loop as inferred from STIX hard X-ray observations

Context. Regularized imaging spectroscopy was introduced for the construction of electron flux images at different energies from count visibilities recorded by the Reuven Ramaty High Energy Solar Spectroscopic Imager (RHESSI). In this work we seek to extend this approach to data from the Spectrometer/Telescope for Imaging X-rays (STIX) on board the Solar Orbiter mission. Aims. Our aims are to demonstrate the feasibility of regularized imaging spectroscopy as a method for analysis of STIX data, and also to show how such an analysis can lead to insights into the physical processes affecting the nonthermal electrons responsible for the hard X-ray emission observed by STIX. Methods. STIX records imaging data in an intrinsically different manner from RHESSI. Rather than sweeping the angular frequency plane in a set of concentric circles (one circle per detector), STIX uses 30 collimators, each corresponding to a specific angular frequency. For this work, we derived an appropriate modification to the previous computational approach for the analysis of the visibilities observed by STIX. This approach also allows for the observed count data to be placed into nonuniformly spaced energy bins. Results. We show that the regularized imaging spectroscopy approach is not only feasible for an analysis of the visibilities observed by STIX, but also more reliable. The application of the regularized imaging spectroscopy technique to several well-observed flares reveals details of the variation of the electron flux spectrum throughout the flare sources. Conclusions. We conclude that the visibility-based regularized imaging spectroscopy approach is well suited for the analysis of STIX data. We also used STIX electron flux spectral images to track, for the first time, the behavior of the accelerated electrons during their path from the acceleration site in the solar corona toward the chromosphere.

FlywayNet: A hidden semi‐Markov model for inferring the structure of migratory bird networks from count data

Abstract Every year, millions of birds migrate between breeding and nonbreeding habitat, but the relative numbers of animals moving between sites are difficult to observe directly. Here we propose FlywayNet, a discrete network model based on observed count data, to determine the most likely migration links between regions using statistical modelling and efficient inference tools. Our approach advances on previous studies by accounting for noisy observations and flexible stopover durations by modelling using interacting hidden semi‐Markov Models. In FlywayNet, individual birds sojourn in stopover nodes for a period of time before moving to other nodes with an unknown probability that we aim to estimate. Exact estimation using existing approaches is not possible, so we designed customised versions of the Monte Carlo expectation‐maximisation and approximate Bayesian computation algorithms for our model. We compare the efficiency and quality of estimation of these approaches on synthetic data and an applied case study. Our algorithms performed well on benchmark problems, with low absolute error and strong correlation between estimated and known parameters. On a case study using citizen science count data of the Far Eastern Curlew (Numenius madagascariensis), an endangered shorebird from the East Asian–Australasian Flyway, the ABC and MCEM algorithms generated contrasting recommendations due to a difference in optimisation criteria and noise in the data. For ABC, we recovered key features of population‐level movements predicted by experts despite the challenges of noisy unstructured data. Understanding connectivity places local conservation efforts and threat mitigation in the global context, yet it has proven difficult to rigorously quantify connectivity at the population level. Our approach provides a flexible framework to infer the structure of migratory networks in birds and other organisms.

Hot-spots detection in count data by Poisson assisted smooth sparse tensor decomposition

Count data occur widely in many bio-surveillance and healthcare applications, e.g. the numbers of new patients of different types of infectious diseases from different cities/counties/states repeatedly over time, say, daily/weekly/monthly. For this type of count data, one important task is the quick detection and localization of hot-spots in terms of unusual infectious rates so that we can respond appropriately. In this paper, we develop a method called Poisson assisted Smooth Sparse Tensor Decomposition (PoSSTenD), which not only detect when hot-spots occur but also localize where hot-spots occur. The main idea of our proposed PoSSTenD method is articulated as follows. First, we represent the observed count data as a three-dimensional tensor including (1) a spatial dimension for location patterns, e.g. different cities/countries/states; (2) a temporal domain for time patterns, e.g. daily/weekly/monthly; (3) a categorical dimension for different types of data sources, e.g. different types of diseases. Second, we fit this tensor into a Poisson regression model, and then we further decompose the infectious rate into two components: smooth global trend and local hot-spots. Third, we detect when hot-spots occur by building a cumulative sum (CUSUM) control chart and localize where hot-spots occur by their LASSO-type sparse estimation. The usefulness of our proposed methodology is validated through numerical simulation studies and a real-world dataset, which records the annual number of 10 different infectious diseases from 1993 to 2018 for 49 mainland states in the United States.

Non-homogeneous Poisson and linear regression models as approaches to study time series with change-points

In this study we consider some stochastic models and a Bayesian approach to analyze count time series data in the presence of one or more change-points. When the observed count data are all different of zero, it is possible to perform an analysis using linear regression models with normal distribution errors for the log-transformed data. When we have the presence of zero counts at different times, the statistical model based on the log-transformation is not suitable. Hence, in this case, it is possible to consider non-homogeneous Poisson processes (NHPP) models with a suitable rate function. In the present work we consider both models (linear regression and Poisson) to analyze three data sets. These sets are data recording the monthly visitors to New Zealand with the purpose of education, yearly tuberculosis incidence data from New York City, and monthly measles incidence data from Brazil. When the NHPP model is used a PLP (power law process) model is assumed for the intensity function. Additionally, in both models, the presence of change-points will be allowed.

Analyzing count data with measurement error

AbstractIn this article, we analyze observed count data such as the number of defects in a steel product where the observed counts are the true counts measured with errors. We account for the measurement error by using a measurement error model based on a latent lognormal (LLN) distribution. We consider making inference about a single population (e.g., from samples of a production lot) and a regression model (e.g., from runs of a designed experiment), where the measurement system properties are known, that is, the parameters of the LLN distribution are known. Then, we consider simultaneous inference for the single population and regression model as well as the measurement system. We demonstrate the proposed methodology with both simulated and real observed counts.

The ARMA Point Process and its Estimation

An autoregressive–moving-average (ARMA) point process model is introduced, which combines self-exciting and shot-noise cluster mechanisms, both useful in a variety of applications. The process is analogous to the ARMA for integer-valued time series, sharing methodological and mathematical similarities. A maximum likelihood estimation procedure, based on MCEM (Monte Carlo Expectation Maximization), is derived and studied. This approach conveniently allows for: (i) trends in immigration/background intensity, (ii) multiple parametric specifications of memory functions and mark distributions, as well as (iii) cases where marks and immigrants are not observed. As such, the ARMA point process provides a flexible framework to disentangle cluster mechanisms in continuously observed count data.

Modeling dynamic correlation in zero-inflated bivariate count data with applications to single-cell RNA sequencing data.

Interactions between biological molecules in a cell are tightly coordinated and often highly dynamic. As a result of these varying signaling activities, changes in gene coexpression patterns could often be observed. The advancements in next‐generation sequencing technologies bring new statistical challenges for studying these dynamic changes of gene coexpression. In recent years, methods have been developed to examine genomic information from individual cells. Single‐cell RNA sequencing (scRNA‐seq) data are count‐based, and often exhibit characteristics such as overdispersion and zero inflation. To explore the dynamic dependence structure in scRNA‐seq data and other zero‐inflated count data, new approaches are needed. In this paper, we consider overdispersion and zero inflation in count outcomes and propose a ZEro‐inflated negative binomial dynamic COrrelation model (ZENCO). The observed count data are modeled as a mixture of two components: success amplifications and dropout events in ZENCO. A latent variable is incorporated into ZENCO to model the covariate‐dependent correlation structure. We conduct simulation studies to evaluate the performance of our proposed method and to compare it with existing approaches. We also illustrate the implementation of our proposed approach using scRNA‐seq data from a study of minimal residual disease in melanoma.

Self‐information‐based weighted CUSUM charts for monitoring Poisson count data with varying sample sizes

AbstractIn many applications, the Poisson count data with varying sample sizes are monitored using statistical process control charts. Among these applications, the weighted CUSUM charts are developed to deal with the effect of the varying sample sizes. However, some of them use limited information of the sample size or the count data while assigning the weights. To gain more information of the process, the self‐information weight functions are developed based on both the sample size and the observed count data. Then, the weighted CUSUM charts are proposed with the self‐information‐based weight. Simulation studies show the self‐information‐based weighted CUSUM charts perform better than the benchmark methods in detecting small shifts. Moreover, the performance of proposed method with estimated parameters is investigated via simulation. Finally, an example is given to illustrate the application of the proposed weighted CUSUM charts.

Zero‐inflated modeling part I: Traditional zero‐inflated count regression models, their applications, and computational tools

AbstractCount regression models maintain a steadfast presence in modern applied statistics as highlighted by their usage in diverse areas like biometry, ecology, and insurance. However, a common practical problem with observed count data is the presence of excess zeros relative to the assumed count distribution. The seminal work of Lambert (1992) was one of the first articles to thoroughly treat the problem of zero‐inflated count data in the presence of covariates. Since then, a vast literature has emerged regarding zero‐inflated count regression models. In this first of two review articles, we survey some of the classic and contemporary literature on parametric zero‐inflated count regression models, with emphasis on the utility of different univariate discrete distributions. We highlight some of the primary computational tools available for estimating and assessing the adequacy of these models. We concurrently emphasize the diverse data problems to which these models have been applied.This article is categorized under: Statistical Models > Generalized Linear Models Software for Computational Statistics > Software/Statistical Software Algorithms and Computational Methods > Maximum Likelihood Methods

Modeling Tree Species Count Data in the Understory and Canopy Layer of Two Mixed Old-Growth Forests in the Dinaric Region

The distribution of tree species has traditionally been analyzed based on tree diameter (DBH) as a continuous variable. However, this approach does not usually provide information on how species are distributed across the area of interest. In this study, an inverse approach was applied to investigate tree distribution patterns in two Dinaric old-growth forest stands composed primarily of European beech, silver fir, and Norway spruce. Specifically, the variance-to-mean relationship of tree counts based on 80 plots (40 in each old-growth stand) were evaluated by using a dispersion index. Understory trees exhibited clumped and random patterns, whereas canopy trees were mostly distributed in a random manner. A regular pattern was only determined for beech and all trees in the canopy layer (two cases out of ten). The observed discrete variables were further compared with three theoretical distributions. It was found that a Poisson, binomial, and negative binomial model best fitted the observed count data, which, based on the dispersion index, exhibited a random, regular, and clumped pattern, respectively. The frequency of plots with low species presence and complete absence of species was also revealed. Consequently, the analysis and modeling of tree counts can be of practical use for species conservation purposes.

Parameter-driven state-space model for integer-valued time series with application

ABSTRACTTime series of counts occur in many different contexts, the counts being usually of certain events or objects in specified time intervals. In this paper we introduce a model called parameter-driven state-space model to analyse integer-valued time series data. A key property of such model is that the distribution of the observed count data is independent, conditional on the latent process, although the observations are correlated marginally. Our simulation shows that the Monte Carlo Expectation Maximization (MCEM) algorithm and the particle method are useful for the parameter estimation of the proposed model. In the application to Malaysia dengue data, our model fits better when compared with several other models including that of Yang et al. (2015)

Parametric Point Estimation of the Geeta Distribution

Geeta distribution is a new discrete random variable distribution defined over all the positive integers with two parameters. This distribution belongs to the family of Location-parameter (LDPD) system and is of the form L – shaped model. Pareto and Yule distributions belong to the same family but these distributions have a disadvantage of having a single parameter which makes them not versatile to meet the needs of modern complex data sets. Geeta distribution is found to be very versatile and flexible to fit observed count data sets and can be used efficiently to model different types of sets. This paper investigates the characteristics of Geeta distribution, such as the existence of the mean, variance, moment generating function, probability generating function and that the sum of probabilities for all values of X for Geeta Distribution model is unity. It is well known that the sample mean is the estimator of a population mean from a given population of interest as a point estimator which assume a single number that is obtained by taking a random sample of a specified size from the entire population, depending on whether the population mean and variance is known or unknown These point estimators were obtained by employing the method of Moments, Maximum Likelihood (MLE) and Bayesian estimator. Further the estimators were subjected to the conditions like unbiasedness, efficiency, sufficiency and completeness which are properties of a good estimator. For the first aspect, the results of the mean, variance, moments and generating functions were achieved that proves the distribution is a probability density function (pdf). The methods of moments and the maximum likelihood and their properties were applied and yielded the desired and expected results for any given probability distribution. The best estimator obtained is best linear unbiased estimator (BLUE).

Statistical models for the number of successful cyber intrusions

We propose several generalized linear models (GLMs) to predict the number of successful cyber intrusions (or “intrusions”) into an organization’s computer network, where the rate at which intrusions occur is a function of the following observable characteristics of the organization: (i) domain name system (DNS) traffic classified by their top-level domains (TLDs); (ii) the number of network security policy violations; and (iii) a set of predictors that we collectively call the “cyber footprint” that is comprised of the number of hosts on the organization’s network, the organization’s similarity to educational institution behavior, and its number of records on scholar.google.com . In addition, we evaluate the number of intrusions to determine whether these events follow a Poisson or negative binomial (NB) probability distribution. We reveal that the NB GLM provides the best fit model for the observed count data, number of intrusions per organization, because the NB model allows the variance of the count data to exceed the mean. We also show that there are restricted and simpler NB regression models that omit selected predictors and improve the goodness-of-fit of the NB GLM for the observed data. With our model simulations, we identify certain TLDs in the DNS traffic as having a significant impact on the number of intrusions. In addition, we use the models and regression results to conclude that the number of network security policy violations is consistently predictive of the number of intrusions.

A Poisson Log-Normal Model for Constructing Gene Covariation Network Using RNA-seq Data.

Constructing expression networks using transcriptomic data is an effective approach for studying gene regulation. A popular approach for constructing such a network is based on the Gaussian graphical model (GGM), in which an edge between a pair of genes indicates that the expression levels of these two genes are conditionally dependent, given the expression levels of all other genes. However, GGMs are not appropriate for non-Gaussian data, such as those generated in RNA-seq experiments. We propose a novel statistical framework that maximizes a penalized likelihood, in which the observed count data follow a Poisson log-normal distribution. To overcome the computational challenges, we use Laplace's method to approximate the likelihood and its gradients, and apply the alternating directions method of multipliers to find the penalized maximum likelihood estimates. The proposed method is evaluated and compared with GGMs using both simulated and real RNA-seq data. The proposed method shows improved performance in detecting edges that represent covarying pairs of genes, particularly for edges connecting low-abundant genes and edges around regulatory hubs.

Applicability of Zero-Inflated Models to Fit the Torrential Rainfall Count Data with Extra Zeros in South Korea

Several natural disasters occur because of torrential rainfalls. The change in global climate most likely increases the occurrences of such downpours. Hence, it is necessary to investigate the characteristics of the torrential rainfall events in order to introduce effective measures for mitigating disasters such as urban floods and landslides. However, one of the major problems is evaluating the number of torrential rainfall events from a statistical viewpoint. If the number of torrential rainfall occurrences during a month is considered as count data, their frequency distribution could be identified using a probability distribution. Generally, the number of torrential rainfall occurrences has been analyzed using the Poisson distribution (POI) or the Generalized Poisson Distribution (GPD). However, it was reported that POI and GPD often overestimated or underestimated the observed count data when additional or fewer zeros were included. Hence, in this study, a zero-inflated model concept was applied to solve this problem existing in the conventional models. Zero-Inflated Poisson (ZIP) model, Zero-Inflated Generalized Poisson (ZIGP) model, and the Bayesian ZIGP model have often been applied to fit the count data having additional or fewer zeros. However, the applications of these models in water resource management have been very limited despite their efficiency and accuracy. The five models, namely, POI, GPD, ZIP, ZIGP, and Bayesian ZIGP, were applied to the torrential rainfall data having additional zeros obtained from two rain gauges in South Korea, and their applicability was examined in this study. In particular, the informative prior distributions evaluated via the empirical Bayes method using ten rain gauges were developed in the Bayesian ZIGP model. Finally, it was suggested to avoid using the POI and GPD models to fit the frequency of torrential rainfall data. In addition, it was concluded that the Bayesian ZIGP model used in this study provided the most accurate results for the count data having additional zeros. Moreover, it was recommended that the ZIP model could be an alternative from a practical viewpoint, as the Bayesian approach used in this study was considerably complex.

Bayesian adaptive two-stage design for determining person-time in Phase II clinical trials with Poisson data

ABSTRACTAdaptive clinical trial designs can often improve drug-study efficiency by utilizing data obtained during the course of the trial. We present a novel Bayesian two-stage adaptive design for Phase II clinical trials with Poisson-distributed outcomes that allows for person-observation-time adjustments for early termination due to either futility or efficacy. Our design is motivated by the adaptive trial from [9], which uses binomial data. Although many frequentist and Bayesian two-stage adaptive designs for count data have been proposed in the literature, many designs do not allow for person-time adjustments after the first stage. This restriction limits flexibility in the study design. However, our proposed design allows for such flexibility by basing the second-stage person-time on the first-stage observed-count data. We demonstrate the implementation of our Bayesian predictive adaptive two-stage design using a hypothetical Phase II trial of Immune Globulin (Intravenous).

Mean and Variance Modeling of Under- and Overdispersed Count Data

This article describes the R package CountsEPPM and its use in determining maximum likelihood estimates of the parameters of extended Poisson process models. These provide a Poisson process based family of flexible models that can handle both underdispersion and overdispersion in observed count data, with the negative binomial and Poisson distributions being special cases. Within CountsEPPM models with mean and variance related to covariates are constructed to match a generalized linear model formulation. Use of the package is illustrated by application to several published datasets.

Assessment of Statistical Approaches to Model Low Count Data: An Empirical Application to Youth Delinquency

Objectives: The aim of this study was to identify the risk factors associated with number of crime committed by youth (Youth Delinquency) between ages 10-17, using Ordinary Least Square (OLS), Poisson Regression model (PRM), Negative Binomial Regression model (NBRM)& Zero Inflated Negative Binomial (ZINB) with the aim to choose the most appropriate model for the observed count data. Methodology: The data in the study was collected from youth whose mothers enrolled in Philadelphia Collaborative Perinatal Project (CPP). School and delinquency record (between ages 10-17) was obtained by the Centre for studies in Criminology and Criminal Law. Literature search suggest that factors associated with child delinquency can be divided into four main factors as Individual, Family, School and Peer. Therefore we included variables in the analysis accordingly. Result: For OLS scatter plot of residuals versus estimated counts showed definite pattern of heterogeneity (non-constant variance). The likelihood-ratio (LR) test of over dispersion yields the significant p-value, which implied that the outcome variable is overdispersed. The plot of the difference between the actual probabilities and the mean predicted probabilities for each model showed that PRM has poor predictions for low counts (0-2). Conclusion: NBRM and ZINB both performed well, however fit statistics revealed that NBRM has provided more closed predication as compare ZINB.NB modeling techniques provides much more compelling and accurate results instead of basic PRM or those available through simple linear or log-linear modeling techniques

Spatial methods for event reconstruction in CLEAN

In CLEAN (Cryogenic Low Energy Astrophysics with Noble gases), a proposed neutrino and dark matter detector, background discrimination is possible if one can determine the location of an ionizing radiation event with high accuracy. Here, we develop spatial methods for event reconstruction, and study their performance in computer simulation experiments. We simulate ionizing radiation events that produce multiple scintillation photons within a spherical detection volume filled with liquid neon. We estimate the radial location of a particular ionizing radiation event based on the observed count data corresponding to that event. The count data are collected by detectors mounted at the spherical boundary of the detection volume. We neglect absorption, but account for Rayleigh scattering. To account for wavelength-shifting of the scintillation light, we assume that photons are absorbed and re-emitted at the detectors. In our study, the detectors incompletely cover the surface area of the sphere. In the first method, we estimate the radial location of the event by maximizing the approximate Poisson likelihood of the observed count data. To correct for scattering and wavelength-shifting, we adjust this estimate using a polynomial calibration model. In the second method, we predict the radial location of the event as a polynomial function of the magnitude of the centroid of the observed count data. The polynomial calibration models are constructed from calibration (training) data. In general, the Maximum Likelihood (ML) method estimate is more accurate than that of the Centroid method estimate. We estimate the expected number of photons emitted by the event by a ML method and a simple method based on the ratio of the number of detected photons and a detection probability factor.