Poisson Data Research Articles

A non-homogeneous count process: marginalizing a Poisson driven Cox process

The paper considers a Cox process where the stochastic intensity function for the Poisson data model is itself a non-homogeneous Poisson process. We show that it is possible to obtain the marginal data process, namely a non-homogeneous count process exhibiting over-dispersion. The model generates intensity functions which are non-decreasing. This is not a restriction in practice since observed data can always be transformed which guarantees a non-decreasing intensity function. We are able, for a specific choice of Cox process, to mathematically marginalize out the latent process leaving with a directly available likelihood function. This makes inference much simpler compared to algorithms which retain the latent process.

Prediction Intervals for Overdispersed Poisson Data and Their Application in Medical and Pre-Clinical Quality Control.

In pre-clinical and medical quality control, it is of interest to assess the stability of the process under monitoring or to validate a current observation using historical control data. Classically, this is done by the application of historical control limits (HCL) graphically displayed in control charts. In many applications, HCL are applied to count data, for example, the number of revertant colonies (Ames assay) or the number of relapses per multiple sclerosis patient. Count data may be overdispersed, can be heavily right-skewed and clusters may differ in cluster size or other baseline quantities (e.g., number of petri dishes per control group or different length of monitoring times per patient). Based on the quasi-Poisson assumption or the negative-binomial distribution, we propose prediction intervals for overdispersed count data to be used as HCL. Variable baseline quantities are accounted for by offsets. Furthermore, we provide a bootstrap calibration algorithm that accounts for the skewed distribution and achieves equal tail probabilities. Comprehensive Monte-Carlo simulations assessing the coverage probabilities of eight different methods for HCL calculation reveal, that the bootstrap calibrated prediction intervals control the type-1-error best. Heuristics traditionally used in control charts (e.g., the limits in Shewhart c- or u-charts or the mean ± 2 SD) fail to control a pre-specified coverage probability. The application of HCL is demonstrated based on data from the Ames assay and for numbers of relapses of multiple sclerosis patients. The proposed prediction intervals and the algorithm for bootstrap calibration are publicly available via the R package predint.

Application of Poisson state-space models and shared frailty models for multistage processes surveillance

ABSTRACT There are many applications in actual multistage processes with Poisson data. Control charts are an essential tool for monitoring quality variables in multistage processes. On the other hand, it is not straightforward to adjust a quality variable for the effect of all influential covariates, and ignoring latent variables introduces unobserved heterogeneity which diminishes the detection power of a monitoring scheme. Hence, the present paper proposes two monitoring schemes namely, the GEWMA control chart based on a Poisson state-space model and the modified EWMA control chart based on a shared frailty model in which latent variables are considered in Phase II. In this regard, expectation – maximization and Bayesian algorithms are used to estimate the parameters of the proposed models. A comparison between the two proposed control charts demonstrates that the GEWMA control chart based on a Poisson state-space model performs better than the modified EWMA control chart using the frailty model. Then, comprehensive simulation experiments are conducted to evaluate the performance of the proposed GEWMA. The results show that it performs satisfactorily under different shifts in single and multiple stages. Also, this control chart helps well to identify the out-of-control stage. Finally, a real case in health-care related processes is used to appraise the performance of the proposed monitoring scheme in practice.

Thompson Sampling-Based Partially Observable Online Change Detection for Exponential Families

This paper proposes a holistic sequential change detection framework for partially observable high-dimensional data streams with exponential-family distributions. The framework first proposes a general composite decomposition for exponential-family distributed data by projecting its natural parameter onto normal bases and abnormal bases, which enables efficient inference for sparse changes. Then, the inference results are used for detection scheme construction, and different types of test statistics can be compacted in our framework. Last, by further designing the test statistic as the reward function in the combinatorial multi-armed bandit problem, a Thompson sampling-based sensor allocation strategy is constructed to select the most anomalous variables. Theoretical properties of the detection framework are discussed. Finally, examples of Gaussian, Poisson, and binomial distributed data streams are given in numerical studies and case studies to evaluate the performance of our proposed method. History: Bianca Maria Colosimo served as the senior editor for this article. Funding: C. Zhang is partially supported by the NSFC [Grants 71932006 and 72271138], the BNSF [Grant 9222014], and the ASFC [Grant 2020Z063058001]. H. Yan is partially supported by NIH [R21 AI157618] and NSF [CMMI 2316654]. Data Ethics & Reproducibility Note: No data ethics considerations are foreseen related to this paper. The code capsule is available on Code Ocean at https://codeocean.com/capsule/8794940/tree/v1 and in the e-Companion to this article (available at https://doi.org/10.1287/ijds.2022.00011 ).

Iterative estimating equations for disease mapping with spatial zero‐inflated Poisson data

Spatial epidemiology often involves the analysis of spatial count data with an unusually high proportion of zero observations. While Bayesian hierarchical models perform very well for zero‐inflated data in many situations, a smooth response surface is usually required for the Bayesian methods to converge. However, for infectious disease data with excessive zeros, a Wombling issue with large spatial variation could make the Bayesian methods infeasible. To address this issue, we develop estimating equations associated with disease mapping by including over‐dispersion and spatial noises in a spatial zero‐inflated Poisson model. Asymptotic properties are derived for the parameter estimates. Simulations and data analysis are used to assess and illustrate the proposed method.

A Bayesian proportional hazards mixture cure model for interval-censored data.

The proportional hazards mixture cure model is a popular analysis method for survival data where a subgroup of patients are cured. When the data are interval-censored, the estimation of this model is challenging due to its complex data structure. In this article, we propose a computationally efficient semiparametric Bayesian approach, facilitated by spline approximation and Poisson data augmentation, for model estimation and inference with interval-censored data and a cure rate. The spline approximation and Poisson data augmentation greatly simplify the MCMC algorithm and enhance the convergence of the MCMC chains. The empirical properties of the proposed method are examined through extensive simulation studies and also compared with the R package "GORCure". The use of the proposed method is illustrated through analyzing a data set from the Aerobics Center Longitudinal Study.

Experimental Design Networks: A Paradigm for Serving Heterogeneous Learners Under Networking Constraints

Significant advances in edge computing capabilities enable learning to occur at geographically diverse locations. In general, the training data needed in those learning tasks are not only heterogeneous but also not fully generated locally. In this paper, we propose an experimental design network paradigm, wherein learner nodes train possibly different Bayesian linear regression models via consuming data streams generated by data source nodes over a network. We formulate this problem as a social welfare optimization problem in which the global objective is defined as the sum of experimental design objectives of individual learners, and the decision variables are the data transmission strategies subject to network constraints. We first show that, assuming Poisson data streams in steady state, the global objective is a continuous DR-submodular function. We then propose a Frank-Wolfe type algorithm that outputs a solution within a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1-1/e$</tex-math> </inline-formula> factor from the optimal. Our algorithm contains a novel gradient estimation component which is carefully designed based on Poisson tail bounds and sampling. Finally, we complement our theoretical findings through extensive experiments. Our numerical evaluation shows that the proposed algorithm outperforms several baseline algorithms both in maximizing the global objective and in the quality of the trained models.

A self-starting non-restarting CUSUM chart for monitoring Poisson count data with time-varying sample sizes

Many studies have focused on monitoring the incidence rate of Poisson count data with time-varying sample sizes, with control charts being widely used. However, almost all of the existing reports mainly focus on solving the problem of time-varying sample sizes. Rare attention is paid to the real-life problems where the data cannot restart. Besides, these studies aim at Phase II with the in-control parameters known, which is unrealistic in real applications. As far as we know, there is no research on the non-restarting and unknown in-control parameters control charts, but it is essential and critical for extending applications of control charts. To address these limitations, this study offered two self-starting non-restarting cumulative sum (CUSUM) control charts based on the modified test to monitor the incidence rate of non-restarting Poisson data with time-varying sample sizes. The methods can be applied to other control charts and take into account both Phase I and Phase II without assumptions of time-varying sample sizes. Moreover, the monitor can be carried out with small Phase I samples. Simulation results show that the second control chart proposed in this study (SSNR-CUSUM-P2) is optimal compared to the common non-restarting control chart (E) and the first control chart proposed in this study (SSNR-CUSUM-P1). The SSNR-CUSUM-P2 chart has the highest similarity to the control chart with known parameters (T) in both post-outbreak false alarm rate and outbreak time point set. Finally, a dataset on German measles patients and a dataset on US employment are used to illustrate the application of the methods.

Linear regression for Poisson count data: a new semi-analytical method with applications to COVID-19 events

This study presents the application of a new semi-analytical method of linear regression for Poisson count data to COVID-19 events. The regression is based on the maximum-likelihood solution for the best-fit parameters presented in an earlier publication, and this study introduces a simple analytical solution for the covariance matrix that completes the problem of linear regression with Poisson data for one independent variable. The analytical nature of both parameter estimates and their covariance matrix is made possible by a convenient factorization of the linear model proposed by J. Scargle. The method makes use of the asymptotic properties of the Fisher information matrix, whose inverse provides the covariance matrix. The combination of simple analytical methods to obtain both the maximum-likelihood estimates of the parameters and their covariance matrix constitutes a new and convenient method for the linear regression of Poisson-distributed count data, which are of common occurrence across a variety of fields. A comparison between this maximum-likelihood linear regression method for Poisson data and two alternative methods often used for the regression of count data—the ordinary least–square regression and the χ2 regression—is provided with the application of these methods to the analysis of recent COVID-19 count data. The study also discusses the relative advantages and disadvantages among these methods for the linear regression of Poisson count data.

Tensor Stein-rules in a generalized tensor regression model

In this paper, we consider an estimation problem in a generalized tensor regression model with multi-mode covariates. We generalize the main results in recent literature in four ways. First, we weaken assumptions underlying the main results of the previous works. In particular, the dependence structure of the error and covariates are as weak as an L2− mixingale array, and the error term does not need to be uncorrelated with regressors. Second, we consider a more general constraint than the one considered in previous works. Third, we establish the asymptotic properties of the tensor estimators. Specifically, we derive the joint asymptotic distribution of the unrestricted tensor estimator (UE) and restricted tensor estimator (RE). Fourth, we propose a class of shrinkage-type estimators in the context of tensor regression, and under a general loss function, we derive sufficient conditions for which the shrinkage estimators dominate the UE. In addition to these interesting contributions, we derive a kind of functional central limit theorem for mixingale vector-valued and we establish some identities which are useful in studying the risk dominance of shrinkage-type tensor estimators. Finally, to illustrate the application of the proposed methods, we corroborate the results by some simulation studies of binary, Normal and Poisson data and we analyze a multi-relational network and neuro-imaging datasets.

The person-time ratio distribution for the exact monitoring of adverse events: Historical vs surveillance Poisson data.

In the postmarket drug and vaccine safety surveillance, when the number of adverse events follows a Poisson distribution, the ratio between the exposed and the unexposed person-time information is the random variable that governs the decision rule about the safety of the drug or vaccine. The probability distribution function of such a ratio is derived in this paper. Exact point and interval estimators for the relative risk are discussed as well as statistical hypothesis testing. To the best of our knowledge, this is the first paper that provides an unbiased estimator for the relative risk based on the person-time ratio. The applicability of this new distribution is illustrated through a real data analysis aimed to detect increased risk of occurrence of Myocarditis/Pericarditis following mRNA COVID-19 vaccination in Manitoba, Canada.

Exact sample size determination for a single Poisson random sample.

Classical power analysis for sample size determination is typically performed in clinical trials. A "hybrid" classical Bayesian or a "fully Bayesian" approach can be alternatively used in order to add flexibility to the design assumptions needed at the planning stage of the study and to explicitly incorporate prior information in the procedure. In this paper, we exploit and compare these approaches to obtain the optimal sample size of a single-arm trial based on Poisson data. We adopt exact methods to establish the rejection of the null hypothesis within a frequentist or a Bayesian perspective and suggest the use of a conservative criterion for sample size determination that accounts for the not strictly monotonic behavior of the power function in the presence of discrete data. A Shiny web app in R has been developed to provide a user-friendly interface to easily compute the optimal sample size according to the proposed criteria and to assure the reproducibility of theresults.

Systematic errors in the maximum-likelihood regression of Poisson count data: introducing the overdispersed χ2 distribution

ABSTRACT This paper presents a new method to estimate systematic errors in the maximum-likelihood regression of count data. The method is applicable in particular to X-ray spectra in situations where the Poisson log-likelihood, or the Cash goodness-of-fit statistic, indicate a poor fit that is attributable to overdispersion of the data. Overdispersion in Poisson data is treated as an intrinsic model variance that can be estimated from the best-fit model, using the maximum-likelihood Cmin statistic. The paper also studies the effects of such systematic errors on the ΔC likelihood-ratio statistic, which can be used to test for the presence of a nested model component in the regression of Poisson count data. The paper introduces an overdispersed χ2 distribution that results from the convolution of a χ2 distribution that models the usual ΔC statistic, and a zero-mean Gaussian that models the overdispersion in the data. This is proposed as the distribution of choice for the ΔC statistic in the presence of systematic errors. The methods presented in this paper are applied to XMM–Newton data of the quasar 1ES 1553+113 that were used to detect absorption lines from an intervening warm-hot intergalactic medium (WHIM). This case study illustrates how systematic errors can be estimated from the data, and their effect on the detection of a nested component, such as an absorption line, with the ΔC statistic.

Towards Circular Economy: Unveiling Heterogeneous Effects of Government Policy Stringency, Environmentally Related Innovation, and Human Capital within OECD Countries

With reference to the existing literature, this paper investigates the heterogenous effect on the attainment of circular economy by government policies in the form of government stringency and government financial support, environmentally related innovations, and human capital. The study was carried out in 26 countries of the OECD from 2010–2019 using the Poisson pseudo-maximum likelihood (PPML) model and data from Eurostat and OECD datasets. Indicators for the independent variables were non-market-based stringency, market-based subsidy, gross domestic expenditure on R&amp;D by source of funds, R&amp;D expenditure intramural, national expenditure on environmental protection environmental protection, environmentally related patents with co-inventors, and employees involved in education and training. The results revealed that a significant effect of government stringency in the form of non-market-based stringency, environmental innovation, government financing on R&amp;D, and national expenditure on environmental protection have significant impact on the attainment of circular economy within OECD countries. Surprisingly, there was no significant effect of the market-based subsidy on domestic material consumption (dmc). We conclude that a blend of government policies is the effective means of achieving a circular economy.

GECAM Localization of High-energy Transients and the Systematic Error

The Gravitational Wave High-energy Electromagnetic Counterpart All-sky Monitor (GECAM) is a pair of microsatellites (i.e., GECAM-A and GECAM-B) dedicated to monitoring gamma-ray transients including the high-energy electromagnetic counterparts of gravitational waves, such as gamma-ray bursts, soft gamma-ray repeaters, solar flares, and terrestrial gamma-ray flashes. Since launch in 2020 December, GECAM-B has detected hundreds of astronomical and terrestrial events. For these bursts, localization is the key for burst identification and classification as well as follow-up observations in multiple wavelengths. Here, we propose a Bayesian localization method with Poisson data with Gaussian background profile likelihood to localize GECAM bursts based on the distribution of burst counts in detectors with different orientations. We demonstrate that this method can work well for all kinds of bursts, especially extremely short ones. In addition, we propose a new method to estimate the systematic error of localization based on a confidence level test, which can overcome some problems of the existing method in the literature. We validate this method by Monte Carlo simulations, and then apply it to a burst sample with accurate location and find that the mean value of the systematic error of GECAM-B localization is ∼2.°5. By considering this systematic error, we can obtain a reliable localization probability map for GECAM bursts. Our methods can be applied to other gamma-ray monitors.

Assessing Area under the Curve as an Alternative to Latent Growth Curve Modeling for Repeated Measures Zero-Inflated Poisson Data: A Simulation Study

Researchers interested in the assessment of substance use trajectories, and predictors of change, have several data analysis options. These include, among others, generalized estimating equations and latent growth curve modeling. One difficulty in the assessment of substance use, however, is the nature of the variables studied. Although counting instances of use (e.g., the number of cigarettes smoked per day) would seem to be the best option, such data present difficulties in that the distribution of these variables is not likely normal. Count variables often follow a Poisson distribution, and when dealing with substance use in the general population, there is a preponderance of zeros (representing not using). As such, substance use counts may approximate a zero-inflated Poisson distribution. Unfortunately, analyses with zero-inflated Poisson random variables are not easily accommodated in many types of software and may be beyond access to most researchers. As such, an easier method would benefit researchers interested in assessing substance use change. The purpose of this study is to assess the area under the curve as an option when dealing with repeated measures data and contrast it to one popular method of longitudinal data analysis, latent growth curve modeling. Using a Monte Carlo simulation study with varying sample sizes, we found that the area under the curve performed well with different sample sizes and compared favorably to the performance of latent growth curve modeling, particularly when dealing with smaller sample sizes. The area under the curve may be a simpler alternative for researchers, especially when dealing with smaller sample sizes.

Adaptive Gaussian Markov random field spatiotemporal models for infectious disease mapping and forecasting

Recent disease mapping literature presents adaptively parameterized spatiotemporal (ST) autoregressive (AR) or conditional autoregressive (CAR) models for Bayesian prediction of COVID-19 infection risks. These models were motivated to capture complex spatiotemporal dynamics and heterogeneities of infection risks. In the present paper, we synthesize, generalize, and unify the ST AR and CAR model constructions for models augmented by adaptive Gaussian Markov random fields, with an emphasis on disease forecasting. A general convolution construction is presented, with illustrative models motivated to (i) characterize local risk dependencies and influences over both spatial and temporal dimensions, (ii) model risk heterogeneities and discontinuities, and (iii) predict and forecast areal-level disease risks and occurrences. The broadened constructions allow rich options of intuitive parameterization for disease mapping and spatial regression. Illustrative parameterizations are presented for Bayesian hierarchical models of Poisson, zero-inflated Poisson, and Bernoulli data models, respectively. They are also discussed in the context of quantifying time-varying or time-invariant effects of (omitted) covariates, with application to prediction and forecasting areal-level COVID-19 infection occurrences and probabilities of zero-infection. The model constructions presented herein have much wider scope in offering a flexible framework for modelling complex spatiotemporal data and for estimation, learning, and forecasting purposes.

Whiteness-based parameter selection for Poisson data in variational image processing

We propose a novel parameter selection strategy for variational imaging problems under Poisson noise corruption. The selection of a suitable value of the regularization parameter, which is crucial for achieving high quality reconstructions, is known to be a particularly hard task in low photon-counting regimes. In this work, we extend the so-called residual whiteness principle originally designed for additive white noise to Poisson data. The proposed strategy relies on exploiting the whiteness property of a suitably standardized Poisson noise process. After deriving the theoretical properties underlying our proposal, we solve the target optimization problem by the alternating direction method of multipliers, in its standard two-blocks version or in a semi-linearized version depending on the imaging problem. Our strategy is extensively tested on image restoration and computed tomography reconstruction problems, and compared to the state-of-the-art discrepancy principle for Poisson noise proposed by Zanella at al. as well as to a nearly exact version of it recently proposed by the authors.

Fast and memory-efficient reconstruction of sparse Poisson data in listmode with non-smooth priors with application to time-of-flight PET

Objective. Complete time of flight (TOF) sinograms of state-of-the-art TOF PET scanners have a large memory footprint. Currently, they contain ∼4 · 109 data bins which amount to ∼17 GB in 32 bit floating point precision. Moreover, their size will continue to increase with advances in the achievable detector TOF resolution and increases in the axial field of view. Using iterative algorithms to reconstruct such enormous TOF sinograms becomes increasingly challenging due to the memory requirements and the computation time needed to evaluate the forward model for every data bin. This is especially true for more advanced optimization algorithms such as the stochastic primal-dual hybrid gradient (SPDHG) algorithm which allows for the use of non-smooth priors for regularization using subsets with guaranteed convergence. SPDHG requires the storage of additional sinograms in memory, which severely limits its application to data sets from state-of-the-art TOF PET systems using conventional computing hardware. Approach. Motivated by the generally sparse nature of the TOF sinograms, we propose and analyze a new listmode (LM) extension of the SPDHG algorithm for image reconstruction of sparse data following a Poisson distribution. The new algorithm is evaluated based on realistic 2D and 3D simulationsn, and a real data set acquired on a state-of-the-art TOF PET/CT system. The performance of the newly proposed LM SPDHG algorithm is compared against the conventional sinogram SPDHG and the listmode EM-TV algorithm. Main results. We show that the speed of convergence of the proposed LM-SPDHG is equivalent the original SPDHG operating on binned data (TOF sinograms). However, we find that for a TOF PET system with 400 ps TOF resolution and 25 cm axial FOV, the proposed LM-SPDHG reduces the required memory from approximately 56 to 0.7 GB for a short dynamic frame with 107 prompt coincidences and to 12.4 GB for a long static acquisition with 5·108 prompt coincidences. Significance. In contrast to SPDHG, the reduced memory requirements of LM-SPDHG enables a pure GPU implementation on state-of-the-art GPUs—avoiding memory transfers between host and GPU—which will substantially accelerate reconstruction times. This in turn will allow the application of LM-SPDHG in routine clinical practice where short reconstruction times are crucial.

DeepAdversaries: examining the robustness of deep learning models for galaxy morphology classification

With increased adoption of supervised deep learning methods for work with cosmological survey data, the assessment of data perturbation effects (that can naturally occur in the data processing and analysis pipelines) and the development of methods that increase model robustness are increasingly important. In the context of morphological classification of galaxies, we study the effects of perturbations in imaging data. In particular, we examine the consequences of using neural networks when training on baseline data and testing on perturbed data. We consider perturbations associated with two primary sources: (a) increased observational noise as represented by higher levels of Poisson noise and (b) data processing noise incurred by steps such as image compression or telescope errors as represented by one-pixel adversarial attacks. We also test the efficacy of domain adaptation techniques in mitigating the perturbation-driven errors. We use classification accuracy, latent space visualizations, and latent space distance to assess model robustness in the face of these perturbations. For deep learning models without domain adaptation, we find that processing pixel-level errors easily flip the classification into an incorrect class and that higher observational noise makes the model trained on low-noise data unable to classify galaxy morphologies. On the other hand, we show that training with domain adaptation improves model robustness and mitigates the effects of these perturbations, improving the classification accuracy up to 23% on data with higher observational noise. Domain adaptation also increases up to a factor of the latent space distance between the baseline and the incorrectly classified one-pixel perturbed image, making the model more robust to inadvertent perturbations. Successful development and implementation of methods that increase model robustness in astronomical survey pipelines will help pave the way for many more uses of deep learning for astronomy.