Exponential Family Research Articles

A method to construct exponential families by representation theory

In this paper, we give a method to construct “good” exponential families systematically by representation theory. More precisely, we consider a homogeneous space G/H as a sample space and construct an exponential family invariant under the transformation group G by using a representation of G. The method generates widely used exponential families such as normal, gamma, Bernoulli, categorical, Wishart, von Mises, Fisher–Bingham and hyperboloid distributions.

An inverse Sanov theorem for exponential families

We prove the large deviation principle (LDP) for posterior distributions arising from subfamilies of full exponential families, allowing misspecification of the model. Moreover, motivated by the so-called inverse Sanov Theorem (see e.g. [11,12]), we prove the LDP for the corresponding maximum likelihood estimator, and we study the relationship between rate functions. In our setting, even in the non misspecified case, it is not true in general that the rate functions for posterior distributions and for maximum likelihood estimators are Kullback–Leibler divergences with exchanged arguments.

Multiple values-inflated time series of counts: modeling and inference based on INGARCH scheme

ABSTRACT In this study, we consider integer-valued autoregressive conditional heteroscedastic (INGARCH) time series models of counts whose conditional density follows an multiple values-inflated one parameter exponential family. We check the stationarity and ergodicity of the models and verify the consistency and asymptotic normality of the conditional maximum likelihood estimator (CMLE) under regularity conditions. Then, as an application, we consider a parameter change test using the cumulative sum (CUSUM) test based on (standardized) residuals and squares of those residuals, then subsequently derive their limiting null distributions. A simulation study is conducted focusing on the 0-1-inflated time series of counts, which affirms the validity of the proposed methods. Moreover, a real data analysis using the number of monthly drug cases in the 32nd police car beat in Pittsburgh is also provided to further strengthen the validity of our methods.

Clustering multivariate count data via Dirichlet-multinomial network fusion

Clustering of multivariate count data has widespread applications in areas such as text analysis and microbiome studies. The need to account for overdispersion generally results in a nonconvex loss function, which does not fit into the existing convex clustering framework. Moreover, prior knowledge of a network over the samples, often available from citation or similarity relationships, is not taken into account. We introduce Dirichlet-multinomial network fusion (DMNet) for clustering multivariate count data, which models the samples via Dirichlet-multinomial distributions with individual parameters and employs a weighted group L1 fusion penalty to pursue homogeneity over a prespecified network. To circumvent the nonconvexity issue, we present two exponential family approximations to the Dirichlet-multinomial distribution, which are amenable to efficient optimization and theoretical analysis. We derive an ADMM algorithm and establish nonasymptotic error bounds for the proposed methods. Our bounds involve a trade-off between the connectivity of the network and its fidelity to the true parameter. The usefulness of our methods is illustrated through simulation studies and two text clustering applications.

Discrete-valued time series based on the exponential family with the multidimensional parameter and their probabilistic and statistical analysis.

We propose herein a new parsimonious Markov model for a discrete-valued time series with conditional probability distributions of observations lying in the exponential family with the multidimensional parameter. A family of explicit consistent asymptotically normal statistical estimators is constructed for the parameters of the proposed model for increasing length of observed time series, and asymptotically effective estimator is found within this constructed family. The obtained results can be used for robust statistical analysis of discrete-valued time series,and for statistical analysis of discrete-valued spatio-temporal data and random fields.

Rényi Cross-Entropy Measures for Common Distributions and Processes with Memory

Two Rényi-type generalizations of the Shannon cross-entropy, the Rényi cross-entropy and the Natural Rényi cross-entropy, were recently used as loss functions for the improved design of deep learning generative adversarial networks. In this work, we derive the Rényi and Natural Rényi differential cross-entropy measures in closed form for a wide class of common continuous distributions belonging to the exponential family, and we tabulate the results for ease of reference. We also summarise the Rényi-type cross-entropy rates between stationary Gaussian processes and between finite-alphabet time-invariant Markov sources.

Revisiting Chernoff Information with Likelihood Ratio Exponential Families.

The Chernoff information between two probability measures is a statistical divergence measuring their deviation defined as their maximally skewed Bhattacharyya distance. Although the Chernoff information was originally introduced for bounding the Bayes error in statistical hypothesis testing, the divergence found many other applications due to its empirical robustness property found in applications ranging from information fusion to quantum information. From the viewpoint of information theory, the Chernoff information can also be interpreted as a minmax symmetrization of the Kullback–Leibler divergence. In this paper, we first revisit the Chernoff information between two densities of a measurable Lebesgue space by considering the exponential families induced by their geometric mixtures: The so-called likelihood ratio exponential families. Second, we show how to (i) solve exactly the Chernoff information between any two univariate Gaussian distributions or get a closed-form formula using symbolic computing, (ii) report a closed-form formula of the Chernoff information of centered Gaussians with scaled covariance matrices and (iii) use a fast numerical scheme to approximate the Chernoff information between any two multivariate Gaussian distributions.

Locally associated graphical models and mixed convex exponential families

The notion of multivariate total positivity has proved to be useful in finance and psychology but may be too restrictive in other applications. In this paper, we propose a concept of local association, where highly connected components in a graphical model are positively associated and study its properties. Our main motivation comes from gene expression data, where graphical models have become a popular exploratory tool. The models are instances of what we term mixed convex exponential families and we show that a mixed dual likelihood estimator has simple exact properties for such families as well as asymptotic properties similar to the maximum likelihood estimator. We further relax the positivity assumption by penalizing negative partial correlations in what we term the positive graphical lasso. Finally, we develop a GOLAZO algorithm based on block-coordinate descent that applies to a number of optimization procedures that arise in the context of graphical models, including the estimation problems described above. We derive results on existence of the optimum for such problems.

Current and Future γ-Ray Searches for Dark Matter Annihilation Beyond the Unitarity Limit

For decades, searches for electroweak-scale dark matter (DM) have been performed without a definitive detection. This lack of success may hint that DM searches have focused on the wrong mass range. A proposed candidate beyond the canonical parameter space is ultraheavy DM (UHDM). In this work, we consider indirect UHDM annihilation searches for masses between 30 TeV and 30 PeV—extending well beyond the unitarity limit at ∼100 TeV—and discuss the basic requirements for DM models in this regime. We explore the feasibility of detecting the annihilation signature, and the expected reach for UHDM with current and future very-high-energy (VHE; >100 GeV) γ-ray observatories. Specifically, we focus on three reference instruments: two Imaging Atmospheric Cherenkov Telescope arrays, modeled on VERITAS and CTA-North, and one extended air shower array, motivated by HAWC. With reasonable assumptions on the instrument response functions and background rate, we find a set of UHDM parameters (mass and cross section) for which a γ-ray signature can be detected by the aforementioned observatories. We further compute the expected upper limits for each experiment. With realistic exposure times, the three instruments can probe DM across a wide mass range. At the lower end, it can still have a point-like cross section, while at higher masses the DM could have a geometric cross section, indicative of compositeness.

Comparison of Some Methods for Estimating the Survival Function and Failure Rate for the Exponentiated Expanded Power Function Distribution


 We have presented the distribution of the exponentiated expanded power function (EEPF) with four parameters, where this distribution was created by the exponentiated expanded method created by the scientist Gupta to expand the exponential distribution by adding a new shape parameter to the cumulative function of the distribution, resulting in a new distribution, and this method is characterized by obtaining a distribution that belongs for the exponential family. We also obtained a function of survival rate and failure rate for this distribution, where some mathematical properties were derived, then we used the method of maximum likelihood (ML) and method least squares developed (LSD) to estimate the parameters and because of the nonlinear relationship between the parameters, numerical algorithms were used to find the estimates of the two methods. They are Newton-Raphson (NR) and Nelder mead (NM) algorithms to improve the estimators, and a Monte Carlo simulation experiment was conducted to evaluate the performance of the two algorithms' estimates, and the average integrated error criterion (IMSE) was used to compare the survival function estimates and the failure rate. The results showed the efficiency of the maximum likelihood method estimates and least squares developed using the two algorithms (NR, NM) where their results were close, and this shows the new distribution efficiency (EEPF) for modeling survival data.

MLE of Jointly Constrained Mean-Covariance of Multivariate Normal Distributions

Estimating the unconstrained mean and covariance matrix is a popular topic in statistics. However, estimation of the parameters of $N_p(\mu,\Sigma)$ under joint constraints such as $\Sigma\mu = \mu$ has not received much attention. It can be viewed as a multivariate counterpart of the classical estimation problem in the $N(\theta,\theta^2)$ distribution. In addition to the usual inference challenges under such non-linear constraints among the parameters (curved exponential family), one has to deal with the basic requirements of symmetry and positive definiteness when estimating a covariance matrix. We derive the non-linear likelihood equations for the constrained maximum likelihood estimator of $(\mu,\Sigma)$ and solve them using iterative methods. Generally, the MLE of covariance matrices computed using iterative methods do not satisfy the constraints. We propose a novel algorithm to modify such (infeasible) estimators or any other (reasonable) estimator. The key step is to re-align the mean vector along the eigenvectors of the covariance matrix using the idea of regression. In using the Lagrangian function for constrained MLE (Aitchison et al. 1958), the Lagrange multiplier entangles with the parameters of interest and presents another computational challenge. We handle this by either iterative or explicit calculation of the Lagrange multiplier. The existence and nature of location of the constrained MLE are explored within a data-dependent convex set using recent results from random matrix theory. A simulation study illustrates our methodology and shows that the modified estimators perform better than the initial estimators from the iterative methods.

THE ZERO, ONE AND ZERO-AND-ONE-INFLATED NEW UNIT-LINDLEY DISTRIBUTIONS

In this paper, we propose the zero, one and zero-and-one-inflated New unit-Lindley distributions as natural extensions of the New unit-Lindley distribution to model continuous responses measured at the following intervals [0, 1), (0, 1] and [0, 1]. They were constructed based on convex combinations between the New unit-Lindley distribution and the distributions degenerate at zero, one, and Bernoulli distribution. They also have a number of interesting properties, such as being members of the exponential family. Besides, they have closed forms for the cumulative distribution functions, quantiles, and moments. Inferential aspects and regression structures arediscussed in this work as well as a Monte Carlo simulation study to evaluate theperformance of the regressors. Finally, we bring an application to real data on the suicide rate in the year 2016.

Change-plane analysis for subgroup detection with a continuous treatment.

Detecting and characterizing subgroups with differential effects of a binary treatment has been widely studied and led to improvements in patient outcomes and population risk management. Under the setting of a continuous treatment, however, such investigations remain scarce. We propose a semiparametric change-plane model and consequently a doubly robust test statistic for assessing the existence of two subgroups with differential treatment effects under a continuous treatment. The proposed testing procedure is valid when either the baseline function for the covariate effects or the generalized propensity score function for the continuous treatment is correctly specified. The asymptotic distributions of the test statistic under the null and local alternative hypotheses are established. When the null hypothesis of no subgroup is rejected, the change-plane parameters that define the subgroups can be estimated. This paper provides a unified framework of the change-plane method to handle various types of outcomes, including the exponential family of distributions and time-to-event outcomes. Additional extensions with nonparametric estimation approaches are also provided. We evaluate the performance of our proposed methods through extensive simulation studies under various scenarios. An application to the Health Effects of Arsenic Longitudinal Study with a continuous environmental exposure of arsenic ispresented.

Assessing, Testing and Estimating the Amount of Fine-Tuning by Means of Active Information.

A general framework is introduced to estimate how much external information has been infused into a search algorithm, the so-called active information. This is rephrased as a test of fine-tuning, where tuning corresponds to the amount of pre-specified knowledge that the algorithm makes use of in order to reach a certain target. A function f quantifies specificity for each possible outcome x of a search, so that the target of the algorithm is a set of highly specified states, whereas fine-tuning occurs if it is much more likely for the algorithm to reach the target as intended than by chance. The distribution of a random outcome X of the algorithm involves a parameter that quantifies how much background information has been infused. A simple choice of this parameter is to use in order to exponentially tilt the distribution of the outcome of the search algorithm under the null distribution of no tuning, so that an exponential family of distributions is obtained. Such algorithms are obtained by iterating a Metropolis–Hastings type of Markov chain, which makes it possible to compute their active information under the equilibrium and non-equilibrium of the Markov chain, with or without stopping when the targeted set of fine-tuned states has been reached. Other choices of tuning parameters are discussed as well. Nonparametric and parametric estimators of active information and tests of fine-tuning are developed when repeated and independent outcomes of the algorithm are available. The theory is illustrated with examples from cosmology, student learning, reinforcement learning, a Moran type model of population genetics, and evolutionary programming.

The Generalized Exponential Extended Exponentiated Family of Distributions: Theory, Properties, and Applications

Here, we propose a new generalized exponential extended exponentiated (NGE3) family of distributions. Some statistical properties of proposed family are gained. The most extreme probability method, maximum likelihood (ML), is utilized for parameter estimation. We explore an exceptional model called NGE3-Exponential (NGE3E). NGE3E is estimated with ML, and the performance of estimators is demonstrated by utilizing a simulation. Moreover, two applications are given to show the significance and adaptability of the proposed model in comparison to some generalized models (GMs).

Multilayer Exponential Family Factor models for integrative analysis and learning disease progression.

Current diagnosis of neurological disorders often relies on late-stage clinical symptoms, which poses barriers to developing effective interventions at the premanifest stage. Recent research suggests that biomarkers and subtle changes in clinical markers may occur in a time-ordered fashion and can be used as indicators of early disease. In this article, we tackle the challenges to leverage multidomain markers to learn early disease progression of neurological disorders. We propose to integrate heterogeneous types of measures from multiple domains (e.g., discrete clinical symptoms, ordinal cognitive markers, continuous neuroimaging, and blood biomarkers) using a hierarchical Multilayer Exponential Family Factor (MEFF) model, where the observations follow exponential family distributions with lower-dimensional latent factors. The latent factors are decomposed into shared factors across multiple domains and domain-specific factors, where the shared factors provide robust information to perform extensive phenotyping and partition patients into clinically meaningful and biologically homogeneous subgroups. Domain-specific factors capture remaining unique variations for each domain. The MEFF model also captures nonlinear trajectory of disease progression and orders critical events of neurodegeneration measured by each marker. To overcome computational challenges, we fit our model by approximate inference techniques for large-scale data. We apply the developed method to Parkinson's Progression Markers Initiative data to integrate biological, clinical, and cognitive markers arising from heterogeneous distributions. The model learns lower-dimensional representations of Parkinson's disease (PD) and the temporal ordering of the neurodegeneration of PD.

Nonparametric Estimation of Repeated Densities with Heterogeneous Sample Sizes

We consider the estimation of densities in multiple subpopulations, where the available sample size in each subpopulation greatly varies. This problem occurs in epidemiology, for example, where different diseases may share similar pathogenic mechanism but differ in their prevalence. Without specifying a parametric form, our proposed method pools information from the population and estimate the density in each subpopulation in a data-driven fashion. Drawing from functional data analysis, low-dimensional approximating density families in the form of exponential families are constructed from the principal modes of variation in the log-densities. Subpopulation densities are subsequently fitted in the approximating families based on likelihood principles and shrinkage. The approximating families increase in their flexibility as the number of components increases and can approximate arbitrary infinite-dimensional densities. We also derive convergence results of the density estimates formed with discrete observations. The proposed methods are shown to be interpretable and efficient in simulation studies as well as applications to electronic medical record and rainfall data. Supplementary materials for this article are available online.

Regression analysis for exponential family data in a finite population setup using two-stage cluster sample

Regression analysis for exponential family data in a finite population setup using two-stage cluster sample

Large-Scale Generalized Linear Models for Longitudinal Data with Grouped Patterns of Unobserved Heterogeneity

ABSTRACT This article provides methods for flexibly capturing unobservable heterogeneity from longitudinal data in the context of an exponential family of distributions. The group memberships of individual units are left unspecified, and their heterogeneity is influenced by group-specific unobservable factor structures. The model includes, as special cases, probit, logit, and Poisson regressions with interactive fixed effects along with unknown group membership. We discuss a computationally efficient estimation method and derive the corresponding asymptotic theory. Uniform consistency of the estimated group membership is established. To test heterogeneous regression coefficients within groups, we propose a Swamy-type test that allows for unobserved heterogeneity. We apply the proposed method to the study of market structure of the taxi industry in New York City. Our method unveils interesting and important insights from large-scale longitudinal data that consist of over 450 million data points.

Optimal equivalence testing in exponential families

Optimal equivalence testing in exponential families