Exponential Family Of Distributions Research Articles

Type II general inverse exponential family of distributions

In this paper, we introduce a new family of distributions based on the T-X transformation, the inverse exponential distribution, the odds function and the Lehmann type II distribution. We investigate its general mathematical properties, including moments, moment generating function, quantile function, entropies and order statistics. A statistical model is constructed from a special case of the family using the Bur III distribution (also known as exponentiated Lomax distribution) as baseline. The estimation of the parameters are performed by the maximum likelihood method and the least square method. Finally, we illustrate its importance by means of two applications to real life data sets.

Estimators and D-optimal experimental designs for mixtures of binary responses

Models of mixture distributions are of great interest in many areas where several populations are mixed up. If the response is binary there is a mixture of Bernoulli distributions, which has practical applications for the classification of texts and images, biochemistry, genetics, robotics, computer science or pattern recognition. A finite mixture of probability distributions includes a set of parameters such as the proportion of the mixture and the parameters of each distribution. This simple case allows us to make some explicit computations of the estimators as well as working on the EM algorithm doing some comparisons. Some of these parameters may depend on one or more covariates through some specific model. Mixtures of exponential family distributions, except in very particular cases, are no longer within the exponential family and this means, among other things, that the expectation for computing the Information Matrix must, in most cases, be approximated. One of the main contributions of this article is its handling of this. Linear and logistic models are considered either for the proportion of one of the two populations (clusters), or for the parameters of the Bernoulli distributions. For each of these cases the analytic expression of the information matrix is calculated and optimal designs determined.

Smooth, identifiable supermodels of discrete DAG models with latent variables

We provide a parameterization of the discrete nested Markov model, which is a supermodel that approximates DAG models (Bayesian network models) with latent variables. Such models are widely used in causal inference and machine learning. We explicitly evaluate their dimension, show that they are curved exponential families of distributions, and fit them to data. The parameterization avoids the irregularities and unidentifiability of latent variable models. The parameters used are all fully identifiable and causally-interpretable quantities.

Boosted negative sampling by quadratically constrained entropy maximization

Learning probability densities for natural language representations is a difficult problem because language is inherently sparse and high-dimensional. Negative sampling is a popular and effective way to avoid intractable maximum likelihood problems, but it requires correct specification of the sampling distribution. Previous state of the art methods rely on heuristic distributions that appear to do well in practice. In this work, we define conditions for optimal sampling distributions and demonstrate how to approximate them using Quadratically Constrained Entropy Maximization(QCEM). Our analysis shows that state of the art heuristics are restrictive approximations to our proposed framework. To demonstrate the merits of our formulation, we apply QCEM to matching synthetic exponential family distributions and to finding high dimensional word embedding vectors for English. We are able to achieve faster inference on synthetic experiments and improve the correlation on semantic similarity evaluations on the Rare Words dataset by 4.8%.

Monte Carlo Methods for Insurance Risk Computation

In this paper we consider the problem of computing tail probabilities of the distribution of a random sum of positive random variables. We assume that the individual claim variables follow a reproducible natural exponential family (NEF) distribution, and that the random number has a NEF counting distribution with a cubic variance function. This specific modeling is supported by data of the aggregated claim distribution of an insurance company. Large tail probabilities are important as they reflect the risk of large losses, however, analytic or numerical expressions are not available. We propose several simulation algorithms which are based on an asymptotic analysis of the distribution of the counting variable and on the reproducibility property of the claim distribution. The aggregated sum is simulated efficiently by importance sampling using an exponential change of measure. We conclude by numerical experiments of these algorithms, based on real car insurance claim data.

Fine Asymptotics for Universal One-to-One Compression of Parametric Sources

Universal source coding at short blocklengths is considered for an i.i.d. exponential family of distributions. The Type Size code has previously been shown to be optimal up to the third-order rate for universal compression of all memoryless sources over finite alphabets. The Type Size code assigns sequences ordered based on their type class sizes to binary strings ordered lexicographically. To generalize this type class approach for parametric sources, a natural scheme is to define two sequences to be in the same type class if and only if they are equiprobable under any model in the parametric class. This natural approach, however, is shown to be suboptimal. A variation of the Type Size code is introduced, where type classes are defined based on neighborhoods of minimal sufficient statistics. The asymptotics of the overflow rate of this variation are derived and a converse result establishes its optimality up to the third-order term.

Application of generalized additive model location, scale and shape (GAMLSS) for rice production in Banyuwangi regency

Banyuwangi is one of the regencies in East Java that has great potential for rice production. In terms of rice production, Banyuwangi occupies the top 5 positions in all East Java districts.The focus of this research is to get a suitable model for rice production and find out how the influence of the predictor variables on the response variable. The models used in this case is Generalized Additive Model Location, Scale and Shape (GAMLSS) with RS algorithm.GAMLSS is an extension of the GAM and GLM. GAMLSS is also a semi-parametric which has advantages, namely (1) the distribution used is wider, because it contains the distribution of the exponential family and other distributions and (2) all of the parameters (μ, σ, υ, τ) can be modeled depending on the distribution used. The analyzed model contains rice productionas a response variable and predictor variables used: total rain fall, harvested area, population in district, students ofsenior high schools, and health centers.In this study, the best model will bedefinedby the smallest AIC and SIC from comparison results of all possibilities in vary smoothings, parameters, and degrees of freedom. Furthermore, the best distribution for rice production will also known in this study.

Node Features Adjusted Stochastic Block Model

Stochastic block model (SBM) and its variants are popular models used in community detection for network data. In this article, we propose a feature-adjusted stochastic block model (FASBM) to capture the impact of node features on the network links as well as to detect the residual community structure beyond that explained by the node features. The proposed model can accommodate multiple node features and estimate the form of feature impacts from the data. Moreover, unlike many existing algorithms that are limited to binary-valued interactions, the proposed FASBM model and inference approaches are easily applied to relational data that generate from any exponential family distribution. We illustrate the methods on simulated networks and on two real-world networks: a brain network and an US air-transportation network.

Generalizing expectation propagation with mixtures of exponential family distributions and an application to Bayesian logistic regression

Expectation propagation (EP) is a widely used deterministic approximate inference algorithm in Bayesian machine learning. Traditional EP approximates an intractable posterior distribution through a set of local approximations which are updated iteratively. In this paper, we propose a generalized version of EP called generalized EP (GEP), which is a new method based on the minimization of KL divergence for approximate inference. However, when the variance of the gradient is large, the algorithm may need a long time to converge. We use control variates and develop a variance reduced version of this method called GEP-CV. We evaluate our approach on Bayesian logistic regression, which provides faster convergence and better performance than other state-of-the-art approaches.

Tropical Sufficient Statistics for Persistent Homology

We show that an embedding in Euclidean space based on tropical geometry generates stable sufficient statistics for barcodes. In topological data analysis, barcodes are multiscale summaries of algebraic topological characteristics that capture the `shape' of data; however, in practice, they have complex structures that make them difficult to use in statistical settings. The sufficiency result presented in this work allows for classical probability distributions to be assumed on the tropical geometric representation of barcodes. This makes a variety of parametric statistical inference methods amenable to barcodes, all while maintaining their initial interpretations. More specifically, we show that exponential family distributions may be assumed, and that likelihood functions for persistent homology may be constructed. We conceptually demonstrate sufficiency and illustrate its utility in persistent homology dimensions 0 and 1 with concrete parametric applications to human immunodeficiency virus and avian influenza data.

Lindley Dağılımında Sansürlü ve Sansürsüz Verilerde Parametre ve Sağ Kalım Fonksiyonları için Nokta ve Aralık Tahmini

Objective: Survival analysis is a statistical method used to compare the life expectancy of patients in health studies and to investigate the effectiveness of the treatments. The Lindley distribution is a widely used distribution in survival analysis in recent years and is used to describe the life of a process or device. The Lindley distribution is a two-parameter continuous distribution that is widely used in a wide range of fields including biology, engineering and medicine. Material and Method: In this study, point and interval estimates of the Lindley distribution were examined for censored and uncensored data. Parameter inferences and confidence intervals are shown. In the application section, real-time data (uncensored) and a simulation data (uncensored and censored) are compared with other distributions of the exponential distribution family. Results: According to the results obtained from the analysis, it was observed that Lindley distribution gave better results in special data structures compared to other distributions of exponential family. Lindley distribution; For uncensored real-life data and censored and uncensored simulation data, it has a lower model selection criterion. Conclusion: The exact distribution of the data to be used in survival analysis is one of the prerequisites for the success of the analysis. The Lindley distribution, with its increasing popularity in recent years, is a distribution that gives successful results in survival analysis.

$e$PCA: High dimensional exponential family PCA

Many applications involve large datasets with entries from exponential family distributions. Our main motivating application is photon-limited imaging, where we observe images with Poisson distributed pixels. We focus on X-ray Free Electron Lasers (XFEL), a quickly developing technology whose goal is to reconstruct molecular structure. In XFEL, estimating the principal components of the noiseless distribution is needed for denoising and for structure determination. However, the standard method, Principal Component Analysis (PCA), can be inefficient in non-Gaussian noise. Motivated by this application, we develop $e$PCA (exponential family PCA), a new methodology for PCA on exponential families. $e$PCA is a fast method that can be used very generally for dimension reduction and denoising of large data matrices with exponential family entries. We conduct a substantive XFEL data analysis using $e$PCA. We show that $e$PCA estimates the PCs of the distribution of images more accurately than PCA and alternatives. Importantly, it also leads to better denoising. We also provide theoretical justification for our estimator, including the convergence rate and the Marchenko–Pastur law in high dimensions. An open-source implementation is available.

Soft Randomized Machine Learning

A new method for entropy-randomized machine learning is proposed based on empirical risk minimization instead of the exact fulfillment of empirical balance conditions. The corresponding machine learning algorithm is shown to generate a family of exponential distributions, and their structure is found.

Approximation of Information Divergences for Statistical Learning with Applications

Abstract In this paper we give a partial response to one of the most important statistical questions, namely, what optimal statistical decisions are and how they are related to (statistical) information theory. We exemplify the necessity of understanding the structure of information divergences and their approximations, which may in particular be understood through deconvolution. Deconvolution of information divergences is illustrated in the exponential family of distributions, leading to the optimal tests in the Bahadur sense. We provide a new approximation of I-divergences using the Fourier transformation, saddle point approximation, and uniform convergence of the Euler polygons. Uniform approximation of deconvoluted parts of I-divergences is also discussed. Our approach is illustrated on a real data example.

Estimation of P(X>Y) for the Positive Exponential Family of Distributions

A positive exponential family of distributions is taken into consideration. Two measures of reliability are discussed. Uniformly minimum variance unbiased estimators (UMVUES) and maximum likelihood estimators (MLES) are developed for the reliability functions. In addition to the UMVUES and MLES, we derive the method of moment estimators (MOME). The performances of two types of estimators are compared through Monte Carlo simulation.

A Non-Linear Filtering Algorithm Based on Alpha-Divergence Minimization.

A non-linear filtering algorithm based on the alpha-divergence is proposed, which uses the exponential family distribution to approximate the actual state distribution and the alpha-divergence to measure the approximation degree between the two distributions; thus, it provides more choices for similarity measurement by adjusting the value of during the updating process of the equation of state and the measurement equation in the non-linear dynamic systems. Firstly, an -mixed probability density function that satisfies the normalization condition is defined, and the properties of the mean and variance are analyzed when the probability density functions and are one-dimensional normal distributions. Secondly, the sufficient condition of the alpha-divergence taking the minimum value is proven, that is when , the natural statistical vector’s expectations of the exponential family distribution are equal to the natural statistical vector’s expectations of the -mixed probability state density function. Finally, the conclusion is applied to non-linear filtering, and the non-linear filtering algorithm based on alpha-divergence minimization is proposed, providing more non-linear processing strategies for non-linear filtering. Furthermore, the algorithm’s validity is verified by the experimental results, and a better filtering effect is achieved for non-linear filtering by adjusting the value of .

Multivariate Analogs of Classical Univariate Discrete Distributions and Their Properties

Some discrete distributions such as Bernoulli, binomial, geometric, negative binomial, Poisson, Polya–Aeppli, and others play an important role in applied problems of probability theory and mathematical statistics. We propose a variant of a multivariate distribution whose components have a given univariate discrete distribution. In fact we consider some very general variant of the so-called reduction method. We find the explicit form of the mass function and generating function of such distribution and study their properties. We prove that our construction is unique in natural exponential families of distributions. Our results are the generalization and unification of many results of other authors.

A general framework for association analysis of heterogeneous data

Multivariate association analysis is of primary interest in many applications. Despite the prevalence of high-dimensional and non-Gaussian data (such as count-valued or binary), most existing methods only apply to low-dimensional data with continuous measurements. Motivated by the Computer Audition Lab 500-song (CAL500) music annotation study, we develop a new framework for the association analysis of two sets of high-dimensional and heterogeneous (continuous/binary/count) data. We model heterogeneous random variables using exponential family distributions, and exploit a structured decomposition of the underlying natural parameter matrices to identify shared and individual patterns for two data sets. We also introduce a new measure of the strength of association, and a permutation-based procedure to test its significance. An alternating iteratively reweighted least squares algorithm is devised for model fitting, and several variants are developed to expedite computation and achieve variable selection. The application to the CAL500 data sheds light on the relationship between acoustic features and semantic annotations, and provides effective means for automatic music annotation and retrieval.

Bayesian tensor factorization for multi-way analysis of multi-dimensional EEG

Abstract Factorization-based analysis of multi-dimensional EEG (Electroencephalography) has become increasingly important in neuroscience research and practices with the capability of extracting latent multi-way features. However, how to sift the most informative factors of routinely noisy EEG remains unclear especially under the circumstance of no a priori knowledge. This study proposes a Bayesian tensor factorization (BTF) model as a “one-stop” solution to the challenges. BTF assumes non-informative priori on potential distribution of factors and noise derived from exponential family distribution. A high-dimensional variational Bayesian inference method is designed to iteratively estimate the posterior distribution of potential factors. The factor vectors whose elements are “small” values can then be identified as redundancy and filtered out afterwards. Finally, the study enables a generic factorization-based method for multi-way analysis of brain states. Results from experiments on synthesized tensors indicate that (1) BTF excels in processing EEG tensor mixed with intensive white noises in comparison with the traditional counterparts; and (2) the non-informative components in factors can be filtered out effectively (rank reduction). Two case studies of factorization-based multi-way analysis assisted by a Multi-Layer Perception (MLP) network have been performed on two real EEG datasets, and (1) seizure detection and (2) sleep stage classification can be achieved with averaged accuracy up to 99.52% and 90.81% respectively.

Flexible Bayesian Models for Inferences From Coarsened, Group-Level Achievement Data

This article proposes a flexible extension of the Fay–Herriot model for making inferences from coarsened, group-level achievement data, for example, school-level data consisting of numbers of students falling into various ordinal performance categories. The model builds on the heteroskedastic ordered probit (HETOP) framework advocated by Reardon, Shear, Castellano, and Ho by allowing group parameters to be modeled with regressions on group-level covariates, and residuals modeled using the flexible exponential family of distributions recommended by Efron. We demonstrate that the alternative modeling framework, termed the “Fay–Herriot heteroskedastic ordered probit” (FH-HETOP) model, is useful for mitigating some of the challenges with direct maximum likelihood estimators from the HETOP model. We conduct a simulation study to compare the costs and benefits of several methods for using the FH-HETOP model to estimate group parameters and functions of them, including posterior means, constrained Bayes estimators, and the “triple goal” estimators of Shen and Louis. We also provide an application of the FH-HETOP model to math proficiency data from the Early Childhood Longitudinal Study. Code for estimating the FH-HETOP model and conducting supporting calculations is provided in a new package for the R environment.