Multinomial Case Research Articles

Uncertainty quantification in logistic regression using random fuzzy sets and belief functions

Evidential likelihood-based inference is a new approach to statistical inference in which the relative likelihood function is interpreted as a possibility distribution. By expressing new data as a function of the parameter and a random variable with known probability distribution, one then defines a random fuzzy set and an associated predictive belief function representing uncertain knowledge about future observations. In this paper, this approach is applied to binomial and multinomial regression. In the binomial case, the predictive belief function can be computed by numerically integrating the possibility distribution of the posterior probability. In the multinomial case, the solution is obtained by a combination of constrained nonlinear optimization and Monte Carlo simulation. In both cases, computations can be considerably simplified using a normal approximation to the relative likelihood. Numerical experiments show that decision rules based on predictive belief functions make it possible to reach lower error rates for different rejection rates, as compared to decisions based on posterior probabilities.

Naive Feature Selection: A Nearly Tight Convex Relaxation for Sparse Naive Bayes

Because of its linear complexity, naive Bayes classification remains an attractive supervised learning method, especially in very large-scale settings. We propose a sparse version of naive Bayes, which can be used for feature selection. This leads to a combinatorial maximum-likelihood problem, for which we provide an exact solution in the case of binary data, or a bound in the multinomial case. We prove that our convex relaxation bounds become tight as the marginal contribution of additional features decreases using a priori duality gap bounds derived from the Shapley–Folkman theorem. We show how to produce primal solutions satisfying these bounds. Both binary and multinomial sparse models are solvable in time almost linear in problem size, representing a very small extra relative cost compared with the classical naive Bayes. Numerical experiments on text data show that the naive Bayes feature selection method is as statistically effective as state-of-the-art feature selection methods such as recursive feature elimination, l1-penalized logistic regression, and LASSO while being orders of magnitude faster (a python implementation can be found at https://github.com/aspremon/NaiveFeatureSelection ). Funding: A. d’Aspremont acknowledges support from the Fonds AXA Pour la Recherche and Kamet Ventures [Machine Learning and Optimisation Joint Research Initiative] and a Google-focused award as well as funding from Agence Nationale de la Recherche [Grant ANR-19-P3IA-0001]. L. El Ghaoui acknowledges support from Berkeley Artificial Intelligence Research and the Tsinghua–Berkeley–Shenzhen Institute.

Optimization in Item Delivery as Risk Management: Multinomial Case Using the New Method of Statistical Inference for Online Decision

Online activity increasing spreads with the power of technological development. Many studies reported the impact of online activities on decision making. From the statistical perspective, decision making is related to statistical inference. In this regard, it is interesting to propose a new method of statistical inference for online decisions. This method is built by the logarithm distribution of the likelihood function, which allows us to determine statistics using the normal statistical test approach iteratively. It means that the inference can be made in an online way every time new data arrive. Compared to classical methods (commonly, chi-squared), the advantage of this method is that it allows us to make decisions without storing large data. In particular, the novelty of this research is expressed in the algorithm, theorem, and corollary for the statistical inference procedure. In detail, this paper’s simulation discusses online statistical tests for multinomial cases and applies them to transportation data for item delivery, namely traffic density. Changes in traffic density resulted in changes to the strategy of item delivery. The goal is to obtain a minimum delivery time for the risk of losses.

Low-rank matrix denoising for count data using unbiased Kullback-Leibler risk estimation

Many statistical studies are concerned with the analysis of observations organized in a matrix form whose elements are count data. When these observations are assumed to follow a Poisson or a multinomial distribution, it is of interest to focus on the estimation of either the intensity matrix (Poisson case) or the compositional matrix (multinomial case) when it is assumed to have a low rank structure. In this setting, it is proposed to construct an estimator minimizing the regularized negative log-likelihood by a nuclear norm penalty. Such an approach easily yields a low-rank matrix-valued estimator with positive entries which belongs to the set of row-stochastic matrices in the multinomial case. Then, as a main contribution, a data-driven procedure is constructed to select the regularization parameter in the construction of such estimators by minimizing (approximately) unbiased estimates of the Kullback-Leibler (KL) risk in such models, which generalize Stein's unbiased risk estimation originally proposed for Gaussian data. The evaluation of these quantities is a delicate problem, and novel methods are introduced to obtain accurate numerical approximation of such unbiased estimates. Simulated data are used to validate this way of selecting regularizing parameters for low-rank matrix estimation from count data. For data following a multinomial distribution, the performances of this approach are also compared to K-fold cross-validation. Examples from a survey study and metagenomics also illustrate the benefits of this methodology for real data analysis.

New Method of Inferential Statistics for Online Decision in Multinomial Cases

New Method of Inferential Statistics for Online Decision in Multinomial Cases

Simultaneous variable selection and estimation for longitudinal ordinal data with a diverging number of covariates

<abstract><p>In this paper, we study the problem of simultaneous variable selection and estimation for longitudinal ordinal data with high-dimensional covariates. Using the penalized generalized estimation equation (GEE) method, we obtain some asymptotic properties for these types of data in the case that the dimension of the covariates $ p_n $ tends to infinity as the number of cluster $ n $ approaches to infinity. More precisely, under appropriate regular conditions, all the covariates with zero coefficients can be examined simultaneously with probability tending to 1, and the estimator of the non-zero coefficients exhibits the asymptotic Oracle properties. Finally, we also perform some Monte Carlo studies to illustrate the theoretical analysis. The main result in this paper extends the elegant work of Wang et al. <sup>[<xref ref-type="bibr" rid="b1">1</xref>]</sup> to the multinomial response variable case.</p></abstract>

Estimating the variance of Shannon entropy

The statistical analysis of data stemming from dynamical systems, including, but not limited to, time series, routinely relies on the estimation of information theoretical quantities, most notably Shannon entropy. To this purpose, possibly the most widespread tool is provided by the so-called plug-in estimator, whose statistical properties in terms of bias and variance were investigated since the first decade after the publication of Shannon's seminal works. In the case of an underlying multinomial distribution, while the bias can be evaluated by knowing support and data set size, variance is far more elusive. The aim of the present work is to investigate, in the multinomial case, the statistical properties of an estimator of a parameter that describes the variance of the plug-in estimator of Shannon entropy. We then exactly determine the probability distributions that maximize that parameter. The results presented here allow one to set upper limits to the uncertainty of entropy assessments under the hypothesis of memoryless underlying stochastic processes.

Asymptotic distribution of sample Shannon entropy in the case of an underlying finite, regular Markov chain.

The inference of Shannon entropy out of sample histograms is known to be affected by systematic and random errors that depend on the finite size of the available data set. This dependence was mostly investigated in the multinomial case, in which states are visited in an independent fashion. In this paper the asymptotic behavior of the distribution of the sample Shannon entropy, also referred to as plug-in estimator, is investigated in the case of an underlying finite Markov process characterized by a regular stochastic matrix. As the size of the data set tends to infinity, the plug-in estimator is shown to become asymptotically normal, though in a way that substantially deviates from the known multinomial case. The asymptotic behavior of bias and variance of the plug-in estimator are expressed in terms of the spectrum of the stochastic matrix and of the related covariance matrix. Effects of initial conditions are also considered. By virtue of the formal similarity with Shannon entropy, the results are directly applicable to the evaluation of permutation entropy.

Fast computation of multinomial coefficients

In a previous publication, we have used the discrete Fourier transform to compute the binomial coefficients. In the present paper, we extend the previously proposed method to compute the multinomial coefficients, analyse its precision and performance. The other methods, analysed in our previous publication, are also extended to the multinomial case. The FFT method presents the best performance to compute all multinomial coefficients at a given level.

Inference on semiparametric multinomial response models

We explore inference on regression coefficients in semiparametric multinomial response models. We consider cross‐sectional, and both static and dynamic panel settings where we focus throughout on inference under sufficient conditions for point identification. The approach to identification uses a matching insight throughout all three models coupled with variation in regressors: with cross‐section data, we match across individuals while with panel data, we match within individuals over time. Across models, we relax the Indpendence of Irrelevant Alternatives (or IIA assumption, see McFadden (1974)) and allow for arbitrary correlation in the unobservables that determine utility of various alternatives. For the cross‐sectional model, estimation is based on a localized rank objective function, analogous to that used in Abrevaya, Hausman, and Khan (2010), and presents a generalization of existing approaches. In panel data settings, rates of convergence are shown to exhibit a curse of dimensionality in the number of alternatives. The results for the dynamic panel data model generalize the work of Honoré and Kyriazidou (2000) to cover the semiparametric multinomial case. A simulation study establishes adequate finite sample properties of our new procedures. We apply our estimators to a scanner panel data set.

A Latent Gaussian Copula Model for Mixed Data Analysis in Brain Imaging Genetics.

Recent advances in imaging genetics make it possible to combine different types of data including medical images like functional magnetic resonance imaging (fMRI) and genetic data like single nucleotide polymorphisms (SNPs) for comprehensive diagnosis of mental disorders. Understanding complex interactions among these heterogeneous data may give rise to a new perspective, while at the same time demand statistical models for their integration. Various graphical models have been proposed for the study of interaction or association networks with continuous, binary, and count data as well as the mixture of them. However, limited efforts have been made for the multinomial case, for instance, SNP data. Our goal is therefore to fill the void by developing a graphical model for the integration of fMRI image and SNP data, which can provide deeper understanding of the unknown neurogenetic mechanism. In this article, we propose a latent Gaussian copula model for mixed data containing multinomial components. We assume that the discrete variable is obtained by discretizing a latent (unobserved) continuous variable and then create a semi-rank based estimator of the graph structure. The simulation results demonstrate that the proposed latent correlation has more steady and accurate performance than several existing methods in detecting graph structure. When applying to a real schizophrenia data consisting of SNP array and fMRI image collected by the Mind Clinical Imaging Consortium (MCIC), the proposed method reveals a set of distinct SNP-brain associations, which are verified to be biologically significant. The proposed model is statistically promising in handling mixed types of data including multinomial components, which can find widespread applications. To promote reproducible research, the R code is available at https://github.com/Aiying0512/LGCM.

Occlusion-based estimation of independent multinomial random variables using occurrence and sequential information

This paper deals with the relatively new field of sequence-based estimation in which the goal is to estimate the parameters of a distribution by utilizing both the information in the observations and in their sequence of appearance. Traditionally, the Maximum Likelihood (ML) and Bayesian estimation paradigms work within the model that the data, from which the parameters are to be estimated, is known, and that it is treated as a set rather than as a sequence. The position that we take is that these methods ignore, and thus discard, valuable sequence -based information, and our intention is to obtain ML estimates by “extracting” the information contained in the observations when perceived as a sequence. The results of Oommen (November 2007) introduced the concepts of Sequence Based Estimation (SBE) for the Binomial distribution, where the authors derived the corresponding MLE results when the samples are taken two-at-a-time, and then extended these for the cases when they are processed three-at-a-time, four-at-a-time etc. This current paper generalizes these results for the multinomial case. The strategy we invoke involves a novel phenomenon called “Occlusion” that has not been reported in the field of estimation. The phenomenon can be described as follows: By occluding (hiding or concealing) certain observations, we map the estimation problem onto a lower-dimensional space, i.e., onto a binomial space. Once these occluded SBEs have been computed, we demonstrate how the overall Multinomial SBE (MSBE) can be obtained by mapping several lower-dimensional estimates, that are all bound by rigid probability constraints, onto the original higher-dimensional space. In each case, we formally prove and experimentally demonstrate the convergence of the corresponding estimates. The estimation methods proposed here have also been tested on real-life datasets from the UCI repository (Frank and Asuncion, 2013), and the accuracies obtained have been remarkable. We also discuss how various MSBEs can be fused to yield a superior MSBE, and present some potential applications of MSBEs. Our new estimates have great potential for practitioners, especially when the cardinality of the observation set is small.

Stochastic discretized learning-based weak estimation: a novel estimation method for non-stationary environments

The task of designing estimators that are able to track time-varying distributions has found promising applications in many real-life problems.Existing approaches resort to sliding windows that track changes by discarding old observations. In this paper, we report a novel estimator referred to as the Stochastic Discretized Weak Estimator (SDWE), that is based on the principles of discretized Learning Automata (LA). In brief, the estimator is able to estimate the parameters of a time varying binomial distribution using finite memory. The estimator tracks changes in the distribution by operating a controlled random walk in a discretized probability space. The steps of the estimator are discretized so that the updates are done in jumps, and thus the convergence speed is increased. Further, the state transitions are both state-dependent and randomized. As far as we know, such a scheme is both novel and pioneering. The results which have first been proven for binomial distributions have subsequently been extended for the multinomial case, using which they can be applied to any univariate distribution using a histogram-based scheme. The most outstanding and pioneering contribution of our work is that of achieving multinomial estimation without relying on a set of binomial estimators, and where the underlying strategy is truly randomized. Interestingly, the estimator possesses a low computational complexity that is independent of the number of parameters of the multinomial distribution. The generalization of these results for other distributions has also been alluded to. The paper briefly reports conclusive experimental results that prove the ability of the SDWE to cope with non-stationary environments with high adaptation and accuracy.

A generalization of the Binomial distribution based on the dependence ratio

We propose a generalization of the Binomial distribution, called DR‐Binomial, which accommodates dependence among units through a model based on the dependence ratio (Ekholm et al., Biometrika, 82, 1995, 847). Properties of the DR‐Binomial are discussed, and the constraints on its parameter space are studied in detail. Likelihood‐based inference is presented, using both the joint and profile likelihoods; the usefulness of the DR‐Binomial in applications is illustrated on a real dataset displaying negative unit‐dependence, and hence under‐dispersion compared with the Binomial. Although the DR‐Binomial turns out to be a reparameterization of Altham's Additive‐Binomial and Kupper–Haseman's Correlated‐Binomial distribution, we believe its introduction is useful, both in terms of interpretability and mathematical tractability and in terms of generalizability to the Multinomial case.

Elemental information matrices and optimal experimental design for generalized regression models

The construction of optimal experimental designs for regression models requires knowledge of the information matrix of a single observation. The latter can be found if the elemental information matrix corresponding to the distribution of the response is known. We present tables of elemental information matrices for distributions that are often used in statistical work. The tables contain matrices for one- and two-parameter distributions. Additionally we describe multivariate normal and multinomial cases. The parameters of response distributions can themselves be parameterized to provide dependence on explanatory variables, thus leading to regression formulations for wide classes of models. We present essential results from optimal experimental design and illustrate our approach with a few examples including bivariate binary responses and gamma regression.

Multinomial Agent's Trust Modeling Using Entropy of the Dirichlet Distribution

Nowadays one of the most important challenges for integrated systems such as social networks is the evaluation of trust for agents which are interacting with each other in the environment. It plays an important role that the trust has been evaluated from the agent’s experiences. In this paper, we propose a new mathematical approach based on the entropy of Dirichlet distribution, to model the agent’s trust to another agent, based on the past observation from him. The use of Dirichlet distribution for trust model, allows us to evaluate the trust in multinomial cases. Also our approach considers the uncertainty and conflict of agent’s behavior for underlying trust model.

Stochastic learning-based weak estimation of multinomial random variables and its applications to pattern recognition in non-stationary environments

In this paper, we formally present a novel estimation method, referred to as the Stochastic Learning Weak Estimator (SLWE), which yields the estimate of the parameters of a binomial distribution, where the convergence of the estimate is weak, i.e. with regard to the first and second moments. The estimation is based on the principles of stochastic learning. The mean of the final estimate is independent of the scheme's learning coefficient, λ , and both the variance of the final distribution and the speed decrease with λ . Similar results are true for the multinomial case, except that the equations transform from being of a scalar type to be of a vector type. Amazingly enough, the speed of the latter only depends on the same parameter, λ , which turns out to be the only non-unity eigenvalue of the underlying stochastic matrix that determines the time-dependence of the estimates. An empirical analysis on synthetic data shows the advantages of the scheme for non-stationary distributions. The paper also briefly reports (without detailed explanation) conclusive results that demonstrate the superiority of SLWE in pattern-recognition-based data compression, where the underlying data distribution is non-stationary. Finally, and more importantly, the paper includes the results of two pattern recognition exercises, the first of which involves artificial data, and the second which involves the recognition of the types of data that are present in news reports of the Canadian Broadcasting Corporation (CBC). The superiority of the SLWE in both these cases is demonstrated.

Extensions of the Ho and Lee interest-rate model to the multinomial case

The paper presents a state dependent multinomial model of intertemporal changes in the term structure of interest rates. The model is a one-factor interest-rate model within the Markov family models for short-term interest rate and it extends the Ho and Lee [J. Finance XLI (5) (1986) 1001] binomial model. We derive the theoretical basis of the multinomial model, suggest a computational framework to evaluate the model's parameters and investigate the suitability of the model for the Italian market.

Median of the p Value Under the Alternative Hypothesis

Due to absence of an universally acceptable magnitude of the Type I error in various fields, p values are used as a well-recognized tool in decision making in all areas of statistical practice. The distribution of p values under the null hypothesis is uniform. However, under the alternative hypothesis the distribution of the p values is skewed. The expected p value (EPV) has been proposed by authors to be used as a measure of the performance of the test. This article proposes the median of the p values (MPV) which is more appropriate for this purpose. We work out many examples to calculate the MPV's directly and also compare the MPV with the EPV. We consider testing equality of distributions against stochastic ordering in the multinomial case and compare the EPV's and MPV's by simulation. A second simulation study for general continuous data is also considered for two samples with different test statistics for the same hypotheses. In both cases MPV performs better than EPV.

Constrained nonparametric maximum‐likelihood estimation for mixture models

AbstractA nonparametric mixture model specifies that observations arise from a mixture distribution, ∫ f(x, θ) dG(θ), where the mixing distribution G is completely unspecified. A number of algorithms have been developed to obtain unconstrained maximum‐likelihood estimates of G, but none of these algorithms lead to estimates when functional constraints are present. In many cases, there is a natural interest in functional ϕ(G), such as the mean and variance, of the mixing distribution, and profile likelihoods and confidence intervals for ϕ(G) are desired. In this paper we develop a penalized generalization of the ISDM algorithm of Kalbfleisch and Lesperance (1992) that can be used to solve the problem of constrained estimation. We also discuss its use in various different applications. Convergence results and numerical examples are given for the generalized ISDM algorithm, and asymptotic results are developed for the likelihood‐ratio test statistics in the multinomial case.