Recovery of Joint Probability Distribution from One-Way Marginals: Low Rank Tensors and Random Projections

Abstract

Joint probability density or joint probability mass function (PDF/PMF) estimation is a fundamental machine learning problem. The number of free parameters scales exponentially with respect to the number of random variables. Hence, most work on nonparametric joint distribution estimation is based on some structural assumptions such as clique factorization adopted by probabilistic graphical models, imposition of low rank on the joint probability tensor and reconstruction from 3-way or 2-way marginals, etc. In the current work, we link random projections of data to the problem of PMF estimation using techniques from tomography. Using it alongside low-rank tensor decomposition, we present an approach to estimate joint distribution from just one-way marginals in a transformed space. We provide a novel algorithm for recovering factors of the tensor from one-way marginals, test it across synthetic and real-world datasets, and also perform MAP inference on the estimated model for classification.