Explanation of Variability and Removal of Confounding Factors from Data through Optimal Transport

Abstract

A methodology based on the theory of optimal transport is developed to attribute variability in data sets to known and unknown factors and to remove such attributable components of the variability from the data. Denoting by x the quantities of interest and by z the explanatory factors, the procedure transforms x into filtered variables y through a z‐dependent map, so that the conditional probability distributions ρ(x|z) are pushed forward into a target distribution μ(y), independent of z. Among all maps and target distributions that achieve this goal, the procedure selects the one that minimally distorts the original data: the barycenter of the ρ(x|z). Connections are found to unsupervised learning and to fundamental problems in statistics such as conditional density estimation and sampling. Particularly simple instances of the methodology are shown to be equivalent to k‐means and principal component analysis. An application is shown to a time series of ground temperature hourly data across the United States.© 2017 Wiley Periodicals, Inc.