Comparing weighted difference and earth mover's distance via Young diagrams

Abstract

We consider two natural statistics on pairs of histograms, in which the n bins have weights 0,…,n−1. The signed difference (D) between the weighted totals of the histograms is, in a sense, refined by the earth mover's distance (EMD), which measures the amount of work required to equalize the histograms. We were recently surprised, however, by how little the EMD actually does refine D in certain real-world applications, which led to the main problem in this paper: what is the probability that EMD=|D|? We derive a formula for this probability, as well as the expected value of |D|, via the combinatorics of Young diagrams and plane partitions. We then generalize our results to an arbitrary number of histograms, where we realize the higher-dimensional analogue |D| as distance on the Type-A root lattice.