Feeling the bern: Adaptive estimators for Bernoulli probabilities of pairwise comparisons

Abstract

We study methods for aggregating pairwise comparison data in order to estimate outcome probabilities for future comparisons. We investigate this problem under a flexible class of models satisfying the strong stochastic transitivity (SST) condition. Prior works have studied the minimax risk for estimation of the pairwise comparison probabilities under the SST model. The minimax risk, however, is a measure of the worst-case risk of an estimator over a large parameter space, and in general provides only a rudimentary understanding of an estimator in problems where the intrinsic difficulty of estimation varies considerably over the parameter space. In this paper, we introduce an adaptivity index, in order to benchmark the performance of an estimator against an oracle estimator. The adaptivity index, in addition to measuring the worst-case risk of an estimator, also captures the extent to which the estimator adapts to the instance-specific difficulty of the underlying problem, relative to an oracle estimator. In the context of this adaptivity index we provide two main results. We propose a three-step, Count-Randomize-Least squares (CRL) estimator, and derive upper bounds on the adaptivity index of this estimator. We complement this result with a complexity-theoretic result, that shows that conditional on the planted clique hardness conjecture, no computationally efficient estimator can achieve a substantially smaller adaptivity index.