Smoothed Analysis with Adaptive Adversaries

Abstract

We prove novel algorithmic guarantees for several online problems in the smoothed analysis model. In this model, at each time step an adversary chooses an input distribution with density function bounded above pointwise by  \(\tfrac{1}{\sigma } \) times that of the uniform distribution; nature then samples an input from this distribution. Here, σ is a parameter that interpolates between the extremes of worst-case and average case analysis. Crucially, our results hold for adaptive adversaries that can base their choice of an input distribution on the decisions of the algorithm and the realizations of the inputs in the previous time steps. An adaptive adversary can nontrivially correlate inputs at different time steps with each other and with the algorithm’s current state; this appears to rule out the standard proof approaches in smoothed analysis. This paper presents a general technique for proving smoothed algorithmic guarantees against adaptive adversaries, in effect reducing the setting of an adaptive adversary to the much simpler case of an oblivious adversary (i.e., an adversary that commits in advance to the entire sequence of input distributions). We apply this technique to prove strong smoothed guarantees for three different problems: (1) Online learning: We consider the online prediction problem, where instances are generated from an adaptive sequence of σ -smooth distributions and the hypothesis class has VC dimension d . We bound the regret by \(\tilde{O}\big (\sqrt {T d\ln (1/\sigma)} + d\ln (T/\sigma) \big) \) and provide a near-matching lower bound. Our result shows that under smoothed analysis, learnability against adaptive adversaries is characterized by the finiteness of the VC dimension. This is as opposed to the worst-case analysis, where online learnability is characterized by Littlestone dimension (which is infinite even in the extremely restricted case of one-dimensional threshold functions). Our results fully answer an open question of Rakhlin et al. [64]. (2) Online discrepancy minimization: We consider the setting of the online Komlós problem, where the input is generated from an adaptive sequence of σ -smooth and isotropic distributions on the ℓ 2 unit ball. We bound the ℓ ∞ norm of the discrepancy vector by \(\tilde{O}\big (\ln ^2\big (\frac{nT}{\sigma }\big) \big) \) . This is as opposed to the worst-case analysis, where the tight discrepancy bound is \(\Theta (\sqrt {T/n}) \) . We show such polylog( nT / σ ) discrepancy guarantees are not achievable for non-isotropic σ -smooth distributions. (3) Dispersion in online optimization: We consider online optimization with piecewise Lipschitz functions where functions with ℓ discontinuities are chosen by a smoothed adaptive adversary and show that the resulting sequence is \(\big ({\sigma }/{\sqrt {T\ell }}, \tilde{O}\big (\sqrt {T\ell } \big)\big) \) -dispersed. That is, every ball of radius \({\sigma }/{\sqrt {T\ell }} \) is split by \(\tilde{O}\big (\sqrt {T\ell } \big) \) of the partitions made by these functions. This result matches the dispersion parameters of Balcan et al. [13] for oblivious smooth adversaries, up to logarithmic factors. On the other hand, worst-case sequences are trivially (0, T )-dispersed.