SIERRA: A robust bilateral feature upsampler for dense prediction

Abstract

Feature upsampling is a fundamental operation in modern deep network architectures. Existing upsamplers, however, are prone to cause negative upsampling—performance of an upsampler falls behind naive interpolation. For instance, the recent dynamic upsampler CARAFE is the best performing operator in semantic segmentation, but it turns out to be the worst one in image matting. In this work, we present robuSt bIlatERal featuRe upsAmpler (SIERRA), a simple, task-robust, plug-and-play, and ultra lightweight upsampler. Its key idea is to use an efficient gradient-prior kernel to modulate a (shifted) distance-prior kernel to control which feature points participate in interpolation, which shares a similar spirit to joint bilateral filtering (JBF). Yet, in contrast to JBF that requires high-res guidance, SIERRA generates kernels from the low-res decoder feature alone. Extensive experiments demonstrate the superiority and robustness of SIERRA on five dense prediction tasks. Code will be available online.