Top-down and bottom-up attentional multiple instance learning for still image action recognition

Abstract

We propose an end-to-end, top-down and bottom-up attentional deep multiple instance learning approach for still image action recognition. Our approach does not rely on neither any attribute nor object labels and/or explicitly trained object detectors. Instead, our approach simultaneously proposes image regions, chooses the action related regions with a top-down attention mechanism, weights each image region with a bottom-up attention layer to mask unrelated pixels to yield a fine-grained pixel-level action related map. Then, it embeds the selected regions into a single image-level feature descriptor and assigns an action label. We evaluate our approach on four benchmark still image action datasets: Stanford40 (Yao et al., 2011), Pascal VOC 2012 (Everingham et al., 2012) MPII (Andriluka et al., 2014) and VCOCO (Gupta and Malik, 2015). Our results demonstrate that, while increasing the overall recognition performance, our framework successfully selects action related image regions and creates pixel grade action masks.