Dual-perspective semantic-aware representation blending for multi-label image recognition with partial labels

Abstract

Recently, multi-label image recognition with partial labels (MLR-PL) has attracted increasing attention, in which only some labels are known while others are unknown for each image. However, current algorithms rely on pre-trained image similarity models or iteratively updating the image classification models to generate pseudo labels for unknown labels. Thus, they depend on a certain amount of annotations and inevitably suffer from obvious performance drops. To address this dilemma, we propose a dual-perspective semantic-aware representation blending (DSRB) framework that blends multi-granularity category-specific semantic representation across different images, from an instance and prototype perspective, respectively, to transfer information of known labels to complement unknown labels. Specifically, an instance-perspective representation blending (IPRB) module is designed to blend the representations of the known labels in an image with the representations of the corresponding unknown labels in another image to complement these unknown labels. Meanwhile, a prototype-perspective representation blending (PPRB) module is introduced to learn more stable representation prototypes for each category and blends the representation of unknown labels with the prototypes of corresponding labels in a location-sensitive manner to complement these unknown labels. Extensive experiments on various datasets show that the proposed DSRB consistently outperforms current state-of-the-art algorithms on all known label proportion settings.