Enhanced task attention with adversarial learning for dynamic multi-task CNN

Abstract

Multi-task deep learning is promising to solve multi-label multi-instance visual recognition tasks. However, flexible information sharing in the task group might bring performance bottlenecks to an individual task. To tackle this problem, we propose a novel learning framework of multi-task Convolutional Neural Network (CNN) to enhance task attention through conditionally tuning the Task Transfer Connections (TTC) with adversarial learning. For the dynamic multi-task CNN, we set up a shared subnet to extract shared features across multiple tasks and a task discriminator shared by all layers to distinguish features of all subnets. The adversarial training is introduced between the shared subnet and the task discriminator to guide each task subnet to focus on its specific task. To apply adversarial learning to the complex labeling system of multiple tasks, we design an even-label strategy for the multi-task model with a shared subnet to make adversarial learning feasible for the complex labeling system of multiple tasks. As a result, the proposed model can constrain the shared subnet’s learning unbiased to any single task and achieve task attention for all task subnets. Experimental results of the ablation study and the TTC analysis validate the effectiveness of the proposed approach.