LiSiam: Localization Invariance Siamese Network for Deepfake Detection

Abstract

Advances in facial manipulation technology have led to increasing indistinguishable and realistic face swap videos, which raises growing concerns about the security risk of deepfakes in the community. Although current deepfake detectors can gain promising performance when handling high-quality faces under within-database settings, most detectors suffer from performance degradation in cross-database evaluation. Moreover, when test faces’ quality is different from training faces, the performance degrades even under within-database settings. To this end, we propose a novel Localization invariance Siamese Network (LiSiam) to enforce localization invariance against different image degradation for deepfake detection. Specifically, our Siamese network-based feature extractor takes the original image and the corresponding quality-degraded image as pairwise inputs and outputs two segmentation maps. A localization invariance loss is further proposed to impose localization consistency between the two segmentation maps. In addition, we design a Mask-guided Transformer to capture the co-occurrence between the forgery region and its surroundings. Finally, a multi-task learning strategy is utilized to obtain a robust and discriminative feature representation and jointly optimize multiple objective functions (i.e., segmentation, classification, and localization invariance losses) in an end-to-end manner. Experimental results on two public datasets, i.e., FaceForensics++ and Celeb-DF, demonstrate the superior performance of our proposed method to state-of-the-art methods.