JRC: Deepfake detection via joint reconstruction and classification

Abstract

Deep learning has enabled realistic face manipulation for malicious purposes (e.g., deepfakes), which poses significant concerns over the integrity of the media in circulation. Most existing deep learning techniques for deepfake detection can achieve promising performance in the intra-dataset evaluation setting, but are unable to perform satisfactorily in the inter-dataset evaluation setting. Most previous methods use a backbone network to extract global features for making predictions and only employ binary supervision to train the network. Classification merely based on the learning of global features often leads to weak generalizability to deepfakes of unseen manipulation methods. In this paper, we design a two-branch Convolutional AutoEncoder (CAE), which considers the reconstruction and classification tasks simultaneously for deepfake detection. This Joint Reconstruction and Classification (JRC) method shares the information learned by one task with the other, each focusing on different aspects, and hence boosts the overall performance. JRC is end-to-end, and experiments demonstrate that it achieves state-of-the-art performance on three commonly-used datasets, particularly in the cross-dataset evaluation setting.