Facial Manipulation Research Articles

Generalization of Forgery Detection With Meta Deepfake Detection Model

Face forgery generating algorithms that produce a range of manipulated videos/images have developed quickly. Consequently, this causes an increase in the production of fake information, making it difficult to identify. Because facial manipulation technologies raise severe concerns, face forgery detection is gaining increasing attention in the area of computer vision. In real-world applications, face forgery detection systems frequently encounter and perform poorly in unseen domains, due to poor generalization. In this paper, we propose a deepfake detection method based on meta-learning called Meta Deepfake Detection (MDD). The goal of the model is to develop a generalized model capable of directly solving new unseen domains without the need for model updates. The MDD algorithm establishes various weights for facial images from various domains. Specifically, MDD uses meta-weight learning to shift information from the source domains to the target domains with meta-optimization steps, which aims for the model to generate effective representations of the source and target domains. We build multi-domain sets using meta splitting strategy to create a meta-train set and meta-test set. Based on these sets, the model determines the gradient descent and obtains backpropagation. The inner and outer loop gradients were aggregated to update the model to enhance generalization. By introducing pair-attention loss and average-center alignment loss, the detection capabilities of the system were substantially enhanced. In addition, we used some evaluation benchmarks established from several popular deepfake datasets to compare the generalization of our proposal in several baselines and assess its effectiveness.

Three-classification face manipulation detection using attention-based feature decomposition

Face manipulation detection has become a recent research hotpot, and many detection methods have been proposed. Most existing detection methods treat face manipulation detection as a vanilla binary classification problem. However, according to the visual effects of facial manipulation, face manipulation approaches are divided into face replacement and face reenactment. In this study, we propose a three-classification face manipulation detection method(TFMD). To implement three-classification face manipulation detection, we introduce a new face forgery feature representation, where the face forgery features are jointly represented by the identity-changing features and the face real-fake features. To decompose the face forgery features into the identity-changing features and the face real-fake features, we design an attention-based feature decomposition module (AFDM). Moreover, we extract high-frequency noise features from the shallow features of the RGB stream to enrich high-frequency information. Through extensive experiments on two datasets, FaceForensics++ and Celeb-DF, we achieve state-of-the-art performance and demonstrate the effectiveness and robustness of our proposed method.

Mixture of Experts for Facial Forgery Detection

With the rapid development of artificial intelligence, facial manipulation technology has also evolved remarkably. This has led to the problem of malicious misuse of technology in a society and also developed the research topic of facial forgery detection. In practice, substantial videos are processed by unknown compression and low-quality methods, which makes the detection of facial forgery extremely challenging. We find that deep neural networks trained for specific forged features have better robustness in the presence of unknown compression and low quality. For this reason, we propose an approach to improve facial forgery detection by implementing a divide-and-conquer idea based on the mixture of experts (MoE) framework. (1) Based on the characteristics of the forgery method itself, the facial forgery detection problem is divided into face-swapping detection and expression-swapping detection. (2) Based on the special similarities and differences between expert models, we design an effective lightweight combination method. (3) We propose the focal loss with adaptive decay (FAD) loss function, which effectively alleviates the problem of over-focusing on noisy samples of the focal loss function and further improves the similarity and difference between expert models. Numerous experiments have demonstrated the effectiveness and superiority of our framework on FaceForensics++ (FF++) datasets and benchmarks.

ViXNet: Vision Transformer with Xception Network for deepfakes based video and image forgery detection

With the advent of image generative technologies, there is a huge growth in the development of facial manipulation techniques that allow people to easily modify media data like videos and images by changing the identity or facial expression of the target person with another person’s face. Colloquially, these manipulated videos and images are termed “deepfakes”. As a result, every piece of content in digital media comes with a question — is this authentic? Hence, there is an unprecedented need for a competent deepfakes detection method. The rapid changes in forging methods make this a very challenging task and thus generalization of the detection methods is also of utmost required. However, the generalization strengths of the prevailing deepfakes detection methods are not satisfactory. In other words, these models perform well when trained and tested on the same dataset but fail to perform satisfactorily when models are trained on one dataset and tested on another. The most modern deep learning aided deepfakes detection techniques looked for a consistent pattern among the leftover artifacts in specific facial regions of the target face rather than the entire face. To this end, we propose a Vision Transformer with Xception Network (ViXNet) to learn the consistency of these almost imperceptible artifacts left by deepfaking methods on the entire facial region. The ViXNet comprises two branches — one tries to learn inconsistencies among local face region specifics by combining patch-wise self-attention module and vision transformer, and the other generates global spatial features using a deep convolutional neural network. To assess the performance of ViXNet, we evaluate it using two different experimental setups — intra-dataset and inter-dataset when using three standard deepfakes video datasets, namely FaceForensics++, and Celeb-DF (V2) and one deepfakes image dataset called Deepfakes. We have attained 98.57% (83.60%), 99.26% (74.78%), and 98.93% (75.13%) AUC scores using intra(inter)-dataset experimental setups on FaceForensics++, Celeb-DF (V2), and Deepfakes datasets respectively. Additionally, we have evaluated ViXNet on the Deepfake Detection Challenge (DFDC) dataset and we have obtained 86.32% AUC score and 79.06% F1-score on the said dataset. Performances of the proposed model are comparable to state-of-the-art methods. Besides, the obtained results ensure the robustness and the generalization ability of the proposed model.

Delving into the Local: Dynamic Inconsistency Learning for DeepFake Video Detection

The rapid development of facial manipulation techniques has aroused public concerns in recent years. Existing deepfake video detection approaches attempt to capture the discrim- inative features between real and fake faces based on tem- poral modelling. However, these works impose supervisions on sparsely sampled video frames but overlook the local mo- tions among adjacent frames, which instead encode rich in- consistency information that can serve as an efficient indica- tor for DeepFake video detection. To mitigate this issue, we delves into the local motion and propose a novel sampling unit named snippet which contains a few successive videos frames for local temporal inconsistency learning. Moreover, we elaborately design an Intra-Snippet Inconsistency Module (Intra-SIM) and an Inter-Snippet Interaction Module (Inter- SIM) to establish a dynamic inconsistency modelling frame- work. Specifically, the Intra-SIM applies bi-directional tem- poral difference operations and a learnable convolution ker- nel to mine the short-term motions within each snippet. The Inter-SIM is then devised to promote the cross-snippet infor- mation interaction to form global representations. The Intra- SIM and Inter-SIM work in an alternate manner and can be plugged into existing 2D CNNs. Our method outperforms the state of the art competitors on four popular benchmark dataset, i.e., FaceForensics++, Celeb-DF, DFDC and Wild- Deepfake. Besides, extensive experiments and visualizations are also presented to further illustrate its effectiveness.

Dual Contrastive Learning for General Face Forgery Detection

With various facial manipulation techniques arising, face forgery detection has drawn growing attention due to security concerns. Previous works always formulate face forgery detection as a classification problem based on cross-entropy loss, which emphasizes category-level differences rather than the essential discrepancies between real and fake faces, limiting model generalization in unseen domains. To address this issue, we propose a novel face forgery detection framework, named Dual Contrastive Learning (DCL), which specially constructs positive and negative paired data and performs designed contrastive learning at different granularities to learn generalized feature representation. Concretely, combined with the hard sample selection strategy, Inter-Instance Contrastive Learning (Inter-ICL) is first proposed to promote task-related discriminative features learning by especially constructing instance pairs. Moreover, to further explore the essential discrepancies, Intra-Instance Contrastive Learning (Intra-ICL) is introduced to focus on the local content inconsistencies prevalent in the forged faces by constructing local region pairs inside instances. Extensive experiments and visualizations on several datasets demonstrate the generalization of our method against the state-of-the-art competitors. Our Code is available at https://github.com/Tencent/TFace.git.

30. COVID-19 and the Yale Response: A Semistructured Interview Study on Plastic Surgery Resident Education and Departmental Adaptation to the Lockdown

Background: COVID-19-lockdowns tasked residencies with developing creative alternatives to their standard curricula. We aimed to capture Yale Plastic Surgery’s response to understand what specific changes should be continued following the pandemic’s conclusion. Methods: A qualitative, semi-structured interview-based approach was utilized. All departmental members during the 2019-2020 academic year were invited for interviews. Results: Responses were collected from ten residents, one fellow, and nine attendings between October 2020 and February 2021. Median interview duration was twenty-two minutes. Residents/fellows noted the change as sudden, with 82% stating the increased unstructured time was challenging to process professionally. 78% of attendings noted a drastic increase in their research output. To increase collaboration between programs and fill student time, institutions opened access to their virtual lectures to each other. Students/attendings reflected highly positively on the experience, and 100% felt that analogous collaboration should continue when possible post-pandemic. Despite this, many emphasized the irreplaceability of in-person visiting lecturers and that the two formats should complement each other going forwards. A modified didactic curriculum involving journal clubs was developed to provide residents with educational time previously spent in the operating room. 90% of residents felt this provided more structure to their education than the original curriculum would have. 90% noted a significant benefit to their knowledge-base. 67% of attendings specifically highlighted the benefit of reduced unstructured free time and increased resident educational engagement. Students were invited to join craniofacial pre-surgical facial manipulation planning meetings between attendings and software representatives. 100% of residents felt observing the planning sessions was beneficial, with many explicitly noting it provided a unique perspective into the surgeon’s thought process behind planned manipulations to which they usually are not privy. 100% of students felt confident at the time of interview that the lockdown would have no lasting effects on their preparation to become a surgeon. 95% of attending physicians felt the long-term surgical ability of students would not be affected by the lockdown. Conclusion: COVID-19 forced the experimentation of novel approaches to resident education. Rapid changes enacted at Yale enabled resident training to advance, and documentation of the success of these changes can inform future curriculum design.

MC-LCR: Multimodal contrastive classification by locally correlated representations for effective face forgery detection

As the remarkable development of facial manipulation technologies is accompanied by severe security concerns, face forgery detection has spurred recent research. Most detection methods train a binary classifier under global supervision to judge whether a face is real or fake. However, advanced manipulations only perform small-scale tampering, posing challenges to comprehensively capturing subtle and local forgery artifacts, especially in high-compression settings and cross-dataset scenarios. To address such limitations, we propose a framework, multimodal contrastive classification by locally correlated representations (MC-LCR), for effective face forgery detection. Instead of specific appearance features, MC-LCR amplifies implicit local discrepancies between authentic and forged faces from both the spatial and frequency domains. A shallow style representation block measures the pairwise correlation of shallow feature maps, encoding local style information to extract more discriminative features in the spatial domain. We observe that subtle forgery artifacts can be further exposed in the patch-wise phase and amplitude spectrum, and that they exhibit different clues. According to the complementarity of amplitude and phase information, we develop a patch-wise amplitude and phase dual attention module to capture locally correlated inconsistencies in the frequency domain. The collaboration of supervised contrastive loss with cross-entropy loss helps the network learn more discriminative and generalized representations. Through extensive experiments and comprehensive studies, we achieve state-of-the-art performance and demonstrate the robustness and generalization of our method.

FaceMD: convolutional neural network-based spatiotemporal fusion facial manipulation detection

FaceMD: convolutional neural network-based spatiotemporal fusion facial manipulation detection

More than perfect: Cosmetic surgery and ageing single women in contemporary China

In this article, I want to connect my inquiry into cosmetic surgery not only to a general category of women, but to its intersections with ageing, singlehood and China. Facilitating dialogues between cultural studies, feminist studies and ageing studies, I argue for the need to be less normative, but more inquisitive, for the need to heed specificities, to hear what the women themselves have to say. Departing from the dominant feminist and neoliberal critique against beautification practices in general and cosmetic surgery in particular, I argue for alternative ways to connect with, and the concomitant refusal to flatten, women’s experience. In other words, I am less interested in how they are used by the beauty industries, but how they use beautification practices – under their specific circumstances. I draw from intersectional perspectives to re-examine the dominant critique against beauty work through this group of single, ageing, Chinese women. What motivates them to agree to acid filling, Botox injection, facial manipulation and other bodily interventions? How do they choose what to do? What factors are involved? How does the surgery impact on their personal and professional life? To answer these questions, I recruited 12 women, born between 1970 and 1989, who have undergone a diversity of interventions. Far from the type or stereotype of women who desire perfection, they articulate their experiences using four repertoires: to demonstrate they are more thoughtful, knowledgeable and mature; to underline their single sisterhood; to distance themselves from the perfectionist longing; and to experience themselves better as single women.

LiSiam: Localization Invariance Siamese Network for Deepfake Detection

Advances in facial manipulation technology have led to increasing indistinguishable and realistic face swap videos, which raises growing concerns about the security risk of deepfakes in the community. Although current deepfake detectors can gain promising performance when handling high-quality faces under within-database settings, most detectors suffer from performance degradation in cross-database evaluation. Moreover, when test faces’ quality is different from training faces, the performance degrades even under within-database settings. To this end, we propose a novel Localization invariance Siamese Network (LiSiam) to enforce localization invariance against different image degradation for deepfake detection. Specifically, our Siamese network-based feature extractor takes the original image and the corresponding quality-degraded image as pairwise inputs and outputs two segmentation maps. A localization invariance loss is further proposed to impose localization consistency between the two segmentation maps. In addition, we design a Mask-guided Transformer to capture the co-occurrence between the forgery region and its surroundings. Finally, a multi-task learning strategy is utilized to obtain a robust and discriminative feature representation and jointly optimize multiple objective functions (i.e., segmentation, classification, and localization invariance losses) in an end-to-end manner. Experimental results on two public datasets, i.e., FaceForensics++ and Celeb-DF, demonstrate the superior performance of our proposed method to state-of-the-art methods.

Learning Forgery Region-Aware and ID-Independent Features for Face Manipulation Detection

Over the past several years, to solve the problem of malicious abuse of facial manipulation technology, face manipulation detection has obtained considerable attention. Although existing works achieved impressive performance on a hold-out test set, their methods suffered a significant performance drop on data from a different distribution than the training set used. In this paper, we conduct an in-depth analysis on existing typical models about poor generalization capability and propose a novel method for face manipulation detection, which can alleviate overfitting and improve the generalization ability by learning forgery region aware and ID-independent features. Specifically, a forgery region guided self-attention module (FR) is introduced to make the model focus on the forgery region and a landmark guided dropout module (LM) is designed to randomly remove features of structured informative regions for destroying identity features. These two regularization modules are then added to the basic classification network, e.g., Xception. With the help of the joint learning framework, both forgery region-aware and ID-independent features are learned for face manipulation detection with better generalization ability. Extensive experiments demonstrate that the proposed method can not only outperform competing state-of-the-art methods on the FaceForensics++ dataset but also achieve superior generalization performance on the Celeb-DF dataset.

A Novel Defensive Strategy for Facial Manipulation Detection Combining Bilateral Filtering and Joint Adversarial Training

Facial manipulation enables facial expressions to be tampered with or facial identities to be replaced in videos. The fake videos are so realistic that they are even difficult for human eyes to distinguish. This poses a great threat to social and public information security. A number of facial manipulation detectors have been proposed to address this threat. However, previous studies have shown that the accuracy of these detectors is sensitive to adversarial examples. The existing defense methods are very limited in terms of applicable scenes and defense effects. This paper proposes a new defense strategy for facial manipulation detectors, which combines a passive defense method, bilateral filtering, and a proactive defense method, joint adversarial training, to mitigate the vulnerability of facial manipulation detectors against adversarial examples. The bilateral filtering method is applied in the preprocessing stage of the model without any modification to denoise the input adversarial examples. The joint adversarial training starts from the training stage of the model, which mixes various adversarial examples and original examples to train the model. The introduction of joint adversarial training can train a model that defends against multiple adversarial attacks. The experimental results show that the proposed defense strategy positively helps facial manipulation detectors counter adversarial examples.

Initiative Defense against Facial Manipulation

Benefiting from the development of generative adversarial networks (GAN), facial manipulation has achieved significant progress in both academia and industry recently. It inspires an increasing number of entertainment applications but also incurs severe threats to individual privacy and even political security meanwhile. To mitigate such risks, many countermeasures have been proposed. However, the great majority methods are designed in a passive manner, which is to detect whether the facial images or videos are tampered after their wide propagation. These detection-based methods have a fatal limitation, that is, they only work for ex-post forensics but can not prevent the engendering of malicious behavior. To address the limitation, in this paper, we propose a novel framework of initiative defense to degrade the performance of facial manipulation models controlled by malicious users. The basic idea is to actively inject imperceptible venom into target facial data before manipulation. To this end, we first imitate the target manipulation model with a surrogate model, and then devise a poison perturbation generator to obtain the desired venom. An alternating training strategy are further leveraged to train both the surrogate model and the perturbation generator. Two typical facial manipulation tasks: face attribute editing and face reenactment, are considered in our initiative defense framework. Extensive experiments demonstrate the effectiveness and robustness of our framework in different settings. Finally, we hope this work can shed some light on initiative countermeasures against more adversarial scenarios.

Local Relation Learning for Face Forgery Detection

With the rapid development of facial manipulation techniques, face forgery has received considerable attention in digital media forensics due to security concerns. Most existing methods formulate face forgery detection as a classification problem and utilize binary labels or manipulated region masks as supervision. However, without considering the correlation between local regions, these global supervisions are insufficient to learn a generalized feature and prone to overfitting. To address this issue, we propose a novel perspective of face forgery detection via local relation learning. Specifically, we propose a Multi-scale Patch Similarity Module (MPSM), which measures the similarity between features of local regions and forms a robust and generalized similarity pattern. Moreover, we propose an RGB-Frequency Attention Module (RFAM) to fuse information in both RGB and frequency domains for more comprehensive local feature representation, which further improves the reliability of the similarity pattern. Extensive experiments show that the proposed method consistently outperforms the state-of-the-arts on widely-used benchmarks. Furthermore, detailed visualization shows the robustness and interpretability of our method.

Detecting facial manipulated videos based on set convolutional neural networks

With the boom of artificial intelligence, facial manipulation technology is becoming more simple and more numerous. At the same time, the technology also has a large and profound negative impact on face forensics, such as Deepfakes. In this paper, in order to aggregate multiframe features to detect facial manipulation videos, we solve facial manipulated video detection from set perspective and propose a novel framework based on set, which is called set convolutional neural network (SCNN). Three instances of the proposed framework SCNN are implemented and evaluated on the Deepfake TIMIT dataset, FaceForensics++ dataset and DFDC Preview datset. The results show that the method outperforms previous methods and can achieve state-of-the-art performance on both datasets. As a perspective, the proposed method is a fusion promotion of single-frame digital video forensics network.

PRRNet: Pixel-Region relation network for face forgery detection

As advanced facial manipulation technologies develop rapidly, one can easily modify an image by changing the identity or the facial expression of the target person, which threatens social security. To address this problem, face forgery detection becomes an important and challenging task. In this paper, we propose a novel network, called Pixel-Region Relation Network (PRRNet), to capture pixel-wise and region-wise relations respectively for face forgery detection. The main motivation is that a facial manipulated image is composed of two parts from different sources, and the inconsistencies between the two parts is a significant kind of evidence for manipulation detection. Specifically, PRRNet contains two serial relation modules, i.e. the Pixel-Wise Relation (PR) module and the Region-Wise Relation (RR) module. For each pixel in the feature map, the PR module captures its similarities with other pixels to exploit the local relations information. Then, the PR module employs a spatial attention mechanism to represent the manipulated region and the original region separately. With the representations of the two regions, the RR module compares them with multiple metrics to measure the inconsistency between these two regions. In particular, the final predictions are obtained totally based on whether the inconsistencies exist. PRRNet achieves the state-of-the-art detection performance on three recent proposed face forgery detection datasets. Besides, our PRRNet shows the robustness when trained and tested on different image qualities.

Detecting handcrafted facial image manipulations and GAN-generated facial images using Shallow-FakeFaceNet

The rapid progress of sophisticated image editing tools has made it easier to manipulate original face images and create fake media content by putting one’s face to another. In addition to image editing tools, creating natural-looking fake human faces can be easily achieved by Generative Adversarial Networks (GANs). However, malicious use of these new media generation technologies can lead to severe problems, such as the development of fake pornography, defamation, or fraud. In this paper, we introduce a novel Handcrafted Facial Manipulation (HFM) image dataset and soft computing neural network models (Shallow-FakeFaceNets) with an efficient facial manipulation detection pipeline. Our neural network classifier model, Shallow-FakeFaceNet (SFFN), shows the ability to focus on the manipulated facial landmarks to detect fake images. The detection pipeline only relies on detecting fake facial images based on RGB information, not leveraging any metadata, which can be easily manipulated. Our results show that our method achieves the best performance of 72.52% in Area Under the Receiver Operating Characteristic (AUROC), gaining 3.99% F1-score and 2.91% AUROC on detecting handcrafted fake facial images, and 93.99% on detecting small GAN-generated fake images, gaining 1.98% F1-score and 10.44% AUROC compared to the best performing state-of-the-art classifier. This study is targeted for developing an automated defense mechanism to combat fake images used in different online services and applications, leveraging our state-of-the-art hand-crafted fake facial dataset (HFM) and the neural network classifier Shallow-FakeFaceNet (SFFN). In addition, our work presents various experimental results that can help guide better applied soft computing research in the future to effectively combat and detect human and GAN-generated fake face images.

Non-Pregnant and Pregnant Women\u2019s Femininity Preferences in Male Faces: Tests Based on Within- and Between-Sex Sexual Dimorphism Facial Manipulations

Past research on women’s preferences for male facial masculinity in Western cultures has produced inconsistent results. Some inconsistency may be related to the use of different facial stimulus manipulations (e.g., between-sex sexual dimorphic facial manipulation or within-sex sexual dimorphic facial manipulation) that do not perfectly avoid non-facial cues, and pregnancy status may also influence women’s face preferences. We therefore recruited pregnant and nonpregnant Chinese women and manipulated the sexual dimorphism of male facial stimuli to explore the influences of manipulation methods, non-facial cues, and pregnancy status on face preferences. Results showed that: (1) in contrast with a general masculinity preference observed in Western cultures, both pregnant and nonpregnant Chinese women preferred feminized and neutral male faces generally; (2) pregnant women’s preference for feminized male faces was stable across manipulation methods, while nonpregnant women preferred feminized male faces except under between-sex sexual dimorphism manipulation; and (3) manipulation methods, rather than non-facial cues, influenced participants’ face preferences. Specifically, women showed the strongest preferences for femininity when face stimuli were manipulated by within-sex sexual dimorphic facial manipulation, followed by unmanipulated faces and between-sex sexual dimorphic facial manipulation. This effect was stronger for nonpregnant women in the unmanipulated condition and for pregnant women in the between-sex sexual dimorphic facial manipulation. This research provides empirical evidence of women’s preferences for sexual dimorphism in male faces in a non-Western culture, as well as the effects of facial manipulation methods, pregnancy status, and the interactions between these factors.

GANprintR: Improved Fakes and Evaluation of the State of the Art in Face Manipulation Detection

The availability of large-scale facial databases, together with the remarkable progresses of deep learning technologies, in particular Generative Adversarial Networks (GANs), have led to the generation of extremely realistic fake facial content, raising obvious concerns about the potential for misuse. Such concerns have fostered the research on manipulation detection methods that, contrary to humans, have already achieved astonishing results in various scenarios. In this study, we focus on the synthesis of entire facial images, which is a specific type of facial manipulation. The main contributions of this study are four-fold: i) a novel strategy to remove GAN “fingerprints” from synthetic fake images based on autoencoders is described, in order to spoof facial manipulation detection systems while keeping the visual quality of the resulting images; ii) an in-depth analysis of the recent literature in facial manipulation detection; iii) a complete experimental assessment of this type of facial manipulation, considering the state-of-the-art fake detection systems (based on holistic deep networks, steganalysis, and local artifacts), remarking how challenging is this task in unconstrained scenarios; and finally iv) we announce a novel public database, named iFakeFaceDB, yielding from the application of our proposed GAN-fingerprint Removal approach (GANprintR) to already very realistic synthetic fake images. The results obtained in our empirical evaluation show that additional efforts are required to develop robust facial manipulation detection systems against unseen conditions and spoof techniques, such as the one proposed in this study.