Facial Manipulation Research Articles

TAENet: Two-branch Autoencoder Network for Interpretable Deepfake Detection

Deepfake detection attracts increasingly attention due to serious security issues caused by facial manipulation techniques. Recently, deep learning-based detectors have achieved promising performance. However, these detectors suffer severe untrustworthy due to the lack of interpretability. Thus, it is essential to work on the interpretibility of deepfake detectors to improve the reliability and traceability of digital evidence. In this work, we propose a two-branch autoencoder network named TAENet for interpretable deepfake detection. TAENet is composed of Content Feature Disentanglement (CFD), Content Map Generation (CMG), and Classification. CFD extracts latent features of real and forged content with dual encoder and feature discriminator. CMG employs a Pixel-level Content Map Generation Loss (PCMGL) to guide the dual decoder in visualizing the latent representations of real and forged contents as real-map and fake-map. In classification module, the Auxiliary Classifier (AC) serves as map amplifier to improve the accuracy of real-map image extraction. Finally, the learned model decouples the input image into two maps that have the same size as the input, providing visualized evidence for deepfake detection. Extensive experiments demonstrate that TAENet can offer interpretability in deepfake detection without compromising accuracy.

An efficient identity-preserving and fast-converging hybrid generative adversarial network inversion framework

In this paper, we present a novel Hybrid Generative Adversarial Network (HGAN) inversion framework that enables facial images to be rapidly inverted while preserving identity and personality characteristics. Accurate inversion of facial images requires high precision in computer vision and is critical to the success of future facial manipulations (age progression, regression, accessory, and hair stylization). However, existing methods often fail to preserve the personality characteristics of the real image, negatively affecting the accuracy of manipulations. In this context, our key contribution lies in using a transformer-based strategy to initiate the generator, which effectively models spatial relationships for detailed image processing. This approach is innovative because it leverages transformer structures to enhance image inversion tasks. Additionally, we introduce a novel loss function to enhance convergence speed and reliability, ensuring high accuracy in identity and personality trait preservation. Experimental results show that our method achieves a reconstruction accuracy of 93% and improves inversion time by 86%. This advancement could significantly impact facial manipulation technologies, laying the foundation for a technological breakthrough with potential applications in secure digital authentication systems and personal data protection. Our method may have a significant impact on privacy and security in future studies, contributing to the development of secure digital authentication systems and enhancing the protection of personal data. Therefore, our work is crucial for advancing the field of facial image manipulation and ensuring the privacy and security of personal data.

Multi-teacher Universal Distillation Based on Information Hiding for Defense Against Facial Manipulation

Multi-teacher Universal Distillation Based on Information Hiding for Defense Against Facial Manipulation

Combined spatial and frequency dual stream network for face forgery detection.

With the development of generative model, the cost of facial manipulation and forgery is becoming lower and lower. Fraudulent data has brought numerous hidden threats in politics, privacy, and cybersecurity. Although many methods of face forgery detection focus on the learning of high frequency forgery traces and achieve promising performance, these methods usually learn features in spatial and frequency independently. In order to combine the information of the two domains, a combined spatial and frequency dual stream network is proposed for face forgery detection. Concretely, a cross self-attention (CSA) module is designed to improve frequency feature interaction and fusion at different scales. Moreover, to augment the semantic and contextual information, frequency guided spatial feature extraction module is proposed to extract and reconstruct the spatial information. These two modules deeply mine the forgery traces via a dual-stream collaborative network. Through comprehensive experiments on different datasets, we demonstrate the effectiveness of proposed method for both within and cross datasets.

Adversarial Robust Safeguard for Evading Deep Facial Manipulation

The non-consensual exploitation of facial manipulation has emerged as a pressing societal concern. In tandem with the identification of such fake content, recent research endeavors have advocated countering manipulation techniques through proactive interventions, specifically the incorporation of adversarial noise to impede the manipulation in advance. Nevertheless, with insufficient consideration of robustness, we show that current methods falter in providing protection after simple perturbations, e.g., blur. In addition, traditional optimization-based methods face limitations in scalability as they struggle to accommodate the substantial expansion of data volume, a consequence of the time-intensive iterative pipeline. To solve these challenges, we propose a learning-based model, Adversarial Robust Safeguard (ARS), to generate desirable protection noise in a single forward process, concurrently exhibiting a heightened resistance against prevalent perturbations. Specifically, our method involves a two-way protection design, characterized by a basic protection component responsible for generating efficacious noise features, coupled with robust protection for further enhancement. In robust protection, we first fuse image features with spatially duplicated noise embedding, thereby accounting for inherent information redundancy. Subsequently, a combination comprising a differentiable perturbation module and an adversarial network is devised to simulate potential information degradation during the training process. To evaluate it, we conduct experiments on four manipulation methods and compare recent works comprehensively. The results of our method exhibit good visual effects with pronounced robustness against varied perturbations at different levels.

Multiclass AI-Generated Deepfake Face Detection Using Patch-Wise Deep Learning Model

In response to the rapid advancements in facial manipulation technologies, particularly facilitated by Generative Adversarial Networks (GANs) and Stable Diffusion-based methods, this paper explores the critical issue of deepfake content creation. The increasing accessibility of these tools necessitates robust detection methods to curb potential misuse. In this context, this paper investigates the potential of Vision Transformers (ViTs) for effective deepfake image detection, leveraging their capacity to extract global features. Objective: The primary goal of this study is to assess the viability of ViTs in detecting multiclass deepfake images compared to traditional Convolutional Neural Network (CNN)-based models. By framing the deepfake problem as a multiclass task, this research introduces a novel approach, considering the challenges posed by Stable Diffusion and StyleGAN2. The objective is to enhance understanding and efficacy in detecting manipulated content within a multiclass context. Novelty: This research distinguishes itself by approaching the deepfake detection problem as a multiclass task, introducing new challenges associated with Stable Diffusion and StyleGAN2. The study pioneers the exploration of ViTs in this domain, emphasizing their potential to extract global features for enhanced detection accuracy. The novelty lies in addressing the evolving landscape of deepfake creation and manipulation. Results and Conclusion: Through extensive experiments, the proposed method exhibits high effectiveness, achieving impressive detection accuracy, precision, and recall, and an F1 rate of 99.90% on a multiclass-prepared dataset. The results underscore the significant potential of ViTs in contributing to a more secure digital landscape by robustly addressing the challenges posed by deepfake content, particularly in the presence of Stable Diffusion and StyleGAN2. The proposed model outperformed when compared with state-of-the-art CNN-based models, i.e., ResNet-50 and VGG-16.

FDML: Feature Disentangling and Multi-view Learning for face forgery detection

Recent advances in realistic facial manipulation techniques have led to a growing interest in forgery detection due to security concerns. The presence of source-dependent information in both the forged images and the learned representations inevitably confuses the detector. To alleviate this issue, we present a Feature Disentangling and Multi-view Learning (FDML) framework to distill forgery-relevant intrinsic features from entangled information in a progressive manner, i.e., from image-level to feature-level. Towards image-level, the input image is first transformed into two complementary views, one using learnable filters to adaptively mine subtle frequency-aware clues, and the other using a novel data augmentation operation called SceneMix to weaken source-specific factors. The intermediate features output from these two branches are fully integrated through a trainable two-branch Hybrid Attention Module (HAM), guiding the effective performance of fused features. For feature-level, to automatically separate forgery-relevant features from source-relevant features and reduce the interference of irrelevant factors in decision making, two feature disentangling schemes are proposed, and finally only forgery-relevant features are used for prediction, which greatly improves the detection performance. Extensive experiments show that our framework achieves competitive performances compared with state-of-the-art works.

Exposing low-quality deepfake videos of Social Network Service using Spatial Restored Detection Framework

The increasing abuse of facial manipulation methods, such as FaceSwap, Deepfakes etc., seriously threatens the authenticity of digital images/videos on the Internet. Therefore, it is of great importance to identify the facial videos to confirm the contents and avoid fake news or rumours. Many researchers have paid great attention to the detection of deepfakes and put forward a number of deep-learning-based detection models.The existing approaches mostly face the performance degradation in detecting low-quality(LQ) videos, i.e. heavily compressed or low-resolution videos through some SNS (Social Network Service), resulting in the limitation in real applications. To address this issue, in this paper, a novel Spatial Restore Detection Framework(SRDF) is proposed for improving the detection performance for LQ videos by restoring spatial features. We designed a feature extraction-enhancement block and a mapping block inspired by super-resolution methods, to restore and enhance texture features. An attention module was introduced to guide the texture features restoration and enhancement stage attending to different local areas and restoring the texture features. Besides, an improved isolated loss was put forward to prevent the expansion of a single area concerned. Moreover, we adopted a regional data augmentation strategy to prompt feature restore and enhancement in the region attended. Extensive experiments conducted on two deepfake datasets have validated the superiority of the proposed method compared to the state-of-the-art, especially in the scenarios of detecting low-quality deepfake videos.

SARGAN: Spatial Attention-Based Residuals for Facial Expression Manipulation

Encoder-decoder based architecture has been widely used in the generator of generative adversarial networks for facial manipulation. However, we observe that the current architecture fails to recover the input image color, rich facial details such as skin color or texture and introduces artifacts as well. In this paper, we present a novel method named SARGAN that addresses the above-mentioned limitations from three perspectives. First, we employed spatial attention-based residual block instead of vanilla residual blocks to properly capture the expression-related features to be changed while keeping the other features unchanged. Second, we exploited a symmetric encoder-decoder network to attend facial features at multiple scales. Third, we proposed to train the complete network with a residual connection which relieves the generator of pressure to generate the input face image thereby producing the desired expression by directly feeding the input image towards the end of the generator. Both qualitative and quantitative experimental results show that our proposed model performs significantly better than state-of-the-art methods. In addition, existing models require much larger datasets for training but their performance degrades on out-of-distribution images. While SARGAN can be trained on smaller facial expressions datasets, which generalizes well on out-of-distribution images including human photographs, portraits, avatars and statues.

Unmasking Deception: Empowering Deepfake Detection with Vision Transformer Network

With the development of image-generating technologies, significant progress has been made in the field of facial manipulation techniques. These techniques allow people to easily modify media information, such as videos and images, by substituting the identity or facial expression of one person with the face of another. This has significantly increased the availability and accessibility of such tools and manipulated content termed ‘deepfakes’. Developing an accurate method for detecting fake images needs time to prevent their misuse and manipulation. This paper examines the capabilities of the Vision Transformer (ViT), i.e., extracting global features to detect deepfake images effectively. After conducting comprehensive experiments, our method demonstrates a high level of effectiveness, achieving a detection accuracy, precision, recall, and F1 rate of 99.5 to 100% for both the original and mixture data set. According to our existing understanding, this study is a research endeavor incorporating real-world applications, specifically examining Snapchat-filtered images.

Multi-attention-based approach for deepfake face and expression swap detection and localization

Advancements in facial manipulation technology have resulted in highly realistic and indistinguishable face and expression swap videos. However, this has also raised concerns regarding the security risks associated with deepfakes. In the field of multimedia forensics, the detection and precise localization of image forgery has become essential tasks. Current deepfake detectors perform well with high-quality faces within specific datasets, but often struggle to maintain their performance when evaluated across different datasets. To this end, we propose an attention-based multi-task approach to improve feature maps for classification and localization tasks. The encoder and the attention-based decoder of our network generate localized maps that highlight regions with information about the type of manipulation. These localized features are shared with the classification network, improving its performance. Instead of using encoded spatial features, attention-based localized features from the decoder’s first layer are combined with frequency domain features to create a discriminative representation for deepfake detection. Through extensive experiments on face and expression swap datasets, we demonstrate that our method achieves competitive performance in comparison to state-of-the-art deepfake detection approaches in both in-dataset and cross-dataset scenarios. Code is available at https://github.com/saimawaseem/Multi-Attention-Based-Approach-for-Deepfake-Face-and-Expression-Swap-Detection-and-Localization.

Semantically Disentangled Variational Autoencoder for Modeling 3D Facial Details.

Parametric face models, like morphable and blendshape models, have shown great potential in face representation, reconstruction, and animation. However, all these models focus on large-scale facial geometry. Facial details like wrinkles are not parameterized in these models, impeding their accuracy and realism. In this paper, we propose a method to learn a Semantically Disentangled Variational Autoencoder (SDVAE) to parameterize facial details and support independent detail manipulation as an extension of an off-the-shelf large-scale face model. Our method utilizes the non-linear capability of Deep Neural Networks for detail modeling, achieving better accuracy and greater representation power compared with linear models. In order to disentangle the semantic factors of identity, expression and age, we propose to eliminate the correlation between different factors in an adversarial manner. Therefore, wrinkle-level details of various identities, expressions, and ages can be generated and independently controlled by changing latent vectors of our SDVAE. We further leverage our model to reconstruct 3D faces via fitting to facial scans and images. Benefiting from our parametric model, we achieve accurate and robust reconstruction, and the reconstructed details can be easily animated and manipulated. We evaluate our method on practical applications, including scan fitting, image fitting, video tracking, model manipulation, and expression and age animation. Extensive experiments demonstrate that the proposed method can robustly model facial details and achieve better results than alternative methods.

FACE SPOOF DETECTION USING CONVOLUTIONAL NEURAL NETWORK

The paper suggests a technique that uses convolutional neural network (CNN) to identify fraudulent facial manipulation. The proposed method comprises teaching an intricate neural network using a comprehensive compilation of genuine and fake facial images. The structure of CNN includes several layers of convolution and pooling, which enable it to identify distinguishing features in the input images. Following its training, the model is employed to differentiate a presented facial image into either authentic or fraudulent. To determine the efficacy of the proposed technique, a standardized data set for identifying counterfeit or altered facial attributes was used. The suggested method presents various benefits compared to current techniques for detecting face spoofing. To start with, utilizing a deep CNN empowers the model to acquire complex and discerning characteristics from the input images, thus augmenting the precision of the categorization mission. Additionally, the suggested method is effective in terms of computational requirements, enabling its utilization in real-time scenarios. The proposed methodology is able to withstand a range of fraudulent tactics used on facial recognition systems, such as print and replay attacks. The findings from this study aid in the progression of face recognition technology by enhancing the accuracy and dependability of fraud detection systems. These improved systems have practical applications in security measures, biometric identification, and digital criminal investigations. The suggested method could substantially enhance the dependability and safety of facial recognition systems, consequently boosting their functional value and credibility.

MRT-Net: Auto-adaptive weighting of manipulation residuals and texture clues for face manipulation detection

With the growing popularity of social media and an outburst of malicious tampering methods, it has become easier than ever to mislead others and create disharmony in society by manipulating social media content. Hence, there is an urgent need to build robust manipulation detection systems. This manuscript proposes a novel end-to-end architecture that extracts discriminative “manipulation residuals” (MR) and “textural” (T) features for facial manipulation detection. Most multi-branch architectures suffer from a common flaw in their fusion stage, where they combine multi-domain features in equal proportion. This is disadvantageous since each feature may not be equally important in making the final prediction. MRT-Net solves this problem by implementing an auto-adaptive weighting mechanism to select an ideal proportion of manipulation residual and textural information that prove complementary to one another. Specifically, two weighting factors for the MR and T features, ∝1 and ∝2 are added as parameters to the proposed neural network and are updated automatically via backpropagation, thereby allowing MRT-Net to find the ideal mix of residual and textural information. Additionally, MRT-Net benefits from a channel attention mechanism that boosts its performance even further. MRT-Net achieves excellent performance on three public benchmark datasets, Deep Fake Detection Challenge (DFDC), CelebDF and FaceForensics++ (FF++). AUC scores achieved are 0.9964 on DFDC, 0.9921 on CelebDF, 0.9910 on FF++(DeepFake), 0.9974 on FF++(Face2Face), 0.9942 on FF++(FaceShifter), 0.9933 on FF++(FaceSwap) and 0.9662 on FF++(NeuralTextures). It also achieves accuracy scores of 0.9760 on DFDC, 0.9815 on CelebDF, 0.9670 on FF++(DeepFake), 0.9767 on FF++(Face2Face), 0.9611 on FF++(FaceShifter), 0.9676 on FF++(FaceSwap) and 0.9025 on FF++(NeuralTextures). These excellent results demonstrate MRT-Net's potency as it comfortably outperforms other state-of-the-art face manipulation detection methods.

Republication de : Ramsay hunt syndrome after parotidectomy: precipitated by facial nerve manipulation?

Republication de : Ramsay hunt syndrome after parotidectomy: precipitated by facial nerve manipulation?

The Same Name Is Not Always the Same: Correlating and Tracing Forgery Methods across Various Deepfake Datasets

Deepfakes are becoming increasingly ubiquitous, particularly in facial manipulation. Numerous researchers and companies have released multiple datasets of face deepfakes labeled to indicate different methods of forgery. However, naming these labels is often arbitrary and inconsistent, leading to the fact that most researchers now choose to use only one of the datasets for research work. However, researchers must use these datasets in practical applications and conduct traceability research. In this study, we employ some models to extract forgery features from various deepfake datasets and utilize the K-means clustering method to identify datasets with similar feature values. We analyze the feature values using the Calinski Harabasz Index method. Our findings reveal that datasets with the same or similar labels in different deepfake datasets exhibit different forgery features. We proposed the KCE system to solve this problem, which combines multiple deepfake datasets according to feature similarity. We analyzed four groups of test datasets and found that the model trained based on KCE combined data faced unknown data types, and Calinski Harabasz scored 42.3% higher than combined by forged names. Furthermore, it is 2.5% higher than the model using all data, although the latter has more training data. It shows that this method improves the generalization ability of the model. This paper introduces a fresh perspective for effectively evaluating and utilizing diverse deepfake datasets and conducting deepfake traceability research.

Constructing Spatio-Temporal Graphs for Face Forgery Detection

Recently, advanced development of facial manipulation techniques threatens web information security, thus, face forgery detection attracts a lot of attention. It is clear that both spatial and temporal information of facial videos contains the crucial manipulation traces, which are inevitably created during the generation process. However, most existing face forgery detectors only focus on the spatial artifacts or the temporal incoherence, and they are struggling to learn a significant and general kind of representations for manipulated facial videos. In this work, we propose to construct spatial-temporal graphs for fake videos to capture the spatial inconsistency and the temporal incoherence at the same time. To model the spatial-temporal relationship among the graph nodes, a novel forgery detector named Spatio-Temporal Graph Network (STGN) is proposed, which contains two kinds of graph-convolution-based units, the Spatial Relation Graph Unit (SRGU) and the Temporal Attention Graph Unit (TAGU). To exploit spatial information, the SRGU models the inconsistency between each pair of patches in the same frame, instead of focusing on the low-level local spatial artifacts which are vulnerable to samples created by unseen manipulation methods. And, the TAGU is proposed to model the long-distance temporal relation among the patches at the same spatial position in different frames with a graph attention mechanism based on the inter-node similarity. With the SRGU and the TAGU, our STGN can combine the discriminative power of spatial inconsistency and the generalization capacity of temporal incoherence for face forgery detection. Our STGN achieves state-of-the-art performances on several popular forgery detection datasets. Extensive experiments demonstrate both the superiority of our STGN on intra manipulation evaluation and the effectiveness for new sorts of face forgery videos on cross manipulation evaluation.

Artifacts-Disentangled Adversarial Learning for Deepfake Detection

Due to the development of facial manipulation technologies, the generated deepfake videos cause a severe trust crisis in society. Existing methods prove that effective extraction of the artifacts introduced during the forgery process is essential for deepfake detection. However, since the features extracted by supervised binary classification contain a lot of artifact-irrelevant information, existing algorithms suffer severe performance degradation in the case of the mismatch between training and testing datasets. To overcome this issue, we propose an Artifacts-Disentangled Adversarial Learning (ADAL) framework to achieve accurate deepfake detection by disentangling the artifacts from irrelevant information. Furthermore, the proposed algorithm provides visual evidence by effectively estimating artifacts. Specifically, Multi-scale Feature Separator (MFS) in the disentanglement generator is designed to precisely transmit the artifact features and optimize the connection between the encoder and decoder. In addition, we design an Artifacts Cycle Consistency Loss (ACCL) which uses the disentangled artifacts to construct new samples and enables pixel-level supervised training for the generator to estimate more accurate artifacts. The symmetric discriminators are paralleled to differentiate the constructed samples from the original images in both fake and real domains, making the adversarial training process more stable. Extensive experiments on existing benchmarks demonstrate that the proposed method outperforms the state-of-the-art approaches.

An Audience Facilitates Facial Feedback: A Social-Context Hypothesis Reconciling Original Study and Nonreplication.

Nonreplications of previously undisputed phenomena tend to leave a theoretical vacuum. This theoretical perspective seeks to fill the gap left by the failure to replicate unobtrusive facial feedback. In the emblematic original study, participants who held a pen between the teeth (i.e., requiring activity of the zygomaticus major muscle) rated cartoons more positively than participants who held the pen between the lips. We argue that the same social mechanisms (e.g., the presence of an audience) modulate facial feedback to emotion as are involved in the feed-forward shaping of facial actions by emotions. Differing social contexts could thus help explain the contrast between original findings and failures to obtain unobtrusive facial feedback. An exploratory analysis that included results only from (unobtrusive) facial-feedback studies without explicit reference to emotion in the facial manipulation provided preliminary support for this hypothesis. Studies with a social context (e.g., due to experimenter presence) showed a medium-sized aggregate facial-feedback effect, whereas studies without a social context (e.g., when facial actions were only filmed), revealed a small effect. Video awareness strengthened facial feedback considerably within an engaging social context, but seemed to reduce it without a social context. We provisionally conclude that a (pro-)social interpretation of facial actions facilitates feedback to (primarily positive) emotion, and suggest further research explicitly manipulating this context.

Information-Containing Adversarial Perturbation for Combating Facial Manipulation Systems

With the development of deep learning technology, the facial manipulation system has become powerful and easy to use. Such systems can modify the attributes of the given facial images, such as hair color, gender, and age. Malicious applications of such systems pose a serious threat to individuals’ privacy and reputation. Existing studies have proposed various approaches to protect images against facial manipulations. Passive defense methods aim to detect whether the face is real or fake, which works for posterior forensics but can not prevent malicious manipulation. Initiative defense methods protect images upfront by injecting adversarial perturbations into images to disrupt facial manipulation systems but can not identify whether the image is fake. To address the limitation of existing methods, we propose a novel two-tier protection method named Information-containing Adversarial Perturbation (IAP), which provides more comprehensive protection for facial images. We use an encoder to map a facial image and its identity message to a cross-model adversarial example which can disrupt multiple facial manipulation systems to achieve initiative protection. Recovering the message in adversarial examples with a decoder serves passive protection, contributing to provenance tracking and fake image detection. We introduce a feature-level correlation measurement that is more suitable to measure the difference between the facial images than the commonly used mean squared error. Moreover, we propose a spectral diffusion method to spread messages to different frequency channels, thereby improving the robustness of the message against facial manipulation. Extensive experimental results demonstrate that our proposed IAP can recover the messages from the adversarial examples with high average accuracy and effectively disrupt the facial manipulation systems.