Avatar Generation Research Articles

FaceDefend: Copyright Protection to Prevent Face Embezzle

With the rapid evolution of deep learning and the advent of artificial intelligence, the metaverse has emerged as a significant technology. Within the metaverse, diverse elements such as rich applications and realistic digital avatars provide users with immersive experiences, but it poses a series of security problems. Current research predominantly focuses on the data storage and transmission processes from the perspective of blockchain and the Internet of Things to achieve the protection of the metaverse. However, there exists a gap in security research on the digital avatar generation process. Given that digital avatars are the primary entities engaging in social activities within the metaverse and are crafted based on real face images, the virtual character can be generated easily by stealing the user’s face image and controlled to interact with others. In order to deal with the above problems, we propose a novel method to prevent the misuse of faces, which maintains the security of the metaverse by protecting facial data and thus preventing its misuse. We explore the common architecture of generative models and propose a defense method based on copyright protection to prevent face embezzling. Firstly, we utilize the copyright protection module to obtain copyright protection information. Secondly, we utilized the defense control module to ensure the representation of the protected images occurs errors in the latent space of the generation model. Therefore the subsequent generation task output fails, which effectively protects the face data and prevents the generation of digital avatars. Furthermore, the results on public datasets and across multiple generative models present unnatural outputs, indicating the excellence of our defense and transfer capabilities.

Fostering Inclusive Recruitment Interviews with Intelligent Digital Humans

This article is about an intelligent digital human model enhanced by artificial intelligence, designed to meet the requirements from a multinational company in need of training for their human resources personnel on bias-free recruitment interviews. We have been creating a new generation of avatars with social intelligence, who are capable not only of presenting a wide variety of topics in a dynamic and engaging manner, but also of interacting with the audience and communicating emotions and moods. We have been customizing avatars for role plays, building them as real interlocutors who facilitate training in how to handle difficult conversations by including aspects such as non-verbal communication, different communication styles, and diversity and inclusion. Practicing conversations with avatars accelerates learning from experience without the risks associated with learning in the field. At the end of each interview, timely feedback is provided so learners can determine how to improve their performance. These digital humans are able to perform like realistic human beings, challenging the interviewer both at a verbal and para-verbal level, as well on the cognitive and the emotional levels – making it easy for the interviewer to get trapped into biases and false assumptions. The key message is this: diversity and inclusion best practices are first of all about mindset.

Expressive Talking Avatars.

Stylized avatars are common virtual representations used in VR to support interaction and communication between remote collaborators. However, explicit expressions are notoriously difficult to create, mainly because most current methods rely on geometric markers and features modeled for human faces, not stylized avatar faces. To cope with the challenge of emotional and expressive generating talking avatars, we build the Emotional Talking Avatar Dataset which is a talking-face video corpus featuring 6 different stylized characters talking with 7 different emotions. Together with the dataset, we also release an emotional talking avatar generation method which enables the manipulation of emotion. We validated the effectiveness of our dataset and our method in generating audio based puppetry examples, including comparisons to state-of-the-art techniques and a user study. Finally, various applications of this method are discussed in the context of animating avatars in VR.

Secure and Trustworthy Artificial Intelligence-extended Reality (AI-XR) for Metaverses

Metaverse is expected to emerge as a new paradigm for the next-generation Internet, providing fully immersive and personalized experiences to socialize, work, and play in self-sustaining and hyper-spatio-temporal virtual world(s). The advancements in different technologies such as augmented reality, virtual reality, extended reality (XR), artificial intelligence (AI), and 5G/6G communication will be the key enablers behind the realization of AI-XR metaverse applications. While AI itself has many potential applications in the aforementioned technologies (e.g., avatar generation, network optimization), ensuring the security of AI in critical applications like AI-XR metaverse applications is profoundly crucial to avoid undesirable actions that could undermine users’ privacy and safety, consequently putting their lives in danger. To this end, we attempt to analyze the security, privacy, and trustworthiness aspects associated with the use of various AI techniques in AI-XR metaverse applications. Specifically, we discuss numerous such challenges and present a taxonomy of potential solutions that could be leveraged to develop secure, private, robust, and trustworthy AI-XR applications. To highlight the real implications of AI-associated adversarial threats, we designed a metaverse-specific case study and analyzed it through the adversarial lens. Finally, we elaborate upon various open issues that require further research interest from the community.

A Study on the Algorithm Configuration for Sharing Career Experience in a Metaverse Platform

This study relates to a system for providing an experience sharing platform in a metaverse environment. In more detail, it is about an experience-sharing platform providing system that enables users to share and trade each other's experiences based on metaverse by allowing users to share information about their careers with each other in a metaverse environment. The algorithm proposed by this study is largely composed of a metaverse interface and a main server. The metaverse interface includes a motion generation unit that generates user motion information according to a signal received through a user terminal owned by a user, an avatar generation unit that generates an avatar that moves in conjunction with the user motion information, and a space generation unit that creates a virtual metaverse space for displaying at least one avatar. The main server is largely composed of input of requirements, matching, experience sharing, and cost payment modules and databases. In addition, the position control and area division module are also composed of the main server including a satisfaction reflecting movement, a virtual horizontal line generation, and an additional division processing parts, which are lower configuration algorithms of them.

A Comprehensive Framework for Face age Synthesis, Voice Transformation, and Cartoon Generation

This paper presents a comprehensive framework addressing age-invariant face recognition (AIFR), face age synthesis (FAS), voice transformation, and avatar generation. Traditional AIFR techniques focus on minimizing age-related variations in face recognition but lack visual interpretability, while FAS methods aim to synthesize faces of different ages but often compromise recognition due to artifacts. To address these limitations, we propose MTLFace, a multi-task learning framework that simultaneously handles AIFR and FAS tasks. MTLFace employs attention-based feature decomposition to separate identity and age-related features spatially, improving interpretability. Additionally, we introduce an identity conditional module for fine-grained face age synthesis, enhancing the naturalness of synthesized faces. Unlike conventional methods that achieve age group-level synthesis, our identity conditional module enables identity-level synthesis, resulting in smoother age transitions and preserving individual facial characteristics. Leveraging high-quality synthesized faces, we enhance AIFR performance via selective fine-tuning, where synthesized faces are used to augment training data, leading to improved robustness against age variations. Furthermore, we contribute a large cross-age face dataset with annotations, facilitating research in age-related tasks. In addition to AIFR and FAS, we explore limitations in existing methodologies for voice transformation and propose advancements to improve speech quality and address current challenges. Lastly, we introduce a novel approach for cartoon face generation, utilizing component-based facial feature extraction and template matching to produce diverse and stylistic cartoon faces. Experimental results across various benchmarks demonstrate the effectiveness of our approaches in achieving superior performance in face recognition, voice transformation, and cartoon face generation, thereby contributing to the advancement of multimodal synthesis technologies with potential applications in entertainment, virtual communication, and assistive technologies

Avatars as Digital Identity: A Case Study of Avatar in Facial Recognition Technology & eKYC by IndoAI

In today’s digital era, avatars have metamorphosized their conventional role, emerging as important tools for safeguarding digital identities in response to the prevalence of facial recognition technology. This article-case study explores the expanding importance of avatars, not only as representations of identity but as protective shields in the digital space. Avatars, once confined to gaming and virtual worlds, now play a crucial role in digital identity formation across various fields. The case study on avatar generation in facial recognition technology with eKYC highlights sophisticated security measures, and fusion of facial recognition, eKYC, artrimetrics and avatar authentication signals a new era where avatars play a pivotal role in retail industry. This study anticipates the growing integration of avatars into our social existence, offering innovative solutions for identity management and security in the digital realm. Scholars Middle East Publishers

Development of an Interactive Live Streaming System for Language Learning

<p>In this paper, we propose Virtual Avatar Interactive Live Streaming System (VAILSS), our approach to streaming 3D avatar performances in interactive storytelling applications using multimodal interactive techniques. This study focuses on developing a virtual live interaction system for VTubers in the context of language learning. A live virtual performance from our system consists of three components: 1) an Avatar generation framework, 2) an AI motion capture system, and 3) an interactive storytelling engine. The system integrates artificial intelligence and uses motion capture to identify facial expressions and movements, enabling virtual characters to provide live storytelling services. Additionally, the system allows for bi-directional interaction with players through tablet touch and voice. The system aims to promote multimodal learning channels for the masses. To gain deeper insights into the effectiveness of virtual role-playing in language learning, we organized a three-week workshop to investigate our system’s impact on user experience. We extended invitations to 17 night-school freshmen from the NFU Department of Applied Foreign Languages, enrolled in an English class focused on tour guides, to participate in a virtual performance. The experiment results demonstrated that the VAILSS positively affected students’ learning outcomes, particularly in enhancing foreign language acquisition.</p> <p> </p>

MuNeRF: Robust Makeup Transfer in Neural Radiance Fields.

There has been a high demand for facial makeup transfer tools in fashion e-commerce and virtual avatar generation. Most of the existing makeup transfer methods are based on the generative adversarial networks. Despite their success in makeup transfer for a single image, they struggle to maintain the consistency of makeup under different poses and expressions of the same person. In this paper, we propose a robust makeup transfer method which consistently transfers the makeup style of a reference image to facial images in any poses and expressions. Our method introduces the implicit 3D representation, neural radiance fields (NeRFs), to ensure the geometric and appearance consistency. It has two separate stages, including one basic NeRF module to reconstruct the geometry from the input facial image sequence, and a makeup module to learn how to transfer the reference makeup style consistently. We propose a novel hybrid makeup loss which is specially designed based on the makeup characteristics to supervise the training of the makeup module. The proposed loss significantly improves the visual quality and faithfulness of the makeup transfer effects. To better align the distribution between the transferred makeup and the reference makeup, a patch-based discriminator that works in the pose-independent UV texture space is proposed to provide more accurate control of the synthesized makeup. Extensive experiments and a user study demonstrate the superiority of our network for a variety of different makeup styles.

Animated Avatar Generation Technology Research Based on Deep Convolutional Generative Adversarial Network Integrated With Self-Attention and Spectral Normalization

Animated Avatar Generation Technology Research Based on Deep Convolutional Generative Adversarial Network Integrated With Self-Attention and Spectral Normalization

HDHumans

Photo-real digital human avatars are of enormous importance in graphics, as they enable immersive communication over the globe, improve gaming and entertainment experiences, and can be particularly beneficial for AR and VR settings. However, current avatar generation approaches either fall short in high-fidelity novel view synthesis, generalization to novel motions, reproduction of loose clothing, or they cannot render characters at the high resolution offered by modern displays. To this end, we propose HDHumans, which is the first method for HD human character synthesis that jointly produces an accurate and temporally coherent 3D deforming surface and highly photo-realistic images of arbitrary novel views and of motions not seen at training time. At the technical core, our method tightly integrates a classical deforming character template with neural radiance fields (NeRF). Our method is carefully designed to achieve a synergy between classical surface deformation and a NeRF. First, the template guides the NeRF, which allows synthesizing novel views of a highly dynamic and articulated character and even enables the synthesis of novel motions. Second, we also leverage the dense pointclouds resulting from the NeRF to further improve the deforming surface via 3D-to-3D supervision. We outperform the state of the art quantitatively and qualitatively in terms of synthesis quality and resolution, as well as the quality of 3D surface reconstruction.

Pose-Guided 3D Human Generation in Indoor Scene

In this work, we address the problem of scene-aware 3D human avatar generation based on human-scene interactions. In particular, we pay attention to the fact that physical contact between a 3D human and a scene (i.e., physical human-scene interactions) requires a geometrical alignment to generate natural 3D human avatar. Motivated by this fact, we present a new 3D human generation framework that considers geometric alignment on potential contact areas between 3D human avatars and their surroundings. In addition, we introduce a compact yet effective human pose classifier that classifies the human pose and provides potential contact areas of the 3D human avatar. It allows us to adaptively use geometric alignment loss according to the classified human pose. Compared to state-of-the-art method, our method can generate physically and semantically plausible 3D humans that interact naturally with 3D scenes without additional post-processing. In our evaluations, we achieve the improvements with more plausible interactions and more variety of poses than prior research in qualitative and quantitative analysis. Project page: https://bupyeonghealer.github.io/phin/.

K-SMPL: 한국인 체형 데이터 기반의 매개화된 인체 모델

The Skinned Multi-Person Linear Model (SMPL) is the most widely used parametric 3D Human Model optimized and learned from CAESAR, a 3D human scanned database created with measurements from 3,800 people living in United States in the 1990s. We point out the lack of racial diversity of body types in SMPL and propose K-SMPL that better represents Korean 3D body shapes. To this end, we develop a fitting algorithm to estimate 2,773 Korean 3D body shapes from Korean body measurement data. By conducting principle component analysis to the estimated Korean body shapes, we construct K-SMPL model that can generate various Korean body shape in 3D. K-SMPL model allows to improve the fitting accuracy over SMPL with respect to the Korean body measurement data. K-SMPL model can be widely used for avatar generation and human shape fitting for Korean.

AvatarCLIP

3D avatar creation plays a crucial role in the digital age. However, the whole production process is prohibitively time-consuming and labor-intensive. To democratize this technology to a larger audience, we propose AvatarCLIP, a zero-shot text-driven framework for 3D avatar generation and animation. Unlike professional software that requires expert knowledge, AvatarCLIP empowers layman users to customize a 3D avatar with the desired shape and texture, and drive the avatar with the described motions using solely natural languages. Our key insight is to take advantage of the powerful vision-language model CLIP for supervising neural human generation, in terms of 3D geometry, texture and animation. Specifically, driven by natural language descriptions, we initialize 3D human geometry generation with a shape VAE network. Based on the generated 3D human shapes, a volume rendering model is utilized to further facilitate geometry sculpting and texture generation. Moreover, by leveraging the priors learned in the motion VAE, a CLIP-guided reference-based motion synthesis method is proposed for the animation of the generated 3D avatar. Extensive qualitative and quantitative experiments validate the effectiveness and generalizability of AvatarCLIP on a wide range of avatars. Remarkably, AvatarCLIP can generate unseen 3D avatars with novel animations, achieving superior zero-shot capability. Codes are available at https://github.com/hongfz16/AvatarCLIP.

Controlling the Sense of Embodiment for Virtual Avatar Applications: Methods and Empirical Study.

BackgroundThe sense of embodiment (SoE) is the feeling of one’s own body, and research on the SoE extends from the rubber hand illusion to the full-body ownership illusion with a virtual avatar.ObjectiveThe key to utilizing a virtual avatar is understanding and controlling the SoE, and it can be extended to several medical applications. In this study, we aimed to clarify these aspects by considering the following three subcomponents of SoE: sense of agency, ownership, and self-location.MethodsWe defined a human avatar (HA), point light avatar (PLA), and out-of-body point light avatar (OBPLA) and compared them in three user studies. In study 1, 28 participants were recruited and the three avatar conditions (HA, PLA, and OBPLA) were compared. In study 2, 29 new participants were recruited, and there were two avatar conditions (HA ad PLA) and two motion synchrony conditions (synchrony and asynchrony). In study 3, 29 other participants were recruited, and there were two avatar conditions (PLA and OBPLA) and two motion synchrony conditions (synchrony and asynchrony). Dependent measures included sense of agency, ownership, and self-location; emotional response; presence; and simulator sickness.ResultsThe findings of study 1 showed that the three avatar generation methodologies can control the sense of ownership and self-location in a stepwise manner while maintaining a high sense of agency. In studies 2 and 3, we found dependencies among the three subcomponents of SoE and observed that they affected users’ subjective experiences.ConclusionsOur findings may have implications for boosting the effects of virtual avatar applications in medical areas, by understanding and controlling the SoE with a full-body illusion.

A Study on the Body Shape Analysis for an Avatar Generation of the Virtual Fitting System-Focusing on Korean Women in their 20's-

In the virtual fitting system, the use of a 3D avatar is not a simple garment model, but it should be able to reproduce the size and shape of the customer using a fitting system. Although various virtual fitting systems have their own 3D avatar sizing systems and provide 3D avatars that match the size of the customer, there are limitations in realizing the actual body shape in actual use by the consumer. The purpose of this study is to realize a 3D avatar with excellent size and conformity for customer use. Therefore, this study aims to provide basic data for the formation of a 3D standard avatar of Korean women aged in their 20's, by comparing and analyzing the degree of the consumer user friendly system change of a body type, and the consumer’s ability in selecting a consumer representative body type. Based on the survey data of 'Size Korea' conducted from 2004 to 2015 at three times, we examined the change of body shape over 10 years. Then, based on the results of 6th and 7th data, 4 factors of the concurrent body shape change of women of the consumer demographic studied were selected through the use of a factor analysis. Following this analysis, the 4 extracted factors were clustered again and finally released 7 representative body types, which were obtained based on height and weight. The size of each representative figure is derived by the use of a regression analysis, and it is used as a basic data for 3D avatar formation of the virtual fitting system.

A Study on i-Fashion 3D Avatar's Consumer satisfaction & Comparison of 3D and Direct Masurement - Based of Domestic University Students

This research is to understand customer satisfaction with virtual fitting based on a 3D body scanner and avatars as well as differences between avatars and the 'real me'. To this end, this research examined Korean college students to facilitate 3D body scanning, avatar generation and surveys. The author used 3D body scan data with direct measurements to identify differences between the 3D body scan data-based 'my avatar' and 'real me' in the virtual dress fitting system. The survey results on 'the level of customer awareness on 3D body scanner' found that the majority of both genders did not know about it and indicated a lower usability to incorporate IT technology into the fashion industry. The question in the 3D body scanning and avatar found an affirmative attitude. Satisfaction levels on the 3D avatars' similarity with 'own body' and garment fitting were positive and indicated a need for further technological improvements to express the avatars identical to customers' own body. More research is necessary for the accuracy of sizes for 3D body scanning that measure body sizes while wearing clothes. Avatars based on such datamay be less similar to 'own body' and cause customer dissatisfaction. Thus, further technology development is required to narrow gaps using data to make avatars that provide more accurate virtual fitting simulation services to customers.

Cartoon Avatar Processing Based on Image Edge Detection

Currently, face recognition plays an important role in many applications ranging from access control, suspect identification to automatic target recognition. Compared to these mature face recognition techniques, another kind of similar technique, which transfers human faces to abstract cartoon faces, is currently neglected until recent years. This paper mainly presents the approach on converting real face photos to cartoon pictures. In this paper, edge detection methods are used to for face and feature recognition, and feature-based sense organs matching methods are also developed for cartoon avatar generation according to facial feature correlation of the cartoon image galleries.