Avatar Creation Research Articles

Building Consistent Characters through Open-Source Generative AI

This paper presents a pipeline for creating consistent characters using open-source generative AI tools, aimed at enhancing educational content. Through the use of XTTS, Fooocus, and FaceFusion, a talking avatar of the marine biologist Raffaele Issel was developed, accurately replicating his facial and vocal features. This study demonstrates how these tools can generate high-quality, engaging educational materials with minimal technical expertise and resources. The methodology facilitates the creation of interactive avatars for use in graphic novels and comic books, enriching museum and educational experiences.

Global point cloud registration network for large transformations

Three-dimensional registration is an established yet challenging problem that is key in many different applications, such as mapping the environment for autonomous vehicles, or modeling people for avatar creation, among others. Registration refers to the process of mapping multiple data into the same coordinate system by means of matching correspondences and transformation estimation. Novel proposals exploit the benefits of deep learning architectures for this purpose, as they learn the best features for the data, providing better matches and hence results. However, the state of the art is usually focused on cases of relatively small transformations, although in certain applications and in a real and practical environment, large transformations are very common. In this paper, we present ReLaTo (Registration for Large Transformations), an architecture that addresses the cases where large transformations happen while maintaining good performance for local transformations. This proposal uses a novel Softmax pooling layer to find correspondences in a bilateral consensus manner between two point sets, sampling the most confident matches. These matches estimate a coarse and global registration using weighted Singular Value Decomposition (SVD). A target-guided denoising step is applied to both the obtained matches and latent features to estimate the final fine registration considering the local geometry. All these steps are carried out following an end-to-end approach, which has been shown to perform better than 7 state-of-the-art registration methods in two datasets commonly used for this task (ModelNet40 and the Karlsruhe Institute of Technology and Toyota Technological Institute dataset, KITTI), especially in the case of large transformations.Graphic abstract

Mesh representation matters: investigating the influence of different mesh features on perceptual and spatial fidelity of deep 3D morphable models

BackgroundDeep 3D morphable models (deep 3DMMs) play an essential role in computer vision. They are used in facial synthesis, compression, reconstruction and animation, avatar creation, virtual try-on, facial recognition systems and medical imaging. These applications require high spatial and perceptual quality of synthesised meshes. Despite their significance, these models have not been compared with different mesh representations and evaluated jointly with point-wise distance and perceptual metrics. MethodsWe compare the influence of different mesh representation features to various deep 3DMMs on spatial and perceptual fidelity of the reconstructed meshes. This paper proves the hypothesis that building deep 3DMMs from meshes represented with global representations leads to lower spatial reconstruction error measured with L1 and L2 norm metrics and underperforms on perceptual metrics. In contrast, using differential mesh representations which describe differential surface properties yields lower perceptual FMPD and DAME and higher spatial fidelity error. The influence of mesh feature normalisation and standardisation is also compared and analysed from perceptual and spatial fidelity perspectives. ResultsThe results presented in this paper provide guidance in selecting mesh representations to build deep 3DMMs accordingly to spatial and perceptual quality objectives and propose combinations of mesh representations and deep 3DMMs which improve either perceptual or spatial fidelity of existing methods.

3DFaceShop: Explicitly Controllable 3D-Aware Portrait Generation.

In contrast to the traditional avatar creation pipeline which is a costly process, contemporary generative approaches directly learn the data distribution from photographs. While plenty of works extend unconditional generative models and achieve some levels of controllability, it is still challenging to ensure multi-view consistency, especially in large poses. In this work, we propose a network that generates 3D-aware portraits while being controllable according to semantic parameters regarding pose, identity, expression and illumination. Our network uses neural scene representation to model 3D-aware portraits, whose generation is guided by a parametric face model that supports explicit control. While the latent disentanglement can be further enhanced by contrasting images with partially different attributes, there still exists noticeable inconsistency in non-face areas when animating expressions. We solve this by proposing a volume blending strategy in which we form a composite output by blending dynamic and static areas, with two parts segmented from the jointly learned semantic field. Our method outperforms prior arts in extensive experiments, producing realistic portraits with vivid expression in natural lighting when viewed from free viewpoints. It also demonstrates generalization ability to real images as well as out-of-domain data, showing great promise in real applications.

Does Sense of Presence Affect People's Politeness? Comparing VRChat and Soul App

Mobile social activities are among the most popular forms of socialization today, and has become an integral part of people's daily lives. With advances in technology and the rise of affordable virtual reality devices, VR social interaction through avatar creation and support for advanced body tracking capabilities has the potential to lead the next revolution in the digitization of social activities. However, whether in VR or on mobile phones, user politeness and friendliness are always crucial factors influencing the user experience of social applications. While past research has examined online politeness, less attention has been paid to comparing politeness perceptions between social VR and mobile applications. In the meantime, given that the core characteristics of VR socializing is its heightened sense of presence, research by many scholars in the relevant field also indicates that presence often positively correlates with positive emotions. Therefore, this study will explore how people perceive and express verbal and non-verbal politeness behaviors in different social applications (VR chat, Soul app), as well as investigate how the sense of presence provided by the devices influences people's expression of politeness. The study's results identified the polite and impolite behaviors that people had perceived in VR and the mobile App. Lastly, the study also reported that VR applications that offer a heightened sense of presence can better facilitate politeness.

Formulating facial mesh tracking as a differentiable optimization problem: a backpropagation-based solution

Facial mesh tracking enables the production of topologically consistent 3D facial meshes from stereo video input captured by calibrated cameras. This technology is an integral part of many digital human applications, such as personalized avatar creation, audio-driven 3D facial animation, and talking face video generation. Currently, most facial mesh tracking methods are built on computer graphics techniques, which involve complex procedures and often necessitate human annotation within pipelines. As a result, these approaches are difficult to implement and hard to generalize across various scenarios. We propose a backpropagation-based solution that formulates facial mesh tracking as a differentiable optimization problem called the BPMT. Our solution leverages visual clues extracted from the stereo input to estimate vertex-wise geometry and texture information. The BPMT is composed of two steps: automatic face analysis and mesh tracking. In the first step, a range of visual clues are automatically extracted from the input, including facial point clouds, multi-view 2D landmarks, 3D landmarks in the world coordinate system, motion fields, and image masks. The second step can be viewed as a differentiable optimization problem, with constraints comprising stereo video input and facial clues. The primary objective is to achieve topologically consistent 3D facial meshes across frames. Additionally, the parameters to be optimized encompass the positions of free-form deformed vertices and a shared texture UV map. Furthermore, the 3D morphable model (3DMM) is introduced as a form of regularization to enhance the convergence of the optimization process. Leveraging the fully developed backpropagation software, we progressively register the facial meshes to the recorded object, generating high-quality 3D faces with consistent topologies. The BPMT requires no manual labeling within the pipeline, making it suitable for producing large-scale stereo facial data. Moreover, our method exhibits a high degree of flexibility and extensibility, positioning it as a promising platform for future research in the community.

Creating a metaverse‐me: Exploring the consumer avatar creation process

AbstractThe metaverse is emerging as a focal point for technological advancement, social interaction, and commerce. Despite brands' enthusiasm for establishing a presence in the metaverse, consumer adoption remains modest, with the initial hurdle being avatar creation. To better understand why consumers are hesitant to engage in the metaverse, this study focuses on avatars as virtual representations of self, particularly during the stage of avatar creation. Drawing on theories of self‐presentation and identity formation, this research investigates how users conceptualise and express their virtual avatars. Through a series of in‐depth semi‐structured interviews, we propose an integrated framework that illustrates how consumers engage in a complex avatar creation process consisting of consolidation, exploration, affirmation, and aspiration. Further, our insights show that avatars serve as extensions of real‐world identities, as well as how this impacts consumption behaviour. These insights are valuable for marketers aiming to develop actionable consumer‐centric metaverse strategies, and researchers investigating consumer behaviour within immersive consumption environments.

Customized Avatars In A Multiplatform Game On Mobile And Virtual Reality For Hospitalized Children In Hemato-Oncology: A Conceptual Design

The hospital is an environment that may induce both anxiety and pain in hospitalized children and their families. The use of games and distraction offers a solution to better manage anxiety and pain. The objective of the present paper is to introduce “A Friend for Life,” a multi-platform game based on the creation of customized avatars to help children hospitalized in the hematology-oncology unit cope with their everyday pain and anxiety. This smartphone and virtual reality application uses the IKEA Effect and transmedia storytelling to improve child engagement in the game. This game was developed as part of AVATAR, a research project that combines collaborative action research with co-design. The project is led by our multidisciplinary team with relevant experience in nursing, pediatrics, hematology-oncology, and new technology. Also, our team includes a former patient and a former patient’s parent. In this paper, we introduce our serious game, “A Friend for Life”, and the collaborative development method that was used in the prototyping phase. Additionally, we will outline our hypothetical conceptual framework and its underlying concepts. Finally, we discuss the game's limitations and directions for future research.

Adolescent Female Users' Avatar Creation in Social Virtual Worlds: Opportunities and Challenges.

Many adolescent females are active online, finding creative ways to express themselves through evolving social media technologies. Social virtual worlds (SVWs), distinguished by extensive avatar customization features, provide them with unique opportunities to craft virtual identities and explore diverse facets of self-presentation. This study investigates adolescent females' construction of avatars in Zepeto, a South Korea-based global SVW platform. Employing social representation theory (SRT) as a theoretical framework, this study conducted in-depth interviews with adolescent female Zepeto users to uncover their perceptions of and motivations behind avatar creation in SVWs, as well as to explore how the interplay of virtual and real worlds presents them with various opportunities and challenges. While the prospects of SVWs remain uncertain, examining how these young users interpret and experience SVWs contributes to identifying potential strategies to enhance the sustainability of these platforms.

Storying My Body in Bits and Bytes

ABSTRACT This study shows how cyberautoethnography can be used to critically interrogate and make sense of how technology and attendant artifacts shape our understanding of embodiment. Specifically, this goal is achieved by applying cyberautoethnography to study Bitmoji avatars, a digital artifact, as a type of digital embodiment. Users can personalize Bitmoji avatars by choosing from a variety of skin tones, body sizes, and other physical characteristics. Thus, similar to the creation of other avatars (such as those in social media and gaming environments), creating and circulating these digital artifacts can be understood as acts of strategic digital (re)presentation and embodiment. Here, I reflect on the process of creating my Bitmoji avatar and analyze conversations using Bitmojis to discuss how I use my Bitmoji to make my body hypervisible in a context that’s typically understood as disembodied. I conclude that the seemingly mundane use of Bitmojis can be understood as a feminist practice that marginalized individuals might engage with in order to extend the political dimensions of everyday, lived experiences into the digital realm. Thus, this critical cyberautoethnography showcases the value of autoethnography for understanding how agency is claimed and limited by contemporary forms of technology-mediated embodiment.

3D Reconstruction and Semantic Modeling of Eyelashes

AbstractHigh‐fidelity digital human modeling has become crucial in various applications, including gaming, visual effects and virtual reality. Despite the significant impact of eyelashes on facial aesthetics, their reconstruction and modeling have been largely unexplored. In this paper, we introduce the first data‐driven generative model of eyelashes based on semantic features. This model is derived from real data by introducing a new 3D eyelash reconstruction method based on multi‐view images. The reconstructed data is made available which constitutes the first dataset of 3D eyelashes ever published. Through an innovative extraction process, we determine the features of any set of eyelashes, and present detailed descriptive statistics of human eyelashes shapes. The proposed eyelashes model, which exclusively relies on semantic parameters, effectively captures the appearance of a set of eyelashes. Results show that the proposed model enables interactive, intuitive and realistic eyelashes modeling for non‐experts, enriching avatar creation and synthetic data generation pipelines.

Relightable and Animatable Neural Avatars from Videos

Lightweight creation of 3D digital avatars is a highly desirable but challenging task. With only sparse videos of a person under unknown illumination, we propose a method to create relightable and animatable neural avatars, which can be used to synthesize photorealistic images of humans under novel viewpoints, body poses, and lighting. The key challenge here is to disentangle the geometry, material of the clothed body, and lighting, which becomes more difficult due to the complex geometry and shadow changes caused by body motions. To solve this ill-posed problem, we propose novel techniques to better model the geometry and shadow changes. For geometry change modeling, we propose an invertible deformation field, which helps to solve the inverse skinning problem and leads to better geometry quality. To model the spatial and temporal varying shading cues, we propose a pose-aware part-wise light visibility network to estimate light occlusion. Extensive experiments on synthetic and real datasets show that our approach reconstructs high-quality geometry and generates realistic shadows under different body poses. Code and data are available at https://wenbin-lin.github.io/RelightableAvatar-page.

AvatarVerse: High-Quality & Stable 3D Avatar Creation from Text and Pose

Creating expressive, diverse and high-quality 3D avatars from highly customized text descriptions and pose guidance is a challenging task, due to the intricacy of modeling and texturing in 3D that ensure details and various styles (realistic, fictional, etc). We present AvatarVerse, a stable pipeline for generating expressive high-quality 3D avatars from nothing but text descriptions and pose guidance. In specific, we introduce a 2D diffusion model conditioned on DensePose signal to establish 3D pose control of avatars through 2D images, which enhances view consistency from partially observed scenarios. It addresses the infamous Janus Problem and significantly stablizes the generation process. Moreover, we propose a progressive high-resolution 3D synthesis strategy, which obtains substantial improvement over the quality of the created 3D avatars. To this end, the proposed AvatarVerse pipeline achieves zero-shot 3D modeling of 3D avatars that are not only more expressive, but also in higher quality and fidelity than previous works. Rigorous qualitative evaluations and user studies showcase AvatarVerse's superiority in synthesizing high-fidelity 3D avatars, leading to a new standard in high-quality and stable 3D avatar creation. Our project page is: https://avatarverse3d.github.io/ .

Avatar creation in the metaverse: A focus on event expectations

The metaverse has revolutionized the way event experiences are virtualized, as this cutting-edge technology is especially capable of fostering dynamic interactivity among avatars. However, few empirical studies have explored the creation of avatars for virtual events in the metaverse. This research delves into the theme in a mixed-methods approach. The results of semi-structured interviews with participants in a conference held in the metaverse highlighted that they would create a single, unique avatar, whose style would be adapted to the characteristics of future events. These findings were assessed later in an online experiment, in which participants created their own avatars for a (utilitarian or hedonic) virtual event in the metaverse. The results confirmed that users would create avatars with formal/informal styles depending on the utilitarian/hedonic value they expected to derive from the event. In addition, the expectancy they feel about the metaverse experience prompts attendees to design avatars similar to themselves, which significantly increases their satisfaction with their digital personas. The technical capabilities of the platform (ease of use and customization) seem also to play an important role in enhancing satisfaction with the avatar. This research suggests specific actions that might be taken to ensure that attendees of virtual events in the metaverse are satisfied with their avatar designs.

TEACHING DIGITAL IDENTITY: OPPORTUNITIES, CHALLENGES, AND ETHICAL CONSIDERATIONS FOR AVATAR CREATION IN EDUCATIONAL SETTINGS

This article serves intends to serve as a guide aimed at both educators and students, focusing on the rapidly evolving field of avatar creation within creative practices. As tools driven by artificial intelligence (AI) democratise the digital realm, we see a surge in the presence of virtual influencers and V-Tubers who use avatars for a myriad of applications. This unfolding landscape offers significant opportunities for personal and artistic expression but also poses ethical challenges and risks that demand careful consideration. Looking towards opportunities brought by advancements in synthetic media, such as easier access, diversity of representation, collaborative possibilities, and heightened realism in avatar creation. Alongside opportunities, the paper investigates ethical questions related to deepfakes, cultural sensitivities, and the need for stringent data privacy measures. Within the scope of this article, synthetic media is understood as computer-generated content that allows individuals to create and adapt media overcoming specialised skills or training required by earlier generations of media producers. The objective of this article is two-fold: to equip educators and their students with actionable knowledge and best practices, and to guide them in harnessing synthetic media's positive potential responsibly. Through education we aim to foster a digital environment that is safer, more inclusive, and ethically sound for future generations. It also outlines best practices for ethical avatar creation, focusing on transparency, data privacy, diversity, fairness, quality, and responsibility. By adhering to these guidelines, we may navigate complexities of this emerging field while minimising its potential negative impacts on society.

FLARE: Fast Learning of Animatable and Relightable Mesh Avatars

Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. While 3D meshes enable efficient processing and are highly portable, they lack realism in terms of shape and appearance. Neural representations, on the other hand, are realistic but lack compatibility and are slow to train and render. Our key insight is that it is possible to efficiently learn high-fidelity 3D mesh representations via differentiable rendering by exploiting highly-optimized methods from traditional computer graphics and approximating some of the components with neural networks. To that end, we introduce FLARE, a technique that enables the creation of animatable and relightable mesh avatars from a single monocular video. First, we learn a canonical geometry using a mesh representation, enabling efficient differentiable rasterization and straightforward animation via learned blendshapes and linear blend skinning weights. Second, we follow physically-based rendering and factor observed colors into intrinsic albedo, roughness, and a neural representation of the illumination, allowing the learned avatars to be relit in novel scenes. Since our input videos are captured on a single device with a narrow field of view, modeling the surrounding environment light is non-trivial. Based on the split-sum approximation for modeling specular reflections, we address this by approximating the prefiltered environment map with a multi-layer perceptron (MLP) modulated by the surface roughness, eliminating the need to explicitly model the light. We demonstrate that our mesh-based avatar formulation, combined with learned deformation, material, and lighting MLPs, produces avatars with high-quality geometry and appearance, while also being efficient to train and render compared to existing approaches.

EmoStory: Emotion Prediction and Mapping in Narrative Stories

A well-designed story is built upon a sequence of plots and events. Each event has its purpose in piquing the audience's interest in the plot; thus, understanding the flow of emotions within the story is vital to its success. A story is usually built up through dramatic changes in emotion and mood to create resonance with the audience. The lack of research in this understudied field warrants exploring several aspects of the emotional analysis of stories. In this paper, we propose an encoder-decoder framework to perform sentence-level emotion recognition of narrative stories on both dimensional and categorical aspects, achieving MAE=0.0846 and 54% accuracy (8-class), respectively, on the EmoTales dataset and a reasonably good level of generalization to an untrained dataset. The first use of attention and multi-head attention mechanisms for emotion representation mapping (ERM) yields state-of-the-art performance in certain settings. We further present the preliminary idea of EmoStory, a concept that seamlessly predicts both dimensional and categorical space in an efficient manner, made possible with ERM. This methodology is useful in only one of the two aspects is available. In the future, these techniques could be extended to model the personality or emotional state of characters in stories, which could benefit the affective assessment of experiences and the creation of emotive avatars and virtual worlds

"To Be or Not to Be Me?": Exploration of Self-Similar Effects ofAvatars on Social Virtual Reality Experiences.

The growing interest in the self-similarity effect of avatars in virtual reality (VR) has spurred the creation of realistic avatars that closely mirror their users. However, despite extensive research on the self-similarity effect in single-user VR environments, our understanding of its impact in social VR settings remains underdeveloped. This shortfall exists despite the unique socio-psychological phenomena arising from the illusion of embodiment that could potentially alter these effects. To fill this gap, this paper provides an in-depth empirical investigation of how avatars' self-similarity influences social VR experiences. Our research uncovers several notable findings: 1) A high level of avatar self-similarity boosts users' sense of embodiment and social presence but has minimal effects on the overall presence and even slightly hinders immersion. These results are driven by increased self-awareness. 2) Among various factors that contribute to the self-similarity of avatars, voice stands out as a significant influencer of social VR experiences, surpassing other representational factors. 3) The impact of avatar self-similarity shows negligible differences between male and female users. Based on these findings, we discuss the pros and cons of incorporating self-similarity into social VR avatars. Our study serves as a foundation for further research in this field.

4DHumanOutfit: A multi-subject 4D dataset of human motion sequences in varying outfits exhibiting large displacements

We present a new dataset of densely sampled spatio-temporal 4D human motion data of different actors, outfits and motions. The dataset contains different actors wearing different outfits while performing different motions in each outfit. It samples the space of 4D human motion along 3 axes with identity, outfit and motion, therefore providing a cube of data where each identity is represented by a planar slice of the cube with all outfits and for all motions. The dataset has numerous potential applications for the processing and creation of digital humans including augmented reality, avatar creation and virtual try on. 4DHumanOutfit is released for research purposes at https://kinovis.inria.fr/4dhumanoutfit/. In addition to image data and 4D reconstructions, the dataset includes reference solutions for each axis. We present independent baselines along each axis that demonstrate the value of these reference solutions for evaluation tasks.

A user experience perspective on heritage tourism in the metaverse: Empirical evidence and design dilemmas for VR

The tourism industry will be shaped by the growth and development of the metaverse in the coming decades. Virtual reality (VR) will enable the creation of virtual worlds, avatars, digital twins, and new social networks. These technologies can be utilized in tourism to enable travelers to preview real-world experiences, to enhance experiences while on-site, to relive experiences after travel, or in some cases to even substitute for travel. Given the metaverse’s transformative potential, empirical investigation of VR is clearly warranted. Studies of VR tourism typically choose a single specific VR application and investigate its impact on adoption or user satisfaction. This application-level focus is a significant limitation. We therefore evaluate multiple heritage tourism applications as well as VR hardware in a comprehensive and structured analysis from the user experience (UX) perspective. Our content analysis of user interviews reveals 13 factors that shape users’ overall perceptions about VR. These factors are grouped into categories related to presentation of the VR content, the content itself, and the functionality of the hardware and software. Our analysis also reveals three design dilemmas for creators of VR heritage tourism content for the metaverse. Implications and an agenda for future research are included.