3D Avatar Research Articles

High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field

One crucial aspect of 3D head avatar reconstruction lies in the details of facial expressions. Although recent NeRF-based photo-realistic 3D head avatar methods achieve high-quality avatar rendering, they still encounter challenges retaining intricate facial expression details because they overlook the potential of specific expression variations at different spatial positions when conditioning the radiance field. Motivated by this observation, we introduce a novel Spatially-Varying Expression (SVE) conditioning. The SVE can be obtained by a simple MLP-based generation network, encompassing both spatial positional features and global expression information. Benefiting from rich and diverse information of the SVE at different positions, the proposed SVE-conditioned NeRF can deal with intricate facial expressions and achieve realistic rendering and geometry details of high-fidelity 3D head avatars. Additionally, to further elevate the geometric and rendering quality, we introduce a new coarse-to-fine training strategy, including a geometry initialization strategy at the coarse stage and an adaptive importance sampling strategy at the fine stage. Extensive experiments indicate that our method outperforms other state-of-the-art (SOTA) methods in rendering and geometry quality on mobile phone-collected and public datasets. Code and data can be found at https://github.com/minghanqin/AvatarSVE.

Abstract 154: Diffuse midline glioma invasion and metastasis rely on cell-autonomous signaling through a phenotypic switch controlled by BMP7

Abstract Diffuse midline gliomas (DMG) are pediatric tumors with negligible two-year survival after diagnosis characterized by their ability to infiltrate the central nervous system. In the hope of controlling the local growth and slowing the disease all patients receive radiotherapy. However, distant progression occurs frequently in DMG patients. Current clues as to what causes tumor infiltration circle mainly around the tumor microenvironment, but there are currently no known determinants to predict the degree of invasiveness. We use patient-derived glioma stem cells (GSCs) to create patient-specific 3D avatars to model interindividual invasion and elucidate the cellular supporting mechanisms. We show that 3D models mirror the invasive behavior of the parental tumors, thus proving the ability of DMG to infiltrate as an autonomous characteristic of tumor cells. Furthermore, we distinguished two modes of migration, mesenchymal and amoeboid-like, and associated the amoeboid-like modality with GSCs derived from the most invasive tumors. Using transcriptomics of both organoids and primary tumors, we further characterized the invasive amoeboid-like tumors as oligodendrocyte progenitor-like, with highly contractile cytoskeleton and reduced adhesion ability. Moreover, we identified the cell-autonomous BMP7-related signaling as a major regulator of DMG motility and invasion, controlling a switch between two migration modalities. Finally, we deciphered MEK, ERK and Rho/ROCK kinases activated downstream of the BMP7 stimulation as actionable targets controlling tumor cell motility. Our findings identify two new therapeutic avenues. First, patient-derived GSCs represent a predictive tool for stratification of patient with increased risk of metastatic progression in order to adapt their irradiation strategies within weeks after diagnosis. Second, autocrine and short-range BMP7-related signaling becomes a druggable target to prevent DMG spread and metastasis. Citation Format: Marco Bruschi, Lilia Midjek, Yassine Ajlil, Stephanie Vairy, Manon Lancien, Samia Ghermaoui, Thomas Kergrohen, Maite Verreault, Ahmed Idbaih, Carlos Alberto Oliveira de Biagi, Ilon Liu, Mariella G. Filbin, Kevin Beccaria, Thomas Blauwblomme, Stephanie Puget, Arnault Tauziede-Espariat, Pascale Varlet, Volodia Dangouloff-Ros, Nathalie Boddaert, Gwenael Le Teuff, Jacques Grill, Guillaume Montagnac, Nadia Elkathib, Marie-Anne Debily, David Castel. Diffuse midline glioma invasion and metastasis rely on cell-autonomous signaling through a phenotypic switch controlled by BMP7 [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 154.

Development of Wearable Devices for Collecting Digital Rehabilitation/Fitness Data from Lower Limbs.

Lower extremity exercises are considered a standard and necessary treatment for rehabilitation and a well-rounded fitness routine, which builds strength, flexibility, and balance. The efficacy of rehabilitation programs hinges on meticulous monitoring of both adherence to home exercise routines and the quality of performance. However, in a home environment, patients often tend to inaccurately report the number of exercises performed and overlook the correctness of their rehabilitation motions, lacking quantifiable and systematic standards, thus impeding the recovery process. To address these challenges, there is a crucial need for a lightweight, unbiased, cost-effective, and objective wearable motion capture (Mocap) system designed for monitoring and evaluating home-based rehabilitation/fitness programs. This paper focuses on the development of such a system to gather exercise data into usable metrics. Five radio frequency (RF) inertial measurement unit (IMU) devices (RF-IMUs) were developed and strategically placed on calves, thighs, and abdomens. A two-layer long short-term memory (LSTM) model was used for fitness activity recognition (FAR) with an average accuracy of 97.4%. An intelligent smartphone algorithm was developed to track motion, recognize activity, and calculate key exercise variables in real time for squat, high knees, and lunge exercises. Additionally, a 3D avatar on the smartphone App allows users to observe and track their progress in real time or by replaying their exercise motions. A dynamic time warping (DTW) algorithm was also integrated into the system for scoring the similarity in two motions. The system's adaptability shows promise for applications in medical rehabilitation and sports.

A systematic literature review of generative adversarial networks (GANs) in 3D avatar reconstruction from 2D images

A systematic literature review of generative adversarial networks (GANs) in 3D avatar reconstruction from 2D images

A Proposal to Support Schizophrenia Patients: Mobile Application Based on Binaural Beats and 3D Avatar

A Proposal to Support Schizophrenia Patients: Mobile Application Based on Binaural Beats and 3D Avatar

A Proposal to Support Schizophrenia Patients: Mobile Application Based on Binaural Beats and 3D Avatar

A Proposal to Support Schizophrenia Patients: Mobile Application Based on Binaural Beats and 3D Avatar

The emotional metaverse: exploring the benefits of predicting emotion from 3D avatars

With the metaverse here to stay, we are seeing ever more advances in capability. This includes the ability to incorporate real-time body and facial animation, so that personalised avatars can display our changing expressions, allowing those we meet to gauge our mood. Given this advance, the opportunity now exists to utilise emotion recognition from avatars. This positional paper explores the benefits of utilising emotion prediction within the metaverse.

TalkingStyle: Personalized Speech-Driven 3D Facial Animation with Style Preservation.

It is a challenging task to create realistic 3D avatars that accurately replicate individuals' speech and unique talking styles for speech-driven facial animation. Existing techniques have made remarkable progress but still struggle to achieve lifelike mimicry. This paper proposes "TalkingStyle", a novel method to generate personalized talking avatars while retaining the talking style of the person. Our approach uses a set of audio and animation samples from an individual to create new facial animations that closely resemble their specific talking style, synchronized with speech. We disentangle the style codes from the motion patterns, allowing our method to associate a distinct identifier with each person. To manage each aspect effectively, we employ three separate encoders for style, speech, and motion, ensuring the preservation of the original style while maintaining consistent motion in our stylized talking avatars. Additionally, we propose a new style-conditioned transformer decoder, offering greater flexibility and control over the facial avatar styles. We comprehensively evaluate TalkingStyle through qualitative and quantitative assessments, as well as user studies demonstrating its superior realism and lip synchronization accuracy compared to current state-of-the-art methods. To promote transparency and further advancements in the field, we also make the source code publicly available at https://github.com/wangxuanx/TalkingStyle.

From pixels to 3D worlds: Enhancing digital collaboration for distributed teams

The rise in distributed and remote work arrangements has been substantial, with expectations of further expansion. Such a trend presents both opportunities and challenges, particularly with respect to digital collaboration. Available literature proposes that virtual interactions result in limited knowledge transfer and fewer creative ideas compared to physical spaces, due to the lack of shared physical space and limitations in facilitating nonverbal communication. To address these challenges, this paper takes a design-science research approach and presents a real-time communication and collaboration tool, that supports distributed teams in collaborating online. The tool incorporates playful features within a low-poly design that allow for the creation of 3D objects in a shared, virtual environment and enables interactions between users, ultimately fostering engagement and improving spatial knowledge representations. Combining 3D avatars with real-time audio, video, and text communication enhances the efficient transfer of knowledge and fosters interpersonal relationships. The artifact undergoes two iterations of evaluation, using user tests in combination with surveys, to ensure research rigor. Early findings demonstrate a considerable level of user engagement with the tool, which leads to enhanced communication and collaboration among users. Furthermore, it facilitates quick visualization of ideas and enhanced sharing of knowledge during user tests.

Bi-directional translation methods of ASL and speech/text

This paper introduces a sign language translation system designed to bridge the communication gap between Hard of Hearing (HOH) or mute individuals, and address challenges in job opportunities and mental health. The system offers bi-directional translation of sign language and speech/text. For sign language to speech/text translation, the system utilizes either camera-based gestures, or data-glove inputs. The reverse translation is facilitated through a 3D avatar demonstrating the corresponding sign language for the input speech or text. Our findings demonstrate the system’s translation accuracy and its potential to empower HOH and mute individuals in various communication scenarios.

PIXGAN-Drone: 3D Avatar of Human Body Reconstruction from Multi-View 2D Images

PIXGAN-Drone: 3D Avatar of Human Body Reconstruction from Multi-View 2D Images

Development of an Interactive Live Streaming System for Language Learning

<p>In this paper, we propose Virtual Avatar Interactive Live Streaming System (VAILSS), our approach to streaming 3D avatar performances in interactive storytelling applications using multimodal interactive techniques. This study focuses on developing a virtual live interaction system for VTubers in the context of language learning. A live virtual performance from our system consists of three components: 1) an Avatar generation framework, 2) an AI motion capture system, and 3) an interactive storytelling engine. The system integrates artificial intelligence and uses motion capture to identify facial expressions and movements, enabling virtual characters to provide live storytelling services. Additionally, the system allows for bi-directional interaction with players through tablet touch and voice. The system aims to promote multimodal learning channels for the masses. To gain deeper insights into the effectiveness of virtual role-playing in language learning, we organized a three-week workshop to investigate our system’s impact on user experience. We extended invitations to 17 night-school freshmen from the NFU Department of Applied Foreign Languages, enrolled in an English class focused on tour guides, to participate in a virtual performance. The experiment results demonstrated that the VAILSS positively affected students’ learning outcomes, particularly in enhancing foreign language acquisition.</p> <p> </p>

Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation.

Recent advances in extended reality (XR) technology have opened the possibility of significantly improving telemedicine systems. This is primarily achieved by transferring 3D information about patient state, which is utilized to create more immersive experiences on VR/AR headsets. In this paper, we propose an XR-based telemedicine collaboration system in which the patient is represented as a 3D avatar in an XR space shared by local and remote clinicians. The proposed system consists of an AR client application running on Microsoft HoloLens 2 used by a local clinician, a VR client application running on the HTC vive Pro used by a remote clinician, and a backend part running on the server. The patient is captured by a camera on the AR side, and the 3D body pose estimation is performed on frames from this camera stream to form a 3D patient avatar. Additionally, the AR and VR sides can interact with the patient avatar via virtual hands, and annotations can be performed on a 3D model. The main contribution of our work is the use of 3D body pose estimation for the creation of a 3D patient avatar. In this way, 3D body reconstruction using depth cameras is avoided, which reduces system complexity and hardware and network resources. Another contribution is the novel architecture of the proposed system, where audio and video streaming are realized using WebRTC protocol. The performance evaluation showed that the proposed system ensures high frame rates for both AR and VR client applications, while the processing latency remains at an acceptable level.

VALID: a perceptually validated Virtual Avatar Library for Inclusion and Diversity

As consumer adoption of immersive technologies grows, virtual avatars will play a prominent role in the future of social computing. However, as people begin to interact more frequently through virtual avatars, it is important to ensure that the research community has validated tools to evaluate the effects and consequences of such technologies. We present the first iteration of a new, freely available 3D avatar library called the Virtual Avatar Library for Inclusion and Diversity (VALID), which includes 210 fully rigged avatars with a focus on advancing racial diversity and inclusion. We also provide a detailed process for creating, iterating, and validating avatars of diversity. Through a large online study (n = 132) with participants from 33 countries, we provide statistically validated labels for each avatar’s perceived race and gender. Through our validation study, we also advance knowledge pertaining to the perception of an avatar’s race. In particular, we found that avatars of some races were more accurately identified by participants of the same race.

Double reverse diffusion for realistic garment reconstruction from images

Creating realistic digital 3D avatars has been getting more attention thanks to the introduction of new multimedia formats such as augmented and virtual reality. An important factor making avatars realistic is clothes. In this paper, we investigate a new method to reconstruct realistic garments from a set of images and body information. Early methods working on realistic images struggle to faithfully reconstruct the garment details. As deep learning is increasingly applied to geometric data which can conveniently represent garments, we devise a novel deep learning-based solution to the garment reconstruction problem. We offer a new perspective on the reconstruction problem and treat it as a reversion of the smoothing diffusion process. To achieve this goal, we propose to deform the smoothed human mesh into a clothed human via a Double Reverse Diffusion (DReD) process. For the first reverse diffusion, we introduce a novel operator called Graph Long Short-Term Memory (GraphLSTM) which recursively diffuses features to produce a deformed mesh by modeling the relationships between vertices. Then, the output mesh can be repeatedly upsampled and deformed by the above pipeline to obtain finer garment details, which can be seen as another reverse diffusion process. To obtain features for the reverse diffusion, we extract pixel-aligned features transferred from images and explore to incorporate the visibility of garments from the image viewpoints. Through detailed experiments on two public datasets, we demonstrate that DReD synthesizes more realistic wrinkled garments with lower errors and offers faster inference than previous methods.

중년 비만 여성의 아바타 모델링을 위한 체형 연구

This study categorized and analyzed the body types of middle-aged obese women, aged 40-59, based on the characteristics of their shapes for the purpose of 3D avatar modeling. Data from 149 middle-aged obese women were utilized, as provided by the 'Size Korea 8th Body Measurement Survey,' consisting of 3D measurement data. Eleven body type factors were extracted. The data was analyzed through factor analysis and classified into four body types.Type 1 represents a curved body shape with pronounced curves, Type 2 is characterized by well-developed lower body and a stooped posture, Type 3 exhibits a straight body shape with minimal curves, and Type 4 features long legs and development of upper body with a leaning posture.

MVP-Human Dataset for 3-D Clothed Human Avatar Reconstruction From Multiple Frames

In this paper, we consider a novel problem of reconstructing a 3D clothed human avatar from multiple frames, independent of assumptions on camera calibration, capture space, and constrained actions. We contribute a large-scale dataset, Multi-View and multi-Pose 3D human (MVP-Human in short) to help address this problem. The dataset contains 400 subjects, each of which has 15 scans in different poses and 8-view images for each pose, providing 6,000 3D scans and 48,000 images in total. In addition, a baseline method that takes multiple images as inputs, and generates a shape-with-skinning avatar in the canonical space, finished in one feed-forward pass is proposed. It first reconstructs the implicit skinning fields in a multi-level manner, and then the image features from multiple images are aligned and integrated to estimate a pixel-aligned implicit function that represents the clothed shape. With the newly collected dataset and the baseline method, it shows promising performance on 3D clothed avatar reconstruction. We release the MVP-Human dataset and the baseline method in, hoping to promote research and development in this field.

Fast-SNARF: A Fast Deformer for Articulated Neural Fields.

Neural fields have revolutionized the area of 3D reconstruction and novel view synthesis of rigid scenes. A key challenge in making such methods applicable to articulated objects, such as the human body, is to model the deformation of 3D locations between the rest pose (a canonical space) and the deformed space. We propose a new articulation module for neural fields, Fast-SNARF, which finds accurate correspondences between canonical space and posed space via iterative root finding. Fast-SNARF is a drop-in replacement in functionality to our previous work, SNARF, while significantly improving its computational efficiency. We contribute several algorithmic and implementation improvements over SNARF, yielding a speed-up of 150×. These improvements include voxel-based correspondence search, pre-computing the linear blend skinning function, and an efficient software implementation with CUDA kernels. Fast-SNARF enables efficient and simultaneous optimization of shape and skinning weights given deformed observations without correspondences (e.g. 3D meshes). Because learning of deformation maps is a crucial component in many 3D human avatar methods and since Fast-SNARF provides a computationally efficient solution, we believe that this work represents a significant step towards the practical creation of 3D virtual humans.

Synthesizing Human Activity for Data Generation.

The problem of gathering sufficiently representative data, such as those about human actions, shapes, and facial expressions, is costly and time-consuming and also requires training robust models. This has led to the creation of techniques such as transfer learning or data augmentation. However, these are often insufficient. To address this, we propose a semi-automated mechanism that allows the generation and editing of visual scenes with synthetic humans performing various actions, with features such as background modification and manual adjustments of the 3D avatars to allow users to create data with greater variability. We also propose an evaluation methodology for assessing the results obtained using our method, which is two-fold: (i) the usage of an action classifier on the output data resulting from the mechanism and (ii) the generation of masks of the avatars and the actors to compare them through segmentation. The avatars were robust to occlusion, and their actions were recognizable and accurate to their respective input actors. The results also showed that even though the action classifier concentrates on the pose and movement of the synthetic humans, it strongly depends on contextual information to precisely recognize the actions. Generating the avatars for complex activities also proved problematic for action recognition and the clean and precise formation of the masks.

MI-Poser

Inside-out tracking of human body poses using wearable sensors holds significant potential for AR/VR applications, such as remote communication through 3D avatars with expressive body language. Current inside-out systems often rely on vision-based methods utilizing handheld controllers or incorporating densely distributed body-worn IMU sensors. The former limits hands-free and occlusion-robust interactions, while the latter is plagued by inadequate accuracy and jittering. We introduce a novel body tracking system, MI-Poser, which employs AR glasses and two wrist-worn electromagnetic field (EMF) sensors to achieve high-fidelity upper-body pose estimation while mitigating metal interference. Our lightweight system demonstrates a minimal error (6.6 cm mean joint position error) with real-world data collected from 10 participants. It remains robust against various upper-body movements and operates efficiently at 60 Hz. Furthermore, by incorporating an IMU sensor co-located with the EMF sensor, MI-Poser presents solutions to counteract the effects of metal interference, which inherently disrupts the EMF signal during tracking. Our evaluation effectively showcases the successful detection and correction of interference using our EMF-IMU fusion approach across environments with diverse metal profiles. Ultimately, MI-Poser offers a practical pose tracking system, particularly suited for body-centric AR applications.