Avatar Animation Research Articles

Universal Facial Encoding of Codec Avatars from VR Headsets

Faithful real-time facial animation is essential for avatar-mediated telepresence in Virtual Reality (VR). To emulate authentic communication, avatar animation needs to be efficient and accurate: able to capture both extreme and subtle expressions within a few milliseconds to sustain the rhythm of natural conversations. The oblique and incomplete views of the face, variability in the donning of headsets, and illumination variation due to the environment are some of the unique challenges in generalization to unseen faces. In this paper, we present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset. We present a self-supervised learning approach, based on a cross-view reconstruction objective, that enables generalization to unseen users. We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency. We present an improved parameterization for precise ground-truth generation that provides robustness to environmental variation. The resulting system produces accurate facial animation for unseen users wearing VR headsets in realtime. We compare our approach to prior face-encoding methods demonstrating significant improvements in both quantitative metrics and qualitative results.

Investigating VTubing as a Reconstruction of Streamer Self-Presentation: Identity, Performance, and Gender

VTubers, or Virtual YouTubers, are live streamers who create streaming content using animated 2D or 3D virtual avatars. In recent years, there has been a significant increase in the number of VTuber creators and viewers across the globe. This practice has drawn research attention into topics such as viewers' engagement behaviors and perceptions, however, as animated avatars offer more identity and performance flexibility than traditional live streaming where one uses their own body, little research has focused on how this flexibility influences how creators present themselves. This research thus seeks to fill this gap by presenting results from a qualitative study of 16 Chinese-speaking VTubers' streaming practices. The data revealed that the virtual avatars that were used while live streaming afforded creators opportunities to present themselves using inflated presentations and resulted in inclusive interactions with viewers. The results also unveiled the inflated, and often sexualized, gender expressions of VTubers while they were situated in misogynistic environments. The socio-technical facets of VTubing were found to potentially reduce sexual harassment and sexism, whilst also raising self-objectification concerns.

Everything2Motion: Synchronizing Diverse Inputs via a Unified Framework for Human Motion Synthesis

In the dynamic field of film and game development, the emergence of human motion synthesis methods has revolutionized avatar animation. Traditional methodologies, typically reliant on single modality inputs like text or audio, employ modality-specific model frameworks, posing challenges for unified model deployment and application. To address this, we propose Everything2Motion, a unified model framework. Everything2Motion consists of three key modules. The Input-Output Modality Modulation module tailors structures for specific multimodal inputs, eliminating the need for modality-specific frameworks. The Query-aware Autoencoder, based on the transformer encoder-decoder architecture, enables efficient latent motion generation. Lastly, the Prior Motion Distillation Decoder, a pretrained module, enhances the final skeleton sequence's naturalness and fluidity. Comprehensive experiments on several public datasets demonstrate the effectiveness of Everything2Motion, highlighting its potential for practical applications and setting a new benchmark in human motion synthesis.

The Future is Calling

The development of animated avatar calls is seen as a milestone in realizing the full potential of both the metaverse and 5G. A new collaboration has found a way to test them, says Rohde & Schwarz.

Learn2Talk: 3D Talking Face Learns from 2D Talking Face.

The speech-driven facial animation technology is generally categorized into two main types: 3D and 2D talking face. Both of these have garnered considerable research attention in recent years. However, to our knowledge, the research into 3D talking face has not progressed as deeply as that of 2D talking face, particularly in terms of lip-sync and perceptual mouth movements. The lip-sync necessitates an impeccable synchronization between mouth motion and speech audio. The speech perception derived from the perceptual mouth movements should resemble that of the driving audio. To mind the gap between the two sub-fields, we propose Learn2Talk, a learning framework that enhances 3D talking face network by integrating two key insights from the field of 2D talking face. Firstly, drawing inspiration from the audio-video sync network, we develop a 3D sync-lip expert model for the pursuit of lip-sync between audio and 3D facial motions. Secondly, we utilize a teacher model, carefully chosen from among 2D talking face methods, to guide the training of the audio-to-3D motions regression network, thereby increasing the accuracy of 3D vertex movements. Extensive experiments demonstrate the superiority of our proposed framework over state-of-the-art methods in terms of lip-sync, vertex accuracy and perceptual movements. Finally, we showcase two applications of our framework: audio-visual speech recognition and speech-driven 3D Gaussian Splatting-based avatar animation. The project page of this paper is: https://lkjkjoiuiu.github.io/Learn2Talk/.

An Application for Mental Health Monitoring Using Facial, Voice, and Questionnaire Information

Depression is a major societal issue. However, depression can be hard to self-diagnose, and people suffering from depression often hesitate to consult with professionals. We discuss the design and initial testings of our prototype application that performs depression detection using multi-modal information such as questionnaires, speech, and face landmarks. The application has an animated avatar ask questions concerning the users’ well-being. To perform screening, we opt for a 2-stage method which first predicts individual HAM-D ratings for better explainability, which may help facilitate the referral process to medical professionals if required. Initial results show that our system archives 0.85 Marco-F1 for the depression detection task.

ViGather: Inclusive Virtual Conferencing with a Joint Experience Across Traditional Screen Devices and Mixed Reality Headsets

Teleconferencing is poised to become one of the most frequent use cases of immersive platforms, since it supports high levels of presence and embodiment in collaborative settings. On desktop and mobile platforms, teleconferencing solutions are already among the most popular apps and accumulate significant usage time---not least due to the pandemic or as a desirable substitute for air travel or commuting. In this paper, we present ViGather, an immersive teleconferencing system that integrates users of all platform types into a joint experience via equal representation and a first-person experience. ViGather renders all participants as embodied avatars in one shared scene to establish co-presence and elicit natural behavior during collocated conversations, including nonverbal communication cues such as eye contact between participants as well as body language such as turning one's body to another person or using hand gestures to emphasize parts of a conversation during the virtual hangout. Since each user embodies an avatar and experiences situated meetings from an egocentric perspective no matter the device they join from, ViGather alleviates potential concerns about self-perception and appearance while mitigating potential 'Zoom fatigue', as users' self-views are not shown. For participants in Mixed Reality, our system leverages the rich sensing and reconstruction capabilities of today's headsets. For users of tablets, laptops, or PCs, ViGather reconstructs the user's pose from the device's front-facing camera, estimates eye contact with other participants, and relates these non-verbal cues to immediate avatar animations in the shared scene. Our evaluation compared participants' behavior and impressions while videoconferencing in groups of four inside ViGather with those in Meta Horizon as a baseline for a social VR setting. Participants who participated on traditional screen devices (e.g., laptops and desktops) using ViGather reported a significantly higher sense of physical, spatial, and self-presence than when using Horizon, while all perceived similar levels of active social presence when using Virtual Reality headsets. Our follow-up study confirmed the importance of representing users on traditional screen devices as reconstructed avatars for perceiving self-presence.

A real-time Arabic avatar for deaf–mute community using attention mechanism

Speech-impaired people use Sign Language (SL), an efficient natural form of communication, all over the world. This paper aims to use deep learning technology in the realm of SL translation and identification. In order to ease communication between hearing-impaired and sighted individuals and to enable the social inclusion of hearing-impaired people in their daily lives, it presents a transformer as a neural machine translation model. The article details the creation of a machine translation system that converts Arabic audio and text into Arabic Sign Language (ArSL) automatically. It does this by utilizing an animated character to produce the correct sign for each spoken word. Since Arabic has few resources, it was challenging to obtain an Arabic-Sign dataset, so we created our own Arabic–Arabic sign gloss, which consists of 12,187 pairs, to train the model. We use bidirectional encoder representations from transformers as an embedding layer to interpret input text tokens and represent an appropriate natural language vector space for deep learning models. To represent the structure of each Arabic word, the Ferasa Part-of-Speech Tagging module was used and then the extracted rules from the ArSL structure were applied. This paper shows a detailed description of a natural language translator (for converting an Arabic word sequence into a sequence of signs belonging to the ArSL) and a 2D avatar animation module (for playing back the signs). In our prototype, we train the software-based module using the attention mechanism. The evaluation was carried out in our developed Arabic sentences with the corresponding Arabic gloss. The proposed model achieves promising results and indicates significant improvements to direct communication between hearing and deaf people, with a training accuracy of 94.71% and an 87.04% testing accuracy for Arabic–Arabic sign gloss translation.

A Trained Humanoid Robot can Perform Human-Like Crossmodal Social Attention and Conflict Resolution

To enhance human-robot social interaction, it is essential for robots to process multiple social cues in a complex real-world environment. However, incongruency of input information across modalities is inevitable and could be challenging for robots to process. To tackle this challenge, our study adopted the neurorobotic paradigm of crossmodal conflict resolution to make a robot express human-like social attention. A behavioural experiment was conducted on 37 participants for the human study. We designed a round-table meeting scenario with three animated avatars to improve ecological validity. Each avatar wore a medical mask to obscure the facial cues of the nose, mouth, and jaw. The central avatar shifted its eye gaze while the peripheral avatars generated sound. Gaze direction and sound locations were either spatially congruent or incongruent. We observed that the central avatar’s dynamic gaze could trigger crossmodal social attention responses. In particular, human performance was better under the congruent audio-visual condition than the incongruent condition. Our saliency prediction model was trained to detect social cues, predict audio-visual saliency, and attend selectively for the robot study. After mounting the trained model on the iCub, the robot was exposed to laboratory conditions similar to the human experiment. While the human performance was overall superior, our trained model demonstrated that it could replicate attention responses similar to humans.

The Research on the Characteristics of Chinese Game Virtual Streamers

With the development of the metaverse, many activities have been infiltrated into various virtual interactive experiences. In the widespread live-streaming industry, the virtual anchors who stream behind various animated avatars are favored by a majority of young people. This paper summarizes five characteristics of Chinese game virtual streaming through observations on Bilibili live streaming platform and social media such as Weibo, TikTok and Little Red Book. The author believes that Chinese game virtual streamers have strong role-playing quality, a variety of live games content, humor, versatility and the ability to follow the latest trends. Through these characteristics, they can enrich the live content to attract more audience attention, also strengthen the interaction with the audience and meet the needs of the audience to a greater extent, thus promoting the development of the virtual streaming industry. The analysis of the characteristics of game virtual streamers can make the streamers better improve and strengthen their own advantages and expand the audience group, as well as enable the audience to have a deep understanding of the virtual streamers, which encourages a more appropriate and rational virtual stream watching experience.

Unintentional synchronization with self-avatar for upper- and lower-body movements

The subjective experience of embodying an avatar when immersed in virtual reality (VR) is known to support the sense of presence and to help with the interaction in a virtual environment. Virtual embodiment is often thought of as the consequence of replacement of the physical body by a virtual one, with a sense of agency for the avatar obtained by making the avatar’s body follow the user’s movements. This unidirectional motor link was, however, challenged by studies observing the opposite effect under different circumstances, for example, in a slow-motion context or when an arm movement was snapped on a predefined axis. These reports are, however, still rare or anecdotal. With the idea of a generalized bidirectional relationship between the user and the avatar in mind, we established a methodology to systematically provoke and study the circumstances under which participants follow the movements of their avatar during long repetitive movements without having been instructed to do so. A preliminary study confirmed that our virtual experimental setup, using full-body motion capture, avatar animation, and virtual mirrors, supports a strong sense of agency and body ownership for the avatar while enabling the experimental manipulation of the avatar’s movement. In the main experimental study, where participants performed repetitive upper- and lower-body movements while their avatar animations were either congruent or out-of-phase, we observed that almost all participants synchronized with their avatar at least once, for ∼47% of trials for lower limb movements and ∼38% for upper limb movements. Participants still reported low agency and ownership for the avatar under the incongruent condition, but, most interestingly, some of them also reported that their movements were not influenced by the avatar despite the behavioral effect. Our methodological approach and results contribute to the characterization of the conditions of occurrence of the self-avatar follower effect and, thereby, to identifying an enriched interaction design for VR, involving complex avatar–user mutual interdependencies.

Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer‐Grade VR Devices

AbstractThe animation of user avatars plays a crucial role in conveying their pose, gestures, and relative distances to virtual objects or other users. Self‐avatar animation in immersive VR helps improve the user experience and provides a Sense of Embodiment. However, consumer‐grade VR devices typically include at most three trackers, one at the Head Mounted Display (HMD), and two at the handheld VR controllers. Since the problem of reconstructing the user pose from such sparse data is ill‐defined, especially for the lower body, the approach adopted by most VR games consists of assuming the body orientation matches that of the HMD, and applying animation blending and time‐warping from a reduced set of animations. Unfortunately, this approach produces noticeable mismatches between user and avatar movements. In this work we present a new approach to animate user avatars that is suitable for current mainstream VR devices. First, we use a neural network to estimate the user's body orientation based on the tracking information from the HMD and the hand controllers. Then we use this orientation together with the velocity and rotation of the HMD to build a feature vector that feeds a Motion Matching algorithm. We built a MoCap database with animations of VR users wearing a HMD and used it to test our approach on both self‐avatars and other users' avatars. Our results show that our system can provide a large variety of lower body animations while correctly matching the user orientation, which in turn allows us to represent not only forward movements but also stepping in any direction.

Expressive Interaction Design Using Facial Muscles as Controllers

Here we describe a proof-of-concept case study focusing on the design and development of a novel computer interface that uses facial muscles to control interactivity within a virtual environment. We have developed a system comprised of skin-mounted electrodes that detect underlying muscle activity through electromyography. The signals from the electrodes are filtered and smoothed, then used as input data to an application that displays a virtual environment with a 3D animated avatar. The user’s expressions control the facial movements of the avatar, thus conveying user emotions through real-time animation of a representative face in a virtual scenario. To achieve this, we collaborated with our Public and Patient Involvement focus group to discuss concepts and design appropriate interactions, while simultaneously developing a prototype system. Programmers and 3D artists worked together to create a system whereby individual user facial muscles are connected to 3D animated models of the same muscle features represented in an avatar, providing the user with an option to receive visual and numerical feedback on the extent of their muscle control. Using the prototype system, people can communicate facial expressions virtually with each other, without the need for a camera. This research is part of an on-going project to develop a facial muscle rehabilitation system that can be optimized to help patients with conditions such as hypomimia.

Topographic design in wearable MXene sensors with in-sensor machine learning for full-body avatar reconstruction

Wearable strain sensors that detect joint/muscle strain changes become prevalent at human–machine interfaces for full-body motion monitoring. However, most wearable devices cannot offer customizable opportunities to match the sensor characteristics with specific deformation ranges of joints/muscles, resulting in suboptimal performance. Adequate wearable strain sensor design is highly required to achieve user-designated working windows without sacrificing high sensitivity, accompanied with real-time data processing. Herein, wearable Ti3C2Tx MXene sensor modules are fabricated with in-sensor machine learning (ML) models, either functioning via wireless streaming or edge computing, for full-body motion classifications and avatar reconstruction. Through topographic design on piezoresistive nanolayers, the wearable strain sensor modules exhibited ultrahigh sensitivities within the working windows that meet all joint deformation ranges. By integrating the wearable sensors with a ML chip, an edge sensor module is fabricated, enabling in-sensor reconstruction of high-precision avatar animations that mimic continuous full-body motions with an average avatar determination error of 3.5 cm, without additional computing devices.

The Blending of Bending: How We Engage with the World of Avatar: The Last Airbender through Memes

ABSTRACT People often use memes to express their ideological stance on real world events. This study departs from a recent COVID-19-related meme which makes use of elements known from the animated television series Avatar: The Last Airbender (ATLA) and Avatar: Legend of Korra (LOK), and asks how it came to be and how stance is conveyed through them. After acknowledging the impact of the series, conceptual blending theory is adopted to investigate the worldbuilding of the macrocosm in ATLA. This is identified as a correlative network, which acts as the blended space of multiple input spaces consisting of intertextual references. The world of ATLA then functions as a new input space which is updated with modern elements, resulting in the blend of LOK. Minor blends are identified in the hybrid animals that occupy the fictional world. Lastly, it is shown how the selection of a particular input element for participation in meme blends already conveys an ideological stance, rather than only emerging through readers’ eyes.

A hybrid BCI for neurofeedback-based attention training: design and preliminary evaluation

Neurofeedback therapies based on brain-computer interfaces (BCI) can be used for the rehabilitation of attention deficit hyperactivity disorder (ADHD), as an alternative to conventional treatments. In this paper the design, implementation and preliminary evaluation of Attenti-ON, a hybrid BCI for ADHD rehabilitation, is presented. Attenti-ON uses a Serious Game commanded by the level of attention estimated by the theta/beta ratio of the electroencephalogram as a neurofeedback and also incorporates a manual activity that focuses attention on the required cognitive task., Attenti-ON consists of an user interface and four modules: the first one acquires the electroencephalography signals from prefrontal electrodes, located at Fp1 and Fp2, and conditions them. The second module procceses the signals and generates a valid command for the third module which consists in the Serious Game, developed in Unity. In this module the speed of an animated avatar is modified by the level of attention. The last module is the manual control consisting of a keyboard or joystick. Two evaluations of Attenti-ON were conducted. The first one was to assess the systems operation by using a signals database. In the second one, the BCIs closed loop-performance on a healthy volunteer was evaluated and aimed at verifying that the speed of the avatar varied according to the level of attention of the volunteer, which was modified by simulated interference situations. These preliminary results suggest that Attenti-ON could be transfered to the clinical setting for its evaluation for the treatment of ADHD.

VGTC Virtual Reality Significant New Researcher Award

The 2022 IEEE VGTC Significant New Researcher Award goes to Mar Gonzalez-Franco of Microsoft, in recognition of her research and incipient leadership in the field of Virtual Reality and spatial computing. Her work has provided new insights on how people behave and perceive avatars, haptics, sound and multisensory signals inside Virtual Reality. She has produced numerous new prototypes advancing the state of the art on haptic controllers and haptic displays. She has been prolific in proposing new theories and standardized methods of evaluation for different phenomena such as embodiment, locomotion, the uncanny valley of haptics, avatar self-recognition and the self-avatar follower effect. Meanwhile, Dr. Gonzalez-Franco has helped democratize access to VR technology by open sourcing of avatars and avatar animation libraries (Microsoft Rocketbox).

Real Time Sign Language Recognition System for Hearing and Speech Impaired People

Abstract: Sign Language is globally used by more than 70 million impaired people to communicate and is characterized by fast, highly articulate motion of hand gesture which is difficult for verbal speakers to understand. This limitation combined with the lack of knowledge about sign language by verbal speakers creates a separation where both parties are unable to effectively communicate, to overcome this limitation we propose a new method for sign language recognition using OpenCV (A python library) which is used for pre-processing images and extracting different skin toned hands from the background. In this method hand gesture are used to make signs which are detected by YOLOv5 algorithm for object detection which is the fastest algorithm till date while Convolutional-Neural-Networks (CNN) are used for training gesture and to classify the images, and further we proposed a system which translates speech into sign language so that the words of the verbal speaker can be transmitted to the deaf/mute. This automated system first detects speech using the JavaScript Web-Speech API and converts it into text because the recognized text is processed using the Natural Language Toolkit and aligns token text with the sign language library (sign language videos) videos according to well-known text and finally shows a compiled output which is displayed through avatar animation for a deaf / dumb person. The proposed system has various advantages like Portability, User-friendly Interface and Voice Module. The software is also very cost-effective which only needs a laptop camera or webcam and hand gesture, system accuracy is compared to high-quality methods and is found to be the best. Keywords: Sign language, Hand gesture, OpenCV, Python library, Image processing, YOLOv5, Object detection, Pose estimation, Machine learning

Exploring disrupting scenarios in the fashion retail and communication paradigms

The growing complexity in the economic sphere, the accelerating rate of change, the new information technology and the globalization of cultural processes create a new landscape. The new tech habits are reshaping customer behaviours and businesses. Contaminations, hybridizations and negotiations increasingly characterize today’s retail landscape. The future of retail is connected to how well brands adapt and take advantage of digital change. Within this framework, the fashion system can catch the opportunity to reflect and redesign the entire production and cultural system. Today, the shopping experience is realized thanks to multiform sensory and informational solicitations and the different distribution and communication channels, both physical and digital, defining new needs, strategies, technologies and even new aesthetic forms. In this renewed context, retail becomes increasingly relevant both in its physical and digital form and above all in the ‘phygital’ one, between virtual and real. A form that creates new sales space through the creation of experiences and interactions between the physical space and the intangibility of the digital world: from the multichannel, we move towards increasingly integrated strategies that use omnichannel sales and communication methods. In this context, new paradigms and new retail concepts emerge, bridging communication and distribution. Communication actions that are increasingly transversal, enriched with new languages and tools capable of reverberating the aura of fashion through the creation of real cultural palimpsests such as the Dior Talks and Possible Conversations by Prada, the Alexander McQueen Floral Challenge or the Trivia by Salvatore Ferragamo. Projects between communication, art and marketing that redefine the fashion point of view on the relationship between real and digital; as well as the Sunnei Canvas project that uses 3D design tools both for the development of the collection and for the creation of animated avatars that will guide the final translation of the brand’s lexicon.

Transformasi Ruang 3D Pada Animasi Expresi Wajah Avatar Berbasis Radial Basis Function

One technique for forming facial animations on avatars is by reusing existing animations, either from other avatar animations or animations from motion data obtained using facial motion capture. This research focuses on the transformation of 3D space for the formation of facial animations on avatars in games or animated films. The transformation is carried out from motion capture data into a 3D avatar face model with 3 face models, namely the human face model, the swan face model and the anoman face model. The motion capture data is transferred according to the feature points of the face model. The results obtained by the facial model feature points will have animations that match the motion capture data. Of the 3 target face models used, the animation results with registration on the human face model have an average standard deviation is 0,0510. The goose face model has an average standard deviation is 0.0034 and the anoman face model has an average standard deviation is 0,0024. With this technique, it is hoped that the formation of facial expression animation on Avatar can be done more quickly because of the reuse of facial motion capture data.