Avatar Animation Research Articles

THGS: Lifelike Talking Human Avatar Synthesis From Monocular Video Via 3D Gaussian Splatting

AbstractDespite the remarkable progress in 3D talking head generation, directly generating 3D talking human avatars still suffers from rigid facial expressions, distorted hand textures and out‐of‐sync lip movements. In this paper, we extend speaker‐specific talking head generation task to talking human avatar synthesis and propose a novel pipeline, THGS, that animates lifelike Talking Human avatars using 3D Gaussian Splatting (3DGS). Given speech audio, expression and body poses as input, THGS effectively overcomes the limitations of 3DGS human re‐construction methods in capturing expressive dynamics, such as mouth movements, facial expressions and hand gestures, from a short monocular video. Firstly, we introduce a simple yet effective Learnable Expression Blendshapes (LEB) for facial dynamics re‐construction, where subtle facial dynamics can be generated by linearly combining the static head model and expression blendshapes. Secondly, a Spatial Audio Attention Module (SAAM) is proposed for lip‐synced mouth movement animation, building connections between speech audio and mouth Gaussian movements. Thirdly, we employ a body pose, expression and skinning weights joint optimization strategy to optimize these parameters on the fly, which aligns hand movements and expressions better with video input. Experimental results demonstrate that THGS can achieve high‐fidelity 3D talking human avatar animation at 150+ fps on a web‐based rendering system, improving the requirements of real‐time applications. Our project page is at https://sora158.github.io/THGS.github.io/.

Combining Unity with machine vision to create low latency, flexible and simple virtual realities

Abstract In recent years, virtual reality arenas have become increasingly popular for quantifying visual behaviours. By using the actions of a constrained animal to control the visual scenery, the animal perceives that it is moving through a virtual world. Importantly, as the animal is constrained in space, behavioural quantification is facilitated. Furthermore, using computer‐generated visual scenery allows for identification of visual triggers of behaviour. We created a novel virtual reality arena combining machine vision with the gaming engine Unity. For tethered flight, we enhanced an existing multi‐modal virtual reality arena, MultiMoVR, but tracked wing movements using DeepLabCut‐live (DLC‐live). For tethered walking animals, we used FicTrac to track the motion of a trackball. In both cases, real‐time tracking was interfaced with Unity to control the location and rotation of the tethered animal's avatar in the virtual world. We developed a user‐friendly Unity Editor interface, CAVE, to simplify experimental design and data storage without the need for coding. We show that both the DLC‐live‐Unity and the FicTrac‐Unity configurations close the feedback loop effectively and quickly. We show that closed‐loop feedback reduces behavioural artefacts exhibited by walking crabs in open‐loop scenarios, and that flying Eristalis tenax hoverflies navigate towards virtual flowers in closed loop. We show examples of how the CAVE interface can enable experimental sequencing control including use of avatar proximity to virtual objects of interest. Our results show that combining Unity with machine vision tools provides an easy and flexible virtual reality environment that can be readily adjusted to new experiments and species. This can be implemented programmatically in Unity, or by using our new tool CAVE, which allows users to design new experiments without additional programming. We provide resources for replicating experiments and our interface CAVE via GitHub, together with user manuals and instruction videos, for sharing with the wider scientific community.

Girl-instrument: Posthumanism, producers, and virtual performers in Japanese music

Vocal synthesizer programs such as Vocaloid use artificial intelligence to create humanlike vocal tracks without human singers. These programs often tie their voice banks to animated avatars, making each user-created song a part of a never-ending oeuvre of work for a distinct virtual performer. These avatars can not only be used in images and video, but also in augmented reality concerts through semi-transparent LED screens that make it look like these animated virtual performers are performing live. ‘Vocaloid’ then is not only a software or a genre of music created using that software, but also denotes a hyperreal virtual performer. Through a feminist critique that utilizes Inter-Asian referencing, this paper problematizes the issue of removing the human from the female body to make it widely usable as a medium of expression and question the usage of Vocaloids in consideration of the future of digital technologies.

Universal Facial Encoding of Codec Avatars from VR Headsets

Faithful real-time facial animation is essential for avatar-mediated telepresence in Virtual Reality (VR). To emulate authentic communication, avatar animation needs to be efficient and accurate: able to capture both extreme and subtle expressions within a few milliseconds to sustain the rhythm of natural conversations. The oblique and incomplete views of the face, variability in the donning of headsets, and illumination variation due to the environment are some of the unique challenges in generalization to unseen faces. In this paper, we present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset. We present a self-supervised learning approach, based on a cross-view reconstruction objective, that enables generalization to unseen users. We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency. We present an improved parameterization for precise ground-truth generation that provides robustness to environmental variation. The resulting system produces accurate facial animation for unseen users wearing VR headsets in realtime. We compare our approach to prior face-encoding methods demonstrating significant improvements in both quantitative metrics and qualitative results.

Investigating VTubing as a Reconstruction of Streamer Self-Presentation: Identity, Performance, and Gender

VTubers, or Virtual YouTubers, are live streamers who create streaming content using animated 2D or 3D virtual avatars. In recent years, there has been a significant increase in the number of VTuber creators and viewers across the globe. This practice has drawn research attention into topics such as viewers' engagement behaviors and perceptions, however, as animated avatars offer more identity and performance flexibility than traditional live streaming where one uses their own body, little research has focused on how this flexibility influences how creators present themselves. This research thus seeks to fill this gap by presenting results from a qualitative study of 16 Chinese-speaking VTubers' streaming practices. The data revealed that the virtual avatars that were used while live streaming afforded creators opportunities to present themselves using inflated presentations and resulted in inclusive interactions with viewers. The results also unveiled the inflated, and often sexualized, gender expressions of VTubers while they were situated in misogynistic environments. The socio-technical facets of VTubing were found to potentially reduce sexual harassment and sexism, whilst also raising self-objectification concerns.

Everything2Motion: Synchronizing Diverse Inputs via a Unified Framework for Human Motion Synthesis

In the dynamic field of film and game development, the emergence of human motion synthesis methods has revolutionized avatar animation. Traditional methodologies, typically reliant on single modality inputs like text or audio, employ modality-specific model frameworks, posing challenges for unified model deployment and application. To address this, we propose Everything2Motion, a unified model framework. Everything2Motion consists of three key modules. The Input-Output Modality Modulation module tailors structures for specific multimodal inputs, eliminating the need for modality-specific frameworks. The Query-aware Autoencoder, based on the transformer encoder-decoder architecture, enables efficient latent motion generation. Lastly, the Prior Motion Distillation Decoder, a pretrained module, enhances the final skeleton sequence's naturalness and fluidity. Comprehensive experiments on several public datasets demonstrate the effectiveness of Everything2Motion, highlighting its potential for practical applications and setting a new benchmark in human motion synthesis.

The Future is Calling

The development of animated avatar calls is seen as a milestone in realizing the full potential of both the metaverse and 5G. A new collaboration has found a way to test them, says Rohde & Schwarz.

Learn2Talk: 3D Talking Face Learns from 2D Talking Face.

The speech-driven facial animation technology is generally categorized into two main types: 3D and 2D talking face. Both of these have garnered considerable research attention in recent years. However, to our knowledge, the research into 3D talking face has not progressed as deeply as that of 2D talking face, particularly in terms of lip-sync and perceptual mouth movements. The lip-sync necessitates an impeccable synchronization between mouth motion and speech audio. The speech perception derived from the perceptual mouth movements should resemble that of the driving audio. To mind the gap between the two sub-fields, we propose Learn2Talk, a learning framework that enhances 3D talking face network by integrating two key insights from the field of 2D talking face. Firstly, drawing inspiration from the audio-video sync network, we develop a 3D sync-lip expert model for the pursuit of lip-sync between audio and 3D facial motions. Secondly, we utilize a teacher model, carefully chosen from among 2D talking face methods, to guide the training of the audio-to-3D motions regression network, thereby increasing the accuracy of 3D vertex movements. Extensive experiments demonstrate the superiority of our proposed framework over state-of-the-art methods in terms of lip-sync, vertex accuracy and perceptual movements. Finally, we showcase two applications of our framework: audio-visual speech recognition and speech-driven 3D Gaussian Splatting-based avatar animation. The project page of this paper is: https://lkjkjoiuiu.github.io/Learn2Talk/.

An Application for Mental Health Monitoring Using Facial, Voice, and Questionnaire Information

Depression is a major societal issue. However, depression can be hard to self-diagnose, and people suffering from depression often hesitate to consult with professionals. We discuss the design and initial testings of our prototype application that performs depression detection using multi-modal information such as questionnaires, speech, and face landmarks. The application has an animated avatar ask questions concerning the users’ well-being. To perform screening, we opt for a 2-stage method which first predicts individual HAM-D ratings for better explainability, which may help facilitate the referral process to medical professionals if required. Initial results show that our system archives 0.85 Marco-F1 for the depression detection task.

ViGather: Inclusive Virtual Conferencing with a Joint Experience Across Traditional Screen Devices and Mixed Reality Headsets

Teleconferencing is poised to become one of the most frequent use cases of immersive platforms, since it supports high levels of presence and embodiment in collaborative settings. On desktop and mobile platforms, teleconferencing solutions are already among the most popular apps and accumulate significant usage time---not least due to the pandemic or as a desirable substitute for air travel or commuting. In this paper, we present ViGather, an immersive teleconferencing system that integrates users of all platform types into a joint experience via equal representation and a first-person experience. ViGather renders all participants as embodied avatars in one shared scene to establish co-presence and elicit natural behavior during collocated conversations, including nonverbal communication cues such as eye contact between participants as well as body language such as turning one's body to another person or using hand gestures to emphasize parts of a conversation during the virtual hangout. Since each user embodies an avatar and experiences situated meetings from an egocentric perspective no matter the device they join from, ViGather alleviates potential concerns about self-perception and appearance while mitigating potential 'Zoom fatigue', as users' self-views are not shown. For participants in Mixed Reality, our system leverages the rich sensing and reconstruction capabilities of today's headsets. For users of tablets, laptops, or PCs, ViGather reconstructs the user's pose from the device's front-facing camera, estimates eye contact with other participants, and relates these non-verbal cues to immediate avatar animations in the shared scene. Our evaluation compared participants' behavior and impressions while videoconferencing in groups of four inside ViGather with those in Meta Horizon as a baseline for a social VR setting. Participants who participated on traditional screen devices (e.g., laptops and desktops) using ViGather reported a significantly higher sense of physical, spatial, and self-presence than when using Horizon, while all perceived similar levels of active social presence when using Virtual Reality headsets. Our follow-up study confirmed the importance of representing users on traditional screen devices as reconstructed avatars for perceiving self-presence.

A real-time Arabic avatar for deaf–mute community using attention mechanism

Speech-impaired people use Sign Language (SL), an efficient natural form of communication, all over the world. This paper aims to use deep learning technology in the realm of SL translation and identification. In order to ease communication between hearing-impaired and sighted individuals and to enable the social inclusion of hearing-impaired people in their daily lives, it presents a transformer as a neural machine translation model. The article details the creation of a machine translation system that converts Arabic audio and text into Arabic Sign Language (ArSL) automatically. It does this by utilizing an animated character to produce the correct sign for each spoken word. Since Arabic has few resources, it was challenging to obtain an Arabic-Sign dataset, so we created our own Arabic–Arabic sign gloss, which consists of 12,187 pairs, to train the model. We use bidirectional encoder representations from transformers as an embedding layer to interpret input text tokens and represent an appropriate natural language vector space for deep learning models. To represent the structure of each Arabic word, the Ferasa Part-of-Speech Tagging module was used and then the extracted rules from the ArSL structure were applied. This paper shows a detailed description of a natural language translator (for converting an Arabic word sequence into a sequence of signs belonging to the ArSL) and a 2D avatar animation module (for playing back the signs). In our prototype, we train the software-based module using the attention mechanism. The evaluation was carried out in our developed Arabic sentences with the corresponding Arabic gloss. The proposed model achieves promising results and indicates significant improvements to direct communication between hearing and deaf people, with a training accuracy of 94.71% and an 87.04% testing accuracy for Arabic–Arabic sign gloss translation.

A Trained Humanoid Robot can Perform Human-Like Crossmodal Social Attention and Conflict Resolution

To enhance human-robot social interaction, it is essential for robots to process multiple social cues in a complex real-world environment. However, incongruency of input information across modalities is inevitable and could be challenging for robots to process. To tackle this challenge, our study adopted the neurorobotic paradigm of crossmodal conflict resolution to make a robot express human-like social attention. A behavioural experiment was conducted on 37 participants for the human study. We designed a round-table meeting scenario with three animated avatars to improve ecological validity. Each avatar wore a medical mask to obscure the facial cues of the nose, mouth, and jaw. The central avatar shifted its eye gaze while the peripheral avatars generated sound. Gaze direction and sound locations were either spatially congruent or incongruent. We observed that the central avatar’s dynamic gaze could trigger crossmodal social attention responses. In particular, human performance was better under the congruent audio-visual condition than the incongruent condition. Our saliency prediction model was trained to detect social cues, predict audio-visual saliency, and attend selectively for the robot study. After mounting the trained model on the iCub, the robot was exposed to laboratory conditions similar to the human experiment. While the human performance was overall superior, our trained model demonstrated that it could replicate attention responses similar to humans.

The Research on the Characteristics of Chinese Game Virtual Streamers

With the development of the metaverse, many activities have been infiltrated into various virtual interactive experiences. In the widespread live-streaming industry, the virtual anchors who stream behind various animated avatars are favored by a majority of young people. This paper summarizes five characteristics of Chinese game virtual streaming through observations on Bilibili live streaming platform and social media such as Weibo, TikTok and Little Red Book. The author believes that Chinese game virtual streamers have strong role-playing quality, a variety of live games content, humor, versatility and the ability to follow the latest trends. Through these characteristics, they can enrich the live content to attract more audience attention, also strengthen the interaction with the audience and meet the needs of the audience to a greater extent, thus promoting the development of the virtual streaming industry. The analysis of the characteristics of game virtual streamers can make the streamers better improve and strengthen their own advantages and expand the audience group, as well as enable the audience to have a deep understanding of the virtual streamers, which encourages a more appropriate and rational virtual stream watching experience.

Unintentional synchronization with self-avatar for upper- and lower-body movements

The subjective experience of embodying an avatar when immersed in virtual reality (VR) is known to support the sense of presence and to help with the interaction in a virtual environment. Virtual embodiment is often thought of as the consequence of replacement of the physical body by a virtual one, with a sense of agency for the avatar obtained by making the avatar’s body follow the user’s movements. This unidirectional motor link was, however, challenged by studies observing the opposite effect under different circumstances, for example, in a slow-motion context or when an arm movement was snapped on a predefined axis. These reports are, however, still rare or anecdotal. With the idea of a generalized bidirectional relationship between the user and the avatar in mind, we established a methodology to systematically provoke and study the circumstances under which participants follow the movements of their avatar during long repetitive movements without having been instructed to do so. A preliminary study confirmed that our virtual experimental setup, using full-body motion capture, avatar animation, and virtual mirrors, supports a strong sense of agency and body ownership for the avatar while enabling the experimental manipulation of the avatar’s movement. In the main experimental study, where participants performed repetitive upper- and lower-body movements while their avatar animations were either congruent or out-of-phase, we observed that almost all participants synchronized with their avatar at least once, for ∼47% of trials for lower limb movements and ∼38% for upper limb movements. Participants still reported low agency and ownership for the avatar under the incongruent condition, but, most interestingly, some of them also reported that their movements were not influenced by the avatar despite the behavioral effect. Our methodological approach and results contribute to the characterization of the conditions of occurrence of the self-avatar follower effect and, thereby, to identifying an enriched interaction design for VR, involving complex avatar–user mutual interdependencies.

Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer‐Grade VR Devices

AbstractThe animation of user avatars plays a crucial role in conveying their pose, gestures, and relative distances to virtual objects or other users. Self‐avatar animation in immersive VR helps improve the user experience and provides a Sense of Embodiment. However, consumer‐grade VR devices typically include at most three trackers, one at the Head Mounted Display (HMD), and two at the handheld VR controllers. Since the problem of reconstructing the user pose from such sparse data is ill‐defined, especially for the lower body, the approach adopted by most VR games consists of assuming the body orientation matches that of the HMD, and applying animation blending and time‐warping from a reduced set of animations. Unfortunately, this approach produces noticeable mismatches between user and avatar movements. In this work we present a new approach to animate user avatars that is suitable for current mainstream VR devices. First, we use a neural network to estimate the user's body orientation based on the tracking information from the HMD and the hand controllers. Then we use this orientation together with the velocity and rotation of the HMD to build a feature vector that feeds a Motion Matching algorithm. We built a MoCap database with animations of VR users wearing a HMD and used it to test our approach on both self‐avatars and other users' avatars. Our results show that our system can provide a large variety of lower body animations while correctly matching the user orientation, which in turn allows us to represent not only forward movements but also stepping in any direction.

Expressive Interaction Design Using Facial Muscles as Controllers

Here we describe a proof-of-concept case study focusing on the design and development of a novel computer interface that uses facial muscles to control interactivity within a virtual environment. We have developed a system comprised of skin-mounted electrodes that detect underlying muscle activity through electromyography. The signals from the electrodes are filtered and smoothed, then used as input data to an application that displays a virtual environment with a 3D animated avatar. The user’s expressions control the facial movements of the avatar, thus conveying user emotions through real-time animation of a representative face in a virtual scenario. To achieve this, we collaborated with our Public and Patient Involvement focus group to discuss concepts and design appropriate interactions, while simultaneously developing a prototype system. Programmers and 3D artists worked together to create a system whereby individual user facial muscles are connected to 3D animated models of the same muscle features represented in an avatar, providing the user with an option to receive visual and numerical feedback on the extent of their muscle control. Using the prototype system, people can communicate facial expressions virtually with each other, without the need for a camera. This research is part of an on-going project to develop a facial muscle rehabilitation system that can be optimized to help patients with conditions such as hypomimia.

Topographic design in wearable MXene sensors with in-sensor machine learning for full-body avatar reconstruction

Wearable strain sensors that detect joint/muscle strain changes become prevalent at human–machine interfaces for full-body motion monitoring. However, most wearable devices cannot offer customizable opportunities to match the sensor characteristics with specific deformation ranges of joints/muscles, resulting in suboptimal performance. Adequate wearable strain sensor design is highly required to achieve user-designated working windows without sacrificing high sensitivity, accompanied with real-time data processing. Herein, wearable Ti3C2Tx MXene sensor modules are fabricated with in-sensor machine learning (ML) models, either functioning via wireless streaming or edge computing, for full-body motion classifications and avatar reconstruction. Through topographic design on piezoresistive nanolayers, the wearable strain sensor modules exhibited ultrahigh sensitivities within the working windows that meet all joint deformation ranges. By integrating the wearable sensors with a ML chip, an edge sensor module is fabricated, enabling in-sensor reconstruction of high-precision avatar animations that mimic continuous full-body motions with an average avatar determination error of 3.5 cm, without additional computing devices.

The Blending of Bending: How We Engage with the World of Avatar: The Last Airbender through Memes

ABSTRACT People often use memes to express their ideological stance on real world events. This study departs from a recent COVID-19-related meme which makes use of elements known from the animated television series Avatar: The Last Airbender (ATLA) and Avatar: Legend of Korra (LOK), and asks how it came to be and how stance is conveyed through them. After acknowledging the impact of the series, conceptual blending theory is adopted to investigate the worldbuilding of the macrocosm in ATLA. This is identified as a correlative network, which acts as the blended space of multiple input spaces consisting of intertextual references. The world of ATLA then functions as a new input space which is updated with modern elements, resulting in the blend of LOK. Minor blends are identified in the hybrid animals that occupy the fictional world. Lastly, it is shown how the selection of a particular input element for participation in meme blends already conveys an ideological stance, rather than only emerging through readers’ eyes.

A hybrid BCI for neurofeedback-based attention training: design and preliminary evaluation

Neurofeedback therapies based on brain-computer interfaces (BCI) can be used for the rehabilitation of attention deficit hyperactivity disorder (ADHD), as an alternative to conventional treatments. In this paper the design, implementation and preliminary evaluation of Attenti-ON, a hybrid BCI for ADHD rehabilitation, is presented. Attenti-ON uses a Serious Game commanded by the level of attention estimated by the theta/beta ratio of the electroencephalogram as a neurofeedback and also incorporates a manual activity that focuses attention on the required cognitive task., Attenti-ON consists of an user interface and four modules: the first one acquires the electroencephalography signals from prefrontal electrodes, located at Fp1 and Fp2, and conditions them. The second module procceses the signals and generates a valid command for the third module which consists in the Serious Game, developed in Unity. In this module the speed of an animated avatar is modified by the level of attention. The last module is the manual control consisting of a keyboard or joystick. Two evaluations of Attenti-ON were conducted. The first one was to assess the systems operation by using a signals database. In the second one, the BCIs closed loop-performance on a healthy volunteer was evaluated and aimed at verifying that the speed of the avatar varied according to the level of attention of the volunteer, which was modified by simulated interference situations. These preliminary results suggest that Attenti-ON could be transfered to the clinical setting for its evaluation for the treatment of ADHD.

VGTC Virtual Reality Significant New Researcher Award

The 2022 IEEE VGTC Significant New Researcher Award goes to Mar Gonzalez-Franco of Microsoft, in recognition of her research and incipient leadership in the field of Virtual Reality and spatial computing. Her work has provided new insights on how people behave and perceive avatars, haptics, sound and multisensory signals inside Virtual Reality. She has produced numerous new prototypes advancing the state of the art on haptic controllers and haptic displays. She has been prolific in proposing new theories and standardized methods of evaluation for different phenomena such as embodiment, locomotion, the uncanny valley of haptics, avatar self-recognition and the self-avatar follower effect. Meanwhile, Dr. Gonzalez-Franco has helped democratize access to VR technology by open sourcing of avatars and avatar animation libraries (Microsoft Rocketbox).