Full-body Avatar Research Articles

A comparison of balance between real and virtual environments: differences, role of visual cues and full-body avatars, a quasi-experimental clinical study

Virtual rehabilitation using Virtual Reality (VR) technology is a promising novel approach to rehabilitation. However, postural responses in VR differ significantly from real life. The introduction of an avatar or visual cues in VR could help rectify this difference. An initial session was used to assess static and dynamic balance performances between VR and real life to set the reference values. A second session involved three VR conditions applied in a randomised order: i.e. full-body avatar, enhanced visual cues, or a combination of both conditions. Performances of the centre of pressure (COP) were recorded on a force plate. Seventy (70) people took part in the first session and 74 in the second. During the first session, a significant difference was observed in left static, right static and right dynamic COP distance (respectively SMD = − 0.40 [− 0.73, − 0.06], p = 0.02, − 0.33 [− 0.67, 0.00], p = 0.05, SMD = − 0.61 [− 0.95, − 0.27], p < 0.001) and a non-significant difference in the left dynamic, SMD = − 0.22 [− 0.56, 0.11], p = 0.19). During the second session it was observed that this difference was corrected mainly by reinforced visual information and to a lesser extent by the presence of a full-body avatar. Balance disruption triggered by the use of virtual reality can be offset by vertical visual information and/or by the presence of a full-body avatar. Further research is required on the effects of a full-body avatar.

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

Accurate and reliable human motion reconstruction is crucial for creating natural interactions of full-body avatars in Virtual Reality (VR) and entertainment applications. As the Metaverse and social applications gain popularity, users are seeking cost-effective solutions to create full-body animations that are comparable in quality to those produced by commercial motion capture systems. In order to provide affordable solutions though, it is important to minimize the number of sensors attached to the subject’s body. Unfortunately, reconstructing the full-body pose from sparse data is a heavily under-determined problem. Some studies that use IMU sensors face challenges in reconstructing the pose due to positional drift and ambiguity of the poses. In recent years, some mainstream VR systems have released 6-degree-of-freedom (6-DoF) tracking devices providing positional and rotational information. Nevertheless, most solutions for reconstructing full-body poses rely on traditional inverse kinematics (IK) solutions, which often produce non-continuous and unnatural poses. In this article, we introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from a reduced set of six tracking devices. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses by learning the human motion manifold from motion capture data. Then, we employ a learned IK component, made of multiple lightweight feed-forward neural networks, to adjust the hands and feet toward the corresponding trackers. We extensively evaluate our method on publicly available motion capture datasets and with real-time live demos. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and can be used for users with different body dimensions and proportions.

Greater interpersonal distance in adults with autism.

Social interactions are often shaped by the space we prefer to maintain between us and others, that is, interpersonal distance. Being too distant or too close to a stranger can often be perceived as odd, and lead to atypical social interactions. This calibration of appropriate interpersonal distance thus constitutes an important social skill. Individuals with autism spectrum disorder (ASD, hereafter autism) often experience difficulties with this skill, and anecdotal accounts suggest atypical interpersonal distances in their social interactions. In the current study, we systematically measured interpersonal distance in individuals with autism using immersive virtual reality (IVR) to recreate a naturalistic interaction with a full body avatar of a similar age. Participants observed their own virtual body in first-person perspective, and the other avatar in two tasks: in the first task, they approached the other avatar (active), in the second one they were approached by the other avatar (passive). Two groups of neurotypical and autistic adults, performed both tasks. Autistic adults showed greater interpersonal distance when compared to non-autistic adults. Additionally, the difference between the passive and active conditions was smaller for non-autistic compared to autistic adults. Across the full sample, greater interpersonal distance was associated with higher autism-related traits. This study provides systematic evidence for greater interpersonal distance in autistic adults using a paradigm with high ecological validity and can be useful in informing the design of appropriate environmental adjustments for shared spaces.

Bodylab

Bodylab is a virtual reality system for creating full-body virtual reality avatars. It provides anatomically realistic mannequins as canvases, which can be painted and textured, sculpted and deformed, adorned with objects and animated with particle effects. The artist can work from a first or third person perspective, and variety of interaction modalities are made available. To assess the system, we invite costume designers and theatrical artists to provide expert reviews. We also describe the use of Bodylab in a live dance performance. We believe Bodylab to be the first virtual reality tool for in virtuo painting and sculpting of full-body avatars.

AvatarReX: Real-time Expressive Full-body Avatars

We present AvatarReX, a new method for learning NeRF-based full-body avatars from video data. The learnt avatar not only provides expressive control of the body, hands and the face together, but also supports real-time animation and rendering. To this end, we propose a compositional avatar representation, where the body, hands and the face are separately modeled in a way that the structural prior from parametric mesh templates is properly utilized without compromising representation flexibility. Furthermore, we disentangle the geometry and appearance for each part. With these technical designs, we propose a dedicated deferred rendering pipeline, which can be executed at a real-time framerate to synthesize high-quality free-view images. The disentanglement of geometry and appearance also allows us to design a two-pass training strategy that combines volume rendering and surface rendering for network training. In this way, patch-level supervision can be applied to force the network to learn sharp appearance details on the basis of geometry estimation. Overall, our method enables automatic construction of expressive full-body avatars with real-time rendering capability, and can generate photo-realistic images with dynamic details for novel body motions and facial expressions.

Embodiment of virtual feet correlates with motor performance in a target-stepping task: a pilot study

Immersive Virtual Reality (IVR) has gained popularity in neurorehabilitation for its potential to increase patients’ motivation and engagement. A crucial yet relatively unexplored aspect of IVR interfaces is the patients’ representation in the virtual world, such as with an avatar. A higher level of embodiment over avatars has been shown to enhance motor performance during upper limb training and has the potential to be employed to enhance neurorehabilitation. However, the relationship between avatar embodiment and gait performance remains unexplored. In this work, we present the results of a pilot study with 12 healthy young participants that evaluates the effect of different virtual lower limb representations on foot placement accuracy while stepping over a trail of 16 virtual targets. We compared three levels of virtual representation: i) a full-body avatar, ii) only feet, and iii) no representation. Full-body tracking is computed using standard VR trackers to synchronize the avatar with the participants’ motions. Foot placement accuracy is measured as the distance between the foot’s center of mass and the center of the selected virtual target. Additionally, we evaluated the level of embodiment over each virtual representation through a questionnaire. Our findings indicate that foot placement accuracy increases with some form of virtual representation, either full-body or foot, compared to having no virtual representation. However, the foot and full-body representations do not show significant differences in accuracy. Importantly, we found a negative correlation between the level of embodiment of the foot representation and the distance between the placed foot and the target. However, no such correlation was found for the full-body representation. Our results highlight the importance of embodying a virtual representation of the foot when performing a task that requires accurate foot placement. However, showing a full-body avatar does not appear to further enhance accuracy. Moreover, our results suggest that the level of embodiment of the virtual feet might modulate motor performance in this stepping task. This work motivates future research on the effect of embodiment over virtual representations on motor control to be exploited for IVR gait rehabilitation.

Impact of Full-Body Avatars in Immersive Multiplayer Virtual Reality Training for Police Forces

Recent advancements in virtual reality technology contributed to the development of immersive games and simulations of any kind. Simulations are a valuable tool for police forces to practice specific skills within threatening, dangerous, and stressful situations without dangerous real-world implications. Effective virtual simulations need to be realistic in terms of actions and also need to elicit emotion, engagement, and stress. To this end, we developed a multiplayer virtual reality training simulation for police forces to explore the effect of full-body avatars, especially regarding the stress level and response to threats. We analyzed the statistical significance and effect sizes of the data from 32 police officers under two conditions: either they could see an assailant with a full-body avatar or only a head and hands. The results reveal an increase in stress level and threat during the immersion with the full-body avatar. This was shown objectively by analyzing the heart rate variability and subjectively by examining a questionnaire in response to a threat to the assailant’s virtual body. These findings indicate that an assailant with a full-body avatar is beneficial during training for police forces, as it increases the feeling of a threat and contributes to a more realistic training experience.

Dressing Avatars

Despite recent progress in developing animatable full-body avatars, realistic modeling of clothing - one of the core aspects of human self-expression - remains an open challenge. State-of-the-art physical simulation methods can generate realistically behaving clothing geometry at interactive rates. Modeling photorealistic appearance, however, usually requires physically-based rendering which is too expensive for interactive applications. On the other hand, data-driven deep appearance models are capable of efficiently producing realistic appearance, but struggle at synthesizing geometry of highly dynamic clothing and handling challenging body-clothing configurations. To this end, we introduce pose-driven avatars with explicit modeling of clothing that exhibit both photorealistic appearance learned from real-world data and realistic clothing dynamics. The key idea is to introduce a neural clothing appearance model that operates on top of explicit geometry: at training time we use high-fidelity tracking, whereas at animation time we rely on physically simulated geometry. Our core contribution is a physically-inspired appearance network, capable of generating photorealistic appearance with view-dependent and dynamic shadowing effects even for unseen body-clothing configurations. We conduct a thorough evaluation of our model and demonstrate diverse animation results on several subjects and different types of clothing. Unlike previous work on photorealistic full-body avatars, our approach can produce much richer dynamics and more realistic deformations even for many examples of loose clothing. We also demonstrate that our formulation naturally allows clothing to be used with avatars of different people while staying fully animatable, thus enabling, for the first time, photorealistic avatars with novel clothing.

An Efficient Redirection of Positional Interactions in Mixed/ Augmented Reality

With the rise in remote work culture and increased computing capabilities of head-mounted displays (HMDs), more immersive, collaborative experiences are desired in remote–local mixed/augmented reality (MR/AR). Photorealistic full-body avatar representations of users in remote workspace interactions have shown to have increased social presence, nonverbal behavior, and engagement. However, a direct mapping of the body pose angles from local to the remote workspace will, in most cases, result in positional errors during human–object interaction, caused by the dissimilarity between remote and local workspaces. Hence, the interaction must be retargeted, but it should be retargeted in such a way that the original intent of the body pose should be preserved. However, these two objectives sometimes contradict each other. As a result, a multi-objective optimization (MO) problem can be formulated where the primary objective is to minimize positional errors and the secondary objective is to preserve the original interaction body pose. The current state-of-the-art solution uses an evolutionary computation-based inverse kinematic (IK) approach to solve the MO problem where the weights between the objectives must be set by the user based on trial and error, leading to a suboptimal solution. In this paper, we present a new dynamic weight allocation approach to this problem, where a user has the flexibility to set a chosen minimum error tolerance, and the weights will be distributed between the objectives based on a dynamic allocation algorithm. We have used a two-pronged approach to test the adaptability and robustness of this mechanism: (i) on motion-captured human animations of varying levels of speeds, error tolerances, redirections and (ii) we conducted an experiment involving 12 human participants and recorded, redirected their actions performed during a book-shelving task in AR. Compared to the static weighting, the dynamic weighted mechanism showed a net ([Formula: see text] objective) decrease in error ranging from 20.5% to 34.42% across varying animation speeds and a decrease in error ranging from 11.44% to 36.2% for the recorded human actions during the AR task, demonstrating its robustness and better pose preservation across interactions.

Virtual reality for immersive multi-user firefighter-training scenarios

BackgroundVirtual reality (VR) applications can be used to provide comprehensive training scenarios that are difficult or impossible to represent in physical configurations. This includes team training for emergency services such as firefighting. Creating a high level of immersion is essential for achieving effective virtual training. In this respect, motion-capture systems offer the possibility of creating highly immersive multi-user training experiences, including full-body avatars. MethodsThis study presents a preliminary prototype that helps extinguish a virtual fire on a container ship as a VR training scenario. The prototype provides a full-body and multi-user VR experience based on the synthesis of position data provided by the motion-capture system and orientation data from the VR headsets. Moreover, the prototype facilitates an initial evaluation of the results. ResultsThe results confirm the value of using VR for training procedures that are difficult to implement in the real world. Furthermore, the results show that motion-capture-based VR technologies are particularly useful for firefighting training, in which participants can collaborate in difficult-to-access environments. However, this study also indicates that increasing the immersion in such training remains a challenge. ConclusionsThis study presents a prototypical VR application that enables the multi-user training of maritime firefighters. Future research should evaluate the initial results, provide more extensive training scenarios, and measure the training progress.

Topographic design in wearable MXene sensors with in-sensor machine learning for full-body avatar reconstruction

Wearable strain sensors that detect joint/muscle strain changes become prevalent at human–machine interfaces for full-body motion monitoring. However, most wearable devices cannot offer customizable opportunities to match the sensor characteristics with specific deformation ranges of joints/muscles, resulting in suboptimal performance. Adequate wearable strain sensor design is highly required to achieve user-designated working windows without sacrificing high sensitivity, accompanied with real-time data processing. Herein, wearable Ti3C2Tx MXene sensor modules are fabricated with in-sensor machine learning (ML) models, either functioning via wireless streaming or edge computing, for full-body motion classifications and avatar reconstruction. Through topographic design on piezoresistive nanolayers, the wearable strain sensor modules exhibited ultrahigh sensitivities within the working windows that meet all joint deformation ranges. By integrating the wearable sensors with a ML chip, an edge sensor module is fabricated, enabling in-sensor reconstruction of high-precision avatar animations that mimic continuous full-body motions with an average avatar determination error of 3.5 cm, without additional computing devices.

QuarantivityVR: Supporting Self-Embodiment for Non-HMD Users in Asymmetric Social VR Games

Abstract The prevalence of immersive head-mounted display (HMD) social virtual reality (VR) applications introduced asymmetric interaction among users within the virtual environment (VE). Therefore, researchers opted for (1) exploring the asymmetric social VR interaction dynamics in only co-located setups, (2) assigning interdependent roles to both HMD and non-HMD users, and (3) representing non-HMD users as abstract avatars in the VE. Therefore, we investigate the feasibility of supporting Self-Embodiment in an asymmetric VR interaction mode in a remote setup. To this end, we designed an asymmetric social VR game, QuarantivityVR, to (1) support sense of self-embodiment for non-HMD users in a remote setting by representing them as realistic full-body avatars within the VE, (2) augment visual-motor synchrony for the non-HMD users to increase their sense of agency and presence by detecting their motion through Kinect sensor and laptop’s webcam. During the game, each player performs three activities in succession, namely movie-guessing, spelling-bee, and answering mathematical questions. We believe that our work will act as a step towards the inclusion of a wide spectrum of users that can not afford full immersion and will aid researchers in creating enjoyable interactions for both users in the physical and virtual spaces.

Reaching articular limits can negatively impact embodiment in virtual reality.

Providing Virtual Reality(VR) users with a 3D representation of their body complements the experience of immersion and presence in the virtual world with the experience of being physically located and more personally involved. A full-body avatar representation is known to induce a Sense of Embodiment (SoE) for this virtual body, which is associated with improvements in task performance, motivation and motor learning. Recent experimental research on embodiment provides useful guidelines, indicating the extent of discrepancy tolerated by users and, conversely, the limits and disruptive events that lead to a break in embodiment (BiE). Based on previous works on the limit of agency under movement distortion, this paper describes, studies and analyses the impact of a very common yet overlooked embodiment limitation linked to articular limits when performing a reaching movement. We demonstrate that perceiving the articular limit when fully extending the arm provides users with an additional internal proprioceptive feedback which, if not matched in the avatar's movement, leads to the disruptive realization of an incorrect posture mapping. This study complements previous works on self-contact and visuo-haptic conflicts and emphasizes the risk of disrupting the SoE when distorting users' movements or using a poorly-calibrated avatar.

AudienceMR: Extending the Local Space for Large-Scale Audience with Mixed Reality for Enhanced Remote Lecturer Experience

AudienceMR is designed as a multi-user mixed reality space that seamlessly extends the local user space to become a large, shared classroom where some of the audience members are seen seated in a real space, and more members are seen through an extended portal. AudienceMR can provide a sense of the presence of a large-scale crowd/audience with the associated spatial context. In contrast to virtual reality (VR), however, with mixed reality (MR), a lecturer can deliver content or conduct a performance from a real, actual, comfortable, and familiar local space, while interacting directly with real nearby objects, such as a desk, podium, educational props, instruments, and office materials. Such a design will elicit a realistic user experience closer to an actual classroom, which is currently prohibitive owing to the COVID-19 pandemic. This paper validated our hypothesis by conducting a comparative experiment assessing the lecturer’s experience with two independent variables: (1) an online classroom platform type, i.e., a 2D desktop video teleconference, a 2D video screen grid in VR, 3D VR, and AudienceMR, and (2) a student depiction, i.e., a 2D upper-body video screen and a 3D full-body avatar. Our experiment validated that AudienceMR exhibits a level of anxiety and fear of public speaking closer to that of a real classroom situation, and a higher social and spatial presence than 2D video grid-based solutions and even 3D VR. Compared to 3D VR, AudienceMR offers a more natural and easily usable real object-based interaction. Most subjects preferred AudienceMR over the alternatives despite the nuisance of having to wear a video see-through headset. Such qualities will result in information conveyance and an educational efficacy comparable to those of a real classroom, and better than those achieved through popular 2D desktop teleconferencing or immersive 3D VR solutions.

“Nice to see you virtually”: Thoughtful design and evaluation of virtual avatar of the other user in AR and VR based telexistence systems

This paper presents two studies investigating how physically remote telexistence users wish to see other users visualized as virtual avatars in a) augmented reality, and b) immersive virtual reality while conducting a collaborative task. To answer this research question, a telexistence system was designed and implemented with simple avatar designs. After that, visual examples of alternative avatar representations for both use cases were designed by thoughtfully altering the visual parameters of 36 virtual avatar examples. The avatar designs were first evaluated in a user study with 16 participants in conjunction with using an implemented telexistence system. As a follow-up an online survey with 43 respondents was used to record their preferences regarding virtual avatar appearance. The results suggest that users prefer the other user to be represented in a photorealistic full-body human avatar in both augmented reality and virtual reality due to its humanlike representation and affordances for interaction. In augmented reality, the choice for a hologram full body avatar was also popular due to its see-through appearance, which prevents a mix-up with a real person in the physical space.

Driving-signal aware full-body avatars

Driving-signal aware full-body avatars

Driving-signal aware full-body avatars

We present a learning-based method for building driving-signal aware full-body avatars. Our model is a conditional variational autoencoder that can be animated with incomplete driving signals, such as human pose and facial keypoints, and produces a high-quality representation of human geometry and view-dependent appearance. The core intuition behind our method is that better drivability and generalization can be achieved by disentangling the driving signals and remaining generative factors, which are not available during animation. To this end, we explicitly account for information deficiency in the driving signal by introducing a latent space that exclusively captures the remaining information, thus enabling the imputation of the missing factors required during full-body animation, while remaining faithful to the driving signal. We also propose a learnable localized compression for the driving signal which promotes better generalization, and helps minimize the influence of global chance-correlations often found in real datasets. For a given driving signal, the resulting variational model produces a compact space of uncertainty for missing factors that allows for an imputation strategy best suited to a particular application. We demonstrate the efficacy of our approach on the challenging problem of full-body animation for virtual telepresence with driving signals acquired from minimal sensors placed in the environment and mounted on a VR-headset.

Evaluation of the impact of different levels of self-representation and body tracking on the sense of presence and embodiment in immersive VR

The main goal of this paper is to investigate the effect of different types of self-representations through floating members (hands vs. hands + feet), virtual full body (hands + feet vs. full-body avatar), walking fidelity (static feet, simulated walking, real walking), and number of tracking points used (head + hands, head + hands + feet, head + hands + feet + hip) on the sense of presence and embodiment through questionnaires. The sample consisted of 98 participants divided into a total of six conditions in a between-subjects design. The HTC Vive headset, controllers, and trackers were used to perform the experiment. Users were tasked to find a series of hidden objects in a virtual environment and place them in a travel bag. We concluded that (1) the addition of feet to floating hands can impair the experienced realism ( $$p=0.039$$ ), (2) both floating members and full-body avatars can be used without affecting presence and embodiment ( $$p>0.05$$ ) as long as there is the same level of control over the self-representation, (3) simulated walking scores of presence and embodiment were similar when compared to static feet and real walking tracking data ( $$p>0.05$$ ), and (4) adding hip tracking overhead, hand and feet tracking (when using a full-body avatar) allows for a more realistic response to stimuli ( $$p=0.002$$ ) and a higher overall feeling of embodiment ( $$p=0.023$$ ).

Enhancing Virtual Walking Sensation Using Self-Avatar in First-Person Perspective and Foot Vibrations

Walking is a fundamental physical activity in humans. Various virtual walking systems have been developed using treadmill or leg-support devices. Using optic flow, foot vibrations simulating footsteps, and a walking avatar, we propose a virtual walking system that does not require limb action for seated users. We aim to investigate whether a full-body or hands-and-feet-only walking avatar with either the first-person (experiment 1) or third-person (experiment 2) perspective can convey the sensation of walking in a virtual environment through optic flows and foot vibrations. The viewing direction of the virtual camera and the head of the full-body avatar were linked to the actual user's head motion. We discovered that the full-body avatar with the first-person perspective enhanced the sensations of walking, leg action, and telepresence, either through synchronous or asynchronous foot vibrations. Although the hands-and-feet-only avatar with the first-person perspective enhanced the walking sensation and telepresence, compared with the no-avatar condition, its effect was less prominent than that of the full-body avatar. However, the full-body avatar with the third-person perspective did not enhance the sensations of walking and leg action; rather, it impaired the sensations of self-motion and telepresence. Synchronous or rhythmic foot vibrations enhanced the sensations of self-motion, waking, leg action, and telepresence, irrespective of the avatar condition. These results suggest that the full-body or hands-and-feet avatar is effective for creating virtual walking experiences from the first-person perspective, but not the third-person perspective, and that the foot vibrations simulating footsteps are effective, regardless of the avatar condition.

How preadolescents and adults remember and experience virtual reality: The role of avatar incarnation, emotion, and sense of presence

Incarnate bodies in the first-person view in virtual reality, especially with a head-mounted display, could impact cognitive processes relying on self-representation, such as memory, emotion, or sense of presence. The goal of this study is to investigate these effects by comparing adults’ and preadolescents’ behaviors. We manipulated incarnation by presenting the experience with or without an avatar (full-body avatar versus VR controllers), while manipulating emotion with presented stimuli. 40 adults and 32 preadolescents were both divided into two experimental groups (avatar versus no-avatar). Valence, arousal, and presence were self-evaluated, and a free recall task was performed. Physiological measurements were carried out to add objective support for our findings. We showed an emotional enhancement of memory in both age groups. Importantly, embodying an avatar permitted preadolescents to have the same memory performance than adults. Memory was predicted by valence and head movements in adults, but no predictors were found for preadolescents. Overall, preadolescents rated virtual environments as more arousing, more positive, and inducing more presence than adults. Surprisingly, avatar incarnation did not impact these measures, we propose that immersion at a sensorimotor level is a more crucial component than incarnation.