Digital Humans Research Articles

Expectancy Violations and Discontinuance Behavior in Live-Streaming Commerce: Exploring Human Interactions with Virtual Streamers.

Virtual streamers, as a typical application of AI-enabled digital humans, are increasingly being utilized in live-streaming commerce due to technological advancements and industry innovations. Although virtual streamers present several benefits, there is potential for adverse effects when they do not align with consumer expectations. Drawing upon expectancy violations theory, this study developed a theoretical model to explore whether and how consumers' expectation violations during human-virtual streamer interactions affect consumers' discontinuance behavior. Through an online questionnaire survey of 307 Chinese consumers with prior experience interacting with virtual streamers, this study used a partial least squares structural equation model to analyze the research model. The empirical results indicated that professionalism expectation violation, empathy expectation violation, and responsiveness expectation violation positively influenced consumers' distrust and dissatisfaction, which subsequently led to discontinuance behavior. This study contributes to the literature on live-streaming commerce, human-AI interaction, and expectancy violation theory. Furthermore, the findings offer valuable insights for practitioners in the field of live-streaming commerce by enabling them to formulate preventive or remedial strategies to mitigate potential negative outcomes when implementing virtual streamers.

A Diffusion Modeling-Based System for Teaching Dance to Digital Human

The introduction of artificial intelligence (AI) has triggered changes in modern dance education. This study investigates the application of diffusion-based modeling and virtual digital humans in dance instruction. Utilizing AI and digital technologies, the proposed system innovatively merges music-driven dance generation with virtual human-based teaching. It achieves this by extracting rhythmic and emotional information from music through audio analysis to generate corresponding dance sequences. The virtual human, functioning as a digital tutor, demonstrates dance movements in real time, enabling students to accurately learn and execute dance postures and rhythms. Analysis of the teaching outcomes, including effectiveness, naturalness, and fluidity, indicates that learning through the digital human results in enhanced user engagement and improved learning outcomes. Additionally, the diversity of dance movements is increased. This system enhances students’ motivation and learning efficacy, offering a novel approach to innovating dance education.

Film and Television Animation Production Technology Based on Expression Transfer and Virtual Digital Human

The world of film and TV animation has witnessed a revolutionary transformation with the combination of Expression transfer and digital virtual Human technology. This paper delves into the superior methodologies and technological improvements in the discipline of animation production, especially specializing in how those technology are redefining the requirements and practices of animation in film and television. Expression transfer technology, a groundbreaking approach in animation, entails the transfer of facial expressions from real actors to lively characters. This technique not most effective enhances the realism of lively characters but additionally lets in for a extra nuanced and emotive performance, bridging the space among conventional animation and stay-action performances. Digital Human era, however, entails creating extraordinarily practical virtual representations of people. Those digital humans aren’t mere caricatures or stylized variations however are reasonable in appearance and movement, way to advancements in motion seize, 3-D modeling, and synthetic intelligence. The mixture of those technologies is main to a new technology in animation where characters aren’t only visually stunning however also exhibit a depth of emotion and realism previously unimaginable. This paper explores diverse case research and applications of these technology in current animation, highlighting their effect on storytelling, person improvement, and viewer engagement. It also addresses the demanding situations and moral considerations in employing these technologies, which includes retaining artistic integrity and the capacity for misuse. The research concludes with a forward-searching attitude on how Expression transfer and digital digital Human technologies are set to redefine the future of movie and television animation, presenting new opportunities for creative expression and narrative intensity.

Digital Human Intelligent Interaction System Based on Multimodal Pre-training Mode

ABSTRACT As the forefront of human–computer intelligent interaction, digital humans have increasingly diverse application scenarios, but also face many challenges. In order to enhance the intelligence level and interaction capability of digital human, the study first optimizes the traditional UniLM model, then introduces the mechanism of multi-head attention, and mitigates the exposure bias by using adversarial training and random replacement of decoder to design an improved UniLM model, and finally applies the improved model to the digital human management system. The results show that the precision rate of the improved UniLM model is improved by 7.68%, 6.4%, and 4.96%, the recall rate is improved by 11.94%, 9.69%, and 8.83%, and the F1-score is improved by 8.34%, 6.41%, and 7.68% compared with the other three models, which proves that it has a better precision rate, robustness, and generalization ability. The perplexity of the improved UniLM model is 135, 95, 76, 71, and 55 under five text lengths, which is significantly lower than the other models, proving that its text generation ability is better. The above results demonstrate the performance of the research-designed digital human system based on the Improved UniLM model, which provides a direction for the further development of digital human technology.

Less Artificial, More Intelligent: Understanding Affinity, Trustworthiness, and Preference for Digital Humans

Practice- and policy-oriented abstract: Companies are increasingly deploying highly realistic digital human agents (DHAs) controlled by advanced AI for online customer service, tasks typically handled by chatbots. We conducted four experiments to assess users’ perceptions (trustworthiness, affinity, and willingness to work with) and behaviors while using DHAs, utilizing quantitative surveys, qualitative interviews, direct observations, and neurophysiological measurements. Our studies involved four DHAs, including two commercial products (found to be immature) and two future-focused ones (where participants believed the AI-controlled DHAs were human-controlled). In the first study, comparing perceptions of a DHA, chatbot, and human agent from descriptions revealed few differences between the DHA and chatbot. The second study, involving actual use of a commercial DHA, showed participants found it uncanny, robotic, or difficult to converse with. The third and fourth studies used a “Wizard of Oz” design, with participants believing a human-controlled DHA was AI-driven. Results showed a preference for human agents via video conferencing, but no significant differences between DHAs and human agents when visual fidelity was controlled. Current DHAs, despite communication issues, trigger more affinity than chatbots. When DHAs match human communication abilities, they are perceived similarly to human agents for simple tasks. This research also suggests DHAs may alleviate algorithm aversion.

Editable Co-Speech Gesture Synthesis Enhanced with Individual Representative Gestures

Co-speech gesture synthesis is a challenging task due to the complexity and uncertainty between gestures and speech. Gestures that accompany speech (i.e., Co-Speech Gesture) are an essential part of natural and efficient embodied human communication, as they work in tandem with speech to convey information more effectively. Although data-driven approaches have improved gesture synthesis, existing deep learning-based methods use deterministic modeling which could lead to averaging out predicted gestures. Additionally, these methods lack control over gesture generation such as user editing of generated results. In this paper, we propose an editable gesture synthesis method based on a learned pose script, which disentangles gestures into individual representative and rhythmic gestures to produce high-quality, diverse and realistic poses. Specifically, we first detect the time of occurrence of gestures in video sequences and transform them into pose scripts. Regression models are then built to predict the pose scripts. Next, learned pose scripts are used for gesture synthesis, while rhythmic gestures are modeled using a variational auto-encoder and a one-dimensional convolutional network. Moreover, we introduce a large-scale Chinese co-speech gesture synthesis dataset with multimodal annotations for training and evaluation, which will be publicly available to facilitate future research. The proposed method allows for the re-editing of generated results by changing the pose scripts for applications such as interactive digital humans. The experimental results show that this method generates more quality, more diverse, and realistic gestures than other existing methods.

3D facial modeling, animation, and rendering for digital humans: A survey

With the continuous advancement of 3D and human–computer interaction technologies, digital human systems have been widely implemented in our daily lives, such as 3D games, online education, and virtual assistants. This paper provides a comprehensive literature survey of the technologies vitally important to digital humans in 3D facial modeling, animation, and rendering. Firstly, we combed through the research development of 3D face modeling technology and summarized the 3D facial models. Then, we summed up various methods for animating the 3D face based on data from images, videos, and the like. Lastly, we delved deeply into the techniques for 3D face rendering, comparing traditional rendering with neural network rendering. In conclusion, we provided a forward-looking perspective on the current challenges and future development of 3D digital humans.

디지털 휴먼 캐릭터의 성격 특성과 안면 디자인의 관계 탐색

Due to the development of AI technology, the use of digital humans and their market value are on the rise. Accordingly, as an alternative approach to communication between digital humans and users and building brand image, the purpose of this study is to suggest design elements by applying the Enneagram, Carl Jung's personality types, and Sasang Constitution's personality characteristics to facial design. To this end, the connection between each personality theory and digital human facial design was identified. Alternative design elements were extracted by finding consistency between the nine personality types of the Enneagram, Carl Jung's four personality types, and the four personality traits of Sasang Constitution. There are four possible facial shapes: rectangular, inverted triangle, oval, and square. These are the criteria for facial proportions and include facial shape and overall facial features, specifically the size and protrusion of eyes, nose, mouth, and forehead, and facial features. The location of the central axis and the degree of sagging were specified. As a result of the study, the significance of this study was that it found consistency between each personality type used as an argument for this study and used this to suggest alternative design guidelines.

Research on Digital Virtual Humans in the Metaverse for Smart Teaching

With the rapid development of information technology and artificial intelligence, the virtual digital world known as the metaverse has become an important extension of the real world. This paper aims to explore the design, implementation, and application of digital virtual humans in the metaverse for smart teaching. Firstly, starting from the concept and characteristics of digital virtual humans, we introduce the current development status and application scenarios of digital virtual humans. Then, we expound on the concept of smart teaching and discuss the roles of digital virtual humans in smart teaching, such as teaching assistants, personalized learning tutors, and simulated learning environments. We also analyze the advantages and challenges of digital virtual humans in smart teaching, including aspects like language understanding, emotion recognition, and privacy security. Finally, this paper discusses the development trends and challenges of digital virtual humans, providing new ideas and directions for the education field, exploring how to achieve smart teaching in the virtual world, in order to better meet the learning needs of students and improve the quality and effectiveness of education.

A report of research on the digital human and digital medicine in China

The progress of science and technology has provided new methods and means for modern medical research, providing strong support and guarantee for modern clinical medicine. Science and technology have not only changed the traditional methods of surgery, but also changed the behavior and service mode of healthcare. The promotion and popularization of digital humans and digital medical research in China have been comprehensively applied in medical education, clinical medicine, aerospace, automotive manufacturing, sports, and artistic performances. This is a new thing, an emerging technology, and a brand-new undertaking.

Action unit intensity regression for facial MoCap aimed towards digital humans

AbstractDue to the increasing demand for virtual avatars, there has been a recent growth in the research and development of frameworks for realistic digital humans, which create a demand for realistic and adaptable facial motion capture systems. Most frameworks belong to private companies or represent high investments, which is why the creation of democratized solutions is relevant for the growth of digital human content creation. This research work proposes a facial motion capture framework for digital humans with the use of machine learning for facial codification intensity regression. The main focus is to use coded face movement intensities to generate realistic expressions on a digital human. The ablation studies performed on the regression models show that Neural Networks, using Histogram of Oriented Gradients as features, and with person-specific normalization, present overall better performance against other methods in the literature. With an RMSE of 0.052, the proposed framework offers reliable results that can be rendered in the face of a MetaHuman.

Fostering Inclusive Recruitment Interviews with Intelligent Digital Humans

This article is about an intelligent digital human model enhanced by artificial intelligence, designed to meet the requirements from a multinational company in need of training for their human resources personnel on bias-free recruitment interviews. We have been creating a new generation of avatars with social intelligence, who are capable not only of presenting a wide variety of topics in a dynamic and engaging manner, but also of interacting with the audience and communicating emotions and moods. We have been customizing avatars for role plays, building them as real interlocutors who facilitate training in how to handle difficult conversations by including aspects such as non-verbal communication, different communication styles, and diversity and inclusion. Practicing conversations with avatars accelerates learning from experience without the risks associated with learning in the field. At the end of each interview, timely feedback is provided so learners can determine how to improve their performance. These digital humans are able to perform like realistic human beings, challenging the interviewer both at a verbal and para-verbal level, as well on the cognitive and the emotional levels – making it easy for the interviewer to get trapped into biases and false assumptions. The key message is this: diversity and inclusion best practices are first of all about mindset.

3D facial animation driven by speech-video dual-modal signals

In recent years, the applications of digital humans have become increasingly widespread. One of the most challenging core technologies is the generation of highly realistic and automated 3D facial animation that combines facial movements and speech. The single-modal 3D facial animation driven by speech typically ignores the weak correlation between speech and upper facial movements as well as head posture. In contrast, the video-driven approach can perfectly solve the posture problem while obtaining natural expressions. However, mapping 2D facial information to 3D facial information may lead to information loss, which make lip synchronization generated by video-driven methods is not as good as the speech-driven methods trained on 4D facial data. Therefore, this paper proposes a dual-modal generation method that uses speech and video information to generate more natural and vivid 3D facial animation. Specifically, the lip movements related to speech are generated by speech-video information, while speech-uncorrelated postures and expressions are generated solely by video information. The speech-driven module is used to extract speech features, and its output lip animation is then used as the foundation for facial animation. The expression and pose module is used to extract temporal visual features for regressing expression and head posture parameters. We fuse speech and video features to obtain chin posture parameters related to lip movements, and use these parameters to fine-tune the lip animation generated form the speech-driven module. This paper introduces multiple consistency losses to enhance the network’s capability to generate expressions and postures. Experiments conducted on the LRS3, TCD-TIMIT and MEAD datasets show that the proposed method achieves better performance on evaluation metrics such as CER, WER, VER and VWER than the current state-of-the-art methods. In addition, a perceptual user study show that over 77% and 70% of cases believe that this paper’s method is better than the comparative algorithms EMOCA and SPECTRE in terms of realism. In terms of lip synchronization, it received over 79% and 66% of cases support, respectively. Both evaluation methods demonstrate the effectiveness of the proposed method.

Anthropometric evaluation through images: Findings from the SCANNER software package

ObjectiveTo compare the waist-to-height ratio (WHtR) agreement between synthetic data and the Smart Computerized Anthropometric NavigatioN and Evaluation Resource (SCANNER) software package. MethodsOne hundred and ten 3D digital humans (55 for each sex) were created to obtain synthetic values. WHtR was obtained through the waist circumference and height division, both in centimeters. These data were programmed and obtained directly from the synthetic models. SCANNER v0.01 was coded by the researchers using Matlab. Differences between the objective WHtR and the one the SCANNER software package estimated were quantified using standard errors, Spearman's correlation and the Bland-Altman plot. ResultsUsing the Spearman correlation, an agreement level of 0.982 was identified. Using the Bland-Altman plot, the agreement level was high, with a Rho value of 0.983 (95% CI: 0.977–0.988). Finally, when the standard errors were quantified, there was an overall error (between the synthetic data created and the computed one) of 0.49%, being higher in men (0.81%) than in women (0.18%). ConclusionsThe SCANNER software package is a straightforward tool that could facilitate the estimation of WHtR in distance participants or patients.

From Screen to Reality: Exploring the Evolution and Integration of Motion Capture Technology for Virtual Digital Humans

From Screen to Reality: Exploring the Evolution and Integration of Motion Capture Technology for Virtual Digital Humans

Neural Garment Dynamics via Manifold‐Aware Transformers

AbstractData driven and learning based solutions for modeling dynamic garments have significantly advanced, especially in the context of digital humans. However, existing approaches often focus on modeling garments with respect to a fixed parametric human body model and are limited to garment geometries that were seen during training. In this work, we take a different approach and model the dynamics of a garment by exploiting its local interactions with the underlying human body. Specifically, as the body moves, we detect local garment‐body collisions, which drive the deformation of the garment. At the core of our approach is a mesh‐agnostic garment representation and a manifold‐aware transformer network design, which together enable our method to generalize to unseen garment and body geometries. We evaluate our approach on a wide variety of garment types and motion sequences and provide competitive qualitative and quantitative results with respect to the state of the art.

DiT-Gesture: A Speech-Only Approach to Stylized Gesture Generation

The generation of co-speech gestures for digital humans is an emerging area in the field of virtual human creation. Prior research has progressed by using acoustic and semantic information as input and adopting a classification method to identify the person’s ID and emotion for driving co-speech gesture generation. However, this endeavor still faces significant challenges. These challenges go beyond the intricate interplay among co-speech gestures, speech acoustic, and semantics; they also encompass the complexities associated with personality, emotion, and other obscure but important factors. This paper introduces “DiT-Gestures”, a speech-conditional diffusion-based and non-autoregressive transformer-based generative model with the WavLM pre-trained model and a dynamic mask attention network (DMAN). It can produce individual and stylized full-body co-speech gestures by only using raw speech audio, eliminating the need for complex multimodal processing and manual annotation. Firstly, considering that speech audio contains acoustic and semantic features and conveys personality traits, emotions, and more subtle information related to accompanying gestures, we pioneer the adaptation of WavLM, a large-scale pre-trained model, to extract the style from raw audio information. Secondly, we replace the causal mask by introducing a learnable dynamic mask for better local modeling in the neighborhood of the target frames. Extensive subjective evaluation experiments are conducted on the Trinity, ZEGGS, and BEAT datasets to confirm WavLM’s and the model’s ability to synthesize natural co-speech gestures with various styles.

GANtlitz: Ultra High Resolution Generative Model for Multi‐Modal Face Textures

AbstractHigh‐resolution texture maps are essential to render photoreal digital humans for visual effects or to generate data for machine learning. The acquisition of high resolution assets at scale is cumbersome, it involves enrolling a large number of human subjects, using expensive multi‐view camera setups, and significant manual artistic effort to align the textures. To alleviate these problems, we introduce GANtlitz (A play on the german noun Antlitz, meaning face), a generative model that can synthesize multi‐modal ultra‐high‐resolution face appearance maps for novel identities. Our method solves three distinct challenges: 1) unavailability of a very large data corpus generally required for training generative models, 2) memory and computational limitations of training a GAN at ultra‐high resolutions, and 3) consistency of appearance features such as skin color, pores and wrinkles in high‐resolution textures across different modalities. We introduce dual‐style blocks, an extension to the style blocks of the StyleGAN2 architecture, which improve multi‐modal synthesis. Our patch‐based architecture is trained only on image patches obtained from a small set of face textures (<100) and yet allows us to generate seamless appearance maps of novel identities at 6k × 4k resolution. Extensive qualitative and quantitative evaluations and baseline comparisons show the efficacy of our proposed system. (see https://www.acm.org/publications/class-2012)

Enhancing Communication Between Sonographers and Patients in Diverse Cultural Settings via Digital Human Role-Playing

Effective communication skills are essential for sonographers to build trust, to explain examination procedures to the patient in non-technical terms, to alleviate anxiety and gain patient consent and collaboration, and to provide information at a pace suitable for the patient. In order to communicate effectively, the sonographer needs to be able to communicate empathetically, adjusting their communication style to meet the needs of different audiences. This is particularly challenging when working with a diverse and multicultural group of patients where the risk of misinterpretation is higher. Students are provided with the opportunity to practice dialogues with virtual patients that are able to interact as real human beings, communicating concerns, emotions, and moods both at a verbal and non-verbal level. Coaching through digital humans accelerates learning from experience without the risks associated with learning in the field.

Improving diversity of speech‐driven gesture generation with memory networks as dynamic dictionaries

AbstractGenerating co‐speech gestures for interactive digital humans remains challenging because of the indeterministic nature of the problem. The authors observe that gestures generated from speech audio or text by existing neural methods often contain less movement shift than expected, which can be viewed as slow or dull. Thus, a new generative model coupled with memory networks as dynamic dictionaries for speech‐driven gesture generation with improved diversity is proposed. More specifically, the dictionary network dynamically stores connections between text and pose features in a list of key‐value pairs as the memory for the pose generation network to look up; the pose generation network then merges the matching pose features and input audio features for generating the final pose sequences. To make the improvements more accurately measurable, a new objective evaluation metric for gesture diversity that can remove the influence of low‐quality motions is also proposed and tested. Quantitative and qualitative experiments demonstrate that the proposed architecture succeeds in generating gestures with improved diversity.