Exploring Familiarity and Knowledgeability in Conversational Virtual Agents

Previous research has shown that people sometimes feel uncomfortable in the presence of objects that appear almost, but not quite, human-like. This phenomenon, known as the Uncanny Valley effect, may present an impediment to the widespread acceptance of humanoid robots in everyday life, e.g. as school teachers, nurses, nannies, or clerks. It is therefore important for the field of human-robot interaction to identify ways in which the Uncanny Valley effect can be overcome. The current study was designed to test the hypothesis that sharing a stressful experience with a robot may serve to establish rapport with that specific robot, leading to a boost in people’s willingness to interact with the robot. To this end, we conducted an experiment in which participants underwent a pleasant experience with one virtual agent and a mildly stressful experience with a second virtual agent (the Induction Phase). Next (the Test Phase), they carried out a new series of tasks, before each of which they selected one virtual agent (either one of the two familiar agents or one of two unfamiliar agents) to be their partner for that task. The results indicate that participants chose to partner with the virtual agent with whom they had shared the stressful experience more often than the virtual agent with whom they had shared the pleasant experience, and more often than either of the other two agents. This finding, as well as the results of questionnaires that were administered at the end of the experiment, support the hypothesis that shared stressful experiences help overcome the Uncanny Valley effect.

The uncanny valley effect describes a phenomenon where humanoid, almost lifelike virtual agents evoke feelings of discomfort in the observers. The Pathogen Avoidance Hypothesis proposes that these feelings are based on a cognitive mechanism that originally evolved to motivate pathogen avoidance. Slight imperfections in virtual agents are thereby misconceived as cues of communicable diseases that elicit disgust and avoidance behavior. Whether uncanny entities also increase mucosal immune responses, particularly when immediate avoidance is impossible, currently remains elusive. The present study examined the link between the uncanny valley and mucosal immune responses by comparing changes in the salivary antibody secretory immunoglobulin A (sIgA). For this, we developed three sets of virtual agents (cartoon, uncanny, and realistic agents) that differed in their human-likeness and uncanniness. Agents were encountered in a social virtual reality (VR) task that required close approach, making avoidance impossible. We found that only the set of uncanny agents who showed slight deviations from normal human appearance, yet were still considered as more human-like than the cartoon agents, evoked a significant increase in sIgA release during social interactions in VR. This conforms to error-management theories suggesting that perceptual systems are biased towards false positive responses to potential contagion cues, reducing the risk of missing actual health threats. Our findings further indicate that presence in virtual reality was likely essential for effective immune activation by uncanny agents, as increased sIgA levels correlated with deeper involvement in the virtual environment. Moreover, the immune response may have been automatically triggered by perceived threats from these agents, as conscious evaluations of interoceptive awareness, state anxiety, and eeriness during VR did not correlate with higher antibody release. Altogether, these data show that perception of uncanniness in VR evokes mucosal immune responses resembling those related to real contagion threats.

The dynamic development of solutions in the field of virtual and augmented reality poses challenges to designers. These challenges relate to both technical conditions, including hardware capabilities and software solutions, as well as psychophysical constructs conditioning the end users’ reception of the generated multimedia message. One of the key elements of the virtual and augmented reality experience is the interaction with the system through a virtual agent represented by an avatar, i.e. a reflection of the image of a participant in the virtual world, carrying on a conversation with the user. This paper presents a proposed software and hardware solution for conducting multifaceted research and comparative analysis of diverse interfaces and human-computer interaction in virtual and augmented reality. In the course of this research, statistically significant results were obtained indicating differences in perception between three types of virtual agents. Each of them represented by different avatars in a specially created research environment that allowed to conduct usability tests under reproducible conditions to study user interaction in virtual reality.

Virtual agents are typical assistance tools for navigation and interaction in Virtual Reality (VR) tour, training, education, etc. It has been demonstrated that the gaits, gestures, gazes, and positions of virtual agents are major factors that affect the user's perception and experience for seated and standing VR. In this paper, we present a novel position-aware virtual agent locomotion method, called PAVAL, that can perform virtual agent positioning (position+orientation) in real time for room-scale VR navigation assistance. We first analyze design guidelines for virtual agent locomotion and model the problem using the positions of the user and the surrounding virtual objects. Then we conduct a one-off preliminary study to collect subjective data and present a model for virtual agent positioning prediction with fixed user position. Based on the model, we propose an algorithm to optimize the object of interest, virtual agent position, and virtual agent orientation in sequence for virtual agent locomotion. As a result, during user navigation in a virtual scene, the virtual agent automatically moves in real time and introduces virtual object information to the user. We evaluate PAVAL and two alternative methods via a user study with humanoid virtual agents in various scenes, including virtual museum, factory, and school gym. The results reveal that our method is superior to the baseline condition.

Human interactions with the trainer during physical training can be highly engaging and motivating [1] and are based on rapport as a dynamic structure of mutual attentiveness and coordination [2]. Human-Agent interaction in virtual reality (VR) aims to establish interaction patterns and rapport with virtual agents similar to real life. Research shows that users react towards virtual agents similar to real people [3] and that rapport is established similar to human rapport [4]. Therefore, rapport with virtual trainers in exergames is used to enhance an engaging and motivating user experience.In this paper we report on the results from an evaluation study on perceptions and interactions with the virtual trainer “Anna” after a 6-week exergame training for senior patients with hypertension. The human-like “Anna” is the key element of interaction design in a gamified series of exergames developed in the bewARe project. Anna was developed as a realistic, full body, female figure (silhouette) to motivate participation in the VR training. The primary goal of our research was to evaluate to what extent senior users can establish rapport with the virtual trainer as a factor contributing to positive user experience and training outcomes. The evaluation was conducted with 23 participants aged 65 and older with diagnosed hypertension. The virtual trainer Anna facilitated user participation in both exergames by giving instructions, modeling movements and providing feedback during the exergames in the HTC Vive Pro Headset. We used the 15-item rapport scale by [5] to measure rapport. The study also applied further research instruments to explore perceptions of the virtual trainer such as the trait list with 9 items describing selected features of the virtual trainer, and the bipolar uncanniness questionnaire with 40 adjectives used to assess possible Uncanny Valley effects described by [6]. The results of the rapport scale indicate that the design of the virtual trainer was effective for establishing rapport especially in terms of building a relationship with the virtual trainer and enhancing the engagement of senior users to participate in the VR training. However, the design was less effective in creating a positive perception of the trainer as a warm, caring and respectful agent. The overall median of the rapport scale was 6 (Min:1,Max:8). The results of the evaluation of the trait list revealed that voice quality, speech pauses and bodily movements were rated highest, followed by head and hand movements. The lowest values were researched for face expression. In the Uncanny Valley questionnaire, the median value for the humannes scale was 1 (Min:-3,Max:3), for the attractiveness 1 (Min:0,Max:3) and for the eeriness 0 (Min:-1,Max:0). Furthermore, the paper explores the relationships between the rapport scores and the perception of senior trainees of selected characteristics of the virtual agent and the uncanniness scale. Finally, given the diverse results from the study, the paper discusses possible design options for enhanced rapport and motivational effects of a virtual trainer based on the analysis of literature in related areas.

Background: Exposure therapy involves exposure to feared stimuli and is considered to be the gold-standard treatment for anxiety disorders. While its application in Virtual Reality (VR) has been very successful for phobic disorders, the effects of exposure to virtual social stimuli in Social Anxiety Disorder are heterogeneous. This difference has been linked to demands on realism and presence, particularly social presence, as a pre-requisite in evoking emotional experiences in virtual social interactions. So far, however, the influence of social presence on emotional experience in social interactions with virtual agents remains unknown.Objective: We investigated the relationship between realism and social presence and the moderating effect of social presence on the relationship between agent behavior and experienced emotions in virtual social interaction.Methods: Healthy participants (N = 51) faced virtual agents showing supportive and dismissive behaviors in two virtual environments (short interactions and oral presentations). At first, participants performed five blocks of short one-on-one interactions with virtual agents (two male and two female agents per block). Secondly, participants gave five presentations in front of an audience of 16 agents. In each scenario, agent behavior was a within subjects factor, resulting in one block of neutral, two blocks of negative, and two blocks of positive agent behavior. Ratings of agent behavior (valence and realism), experience (valence and arousal), and presence (physical and social) were collected after every block. Moderator effects were investigated using mixed linear models with random intercepts. Correlations were analyzed via repeated measures correlations.Results: Ratings of valence of agent behaviors showed reliable relationships with experienced valence and less reliable relationships with experienced arousal. These relationships were moderated by social presence in the presentation scenario. Results for the interaction scenario were weaker but potentially promising for experimental studies. Variations in social presence and realism over time were correlated but social presence proved a more reliable moderator.Conclusion: Our findings emphasize the role of social presence for emotional experience in response to specific agent behaviors in virtual social interactions. While these findings should be replicated with experimental designs and in clinical samples, variability in social presence might account for heterogeneity in efficacy of virtual exposure to treat social anxiety disorder.

This research paper explores the integration of Large Language Models (LLMs) into Virtual Reality (VR) environments to enhance human-like interactions with non-player Characters (NPCs) in professional training simulations. Current VR systems suffer from repetitive, scripted dialogues that lack naturalness and adaptability, reducing user engagement and learning outcomes. To address this, the study developed a proof-of-concept VR system using state-of-the-art LLMs, including GPT-4, Claude 3.5, and Large Language Model Meta AI (LLAMA), evaluated through the Massive Multitask Language Understanding (MMLU) benchmark. The Agile methodology was employed to iteratively refine the system based on user feedback, optimising NPC interactions for contextual relevance and realism. Results demonstrated significant improvements in naturalness, engagement, and context maintenance, with LLaMA-powered NPCs outperforming others in user acceptance testing. These findings underscore the potential of LLMs to revolutionise VR based training by delivering lifelike, context-aware dialogues and provide a robust foundation for future research in AI-driven immersive environments.

Scientific evidence suggests that virtual reality (VR) could potentially help patients tolerate painful medical procedures and conditions. The aim of this study was to evaluate the efficacy of virtual reality on pain tolerance and threshold. A within-subjects experimental study design was conducted on 53 female students at Qassim University in Saudi Arabia. Each participant completed three rounds of assessment: one baseline (no VR) and two VR immersion (passive and interactive) in random order sequence. During each round, participants submerged their non-dominant hand into an ice bath; pain threshold and tolerance were measured as outcomes and analyzed using repeated measures ANOVA. Participants had both higher pain threshold andtolerance during interactive and passive VR rounds incomparison to the non-VR baseline assessment (p<0.05). Participants had greater pain tolerance during the interactive VR condition compared to the passive VR condition (p<0.001). VR experiences increase pain threshold and tolerance with minimal side effects, and the larger effects were demonstrated using interactive games. Interactive VR gaming should be considered and tested as a treatment for pain.

As digitization has transformed media and Virtual Reality (VR) and Augmented Reality (AR) have evolved from a research area to a commodity, digital media content creators are experimenting with VR digital storytelling. Most VR experiences are developed in a lean-back 360 immersive format with limited viewer interaction. This paper presents an innovative, first person Point of View (POV) highly interactive VR documentary titled "The Revolution of 1897 in Crete" aiming to enhance interactivity, eliminate character-related uncanny valley effects, promote non-obtrusive subtitle placement and techniques for familiarization with complicated VR controllers. Viewers are active participants through interactive tasks involving sound and music, supporting "doing" rather than just observing. We initially analyze interactive VR experiences to identify the specific characteristics of this new medium, composing a VR grammar. We evaluate task load and usability of the presented interactive and multimodal VR documentary. We propose design recommendations for creators of VR and AR media experiences in any context including cultural heritage.

Virtual characters can elicit an uncomfortable sensation usually known as Uncanny Valley (UV). Evidence suggests that animation exacerbates the UV effect. This paper revisits the UV hypothesis to assess its effects on people’s perception of virtual characters, to try to answer the questions “What happens to the perceptual comfort if, in addition to images and videos of characters, we include those characters in an interactive Virtual Reality (VR) environment?”, and “Considering the VR environment, do people feel equally comfortable if we increase the number of characters?”. The results indicate that there are differences in the perception of scenarios with low and high densities.

Virtual reality (VR) environments are increasingly popular for various applications, and the appearance of virtual characters is a critical factor that influences user behaviors. In this study, we aimed to investigate the impact of avatar and agent appearances on pre-touch proxemics in VR. To achieve this goal, we designed experiments utilizing three user avatars (man/woman/robot) and three virtual agents (man/woman/robot). Specifically, we measured the pre-touch reaction distances to the face and body, which are the distances at which a person starts to feel uncomfortable before being touched. We examined how these distances varied based on the appearances of avatars, agents, and user gender. Our results revealed that the appearance of avatars and agents significantly impacted pre-touch reaction distances. Specifically, those using a female avatar tended to maintain larger distances before their face and body to be touched, and people also preferred greater distances before being touched by a robot agent. Interestingly, we observed no effects of user gender on pre-touch reaction distances. These findings have implications for the design and implementation of VR systems, as they suggest that avatar and agent appearances play a significant role in shaping users' perceptions of pre-touch proxemics. Our study highlights the importance of considering these factors when creating immersive and socially acceptable VR experiences.

Virtual reality (VR) has emerged as a viable tool for immersive learning, yet the impact of individual differences on user interaction and training outcomes in VR remains somewhat underexplored. This study aims to address this gap by examining the relationship between a number of individual difference and personality variables, interactivity (active vs. passive engagement), and cognitive load in VR. To investigate this, 79 participants were recruited from a university participant pool. One of the variables that emerged as related to performance was handedness(i.e., whether a user is left- or right-handed). Previous research has shown that handedness in VR can explain differences in movement speed, interactivity, and embodiment. Specifically, aligning controls and interactions with the user's dominant hand preferences. In our study, 72 right-handed and 7 left-handed individuals(10%, which is representative of handedness in the population)completed a series of procedural tasks in VR. These tasks were designed with varying levels of interactivity—the active instruction condition affording more user-driven engagement than the passive instruction condition. Performance metrics such as accuracy in the VR post-training test were collected, while cognitive load was measured post-training using a subjective questionnaire. Data analyses were performed using Analyses of Variance and multiple regressions. Results revealed no significant effect of handedness on cognitive load and no interaction effects between handedness and interactivity in the VR post-training scores. However, we found that handedness and cognitive load were significant predictors of procedural training outcomes. These findings suggest that handedness may influence training in VR, underscoring its potential role in shaping learning outcomes. Based on these results, we offer practical considerations for implementing VR-based training across safety critical industries like aviation, defense, and healthcare. These include recommendations for designing VR interfaces that accommodate handedness variability and guidelines for optimizing interactivity to enhance learning outcomes for both right-handed and left-handed users.

The proposed research introduces an innovative Virtual Reality (VR) and Large Language Model (LLM) architecture to enhance the learning process across diverse educational contexts, ranging from school to industrial settings. everaging the capabilities of LLMs and Retrieval-Augmented Generation (RAG), the architecture centers around an immersive VR application. This application empowers students of all backgrounds to interactively engage with their environment by posing questions and receiving informative responses in text format and with visual hints in VR, thereby fostering a dynamic learning experience. LLMs with RAG act as the backbones of this architecture, facilitating the integration of private or domain-specific data into the learning process. By seamlessly connecting various data sources through data connectors, RAG overcomes the challenge of disparate and siloed information repositories, including APIs, PDFs, SQL databases, and more. The data indexes provided by RAG solutions further streamline this process by structuring the ingested data into formats optimized for consumption by LLMs. An empirical study was conducted to evaluate the effectiveness of this VR and LLM architecture. Twenty participants, divided into Experimental and Control groups, were selected to assess the impact on their learning process. The Experimental group utilized the immersive VR application, which allowed interactive engagement with the educational environment, while the Control group followed traditional learning methods. The study revealed significant improvements in learning outcomes for the Experimental group, demonstrating the potential of integrating VR and LLMs in enhancing comprehension and engagement in learning contexts. This study presents an innovative approach that capitalizes on the synergy between LLMs and immersive VR technology, opening avenues for a transformative learning experience that transcends traditional boundaries and empowers learners across a spectrum of educational landscapes.

Human-environment interactions normally occur in the physical milieu, and thus by medium of the body and within the space immediately adjacent to and surrounding the body; the peri-personal space (PPS). However, human interactions increasingly occur with or within virtual environments, and hence novel approaches and metrics must be developed to index human-environment interactions in virtual reality (VR). Here we present a multisensory task that measures the spatial extent of human PPS in real, virtual, and augmented realities. We validated it in a mixed reality ecosystem in which real environment and virtual objects are blended together in order to administer and control visual, auditory, and tactile stimuli in ecologically valid conditions. Within this mixed-reality environment, participants are asked to respond as fast as possible to tactile stimuli on their body, while task-irrelevant visual or audio-visual stimuli approach their body. Results demonstrate that, in analogy with observations derived from monkey electrophysiology and in real environmental surroundings, tactile detection is enhanced when visual or auditory stimuli are close to the body, and not when far from it. We then calculate the location where this multisensory facilitation occurs as a proxy of the boundary of PPS. We observe that mapping of PPS via audio-visual, as opposed to visual alone, looming stimuli results in sigmoidal fits – allowing for the bifurcation between near and far space – with greater goodness of fit. In sum, our approach is able to capture the boundaries of PPS on a spatial continuum, at the individual-subject level, and within a fully controlled and previously laboratory-validated setup, while maintaining the richness and ecological validity of real-life events. The task can therefore be applied to study the properties of peri-personal space in humans and to index the features governing human-environment interactions in virtual or mixed reality. We propose PPS as an ecologically valid and neurophysiologically established metric in the study of the impact of VR and related technologies on society and individuals.

Agency, or the capacity to take intentional actions, is considered one of the primary affordances of virtual reality (VR) for learning. VR is expected to increase learners' agency because it allows for full‐body interactivity from a first‐person perspective, giving them novel ways of interacting with the digital environment. Yet, agency in immersive learning has not been well‐studied relative to other affordances like presence, and more evidence is needed to understand how varied media and designs heighten or diminish agency. This mixed‐method study addressed this need by developing and validating measures of sense of agency with 30 high school students who used VR field trips in their engineering class over four lessons. By comparing immersive videos to video game‐like interactive graphical environments, the study illustrates some of the complexities of agency in VR. The findings indicate agency is not a unidimensional construct nor is it equivalent to full‐body interactivity in VR as learners felt some types of agency when using immersive videos. Furthermore, learners' identities moderated associations between the type of VR media and their sense of agency, and agency did not change over time as the novelty of VR waned. These results suggest VR designers should consider varied ways of interacting in VR that are beneficial for learning. They also support the use of immersive videos when the educator's goal is to increase agency over learning or focus, and provide measures and direction for future research to assess the relationship between varied types of agency, features of VR experiences and learning outcomes. Practitioner notes What is already known about this topic Virtual reality (VR) can enhance learning by giving learners a strong sense of presence in the virtual environment and giving them agency over their learning through novel forms of interactivity. Many studies have looked at increasing the learner's presence in VR, but fewer have assessed learners' agency. Prior work found the impact of increased interactivity on learning was mixed: sometimes it increased learning outcomes and motivation, other times it did not. What this paper adds This study develops and validates measures of learners' sense of agency within VR learning environments. This study finds that more interactivity in VR increases how much agency learners feel over their actions but not necessarily their agency over learning and attention. This study also finds variation in students' experiences of agency based on their culturally defined sense of self‐construal. Implications for practice and/or policy Practitioners interested in immersive learning technologies should consider the design of the media used rather than focusing only on the device's capability. Immersive videos may be effective tools for enhancing student agency, depending on the aims of the learning experience. Designers and educators should consider learners' identities such as self‐construal, and understand student experiences may vary.