Robot Avatar Research Articles

Speech-Driven Avatar Robot System with Changing Complexion for the Visualization of an Interactive Atmosphere

Smooth interactions between talkers can be realized by transmitting and receiving mutual video images and voices in remote communication. However, in such remote communication, it is difficult to generate a sense of unity and an interactive atmosphere because humans recognize screens as a boundary of the physical space. Therefore, it is essential to develop a communication system that can generate and share an interactive atmosphere and interaction-activated communication even if talkers are in remote places. In this study, we developed a speech-driven avatar robot system incorporating an estimation model that simulates the degree of activated communication based on the talker’s speech. The developed avatar robot system can visualize an interactive atmosphere while changing the complexion based on an estimated value. The effectiveness of the developed system was demonstrated by means of sensory evaluations.

Revitalizing Human-Robot Interaction: Phygital Twin Driven Robot Avatar for China–Sweden Teleoperation

Revitalizing Human-Robot Interaction: Phygital Twin Driven Robot Avatar for China–Sweden Teleoperation

Effect of having and switching multiple avatars on the operator's right to talk and receive social support.

People with communication difficulties encounter several challenges in their daily online interactions, such as a limited right to talk (RoT), insufficient social support (SS), and a low sense of being attended to (SoBA). Computer-mediated technologies are limited in addressing such problems owing to their limited capacity in transferring verbal and nonverbal cues between users. In this study, to address the limited RoT, low SS, and low SoBA challenges, we proposed a robotic video conference system with two teleoperated robot avatars. The proposed system was compared with another robotic video conference system that adopts only one teleoperated robot avatar. In the field experiment, 37 participants took part in two discussion sessions using each system type, where RoT, SS, and SoBA were adopted as the measured indices. The proposed system significantly increased the users' RoT and SS compared with other robotic video conference systems. This study contributes to the literature by demonstrating the effect exerted by the type of robotic video conference adopted on users' feelings about RoT, SS, and SoBA.

NimbRo Wins ANA Avatar XPRIZE Immersive Telepresence Competition: Human-Centric Evaluation and Lessons Learned

AbstractRobotic avatar systems can enable immersive telepresence with locomotion, manipulation, and communication capabilities. We present such an avatar system, based on the key components of immersive 3D visualization and transparent force-feedback telemanipulation. Our avatar robot features an anthropomorphic upper body with dexterous hands. The remote human operator drives the arms and fingers through an exoskeleton-based operator station, which provides force feedback both at the wrist and for each finger. The robot torso is mounted on a holonomic base, providing omnidirectional locomotion on flat floors, controlled using a 3D rudder device. Finally, the robot features a 6D movable head with stereo cameras, which stream images to a VR display worn by the operator. Movement latency is hidden using spherical rendering. The head also carries a telepresence screen displaying an animated image of the operator’s face, enabling direct interaction with remote persons. Our system won the $10 M ANA Avatar XPRIZE competition, which challenged teams to develop intuitive and immersive avatar systems that could be operated by briefly trained judges. We analyze our successful participation in the semifinals and finals and provide insight into our operator training and lessons learned. In addition, we evaluate our system in a user study that demonstrates its intuitive and easy usability.

Design and development of a gamified tutorial for iVR serious games

Serious games, including immersive Virtual Reality (iVR) experiences, can be challenging for players due to their unfamiliar control systems and mechanics. This study focuses on designing a gamified tutorial for iVR serious games that not only teaches iVR interactions but also enhances user enjoyment and engagement. The tutorial consists of progressively challenging mini-games that adapt to the user's performance. Tips and recommendations are provided through a robot avatar if users struggle or make mistakes. An optional narrative is included to enhance user engagement, but it is not mandatory for the iVR experience. Gamification elements, such as point collection and progress updates, are incorporated into the tutorial. It can be played independently or as an introduction to iVR serious games. The goal is to use gamification principles to maintain user engagement and flow while enhancing the learning experience in the virtual world.

Mood induction through imitation of full-body movements with different affective intentions.

Theories of human emotion, including some emotion embodiment theories, suggest that our moods and affective states are reflected in the movements of our bodies. We used the reverse process for mood regulation; modulate body movements to regulate mood. Dancing is a type of full-body movement characterized by affective expressivity and, hence, offers the possibility to express different affective states through the same movement sequences. We tested whether the repeated imitation of a dancer performing two simple full-body dance movement sequences with different affective expressivity (happy or sad) could change mood states. Computer-based systems, using avatars as dance models to imitate, offer a series of advantages such as independence from physical contact and location. Therefore, we compared mood induction effects in two conditions: participants were asked to imitate dance movements from one of the two avatars showing: (a) videos of a human dancer model or (b) videos of a robot dancer model. The mood induction was successful for both happy and sad imitations, regardless of condition (human vs. robot avatar dance model). Moreover, the magnitude of happy mood induction and how much participants liked the task predicted work-related motivation after the mood induction. We conclude that mood regulation through dance movements is possible and beneficial in the work context.

Empathic embarrassment towards non-human agents in virtual environments

Humans feel empathic embarrassment by witnessing others go through embarrassing situations. We examined whether we feel such empathic embarrassment even with robot avatars. Participants observed a human avatar and a robot avatar face a series of embarrassing and non-embarrassing scenarios. We collected data for their empathic embarrassment and the cognitive empathy on a 7-point Likert scale. Both empathic embarrassment and cognitive empathy were significantly higher in the embarrassed condition compared to the non-embarrassed condition with both avatars, and the cognitive empathy was significantly higher with the human avatar. There was a tendency of participants showing a higher level of skin conductance while watching the human avatar go through embarrassing situations compared to the robot avatar. A following experiment showed that the average plausibility of the embarrassed condition was significantly higher with the human avatar compared to the robot avatar. However, plausibility scores for emotion were not significantly different among the conditions. These results suggest that humans can feel empathic embarrassment as well as cognitive empathy for robot avatars while cognitive empathy for robot avatars is comparatively lower, and that part of the empathic difference between human and robot avatars might be due to the difference of their plausibility.

Thermal Imaging in Robotics as a Privacy-Enhancing or Privacy-Invasive Measure? Misconceptions of Privacy when Using Thermal Cameras in Robots

This article examines the use of thermal imaging in human–robot interactions and its implications for privacy. Thermal imaging is used in robotics research to conceal the identity of individuals by masking identifiable information, such as facial details. The assumption is that by removing clear facial images, individuals become unidentifiable, thus protecting their privacy. However, the definition of personal data does not require information to be linked to a visual observation of a person. Removing the facial image of a person does not exclude the information from being personal data. However, thermal imaging can reveal personal information that is hidden from view. The article argues that privacy in robotics research is often limited to informational privacy and neglects other dimensions of privacy. It emphasises the need for a broader understanding of privacy in the context of robotics, considering the physical embodiment and data processing capabilities of robots. The conclusion of the article suggests that thermal imaging should not be automatically assumed to be more privacy-preserving than regular RGB cameras. It emphasises the importance of understanding personal data and the specific purposes of robot functions when choosing between different sensors. The article calls for a comprehensive approach to privacy in human–robot interaction, taking into account the broader impact of invasive technologies.

Interface using eye-gaze and tablet input for an avatar robot control in class participation support system

This study presents the development of a novel interface that employs eye-gaze and tablet input for controlling an avatar robot, enabling students to remotely participate in classroom activities from home or healthcare facilities. The proposed system integrates eye-gaze input interface for camera attitude control, and tablet input interface for the movement control of the avatar robot, resulting in an intuitive user experience. This proposed dual-input mechanism allows users to seamlessly acquire environmental information in a remote classroom, and interact with classmates and teachers in the classroom. Experimental results corroborate the effectiveness of the proposed method in controlling an avatar robot, enhancing remote classroom participation.

Co-designing a social robot for child health care

Social robots provide new opportunities to support general child healthcare programs. However, it is still unclear how social robots could be used in this context and how corresponding behaviours should be designed. To ensure satisfying implementations of such new technological solutions, it is essential to include the end-users in the designing process. We have conducted a co-design study at two primary schools based on three complementary, creative methods: Draw-write-and-tell and/or story-writing-and-telling, Theatre play, and Robot avatar programming. A total of 46 children aged 7–12 years old participated in four robot co-design workshops. The drawings, stories and theatre plays were analysed, resulting in evaluations of 10 scenarios as well as 21 new scenarios and 7 main user requirements for social robots providing mental support in general child healthcare. Evaluation of the activities highlight their stimulation of out-of-the-box thinking and the development of creative solutions (i.e. drawings/stories/theatre plays resulted in robot designs, scenarios and requirements), while children’s reflections show them being enjoyable for participation. The inputs gathered during these co-design workshops will greatly influence future work on the design and application of social robots in the child healthcare domain.

Cultural differences in joint attention and engagement in mutual gaze with a robot face

Joint attention is a pivotal mechanism underlying human ability to interact with one another. The fundamental nature of joint attention in the context of social cognition has led researchers to develop tasks that address this mechanism and operationalize it in a laboratory setting, in the form of a gaze cueing paradigm. In the present study, we addressed the question of whether engaging in joint attention with a robot face is culture-specific. We adapted a classical gaze-cueing paradigm such that a robot avatar cued participants’ gaze subsequent to either engaging participants in eye contact or not. Our critical question of interest was whether the gaze cueing effect (GCE) is stable across different cultures, especially if cognitive resources to exert top-down control are reduced. To achieve the latter, we introduced a mathematical stress task orthogonally to the gaze cueing protocol. Results showed larger GCE in the Singapore sample, relative to the Italian sample, independent of gaze type (eye contact vs. no eye contact) or amount of experienced stress, which translates to available cognitive resources. Moreover, since after each block, participants rated how engaged they felt with the robot avatar during the task, we observed that Italian participants rated as more engaging the avatar during the eye contact blocks, relative to no eye contact while Singaporean participants did not show any difference in engagement relative to the gaze. We discuss the results in terms of cultural differences in robot-induced joint attention, and engagement in eye contact, as well as the dissociation between implicit and explicit measures related to processing of gaze.

Virtual agents and risk-taking behavior in adolescence: the twofold nature of nudging

Peer pressure can influence risk-taking behavior and it is particularly felt during adolescence. With artificial intelligence (AI) increasingly present in a range of everyday human contexts, including virtual environments, it is important to examine whether AI can have an impact on human’s decision making processes and behavior. By using the balloon analogue risk task (BART) evaluating propensity to take risk, in this study 113 adolescents' risk-taking behavior was measured when playing alone and in the presence of either a robot avatar or human avatar. In the avatar conditions, participants performed the BART while the avatars either (1) verbally incited risk-taking or (2) discouraged risk-taking (experimental tasks). Risk-taking behavior in the BART was assessed in terms of total number of pumps, gain and explosions. Tendency to impulsivity was also evaluated, as well as the effects of age and gender on risky behavior. The main finding showed a significant effect of both avatars on risk-taking tendency, with riskier behavior during incitement than discouragement conditions, the latter being also substantially different from the playing-alone condition. The results of this study open up new questions in a very sensitive and timely topic and offer various insights into the effect of nudging on adolescents’ behavior in virtual contexts.

Adaptive Robotic Information Gathering via non-stationary Gaussian processes

Robotic Information Gathering (RIG) is a foundational research topic that answers how a robot (team) collects informative data to efficiently build an accurate model of an unknown target function under robot embodiment constraints. RIG has many applications, including but not limited to autonomous exploration and mapping, 3D reconstruction or inspection, search and rescue, and environmental monitoring. A RIG system relies on a probabilistic model’s prediction uncertainty to identify critical areas for informative data collection. Gaussian processes (GPs) with stationary kernels have been widely adopted for spatial modeling. However, real-world spatial data is typically non-stationary—different locations do not have the same degree of variability. As a result, the prediction uncertainty does not accurately reveal prediction error, limiting the success of RIG algorithms. We propose a family of non-stationary kernels named Attentive Kernel (AK), which is simple and robust and can extend any existing kernel to a non-stationary one. We evaluate the new kernel in elevation mapping tasks, where AK provides better accuracy and uncertainty quantification over the commonly used stationary kernels and the leading non-stationary kernels. The improved uncertainty quantification guides the downstream informative planner to collect more valuable data around the high-error area, further increasing prediction accuracy. A field experiment demonstrates that the proposed method can guide an Autonomous Surface Vehicle (ASV) to prioritize data collection in locations with significant spatial variations, enabling the model to characterize salient environmental features.

Inflatable Humanoid Cybernetic Avatar for Physical Human–Robot Interaction

In a digital twin, a humanoid robot can be the counterpart of a simulated agent in the real world. In addition, a human, virtual avatar, and avatar robot might constitute digital triplets. We propose an inflatable cybernetic avatar (CA) with a humanoid upper body using an inflatable structure that can represent gestures. This inflatable CA is much lighter, safer, and cheaper than conventional humanoid robots and can be folded when deflated. These properties are ideal for physical human–robot interaction (pHRI) and allow real-time collection of human behavior through interaction. In the experiment, basic movements such as nodding and raising arms were measured using motion capture systems. This paper demonstrates the proposed inflatable CA in a hybrid event. We also conducted an experiment to measure the touch interactions using tactile sensors attached to the fabric of the inflatable part. A psychologically secure inflatable humanoid CA is a promising platform for physical interaction experiments.

Design Metaphors for Understanding User Expectations of Socially Interactive Robot Embodiments

The physical design of a robot suggests expectations of that robot’s functionality for human users and collaborators. When those expectations align with the robot’s true capabilities, users are more likely to adopt the technologies for their intended use. However, the relationship between expectations and socially interactive robot design is not well understood. This article applies the concept of design metaphors to robot design and contributes the Metaphors for Understanding Functional and Social Anticipated Affordances dataset of 165 extant robots and the expectations users place on them. We used Mechanical Turk to crowd-source user expectation over three user studies. The first study ( N = 382) associated crowd-sourced design metaphors to different robot embodiments. The second study ( N = 803) assessed initial social expectations of robot embodiments. The final study ( N = 805) addressed the degree of abstraction of the design metaphors and the functional expectations projected on robot embodiments. We performed analyses to gain insights into how design metaphors can be used to understand social and functional expectations of robots and how these data can be visualized to be useful for study designers and robot designers. Together, these results can serve to guide robot designers toward aligning user expectations with true robot capabilities, facilitating positive human–robot interaction.

Avatar-mediated service encounters: impacts and research agenda

ABSTRACT Technology has been expanding the service encounter concept. Avatars, including virtual and robotic avatars, have been gaining popularity as an emerging technology to generate more human-like and even enhanced remote interactions in technology-mediated service encounters. However, service researchers have paid lesser attention to human-controlled avatar technologies compared to service robots as autonomous avatars. In response to the emerging business and research interests, the technology-mediated service encounter model needs to be updated by integrating avatar technologies. To address this gap, this study develops a conceptual framework of avatar-mediated service encounters. This concept amalgamates features of traditional technology-mediated service encounters and service robots from the aspects of service flexibility and interaction modality. The applications of avatar technologies are categorized based on two axes – user type and avatar embodiment type – and the impacts and research agenda are outlined for each category. The proposed framework contributes to improving remote service experiences and realizing resilient service workplaces.

Spatial Computing in Modular Spiking Neural Networks with a Robotic Embodiment

One of the challenges in modern neuroscience is creating a brain-on-a-chip. Such a semiartificial device based on neural networks grown in vitro should interact with the environment when embodied in a robot. A crucial point in this endeavor is developing a neural network architecture capable of associative learning. This work proposes a mathematical model of a midscale modular spiking neural network (SNN) to study learning mechanisms within the brain-on-a-chip context. We show that besides spike-timing-dependent plasticity (STDP), synaptic and neuronal competitions are critical factors for successful learning. Moreover, the shortest pathway rule can implement the synaptic competition responsible for processing conditional stimuli coming from the environment. This solution is ready for testing in neuronal cultures. The neuronal competition can be implemented by lateral inhibition actuating over the SNN modulus responsible for unconditional responses. Empirical testing of this approach is challenging and requires the development of a technique for growing cultures with a given ratio of excitatory and inhibitory neurons. We test the modular SNN embedded in a mobile robot and show that it can establish the association between touch (unconditional) and ultrasonic (conditional) sensors. Then, the robot can avoid obstacles without hitting them, relying on ultrasonic sensors only.

Merging BSP based swarm dynamics and distributed ledger technologies for smart marine infrastructures

Distributed ledger technologies, together with AI, smart systems and robotics could provide a scalable and robust platform for the smart underwater and surface marine infrastructures of the close future. They will be harbours or ports, marine farms or remote tourism facilities - only accessible through robotic avatars - for protected areas or for the elders. Autonomous Marine Vehicles, IoT networks, and humans will coexist in highly heterogeneous multivendor multiplatform environments where market transactions and complex administrative procedures will be ubiquitous. Some blockchains such as the Ethereum network are able to provide distributed scalable computing and trustable functionalities, capable of managing both technical interactions and market transactions among very diversified autonomous agents. For these reasons they seem to provide a valuable backbone for the smart marine infrastructure of the coming decades. In this paper we outline our research and innovation strategy and present our results showing the potential benefits of a subsidiary architecture integrating distributed ledger technologies with swarms of autonomous surface robots implemented by a Belief Space Planning approach.

AI-driven Family Interaction Over Melded Space and Time

Computer-mediated interaction services connect people over a distance. However, we address that those people are often “locked in a frame”—which includes an interaction mode, a point in time, or a context of either person. We observe that such lock-ins make it difficult to shape the interaction to be mutually symmetric. In this article, we propose a semantic-equivalent melding of space and time to provide a new form of empathetic interaction. We present HomeMeld and MomentMeld, which aim to meld space and time by applying AI, respectively. HomeMeld provides a sense of living together to a family living apart with AI-driven autonomous robotic avatars. MomentMeld utilizes an ensemble of visual AI to create interaction topics by matching semantic-equivalent photos. In-the-wild experiments reveal that HomeMeld and MomentMeld open new possibilities for empathetic interaction. Finally, we introduce a new interaction service leveraging technical synergy of HomeMeld and MomentMeld.

Evaluation of conveying spatial information by pointing gestures of a tele-immersion robot avatar

Evaluation of conveying spatial information by pointing gestures of a tele-immersion robot avatar