Human-robot Interaction Research Articles

Diverse Humanoid Robot Pose Estimation from Images Using Only Sparse Datasets

We present a novel dataset for humanoid robot pose estimation from images, addressing the critical need for accurate pose estimation to enhance human–robot interaction in extended reality (XR) applications. Despite the importance of this task, large-scale pose datasets for diverse humanoid robots remain scarce. To overcome this limitation, we collected sparse pose datasets for commercially available humanoid robots and augmented them through various synthetic data generation techniques, including AI-assisted image synthesis, foreground removal, and 3D character simulations. Our dataset is the first to provide full-body pose annotations for a wide range of humanoid robots exhibiting diverse motions, including side and back movements, in real-world scenarios. Furthermore, we introduce a new benchmark method for real-time full-body 2D keypoint estimation from a single image. Extensive experiments demonstrate that our extended dataset-based pose estimation approach achieves over 33.9% improvement in accuracy compared to using only sparse datasets. Additionally, our method demonstrates the real-time capability of 42 frames per second (FPS) and maintains full-body pose estimation consistency in side and back motions across 11 differently shaped humanoid robots, utilizing approximately 350 training images per robot.

Older adults and robot literacy

ABSTRACT Robots have acquired roles in the provision of information and services. Older adults may interact with robots in many settings, including their homes and public places and, for example, care settings in or outside their homes. In older adults’ care, robots are used to assist their daily tasks, such as cleaning, eating, dressing, bathing, social interaction, entertainment, and medication taking. Consequently, older adults are required to have robot literacy, which has been suggested as a subset of media literacy. However, up to date, only a few researchers have defined robot literacy. In this conceptual article, we first discuss robots in the everyday lives of older adults. Second, drawing on previous studies on media literacies, digital competences, human-robot interaction and other relevant research fields, we define robot literacy as a multidimensional set of skills comprising the following seven skill dimensions: 1) awareness of robots; 2) interaction with robots; 3) understanding and evaluation of the information provided by robots; 4) understanding the data security and privacy of robots; 5) programming of robots; 6) ethical reflection; and 7) providing and receiving social support. Finally, we discuss the applicability of the concept of robot literacy and the limitations of our definition of the concept.

Supporting Human–Robot Interaction in Manufacturing with Augmented Reality and Effective Human–Computer Interaction: A Review and Framework

The integration of Augmented Reality (AR) into Human–Robot Interaction (HRI) represents a significant advancement in collaborative technologies. This paper provides a comprehensive review of AR applications within HRI with a focus on manufacturing, emphasizing their role in enhancing collaboration, trust, and safety. By aggregating findings from numerous studies, this research highlights key challenges, including the need for improved Situational Awareness, enhanced safety, and more effective communication between humans and robots. A framework developed from the literature is presented, detailing the critical elements of AR necessary for advancing HRI. The framework outlines effective methods for continuously evaluating AR systems for HRI. The framework is supported with the help of two case studies and another ongoing research endeavor presented in this paper. This structured approach focuses on enhancing collaboration and safety, with a strong emphasis on integrating best practices from Human–Computer Interaction (HCI) centered around user experience and design.

LQR-based control strategy for improving human–robot companionship and natural obstacle avoidance

In the dynamic and unstructured environment of human–robot symbiosis, companion robots require natural human–robot interaction and autonomous intelligence through multimodal information fusion to achieve effective collaboration. Nevertheless, the control precision and coordination of the accompanying actions are not satisfactory in practical applications. This is primarily attributed to the difficulties in the motion coordination between the accompanying target and the mobile robot. This paper proposes a companion control strategy based on the Linear Quadratic Regulator (LQR) to enhance the coordination and precision of robot companion tasks. This method enables the robot to adapt to sudden changes in the companion target’s motion. Besides, the robot could smoothly avoid obstacles during the companion process. Firstly, a human–robot companion interaction model based on nonholonomic constraints is developed to determine the relative position and orientation between the robot and the companion target. Then, an LQR-based companion controller incorporating behavioral dynamics is introduced to simultaneously avoid obstacles and track the companion target’s direction and velocity. Finally, various simulations and real-world human–robot companion experiments are conducted to regulate the relative position, orientation, and velocity between the target object and the robot platform. Experimental results demonstrate the superiority of this approach over conventional control algorithms in terms of control distance and directional errors throughout system operation. The proposed LQR-based control strategy ensures coordinated and consistent motion with target persons in social companion scenarios.

Design and Construction of 5-Dof EMG Based Robotic Arm System

Electromyography (EMG) is an alternate method of obtaining muscle signal outputs. As a result, the current development in designing my electric plans has piqued the attention of researchers in this subject. This is because Standard controllers lack essential components, limiting the use of limbs to operate equipment, namely an arm controlled by robotics. EMG signals are subject to noise, including crosstalk, motion artifacts, ambient noise, and intrinsic noise. Electromyography preparation requires careful selection of muscle groups, electrode placement, and environment quality, all of which impact the signal output. The goal of this study is to create an EMG-based robotic arm control system that can be used to help the elderly, individuals with impairments, and those who operate in dangerous places. Initially, a literature review on current human-robot interaction approaches and analysis of the kinematics of a 5 DOF robotic arm has been conducted. Then, utilizing Electromyogram (EMG) data obtained from the muscles of the elbow, away for controlling a 5 DOF robotic arm is provided. Two accelerometers are utilized to record the human arm's gesture and posture, this information is then communicated to the robotic arm as an input. In the experiments, wrist rotation, left, right, up, and down movements were used to establish controlled motion of a 5 DOF robotic arm. The project's goal has been met with the successful development of an EMG-controlled robotic arm. The robotic arm may yet be improved by including Implementing a wireless sensor network with multiple channels.

Bridging the gap: Exploring the nexus of service robot personalization, customer identification, and citizenship behaviors

This study explores the multifaceted impact of service robot personalization on customer citizenship behavior, integrating cognitive and affective identification as serial mediators. Leveraging social identity theory, this study examines the intricate psychological processes underlying human-robot interactions. Additionally, the study introduces customer technology readiness as a moderator, acknowledging individual differences in responses to personalization. Results indicate that personalization positively influences alignment perception and identification, subsequently fostering customer citizenship behavior. Furthermore, technology readiness enhances these relationships, emphasizing the need for tailored strategies based on customers' technological predispositions. This research contributes a nuanced understanding of service robot interactions, guiding businesses in strategically deploying personalized interventions to align with diverse customer expectations and preferences.

Integrated Deep Learning Architectures for Perception, Control, and Decision-making in Robotics: a Framework for Sensing, Cognition, and Transparent Decision-making

The increasing complexity of robotic applications demands innovative approaches for addressing problems that lack analytical solutions, with deep learning (DL) emerging as a key tool for enabling robots to learn and adapt in dynamic environments. This survey reviews existing DL techniques in robotics, categorizing the major challenges and exploring successful solutions that leverage DL for perception, control, and decision-making. We discuss the use of modular versus end-to-end DL architectures, providing guidelines for selecting appropriate model structures and training strategies. The review also examines advancements in neuro-robotics systems (NRS), where the convergence of neuroscience and robotics is driving the development of robots with embodied intelligence, enabling more natural human-robot interactions. Recent progress in neural mechanisms for perception, cognition, learning, and control is highlighted, offering insights into creating future neuro-robots. Furthermore, we address the challenge of uncertainty estimation in neural networks, crucial for reliable robotic decision-making, and evaluate existing frameworks such as Bayesian belief networks and Monte Carlo sampling that improve uncertainty modeling without requiring architectural changes. Lastly, we explore efforts to enhance transparency in robotic systems through integrated reasoning and learning methods, focusing on architectures that combine logical reasoning with deep learning to provide explainable decision-making. This survey aims to offer a structured overview of current research and guide future developments in neuro-robotics.

Hierarchical Co-Consistency Quantization and Information Refining Binary Network for Facial Expression Recognition in Human–Robot Interaction

Hierarchical Co-Consistency Quantization and Information Refining Binary Network for Facial Expression Recognition in Human–Robot Interaction

A lightweight network-based sign language robot with facial mirroring and speech system

Human–robot interaction is an essential capability for humanoid robots to enter the physical world and become companions in people’s lives, learning, and work. While the majority of current research focuses on the voice-based interactions of robots, yet over 60% of communication occurs through nonverbal behaviors, such as facial expressions and hand gestures. Endowing robots with the ability to communicate through nonverbal behavior not only enhances the interactive experience with robots but also provides a potential communication tool for individuals with hearing or speech impairments. Here, we develop a humanoid robot capable of adjusting facial movements by driving servos, and design a novel framework for the robot to integrate sign language recognition and facial landmark detection algorithms. This framework facilitates the robot recognize sign language and translate it into spoken language, while also imitating the facial expressions of the signers. To achieve this, we also propose a lightweight deep learning network called RealTimeSignNet for real-time sign language recognition. Leveraging lightweight 3D convolution modules and time-dependent constraints, this model adapts to various time scales, ensuring efficient processing of sign language recognition tasks. Experimental results demonstrate the outstanding performance of the RealTimeSignNet model on mainstream sign language datasets, achieving an accuracy of 88.1% on the large continuous sign language dataset (continuous SLR), 98.2% on the isolated sign language dataset (SLR 500), and 91.50% on the English sign language dataset (WLAS). The overall assessment demonstrates that our humanoid robot is capable of recognizing sign language and translating it into spoken language, while imitating the facial emotions, providing a comprehensive solution to the communication challenges faced by individuals with hearing and speech impairments.

Fuzzy-Based Control for Multiple Tasks With Human–Robot Interaction

Fuzzy-Based Control for Multiple Tasks With Human–Robot Interaction

Referring Image Segmentation With Fine-Grained Semantic Funneling Infusion.

Recently, referring image segmentation has attracted wide attention given its huge potential in human-robot interaction. Networks to identify the referred region must have a deep understanding of both the image and language semantics. To do so, existing works tend to design various mechanisms to achieve cross-modality fusion, for example, tile and concatenation and vanilla nonlocal manipulation. However, the plain fusion usually is either coarse or constrained by the exorbitant computation overhead, finally causing not enough understanding of the referent. In this work, we propose a fine-grained semantic funneling infusion (FSFI) mechanism to solve the problem. The FSFI introduces a constant spatial constraint on the querying entities from different encoding stages and dynamically infuses the gleaned language semantic into the vision branch. Moreover, it decomposes the features from different modalities into more delicate components, allowing the fusion to happen in multiple low-dimensional spaces. The fusion is more effective than the one only happening in one high-dimensional space, given its ability to sink more representative information along the channel dimension. Another problem haunting the task is that the instilling of high-abstract semantic will blur the details of the referent. Targetedly, we propose a multiscale attention-enhanced decoder (MAED) to alleviate the problem. We design a detail enhancement operator (DeEh) and apply it in a multiscale and progressive way. Features from the higher level are used to generate attention guidance to enlighten the lower-level features to more attend to the detail regions. Extensive results on the challenging benchmarks show that our network performs favorably against the state-of-the-arts (SOTAs).

SONAR: An Adaptive Control Architecture for Social Norm Aware Robots

AbstractRecent advances in robotics and artificial intelligence have made it necessary or desired for humans to get involved in interactions with social robots. A key factor for the human acceptance of these robots is their awareness of environmental and social norms. In this paper, we introduce SONAR (for SOcial Norm Aware Robots), a novel robot-agnostic control architecture aimed at enabling social agents to autonomously recognize, act upon, and learn over time social norms during interactions with humans. SONAR integrates several state-of-the-art theories and technologies, including the belief-desire-intention (BDI) model of reasoning and decision making for rational agents, fuzzy logic theory, and large language models, to support adaptive and norm-aware autonomous decision making. We demonstrate the feasibility and applicability of SONAR via real-life experiments involving human-robot interactions (HRI) using a Nao robot for scenarios of casual conversations between the robot and each participant. The results of our experiments show that our SONAR implementation can effectively and efficiently be used in HRI to provide the robot with environmental and social and norm awareness. Compared to a robot with no explicit social and norm awareness, introducing social and norm awareness via SONAR results in interactions that are perceived as more positive and enjoyable by humans, as well as in higher perceived trust in the social robot. Moreover, we investigate, via computer-based simulations, the extent to which SONAR can be used to learn and adapt to the social norms of different societies. The results of these simulations illustrate that SONAR can successfully learn adequate behaviors in a society from a relatively small amount of data. We publicly release the source code of SONAR, along with data and experiments logs.

Supporting Human–Robot Interaction by Projected Augmented Reality and a Brain Interface

Supporting Human–Robot Interaction by Projected Augmented Reality and a Brain Interface

A Review on Human Robot Interaction in Healthcare

Robots are increasingly being used in healthcare delivery and their effectiveness can be determined through understanding their personalities. Nonetheless, healthcare human robot interaction (H-HRI) has not had a systematic or comprehensive grasp of personality. In the past Healthcare has presented benefits for humanoid robots. However, there is no systematic and comprehensive understanding of personality in health care human robot interaction (H-HRI). This paper examines some common scenarios in medicine and provides an exhaustive way to measure trust. Such a proposed paradigm highlights the importance of unambiguous communication among doctors as well as their expectations regarding robot intervention into a task that is centered on humans. Additionally, it shows how patients receive help from robots. Finally, suggestions are made that aim at improving patient’s confidence because the former contributes towards improved therapy efficiency levels. It Briefly Discusses Five Character Traits Also Personality Traits. These entail human-robot interaction factors.

Human-Robot Interaction and Social Robot: The Emerging Field of Healthcare Robotics and Current and Future Perspectives for Spinal Care.

Recent advances in robotics technology and artificial intelligence (AI) have sparked increased interest in humanoid robots that resemble humans and social robots capable of interacting socially. Alongside this trend, a new field of robot research called human-robot interaction (HRI) is gaining prominence. The aim of this review paper is to introduce the fundamental concepts of HRI and social robots, examine their current applications in the medical field, and discuss the current and future prospects of HRI and social robots in spinal care. HRI is an interdisciplinary field where robotics, AI, social sciences, design, and various disciplines collaborate organically to develop robots that successfully interact with humans as the ultimate goal. While social robots are not yet widely deployed in clinical environments, ongoing HRI research encompasses various areas such as nursing and caregiving support, social and emotional assistance, rehabilitation and cognitive enhancement for the elderly, medical information provision and education, as well as patient monitoring and data collection. Although still in its early stages, research related to spinal care includes studies on robotic support for rehabilitation exercises, assistance in gait training, and questionnaire-based assessments for spinal pain. Future applications of social robots in spinal care will require diverse HRI research efforts and active involvement from spinal specialists.

Bridging vision and touch: advancing robotic interaction prediction with self-supervised multimodal learning.

Predicting the consequences of the agent's actions on its environment is a pivotal challenge in robotic learning, which plays a key role in developing higher cognitive skills for intelligent robots. While current methods have predominantly relied on vision and motion data to generate the predicted videos, more comprehensive sensory perception is required for complex physical interactions such as contact-rich manipulation or highly dynamic tasks. In this work, we investigate the interdependence between vision and tactile sensation in the scenario of dynamic robotic interaction. A multi-modal fusion mechanism is introduced to the action-conditioned video prediction model to forecast future scenes, which enriches the single-modality prototype with a compressed latent representation of multiple sensory inputs. Additionally, to accomplish the interactive setting, we built a robotic interaction system that is equipped with both web cameras and vision-based tactile sensors to collect the dataset of vision-tactile sequences and the corresponding robot action data. Finally, through a series of qualitative and quantitative comparative study of different prediction architecture and tasks, we present insightful analysis of the cross-modality influence between vision, tactile and action, revealing the asymmetrical impact that exists between the sensations when contributing to interpreting the environment information. This opens possibilities for more adaptive and efficient robotic control in complex environments, with implications for dexterous manipulation and human-robot interaction.

Sensorless Human–Robot Interaction: Real‐Time Estimation of Co‐Grasped Object Mass and Human Wrench for Compliant Interaction

Human–robot physical interaction in shared object manipulation can fully leverage the strengths of both humans and collaborative robots, achieving better interaction outcomes. However, in typical physical interactions, the inertia parameters of the object are either assumed to be known or ignored, simplifying the interaction to be directly between the human and the collaborative robot. This simplification can lead to inaccuracies in the robot's understanding of human intent due to deviations in object dynamics. This article presents a method that eliminates the need for a force sensor on the human side during human–robot collaboration. Using an extended Kalman filter, the method estimates object parameters online and applies a disturbance observer to determine human force. This approach reduces human effort and improves tracking performance compared to traditional methods. It also avoids the hassle of installing and removing sensors in complex scenarios. By estimating object mass and human force in real time, the robot can adjust its behavior to ensure safer and smoother interactions. Furthermore, the system can handle various objects without requiring specific programming for each case.

Evaluation of Robot Emotion Expressions for Human–Robot Interaction

Evaluation of Robot Emotion Expressions for Human–Robot Interaction

Enhancing Autonomous Agents Through Narrative Memory: A Comprehensive Framework for Improved Performance and Interaction

The process of organizing information and experiences into a narrative style is known as narrative memory, and it presents a cutting-edge method for improving the performance of autonomous agents. This essay describes the idea of narrative memory, as well as its elements, construction of narrative structures, and uses. The four main elements of narrative memory are timelines, characters, events, and context. Events are important things that happen, such interactions or acts, and characters are people, other agents, and the autonomous agent. The context gives the underlying knowledge required to comprehend these events and individuals, while the chronology sets the order of occurrences, creating a cause-and-effect link. The process of creating narrative frameworks includes gathering data, identifying events, crafting stories, and refreshing memories. Three methods are used to collect data: external sources, user input, and sensors. This data is used to identify significant events, which are subsequently arranged into logical tales. New situations and experiences are constantly added to the narrative memory. Improved predictive analysis, easier learning and adaptability, and improved human-robot interaction are just a few of the uses of narrative memory in autonomous agents. Robots can interact with people more successfully if they can comprehend human narratives and reply with responses that are contextually relevant. Agents may learn and adapt by identifying patterns in their prior experiences, and they can make better decisions by forecasting future occurrences with the help of historical narratives. Managing the complexity of real-world events, addressing ambiguity and uncertainty, and guaranteeing the ethical use of narrative data are challenges in the implementation of narrative memory. In order to improve narrative memory frameworks, future research options include creating sophisticated story generating tools, integrating narrative memory with other memory systems, and working across disciplines. The goal of this roadmap is to enhance autonomous systems' comprehension, communication, and decision-making skills by utilizing the power of tales.

인간과 로봇의 상호작용을 위한 로봇 UX디자인에 대한 고찰

The purpose of this paper is to conduct an in-depth review of robot user experience (UX) design to optimize the interaction between humans and robots. As a research method, the existing literature on robot UX design was reviewed, related theories and concepts were summarized, and various types of human-robot interaction were analyzed. By analyzing various robot UX design cases in various fields such as service, healthcare, and education, and suggesting the direction of future robot UX design with User Experience Design, the importance and possibility of robot UX design in the future society are presented. This study provides useful guidelines for scholars and developers interested in robot UX design and will provide important implications for future research and actual design process.