Human-robot Interaction System Research Articles

Body Movement Mirroring and Synchrony in Human–Robot Interaction

This review article provides an overview of papers that have studied body movement mirroring and synchrony within the field of human-robot interaction. The papers included in this review cover system studies, which focus on evaluating the technical aspects of mirroring and synchrony robotic systems, and user studies, which focus on measuring particular interaction outcomes or attitudes towards robots expressing mirroring and synchrony behaviors. We review the papers in terms of the employed robotic platforms and the focus on parts of the body, the techniques used to sense and react to human motion, the evaluation methods, the intended applications of the human-robot interaction systems and the scenarios utilized in user studies. Finally, challenges and possible future directions are considered and discussed.

On Multi-Fidelity Impedance Tuning for Human-Robot Cooperative Manipulation

Abstract We examine how a human-robot interaction (HRI) system may be designed when input output data from previous experiments are available. Our objective is to learn an optimal impedance in the assistance design for a cooperative manipulation task with a new operator. Due to the variability between individuals, the design parameters that best suit one operator of the robot may not be the best parameters for another one. However, by incorporating historical data using a linear auto-regressive (AR-1) Gaussian process, the search for a new operator's optimal parameters can be accelerated. We lay out a framework for optimizing the human-robot cooperative manipulation that only requires input-output data. We characterize the learning performance using a notion called regret, establish how the AR-1 model improves the bound on the regret, and numerically illustrate this improvement in the context of a human-robot cooperative manipulation task. Further, we show how our approach's input-output nature provides robustness against modeling error through an additional numerical study.

Intelligent wheelchair human–robot interactive system based on human posture recognition

Intelligent wheelchair human–robot interactive system based on human posture recognition

The integration of Natural Language Processing (NLP) in Human-Robot Interaction (HRI) represents a significant advancement towards achieving more natural and effective communication between humans and robots. This research explores the application of state-of-the-art NLP techniques to enhance HRI, focusing on improving robots' abilities to understand and generate human language. Key components of our approach include advanced speech recognition, natural language

The integration of Natural Language Processing (NLP) in Human-Robot Interaction (HRI) represents a significant advancement towards achieving more natural and effective communication between humans and robots. This research explores the application of state-of-the-art NLP techniques to enhance HRI, focusing on improving robots' abilities to understand and generate human language. Key components of our approach include advanced speech recognition, natural language understanding (NLU), dialogue management, and natural language generation (NLG). We designed and implemented an HRI system that leverages models such as BERT for language understanding and GPT-3 for generating contextually appropriate responses. Our methodology involves integrating these NLP models with a robotics platform, ensuring real-time interaction capabilities while maintaining a high level of accuracy and context awareness. The system was evaluated through a series of user studies, measuring performance metrics such as accuracy, latency, and user satisfaction. Results indicate that our NLP-enhanced HRI system significantly improves the quality of interactions, demonstrating superior understanding and responsiveness compared to traditional systems. This paper discusses the implementation challenges, including computational constraints and ambiguity resolution, and provides insights into user feedback and system performance. Future work will focus on enhancing context management, exploring multimodal interaction, and addressing ethical considerations in deploying advanced HRI systems. Our findings underscore the potential of NLP to transform human-robot communication, paving the way for more intuitive and effective robotic assistants in various domains. Keywords: Human-Robot Interaction (HRI), Natural Language Processing (NLP), Conversational AI, Speech Recognition, Natural Language Understanding (NLU), Natural Language Generation (NLG), Multimodal Interaction, Dialogue Systems, Context Awareness, Emotion Recognition, Machine Learning in HRI, Personalized Interaction, User Experience (UX) in HRI, Human-Centered Design, Collaborative Robots (Cobots)

User-agnostic adaptation of human locomotion intent: Leveraging Teacher-Student-Learning and ensemble modeling

Wearable robot control in complicated contexts requires an understanding of human locomotion intent and behaviors. However, signals from human–robot interfaces are typically reliant on the user, leading to poor performance of the model trained on training user when applied to new end-point users. This study aims to address this problem by developing a novel Teacher-Student-Learning (TSL) approach using Heterogeneous Ensemble Hypotheses (HEH) to achieve unsupervised user-agnostic adaptation. The motivation behind using the Teacher-Student-Learning architecture is to leverage the diverse features extracted by HEH while maintaining computational efficiency. HEH reduces the difference between labeled training users and unlabeled end-point users by incorporating multiple diverse feature generators to extract a wide range of features, thereby increasing classification accuracy and reducing variance. However, this approach sacrifices efficiency due to the ensemble nature of HEH. To address this trade-off, the knowledge from HEH is distilled into a single network using TSL, ensuring precision and efficiency. The proposed approach is evaluated on two publicly available human locomotion datasets and a 2D moon dataset. Experimental results demonstrate the effectiveness of the proposed method, outperforming all other methods. Tested with three datasets, the proposed method can classify end-point users’ data with an average accuracy of 98.9%, 96.7%, and 96.9% while yielding a low processing time (1 ms). Compared to a benchmark method, the suggested strategy improves the average accuracy by 1.5% and 7.2% for categorizing the target users’ locomotor modes and stabilizes the learning curves. The proposed method represents a significant contribution to the field of human activity recognition and human intent prediction for human–robot interaction systems, enhancing the efficiency and generalization capacity of these systems.

Adaptive Position Constrained Assist-as-Needed Control for Rehabilitation Robots

In rehabilitation practice, motivating patients with neurological injuries to actively increase muscle activity and ensure their safety are important. Therefore, this study proposed a position-constrained assist-as-needed (AAN) control method for rehabilitation robots. A human-robot interaction system with position constraints was first established based on prescribed performance. Aiming at implementing the AAN strategy, the robot assistance level metric (RALM), a constructed global continuous differentiable function incorporating dead-zone and saturation characteristics, was introduced to quantify the robotic assistance and facilitate seamless operation. To bridge the gap between the position constraints and the AAN strategy, a sliding manifold was constructed for the constrained human-robot dynamic system, where RALM was regarded as a weight factor to achieve a human-dominated mode, a robot-dominated mode, and their smooth transition, regarded as a human-robot shared mode. The stability of the closed-loop system was guaranteed by using the Lyapunov theory, and the proposed controller was verified by several physical experiments on a knee exoskeleton driven by pneumatic muscles (PMs).

A Two-Stage Selective Fusion Framework for Joint Intent Detection and Slot Filling.

Spoken language understanding (SLU) is the core of the speech-centric human-robot interaction system, which mainly involves intent detection and slot filling. The recent SLU research focuses on the joint modeling of the two tasks due to their correlation. Furthermore, the slot information consists of slot position and slot type. Although the slot types are semantically related to the intent, the slot positions of the same intent may vary a lot in different utterances due to the diversity of spoken language. Thus, the conventional one-stage slot filling task may introduce unrelated information for slot position prediction in the slot-intent interaction of the joint modeling. Therefore, we propose a novel two-stage selective fusion framework for joint intent detection and slot filling. Unlike the previous one-stage framework, the proposed framework decomposes the slot filling into two stages, i.e., the slot proposal and slot classification. The slot proposal network consisting of BERT and bidirectional long short-term memory (Bi-LSTM)-conditional random field (CRF) predicts the slot positions. Instead of the tokenwise fusion in the existing methods, the slot-intent feature fusion is only performed in the slot classification. A selective fusion mechanism is designed to facilitate the slot-intent interaction within each slot candidate for more accurate slot-type classification. Experiments on five standard benchmarks (i.e., ATIS, SNIPS, MixATIS, MixSNIPS, and DSTC4) show that the proposed framework achieves the best performance in comparison with several state-of-the-art methods.

Kinematic-driven human-robot interaction system with deep learning for flexible acupuncture needling manipulations

Acupuncture is an important therapeutic method in traditional Chinese medicine and essentially characterized by flexible needling manipulations that are complicated for repetitions. In this work, we designed a bio-inspired acupuncture robot combined with human–machine interaction system that can implement flexible and precise needle manipulations as manual acupuncture. In the proposed framework, the manual needling process was recorded with online imaging techniques. With a deep learning-based acupuncture estimator, kinetic parameters of lift insertion (LI) and twist rotation (TR) manipulations were extracted in three-dimensional space and the precent of correct key points for operator’s hand pose estimation achieved 94.03%. It is founded that the variation of operator’s finger vector angle for LI manipulation was much weaker than TR manipulation. Furthermore, inspired by kinematic characteristics of manual acupuncture, a six-degree-of-freedom robot was used to simulate the coordinated motion of hand joints in needling manipulations through robot kinematics analysis and trajectory planning with polynomial interpolation method. Numerical and experiential results showed that the acupuncture robot can mimic various needling manipulations as manual acupuncture. The flexibility and stability of the robotic acupuncture framework were evaluated for a wide range of needling parameters. It was demonstrated that the robot can follow the periodic dynamics of LI and TR manipulations with various frequencies and amplitudes, and perform accurate acupuncture operations with smooth needling trajectories as joint jitters during needle manipulations were completely eliminated. The presented results provide a new strategy for modern applications of acupuncture treatment with robotic and flexible needling manipulations to improve the therapeutic efficacy.

Mechanically Robust and Linearly Sensitive Soft Piezoresistive Pressure Sensor for a Wearable Human-Robot Interaction System.

Soft piezoresistive pressure sensors play an underpinning role in enabling a plethora of future Internet of Things (IoT) applications such as human-robot interaction (HRI) technologies, wearable devices, and metaverse ecosystems. Despite significant attempts to enhance the performance of these sensors, existing sensors still fall short of achieving high strain tolerance and linearity simultaneously. Herein, we present a low-cost, facile, and scalable approach to fabricating a highly strain-tolerant and linearly sensitive soft piezoresistive pressure sensor. Our design utilizes thin nanocracked gold films (NC-GFs) deposited on poly(dimethylsiloxane) (PDMS) as electrodes of the sensor. The large mismatch stress between gold (Au) and PDMS induces the formation of secondary wrinkles along the pyramidal-structured electrode under pressure; these wrinkles function as protuberances on the electrode and enable exceptional linear sensitivity of 4.2 kPa-1 over a wide pressure range. Additionally, our pressure sensor can maintain its performance even after severe mechanical deformations, including repeated stretching up to 30% strain, due to the outstanding strain tolerance of NC-GF. Our sensor's impressive sensing performance and mechanical robustness make it suitable for diverse IoT applications, as demonstrated by its use in wearable pulse monitoring devices and human-robot interaction systems.

A Continuously Variable Transmission-Based Variable Stiffness Actuator for pHRI: Design Optimization and Performance Verification

Abstract Physical human–robot interfaces (pHRIs) enabled the robots to work alongside the human workers complying with the regulations set for physical human–robot interaction systems. A variety of actuation systems named variable stiffness/impedance actuators (VSAs) are configured to be used in these systems’ design. Recently, we introduced a new continuously variable transmission (CVT) mechanism as an alternative solution in configuring VSAs for pHRI. The optimization of this CVT has significant importance to enhance its application area and to detect the limitations of the system. Thus, in this paper, we present a design optimization approach (an adjustment strategy) for this system based on the design goals, desired force, and minimization of the size of the system. To implement such design goals, the static force analysis of the CVT is performed and validated. Furthermore, the fabrication of the optimized prototype is presented, and the experimental verification is performed considering the requirements of VSAs: independent position and stiffness variation, and shock absorbing. Finally, the system is calibrated to display 6 N continuous output force throughout its transmission variation range.

Application of hedge algebras algorithm for mobile robot controller via hand gestures and wireless protocol

With the superior development of technology, mobile robots have become an essential part of humans’ daily life. Consequently, interacting and dealing with them pushes us to develop and propose different suitable Human-Robot Interaction (HRI) systems that can detect the interacted user’s actions and achieve the desired output in real-time. In this paper, we propose a closed-loop smart mechanism for two main agents: the hand gloves’ controller and the mobile robot. To be more specific, the developed model employs flex sensors to measure the curve of the finger. The sensor signals are then processed by aiding the Hedge Algebras Algorithm to control the movement direction and customize the speed of the mobile robot via wireless communication. Numerical simulation and experiments demonstrate that the mobile robot could operate reliably, respond rapidly to control signals, and vary its speed continually based on the different finger gestures. Besides, the control results are also compared with those obtained from the traditional fuzzy controller to prove the superiority and efficiency of the proposed method.

Differential Game-Based Control for Nonlinear Human-Robot Interaction System With Unknown Desired Trajectory.

Differential game is an effective technique to describe the negotiation between the humans and robots, which is widely used to realize the trajectory tracking tasks in the human-robot interaction (HRI). However, most existing works consider the control-affine HRI systems and assume the desired trajectory is available to both the human and the robot, which limit the scope of applications. To overcome these difficulties, this work focuses on the nonaffine HRI system and supposes that the desired trajectory is not available to the robot. A novel differential game framework encoding the desired trajectory estimator is proposed, where the desired trajectory is estimated via the Gaussian process regression (GPR) technique. To address the challenge arising from the nonlinearity of the HRI system, we equivalently transform the original problem into the one in a differentially flat space, and seek the equilibrium strategies for the transformed problem substitutionally. We further prove that the trajectory tracking error satisfies a probabilistic bound, whose confidence interval tightens as the decrease of noise variance during the interaction. Comparative simulation results show that our method outperforms the learning-based method in terms of robustness, parameters setting, and time consumption. Experiment results further show that the tracking error under the proposed human-robot cooperative algorithm is reduced by 55% compared to the human direct control.

Human Robot-Interaction: a conceptual framework for task performance analysis

Human-robot interaction (HRI) is one of the most promising technologies in academic, social, and industrial contexts. Many industries consider HRI as an opportunity to combine the precision, repeatability, and speed of robotic automation with the human’s flexibility, ability, and inclination to problem-solving. Nevertheless, human-centred factors and their impact on performance are still underestimated in this research area resulting in an important research and application gap. This paper addresses this gap by developing a conceptual framework that identifies and combines several key elements from the human-centred and system-centred design paradigms. The aim is to support the design and implementation of human-centred working environments that consider human roles and well-being while maintaining high production performances. This can improve the digital transition to Industry 5.0 enabling the combination of two visions and avoiding the risk of implementing HRI without attention to human factors. This study will contribute to assist practitioners and researchers in identifying the relevant elements to evaluate performance in HRI systems.

Brain–Computer Interface Integrated With Augmented Reality for Human–Robot Interaction

Brain-computer interface (BCI) has been gradually used in human-robot interaction systems. Steady-state visual evoked potential (SSVEP) as a paradigm of electroencephalography (EEG) has attracted more attention in the BCI system research due to its stability and efficiency. However, an independent monitor is needed in the traditional SSVEP-BCI system to display stimulus targets, and the stimulus targets map fixedly to some pre-set commands. These limit the development of the SSVEP-BCI application system in complex and changeable scenarios. In this study, the SSVEP-BCI system integrated with augmented reality (AR) is proposed. Furthermore, a stimulation interface is made by merging the visual information of the objects with stimulus targets, which can update the mapping relationship between stimulus targets and objects automatically to adapt to the change of the objects in the workspace. During the online experiment of the AR-based SSVEP-BCI cue-guided task with the robotic arm, the success rate of grasping is 87.50 ± 3.10% with the SSVEP-EEG data recognition time of 0.5 s based on FB-tCNN. The proposed AR-based SSVEP-BCI system enables the users to select intention targets more ecologically and can grasp more kinds of different objects with a limited number of stimulus targets, resulting in the potential to be used in complex and changeable scenarios.

Towards a more anthropomorphic interaction with robots in museum settings: An experimental study

In recent years, social robots have been increasingly used in the cultural sector to make artworks and exhibits more appealing to a larger audience. On the other hand, visitors, more accustomed to using novel technologies, are increasingly attracted to the possibility of exploiting new ways of interaction. However, more research needs to be done on how social robots successfully interact verbally and non-verbally with visitors. There are few studies on the impact of these technologies and, in particular, a greater level of involvement in using social robots instead of the more usual voice apps to access interactively and acquire information about a work of art or a museum exhibition. This paper proposes a human–robot interaction system based on a three-level architecture able to process and merge heterogeneous sensory information and obtain a set of anthropomorphic mechanisms and metaphors to make the interaction with visitors in museum settings as natural as possible. Moreover, an experimental study has been proposed to assess the impact of this proposed solution, embedded in the social robot Pepper, on museum visitors in the context of an exhibit on Leonardo da Vinci. In particular, the study first aimed to find a potential difference in the interactions between visitors grouped by age and interfacing with the robot equipped with the proposed system or a voice app. Secondly, the study focused on evaluating the effectiveness and usefulness of the robot equipped with the proposed interaction system. A comparative User Experience was conducted between the robot and voice app and a User Experience only for the robot using a specific questionnaire for robotic applications. The results showed a good ability of the proposed system integrated into the robot to make interaction with visitors more pleasant, attractive, and friendly. Engagement and dialogue with them are more natural, stable, and lasting. The cultural information is conveyed more easily and clearly.

Design and Modeling of a Smart Torque-Adjustable Rotary Electroadhesive Clutch for Application in Human–Robot Interaction

The increasing need for sharing workspace and interactive physical tasks between robots and humans has raised concerns regarding the safety of such operations. In this regard, controllable clutches have shown great potential for addressing important safety concerns at the hardware level by separating the high-impedance actuator from the end-effector by providing the power transfer from electromagnetic source to the human. However, the existing clutches suffer from high power consumption and large weight, which make them undesirable from the design point of view. In this article, for the first time, the design and development of a novel, lightweight, and low-power torque-adjustable rotary clutch using electroadhesive materials are presented. The performance of three different pairs of clutch plates is investigated in the context of the smoothness and quality of output torque. The performance degradation issue due to the polarization of the insulator is addressed through the utilization of an alternating current waveform activation signal. Moreover, the effect of the activation frequency on the output torque and power consumption of the clutch is investigated. Finally, a time-dependent model for the output torque of the clutch is presented, and the performance of the clutch was evaluated through experiments, including physical human–robot interaction. The proposed clutch offers a torque-to-power consumption ratio that is six times better than commercial magnetic particle clutches. The proposed clutch presents great potential for developing safe, lightweight, and low-power physical human–robot interaction systems, such as exoskeletons and robotic walkers.

Human–robot collaborative interaction with human perception and action recognition

This paper presents a human–robot interaction system (HRIS) that utilizes human perception and action recognition to enable the robot to understand human intentions and flexibly interact with humans. A monocular multi-person three-dimensional (3D) pose estimation method is first proposed to perceive multi-person two-dimensional (2D) and 3D poses in interaction scenarios. Furthermore, a 3D skeleton poses tracking approach is adopted to locate the identity of each person in consecutive frames and enhance interactive stability. Then, an action recognition model is developed, which exploits tracked pose features to recognize the intentions of humans. An action-controlled interaction system is built with a modular approach to ensure flexibility in meeting multiple task requirements and facilitating flexible interaction. In the system, a distance-based safety solution is designed to avoid collisions between humans and robots. Finally, experimental results are presented to demonstrate the feasibility and effectiveness of the proposed methods and system.

Recent Advances in Sensor-Actuator Hybrid Soft Systems: Core Advantages, Intelligent Applications, and Future Perspectives.

The growing demand for soft intelligent systems, which have the potential to be used in a variety of fields such as wearable technology and human-robot interaction systems, has spurred the development of advanced soft transducers. Among soft systems, sensor-actuator hybrid systems are considered the most promising due to their effective and efficient performance, resulting from the synergistic and complementary interaction between their sensor and actuator components. Recent research on integrated sensor and actuator systems has resulted in a range of conceptual and practical soft systems. This review article provides a comprehensive analysis of recent advances in sensor and actuator integrated systems, which are grouped into three categories based on their primary functions: i) actuator-assisted sensors for intelligent detection, ii) sensor-assisted actuators for intelligent movement, and iii) sensor-actuator interactive devices for a hybrid of intelligent detection and movement. In addition, several bottlenecks in current studies are discussed, and prospective outlooks, including potential applications, are presented. This categorization and analysis will pave the way for the advancement and commercialization of sensor and actuator-integrated systems.

Development of a Human-Robot Interaction System for Industrial Applications

The use of robots is increasing day by day. In this study, it was aimed to develop manufacturing-assistant robot software for small production plants involving non-mass production. The main purpose of this study is to eliminate the difficulty of recruiting an expert robot operator thanks to the developed software and to facilitate the use of robots for non-experts. The developed software consists of three parts: the convolutional neural network (CNN), process selection-trajectory generation, and trajectory regulation modules. Before the operations in these modules are executed, operators record the desired process and the trajectory of the process in the video by hand gestures and index finger. Then recorded video is separated into images. The separated images are classified by the CNN module and the positions of landmarks (joint and fingernail of index finger) were calculated by the same module and using the images. Eight different pre-trained CNN structures were tested in the CNN module, and the best result Xception structure (test loss = 0.0051) was used. The desired process was determined and the trajectory of the process was created with the CNN output data. The connection of the generated trajectory with the object was detected by the trajectory regulation module, and unnecessary trajectory parts were cleaned. Regulated trajectory and desired tasks such as welding or sealing were simulated via an industrial robot in a simulation environment. As a result, an industrial robot could be programmed by non-expert operators for companies whose production line is not standard by using the developed software.

Fielded Human-Robot Interaction for a Heterogeneous Team in the DARPA Subterranean Challenge

Human supervision of multiple fielded robots is a challenging task which requires a thoughtful design and implementation of both the underlying infrastructure and the human interface. It also requires a skilled human able to manage the workload and understand when to trust the autonomy, or manually intervene. We present an end-to-end system for human-robot interaction with a heterogeneous team of robots in complex, communication-limited environments. The system includes the communication infrastructure, autonomy interaction, and human interface elements. Results of the DARPA Subterranean Challenge Final Systems Competition are presented as a case study of the design and analyze the shortcomings of the system.