Human-robot Interaction Tasks Research Articles

Optimal design and fabrication of frame structure for dual-arm service robots: An effective approach for human–robot interaction

Rapid advancement in robotics technology has paved the way for developing mobile service robots capable of human interaction and assistance. In this paper, we propose a comprehensive approach to design, fabricate, and optimize the overall structure of a dual-arm service robot. The conceptual design phase focuses on both critical components, the mobile platform and the manipulation system, essential for seamless navigation and effective task execution. In the proposed system, the distribution of the robot payload in terms of region, maximum stress, and displacement is examined, comprehensively analyzed, and compared with the relevant works. In addition, to enhance the system’s efficiency while minimizing its weight, we introduce a lightweight design approach in which Finite Element Analysis is utilized to optimize the frame structure. Subsequently, we fabricate a physical prototype based on the derived model. Finally, we provide a kinematic model for our dual-arm service robot and demonstrate its efficacy in both control and human–robot interaction (HRI) tasks. Experimental results indicate that the proposed dual arm design can achieve a significant weight reduction of 25% from the original design while still performing actions smoothly for HRI tasks.

Passive Model-Predictive Impedance Control for Safe Physical Human–Robot Interaction

Various cognitive systems have been designed to model the position and stiffness profiles of human behavior and then to drive robots by mimicking the human’s behavior to accomplish physical human-robot interaction tasks through a properly designed impedance controller. However, some studies have shown that variable stiffness parameters of the impedance controller can cause the violation of the passivity constraint of the robot states, and make the robot’s stored energy exceed the external energy injected from the human user, thus leading to the unsafe human-robot interaction. To solve this problem, this paper proposes a novel passive model predictive impedance control method including two control loops. In the bottom-loop of the proposed controller, the robot is driven by a variable impedance controller to achieve the desired compliant interaction behavior. In the top-loop of the proposed controller, the model predictive control (MPC) is used to ensure that the robot states satisfy the passivity constraint by calculating a complementary torque to limit the stored energy of the robot. The passivity of the closed-loop robot system and the feasibility of MPC are guaranteed by theoretical analysis, ensuring the safety of the robotic movement in the human-robot interaction. The effectiveness of the proposed method is demonstrated by the simulation and experiment on the Franka Emika Panda robot.

Attention Analysis in Robotic-Assistive Therapy for Children With Autism.

Children with Autism Spectrum Disorder (ASD) show severe attention deficits, hindering their capacity to acquire new skills. The automatic assessment of their attention response would provide the therapists with an important biomarker to better quantify their behaviour and monitor their progress during therapy. This work aims to develop a quantitative model, to evaluate the attention response of children with ASD, during robotic-assistive therapeutic sessions. Previous attempts to quantify the attention response of autistic subjects during human-robot interaction tasks were limited to restrained child movements. Instead, we developed an accurate quantitative system to assess the attention of ASD children in unconstrained scenarios. Our approach combines gaze extraction (Gaze360 model) with the definition of angular Areas-of-Interest, to characterise periods of attention towards elements of interest in the therapy environment during the session. The methodology was tested with 12 ASD children, achieving a mean test accuracy of 79.5 %. Finally, the proposed attention index was consistent with the therapists' evaluation of patients, allowing a meaningful interpretation of the automatic evaluation. This encourages the future clinical use of the proposed system.

Validation of the Human Arm Stiffness Estimation Method Developed for Overground Physical Interaction Experiments.

To build a physically interactive robot for overground applications, it is crucial to first understand the biomechanics of humans underlying overground physical human-robot interaction (pHRI) tasks. Estimating human arm stiffness during overground interactive tasks is a promising first step toward this goal. For this, an arm stiffness estimation technique was developed in our previous works that consider the unique challenges involving overground pHRI, such as the need to estimate the arm stiffness from a short duration of data with fewer repetitions. In this work, our stiffness estimation method is further validated with a passive spring setup with known stiffness values, as well as with a human experiment setup that resembles the widely used seated reaching tasks. Results show that our method can estimate the passive spring stiffness within 0.5% of error. We also show that the human arm stiffness measured through our method is comparable to those reported in well-known literature. In addition, our method was able to discern experimental conditions such as early vs. late trials or differences in arm movement conditions. Implications of these results are discussed further.Clinical Relevance- This method can aid the design and development of overground interactive robots for human movement assistance and rehabilitation.

Human-robot collaborative task planning using anticipatory brain responses.

Human-robot interaction (HRI) describes scenarios in which both human and robot work as partners, sharing the same environment or complementing each other on a joint task. HRI is characterized by the need for high adaptability and flexibility of robotic systems toward their human interaction partners. One of the major challenges in HRI is task planning with dynamic subtask assignment, which is particularly challenging when subtask choices of the human are not readily accessible by the robot. In the present work, we explore the feasibility of using electroencephalogram (EEG) based neuro-cognitive measures for online robot learning of dynamic subtask assignment. To this end, we demonstrate in an experimental human subject study, featuring a joint HRI task with a UR10 robotic manipulator, the presence of EEG measures indicative of a human partner anticipating a takeover situation from human to robot or vice-versa. The present work further proposes a reinforcement learning based algorithm employing these measures as a neuronal feedback signal from the human to the robot for dynamic learning of subtask-assignment. The efficacy of this algorithm is validated in a simulation-based study. The simulation results reveal that even with relatively low decoding accuracies, successful robot learning of subtask-assignment is feasible, with around 80% choice accuracy among four subtasks within 17 minutes of collaboration. The simulation results further reveal that scalability to more subtasks is feasible and mainly accompanied with longer robot learning times. These findings demonstrate the usability of EEG-based neuro-cognitive measures to mediate the complex and largely unsolved problem of human-robot collaborative task planning.

Error-related potential-based shared autonomy via deep recurrent reinforcement learning

Objective. Error-related potential (ErrP)-based brain–computer interfaces (BCIs) have received a considerable amount of attention in the human–robot interaction community. In contrast to traditional BCI, which requires continuous and explicit commands from an operator, ErrP-based BCI leverages the ErrP, which is evoked when an operator observes unexpected behaviours from the robot counterpart. This paper proposes a novel shared autonomy model for ErrP-based human–robot interaction. Approach. We incorporate ErrP information provided by a BCI as useful observations for an agent and formulate the shared autonomy problem as a partially observable Markov decision process. A recurrent neural network-based actor-critic model is used to address the uncertainty in the ErrP signal. We evaluate the proposed framework in a simulated human-in-the-loop robot navigation task with both simulated users and real users. Main results. The results show that the proposed ErrP-based shared autonomy model enables an autonomous robot to complete navigation tasks more efficiently. In a simulation with 70% ErrP accuracy, agents completed the task 14.1% faster than in the no ErrP condition, while with real users, agents completed the navigation task 14.9% faster. Significance. The evaluation results confirmed that the shared autonomy via deep recurrent reinforcement learning is an effective way to deal with uncertain human feedback in a complex human–robot interaction task.

Towards Safe Physical Human-Robot Interaction by Exploring the Rapid Stiffness Switching Feature of Discrete Variable Stiffness Actuation

Safety holds the prime importance in direct physical human-robot interaction (pHRI) tasks. Robots should have the ability to handle unexpected collisions in unstructured environments. Collision avoidance based on exteroceptive sensors can work in these scenarios. However, it may not be sufficient, especially considering that the relative motion between robots and humans can be fast and hardly predictable. This highlights the importance of fast and reliable detection and reaction techniques for the collisions. Rapid switching to intrinsic complaint mode upon collisions is a promising solution for this requirement. Recently, we have proposed a new design of the discrete variable stiffness actuator (DVSA), which has the capability of instantaneously switching its stiffness between different predefined levels. We believe that this rapid stiffness switching feature can significantly improve safety during collisions. In this letter, we combined a software-based collision detection method with a hardware-based rapid stiffness switching technique. The proposed strategy has been implemented on a DVSA-based manipulator to evaluate its safety performance in the sudden dynamic collision and static near-singular clamping collision scenarios. The results clearly indicate that the proposed strategy can significantly mitigate the impact of unexpected collisions and improve safety during pHRI.

Performance Evaluation of Convolutional Networks on Heterogeneous Architectures for Applications in Autonomous Robotics

Humanoid robots find application in human-robot interaction tasks. However, despite their capabilities, their sequential computing system limits the execution of computationally expensive algorithms such as convolutional neural networks, which have demonstrated good performance in recognition tasks. As an alternative to sequential computing units, Field-Programmable Gate Arrays and Graphics Processing Units have a high degree of parallelism and low power consumption. This study aims to improve the visual perception of a humanoid robot called NAO using these embedded systems running a convolutional neural network. The methodology adopted here is based on image acquisition and transmission using simulation software: Webots and Choreographe. In each embedded system, an object recognition stage is performed using commercial convolutional neural network acceleration frameworks. Xilinx® Ultra96™, Intel® Cyclone® V-SoC and NVIDIA® Jetson™ TX2 cards were used, and Tinier-YOLO, AlexNet, Inception-V1 and Inception V3 transfer-learning networks were executed. Real-time metrics were obtained when Inception V1, Inception V3 transfer-learning and AlexNet were run on the Ultra96 and Jetson TX2 cards, with frame rates between 28 and 30 frames per second. The results demonstrated that the use of these embedded systems and convolutional neural networks can provide humanoid robots such as NAO with greater visual recognition in tasks that require high accuracy and autonomy.

Locally active globally stable dynamical systems: Theory, learning, and experiments

State-dependent dynamical systems (DSs) offer adaptivity, reactivity, and robustness to perturbations in motion planning and physical human–robot interaction tasks. Learning DS-based motion plans from non-linear reference trajectories is an active research area in robotics. Most approaches focus on learning DSs that can (i) accurately mimic the demonstrated motion, while (ii) ensuring convergence to the target, i.e., they are globally asymptotically (or exponentially) stable. When subject to perturbations, a compliant robot guided with a DS will continue following the next integral curves of the DS towards the target. If the task requires the robot to track a specific reference trajectory, this approach will fail. To alleviate this shortcoming, we propose the locally active globally stable DS (LAGS-DS), a novel DS formulation that provides both global convergence and stiffness-like symmetric attraction behaviors around a reference trajectory in regions of the state space where trajectory tracking is important. This allows for a unified approach towards motion and impedance encoding in a single DS-based motion model, i.e., stiffness is embedded in the DS. To learn LAGS-DS from demonstrations we propose a learning strategy based on Bayesian non-parametric Gaussian mixture models, Gaussian processes, and a sequence of constrained optimization problems that ensure estimation of stable DS parameters via Lyapunov theory. We experimentally validated LAGS-DS on writing tasks with a KUKA LWR 4+ arm and on navigation and co-manipulation tasks with iCub humanoid robots.

Smoothly Switched Adaptive Torque Tracking for Functional Electrical Stimulation Cycling

Motorized functional electrical stimulation (FES) cycling can serve as a rehabilitation strategy for individuals affected by neurological injuries. It is unique among human-robot interaction tasks because the cycle's motor must be simultaneously controlled alongside the rider's leg muscles (using neuromuscular electrical stimulation). In this letter, two tracking objectives are proposed for the FES cycle: 1) stimulate the rider's leg muscles to track a desired cadence, and 2) use the cycle's motor to indirectly track a desired torque with an adaptive admittance controller. A combined Lyapunov and passivity based switched systems stability analysis is conducted to prove the cycle's motor is able to globally exponentially track the admittance trajectory and stabilize the overall system. Additionally, this letter showcases a method for the rider to smoothly enable and disable torque tracking. Experiments are presented on four participants to show the efficacy of the controllers.

A Hybrid Systems Approach to Dual-Objective Functional Electrical Stimulation Cycling

Motorized functional electrical stimulation (FES) cycling can serve as a physical rehabilitation strategy for individuals whose lower limbs are affected by neurological injuries. Motorized FES cycling is unique among human-robot interaction tasks because the cycle’s motor and rider’s leg muscles must be simultaneously controlled. In this letter, two tracking objectives are proposed for the combined cycle-rider system. First, the rider’s leg muscles are stimulated to pedal the cycle at a desired cadence, and second, the cycle’s motor is used to regulate the interaction torque between the cycle and rider using an admittance controller. A novel hybrid systems analysis using Lyapunov- and passivity-based techniques is conducted to prove that the cycle’s motor can globally exponentially regulate the admittance error system and prove that the cadence error system is output feedback passive. Experiments conducted on five participants illustrate the efficacy of the controllers with an average admittance error of -0.01±0.70 RPM at an average cadence of 45.51±1.77 RPM for a desired cadence of 50 RPM.

Mechanical Structure Design and Actuation Characteristics Analysis of the Parallel Driven Variable Stiffness Actuator With Unrestricted Rotation Range of the Output Shaft

Among many types of flexible robot actuators, the variable stiffness actuator (VSA) is an actuator that can adjust the compliance of the actuator output shaft by adjusting the equivalent stiffness of its internal elastic element. Compared with the traditional rigid actuators, the VSAs have inherent flexibility and adjustable stiffness because they contain elastic elements and stiffness adjusting mechanisms. They have advantages in human-robot physical interaction security and task adaptability. The independent controllability of position and stiffness and the ability of stiffness adjustment make the VSAs have research value and application prospects in the fields of rehabilitation training equipment, prosthetics, wearable devices and so on. The mechanical scheme design of the VSA is an open research field. At present, there is no perfect mechanical structure design of any type of VSA. Optimizing and improving the mechanical structure design of the VSA is helpful to improve its actuation characteristics and application value. Due to the limitation of the mechanical structure design, most of the existing VSAs have limited rotation angle range of the output shaft, which limits their application. The purpose of this paper is to design the parallel driven VSA whose rotation angle range of output shaft is not limited. Moreover, inspired by the mechanical structure design ideas of some stiffness adjustment mechanisms with good implementation schemes and elastic elements with good force transmission characteristics, this paper makes some improvement, synthesis and innovation in the mechanical structure design of the VSA, so as to improve the structural compactness design, assembly modular design, mechanical transmission reliability design and actuation characteristics. In this paper, six types of parallel driven VSAs with different mechanical structure design schemes are proposed. Their common point is that the rotation angle range of the output shaft is not limited, and the differences are that they have different implementation schemes of variable stiffness mechanisms and elastic elements. The transmission structure diagrams of the designed VSAs show the working principle and stiffness adjustment principle. The actuation characteristics of the designed VSAs are analyzed and their differences are compared. The detailed mechanical structure design of the VSA is shown. In the designed VSAs, the model of the VSA based on the symmetrical Archimedes spiral cam groove mechanism is printed and assembled to show the feasibility of the implementation of the designed mechanical structure scheme. Finally, a manual adjustment experiment verifies the stiffness adjustment ability of the assembled VSA model, and shows that the rotation angle range of the output shaft of the assembled VSA model is unlimited.

Where is my hand? Deep hand segmentation for visual self-recognition in humanoid robots

The ability to distinguish between the self and the background is of paramount importance for robotic tasks. The particular case of hands, as the end effectors of a robotic system that more often enter into contact with other elements of the environment, must be perceived and tracked with precision to execute the intended tasks with dexterity and without colliding with obstacles. They are fundamental for several applications, from Human–Robot Interaction tasks to object manipulation. Modern humanoid robots are characterized by high number of degrees of freedom which makes their forward kinematics models very sensitive to uncertainty. Thus, resorting to vision sensing can be the only solution to endow these robots with a good perception of the self, being able to localize their body parts with precision. In this paper, we propose the use of a Convolution Neural Network (CNN) to segment the robot hand from an image in an egocentric view. It is known that CNNs require a huge amount of data to be trained. To overcome the challenge of labeling real-world images, we propose the use of simulated datasets exploiting domain randomization techniques. We fine-tuned the Mask-RCNN network for the specific task of segmenting the hand of the humanoid robot Vizzy. We focus our attention on developing a methodology that requires low amounts of data to achieve reasonable performance while giving detailed insight on how to properly generate variability in the training dataset. Moreover, we analyze the fine-tuning process within the complex model of Mask-RCNN, understanding which weights should be transferred to the new task of segmenting robot hands. Our final model was trained solely on synthetic images and achieves an average IoU of 82% on synthetic validation data and 56.3% on real test data. These results were achieved with only 1000 training images and 3 h of training time using a single GPU.

Real-time person orientation estimation and tracking using colored point clouds

Robustly estimating the orientations of people is a crucial precondition for a wide range of applications. Especially for autonomous systems operating in populated environments, the orientation of a person can give valuable information to increase their acceptance. Given people’s orientations, mobile systems can apply navigation strategies which take people’s proxemics into account or approach them in a human like manner to perform human robot interaction (HRI) tasks. In this paper, we present an approach for person orientation estimation based on computationally efficient features extracted from colored point clouds, formerly used for a two-class person attribute classification. The classification approach has been extended to the continuous domain while treating the problem of orientation estimation in real time. Furthermore, we present an approach for tracking estimated orientations over time using a Bayesian filter. We will show that tracking can increase the accuracy of orientations by up to 3.69° on a dataset recorded with a mobile robot. Best results on this highly challenging dataset are achieved with a regression approach for orientation estimation in combination with tracking. The mean angular error of just 16.49° proofs the applicability in real-world scenarios.

Lio-A Personal Robot Assistant for Human-Robot Interaction and Care Applications.

Lio is a mobile robot platform with a multi-functional arm explicitly designed for human-robot interaction and personal care assistant tasks. The robot has already been deployed in several health care facilities, where it is functioning autonomously, assisting staff and patients on an everyday basis. Lio is intrinsically safe by having full coverage in soft artificial-leather material as well as collision detection, limited speed and forces. Furthermore, the robot has a compliant motion controller. A combination of visual, audio, laser, ultrasound and mechanical sensors are used for safe navigation and environment understanding. The ROS-enabled setup allows researchers to access raw sensor data as well as have direct control of the robot. The friendly appearance of Lio has resulted in the robot being well accepted by health care staff and patients. Fully autonomous operation is made possible by a flexible decision engine, autonomous navigation and automatic recharging. Combined with time-scheduled task triggers, this allows Lio to operate throughout the day, with a battery life of up to 8 hours and recharging during idle times. A combination of powerful computing units provides enough processing power to deploy artificial intelligence and deep learning-based solutions on-board the robot without the need to send any sensitive data to cloud services, guaranteeing compliance with privacy requirements. During the COVID-19 pandemic, Lio was rapidly adjusted to perform additional functionality like disinfection and remote elevated body temperature detection. It complies with ISO13482 - Safety requirements for personal care robots, meaning it can be directly tested and deployed in care facilities.

Investigating Strategies for Robot Persuasion in Social Human-Robot Interaction.

Persuasion is a fundamental aspect of how people interact with each other. As robots become integrated into our daily lives and take on increasingly social roles, their ability to persuade will be critical to their success during human-robot interaction (HRI). In this article, we present a novel HRI study that investigates how a robot's persuasive behavior influences people's decision making. The study consisted of two small social robots trying to influence a person's answer during a jelly bean guessing game. One robot used either an emotional or logical persuasive strategy during the game, while the other robot displayed a neutral control behavior. The results showed that the Emotion strategy had significantly higher persuasive influence compared to both the Logic and Control conditions. With respect to participant demographics, no significant differences in influence were observed between age or gender groups; however, significant differences were observed when considering participant occupation/field of study (FOS). Namely, participants in business, engineering, and physical sciences fields were more influenced by the robots and aligned their answers closer to the robot's suggestion than did those in the life sciences and humanities professions. The discussions provide insight into the potential use of robot persuasion in social HRI task scenarios; in particular, considering the influence that a robot displaying emotional behaviors has when persuading people.

Incremental Instance-Oriented 3D Semantic Mapping via RGB-D Cameras for Unknown Indoor Scene

Scene parsing plays a crucial role when accomplishing human-robot interaction tasks. As the “eye” of the robot, RGB-D camera is one of the most important components for collecting multiview images to construct instance-oriented 3D environment semantic maps, especially in unknown indoor scenes. Although there are plenty of studies developing accurate object-level mapping systems with different types of cameras, these methods either process the instance segmentation problem in completed mapping or suffer from a critical real-time issue due to heavy computation processing required. In this paper, we propose a novel method to incrementally build instance-oriented 3D semantic maps directly from images acquired by the RGB-D camera. To ensure an efficient reconstruction of 3D objects with semantic and instance IDs, the input RGB images are operated by a real-time deep-learned object detector. To obtain accurate point cloud cluster, we adopt the Gaussian mixture model as an optimizer after processing 2D to 3D projection. Next, we present a data association strategy to update class probabilities across the frames. Finally, a map integration strategy fuses information about their 3D shapes, locations, and instance IDs in a faster way. We evaluate our system on different indoor scenes including offices, bedrooms, and living rooms from the SceneNN dataset, and the results show that our method not only builds the instance-oriented semantic map efficiently but also enhances the accuracy of the individual instance in the scene.

Hand gesture recognition using multimodal data fusion and multiscale parallel convolutional neural network for human–robot interaction

AbstractHand gesture recognition plays an important role in human–robot interaction. The accuracy and reliability of hand gesture recognition are the keys to gesture‐based human–robot interaction tasks. To solve this problem, a method based on multimodal data fusion and multiscale parallel convolutional neural network (CNN) is proposed in this paper to improve the accuracy and reliability of hand gesture recognition. First of all, data fusion is conducted on the sEMG signal, the RGB image, and the depth image of hand gestures. Then, the fused images are generated to two different scale images by downsampling, which are respectively input into two subnetworks of the parallel CNN to obtain two hand gesture recognition results. After that, hand gesture recognition results of the parallel CNN are combined to obtain the final hand gesture recognition result. Finally, experiments are carried out on a self‐made database containing 10 common hand gestures, which verify the effectiveness and superiority of the proposed method for hand gesture recognition. In addition, the proposed method is applied to a seven‐degree‐of‐freedom bionic manipulator to achieve robotic manipulation with hand gestures.

An Improved Attention-Based Spatiotemporal-Stream Model for Action Recognition in Videos

Action recognition is an important yet challenging task in computer vision. Attention mechanism not only tells where to focus but when to focus. It plays a key role in extracting discriminative spatial and temporal features for solving the task. In this paper, we propose an improved spatiotemporal attention model based on the two-stream structure to recognize the different actions in videos. Specifically, we first extract the intra-frame spatial features and inter-frame optical flow features for each video data. Then we implement an effective attention module, which sequentially infers attention maps along three separate dimensions: channel, spatial and temporal. After adaptive feature refinement based on the attention maps, we perform a temporal pooling process to squeeze the temporal dimension. Then, these achieved spatial and temporal features are fed into the spatial LSTM and temporal LSTM, respectively. Finally, we fuse the spatial feature, temporal feature and two-stream fusion feature to classify the actions in videos. Additionally, we also collect and construct a new Ping-Pong action dataset for subsequent human-robot interaction task from YouTube. It contains 2400 labeled videos for 4 categories. We compare with other action recognition algorithms and validate the feasibility and effectiveness of the proposed method on Ping-Pong action dataset and HMDB51 dataset.

A corroborative approach to verification and validation of human–robot teams

We present an approach for the verification and validation (V&V) of robot assistants in the context of human–robot interactions, to demonstrate their trustworthiness through corroborative evidence of their safety and functional correctness. Key challenges include the complex and unpredictable nature of the real world in which assistant and service robots operate, the limitations on available V&V techniques when used individually, and the consequent lack of confidence in the V&V results. Our approach, called corroborative V&V, addresses these challenges by combining several different V&V techniques; in this paper we use formal verification (model checking), simulation-based testing, and user validation in experiments with a real robot. This combination of approaches allows V&V of the human–robot interaction task at different levels of modeling detail and thoroughness of exploration, thus overcoming the individual limitations of each technique. We demonstrate our approach through a handover task, the most critical part of a complex cooperative manufacturing scenario, for which we propose safety and liveness requirements to verify and validate. Should the resulting V&V evidence present discrepancies, an iterative process between the different V&V techniques takes place until corroboration between the V&V techniques is gained from refining and improving the assets (i.e., system and requirement models) to represent the human–robot interaction task in a more truthful manner. Therefore, corroborative V&V affords a systematic approach to “meta-V&V,” in which different V&V techniques can be used to corroborate and check one another, increasing the level of certainty in the results of V&V.