Framework For Human-robot Interaction Research Articles

Three‐dimensional mapping and immersive human–robot interfacing utilize Kinect‐style depth cameras and virtual reality for agricultural mobile robots

AbstractNavigation and autonomous operation in farmland or greenhouse environments present significant challenges for agricultural mobile robots on account of unstructured scene conditions and complex task requirements. The immersive interfaces based on human–robot interaction have been extensively developed, which bridge the high‐level decision of human beings and the precise sensing and high motion ability of robots. In this study, we propose a human–robot interaction framework based on three‐dimensional mapping and virtual reality (VR) visualization. On the construction client, dense three‐dimensional point‐cloud maps are created based on the simultaneous localization and mapping real‐time appearance‐based mapping algorithm with a Kinect‐style depth camera. Subsequently, point‐cloud filtering and mesh creation methods are introduced for postprocessing the map models. A data communication network is employed to transmit the optimized models to the VR‐based exploration client. Utilizing VR visualization, the operator can gain an intuitive and comprehensive understanding of the environments to be explored. The three‐dimensional agricultural scenes are mapped into VR models through the system, which can combine the physical worlds with the VR spaces in a better manner. Compared with the video streaming‐based method for three‐dimensional mapping, the map models with colors and textures are entirely displayed in the VR headset, thus providing robust and global human–robot interaction interfaces and enhancing the immersive exploration for human beings to a considerable extent. Experimental results indicate that the three‐dimensional map models exhibit relatively sufficient construction integrity and yield favorable optimization effects. Additionally, according to a user questionnaire survey, immersive VR‐based interfaces can be more effectively utilized for the three‐dimensional mapping of mobile robots. Research evidence and results demonstrate that our proposed system framework offers positive assistance and support for enhancing human–robot interaction and executing specific robotic tasks in unstructured field agricultural environments. Code implementation and related data sets are shared at the link https://github.com/WangDongBUAA/Mapping_Interfacing.

A Physical Human–Robot Interaction Framework for Trajectory Adaptation Based on Human Motion Prediction and Adaptive Impedance Control

A Physical Human–Robot Interaction Framework for Trajectory Adaptation Based on Human Motion Prediction and Adaptive Impedance Control

Affordance-Based Human–Robot Interaction With Reinforcement Learning

Planning precise manipulation in robotics to perform grasp and release-related operations, while interacting with humans is a challenging problem. Reinforcement learning (RL) has the potential to make robots attain this capability. In this paper, we propose an affordance-based human-robot interaction (HRI) framework, aiming to reduce the action space size that would considerably impede the exploration efficiency of the agent. The framework is based on a new algorithm called Contextual Q-learning (CQL). We first show that the proposed algorithm trains in a reduced amount of time (2.7 seconds) and reaches an 84% of success rate. This suits the robot’s learning efficiency to observe the current scenario configuration and learn to solve it. Then, we empirically validate the framework for implementation in HRI real-world scenarios. During the HRI, the robot uses semantic information from the state and the optimal policy of the last training step to search for relevant changes in the environment that may trigger the generation of a new policy.

Learning Post-Stroke Gait Training Strategies by Modeling Patient-Therapist Interaction.

For safe and effective robot-aided gait training, it is essential to incorporate the knowledge and expertise of physical therapists. Toward this goal, we directly learn from physical therapists' demonstrations of manual gait assistance in stroke rehabilitation. Lower-limb kinematics of patients and assistive force applied by therapists to the patient's leg are measured using a wearable sensing system which includes a custom-made force sensing array. The collected data is then used to characterize a therapist's strategies in response to unique gait behaviors found within a patient's gait. Preliminary analysis shows that knee extension and weight-shifting are the most important features that shape a therapist's assistance strategies. These key features are then integrated into a virtual impedance model to predict the therapist's assistive torque. This model benefits from a goal-directed attractor and representative features that allow intuitive characterization and estimation of a therapist's assistance strategies. The resulting model is able to accurately capture high-level therapist behaviors over the course of a full training session (r2=0.92, RMSE=0.23Nm) while still explaining some of the more nuanced behaviors contained in individual strides (r2=0.53, RMSE=0.61Nm). This work provides a new approach to control wearable robotics in the sense of directly encoding the decision-making process of physical therapists into a safe human-robot interaction framework for gait rehabilitation.

Unmanned Ground Vehicle Platooning Under Cyber Attacks: A Human-Robot Interaction Framework

In this paper, we propose a human-robot interaction (HRI) framework to reduce the vulnerability of an unmanned ground vehicle (UGV) platoon under cyber attacks. An observer-based autonomous resilient control strategy is first designed to mitigate the effects of vehicle-to-vehicle (V2V) cyber attacks. Rigorous proof of the safety conditions of the platoon under V2V cyber attacks is derived. Next, to facilitate human supervision in emergencies and uncertainties, a decision-making aid system is developed, which consists of an anomaly reporting system (ARS), a trust-based information management system (TIMS), and a graphical user interface (GUI). ARS is designed to detect the abnormality of the neighbor UGVs based on the model-based residual generation and analysis. TIMS is developed to rule out information with low trustworthiness, which is fulfilled by a Bayesian-based information pre-processor followed by an enhanced median filter with trustworthiness regulation. It is rigorously proved that TIMS can recover the tempered information under the mild cyber attack scenario autonomously. To deal with unexpected severe vehicle-to-cloud (V2C) cyber attacks, human observations can also be leveraged to bolster the resilience of the TIMS. Representative simulation and humans-in-the-loop experiment (fulfilled by the designed GUI) demonstrate that the proposed framework can effectively guide human operators when working with UGV platoons under cyber attacks. On average, compared to a multi-screen surveillance benchmark approach, the proposed framework can achieve 78.4% decrements of platoon vulnerability and 39.0% decrements in human workload.

An Integrated Real-Time Hand Gesture Recognition Framework for Human–Robot Interaction in Agriculture

Incorporating hand gesture recognition in human–robot interaction has the potential to provide a natural way of communication, thus contributing to a more fluid collaboration toward optimizing the efficiency of the application at hand and overcoming possible challenges. A very promising field of interest is agriculture, owing to its complex and dynamic environments. The aim of this study was twofold: (a) to develop a real-time skeleton-based recognition system for five hand gestures using a depth camera and machine learning, and (b) to enable a real-time human–robot interaction framework and test it in different scenarios. For this purpose, six machine learning classifiers were tested, while the Robot Operating System (ROS) software was utilized for “translating” the gestures into five commands to be executed by the robot. Furthermore, the developed system was successfully tested in outdoor experimental sessions that included either one or two persons. In the last case, the robot, based on the recognized gesture, could distinguish which of the two workers required help, follow the “locked” person, stop, return to a target location, or “unlock” them. For the sake of safety, the robot navigated with a preset socially accepted speed while keeping a safe distance in all interactions.

SAR-Connect: A Socially Assistive Robotic System to Support Activity and Social Engagement of Older Adults.

Multi-domain activities that incorporate physical, cognitive, and social stimuli can enhance older adults' overall health and quality of life. Several robotic platforms have been developed to provide these therapies in a quantifiable manner to complement healthcare personnel in resource-strapped long-term care settings. However, these platforms are primarily limited to one-to-one human robot interaction (HRI) and thus do not enhance social interaction. In this paper, we present a novel HRI framework and a realized platform called SAR-Connect to foster robot-mediated social interaction among older adults through carefully designed tasks that also incorporate physical and cognitive stimuli. SAR-Connect seamlessly integrates a humanoid robot with a virtual reality-based activity platform and a multimodal data acquisition module including game interaction, audio, visual and electroencephalography responses of the participants. Results from a laboratory-based user study with older adults indicates the potential of SAR-Connect that showed this system could 1) involve one or multiple older adults to perform multi-domain activities and provide dynamic guidance, 2) engage them in the robot-mediated task and foster human-human interaction, and 3) quantify their social and activity engagement from multiple sensory modalities.

Human-Robot Interaction via a Joint-Initiative Supervised Autonomy (JISA) Framework

In this paper, we propose and validate a Joint-Initiative Supervised Autonomy (JISA) framework for Human-Robot Interaction (HRI). Through JISA, a robot maintains a measure of its self-confidence (SC) while performing a task, and only prompts the human supervisor for help when its SC drops. At the same time, a human supervisor can intervene in the tasks being performed, based on his/her Situation Awareness (SA). This paper outlines JISA, presents a guideline for defining the tasks to be supervised, and defines the robot SC and human SA attributes. The paper presents a generic block-diagram to apply JISA in various autonomous systems, and provides detailed proof-of-concept (POC) examples for applying JISA. The selected POCs aim to prove the applicability and utility of JISA through two different applications: grid-based collaborative simultaneous localization and mapping (SLAM) and automated jigsaw puzzle reconstruction. In both POCs, the robot SC and human SA attributes are defined in accordance with JISA’s guidelines. In addition, augmented reality (AR) and two-dimensional graphical user interfaces (GUI) are custom-designed to enhance the human SA and allow intuitive interaction between the human and the agent. The superiority of the JISA framework over existing full-autonomy approaches is demonstrated in experiments. In SLAM, JISA eliminates the need for post processing of SLAM maps and reduces the required mapping time by \({\sim }50\%\). In automated puzzle reconstruction, JISA outperforms two solutions: full autonomy and on-demand human intervention prompted by the agent, reaching a reconstruction accuracy of 97.2%.

Adaptive Fuzzy Robust Tracking Control Using Human Electromyogram Signals for Elastic Joint Robots

Sliding mode control is often used for systems with parametric uncertainties due to its desirable robustness and stability, but this approach carries undesirable chattering. Similarly, joint elasticity is a common phenomenon induced by transmission systems in robots, but it presents additional complexity in robot dynamics that could lead to robot vibrations or even instability. Coupling these two phenomena presents further compounded challenges, particularly when faced with the human interface's added uncertainties. Here, a stable voltage-based adaptive fuzzy strategy to sliding mode control is proposed for an elastic joint robot arm that uses a human's upper limb electromyogram (EMG) signals to direct its movement. The concurrent use of EMG with the elastic joint arm provides a suitable framework for human-robot interaction. EMG signals represent human's ‘intention’ on motion, i.e., they move between 50–100 ms before the mechanical motion begins. Hence this strategy potentially builds better synchronization between the robot and human intention. Furthermore, the adaptive fuzzy strategy eliminates the system chattering while also providing robustness against parametric uncertainties and time delay. Lyapunov analysis also shows bounded-input bounded-output stability of the closed-loop system. Finally, the proposed approach is experimentally implemented in the Sport Science Research Institute. Comparisons with a competing strategy, as well as a torque mode controller, shows that the proposed approach leads to a computationally faster and more accurate controller.

Action recognition for educational proposals applying concepts of Social Assistive Robotics

Action recognition has been gaining interest in research due to its great number of applications. So, the main contribution of this manuscript is a human–robot interaction framework, relying on dimension reduction of the system’s inputs in order to require a smaller dataset for training of an Artificial Neural Network. Our motivation is the development of a Social Assistive Robotics application. In summary, we choose nine standard actions to guide a robot, and two neutral ones to represent stand-by or resting cases. The dataset is created by people with different body shape, for robustness purposes, using only 5 to 10 samples of each class per person. Offline and online tests validate the method’s accuracy and confusion matrices clarify the results. A TicTacToe game using a ground robot exemplify a real world application, where each action represents a desired spot in the game. The results confirm a high accuracy, above 96.7%, in all the tests. Based on this, we can conclude our preprocessing strategy and classifier are capable of identifying the action patterns, even for a tiny dataset; thus, it is recommended for educational proposals due to its simplicity.

An Attachment Framework for Human-Robot Interaction

An Attachment Framework for Human-Robot Interaction

A design of Human and overhead Robot Interaction (HoRI) framework for cooperative robotic applications in copper industry

This paper deals with a Human overhead Robot Interaction (HoRI) framework which is focused on enabling the interaction of human and robots in shared workspaces involving both fixed and overhead robotic structures. Scope of this study is to design a safe human robot interaction framework in the shopfloor aiming to minimize ergonomics issues and to increase the overall productivity. Vision and safety sensors are utilized to ensure the efficient and safe interaction among the operators and robots. Scope of the paper is the presentation of a conceptual implementation of the proposed framework in copper industry.

Towards a Modular and Distributed End-User Development Framework for Human-Robot Interaction

In an effort towards the democratization of Robotics, this article presents a novel End-User Development framework called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Robot Interfaces From Zero Experience</i> (RIZE). The framework provides a set of useful software tools for the creation of robot-oriented software architectures and programming interfaces, as well as the modeling and execution of robot behaviors, with a specific emphasis on social behaviors. Programming interfaces built on top of RIZE enable professionals with different backgrounds and interests to design, adapt, and scale-up robotics applications. As an example of a programming interface, we present Open RIZE, which exploits an End-User Programming paradigm combining blocks, tables, and forms-filling interfaces. Unlike previous approaches, robot behavioral code generated by Open RIZE is intrinsically modular, re-usable, scalable, neutral to the employed programming language, and platform-agnostic. In the article, we present the main design guidelines and features of Open RIZE. Additionally, we perform an initial usability evaluation of the Open RIZE interface in an online workshop. Preliminary results using the System Usability Scale with 10 novice end-users indicate that Open RIZE is easy-to-use and learn.

A Dynamic Behavior Control Framework for Physical Human-Robot Interaction

The problem of dynamic behavior control for human-robot collaborative system is studied in this paper. A general dynamic behavior control framework is presented, based on which the interaction control strategies for single-point human-robot collaboration and multi-point human-robot interaction are studied. Through typical human-robot collaborative operation scenarios, the causes of role conflicts under different human-robot collaboration control strategies are analyzed, and the collaborative control algorithm integrating the active motion imitation and passive motion following is presented to reduce the conflicts during the human-robot collaboration process. For multi-point human-robot interaction, the interaction problems are classified and the interaction control strategies for single-task and multi-task scenarios are designed, both in the Cartesian space and the joint space. For multi-task interaction, a smooth task transition strategy from the collaboration control to the safety control is presented to improve the security during the task switching process. Experimental studies verify the effectiveness of the presented method.

A Socially Adaptable Framework for Human-Robot Interaction.

In our everyday lives we regularly engage in complex, personalized, and adaptive interactions with our peers. To recreate the same kind of rich, human-like interactions, a social robot should be aware of our needs and affective states and continuously adapt its behavior to them. Our proposed solution is to have the robot learn how to select the behaviors that would maximize the pleasantness of the interaction for its peers. To make the robot autonomous in its decision making, this process could be guided by an internal motivation system. We wish to investigate how an adaptive robotic framework of this kind would function and personalize to different users. We also wish to explore whether the adaptability and personalization would bring any additional richness to the human-robot interaction (HRI), or whether it would instead bring uncertainty and unpredictability that would not be accepted by the robot's human peers. To this end, we designed a socially adaptive framework for the humanoid robot iCub. As a result, the robot perceives and reuses the affective and interactive signals from the person as input for the adaptation based on internal social motivation. We strive to investigate the value of the generated adaptation in our framework in the context of HRI. In particular, we compare how users will experience interaction with an adaptive versus a non-adaptive social robot. To address these questions, we propose a comparative interaction study with iCub whereby users act as the robot's caretaker, and iCub's social adaptation is guided by an internal comfort level that varies with the stimuli that iCub receives from its caretaker. We investigate and compare how iCub's internal dynamics would be perceived by people, both in a condition when iCub does not personalize its behavior to the person, and in a condition where it is instead adaptive. Finally, we establish the potential benefits that an adaptive framework could bring to the context of repeated interactions with a humanoid robot.

Online Recognition of Incomplete Gesture Data to Interface Collaborative Robots

Online recognition of gestures is critical for intuitive human-robot interaction (HRI) and further push collaborative robotics into the market, making robots accessible to more people. The problem is that it is difficult to achieve accurate gesture recognition in real unstructured environments, often using distorted and incomplete multisensory data. This paper introduces an HRI framework to classify large vocabularies of interwoven static gestures (SGs) and dynamic gestures (DGs) captured with wearable sensors. DG features are obtained by applying data dimensionality reduction to raw data from sensors (resampling with cubic interpolation and principal component analysis). Experimental tests were conducted using the UC2017 hand gesture dataset with samples from eight different subjects. The classification models show an accuracy of 95.6% for a library of 24 SGs with a random forest and 99.3% for 10 DGs using artificial neural networks. These results compare equally or favorably with different commonly used classifiers. Long short-term memory deep networks achieved similar performance in online frame-by-frame classification using raw incomplete data, performing better in terms of accuracy than static models with specially crafted features, but worse in training and inference time. The recognized gestures are used to teleoperate a robot in a collaborative process that consists in preparing a breakfast meal.

Estimation of human impedance and motion intention for constrained human–robot interaction

In this paper, a complete framework for safe and efficient physical human-robot interaction (pHRI) is developed for robot by considering both issues of adaptation to the human partner and ensuring the motion constraints during the interaction. We consider the robot's learning of not only human motion intention, but also the human impedance. We employ radial basis function neural networks (RBFNNs) to estimate human motion intention in real time, and least square method is utilized in robot learning of human impedance. When robot has learned the impedance information about human, it can adjust its desired impedance parameters by a simple tuning law for operative compliance. An adaptive impedance control integrated with RBFNNs and full-state constraints is also proposed in our work. We employ RBFNNs to compensate for uncertainties in the dynamics model of robot and barrier Lyapunov functions are chosen to ensure that full-state constraints are not violated in pHRI. Results in simulations and experiments show the better performance of our proposed framework compared with traditional methods.

Developing a Portable Human–Robot Interaction (HRI) Framework for Outdoor Robots Through Selective Compartmentalization

One of the challenges of outdoor robots is developing effective portable Human-Robot-Interaction (HRI) frameworks. Hand-held devices offer a practical solution. By equipping these devices with robot software, they can be made to interact with the outdoor robots. Android devices are ideal as they are open source and can be integrated with robots powered by the Robot Operating System (ROS), also open source. However, due to the limits of rosjava, the mechanism that links ROS with android, and the conflicting modes of operation between ROS and android, current implementations of ROS-android offer limited robot applications that do not support advanced operations such as autonomous navigation and others. This paper implements selective compartmentalization to overcome these limitations, by combining ROS with android through a number of ROS and android bridges that would facilitate the development of advanced robot applications. Through the proposed method, authors were able to develop a portable HRI framework that allowed human operators to supervise an outdoor mobile robot while it performed an autonomous task. From their mobile devices, users were able to initialize the robot, configure its motion, and monitor its progress. Also, users were able to reprogram the robot to perform new tasks (not previously planned) through a creative use of features offered in the developed HRI framework. Also, user cognitive effort was reported to be low as evident by the positive score on the NASA-TLX scale test which was corroborated with robot performance data. This paper presents the detailed development and implementation steps.

Object affordance based multimodal fusion for natural Human-Robot interaction

Spoken language based natural Human-Robot Interaction (HRI) requires robots to have the ability to understand spoken language, and extract the intention-related information from the working scenario. For grounding the intention-related object in the working environment, object affordance recognition could be a feasible way. To this end, we propose a dataset and a deep CNN based architecture to learn the human-centered object affordance. Furthermore, we present an affordance based multimodal fusion framework to realize intended object grasping according to the spoken instructions of human users. The proposed framework contains an intention semantics extraction module which is employed to extract the intention from spoken language, a deep Convolutional Neural Network (CNN) based object affordance recognition module which is applied to recognize human-centered object affordance, and a multimodal fusion module which is adopted to bridge the extracted intentions and the recognized object affordances. We also complete multiple intended object grasping experiments on a PR2 platform to validate the feasibility and practicability of the presented HRI framework.

Gesture-based human-robot interaction for human assistance in manufacturing

The paradigm for robot usage has changed in the last few years, from a scenario in which robots work isolated to a scenario where robots collaborate with human beings, exploiting and combining the best abilities of robots and humans. The development and acceptance of collaborative robots is highly dependent on reliable and intuitive human-robot interaction (HRI) in the factory floor. This paper proposes a gesture-based HRI framework in which a robot assists a human co-worker delivering tools and parts, and holding objects to/for an assembly operation. Wearable sensors, inertial measurement units (IMUs), are used to capture the human upper body gestures. Captured data are segmented in static and dynamic blocks recurring to an unsupervised sliding window approach. Static and dynamic data blocks feed an artificial neural network (ANN) for static, dynamic, and composed gesture classification. For the HRI interface, we propose a parameterization robotic task manager (PRTM), in which according to the system speech and visual feedback, the co-worker selects/validates robot options using gestures. Experiments in an assembly operation demonstrated the efficiency of the proposed solution.