Framework For Human-robot Interaction Research Articles

Affordance-Based Human–Robot Interaction With Reinforcement Learning

Planning precise manipulation in robotics to perform grasp and release-related operations, while interacting with humans is a challenging problem. Reinforcement learning (RL) has the potential to make robots attain this capability. In this paper, we propose an affordance-based human-robot interaction (HRI) framework, aiming to reduce the action space size that would considerably impede the exploration efficiency of the agent. The framework is based on a new algorithm called Contextual Q-learning (CQL). We first show that the proposed algorithm trains in a reduced amount of time (2.7 seconds) and reaches an 84% of success rate. This suits the robot’s learning efficiency to observe the current scenario configuration and learn to solve it. Then, we empirically validate the framework for implementation in HRI real-world scenarios. During the HRI, the robot uses semantic information from the state and the optimal policy of the last training step to search for relevant changes in the environment that may trigger the generation of a new policy.

Learning Post-Stroke Gait Training Strategies by Modeling Patient-Therapist Interaction.

For safe and effective robot-aided gait training, it is essential to incorporate the knowledge and expertise of physical therapists. Toward this goal, we directly learn from physical therapists' demonstrations of manual gait assistance in stroke rehabilitation. Lower-limb kinematics of patients and assistive force applied by therapists to the patient's leg are measured using a wearable sensing system which includes a custom-made force sensing array. The collected data is then used to characterize a therapist's strategies in response to unique gait behaviors found within a patient's gait. Preliminary analysis shows that knee extension and weight-shifting are the most important features that shape a therapist's assistance strategies. These key features are then integrated into a virtual impedance model to predict the therapist's assistive torque. This model benefits from a goal-directed attractor and representative features that allow intuitive characterization and estimation of a therapist's assistance strategies. The resulting model is able to accurately capture high-level therapist behaviors over the course of a full training session (r2=0.92, RMSE=0.23Nm) while still explaining some of the more nuanced behaviors contained in individual strides (r2=0.53, RMSE=0.61Nm). This work provides a new approach to control wearable robotics in the sense of directly encoding the decision-making process of physical therapists into a safe human-robot interaction framework for gait rehabilitation.

Unmanned Ground Vehicle Platooning Under Cyber Attacks: A Human-Robot Interaction Framework

In this paper, we propose a human-robot interaction (HRI) framework to reduce the vulnerability of an unmanned ground vehicle (UGV) platoon under cyber attacks. An observer-based autonomous resilient control strategy is first designed to mitigate the effects of vehicle-to-vehicle (V2V) cyber attacks. Rigorous proof of the safety conditions of the platoon under V2V cyber attacks is derived. Next, to facilitate human supervision in emergencies and uncertainties, a decision-making aid system is developed, which consists of an anomaly reporting system (ARS), a trust-based information management system (TIMS), and a graphical user interface (GUI). ARS is designed to detect the abnormality of the neighbor UGVs based on the model-based residual generation and analysis. TIMS is developed to rule out information with low trustworthiness, which is fulfilled by a Bayesian-based information pre-processor followed by an enhanced median filter with trustworthiness regulation. It is rigorously proved that TIMS can recover the tempered information under the mild cyber attack scenario autonomously. To deal with unexpected severe vehicle-to-cloud (V2C) cyber attacks, human observations can also be leveraged to bolster the resilience of the TIMS. Representative simulation and humans-in-the-loop experiment (fulfilled by the designed GUI) demonstrate that the proposed framework can effectively guide human operators when working with UGV platoons under cyber attacks. On average, compared to a multi-screen surveillance benchmark approach, the proposed framework can achieve 78.4% decrements of platoon vulnerability and 39.0% decrements in human workload.

An Integrated Real-Time Hand Gesture Recognition Framework for Human–Robot Interaction in Agriculture

Incorporating hand gesture recognition in human–robot interaction has the potential to provide a natural way of communication, thus contributing to a more fluid collaboration toward optimizing the efficiency of the application at hand and overcoming possible challenges. A very promising field of interest is agriculture, owing to its complex and dynamic environments. The aim of this study was twofold: (a) to develop a real-time skeleton-based recognition system for five hand gestures using a depth camera and machine learning, and (b) to enable a real-time human–robot interaction framework and test it in different scenarios. For this purpose, six machine learning classifiers were tested, while the Robot Operating System (ROS) software was utilized for “translating” the gestures into five commands to be executed by the robot. Furthermore, the developed system was successfully tested in outdoor experimental sessions that included either one or two persons. In the last case, the robot, based on the recognized gesture, could distinguish which of the two workers required help, follow the “locked” person, stop, return to a target location, or “unlock” them. For the sake of safety, the robot navigated with a preset socially accepted speed while keeping a safe distance in all interactions.

SAR-Connect: A Socially Assistive Robotic System to Support Activity and Social Engagement of Older Adults.

Multi-domain activities that incorporate physical, cognitive, and social stimuli can enhance older adults' overall health and quality of life. Several robotic platforms have been developed to provide these therapies in a quantifiable manner to complement healthcare personnel in resource-strapped long-term care settings. However, these platforms are primarily limited to one-to-one human robot interaction (HRI) and thus do not enhance social interaction. In this paper, we present a novel HRI framework and a realized platform called SAR-Connect to foster robot-mediated social interaction among older adults through carefully designed tasks that also incorporate physical and cognitive stimuli. SAR-Connect seamlessly integrates a humanoid robot with a virtual reality-based activity platform and a multimodal data acquisition module including game interaction, audio, visual and electroencephalography responses of the participants. Results from a laboratory-based user study with older adults indicates the potential of SAR-Connect that showed this system could 1) involve one or multiple older adults to perform multi-domain activities and provide dynamic guidance, 2) engage them in the robot-mediated task and foster human-human interaction, and 3) quantify their social and activity engagement from multiple sensory modalities.

Human-Robot Interaction via a Joint-Initiative Supervised Autonomy (JISA) Framework

In this paper, we propose and validate a Joint-Initiative Supervised Autonomy (JISA) framework for Human-Robot Interaction (HRI). Through JISA, a robot maintains a measure of its self-confidence (SC) while performing a task, and only prompts the human supervisor for help when its SC drops. At the same time, a human supervisor can intervene in the tasks being performed, based on his/her Situation Awareness (SA). This paper outlines JISA, presents a guideline for defining the tasks to be supervised, and defines the robot SC and human SA attributes. The paper presents a generic block-diagram to apply JISA in various autonomous systems, and provides detailed proof-of-concept (POC) examples for applying JISA. The selected POCs aim to prove the applicability and utility of JISA through two different applications: grid-based collaborative simultaneous localization and mapping (SLAM) and automated jigsaw puzzle reconstruction. In both POCs, the robot SC and human SA attributes are defined in accordance with JISA’s guidelines. In addition, augmented reality (AR) and two-dimensional graphical user interfaces (GUI) are custom-designed to enhance the human SA and allow intuitive interaction between the human and the agent. The superiority of the JISA framework over existing full-autonomy approaches is demonstrated in experiments. In SLAM, JISA eliminates the need for post processing of SLAM maps and reduces the required mapping time by \({\sim }50\%\). In automated puzzle reconstruction, JISA outperforms two solutions: full autonomy and on-demand human intervention prompted by the agent, reaching a reconstruction accuracy of 97.2%.

Adaptive Fuzzy Robust Tracking Control Using Human Electromyogram Signals for Elastic Joint Robots

Sliding mode control is often used for systems with parametric uncertainties due to its desirable robustness and stability, but this approach carries undesirable chattering. Similarly, joint elasticity is a common phenomenon induced by transmission systems in robots, but it presents additional complexity in robot dynamics that could lead to robot vibrations or even instability. Coupling these two phenomena presents further compounded challenges, particularly when faced with the human interface's added uncertainties. Here, a stable voltage-based adaptive fuzzy strategy to sliding mode control is proposed for an elastic joint robot arm that uses a human's upper limb electromyogram (EMG) signals to direct its movement. The concurrent use of EMG with the elastic joint arm provides a suitable framework for human-robot interaction. EMG signals represent human's ‘intention’ on motion, i.e., they move between 50–100 ms before the mechanical motion begins. Hence this strategy potentially builds better synchronization between the robot and human intention. Furthermore, the adaptive fuzzy strategy eliminates the system chattering while also providing robustness against parametric uncertainties and time delay. Lyapunov analysis also shows bounded-input bounded-output stability of the closed-loop system. Finally, the proposed approach is experimentally implemented in the Sport Science Research Institute. Comparisons with a competing strategy, as well as a torque mode controller, shows that the proposed approach leads to a computationally faster and more accurate controller.

Action recognition for educational proposals applying concepts of Social Assistive Robotics

Action recognition has been gaining interest in research due to its great number of applications. So, the main contribution of this manuscript is a human–robot interaction framework, relying on dimension reduction of the system’s inputs in order to require a smaller dataset for training of an Artificial Neural Network. Our motivation is the development of a Social Assistive Robotics application. In summary, we choose nine standard actions to guide a robot, and two neutral ones to represent stand-by or resting cases. The dataset is created by people with different body shape, for robustness purposes, using only 5 to 10 samples of each class per person. Offline and online tests validate the method’s accuracy and confusion matrices clarify the results. A TicTacToe game using a ground robot exemplify a real world application, where each action represents a desired spot in the game. The results confirm a high accuracy, above 96.7%, in all the tests. Based on this, we can conclude our preprocessing strategy and classifier are capable of identifying the action patterns, even for a tiny dataset; thus, it is recommended for educational proposals due to its simplicity.

An Attachment Framework for Human-Robot Interaction

An Attachment Framework for Human-Robot Interaction

A design of Human and overhead Robot Interaction (HoRI) framework for cooperative robotic applications in copper industry

This paper deals with a Human overhead Robot Interaction (HoRI) framework which is focused on enabling the interaction of human and robots in shared workspaces involving both fixed and overhead robotic structures. Scope of this study is to design a safe human robot interaction framework in the shopfloor aiming to minimize ergonomics issues and to increase the overall productivity. Vision and safety sensors are utilized to ensure the efficient and safe interaction among the operators and robots. Scope of the paper is the presentation of a conceptual implementation of the proposed framework in copper industry.

Towards a Modular and Distributed End-User Development Framework for Human-Robot Interaction

In an effort towards the democratization of Robotics, this article presents a novel End-User Development framework called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Robot Interfaces From Zero Experience</i> (RIZE). The framework provides a set of useful software tools for the creation of robot-oriented software architectures and programming interfaces, as well as the modeling and execution of robot behaviors, with a specific emphasis on social behaviors. Programming interfaces built on top of RIZE enable professionals with different backgrounds and interests to design, adapt, and scale-up robotics applications. As an example of a programming interface, we present Open RIZE, which exploits an End-User Programming paradigm combining blocks, tables, and forms-filling interfaces. Unlike previous approaches, robot behavioral code generated by Open RIZE is intrinsically modular, re-usable, scalable, neutral to the employed programming language, and platform-agnostic. In the article, we present the main design guidelines and features of Open RIZE. Additionally, we perform an initial usability evaluation of the Open RIZE interface in an online workshop. Preliminary results using the System Usability Scale with 10 novice end-users indicate that Open RIZE is easy-to-use and learn.

A Dynamic Behavior Control Framework for Physical Human-Robot Interaction

The problem of dynamic behavior control for human-robot collaborative system is studied in this paper. A general dynamic behavior control framework is presented, based on which the interaction control strategies for single-point human-robot collaboration and multi-point human-robot interaction are studied. Through typical human-robot collaborative operation scenarios, the causes of role conflicts under different human-robot collaboration control strategies are analyzed, and the collaborative control algorithm integrating the active motion imitation and passive motion following is presented to reduce the conflicts during the human-robot collaboration process. For multi-point human-robot interaction, the interaction problems are classified and the interaction control strategies for single-task and multi-task scenarios are designed, both in the Cartesian space and the joint space. For multi-task interaction, a smooth task transition strategy from the collaboration control to the safety control is presented to improve the security during the task switching process. Experimental studies verify the effectiveness of the presented method.

A Socially Adaptable Framework for Human-Robot Interaction.

In our everyday lives we regularly engage in complex, personalized, and adaptive interactions with our peers. To recreate the same kind of rich, human-like interactions, a social robot should be aware of our needs and affective states and continuously adapt its behavior to them. Our proposed solution is to have the robot learn how to select the behaviors that would maximize the pleasantness of the interaction for its peers. To make the robot autonomous in its decision making, this process could be guided by an internal motivation system. We wish to investigate how an adaptive robotic framework of this kind would function and personalize to different users. We also wish to explore whether the adaptability and personalization would bring any additional richness to the human-robot interaction (HRI), or whether it would instead bring uncertainty and unpredictability that would not be accepted by the robot's human peers. To this end, we designed a socially adaptive framework for the humanoid robot iCub. As a result, the robot perceives and reuses the affective and interactive signals from the person as input for the adaptation based on internal social motivation. We strive to investigate the value of the generated adaptation in our framework in the context of HRI. In particular, we compare how users will experience interaction with an adaptive versus a non-adaptive social robot. To address these questions, we propose a comparative interaction study with iCub whereby users act as the robot's caretaker, and iCub's social adaptation is guided by an internal comfort level that varies with the stimuli that iCub receives from its caretaker. We investigate and compare how iCub's internal dynamics would be perceived by people, both in a condition when iCub does not personalize its behavior to the person, and in a condition where it is instead adaptive. Finally, we establish the potential benefits that an adaptive framework could bring to the context of repeated interactions with a humanoid robot.

Online Recognition of Incomplete Gesture Data to Interface Collaborative Robots

Online recognition of gestures is critical for intuitive human-robot interaction (HRI) and further push collaborative robotics into the market, making robots accessible to more people. The problem is that it is difficult to achieve accurate gesture recognition in real unstructured environments, often using distorted and incomplete multisensory data. This paper introduces an HRI framework to classify large vocabularies of interwoven static gestures (SGs) and dynamic gestures (DGs) captured with wearable sensors. DG features are obtained by applying data dimensionality reduction to raw data from sensors (resampling with cubic interpolation and principal component analysis). Experimental tests were conducted using the UC2017 hand gesture dataset with samples from eight different subjects. The classification models show an accuracy of 95.6% for a library of 24 SGs with a random forest and 99.3% for 10 DGs using artificial neural networks. These results compare equally or favorably with different commonly used classifiers. Long short-term memory deep networks achieved similar performance in online frame-by-frame classification using raw incomplete data, performing better in terms of accuracy than static models with specially crafted features, but worse in training and inference time. The recognized gestures are used to teleoperate a robot in a collaborative process that consists in preparing a breakfast meal.

Estimation of human impedance and motion intention for constrained human–robot interaction

In this paper, a complete framework for safe and efficient physical human-robot interaction (pHRI) is developed for robot by considering both issues of adaptation to the human partner and ensuring the motion constraints during the interaction. We consider the robot's learning of not only human motion intention, but also the human impedance. We employ radial basis function neural networks (RBFNNs) to estimate human motion intention in real time, and least square method is utilized in robot learning of human impedance. When robot has learned the impedance information about human, it can adjust its desired impedance parameters by a simple tuning law for operative compliance. An adaptive impedance control integrated with RBFNNs and full-state constraints is also proposed in our work. We employ RBFNNs to compensate for uncertainties in the dynamics model of robot and barrier Lyapunov functions are chosen to ensure that full-state constraints are not violated in pHRI. Results in simulations and experiments show the better performance of our proposed framework compared with traditional methods.

Object affordance based multimodal fusion for natural Human-Robot interaction

Spoken language based natural Human-Robot Interaction (HRI) requires robots to have the ability to understand spoken language, and extract the intention-related information from the working scenario. For grounding the intention-related object in the working environment, object affordance recognition could be a feasible way. To this end, we propose a dataset and a deep CNN based architecture to learn the human-centered object affordance. Furthermore, we present an affordance based multimodal fusion framework to realize intended object grasping according to the spoken instructions of human users. The proposed framework contains an intention semantics extraction module which is employed to extract the intention from spoken language, a deep Convolutional Neural Network (CNN) based object affordance recognition module which is applied to recognize human-centered object affordance, and a multimodal fusion module which is adopted to bridge the extracted intentions and the recognized object affordances. We also complete multiple intended object grasping experiments on a PR2 platform to validate the feasibility and practicability of the presented HRI framework.

Gesture-based human-robot interaction for human assistance in manufacturing

The paradigm for robot usage has changed in the last few years, from a scenario in which robots work isolated to a scenario where robots collaborate with human beings, exploiting and combining the best abilities of robots and humans. The development and acceptance of collaborative robots is highly dependent on reliable and intuitive human-robot interaction (HRI) in the factory floor. This paper proposes a gesture-based HRI framework in which a robot assists a human co-worker delivering tools and parts, and holding objects to/for an assembly operation. Wearable sensors, inertial measurement units (IMUs), are used to capture the human upper body gestures. Captured data are segmented in static and dynamic blocks recurring to an unsupervised sliding window approach. Static and dynamic data blocks feed an artificial neural network (ANN) for static, dynamic, and composed gesture classification. For the HRI interface, we propose a parameterization robotic task manager (PRTM), in which according to the system speech and visual feedback, the co-worker selects/validates robot options using gestures. Experiments in an assembly operation demonstrated the efficiency of the proposed solution.

An Efficient ORB based Face Recognition framework for Human-Robot Interaction

In Human-Robot Interaction (HRI), quick and efficient FR techniques are often required in service robots. In a real time scenario, it is absolute that face image patterns observed by robots depends often on variations such as pose, light conditions, location of the robots (view point), etc. In addition to these constraints, the service robots are expected to be quick enough for FR so that they can be deployed in applications such as counting people, security and surveillance, directing humans, etc. In this paper, ORB, a computational expensive and quick feature extraction technique is used, which has been a panacea for the above mentioned constraints. One of the dimensionality reduction techniques called PCA (a tool which reduces high dimensional data to lower dimension while keeping most of the data) with its sublime advantages of reduction of storage and time is often used. But, in the FR system, the linear uncorrelated components of PCA doesn’t consider the non-linear factors such as occlusion and in such cases PCA fails to find the good representative direction. Kernel PCA (KPCA) which uses kernel methods considers even the non-linear factors and is proven to be more suitable than PCA, thus producing better results. By considering all these factors, our paper proposes a novel technique ORB-KPCA for FR along with Threshold Based Filtering (TBF). The proposed technique is proven to be efficient in both time and space by experimenting on three benchmark datasets (ORL, Faces96 and Grimace).

Investigating People’s Rapport Building and Hindering Behaviors When Working with a Collaborative Robot

Modern industrial robots are increasingly moving toward collaborating with people on complex tasks as team members, and away from working in isolated cages that are separated from people. Collaborative robots are programmed to use social communication techniques with people, enabling human team members to use their existing inter-personal skills to work with robots, such as speech, gestures, or gaze. Research is increasingly investigating how robots can use higher-level social structures such as team dynamics or conflict resolution. One particularly important aspect of human–human teamwork is rapport building: these are everyday social interactions between people that help to develop professional relationships by establishing trust, confidence, and collegiality, but which are formally peripheral to a task at hand. In this paper, we report on our investigations of how and if people apply similar rapport-building behaviors to robot collaborators. First, we synthesized existing human–human rapport knowledge into an initial human–robot interaction framework; this framework includes verbal and non-verbal behaviors, both for rapport building and rapport hindering, that people can be expected to exhibit. We developed a novel mock industrial task scenario that emphasizes ecological validity, and creates a range of social interactions necessary for investigating rapport. Finally, we report on a qualitative study that investigates how people use rapport hindering or building behaviors in our industrial scenario, which reflects how people may interact with robots in industrial settings.

Human Behavior and Hand Gesture Classification for Smart Human-robot Interaction

This paper presents an intuitive human-robot interaction (HRI) framework for gesture and human behavior recognition. It relies on a vision-based system as interaction technology to classify gestures and a 3-axis accelerometer for behavior classification (stand, walking, etc.). An intelligent system integrates static gesture recognition recurring to artificial neural networks (ANNs) and dynamic gesture recognition using hidden Markov models (HMM). Results show a recognition rate of 95% for a library of 22 gestures and 97% for a library of 6 behaviors. Experiments show a robot controlled using gestures in a HRI process.