Human-robot Interaction Tasks Research Articles

Adding Proximity Sensing Capability to Tactile Array Based on Off-the-Shelf FSR and PSoC

Tactile sensors have been incorporated into robots to help in many tasks, such as preventing damage to humans in collaborative works or assistance or in dexterous manipulation. Proximity sensors are also common in robotics and are proposed to implement pretouch in multimodal realizations. Many tactile sensors are based on off-the-shelf force-sensing resistors (FSRs). The parasitic capacitance associated with the electrodes of the FSR can be exploited to measure proximity. Moreover, it is possible to implement both force and proximity signal acquisition with a single chip. This article presents a multimodal proximity and tactile patch based on commercial FSR plus a programmable system on chip (PSoC). The ability to measure proximity is achieved without adding any extra element. Discussions about key design issues as well as results that show the performance of the sensor patch are presented. As proximity sensing array, the range is in the order of the size of the electrodes, and it depends on the properties of the object nearby, as expected. The best performance is observed for conductive objects. Therefore, this approach is especially interesting in human-robot interaction tasks, such as shaping robotic hands or grippers around the human body. Furthermore, dielectric objects are also detected, so it can also be used to implement pretouch in manipulation tasks.

Online control programming algorithm for human–robot interaction system with a novel real-time human gesture recognition method

This article proposes an online control programming algorithm for human–robot interaction systems, where robot actions are controlled by the recognition results of gestures performed by human operators based on visual images. In contrast to traditional robot control systems that use pre-defined programs to control a robot where the robot cannot change its tasks freely, this system allows the operator to train online and replan human–robot interaction tasks in real time. The proposed system is comprised of three components: an online personal feature pretraining system, a gesture recognition system, and a task replanning system for robot control. First, we collected and analyzed features extracted from images of human gestures and used those features to train the recognition program in real time. Second, a multifeature cascade classifier algorithm was applied to guarantee both the accuracy and real-time processing of our gesture recognition method. Finally, to confirm the effectiveness of our algorithm, we selected a flight robot as our test platform to conduct an online robot control experiment based on the visual gesture recognition algorithm. Through extensive experiments, the effectiveness and efficiency of our method has been confirmed.

Evaluation of Calf Muscle Reflex Control in the ‘Ankle Strategy’ during Upright Standing Push-Recovery

Revealing human internal control mechanisms during environmental interaction remains paramount and helpful in solving issues related to human-robot interaction. Muscle reflexes, which can directly and rapidly modify the dynamic behavior of joints, are the fundamental control loops of the Central Nervous System. This study investigates the calf muscle reflex control in the “ankle strategy” for human push-recovery movement. A time-increasing searching method is proposed to evaluate the feasibility of the reflex model in terms of predicting real muscle activations. Constraints with physiological implications are imposed to find the appropriate reflex gains. The experimental results show that the reflex model fits over 90% of the forepart of muscle activation. With the increasing of time, the Variance Accounted For (VAF) values drop to below 80% and reflex gains lose the physiology meaning. By dividing the muscle activation into two parts, the reflex formula is still workable for the rest part, with different gains and lower VAF values. This result may indicate that reflex control could more likely dominate the forepart of the push-recovery motion and an analogous control mechanism is still feasible for the rest of the motion part, with different gains. The proposed method provides an alternative way to obtain the human internal control mechanism desired for human-robot interaction task.

Impedance-Based Gaussian Processes for Modeling Human Motor Behavior in Physical and Non-Physical Interaction.

Modeling of human motor intention plays an essential role in predictively controlling a robotic system in human-robot interaction tasks. In most machine learning techniques, human motor behavior is modeled as a generic stochastic process. However, the integration of a priori knowledge about underlying system structures can provide insights on otherwise unobservable intrinsic states that yield the superior prediction performance and increased generalization capabilities. We present a novel method for modeling human motor behavior that explicitly includes a neuroscientifically supported model of human motor control, in which the dynamics of the human arm are modeled by a mechanical impedance that tracks a latent desired trajectory. We adopt a Bayesian setting by defining Gaussian process (GP) priors for the impedance elements and the latent desired trajectory. This enables exploitation of a priori human arm impedance knowledge for regression of interaction forces through inference of a latent desired human trajectory. The method is validated using simulated data, with particular focus on effects of GP prior parameterization and intention estimation capabilities. The superior prediction performance is shown with respect to a naive GP prior. Anexperiment with human participants evaluates generalization capabilities and effects of training data sparsity. We derive the correlations of an impedance-based GP model of human motor behavior that exploits a priori knowledge. The model effectively predicts interaction forces by inferring a latent desired human trajectory in previously observed as well as unobserved regions of the input space.

Cultural differences in speed adaptation in human-robot interaction tasks

Abstract In social interactions, human movement is a rich source of information for all those who take part in the collaboration. In fact, a variety of intuitive messages are communicated through motion and continuously inform the partners about the future unfolding of the actions. A similar exchange of implicit information could support movement coordination in the context of Human-Robot Interaction. In this work, we investigate how implicit signaling in an interaction with a humanoid robot can lead to emergent coordination in the form of automatic speed adaptation. In particular, we assess whether different cultures – specifically Japanese and Italian – have a different impact on motor resonance and synchronization in HRI. Japanese people show a higher general acceptance toward robots when compared with Western cultures. Since acceptance, or better affiliation, is tightly connected to imitation and mimicry, we hypothesize a higher degree of speed imitation for Japanese participants when compared to Italians. In the experimental studies undertaken both in Japan and Italy, we observe that cultural differences do not impact on the natural predisposition of subjects to adapt to the robot.

Dynamic Virtual Fixtures Based on Path Following Control

This paper presents a path following control (PFC) approach to generate guidance virtual fixtures with dynamic constraints (time-varying constraints) for physical human-robot interaction tasks. Using input-output feedback linearization and a parallel transport frame, the system dynamics are linearized and decomposed into tangential and orthogonal direction with respect to a time-varying path, which is parametrized as a regular threefold continuously differentiable curve. The presented PFC approach is combined with compliance control, which enables the systematic generation of virtual fixtures. An experiment on a six-axis industrial robot reveals the potential of this approach to generate virtual fixtures with dynamic constraints.

A Path/Surface Following Control Approach to Generate Virtual Fixtures

The workspace of a robot can be restricted by virtual fixtures to assist an operator in physical human–robot interaction tasks. This paper introduces a combination of surface following control (SFC) with compliance control and presents a path/SFC approach to systematically generate virtual fixtures. This approach allows implementation of numerous types of constraints like guidance and forbidden region virtual fixtures, hard and soft constraints, as well as static and dynamic virtual fixtures, and their combinations. Additionally, closed-loop stability proofs of the proposed control concepts are given. The flexibility of the presented approach is demonstrated by a series of measurement results from an industrial robot.

Semantics-based platform for context-aware and personalized robot interaction in the internet of robotic things

Robots are moving from well-controlled lab environments to the real world, where an increasing number of environments has been transformed into smart sensorized IoT spaces. Users will expect these robots to adapt to their preferences and needs, and even more so for social robots that engage in personal interactions. In this paper, we present declarative ontological models and a middleware platform for building services that generate interaction tasks for social robots in smart IoT environments. The platform implements a modular, data-driven workflow that allows developers of interaction services to determine the appropriate time, content and style of human-robot interaction tasks by reasoning on semantically enriched IoT sensor data. The platform also abstracts the complexities of scheduling, planning and execution of these tasks, and can automatically adjust parameters to the personal profile and current context. We present motivational scenarios in three environments: a smart home, a smart office and a smart nursing home, detail the interfaces and executional paths in our platform and present a proof-of-concept implementation.

Relieving operators’ workload: Towards affective robotics in industrial scenarios

This paper proposes a novel approach based on affective robotics that can be applied to industrial applications. Considering a human-robot interaction task, we propose to analyze the mental workload of the operator, and subsequently adapt the behavior of the robotic system, introducing assistive technologies. These technologies would prevent the performances deterioration caused by the human stress, helping him/her only when needed and decreasing the user’s mental workload. This represents a general methodology, which can be applied to several industrial applications, leading to increase the overall performances of human-robot interaction exploiting principles of human-centered design. As a case study, we consider a teleoperation task, where virtual fixtures are utilized as an assistive technology. The stress of the operator is monitored in terms of heart rate variability, measured by means of a wearable sensor tied at the operator’s wrist. Experimental validation of the proposed architecture is performed on a group of 15 users that teleoperate an industrial robot for performing a pick and place task.

Physically Plausible Wrench Decomposition for Multieffector Object Manipulation

When manipulating an object with multiple effectors such as in multidigit grasping or multiagent collaboration, forces and torques (i.e., wrench) applied to the object at different contact points generally do not fully contribute to the resultant object wrench, but partly compensate each other. The current literature, however, lacks a physically plausible decomposition of the applied wrench into its manipulation and internal components. We formulate the wrench decomposition as a convex optimization problem, minimizing the Euclidean norms of manipulation forces and torques. Physical plausibility in the optimization solution is ensured by constraining the internal and manipulation wrench by the applied wrench. We analyze specific cases of three-fingered grasping and 2-D beam manipulation, and show the applicability of our method to general object manipulation with multiple effectors. The wrench decomposition method is then extended to quantification of measures that are important in evaluating physical human–human and human–robot interaction tasks. We validate our approach via comparison to the state of the art in simulation and via application to a human–human object transport study.

Learning to Detect Pointing Gestures From Wearable IMUs

We propose a learning-based system for detecting when a user performs a pointing gesture, using data acquired from IMU sensors, by means of a 1D convolutional neural network. We quantitatively evaluate the resulting detection accuracy, and discuss an application to a human-robot interaction task where pointing gestures are used to guide a quadrotor landing.

Implementation of a Refusable Human-Robot Interaction Task with Humanoid Robot by Connecting Soar and ROS

This paper proposes combination of a cognitive agent architecture named Soar (State, operator, and result) and ROS (Robot Operating System), which can be a basic framework for a robot agent to interact and cope with its environment more intelligently and appropriately. The proposed Soar-ROS human-robot interaction (HRI) agent understands a set of human’s commands by voice recognition and chooses to properly react to the command according to the symbol detected by image recognition, implemented on a humanoid robot. The robotic agent is allowed to refuse to follow an inappropriate command like “go” after it has seen the symbol ‘X’ which represents that an abnormal or immoral situation has occurred. This simple but meaningful HRI task is successfully experimented on the proposed Soar-ROS platform with a small humanoid robot, which implies that extending the present hybrid platform to artificial moral agent is possible.

A Fast, Robust, and Incremental Model for Learning High-Level Concepts From Human Motions by Imitation

Social robots are becoming a companion in everyday life. To be well accepted by humans, they should efficiently understand meanings of their partners’ motions and body language and respond accordingly. Learning concepts by imitation brings them this ability in a user-friendly way. This paper presents a fast and robust model for incremental learning of concepts by imitation (ILoCI). In ILoCI, observed multimodal spatiotemporal demonstrations are incrementally abstracted and generalized based on their perceptual and functional similarities during the imitation. Perceptually similar demonstrations are abstracted by a dynamic model of the mirror neuron system. The functional similarities of demonstrations are also learned through a limited number of interactions with the teacher. Incremental relearning of acquired concepts together through memory rehearsal enables the learner to gradually extract and utilize the common structural relations among demonstrations to expedite the learning process especially at the initial stages. Performance of ILoCI is assessed using a standard benchmark dataset and a human–robot interaction task in which a humanoid robot learns to abstract teacher's hand motions during imitation. Its performance is also evaluated on occluded observations that are probable in real environments. The results show efficiency of ILoCI in concept acquisition, recognition, prediction, and generation in addition to its robustness to occlusions and high variability in observations.

Inverse Kinematics Based Human Mimicking System using Skeletal Tracking Technology

Humanoid robots needs to have human-like motions and appearance in order to be well-accepted by humans. Mimicking is a fast and user-friendly way to teach them human-like motions. However, direct assignment of observed human motions to robot's joints is not possible due to their physical differences. This paper presents a real-time inverse kinematics based human mimicking system to map human upper limbs motions to robot's joints safely and smoothly. It considers both main definitions of motion similarity, between end-effector motions and between angular configurations. Microsoft Kinect sensor is used for natural perceiving of human motions. Additional constraints are proposed and solved in the projected null space of the Jacobian matrix. They consider not only the workspace and the valid motion ranges of the robot's joints to avoid self-collisions, but also the similarity between the end-effector motions and the angular configurations to bring highly human-like motions to the robot. Performance of the proposed human mimicking system is quantitatively and qualitatively assessed and compared with the state-of-the-art methods in a human-robot interaction task using Nao humanoid robot. The results confirm applicability and ability of the proposed human mimicking system to properly mimic various human motions.

Single-Camera Automatic Landmarking for People Recognition with an Ensemble of Regression Trees

Active Appearance Model (AAM) is a computer vision procedure for statistical matching of object shape and appearance between images. A main drawback in this technique comes from the construction of the shape mesh. Since landmarks must be manually placed when training shapes, AAM is a very time consuming procedure and it cannot be automatically applied on new objects observed in the images. An approach for automatic landmarking of body shapes on still images for AAM training is introduced in this paper. Several works exist applying automatic landmarking on faces or body joints. Here, we explore the possibility to extend one of these methods to full body contours and demonstrate it is a plausible approach in terms of accuracy and speed measures in experimentation. Our proposal represents a new research line in human body pose tracking with a single-view camera. Hence, implementation in real-time would lead to people being recognized by robots endowed with minimal vision resources, like a webcam, in human-robot interaction tasks.

Real-Time Hand Posture Recognition for Human-Robot Interaction Tasks.

In this work, we present a multiclass hand posture classifier useful for human-robot interaction tasks. The proposed system is based exclusively on visual sensors, and it achieves a real-time performance, whilst detecting and recognizing an alphabet of four hand postures. The proposed approach is based on the real-time deformable detector, a boosting trained classifier. We describe a methodology to design the ensemble of real-time deformable detectors (one for each hand posture that can be classified). Given the lack of standard procedures for performance evaluation, we also propose the use of full image evaluation for this purpose. Such an evaluation methodology provides us with a more realistic estimation of the performance of the method. We have measured the performance of the proposed system and compared it to the one obtained by using only the sampled window approach. We present detailed results of such tests using a benchmark dataset. Our results show that the system can operate in real time at about a 10-fps frame rate.

Human robot cooperation planner using plans embedded in objects

Abstract: A human robot cooperation (HRC) planner for joint assembly tasks is presented. By combining first order logic (FoL) planning and object oriented programming, plans embedded in objects (PeO) can be produced. This concept shifts the focus of planning from actions to objects, thus defining them in terms of objects and combinations of their actions. The produced plans are represented using tree structures, which capture the task execution status in terms of completeness, agents’ roles and actions’ order. The planner and its application program interface (API) were written and developed using SWI-Prolog. This API allowed the planner to take various inputs to create different plans, and simultaneously to get a variety of useful outputs. The planner was tested in a collaborative pipe assembly task as a part of a bigger architecture implemented on the Baxter robot. It was able to generate the plan and achieve successful execution. This proof-of-concept planner shows great potential; allowing for a novel description in the HRC domain and execution of human-robot interaction tasks with industrial production in mind. However, first results are presented here; more work is required to take the work to higher technology readiness levels.

Stiffness-based tuning of an adaptive impedance controller for robot-assisted rehabilitation of upper limbs.

In this paper, the tuning procedure of an adaptive impedance control approach, for upper limb rehabilitation therapies assisted by robots, is presented. The main feature of the proposed approach is a custom tuning of the impedance parameters for the controller, based on the stiffness estimation of users (patients), thus achieving a suitable robot-assisted rehabilitation system according with the different conditions of user's mobility. A set of simulation results are presented, in order to verify the suitable performance of the proposed approach in human-robot interaction tasks.

Estimation of perturbations in robotic behavior using dynamic mode decomposition

Physical human–robot interaction tasks require robots that can detect and react to external perturbations caused by the human partner. In this contribution, we present a machine learning approach for detecting, estimating, and compensating for such external perturbations using only input from standard sensors. This machine learning approach makes use of Dynamic Mode Decomposition (DMD), a data processing technique developed in the field of fluid dynamics, which is applied to robotics for the first time. DMD is able to isolate the dynamics of a nonlinear system and is therefore well suited for separating noise from regular oscillations in sensor readings during cyclic robot movements. In a training phase, a DMD model for behavior-specific parameter configurations is learned. During task execution, the robot must estimate the external forces exerted by a human interaction partner. We compare the DMD-based approach to other interpolation schemes. A variant, sparsity promoting DMD, is particularly well suited for high-noise sensors. Results of a user study show that our DMD-based machine learning approach can be used to design physical human–robot interaction techniques that not only result in robust robot behavior but also enjoy a high usability.

A hybrid approach to dialogue management based on probabilistic rules

We present a new modelling framework for dialogue management based on the concept of probabilistic rules. Probabilistic rules are defined as structured mappings between logical conditions and probabilistic effects. They function as high-level templates for probabilistic graphical models and may include unknown parameters whose values are estimated from data using Bayesian inference. Thanks to their use of logical abstractions, probabilistic rules are able to encode the probability and utility models employed in dialogue management in a compact and human-readable form. As a consequence, they can reduce the amount of dialogue data required for parameter estimation and allow system designers to directly incorporate their expert domain knowledge into the dialogue models.Empirical results of a user evaluation in a human–robot interaction task with 37 participants show that a dialogue manager structured with probabilistic rules outperforms both purely hand-crafted and purely statistical methods on a range of subjective and objective quality metrics. The framework is implemented in a software toolkit called OpenDial, which can be used to develop various types of dialogue systems based on probabilistic rules.