Human-robot Interaction Applications Research Articles

Dynamic gesture recognition by directional pulse coupled neural networks for human-robot interaction in real time

Dynamic gesture recognition is a major topic for most real-time human-robot interaction applications. Recently, increasingly people have focused on dynamic gesture recognition based on 3D representation. In this paper, the gesture recognition is converted into the shortest path problem by transforming the feature matrix to an undirected graph and a novel dynamic gesture recognition algorithm of directional pulse coupled neuron network (DPCNN) is proposed for real time human-robot interactions. The DPCNN can select the firing direction by giving different excitations to neighbor neurons and reduce the effects of useless neurons. Furthermore, to reduce the recognition time, an early gesture recognition method based on the adaptive window is introduced to recognize the unfinished gestures. DPCNN reduces the computation time and achieves a high recognition rate on three public datasets compared with other algorithms which improves the efficiency of real-time dynamic gesture recognition and ensures a friendly experience for human-robot interactions.

A Survey of Human Action Analysis in HRI Applications

The human action is an important information source for human social interaction, and it simultaneously plays a crucial role in human–robot interaction (HRI). For a natural and fluent interaction, robots are required to understand human actions and have the capacity to predict action intentions and to imitate human actions for an appropriate response. Currently, existing survey papers for the action recognition mainly summarize algorithms that perform action recognition in experimental scenarios, and survey papers of the HRI mainly introduced various interaction interfaces in the HRI. Different from these surveys, we focus on the human action analysis on robot platforms for the HRI application, including the body motion and gestures. We review the existing HRI related references involving the action recognition, prediction, and the robot imitation of the human action. Moreover, we give a summary of robot platforms and action datasets that are frequently used in the study of HRI. Finally, we give an analysis on the development trend and future research directions of action analysis for the HRI applications.

Robust Sliding Mode Control for Robots Driven by Compliant Actuators

Compliant actuators can offer many attractive features over stiff actuators in real human–robot interaction applications, such as low output impedance, smooth force transmission, and shock tolerance. This brief focuses on a robust sliding mode control (SMC) methodology for robotic systems with compliant actuators. First, a continuous SMC design is introduced due to its advantages of strong robustness and chattering attenuation. However, this continuous SMC structure cannot guarantee a high tracking performance in the presence of mismatched disturbances in compliantly actuated robots. Meanwhile, in many application fields, compliantly actuated robots are affected by different kinds of time-varying disturbances, including external environmental disturbances, internal parameter uncertainties, and frictions, which may be in the form of constant, ramp, and parabolic disturbances. To estimate such unknown disturbances, a generalized proportional integral observer (GPIO) technique is employed. By designing a new sliding surface with the help of disturbance estimation, a GPIO-based continuous SMC method is synthesized, which is used to deal with matched/mismatched time-varying disturbances. A detailed stability analysis of the closed-loop system is also presented. Experimental results under three different test conditions are provided to illustrate the promising tracking performance of the proposed control strategy.

Minimum distance calculation using laser scanner and IMUs for safe human-robot interaction

Abstract In this study we investigate the use of a laser scanner/range-finder and inertial measurement units (IMUs) for the application of human-robot interaction in a dynamic environment with moving obstacles/humans. Humans and robots are represented by capsules, allowing to calculate the human-robot minimum distance on-the-fly. A major challenge is to capture the capsules pose. Data from a laser scanner and IMUs attached to the human body are fused to define the torso relative position and the upper body (arms and chest) configuration, respectively. Collision avoidance is achieved with a customized potential field’s method that allows to adjust the pre-defined robot paths established off-line while keeping the task target. The proposed framework is validated in real environment using a SICK laser scanner, IMUs and a KUKA iiwa robot. Experiments demonstrated the robustness of the proposed approach in capturing human motion, calculating the human-robot minimum distance and the robot behavior that smoothly avoids collisions with the human.

Discrete Hand Motion Intention Decoding Based on Transient Myoelectric Signals

In the application of human-robot interaction (HRI) rehabilitation exercise controlled by Electromyogram (EMG), if the discrete motion intention decoded by EMG signals is within the range of electromechanical delay (EMD), through the rehabilitation training, it will largely enhance the user's feeling feedback and fully activate the brain plasticity. So the decoding of hand movement intention by transient EMG is investigated to reach ideal characteristics needed in the HRI. The high-density EMG signal (HDEMG) database CapgMyo was used to decode the motion intention based on the transient EMG signals within the EMD rather than the whole recorded signals. We investigate the impact of the different decoding window lengths (WL) and training sets constructing methods when using several machine learning algorithms. In addition, the transient EMG signals decoding performance of the sparse multi-electrode EMG signal database NinaPro performing the same hand movement was compared. The visual inspection of the EMG map was used to determine the onset of HDEMG. The proposed approach was tested on EMG decoding window length of 150 ms, demonstrating a mean±SD testing performance of 94.21%±4.84% after voting. However, it is worth noting that sparse EMG signal did not achieve the desired decoding accuracy. The result showed that the high-density EMG signal could be used to decode the motion intention within EMD by simple machine learning algorithms, and extending the window length of training set could improve the decoding accuracy.

Cross-domain facial expression recognition via an intra-category common feature and inter-category Distinction feature fusion network

Facial expression recognition is crucial for various human-robot interaction applications, which requires facial expression analysis having a broad generalization. However, existing researches focus on the recognition in databases containing a limited number of samples. In this paper, we propose a novel feature fusion network for facial expression recognition in a cross-domain manner in order to realize the facial expression recognition in extensive scenarios. The proposed network consists of an Intra-category Common feature representation (IC) channel and an Inter-category Distinction feature representation (ID) channel for facial expression representation, and finally combine learned features of the two channels for facial expression recognition in cross databases. The IC channel learns the common features of intra-category facial expressions, and the ID channel learns the characteristic features of different categories. We evaluate the proposed approach in various experiment settings for cross-domain recognition, and achieves the state-of-the-art performances. We also evaluate the proposed approach for expression recognition in single databases, and also obtains the outstanding performance in the CK+, MMI, SFEW and RAF databases.

Reframing HRI Design Opportunities for Social Robots: Lessons Learnt from a Service Robotics Case Study Approach Using UX for HRI

Over the last few decades, semi-autonomous machine’s technology started to promote awareness towards the importance of human–robot interaction (HRI) for improving daily activities. More affordable social robots are being commercially released and in order to implement viable applications of HRI, a combination human-computer interaction and user experience methodologies could play a pivotal role in assessing new scenarios and evaluating new investigations. However, literature shows that it is still challenging to reach an optimal user experience with robotic companions. The aim of the study was to determine the chance to enhance the user experience with a semi-autonomous social robot, using user experience and human–computer interaction methodologies. In this study, a social robotic companion has been developed and prototyped in order to be adopted in a specific public environment such as a company workspace. The challenges emerged from this peculiar environment triggered the need for a more productive and comfortable office for the employees, and, at the same time, the usability, acceptance and likeability of the robotic companion have been evaluated. The results emphasize that, since HRI is highly interdisciplinary, the benefits of combining approaches from other fields could positively benefit from a meaningful social interaction with the users.

Soft Inflatable Sensing Modules for Safe and Interactive Robots

Recent advancements of technologies in human-robot interaction (HRI) significantly expanded the applications of robots from specialized areas to our daily lives. However, safety is not always guaranteed in traditional robots since they are mostly made of rigid materials and structures, which cause serious injuries to humans upon physical collisions. In this letter, we propose a soft inflatable module with self-contained tactile sensing for safe HRI applications. The proposed sensing module is made of a highly stretchable elastomer skin containing embedded microfluidic tactile sensors and a rigid bone structure. The module not only actively inflates and deflates, but also detects surface contacts using microfluidic soft pressure sensors to prevent human injuries caused by unintended collisions with robots and allowing follow-up actions to be taken, such as halting and evading. This letter describes the design and fabrication of the proposed inflatable sensing module, characterizes the performance of the soft sensors with different inflation levels, and evaluates the safety aspect of the module through collision tests. Finally, we present an example of applications for the proposed system by integrating multiple modules into a robotic arm with 4 DOF. The arm was tested for evasion at different levels of contacts.

Cartesian Impedance Control for Physical Human–Robot Interaction Using Virtual Decomposition Control Approach

This paper presents and verifies a novel Cartesian impedance control of the flexible joint manipulator for physical human–robot interaction application based on virtual decomposition control approach. Firstly, the Cartesian impedance control based on virtual decomposition control (VDC) is presented, and the asymptotical stability of the controller is proven by Lyapunov stability theorem. Compared with traditional methods based on singular perturbation, this method can greatly reduce the computational loads and is more suitable for real-time application. Then, a Cartesian force-feedback path planning combined with Cartesian impedance control based on VDC was used to keep the real contact force within the desired value to protect the manipulator and objects as a force, position, velocity and acceleration sensor, and the sensor can be configured freely by regulating the stiffness, damping and inertia, so the manipulator can interact with human (or unknown environment) in a friendly manner. The experimental results illustrate the validity of the developed VDC-based impedance control approach.

A real-time spike-timing classifier of spatio-temporal patterns

Abstract Considering the problem of recognizing non-verbal cues in Human–Robot Interaction applications, this paper proposes a novel real-time unsupervised spike timing neural network for recognition and early detection of spatio-temporal human gestures. Two spiking network classifiers one based on Izhikevich neuron model, and the other one based on Integrate-and-Fire-or-Burst neuron model have been implemented in CUDA, and allow the classification to be performed in real-time. To evaluate the performance of this proposal, we test the case of a physical robot observing air-handwritings of human gesture. The proposed approaches run in real-time, thus they are suitable for human–robot applications; they allow real-time early classifying human gestures and actions while they require a very small number of training samples. In comparing to other prominent techniques, our approaches demonstrate superior accuracy and are suitable for early classification of different types of human actions in time-sensitive mobile applications such as robotics.

A Position and Torque Switching Control Method for Robot Collision Safety

With the increasing number of human-robot interaction applications, robot control characteristics and their effects on safety as well as performance should be taken account into the robot control system. In this paper, a position and torque switching control method was proposed to improve the robot safety and performance, when robots and humans work in the same space. The switching control method includes two modes, the position control mode using a proportion-integral (PI) algorithm, and the torque control mode using sliding mode control (SMC) algorithm for eliminating swing. Under the normal condition, the robot works in position control mode for trajectory tracking with quick response. Once the robot and human collide, the robot will switch to torque control mode immediately, and the impact force will be restricted within a safe range. When the robot and human detach, the robot will resume to position control mode automatically. Moreover, for a better performance, the joint torque is detected from direct-current (DC) motor′s current rather than the torque sensor. The experiment results show that the proposed approach is effective and feasible.

Skeleton-based bio-inspired human activity prediction for real-time human–robot interaction

Activity prediction is an essential task in practical human-centered robotics applications, such as security, assisted living, etc., which is targeted at inferring ongoing human activities based on incomplete observations. To address this challenging problem, we introduce a novel bio-inspired predictive orientation decomposition (BIPOD) approach to construct representations of people from 3D skeleton trajectories. BIPOD is invariant to scales and viewpoints, runs in real-time on basic computer systems, and is able to recognize and predict activities in an online fashion. Our approach is inspired by biological research in human anatomy. To capture spatio-temporal information of human motions, we spatially decompose 3D human skeleton trajectories and project them onto three anatomical planes (i.e., coronal, transverse and sagittal planes); then, we describe short-term time information of joint motions and encode high-order temporal dependencies. By using Extended Kalman Filters to estimate future skeleton trajectories, we endow our BIPOD representation with the critical capabilities to reduce noisy skeleton observation data and predict the ongoing activities. Experiments on benchmark datasets have shown that our BIPOD representation significantly outperforms previous methods for real-time human activity classification and prediction from 3D skeleton trajectories. Empirical studies using TurtleBot2 and Baxter humanoid robots have also validated that our BIPOD method obtains promising performance, in terms of both accuracy and efficiency, making BIPOD a fast, simple, yet powerful representation for low-latency online activity prediction in human–robot interaction applications.

A Pneumatic Tactile Sensor for Co-Operative Robots.

Tactile sensors of comprehensive functions are urgently needed for the advanced robot to co-exist and co-operate with human beings. Pneumatic tactile sensors based on air bladder possess some noticeable advantages for human-robot interaction application. In this paper, we construct a pneumatic tactile sensor and apply it on the fingertip of robot hand to realize the sensing of force, vibration and slippage via the change of the pressure of the air bladder, and we utilize the sensor to perceive the object’s features such as softness and roughness. The pneumatic tactile sensor has good linearity, repeatability and low hysteresis and both its size and sensing range can be customized by using different material as well as different thicknesses of the air bladder. It is also simple and cheap to fabricate. Therefore, the pneumatic tactile sensor is suitable for the application of co-operative robots and can be widely utilized to improve the performance of service robots. We can apply it to the fingertip of the robot to endow the robotic hand with the ability to co-operate with humans and handle the fragile objects because of the inherent compliance of the air bladder.

Learning models of Human-Robot Interaction from small data.

This paper offers a new approach to learning discrete models for human-robot interaction (HRI) from small data. In the motivating application, HRI is an integral part of a pediatric rehabilitation paradigm that involves a play-based, social environment aiming at improving mobility for infants with mobility impairments. Designing interfaces in this setting is challenging, because in order to harness, and eventually automate, the social interaction between children and robots, a behavioral model capturing the causality between robot actions and child reactions is needed. The paper adopts a Markov decision process (MDP) as such a model, and selects the transition probabilities through an empirical approximation procedure called smoothing. Smoothing has been successfully applied in natural language processing (NLP) and identification where, similarly to the current paradigm, learning from small data sets is crucial. The goal of this paper is two-fold: (i) to describe our application of HRI, and (ii) to provide evidence that supports the application of smoothing for small data sets.

Human-Robot Interaction and Neuroprosthetics: A review of new technologies

New technologies in the field of neuroprosthetics and robotics are leading to the development of innovative commercial products based on user-centered, functional processes of cognitive neuroscience and perceptron studies. The aim of this review is to analyze this innovative path through the description of some of the latest neuroprosthetics and human-robot interaction applications, in particular the brain-computer interface (BCI) linked to haptic systems, interactive robotics, and autonomous systems. These issues will be addressed by analyzing developmental robotics and examples of neurorobotics research. These new devices show the benefits of using an interdisciplinary approach based on cognitive neuroscience, embodied and situated cognition, neural network, and deep learning. All of these products share the capability to adapt, online, to the dynamic environment and to the user's actions. As the role of the human tutor is key in this learning process, these systems permit a natural and dynamic approach to interaction among people, neuroprosthetics, and robotics self-extensions.

Real-time ensemble based face recognition system for NAO humanoids using local binary pattern

NAO humanoid robots are being used in many human-robot interaction applications. One of the important existing challenges is developing an accurate real-time face recognition system which does not require to have high computational cost. In this research work a real-time face recognition system by using block processing of local binary patterns of the face images captured by NAO humanoid is proposed. Majority voting and best score ensemble approaches have been used in order to boost the recognition results obtained in different colour channels of YUV colour space, which is a default colour space provided by the camera of NAO humanoid. The proposed method has been adopted on NAO humanoid and tested under real-world conditions. The recognition results were boosted in the real-time scenario by employing majority voting on the intra-sequence decisions with window size of 5. The experimental results are showing that the proposed face recognition algorithm overcomes the conventional and state-of-the-art techniques.

Bidirectional invariant representation of rigid body motions and its application to gesture recognition and reproduction

In this paper we propose a new bidirectional invariant motion descriptor of a rigid body. The proposed invariant representation is not affected by rotations, translations, time, linear and angular scaling. Invariant properties of the proposed representation enable to recognize gestures in realistic scenarios with unexpected variations (e.g., changes in user’s initial pose, execution time or an observation point), while Cartesian trajectories are sensitive to these changes. The proposed invariant representation also allows reconstruction of the original motion trajectory, which is useful for human-robot interaction applications where a robot recognizes human actions and executes robot’s proper behaviors using same descriptors. By removing the dependency on absolute pose and scaling factors of the Cartesian trajectories the proposed descriptor achieves flexibility to generate different motion instances from the same invariant representation. In order to illustrate the effectiveness of our proposed descriptor in motion recognition and generation, it is tested on three datasets and experiments on a NAO humanoid robot and a KUKA LWR IV $$+$$ manipulator and compared with other existing invariant representations.

Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods

Head pose estimation is an old problem that is recently receiving new attention because of possible applications in human-robot interaction, augmented reality and driving assistance. However, most of the existing work has been tested in controlled environments and is not robust enough for real-world applications. In order to handle these limitations we propose an approach based on Convolutional Neural Networks (CNNs) supplemented with the most recent techniques adopted from the deep learning community. We evaluate the performance of four architectures on recently released in-the-wild datasets. Moreover, we investigate the use of dropout and adaptive gradient methods giving a contribution to their ongoing validation. The results show that joining CNNs and adaptive gradient methods leads to the state-of-the-art in unconstrained head pose estimation.

A Quantitative Analysis of Dressing Dynamics for Robotic Dressing Assistance

Assistive robots have a great potential to address issues related to an ageing population and an increased demand for caregiving. Successful deployment of robots working in close proximity with people requires consideration of both safety and human-robot interaction. One of the established activities of daily living where robots could play an assistive role is dressing. Using the correct force profile for robot control will be essential in this application of human-robot interaction requiring careful exploration of factors related to the user’s pose and the type of garments involved. In this paper a Baxter robot was used to dress a jacket onto a mannequin and human participants considering several combinations of user pose and clothing type (base layers), whilst recording dynamic data from the robot, a load cell and an IMU. We also report on suitability of these sensors for identifying dressing errors, e.g. fabric snagging. Data was analyzed by comparing the overlap of confidence intervals to determine sensitivity to dressing. We expand the analysis to include classification techniques such as decision tree and support vector machines using k-fold cross-validation. The 6-axis load cell successfully discriminated between clothing types with predictive model accuracies between 72-97%. Used independently, the IMU and Baxter sensors were insufficient to discriminate garment types with the IMU showing 40-72% accuracy, but when used in combination this pair of sensors achieved an accuracy similar to the more expensive load cell (98%). When observing dressing errors (snagging) Baxter’s sensors and the IMU data demonstrated poor sensitivity but applying machine learning methods resulted in model with high predicative accuracy and low false negative rates (≤5%). The results show that the load cell could be used independently for this application with good accuracy but a combination of the lower cost sensors could also be used without a significant loss in precision which will be a key element in the robot control architecture for safe human-robot interaction.

A Bayesian hierarchy for robust gaze estimation in human–robot interaction

In this text, we present a probabilistic solution for robust gaze estimation in the context of human–robot interaction. Gaze estimation, in the sense of continuously assessing gaze direction of an interlocutor so as to determine his/her focus of visual attention, is important in several important computer vision applications, such as the development of non-intrusive gaze-tracking equipment for psychophysical experiments in neuroscience, specialised telecommunication devices, video surveillance, human–computer interfaces (HCI) and artificial cognitive systems for human–robot interaction (HRI), our application of interest. We have developed a robust solution based on a probabilistic approach that inherently deals with the uncertainty of sensor models, but also and in particular with uncertainty arising from distance, incomplete data and scene dynamics. This solution comprises a hierarchical formulation in the form of a mixture model that loosely follows how geometrical cues provided by facial features are believed to be used by the human perceptual system for gaze estimation. A quantitative analysis of the proposed framework's performance was undertaken through a thorough set of experimental sessions. Results show that the framework performs according to the difficult requirements of HRI applications, namely by exhibiting correctness, robustness and adaptiveness.