Human-robot Interaction System Research Articles

Research on Discriminative Skeleton-Based Action Recognition in Spatiotemporal Fusion and Human-Robot Interaction

A novel posture motion-based spatiotemporal fused graph convolutional network (PM-STGCN) is presented for skeleton-based action recognition. Existing methods on skeleton-based action recognition focus on independently calculating the joint information in single frame and motion information of joints between adjacent frames from the human body skeleton structure and then combine the classification results. However, that does not take into consideration of the complicated temporal and spatial relationship of the human body action sequence, so they are not very efficient in distinguishing similar actions. In this work, we enhance the ability of distinguishing similar actions by focusing on spatiotemporal fusion and adaptive feature extraction for high discrimination information. Firstly, the local posture motion-based attention (LPM-TAM) module is proposed for the purpose of suppressing the skeleton sequence data with a low amount of motion in the temporal domain, and the representation of motion posture features is concentrated. Besides, the local posture motion-based channel attention module (LPM-CAM) is introduced to make use of the strongly discriminative representation between different action classes of similarity. Finally, the posture motion-based spatiotemporal fusion (PM-STF) module is constructed which fuses the spatiotemporal skeleton data by filtering out the low-information sequence and enhances the posture motion features adaptively with high discrimination. Extensive experiments have been conducted, and the results demonstrate that the proposed model is superior to the commonly used action recognition methods. The designed human-robot interaction system based on action recognition has competitive performance compared with the speech interaction system.

A bang-bang control model predicts the triphasic muscles activity during hand reaching.

There are numerous ways to reach for an apple hanging from a tree. Yet, our motor system uses a specific muscle activity pattern that features activity bursts and silent periods. We suggest that these bursts are an evidence against the common view that the brain controls the commands to the muscles in a smooth continuous manner. Instead, we propose a model in which a motor plan is transformed into a piecewise-constant control signal that is low-pass filtered and transmitted to the muscles with different muscle-specific delays. We use a Markov chain Monte Carlo (MCMC) method to identify transitions in the state of the muscles following initial activation and show that fitting a bang-bang control model to the kinematics of movement predicts these transitions in the state of the muscles. Such a bang-bang controller suggests that the brain reduces the complexity of the problem of ballistic movements control by sending commands to the muscles at sparse times. Identifying this bang-bang controller can be useful to develop efficient controllers for neuroprostheses and other physical human-robot interaction systems.NEW & NOTEWORTHY While ballistic hand reaching movements are characterized by smooth position and velocity signals, the activity of the muscles exhibits bursts and silent periods. Here, we propose that a model based on bang-bang control provides the link between the abrupt changes in the muscle activity and the smooth reaching trajectory. Using bang-bang control instead of continuous control may simplify the design of prostheses and other physical human-robot interaction systems.

Compliant Manipulation Method for a Nursing Robot Based on Physical Structure of Human Limb

Nursing robots can be a good substitute to do nursing works for severely disabled patients who have completely lost the ability of movement. In order to improve the safety and comfort of human-robot interaction, we propose a novel compliant manipulation method for a nursing robot based on physical structure of human limb, which is applicable to severely disabled patients and can reduce the high performance requirements of sensors in existing human-robot interaction systems. The method aims to obtain the optimal manipulative force in static and dynamic manipulations. In the aspect of static calculation, static equilibrium conditions are used to obtain optimal static manipulative force. And in the aspect of dynamic calculation, lagrangian method is used to obtain optimal dynamic manipulative force based on predefined movement trajectory. Adams simulations and experiments are executed by manipulating limb model to conduct rehabilitation movements. Experimental results show that the proposed method can provide compliant manipulation method for a nursing robot.

Twisted Rubber Variable-Stiffness Artificial Muscles.

Variable-stiffness artificial muscles are important in many applications including running and hopping robots, human–robot interaction, and active suspension systems. Previously used technologies include pneumatic muscles, layer and granular jamming, series elastic actuators, and shape memory polymers. All these are limited in terms of cost, complexity, the need for fluid power supplies, or controllability. In this article, we present a new concept for variable-stiffness artificial muscles (the twisted rubber artificial muscle, TRAM) made from twisted rubber cord that overcomes these limitations. Rubber cord is inexpensive, readily available, and inherently compliant. When an extended piece of rubber cord is twisted, the tensile force it exerts is reduced and its stiffness is altered. This behavior makes twisted rubber ideal for use as an artificial muscle, because its output force and natural stiffness are both controllable by varying twist angle. We investigate the behavior of four types of rubber cord and evaluate which type of rubber allows for the greatest reversible reduction in average stiffness (fluoroelastomer [FKM standard] rubber, 56.42% reduction) and initial stiffness (silicone rubber, 92.62%). Tensile force and stiffness can be further altered by increasing the twist angle of the artificial muscle beyond a threshold angle, which initiates nonlinear buckling behavior. This, however, can cause plastic deformation of the artificial muscle. Using a single TRAM, we show how the equilibrium position and natural frequency of a system can be simultaneously altered by controlling twist angle. We further demonstrate independent position and stiffness control of a functional robotic arm system using an antagonistic pair of TRAMs. TRAMs are ready for immediate inclusion in a wide range of robotic systems.

Human Interaction Smart Subsystem-Extending Speech-Based Human-Robot Interaction Systems with an Implementation of External Smart Sensors.

This paper presents a more detailed concept of Human-Robot Interaction systems architecture. One of the main differences between the proposed architecture and other ones is the methodology of information acquisition regarding the robot’s interlocutor. In order to obtain as much information as possible before the actual interaction took place, a custom Internet-of-Things-based sensor subsystems connected to Smart Infrastructure was designed and implemented, in order to support the interlocutor identification and acquisition of initial interaction parameters. The Artificial Intelligence interaction framework of the developed robotic system (including humanoid Pepper with its sensors and actuators, additional local, remote and cloud computing services) is being extended with the use of custom external subsystems for additional knowledge acquisition: device-based human identification, visual identification and audio-based interlocutor localization subsystems. These subsystems were deeply introduced and evaluated in this paper, presenting the benefits of integrating them into the robotic interaction system. In this paper a more detailed analysis of one of the external subsystems—Bluetooth Human Identification Smart Subsystem—was also included. The idea, use case, and a prototype, integration of elements of Smart Infrastructure systems and the prototype implementation were performed in a small front office of the Weegree company as a decent test-bed application area.

Identification of Upper-Limb Movements Based on Muscle Shape Change Signals for Human-Robot Interaction.

Towards providing efficient human-robot interaction, surface electromyogram (EMG) signals have been widely adopted for the identification of different limb movement intentions. Since the available EMG signal sensors are highly susceptible to external interferences such as electromagnetic artifacts and muscle fatigues, the quality of EMG recordings would be mostly corrupted, which may decay the performance of EMG-based control systems. Given the fact that the muscle shape changes (MSC) would be different when doing various limb movements, the MSC signal would be nonsensitive to electromagnetic artifacts and muscle fatigues and maybe promising for movement intention recognition. In this study, a novel nanogold flexible and stretchable sensor was developed for the acquisition of MSC signals utilized for decoding multiple classes of limb movement intents. More precisely, four sensors were used to measure the MSC signals from the right forearm of each subject when they performed seven classes of movements. Also, six different features were extracted from the measured MSC signals, and a linear discriminant analysis- (LDA-) based classifier was built for movement classification tasks. The experimental results showed that using MSC signals could achieve an average recognition rate of about 96.06 ± 1.84% by properly placing the four flexible and stretchable sensors on the forearm. Additionally, when the MSC sampling rate was greater than 100 Hz and the analysis window length was greater than 20 ms, the movement recognition accuracy would be only slightly increased. These pilot results suggest that the MSC-based method should be feasible in movement identifications for human-robot interaction, and at the same time, they provide a systematic reference for the use of the flexible and stretchable sensors in human-robot interaction systems.

Simultaneously Encoding Movement and sEMG-Based Stiffness for Robotic Skill Learning

Transferring human stiffness regulation strategies to robots enables them to effectively and efficiently acquire adaptive impedance control policies to deal with uncertainties during the accomplishment of physical contact tasks in an unstructured environment. In this article, we develop such a physical human-robot interaction system which allows robots to learn variable impedance skills from human demonstrations. Specifically, the biological signals, i.e., surface electromyography are utilized for the extraction of human arm stiffness features during the task demonstration. The estimated human arm stiffness is then mapped into a robot impedance controller. The dynamics of both movement and stiffness are simultaneously modeled by using a model combining the hidden semi-Markov model and the Gaussian mixture regression. More importantly, the correlation between the movement information and the stiffness information is encoded in a systematic manner. This approach enables capturing uncertainties over time and space and allows the robot to satisfy both position and stiffness requirements in a task with modulation of the impedance controller. The experimental study validated the proposed approach.

Design and verification of a human–robot interaction system for upper limb exoskeleton rehabilitation

This paper presents the design of a motion intent recognition system, based on an altitude signal sensor, to improve the human–robot interaction performance of upper limb exoskeleton robots during rehabilitation training. A modified adaptive Kalman filter combined with clipping filtering is proposed for the control system to mitigate the noise and time delay of the collected signal. The clipping filtering method was used to filter the accidental error and avoid the safety problem caused by a mistrigger. A modified adaptive Kalman filter was used to account for the sudden change of the motion state during rehabilitation training. The results show that the intent recognition system designed herein can accurately recognize the human–robot interaction information, and estimate the intent of human motion in time. Therefore, it can be concluded that the designed system effectively follows the predicted motion intent with the proposed method, which is a significant improvement for human–robot interaction control of upper limb extremity rehabilitation robots.

An interactive system for humanoid robot SHFR-III

The natural interaction between human and robot is full of challenges but indispensable. In this article, a human–robot interactive system is designed for humanoid robot SHFR-III. The system consists of three subsystems: multi-sensor positioning subsystem, emotional interaction subsystem, and dialogue subsystem. The multi-sensor positioning subsystem is designed to improve the positioning accuracy, the emotional interaction subsystem uses bimodal emotional recognition model and fuzzy emotional decision-making model to realize the emotion recognition and expression feedback to the interactive objects, and the dialogue subsystem with personal information can complete the response consistent with the default information and avoid conflicting responses .The experimental results show that the multi-sensor positioning subsystem has good environmental adaptability and positioning accuracy, the emotional interaction subsystem can achieve human-like emotional feedback, and the dialogue subsystem can achieve more natural, logical, and consistent responses.

Switching Structured Prediction for Simple and Complex Human Activity Recognition.

Automatic human activity recognition is an integral part of any interactive application involving humans (e.g., human-robot interaction systems). One of the main challenges for activity recognition is the diversity in the way individuals often perform activities. Furthermore, changes in any of the environment factors (i.e., illumination, complex background, human body shapes, viewpoint, etc.) intensify this challenge. In addition, there are different types of activities that robots need to interpret for seamless interaction with humans. Some activities are short, quick, and simple (e.g., sitting), while others may be detailed/complex, and spread throughout a long span of time (e.g., washing mouth). In this article, we recognize the activities within the context of graphical models in a sequence-labeling framework based on skeleton data. We propose a new structured prediction strategy based on probabilistic graphical models (PGMs) to recognize both types of activities (i.e., complex and simple). These activity types are often spanned in very diverse subspaces in the space of all possible activities, which would require different model parameterizations. In order to deal with these parameterization and structural breaks across models, a category-switching scheme is proposed to switch over the models based on the activity types. For parameter optimization, we utilize a distributed structured prediction technique to implement our model in a distributed setting. The method is tested on three widely used datasets (CAD-60, UT-Kinect, and Florence 3-D) that cover both activity types. The results illustrate that our proposed method is able to recognize simple and complex activities while the previous work concentrated on only one of these two main types.

Human-robot interaction in Industry 4.0 based on an Internet of Things real-time gesture control system

In the context of fast-growing digitization of industrial environments, Industry 4.0 aims to improve key elements to achieve more efficient processes, flexibility in customizing products and reduction in energy consumption, among other objectives. This paper presents a system that exploits the Internet of Things (IoT), massive data computation, and human-robot collaboration to reach these goals. The described system combines technological and human-centered aspects to enhance human-robot interaction. In fact, the human factor cannot be left aside when technological advances affecting society are foreseen. The proposal has been tested on a gesture control system that enables a natural interaction with a robotic arm through the use of IoT-oriented inertial measurement unit devices. These devices capture the movements of both human’s arms. Experiments of a technical nature have been run to measure accuracy and latency. In addition, human-centered tests have been conducted with real users to determine the level of intuitiveness and acceptance of the proposed gesture control. The results obtained demonstrate that the proposal meets the demands in terms of real-time, success rate, flexibility and scalability, which are fundamental requirements in Industry 4.0. The usability results have enabled drawing useful conclusions on the use of such human-robot interaction systems.

Robots and Wizards: An Investigation Into Natural Human–Robot Interaction

The goal of the study was to research different communication modalities needed for intuitive Human-Robot Interaction. This study utilizes a Wizard of Oz prototyping method to enable a restriction-free, intuitive interaction with an industrial robot. The data from 36 test subjects suggests a high preference for speech input, automatic path planning and pointing gestures. The catalogue developed during this experiment contains intrinsic gestures suggesting that the two most popular gestures per action can be sufficient to cover the majority of users. The system scored an average of 74% in different user interface experience questionnaires, while containing forced flaws. These findings allow a future development of an intuitive Human-Robot interaction system with high user acceptance.

Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection

Electroencephalogram (EEG) emotion recognition based on a hybrid feature extraction method in empirical mode decomposition domain combining with optimal feature selection based on sequence backward selection is proposed, which can reflect subtle information of multiscale components of unstable and nonlinear EEG signals and remove the reductant features to improve the performance of emotion recognition. The proposal is tested on DEAP dataset, in which the emotional states in the Valance dimension and Arousal dimension are classified by both ${K}$ -nearest neighbor and support vector machine, respectively. In the experiments, temporal windows of different length and three kinds of rhythms of EEG signal are taken into account for comparison, from which the results show that EEG signal with 1s temporal window achieves highest recognition accuracy of 86.46% in Valence dimension and 84.90% in Arousal dimension, respectively, which is superior to some state-of-the-art works. The proposed method would be applied to real-time emotion recognition in multimodal emotional communication-based humans–robots interaction system.

Human–robot interaction in industrial collaborative robotics: a literature review of the decade 2008–2017

ABSTRACT Currently, a large number of industrial robots have been deployed to replace or assist humans to perform various repetitive and dangerous manufacturing tasks. However, based on current technological capabilities, such robotics field is rapidly evolving so that humans are not only sharing the same workspace with robots, but also are using robots as useful assistants. Consequently, due to this new type of emerging robotic systems, industrial collaborative robots or cobots, human and robot co-workers have been able to work side-by-side as collaborators to accomplish tasks in industrial environments. Therefore, new human–robot interaction systems have been developed for such systems to be able to utilize the capabilities of both humans and robots. Accordingly, this article presents a literature review of major recent works on human–robot interactions in industrial collaborative robots, conducted during the last decade (between 2008 and 2017). Additionally, the article proposes a tentative classification of the content of these works into several categories and sub-categories. Finally, this paper addresses some challenges of industrial collaborative robotics and explores future research issues.

On the design of a human–robot interaction strategy for commercial vehicle driving based on human cognitive parameters

A proper design of human–robot interaction strategies based on human cognitive factors can help to compensate human limitations for safety purposes. This work is focused on the development of a human–robot interaction system for commercial vehicle (Renault Twizy) driving, that uses driver cognitive parameters to improve driver’s safety during day and night tasks. To achieve this, eye blink behavior measurements are detected using a convolutional neural network, which is capable of operating under variable illumination conditions using an infrared camera. Percentage of eye closure measure values along with blink frequency are used to infer diver’s sleepiness level. The use of such algorithm is validated with experimental tests for subjects under different sleep-quality conditions. Additional cognitive parameters are also analyzed for the human–robot interaction system such as driver sleep quality, distraction level, stress level, and the effects related to not wearing glasses. Based on such driver cognitive state parameters, a human–robot interaction strategy is proposed to limit the speed of a Renault Twizy vehicle by intervening its acceleration and braking system. The proposed human–robot interaction strategy can increase safety during driving tasks for both users and pedestrians.

Sensorless force estimation of end-effect upper limb rehabilitation robot system with friction compensation

In human robot interaction systems, human intent detection plays an important role to improve the interactive performances and then the rehabilitation effects. A study is proposed to estimate the interactive forces that indirectly detect the human motion intent. A disturbance observer is designed to estimate interactive torques and friction forces without force sensors, and then a friction force model is constructed to estimate the friction force in the robot system. To detect the human–robot interaction force, we subtract the friction force from disturbance observer estimation result. Several experiments were performed to test the performances of the proposed methods. Those methods were applied in an end-effect upper limb rehabilitation robot system. The results show that the precision of the estimated sensor force can increase 5% than the force sensor. The senseless force estimation method we proposed in this article can be an alternative option in force control tasks when force sensors are not suitable.

Online control programming algorithm for human–robot interaction system with a novel real-time human gesture recognition method

This article proposes an online control programming algorithm for human–robot interaction systems, where robot actions are controlled by the recognition results of gestures performed by human operators based on visual images. In contrast to traditional robot control systems that use pre-defined programs to control a robot where the robot cannot change its tasks freely, this system allows the operator to train online and replan human–robot interaction tasks in real time. The proposed system is comprised of three components: an online personal feature pretraining system, a gesture recognition system, and a task replanning system for robot control. First, we collected and analyzed features extracted from images of human gestures and used those features to train the recognition program in real time. Second, a multifeature cascade classifier algorithm was applied to guarantee both the accuracy and real-time processing of our gesture recognition method. Finally, to confirm the effectiveness of our algorithm, we selected a flight robot as our test platform to conduct an online robot control experiment based on the visual gesture recognition algorithm. Through extensive experiments, the effectiveness and efficiency of our method has been confirmed.

Adolescents Environmental Emotion Perception by Integrating EEG and Eye Movements.

Giving a robot the ability to perceive emotion in its environment can improve human-robot interaction (HRI), thereby facilitating more human-like communication. To achieve emotion recognition in different built environments for adolescents, we propose a multi-modal emotion intensity perception method using an integration of electroencephalography (EEG) and eye movement information. Specifically, we first develop a new stimulus video selection method based on computation of normalized arousal and valence scores according to subjective feedback from participants. Then, we establish a valence perception sub-model and an arousal sub-model by collecting and analyzing emotional EEG and eye movement signals, respectively. We employ this dual recognition method to perceive emotional intensities synchronously in two dimensions. In the laboratory environment, the best recognition accuracies of the modality fusion for the arousal and valence dimensions are 72.8 and 69.3%. The experimental results validate the feasibility of the proposed multi-modal emotion recognition method for environment emotion intensity perception. This promising tool not only achieves more accurate emotion perception for HRI systems but also provides an alternative approach to quantitatively assess environmental psychology.

A SEMG-angle model based on HMM for human robot interaction.

BACKGROUND: An important part of the rehabilitation process using exoskeleton robots has been the creation of a friendly Human Robot Interaction (HRI) system.OBJECTIVE: In order to combine SEMG signal into the HRI system, a SEMG-angle model based on Hidden Markov Model (HMM) was put forward in this paper.METHODS: Feature extraction as a critical issue of signal preprocessing was handled by Principal Component Analysis (PCA) which realized signal data dimension reduction and solved the common problem of redundant features. A comparison study was given to show the different performance of various EMG-angle model separately based on HMM, Back Propagation (BP) neural network and Radial Basis Function (RBF) neural network.RESULTS: The HMM modeling method which with lower calculation complexity can achieve a better modeling performance (average accuracy 93.063%) compared with BP neural network (average accuracy 88.180%) and RBF neural network (average accuracy 88.752%).CONCLUSIONS: SEMG signals have some characteristic properties which is similar to a quasi-stationary filtered white noise stochastic process, the structure of HMMs makes it ideally suited for classification and modeling SEMG signals, and the results of this study show that it can achieve a better performance than the commonly used methods (BP and RBF).

An Initiative Service Method Based on Fuzzy Analytical Hierarchy Process and Context Intention Inference for Drinking Service Robot

To ensure that people can replenish the right amount of water when they are short of it, an initiative service method based on the fuzzy analytical hierarchy process (FAHP) and context intention inference for the drinking service robot is proposed, in which the factors that affect people’s drinking demand including individual factors (e.g., physical activity, age, and gender) and environmental factors (e.g., temperature and humidity) are taken into account for drinking service. In addition, a two-level hierarchy structure for human intention together with intention degree is proposed, in which weights between context features and intention are calculated by FAHP, which imitates human’s subjective judgment process mathematically and mentally. Moreover, the relationship between human intention and their demands is established by a demand analysis model for providing satisfactory service of robots. Drinking service experiments are performed in a laboratory scenario using a humans–robots interaction system, from which the experimental results show that the resultant ${p}$ -value in Condition 1 is 0.455, and the resultant ${p}$ -value in Condition 2 is 0.09, and the averaged values of satisfaction feedback are 2.15 and 1.85 (satisfaction from 1 to 5 decreased in turn), respectively, approximately satisfied, which demonstrates the feasibility of the proposal.