Robot Interaction Applications Research Articles

A simulation‐assisted point cloud segmentation neural network for human–robot interaction applications

AbstractWith the advancement of industrial automation, the frequency of human–robot interaction (HRI) has significantly increased, necessitating a paramount focus on ensuring human safety throughout this process. This paper proposes a simulation‐assisted neural network for point cloud segmentation in HRI, specifically distinguishing humans from various surrounding objects. During HRI, readily accessible prior information, such as the positions of background objects and the robot's posture, can generate a simulated point cloud and assist in point cloud segmentation. The simulation‐assisted neural network utilizes simulated and actual point clouds as dual inputs. A simulation‐assisted edge convolution module in the network facilitates the combination of features from the actual and simulated point clouds, updating the features of the actual point cloud to incorporate simulation information. Experiments of point cloud segmentation in industrial environments verify the efficacy of the proposed method.

Antagonistic Magneto-Rheological Actuators with Inherent Output Boundedness: An Ideal Solution for High-Performance and Human-Safe Actuation

This paper studies the working principles of antagonistic magneto-rheological (MR) actuators, i.e., a combination of an electric motor and a pair of MR clutches in an antagonistic configuration, for compliant actuation in robotics. The study focuses on the unique boundedness property exhibited by MR actuators, which limits the output torques delivered to the load, independent of the received input torque and/or control commands. This inherent property is of significant importance for ensuring human safety in human–robot interaction applications. Through a comprehensive analysis, we provide analytical proof of the inherent output boundedness of antagonistic MR actuators and validate our findings through experimental results. Our research demonstrates that these actuators are well-suited for safe operations in robotic applications, eliminating the need for additional sensor measurements or complex control strategies. This promising capability enables the avoidance of trade-offs between actuator performance, complexity, and cost. The insights gained from this study contribute to advancing compliant actuation technology, paving the way for high-performance and human-safe robotic systems.

Hi-ROS: Open-source multi-camera sensor fusion for real-time people tracking

This paper presents Hi-ROS (Human Interaction in ROS), an open source framework focused on real-time accurate assessment of human motion. The system offers a series of tools to track multiple people in real-time by exploiting a calibrated camera network. No assumptions are made about the typology or number of cameras, nor about the body pose estimation algorithm used to extract the 3D poses of the people in the scene. The tools provided by Hi-ROS include a Skeleton Tracker to ensure temporal consistency of the detected poses, a Skeleton Merger to fuse the tracks from multiple cameras, thus limiting flickering phenomena, a Skeleton Optimizer to ensure limb length consistency, and a Skeleton Filter to perform real-time smoothing of the detected joint trajectories. Accuracy, tracking robustness, and real-time performance of the proposed system were evaluated on a public dataset, containing both single-person and multi-person sequences with up to 4 people interacting. The results obtained using different subsets of the proposed tools show how the complete Hi-ROS pipeline provides accurate and reliable estimates also in challenging scenarios, with a reduction of the RMSE of up to 27% with respect to a pure tracking approach. This work aims to push forward the development of unobtrusive human–robot interaction applications, multi-person automated posture analyses, rehabilitation performance assessments, and any possible application enabled by real-time accurate assessment of human motion via markerless motion capture.

Parameterization and Design of Telepresence Robot to Avoid Obstacles

Background: The development of telepresence robots is getting much attention in various areas of human–robot interaction, healthcare systems and military applications because of multiple advantages such as safety improvement, lower energy and fuel consumption, exploitation of road networks, reduced traffic congestion and greater mobility. Methods: In the critical decision-making process during the motion of a robot, intelligent motion planning takes an important and challenging role. It includes obstacle avoidance, searching for the safest path to follow, generating appropriate behavior and comfortable trajectory generation by optimization while keeping road boundaries and traffic rules as important concerns. Results: This paper presents a state machine algorithm for avoiding obstacles and speed control design to a cognitive architecture named auto-MERLIN. This research empirically tested the proposed solutions by providing implementation details and diagrams for establishing the path planning and obstacle tests. Conclusions: The results validate the usability of our approach and show auto-MERLIN as a ready robot for short- and long-term tasks, showing better results than using a default system, particularly when deployed in highly interactive scenarios. The stable speed control of the auto-MERLIN in case of detecting any obstacle was shown.

SkinCell: A Modular Tactile Sensor Patch for Physical Human–Robot Interaction

In this article, we present the design, fabrication, integration, and performance evaluation of SkinCell sensors, a novel tactile array targeting human–robot interaction applications. A polyimide sheet is selected as the substrate of the sensor, and gold electrodes with circular interdigitated finger patterns are deposited through dc sputtering, creating a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula> 4 sensor array with 16 tactile elements. Sensing elements are completed with a thin layer of spin-coated poly (3,4-ethylenedioxythiophene) polystyrene sulfonate (PEDOT: PSS) organic polymer solution. Two individual sensor sheets are being laminated in a back-to-back fashion to create a half-Wheatstone-bridge configuration to reduce temperature coupling while improving sensitivity. The sensor array is then sandwiched between two layers of silicone elastomer with internal dimples and cavities to improve sensitivity, repeatability, and center of force detection resolution. A COMSOL simulation was performed to study the influence of adding the silicone encapsulation as well as design parameters to the sensor substrate. Our simulation results indicate an effective improvement of touch, i.e., voltage sensitivity by employing the featured encapsulation. Characterization experiments have been conducted to identify the sensitivity profile of each sensor on the array, which indicates an average sensitivity of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$26.5~\mu $ </tex-math></inline-formula> V/N. In subsequent experiments, the calibration profiles are used to identify the resultant force during pressing, as well as the location of the center of pressure (COP) on the SkinCell sensor. Finally, SkinCell sensors have been integrated into the OctoCan, a structural electronic device that can acquire squeezing pressure data during physical human–robot interaction (pHRI).

Integrating Virtual, Mixed, and Augmented Reality to Human–Robot Interaction Applications Using Game Engines: A Brief Review of Accessible Software Tools and Frameworks

This article identifies and summarizes software tools and frameworks proposed in the Human–Robot Interaction (HRI) literature for developing extended reality (XR) experiences using game engines. This review includes primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can control or interact with real robotic platforms using devices that extend the user’s reality. The objective of this article is not to present an extensive list of applications and tools. Instead, we present recent, relevant, common, and accessible frameworks and software tools implemented in research articles published in high-impact robotics conferences and journals. For this, we searched papers published during a seven-years period between 2015 and 2022 in relevant databases for robotics (Science Direct, IEEE Xplore, ACM digital library, Springer Link, and Web of Science). Additionally, we present and classify the application context of the reviewed articles in four groups: social robotics, programming of industrial robots, teleoperation of industrial robots, and Human–Robot collaboration (HRC).

Dataset with Tactile and Kinesthetic Information from a Human Forearm and Its Application to Deep Learning.

There are physical Human-Robot Interaction (pHRI) applications where the robot has to grab the human body, such as rescue or assistive robotics. Being able to precisely estimate the grasping location when grabbing a human limb is crucial to perform a safe manipulation of the human. Computer vision methods provide pre-grasp information with strong constraints imposed by the field environments. Force-based compliant control, after grasping, limits the amount of applied strength. On the other hand, valuable tactile and proprioceptive information can be obtained from the pHRI gripper, which can be used to better know the features of the human and the contact state between the human and the robot. This paper presents a novel dataset of tactile and kinesthetic data obtained from a robot gripper that grabs a human forearm. The dataset is collected with a three-fingered gripper with two underactuated fingers and a fixed finger with a high-resolution tactile sensor. A palpation procedure is performed to record the shape of the forearm and to recognize the bones and muscles in different sections. Moreover, an application for the use of the database is included. In particular, a fusion approach is used to estimate the actual grasped forearm section using both kinesthetic and tactile information on a regression deep-learning neural network. First, tactile and kinesthetic data are trained separately with Long Short-Term Memory (LSTM) neural networks, considering the data are sequential. Then, the outputs are fed to a Fusion neural network to enhance the estimation. The experiments conducted show good results in training both sources separately, with superior performance when the fusion approach is considered.

Real-Time Short-Term Pedestrian Trajectory Prediction Based on Gait Biomechanics.

The short-term prediction of a person’s trajectory during normal walking becomes necessary in many environments shared by humans and robots. Physics-based approaches based on Newton’s laws of motion seem best suited for short-term predictions, but the intrinsic properties of human walking conflict with the foundations of the basic kinematical models compromising their performance. In this paper, we propose a short-time prediction method based on gait biomechanics for real-time applications. This method relays on a single biomechanical variable, and it has a low computational burden, turning it into a feasible solution to implement in low-cost portable devices. We evaluate its performance from an experimental benchmark where several subjects walked steadily over straight and curved paths. With this approach, the results indicate a performance good enough to be applicable to a wide range of human–robot interaction applications.

Face mediated human–robot interaction for remote medical examination

Realtime visual feedback from consequences of actions is useful for future safety-critical human–robot interaction applications such as remote physical examination of patients. Given multiple formats to present visual feedback, using face as feedback for mediating human–robot interaction in remote examination remains understudied. Here we describe a face mediated human–robot interaction approach for remote palpation. It builds upon a robodoctor–robopatient platform where user can palpate on the robopatient to remotely control the robodoctor to diagnose a patient. A tactile sensor array mounted on the end effector of the robodoctor measures the haptic response of the patient under diagnosis and transfers it to the robopatient to render pain facial expressions in response to palpation forces. We compare this approach against a direct presentation of tactile sensor data in a visual tactile map. As feedback, the former has the advantage of recruiting advanced human capabilities to decode expressions on a human face whereas the later has the advantage of being able to present details such as intensity and spatial information of palpation. In a user study, we compare these two approaches in a teleoperated palpation task to find the hard nodule embedded in the remote abdominal phantom. We show that the face mediated human–robot interaction approach leads to statistically significant improvements in localizing the hard nodule without compromising the nodule position estimation time. We highlight the inherent power of facial expressions as communicative signals to enhance the utility and effectiveness of human–robot interaction in remote medical examinations.

Design, modeling, and control of a Series Elastic Actuator with Discretely Adjustable Stiffness (SEADAS)

The recent application of the principle of discrete adjustment of stiffness has substantially enhanced the performance of Clutched Elastic Actuators (CEA), especially in terms of their capability for rapid stiffness switching. This key improvement plays a vital role in safety considerations, which are of particular relevance to physical Human–Robot Interaction (pHRI) applications. However, currently the only proposed CEA for this purpose suffers from bulkiness, which limits its potential application in multi-degree-of-freedom manipulators. In this work, we introduce a Series Elastic Actuator with Discretely Adjustable Stiffness (SEADAS) as a novel CEA, in which the stiffness can be modulated by adding/subtracting the appropriate number of elastic elements via clutches. The novelty lies in the unique design topology, where three concentric shafts are connected in series through clutches. Each shaft is further attached to a pair of elastic elements. The number of involved spring pairs, which depends on the engagement/disengagement of the clutches, allows for the attainment of three different levels of stiffness. The realized design is significantly compact relative to the previously proposed CEA that was based on this principle. This paper details the working principle, design and development of the actuator, stiffness and dynamic modeling, as well as a gain-scheduled PID controller used for accurate, stiffness-adaptive joint position control.

Stiffness Modulation of a Cable-Driven Serial-Chain Manipulator Via Cable Routing Alteration

Abstract Cable-driven serial-chain manipulators, CDSMs, are widely used from industries to human–robot interaction applications demanding diverse performance requirements. The CDSMs inherent characteristic of flexibility in altering system parameters facilitates the possibility of varying its performance as per the desired application and thus has been explored in the literature. Among the various performance measures, stiffness plays a vital role in manipulators interaction with unknown environments. Works in the literature reported varying CDSMs system parameters to tune the stiffness characteristics and highlighted the tensionable workspace’s sensitivity to the changes in the system parameters. The current work demonstrates the potential of co-shared cable routing in CDSMs along with a set of design conditions to provide a broader range of stiffness characteristics for a constant tensionable workspace. The results are presented using a planar two degrees-of-freedom (DOFs) CDSM, and the stiffness changes are validated experimentally. This outcome shows that CDSM with co-shared cable routing can render a wide range of stiffness behavior.

An intelligent facial expression recognition system with emotion intensity classification

Facial expressions play a crucial role in emotion recognition as compared to other modalities. In this work, an integrated network, which is capable of recognizing emotion intensity levels from facial images in real time using deep learning technique is proposed. The cognitive study of facial expressions based on expression intensity levels are useful in applications such as healthcare, coboting, Industry 4.0 etc. This work proposes to augment emotion recognition with 2 other important parameters, valence and emotion intensity. This helps in better automated responses by a machine to an emotion. The valence model helps in classifying emotion as positive and negative emotions and discrete model classifies emotions as happy, anger, disgust, surprise and neutral state using Convolution Neural Network (CNN). Feature extraction and classification are carried out using CMU Multi-PIE database. The proposed architecture achieves 99.1% and 99.11% accuracy for valence model and discrete model respectively for offline image data with 5-fold cross validation. The average accuracy achieved in real time for valance model and discrete model is 95% & 95.6% respectively. Also, this work contributes to build a new database using facial landmarks, with three intensity levels of facial expressions which helps to classify expressions into low, mild and high intensities. The performance is also tested for different classifiers. The proposed integrated system is configured for real time Human Robot Interaction (HRI) applications on a test bed consisting of Raspberry Pi and RPA platform to assess its performance.

A Smart Capacitive Sensor Skin with Embedded Data Quality Indication for Enhanced Safety in Human-Robot Interaction.

Smart sensors are an integral part of the Fourth Industrial Revolution and are widely used to add safety measures to human–robot interaction applications. With the advancement of machine learning methods in resource-constrained environments, smart sensor systems have become increasingly powerful. As more data-driven approaches are deployed on the sensors, it is of growing importance to monitor data quality at all times of system operation. We introduce a smart capacitive sensor system with an embedded data quality monitoring algorithm to enhance the safety of human–robot interaction scenarios. The smart capacitive skin sensor is capable of detecting the distance and angle of objects nearby by utilizing consumer-grade sensor electronics. To further acknowledge the safety aspect of the sensor, a dedicated layer to monitor data quality in real-time is added to the embedded software of the sensor. Two learning algorithms are used to implement the sensor functionality: (1) a fully connected neural network to infer the position and angle of objects nearby and (2) a one-class SVM to account for the data quality assessment based on out-of-distribution detection. We show that the sensor performs well under normal operating conditions within a range of 200 and also detects abnormal operating conditions in terms of poor data quality successfully. A mean absolute distance error of was achieved without data quality indication. The overall performance of the sensor system could be further improved to by monitoring the data quality, adding an additional layer of safety for human–robot interaction.

Skeleton Tracking Accuracy and Precision Evaluation of Kinect V1, Kinect V2, and the Azure Kinect

The Azure Kinect, the successor of Kinect v1 and Kinect v2, is a depth sensor. In this paper we evaluate the skeleton tracking abilities of the new sensor, namely accuracy and precision (repeatability). Firstly, we state the technical features of all three sensors, since we want to put the new Azure Kinect in the context of its previous versions. Then, we present the experimental results of general accuracy and precision obtained by measuring a plate mounted to a robotic manipulator end effector which was moved along the depth axis of each sensor and compare them. In the second experiment, we mounted a human-sized figurine to the end effector and placed it in the same positions as the test plate. Positions were located 400 mm from each other. In each position, we measured relative accuracy and precision (repeatability) of the detected figurine body joints. We compared the results and concluded that the Azure Kinect surpasses its discontinued predecessors, both in accuracy and precision. It is a suitable sensor for human–robot interaction, body-motion analysis, and other gesture-based applications. Our analysis serves as a pilot study for future HMI (human–machine interaction) designs and applications using the new Kinect Azure and puts it in the context of its successful predecessors.

A Bayesian Deep Neural Network for Safe Visual Servoing in Human-Robot Interaction.

Safety is an important issue in human–robot interaction (HRI) applications. Various research works have focused on different levels of safety in HRI. If a human/obstacle is detected, a repulsive action can be taken to avoid the collision. Common repulsive actions include distance methods, potential field methods, and safety field methods. Approaches based on machine learning are less explored regarding the selection of the repulsive action. Few research works focus on the uncertainty of the data-based approaches and consider the efficiency of the executing task during collision avoidance. In this study, we describe a system that can avoid collision with human hands while the robot is executing an image-based visual servoing (IBVS) task. We use Monte Carlo dropout (MC dropout) to transform a deep neural network (DNN) to a Bayesian DNN, and learn the repulsive position for hand avoidance. The Bayesian DNN allows IBVS to converge faster than the opposite repulsive pose. Furthermore, it allows the robot to avoid undesired poses that the DNN cannot avoid. The experimental results show that Bayesian DNN has adequate accuracy and can generalize well on unseen data. The predictive interval coverage probability (PICP) of the predictions along x, y, and z directions are 0.84, 0.94, and 0.95, respectively. In the space which is unseen in the training data, the Bayesian DNN is also more robust than a DNN. We further implement the system on a UR10 robot, and test the robustness of the Bayesian DNN and the IBVS convergence speed. Results show that the Bayesian DNN can avoid the poses out of the reach range of the robot and it lets the IBVS task converge faster than the opposite repulsive pose. 1

Robust Sliding Mode Control for Robots Driven by Compliant Actuators

Compliant actuators can offer many attractive features over stiff actuators in real human–robot interaction applications, such as low output impedance, smooth force transmission, and shock tolerance. This brief focuses on a robust sliding mode control (SMC) methodology for robotic systems with compliant actuators. First, a continuous SMC design is introduced due to its advantages of strong robustness and chattering attenuation. However, this continuous SMC structure cannot guarantee a high tracking performance in the presence of mismatched disturbances in compliantly actuated robots. Meanwhile, in many application fields, compliantly actuated robots are affected by different kinds of time-varying disturbances, including external environmental disturbances, internal parameter uncertainties, and frictions, which may be in the form of constant, ramp, and parabolic disturbances. To estimate such unknown disturbances, a generalized proportional integral observer (GPIO) technique is employed. By designing a new sliding surface with the help of disturbance estimation, a GPIO-based continuous SMC method is synthesized, which is used to deal with matched/mismatched time-varying disturbances. A detailed stability analysis of the closed-loop system is also presented. Experimental results under three different test conditions are provided to illustrate the promising tracking performance of the proposed control strategy.

A real-time spike-timing classifier of spatio-temporal patterns

Abstract Considering the problem of recognizing non-verbal cues in Human–Robot Interaction applications, this paper proposes a novel real-time unsupervised spike timing neural network for recognition and early detection of spatio-temporal human gestures. Two spiking network classifiers one based on Izhikevich neuron model, and the other one based on Integrate-and-Fire-or-Burst neuron model have been implemented in CUDA, and allow the classification to be performed in real-time. To evaluate the performance of this proposal, we test the case of a physical robot observing air-handwritings of human gesture. The proposed approaches run in real-time, thus they are suitable for human–robot applications; they allow real-time early classifying human gestures and actions while they require a very small number of training samples. In comparing to other prominent techniques, our approaches demonstrate superior accuracy and are suitable for early classification of different types of human actions in time-sensitive mobile applications such as robotics.

Skeleton-based bio-inspired human activity prediction for real-time human–robot interaction

Activity prediction is an essential task in practical human-centered robotics applications, such as security, assisted living, etc., which is targeted at inferring ongoing human activities based on incomplete observations. To address this challenging problem, we introduce a novel bio-inspired predictive orientation decomposition (BIPOD) approach to construct representations of people from 3D skeleton trajectories. BIPOD is invariant to scales and viewpoints, runs in real-time on basic computer systems, and is able to recognize and predict activities in an online fashion. Our approach is inspired by biological research in human anatomy. To capture spatio-temporal information of human motions, we spatially decompose 3D human skeleton trajectories and project them onto three anatomical planes (i.e., coronal, transverse and sagittal planes); then, we describe short-term time information of joint motions and encode high-order temporal dependencies. By using Extended Kalman Filters to estimate future skeleton trajectories, we endow our BIPOD representation with the critical capabilities to reduce noisy skeleton observation data and predict the ongoing activities. Experiments on benchmark datasets have shown that our BIPOD representation significantly outperforms previous methods for real-time human activity classification and prediction from 3D skeleton trajectories. Empirical studies using TurtleBot2 and Baxter humanoid robots have also validated that our BIPOD method obtains promising performance, in terms of both accuracy and efficiency, making BIPOD a fast, simple, yet powerful representation for low-latency online activity prediction in human–robot interaction applications.

On human–robot interaction of a 3-DOF decoupled parallel mechanism based on the design and construction of a novel and low-cost 3-DOF force sensor

This paper addresses the extension of a 3-degrees-of-freedom (3-DOF) decoupled parallel mechanism for human–robot interaction purposes. To this end, a low-cost 3-DOF force sensor for human–robot interaction applications is proposed, designed and constructed. In the latter force sensor, five load cells are placed in order to identify the amount of the applied force along each Cartesian direction. In addition, an experimental identification procedure based on least square method is carried out in order to obtain the first and third degree polynomial models of the sensor output model. From the practical tests it has been reveled that the force sensor has a reasonable precision of 0.1 N in both x and y-axes and 0.2 N in z-axis, within a range of 5 N which is suitable for human–robot interaction applications. Then, using the proposed force sensor, two control methods, namely “position control” and “speed control” are applied for human–robot interaction purposes and their performances are compared.

Affective Haptics: Current Research and Future Directions

Touch plays a prominent role in communicating emotions and intensifying interpersonal communication. Affective haptics is an emerging field, which focuses on the analysis, design, and evaluation of systems that can capture, process, or display emotions through the sense of touch. The objective of this paper is to present an overview of the recent achievements in affective haptics and to discuss how the sense of touch can elicit or influence human emotions. We first introduce a definition to the term affective haptics and describe its multidisciplinary nature—as a field that integrates ideas from affective computing, haptic technology, and user experience. Second, we provide a thorough discussion about the effectiveness of using the haptic channel to communicate affective information through direct and mediated means. Third, we present a variety of applications in the area ranging from interhuman social interaction systems to human robot interaction applications. Finally, we discuss some of the key findings discerned from the various surveyed papers, and present some of the challenges and trends in this field. We extract the following conclusions pertaining to affective haptics: 1) haptic stimulation can be successfully used to achieve a higher level of emotional immersion during media consumption or emotional telepresence; 2) existing research has demonstrated that haptics is effective in communicating valence and arousal, and the emotions of happiness, sadness, anger and fear, and less focus have been given to the communication of disgust and surprise; 3) the haptic-based affect detection remains an understudied topic, whereas the haptic-based affect display is a well-established subject; and 4) the interpretation of the haptic stimulation by human beings is highly contextual.