Human-robot Interaction Applications Research Articles

Empathetic Speech Synthesis and Testing for Healthcare Robots

One of the major factors that affect the acceptance of robots in Human-Robot Interaction applications is the type of voice with which they interact with humans. The robot’s voice can be used to express empathy, which is an affective response of the robot to the human user. In this study, the aim is to find out if social robots with empathetic voice are acceptable for users in healthcare applications. A pilot study using an empathetic voice spoken by a voice actor was conducted. Only prosody in speech is used to express empathy here, without any visual cues. Also, the emotions needed for an empathetic voice are identified. It was found that the emotions needed are not only the stronger primary emotions, but also the nuanced secondary emotions. These emotions are then synthesised using prosody modelling. A second study, replicating the pilot test is conducted using the synthesised voices to investigate if empathy is perceived from the synthetic voice as well. This paper reports the modelling and synthesises of an empathetic voice, and experimentally shows that people prefer empathetic voice for healthcare robots. The results can be further used to develop empathetic social robots, that can improve people’s acceptance of social robots.

Compact Gearboxes for Modern Robotics: A Review.

On the eve of Human-Robot-Interaction (HRI) becoming customary in our lives, the performance of HRI robotic devices remains strongly conditioned by their gearboxes. In most industrial robots, two relatively unconventional transmission technologies—Harmonic Drives© and Cycloid Drives—are usually found, which are not so broadly used in other industries. Understanding the origin of this singularity provides valuable insights in the search for suitable, future robotic transmission technologies. In this paper we propose an assessment framework strongly conditioned by HRI applications, and we use it to review the performance of conventional and emerging robotic gearbox technologies, for which the design criterion is strongly shifted toward aspects like weight and efficiency. The framework proposes to use virtual power as a suitable way to assess the inherent limitations of a gearbox technologies to achieve high efficiencies. This paper complements the existing research dealing with the complex interaction between gearbox technologies and the actuators, with a new gearbox-centered perspective particularly focused on HRI applications.

Stiffness modeling of a variable stiffness compliant link

Recently, a variable stiffness robotic link based on the rotating beam concept has been developed for applications in physical human robot interaction. A substantial challenge for design of such links is the modeling of stiffness behavior to permit stiffness control. In this paper we present a general 3D model of the link stiffness using screw theory and compliance matrices as well as a planar model for the lateral and torsional stiffness. Since axial buckling is a major failure mode, we also derive an analytical model for predicting axial buckling behavior. The analytical models are compared to the finite element method and experimental results. One of the challenges involved in design and analysis of variable stiffness links is the parasitic compliance of the mechanical elements that support and drive the active portion of the mechanism. For the design analyzed in this paper, we use the models we derive to identify the major sources of parasitic compliance and suggest optimizations to minimize their effects. These results can be used as guidelines for designing variable stiffness links.

Lio-A Personal Robot Assistant for Human-Robot Interaction and Care Applications.

Lio is a mobile robot platform with a multi-functional arm explicitly designed for human-robot interaction and personal care assistant tasks. The robot has already been deployed in several health care facilities, where it is functioning autonomously, assisting staff and patients on an everyday basis. Lio is intrinsically safe by having full coverage in soft artificial-leather material as well as collision detection, limited speed and forces. Furthermore, the robot has a compliant motion controller. A combination of visual, audio, laser, ultrasound and mechanical sensors are used for safe navigation and environment understanding. The ROS-enabled setup allows researchers to access raw sensor data as well as have direct control of the robot. The friendly appearance of Lio has resulted in the robot being well accepted by health care staff and patients. Fully autonomous operation is made possible by a flexible decision engine, autonomous navigation and automatic recharging. Combined with time-scheduled task triggers, this allows Lio to operate throughout the day, with a battery life of up to 8 hours and recharging during idle times. A combination of powerful computing units provides enough processing power to deploy artificial intelligence and deep learning-based solutions on-board the robot without the need to send any sensitive data to cloud services, guaranteeing compliance with privacy requirements. During the COVID-19 pandemic, Lio was rapidly adjusted to perform additional functionality like disinfection and remote elevated body temperature detection. It complies with ISO13482 - Safety requirements for personal care robots, meaning it can be directly tested and deployed in care facilities.

Enhanced torque estimation method from multi-channel surface electromyography compensating electrode location variation.

Recognizing human intentions from the human counterpart is very important in human-robot interaction applications. Surface electromyography(sEMG) has been considered as a potential source for motion intention because the signal represents the on-set timing and amplitude of muscle activation. It is also reported that sEMG has the advantage of knowing body movements ahead of actual movement. However, sEMG based applications suffer from electrode location variation because sEMG shows different characteristics whenever the skin condition is different. They need to recreate the estimation model if electrodes are attached to different locations or conditions. In this paper, we developed a sEMG torque estimation model for electrode location variation. A decomposition model of sEMG signals was developed to discriminate the muscle source signals for electrode location variation, and we verified this model without making a new torque estimation model. Torque estimation accuracy using the proposed method was increased by 24.8% and torque prediction accuracy was increased by 47.7% for the electrode location variation in comparison with the method without decomposition. Therefore, the proposed sEMG decomposition method showed an enhancement in torque estimation for electrode location variation.

3D-Aware Scene Change Captioning From Multiview Images

In this letter, we propose a framework that recognizes and describes changes that occur in a scene observed from multiple viewpoints in natural language text. The ability to recognize and describe changes that occurred in a 3D scene plays an essential role in a variety of human-robot interaction applications. However, most current 3D vision studies have focused on understanding the static 3D scene. Existing scene change captioning approaches recognize and generate change captions from single-view images. Those methods have limited ability to deal with camera movement, object occlusion, which are common in real-world settings. To resolve these problems, we propose a framework that observes every scene from multiple viewpoints and describes the scene change based on an understanding of the underlying 3D structure of scenes. We build three synthetic datasets consisting of primitive 3D object and scanned real object models for evaluation. The results indicate that our method outperforms the previous state-of-the-art 2D-based method by a large margin in terms of sentence generation and change understanding correctness. In addition, our method is more robust to camera movements compared to the previous method and also performs better for scenes with occlusions. Moreover, our method also shows encouraging results in a realistic scene-setting, which indicates the possibility of adapting our framework to a more complicated and extensive scene-settings.

Conceptual mechanical design of antagonistic variable stiffness joint based on equivalent quadratic torsion spring.

The variable stiffness joint is a kind of flexible actuator with variable stiffness characteristics suitable for physical human-robot interaction applications. In the existing variable stiffness joints, the antagonistic variable stiffness joint has the advantages of simple implementation of variable stiffness mechanism and easy modular design of the nonlinear elastic element. The variable stiffness characteristics of antagonistic variable stiffness joints are realized by the antagonistic actuation of two nonlinear springs. A novel design scheme of the equivalent nonlinear torsion spring with compact structure, large angular displacement range, and desired stiffness characteristics is presented in this article. The design calculation for the equivalent quadratic torsion spring is given as an example, and the actuation characteristics of the antagonistic variable stiffness joint based on the equivalent quadratic torsion spring are illustrated. Based on the design idea of constructing the antagonistic variable stiffness joint with compact structure and high compliance, as well as the different design requirements of the joints at different positions of the multi-degrees of freedom robot arm, nine types of mechanical schemes of antagonistic variable stiffness joint with the open design concept are proposed in this article. Finally, the conceptual joint configuration schemes of the robot arm based on the antagonistic variable stiffness joint show the application scheme of the designed antagonistic variable stiffness joint in the multi-degrees of freedom robot.

Intention-Related Natural Language Grounding via Object Affordance Detection and Intention Semantic Extraction.

Similar to specific natural language instructions, intention-related natural language queries also play an essential role in our daily life communication. Inspired by the psychology term “affordance” and its applications in Human-Robot interaction, we propose an object affordance-based natural language visual grounding architecture to ground intention-related natural language queries. Formally, we first present an attention-based multi-visual features fusion network to detect object affordances from RGB images. While fusing deep visual features extracted from a pre-trained CNN model with deep texture features encoded by a deep texture encoding network, the presented object affordance detection network takes into account the interaction of the multi-visual features, and reserves the complementary nature of the different features by integrating attention weights learned from sparse representations of the multi-visual features. We train and validate the attention-based object affordance recognition network on a self-built dataset in which a large number of images originate from MSCOCO and ImageNet. Moreover, we introduce an intention semantic extraction module to extract intention semantics from intention-related natural language queries. Finally, we ground intention-related natural language queries by integrating the detected object affordances with the extracted intention semantics. We conduct extensive experiments to validate the performance of the object affordance detection network and the intention-related natural language queries grounding architecture.

산업용 상지 웨어러블 로봇에 대한 정성적 사용성 평가 가이드라인 연구

Objective: This study is to identify the components and qualitative criteria for assessing the usability of wearable robots in industrial use. Background: Wearable robot technology is a state-of-the-art application of humanrobot interaction (HRI) but research on its usability from the users point of view is insufficient. It is necessary for the wearable robot industry to establish safety standards and usability assessment system for deploying the technology on a commercial scale. Method: Sets of user requirements were identified from previous researches regarding with analogous products including walking aids, assistive and wearable devices. Qualitative assessment criteria were developed on the basis of user requirements classified by usability components and the structure of wearable robots. Results: The usability of wearable robots needs to be evaluated by 6 components: wearability, effectiveness, learnability, efficiency, utility, and safety. Qualitative assessment criteria consisted of 38 questionnaires corresponding to the identified user requirements that belong to each component. Conclusion: The 6 components with 38 assessment criteria would be utilized as the basis for further development of the usability assessment standard. Application The assessment criteria provide ideas and checklists for designing wearable robots with good usability.

Multi-View Visual Question Answering with Active Viewpoint Selection.

This paper proposes a framework that allows the observation of a scene iteratively to answer a given question about the scene. Conventional visual question answering (VQA) methods are designed to answer given questions based on single-view images. However, in real-world applications, such as human–robot interaction (HRI), in which camera angles and occluded scenes must be considered, answering questions based on single-view images might be difficult. Since HRI applications make it possible to observe a scene from multiple viewpoints, it is reasonable to discuss the VQA task in multi-view settings. In addition, because it is usually challenging to observe a scene from arbitrary viewpoints, we designed a framework that allows the observation of a scene actively until the necessary scene information to answer a given question is obtained. The proposed framework achieves comparable performance to a state-of-the-art method in question answering and simultaneously decreases the number of required observation viewpoints by a significant margin. Additionally, we found our framework plausibly learned to choose better viewpoints for answering questions, lowering the required number of camera movements. Moreover, we built a multi-view VQA dataset based on real images. The proposed framework shows high accuracy (94.01%) for the unseen real image dataset.

Arbitrary-View Human Action Recognition: A Varying-View RGB-D Action Dataset

Current researches of action recognition which focus on single-view and multi-view recognition can hardly satisfy the requirements of human-robot interaction (HRI) applications for recognizing human actions from arbitrary views. Arbitrary-view recognition is still a challenging issue due to view changes and visual occlusions. In addition, the lack of datasets also sets up barriers. To provide data for arbitrary-view action recognition, we collect a new large-scale RGB-D action dataset for arbitrary-view action analysis, including RGB videos, depth and skeleton sequences. The dataset includes action samples captured in 8 fixed viewpoints and varying-view sequences which cover the entire 360° view angles. In total, 118 persons are invited to act 40 action categories. Our dataset involves more participants, more viewpoints and a large number of samples. More importantly, it is the first dataset containing the entire 360° varying-view sequences. The dataset provides sufficient data for multi-view, cross-view and arbitrary-view action analysis. Besides, we propose a View-guided Skeleton CNN (VS-CNN) to tackle the problem of arbitrary-view action recognition. Experiment results show that the VS-CNN achieves superior performance, and our dataset provides valuable but challenging data for the evaluation of arbitrary-view recognition.

Design & characterization of a bio-inspired 3-DOF Tactile/Force sensor and implementation on a 3-DOF decoupled parallel mechanism for human-robot interaction purposes

This paper presents the design and implementation of a Tactile/Force sensor which has been used on a 3-DOF decoupled parallel mechanism for Human-Robot Interaction purposes.The sensor, called HexaTactile, is a soft tactile sensor array based on six MEMS barometers, where each of them is covered by a silicone layer in the form of an incomplete pyramid. HexaTactile consists of six soft and highly sensitive tactile modules which are placed on six sides of a cube to allow simultaneous measurement of the force in the positive and negative directions along the x, y and z axes. Some of the advantages of this sensor can be regarded as its high precision, excellent linearity (coefficient of determination r2=0.99), low cost and low noise. The accuracy of the sensor is 0.01 N, within a range of 4 N and therefore HexaTactile can be suitably attached to a robot end-effector for human-robot interaction applications. Then, using the proposed force sensor some control scenarios, including fixed admittance control and active admittance control are applied for human-robot interaction purposes. From the experimental tests, it has been revealed that the active admittance control solved the drawbacks of fixed admittance control, for the considered case studies.

Cost-Efficient Flexible Supercapacitive Tactile Sensor With Superior Sensitivity and High Spatial Resolution for Human-Robot Interaction

Tactile sensing is crucial for the safety, accuracy and robustness of the human-robot interactions in the fields of wearable equipment, service robots and healthcare robots. Although many efforts have been made, it still requires much work to develop functional and reliable flexible tactile sensors with superior sensitivity, wide measurement range, high spatial resolution and low cost based on simple structures and easy fabrication. Here, this paper introduces a flexible supercapacitive tactile sensor with outstanding and balanced performance. The tactile sensor contains two layers of flexible electrodes and a layer of ionic-gel coated microfiber matrix to form a supercapacitive sensing structure. The flexible electrodes and the ionic microfiber matrix are processed with scalable techniques such as screen printing and gel-coating, which guarantee the ultra-flexibility and low fabrication cost. The experimental data suggests strong linearity in the pressure-capacitance relationship and high sensitivity (135.9 nF $\cdot $ kPa $^{-1}\cdot $ cm2 or 27.11 kPa−1). Wide pressure measurement range (from 0 kPa to 1200 kPa) is also achieved by balancing structure parameters. Dynamic responses of the tactile sensors could accurately reflect the applied pressure cycles from human-finger tapping and machine pressing. The tactile sensor can map the pressure distribution with a high spatial resolution (>2 points $\cdot $ mm−2) when connected with the specially designed electric circuitry. The spatial resolution from sub-mm to large area makes it promising for various sensing applications in human-robot interactions, from finger touch to body contact. The developed tactile sensor in this study owns superior applicability and universality which makes it a trustworthy candidate to benefit various applications in robotics, flexible electronics and bioengineered equipment.

A Method to Extract Motion Velocities of Limb and Trunk in Human Locomotion

Human motion monitoring is important in applications of movies, animation, sport training, physical rehabilitation, and human-robot interaction. There is a demand for a simple and easy-to-use method to recognize motions of limbs and trunk. In this paper, we developed a wearable velocity tracking device using two orthogonally-placed micro flow sensors to implement three-dimensional motion velocity measurement. In addition, we proposed a functional link artificial neural network (FLANN) model to extract trunk velocity and relative limb velocity from an absolute limb motion detected by the wearable tracking device according to their different dynamic features. Experiments were conducted to validate the effectiveness of the velocity tracking methodology. Results showed the proposed method with wearable device enables real-time measurements of motion velocities of limb and trunk, which were free of accumulated error, robust for dynamic walking and running, and simple to use.

Robust Tracking Control of Variable Stiffness Joint Based on Feedback Linearization and Disturbance Observer With Estimation Error Compensation

The variable stiffness joint (VSJ) has the characteristics of independent and controllable position and stiffness. The variable stiffness characteristics and inherent flexibility make the VSJ suitable to be used as the actuation joint of the physical human-robot interaction application robot, so as to improve the task adaptability of the robot and physical human-robot interaction safety. The VSJ based on equivalent lever mechanism has the advantages of low energy consumption in stiffness adjustment, so there are many researches on this type of VSJ. The tracking control of output link angular position and joint output stiffness are two basic control targets of the VSJ. For the system dynamic model of the VSJ based on equivalent lever mechanism, considering the unknown parametric perturbations, the unknown friction torques acting on the drive units, the unknown external disturbance acting on the output link and the control input saturation constraints, a robust tracking controller based on feedback linearization, disturbance observer with anti-windup measures, sliding mode control and estimation error compensator is designed to improve the tracking control accuracy of the position and stiffness of the VSJ. The simultaneous tracking control of position and stiffness of the VSJ can be achieved by the designed controller, and the simulation results show the effectiveness and robustness of the proposed controller. Moreover, the simulation results show that the proposed estimation error compensator for the disturbance observer with fixed preset observation gain can effectively reduce the system output tracking error and improve the anti-disturbance characteristics of the controller.

MakeSense: Automated Sensor Design for Proprioceptive Soft Robots.

Soft robots have applications in safe human-robot interactions, manipulation of fragile objects, and locomotion in challenging and unstructured environments. In this article, we present a computational method for augmenting soft robots with proprioceptive sensing capabilities. Our method automatically computes a minimal stretch-receptive sensor network to user-provided soft robotic designs, which is optimized to perform well under a set of user-specified deformation-force pairs. The sensorized robots are able to reconstruct their full deformation state, under interaction forces. We cast our sensor design as a subselection problem, selecting a minimal set of sensors from a large set of fabricable ones, which minimizes the error when sensing specified deformation-force pairs. Unique to our approach is the use of an analytical gradient of our reconstruction performance measure with respect to selection variables. We demonstrate our technique on a bending bar and gripper example, illustrating more complex designs with a simulated tentacle.

Optimization of Real-Time EEG Artifact Removal and Emotion Estimation for Human-Robot Interaction Applications.

Affective human-robot interaction requires lightweight software and cheap wearable devices that could further this field. However, the estimation of emotions in real-time poses a problem that has not yet been optimized. An optimization is proposed for the emotion estimation methodology including artifact removal, feature extraction, feature smoothing, and brain pattern classification. The challenge of filtering artifacts and extracting features, while reducing processing time and maintaining high accuracy results, is attempted in this work. First, two different approaches for real-time electro-oculographic artifact removal techniques are tested and compared in terms of loss of information and processing time. Second, an emotion estimation methodology is proposed based on a set of stable and meaningful features, a carefully chosen set of electrodes, and the smoothing of the feature space. The methodology has proved to perform on real-time constraints while maintaining high accuracy on emotion estimation on the SEED database, both under subject dependent and subject independent paradigms, to test the methodology on a discrete emotional model with three affective states.

Human–Robot Interaction Control Based on a General Energy Shaping Method

Coexistence of the human and robot in the same workspace requires the robot to perform robot tasks, such as trajectory tracking, and also interaction tasks, such as keeping a safe distance with the human. According to various human–robot interaction (HRI) scenarios, different interaction tasks usually require different task requirements or specifications, leading to different control strategies. Besides, due to different natures of the robot tasks and interaction tasks, different controllers may be required when the task is switched from one to another. So far, there is no theoretical framework that integrates different robot and interaction task requirements into a unified robot control strategy. In this article, a general HRI control framework is proposed for the scenario of the human and robot coexisting in the same workspace. We propose a general potential energy function that can be used to derive a stable and unified controller for various robot tasks and HRI tasks. By using the proposed method, various tasks can be specified at a user level by simply adjusting certain task parameters. Interactive weights are also defined to specify the interaction behaviors of robots according to different HRI applications. A human-dominant interaction mode is presented to demonstrate the applications of the proposed method. We show how the control framework can be applied to existing robot control systems with the velocity control or torque control mode by developing a joint velocity reference command and an adaptive controller. The stability of the adaptive control system is shown by using Lyapunov-like analysis. The control methods are implemented on two industrial manipulators and experimental results are presented to illustrate the effectiveness of the proposed control framework.

Video Description

Video description is the automatic generation of natural language sentences that describe the contents of a given video. It has applications in human-robot interaction, helping the visually impaired and video subtitling. The past few years have seen a surge of research in this area due to the unprecedented success of deep learning in computer vision and natural language processing. Numerous methods, datasets, and evaluation metrics have been proposed in the literature, calling the need for a comprehensive survey to focus research efforts in this flourishing new direction. This article fills the gap by surveying the state-of-the-art approaches with a focus on deep learning models; comparing benchmark datasets in terms of their domains, number of classes, and repository size; and identifying the pros and cons of various evaluation metrics, such as SPICE, CIDEr, ROUGE, BLEU, METEOR, and WMD. Classical video description approaches combined subject, object, and verb detection with template-based language models to generate sentences. However, the release of large datasets revealed that these methods cannot cope with the diversity in unconstrained open domain videos. Classical approaches were followed by a very short era of statistical methods that were soon replaced with deep learning, the current state-of-the-art in video description. Our survey shows that despite the fast-paced developments, video description research is still in its infancy due to the following reasons: Analysis of video description models is challenging, because it is difficult to ascertain the contributions towards accuracy or errors of the visual features and the adopted language model in the final description. Existing datasets neither contain adequate visual diversity nor complexity of linguistic structures. Finally, current evaluation metrics fall short of measuring the agreement between machine-generated descriptions with that of humans. We conclude our survey by listing promising future research directions.

A Research on Application of Human-Robot Interaction using Artifical Intelligence

This work presents an online robot instructing with common human – robot connection. Normal human-PC association is a critical interface to acknowledge benevolent joint effort of canny robot and human. The greater part of the correspondence between people is done through discourse and signal, and the connection among discourse and motion is regular and natural. Robot educating by methods for discourse acknowledgment [12] is another route for instructing and playback, which utilizes the common discernment channels of individuals. This paper centers around a training technique dependent on the common human-PC communication. The task is to teach the Robot to compose by giving three unique information sources like Voice order, Camera based Video info or utilizing MEMS equipment interface utilizing Zigbee. Voice direction can be perceived utilizing Android application. Motion will perceived utilizing framework camera, utilizing PCA calculation framework will order the pictures. MEMS sensor is wired equipment hardware which will contain the quantity of likelihood blend to order the robot. Motions are segregated by applying a most extreme data model, with highlights separated utilizing principal component analysis (PCA). The proposed interface could be stretched out to the genuine modern scene. By utilizing signal and discourse, administrators can control the robot without complex tasks. The outcomes show that the online robot instructing framework can effectively show robot controllers