Human-robot Interaction System Research Articles

On the perception and handling of deformable objects – A robotic cell for white goods industry

• Design and implementation of a robotic system for the manipulation and assembly of linear non-rigid components. • Multi-sensor system for flexible part cognition and safe human-robot interaction. • Dexterous gripper for non-rigid material manipulation. • Control framework for process coordination and safe human -robot collaboration. • Validation and testing in a white goods industry case study. Advancements in mechatronics, perception and computational methods allowed robotics to focus on non-rigid part manipulation, which is characterized by high complexity due to the unpredictable and compliant behavior of flexible materials. This paper presents the automation of a manufacturing process involving linear non-rigid components through the implementation of a robotic system that addresses challenges related to flexible material perception and handling. The corresponding technological approach consists of five steps namely safe material supply, detection, grasping, cross section recognition, and assembly. The proposed robotic cell comprises: (1) a multi-sensor system for flexible part recognition and safe human-robot interaction (2) a collaborative robot equipped with a dexterous gripper for flexible material manipulation, and (3) a control framework for process coordination. The robotic system has been tested on a case study originating from the white goods industry, where rubber gaskets are manipulated and assembled on household appliances. Experimental results proved that the implemented robotic system can tackle the challenges of flexible material manipulation with great consistency in the aspects of robustness and repeatability.

Gender detection in children’s speech utterances for human-robot interaction

The human voice speech essentially includes paralinguistic information used in many real-time applications. Detecting the children’s gender is considered a challenging task compared to the adult’s gender. In this study, a system for human-robot interaction (HRI) is proposed to detect the gender in children’s speech utterances without depending on the text. The robot's perception includes three phases: Feature’s extraction phase where four formants are measured at each glottal pulse and then a median is calculated across these measurements. After that, three types of features are measured which are formant average (AF), formant dispersion (DF), and formant position (PF). Feature’s standardization phase where the measured feature dimensions are standardized using the z-score method. The semantic understanding phase is where the children’s gender is detected accurately using the logistic regression classifier. At the same time, the action of the robot is specified via a speech response using the text to speech (TTS) technique. Experiments are conducted on the Carnegie Mellon University (CMU) Kids dataset to measure the suggested system’s performance. In the suggested system, the overall accuracy is 98%. The results show a relatively clear improvement in terms of accuracy of up to 13% compared to related works that utilized the CMU Kids dataset.

Interaction Task Motion Learning for Human–Robot Interaction Control

Traditional applications of robot manipulators are mainly limited to tracking control tasks, in which the desired objectives are specified as desired positions or trajectories. In such applications, a convenient and easy way of programming robots is the traditional teach-and-playback method. However, advances in sensing and robotic technologies have led to the requirements of more demanding tasks, in which robots may need to interact with a human or follow the human’s instructions in performing a sequence of more complex tasks. In such applications, it is not sufficient to just learn the positions or motion and play it back using a robot controller. In this article, a task learning approach is proposed for human–robot interaction systems, where a set of interaction behaviors is formulated and solved by specifying the task requirements in terms of potential energy. The motion behaviors demonstrated by humans can thus be acquired by the robot by seeking the appropriate task parameters of the dynamic potential energy function. To play back and combine the tasks in a sequential way by using a single controller, a new robot controller is also proposed. Lyapunov-like analysis is adopted to guarantee the stability of the control system, and experimental results are given to validate the performance of the proposed learning and control strategies.

모바일 매니퓰레이터의 엘리베이터 조작을 위한 HRI 시스템 구현

Unlike conventional industrial robots that operate in a limited manufacturing environment, service robots should be able to operate in various environments. Where there are many complex high-rise buildings, service robots require the ability to ride and control elevators. Furthermore, a human-robot interaction (HRI) approach is critical because service robots have the characteristics of interacting directly with humans. Therefore, this paper proposes an HRI system that enables direct elevator button manipulation to enable mobile manipulator robots to perform tasks in various high-rise building environments. The system is divided into three subsystems: elevator button recognition, command reception through voice recognition, and elevator button operation. Elevator button recognition and command reception through voice recognition receive the required topics and communicate with the lower end of the elevator button operation. Although the three distinct substructures interact with each other and are independent, they can be replaced by different algorithms at any time. Because of the same characteristics, the system can be applied to various mobile manipulators with different specifications. The experiment proceeds with pressing the button on floors B1-4 through voice recognition in a simulation environment and pressing the button on floors 5-7 without voice recognition. The experimental results compared the central positions of the button and end effector’s x, y, and z coordinates after control. The system, verified through simulation, has since been applied to real robots.

Affective video recommender systems: A survey.

Traditional video recommendation provides the viewers with customized media content according to their historical records (e.g., ratings, reviews). However, such systems tend to generate terrible results if the data is insufficient, which leads to a cold-start problem. An affective video recommender system (AVRS) is a multidiscipline and multimodal human-robot interaction (HRI) system, and it incorporates physical, physiological, neuroscience, and computer science subjects and multimedia resources, including text, audio, and video. As a promising research domain, AVRS employs advanced affective analysis technologies in video resources; therefore, it can solve the cold-start problem. In AVRS, the viewers’ emotional responses can be obtained from various techniques, including physical signals (e.g., facial expression, gestures, and speech) and internal signals (e.g., physiological signals). The changes in these signals can be detected when the viewers face specific situations. The physiological signals are a response to central and autonomic nervous systems and are mostly involuntarily activated, which cannot be easily controlled. Therefore, it is suitable for reliable emotion analysis. The physical signals can be recorded by a webcam or recorder. In contrast, the physiological signals can be collected by various equipment, e.g., psychophysiological heart rate (HR) signals calculated by echocardiogram (ECG), electro-dermal activity (EDA), and brain activity (GA) from electroencephalography (EEG) signals, skin conductance response (SCR) by a galvanic skin response (GSR), and photoplethysmography (PPG) estimating users’ pulse. This survey aims to provide a comprehensive overview of the AVRS domain. To analyze the recent efforts in the field of affective video recommendation, we collected 92 relevant published articles from Google Scholar and summarized the articles and their key findings. In this survey, we feature these articles concerning AVRS from different perspectives, including various traditional recommendation algorithms and advanced deep learning-based algorithms, the commonly used affective video recommendation databases, audience response categories, and evaluation methods. Finally, we conclude the challenge of AVRS and provide the potential future research directions.

A Universal Decoupled Training Framework for Human Parsing.

Human parsing is an important technology in human–robot interaction systems. At present, the distribution of multi-category human parsing datasets is unbalanced, and the samples present a long-tailed distribution, which directly affects the performance of human parsing. Meanwhile, the similarity between different categories leads the model to predict false parsing results. To solve the above problems, a general decoupled training framework called Decoupled Training framework based on Pixel Resampling (DTPR) was proposed to solve the long-tailed distribution, and a new sampling method named Pixel Resampling based on Accuracy distribution (PRA) for semantic segmentation was also proposed and applied to this decoupled training framework. The framework divides the training process into two phases, the first phase is to improve the model feature extraction ability, and the second phase is to improve the performance of the model on tail categories. The training framework was evaluated in MHPv2.0 and LIP datasets, and tested in both high-precision and real-time SOTA models. The MPA metric of model trained by DTPR in above two datasets increased by more than 6%, and the mIoU metric increased by more than 1% without changing the model structure.

Human–Robot Interaction: A Review and Analysis on Variable Admittance Control, Safety, and Perspectives

Human–robot interaction (HRI) is a broad research topic, which is defined as understanding, designing, developing, and evaluating the robotic system to be used with or by humans. This paper presents a survey on the control, safety, and perspectives for HRI systems. The first part of this paper reviews the variable admittance (VA) control for human–robot co-manipulation tasks, where the virtual damping, inertia, or both are adjusted. An overview of the published research for the VA control approaches, their methods, the accomplished collaborative co-manipulation tasks and applications, and the criteria for evaluating them are presented and compared. Then, the performance of various VA controllers is compared and investigated. In the second part, the safety of HRI systems is discussed. The various methods for detection of human–robot collisions (model-based and data-based) are investigated and compared. Furthermore, the criteria, the main aspects, and the requirements for the determination of the collision and their thresholds are discussed. The performance measure and the effectiveness of each method are analyzed and compared. The third and final part of the paper discusses the perspectives, necessity, influences, and expectations of the HRI for future robotic systems.

A Review on HRI for Child Learning

Human robot interaction (HRI) becomes an effective method of interacting with kids. Many HRI's system are currently available. In this paper, a review of various robotic tutor system based on design aspects, safety aspects, mechanical and electronic aspects of robots, learning outcome of kids, type of interaction, configuration of various learning modules, real time behavior monitoring and socio centric aspect are presented in this paper. As the robot tutoring has several benefits over human tutoring because these kinds of system have various pre-installed learning modules so that without any human monitoring child learn its concept. The robotic system is now a days become very effective for the learner for real time applications

Face Recognition and Tracking Framework for Human–Robot Interaction

Recently, face recognition became a key element in social cognition which is used in various applications including human–robot interaction (HRI), pedestrian identification, and surveillance systems. Deep convolutional neural networks (CNNs) have achieved notable progress in recognizing faces. However, achieving accurate and real-time face recognition is still a challenging problem, especially in unconstrained environments due to occlusion, lighting conditions, and the diversity in head poses. In this paper, we present a robust face recognition and tracking framework in unconstrained settings. We developed our framework based on lightweight CNNs for all face recognition stages, including face detection, alignment and feature extraction, to achieve higher accuracies in these challenging circumstances while maintaining the real-time capabilities required for HRI systems. To maintain the accuracy, a single-shot multi-level face localization in the wild (RetinaFace) is utilized for face detection, and additive angular margin loss (ArcFace) is employed for recognition. For further enhancement, we introduce a face tracking algorithm that combines the information from tracked faces with the recognized identity to use in the further frames. This tracking algorithm improves the overall processing time and accuracy. The proposed system performance is tested in real-time experiments applied in an HRI study. Our proposed framework achieves real-time capabilities with an average of 99%, 95%, and 97% precision, recall, and F-score respectively. In addition, we implemented our system as a modular ROS package that makes it straightforward for integration in different real-world HRI systems.

Overview of some Command Modes for Human-Robot Interaction Systems

Interaction and command modes as well as their combination are essential features of modern and futuristic robotic systems interacting with human beings in various dynamical environments. This paper presents a synthetic overview concerning the most command modes used in Human-Robot Interaction Systems (HRIS). It includes the first historical command modes which are namely tele-manipulation, off-line robot programming, and traditional elementary teaching by demonstration. It then introduces the most recent command modes which have been fostered later on by the use of artificial intelligence techniques implemented on more powerful computers. In this context, we will consider specifically the following modes: interactive programming based on the graphical-user-interfaces, voice-based, pointing-on-image-based, gesture-based, and finally brain-based commands.

A Tool for Organizing Key Characteristics of Virtual, Augmented, and Mixed Reality for Human–Robot Interaction Systems: Synthesizing VAM-HRI Trends and Takeaways

Frameworks have begun to emerge to categorize Virtual, Augmented, and Mixed Reality (VAM) technologies that provide immersive, intuitive interfaces to facilitate Human-Robot Interaction. These frameworks, however, fail to capture key characteristics of the growing subfield of VAM-HRI and can be difficult to consistently apply due to continuous scales. This work builds upon these prior frameworks through the creation of a Tool for Organizing Key Characteristics of VAM-HRI Systems (TOKCS). TOKCS discretizes the continuous scales used within prior works for more consistent classification and adds additional characteristics related to a robot's internal model, anchor locations, manipulability, and the system's software and hardware. To showcase the tool's capability, TOKCS is applied to the ten papers from the fourth VAM-HRI workshop and examined for key trends and takeaways. These trends highlight the expressive capability of TOKCS while also helping frame newer trends and future work recommendations for VAM-HRI research.

Analysis and design of a macro-mini robotic system for physical human-robot interaction

The macro-mini architecture has been studied in the past to help with physical human-robot interaction. This paper focuses on the investigation of an active macro/active mini robotic system for the manipulation of large payloads with the objective of designing collaborative robotic systems that provide human users with a natural, effective and intuitive interface. More precisely, an analysis of the system is presented with the goal of determining the range of motion necessary for the active mini robot to ensure a favourable user perception, considering several limitations. To this end, the kinematic and dynamic models of the system are described. The different limitations, such as safety standards, kinematic and dynamic capabilities of both robots (macro and mini), human maximum allowable force and others, and their impact on the interaction bandwidth are stated. A simple controller used for simulations is introduced. Finally, simulation results are presented and the phase shift between the macro robot and the mini robot is observed, for a simple harmonic motion of the payload. Along with the phase shift, the mathematical models derived are used to obtain the behaviour of the mini robot's amplitude as a function of the frequency. For the chosen limitations (based on realistic values), it is shown that reducing the payload's maximum allowed velocity by half (from 1ms−1 to0.5ms−1) reduces the necessary motion range of the mini by a factor of approximately 4.

Data-Driven Human-Robot Interaction Without Velocity Measurement Using Off-Policy Reinforcement Learning

In this paper, we present a novel data-driven design method for the human-robot interaction (HRI) system, where a given task is achieved by cooperation between the human and the robot. The presented HRI controller design is a two-level control design approach consisting of a task-oriented performance optimization design and a plant-oriented impedance controller design. The task-oriented design minimizes the human effort and guarantees the perfect task tracking in the outer-loop, while the plant-oriented achieves the desired impedance from the human to the robot manipulator end-effector in the inner-loop. Data-driven reinforcement learning techniques are used for performance optimization in the outer-loop to assign the optimal impedance parameters. In the inner-loop, a velocity-free filter is designed to avoid the requirement of end-effector velocity measurement. On this basis, an adaptive controller is designed to achieve the desired impedance of the robot manipulator in the task space. The simulation and experiment of a robot manipulator are conducted to verify the efficacy of the presented HRI design framework.

Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots.

Collaborative robots are currently deployed in professional environments, in collaboration with professional human operators, helping to strike the right balance between mechanization and manual intervention in manufacturing processes required by Industry 4.0. In this paper, the contribution of gesture recognition and pose estimation to the smooth introduction of cobots into an industrial assembly line is described, with a view to performing actions in parallel with the human operators and enabling interaction between them. The proposed active vision system uses two RGB-D cameras that record different points of view of gestures and poses of the operator, to build an external perception layer for the robot that facilitates spatiotemporal adaptation, in accordance with the human's behavior. The use-case of this work is concerned with LCD TV assembly of an appliance manufacturer, comprising of two parts. The first part of the above-mentioned operation is assigned to a robot, strengthening the assembly line. The second part is assigned to a human operator. Gesture recognition, pose estimation, physical interaction, and sonic notification, create a multimodal human-robot interaction system. Five experiments are performed, to test if gesture recognition and pose estimation can reduce the cycle time and range of motion of the operator, respectively. Physical interaction is achieved using the force sensor of the cobot. Pose estimation through a skeleton-tracking algorithm provides the cobot with human pose information and makes it spatially adjustable. Sonic notification is added for the case of unexpected incidents. A real-time gesture recognition module is implemented through a Deep Learning architecture consisting of Convolutional layers, trained in an egocentric view and reducing the cycle time of the routine by almost 20%. This constitutes an added value in this work, as it affords the potential of recognizing gestures independently of the anthropometric characteristics and the background. Common metrics derived from the literature are used for the evaluation of the proposed system. The percentage of spatial adaptation of the cobot is proposed as a new KPI for a collaborative system and the opinion of the human operator is measured through a questionnaire that concerns the various affective states of the operator during the collaboration.

A Parallel Robot with Torque Monitoring for Brachial Monoparesis Rehabilitation Tasks

Robots for rehabilitation tasks require a high degree of safety for the interaction with both the patients and for the operators. In particular, high safety is a stable and intuitive control of the moving elements of the system combined with an external system of sensors able to monitor the position of every aspect of the rehabilitation system (operator, robot, and patient) and overcome in a certain measure all the events that may occur during the robotic rehabilitation procedure. This paper presents the development of an internal torque monitoring system for ASPIRE. This is a parallel robot designed for shoulder rehabilitation, which enables the use of strategies towards developing a HRI (human–robot interaction) system for the therapy. A complete analysis regarding the components of the robotic system is carried out with the purpose of determining the dynamic behavior of the system. Next, the proposed torque monitoring system is developed with respect to the previously obtained data. Several experimental tests are performed using healthy subjects being equipped with a series of biomedical sensors with the purpose of validating the proposed torque monitoring strategy and, at the same time, to satisfy the degree of safety that is requested by the medical procedure.

Domain and View-Point Agnostic Hand Action Recognition

Hand action recognition is a special case of action recognition with applications in human-robot interaction, virtual reality or life-logging systems. Building action classifiers able to work for such heterogeneous action domains is very challenging. There are very subtle changes across different actions from a given application but also large variations across domains (e.g. virtual reality vs life-logging). This work introduces a novel skeleton-based hand motion representation model that tackles this problem. The framework we propose is agnostic to the application domain or camera recording view-point. When working on a single domain (intra-domain action classification) our approach performs better or similar to current state-of-the-art methods on well-known hand action recognition benchmarks. And, more importantly, when performing hand action recognition for action domains and camera perspectives which our approach has not been trained for (cross-domain action classification), our proposed framework achieves comparable performance to intra-domain state-of-the-art methods. These experiments show the robustness and generalization capabilities of our framework.

Programming Robots by Demonstration Using Augmented Reality.

The world is living the fourth industrial revolution, marked by the increasing intelligence and automation of manufacturing systems. Nevertheless, there are types of tasks that are too complex or too expensive to be fully automated, it would be more efficient if the machines were able to work with the human, not only by sharing the same workspace but also as useful collaborators. A possible solution to that problem is on human–robot interaction systems, understanding the applications where they can be helpful to implement and what are the challenges they face. This work proposes the development of an industrial prototype of a human–machine interaction system through Augmented Reality, in which the objective is to enable an industrial operator without any programming experience to program a robot. The system itself is divided into two different parts: the tracking system, which records the operator’s hand movement, and the translator system, which writes the program to be sent to the robot that will execute the task. To demonstrate the concept, the user drew geometric figures, and the robot was able to replicate the operator’s path recorded.

A multimodal emotion recognition method based on facial expressions and electroencephalography

Human-robot interaction (HRI) systems play a critical role in society. However, most HRI systems nowadays still face the challenge of disharmony, resulting in an inefficient communication between the human and the robot. In this paper, a multimodal emotion recognition method is proposed to establish an HRI system with a low sense of disharmony. This method is based on facial expressions and electroencephalography (EEG). The image classification method of facial expressions and the suitable feature extraction method of EEG were investigated based on the public datasets. And then these methods were applied to both images and EEG data acquired by ourselves. In addition, the Monte Carlo method was used to merge the results and solve the problem of having a small dataset. The multimodal emotion recognition method was combined with the HRI system, where it achieved a recognition rate of 83.33%. Furthermore, in order to evaluate the HRI system from the user's point of view, a perceptual assessment method was proposed to evaluate our system, in which the system was scored by the participants based on their experience, achieving an average score of 7 (the scores were ranged from 0 to 10). Experimental results demonstrate the effectiveness and feasibility of the multimodal emotion recognition method, which can be useful to reduce the sense of disharmony of HRI systems.

CHARMIE: A Collaborative Healthcare and Home Service and Assistant Robot for Elderly Care

The global population is ageing at an unprecedented rate. With changes in life expectancy across the world, three major issues arise: an increasing proportion of senior citizens; cognitive and physical problems progressively affecting the elderly; and a growing number of single-person households. The available data proves the ever-increasing necessity for efficient elderly care solutions such as healthcare service and assistive robots. Additionally, such robotic solutions provide safe healthcare assistance in public health emergencies such as the SARS-CoV-2 virus (COVID-19). CHARMIE is an anthropomorphic collaborative healthcare and domestic assistant robot capable of performing generic service tasks in non-standardised healthcare and domestic environment settings. The combination of its hardware and software solutions demonstrates map building and self-localisation, safe navigation through dynamic obstacle detection and avoidance, different human-robot interaction systems, speech and hearing, pose/gesture estimation and household object manipulation. Moreover, CHARMIE performs end-to-end chores in nursing homes, domestic houses, and healthcare facilities. Some examples of these chores are to help users transport items, fall detection, tidying up rooms, user following, and set up a table. The robot can perform a wide range of chores, either independently or collaboratively. CHARMIE provides a generic robotic solution such that older people can live longer, more independent, and healthier lives.

Basic human–robot interaction system running on an embedded platform

Robotics will be a dominant area in society throughout future generations. Its presence is currently increasing in most daily life settings, with devices and mechanisms that facilitate the accomplishment of diverse tasks, as well as in work scenarios, where machines perform more and more jobs. This increase in the presence of autonomous robotic systems in society is due to their great efficiency and security compared to human capacity, which is thanks mainly to the enormous precision of their sensor and actuator systems. Among these, vision sensors are of the utmost importance. Humans and many animals naturally enjoy powerful perception systems, but, in robotics, this constitutes a constant line of research. In addition to having a high capacity for reasoning and decision-making, these robots incorporate important advances in their perceptual systems, allowing them to interact effectively in the working environments of this new industrial revolution. Drawing on the most basic interaction between humans, looking at the face, an innovative system is presented in this paper, which was developed for an autonomous and DIY robot. This system is composed of three modules. First, the face detection component, which detects human faces in the current image. Second, the scene representation algorithm, which offers a wider field of view than that of the single camera used, mounted on a servo-pan unit. Third, the active memory component, which was designed and implemented according to two competing dynamics: life and salience. The algorithm intelligently moves the servo-pan unit with the aim of finding new faces, follow existing ones and forgetting those that no longer appear on the scene. The system was developed and validated using a low-cost platform based on a Raspberry Pi3 board.