Human-robot Interaction Applications Research Articles

An emotion recognition system: bridging the gap between humans-machines interaction

Human emotion recognition has emerged as a vital research area in recent years due to its widespread applications in psychology, healthcare, education, entertainment, and human-robot interaction. This research article comprehensively analyzes a machine learning-based six-emotion classification algorithm, focusing on its development, evaluation, and potential applications. The study aims to assess the algorithm's performance, identify its limitations, and discuss the importance of selecting appropriate image descriptors for accurate emotion classification. The algorithm achieved an overall accuracy of 92.23\%, showcasing its potential in various fields. However, the classification of specific emotions, particularly ``excited'' and ``afraid'', demonstrated some limitations, suggesting further refinement. The study also highlights the significance of choosing suitable image descriptors, with the manual distance calculation used in the framework proving effective. This article offers insights into developing and evaluating a six-emotion classification algorithm using a machine learning framework, emphasizing its strengths, limitations, and possible applications in multiple domains. The findings contribute to ongoing efforts in creating robust, accurate, and versatile emotion recognition systems that cater to the diverse needs of various applications across psychology, healthcare, robotics, education, and entertainment.

Flexible Actuators Based on Conductive Polymer Ionogels and Their Electromechanical Modeling.

High-performance flexible actuators, integral components of soft robotics, hold promise for advancing applications in safe human-robot interactions, healthcare, and various other fields. Notable among these actuators are flexible electrochemical systems, recognized for their merits in low-voltage manipulation, rapid response speed, and cost-effectiveness. However, the optimization of output strain, response speed, and stability presents a significant challenge in this domain. Despite the application of diverse electrochemically active materials to enhance actuation performance, a critical need persists for corresponding electrical-mechanical models to comprehensively grasp actuation mechanisms. In this study, we introduce a novel electrochemical actuator that utilizes conductive polymer ionogel as active electrodes. This ionogel exhibits exceptional properties, including high conductivity, flexibility, and electrochemical activity. Our electrochemical actuators exhibit noteworthy bending strain capabilities and rapid response rates, achieving frequencies up to 10 Hz at a modest voltage of 1 V. An analytical model integrating ion migration and dynamic processes has been established to elucidate actuator behavior. Simulation results highlight that electrodes characterized by low resistance and high capacitance are optimal for simultaneous enhancement of bending strain and blocking force. However, the augmentation of Young's modulus, while increasing blocking force, compromises bending strain. Furthermore, a larger aspect ratio proves beneficial for unidirectional stress output, leading to increased bending strain, while actuator blocking force diminishes with greater length. These findings underscore the intricate interplay between material properties and dimensions in optimizing the performance of flexible electrochemical actuators. This work provides important practical and theoretical guidance for the manufacture of high-performance flexible actuators and the search for new smart materials.

Variable admittance control for safe physical human–robot interaction considering intuitive human intention

The trajectory planning of the arm is greatly facilitated by the physical direct teaching technology. Variable admittance control is a promising technology when a robot is interacting with a variable environment, such as a human whose stiffness might change during the interaction. Nevertheless, a canonical admittance controller with imperfect parameters may lead to robot oscillation, which brings more challenges to a variable admittance controller. In this paper, we propose an energy-based variable admittance controller with intrinsic oscillation suppression property. During the physical human–robot interaction (pHRI), the human intention is predicted based on the robot’s state and interaction force. The admittance parameters are tuned automatically to conform the robot’s motion to human intention. When the oscillation is detected online by the proposed wavelet module, our variable admittance model reveals oscillation suppression ability because of dissipating the energy generated by high-frequency oscillation. We compared the proposed variable admittance controller with other admittance controller approaches in both simulation and actual robot experiments. The proposed method shows significant improvement in oscillation suppression and human energy conservation in the human–robot interaction application.

Design, Fabrication, and Characterization of Inkjet-Printed Organic Piezoresistive Tactile Sensor on Flexible Substrate.

In this paper, we propose a novel tactile sensor with a "fingerprint" design, named due to its spiral shape and dimensions of 3.80 mm × 3.80 mm. The sensor is duplicated in a four-by-four array containing 16 tactile sensors to form a "SkinCell" pad of approximately 45 mm by 29 mm. The SkinCell was fabricated using a custom-built microfabrication platform called the NeXus which contains additive deposition tools and several robotic systems. We used the NeXus' six-degrees-of-freedom robotic platform with two different inkjet printers to deposit a conductive silver ink sensor electrode as well as the organic piezoresistive polymer PEDOT:PSS-Poly (3,4-ethylene dioxythiophene)-poly(styrene sulfonate) of our tactile sensor. Printing deposition profiles of 100-micron- and 250-micron-thick layers were measured using microscopy. The resulting structure was sintered in an oven and laminated. The lamination consisted of two different sensor sheets placed back-to-back to create a half-Wheatstone-bridge configuration, doubling the sensitivity and accomplishing temperature compensation. The resulting sensor array was then sandwiched between two layers of silicone elastomer that had protrusions and inner cavities to concentrate stresses and strains and increase the detection resolution. Furthermore, the tactile sensor was characterized under static and dynamic force loading. Over 180,000 cycles of indentation were conducted to establish its durability and repeatability. The results demonstrate that the SkinCell has an average spatial resolution of 0.827 mm, an average sensitivity of 0.328 mΩ/Ω/N, expressed as the change in resistance per force in Newtons, an average sensitivity of 1.795 µV/N at a loading pressure of 2.365 PSI, and a dynamic response time constant of 63 ms which make it suitable for both large area skins and fingertip human-robot interaction applications.

Antagonistic Magneto-Rheological Actuators with Inherent Output Boundedness: An Ideal Solution for High-Performance and Human-Safe Actuation

This paper studies the working principles of antagonistic magneto-rheological (MR) actuators, i.e., a combination of an electric motor and a pair of MR clutches in an antagonistic configuration, for compliant actuation in robotics. The study focuses on the unique boundedness property exhibited by MR actuators, which limits the output torques delivered to the load, independent of the received input torque and/or control commands. This inherent property is of significant importance for ensuring human safety in human–robot interaction applications. Through a comprehensive analysis, we provide analytical proof of the inherent output boundedness of antagonistic MR actuators and validate our findings through experimental results. Our research demonstrates that these actuators are well-suited for safe operations in robotic applications, eliminating the need for additional sensor measurements or complex control strategies. This promising capability enables the avoidance of trade-offs between actuator performance, complexity, and cost. The insights gained from this study contribute to advancing compliant actuation technology, paving the way for high-performance and human-safe robotic systems.

End-to-End Speaker Profiling Using 1D CNN Architectures and Filter Bank Initialization

The automatic estimation of speaker characteristics, such as height, age, and gender, has various applications in forensics, surveillance, customer service, and many human-robot interaction applications. These applications are often required to produce a response promptly. This work proposes a novel approach to speaker profiling by combining filter bank initializations, such as continuous wavelets and gammatone filter banks, with one-dimensional (1D) convolutional neural networks (CNN) and residual blocks. The proposed end-to-end model goes from the raw waveform to an estimated height, age, and gender of the speaker by learning speaker representation directly from the audio signal without relying on handcrafted and pre-computed acoustic features. The conducted experiments on the TIMIT dataset show that the proposed approach outperforms many previous studies on speaker profiling with a mean absolute error (MAE) of 5.18 and 4.91 cm in height estimation and MAE of 5.36 and 6.07 years in age estimation for males and females, respectively, and achieving an accuracy of 99.98% in gender prediction.

Safe High Stiffness Impedance Control for Series Elastic Actuators using Collocated Position Feedback

Physical human-robot interaction applications requiring safety and compliance characteristics usually employ mechanical devices such as series elastic actuators (SEAs). So far, impedance control schemes for SEAs have been investigated to overcome the limited displayable stiffness at the end point, represented by the stiffness of a physical spring, in a passive manner. SEAs usually mount two encoders on both the motor and load sides for position measurements. However, in several applications, the installation of a load-side encoder is not feasible from both cost and manufacturing perspectives. In this scenario, a Kalman Filter based on micro electro mechanical system accelerometers has been proposed for estimating the external force and load side quantities, without the need for load-side encoders. Herein, a novel impedance control scheme leveraging this approach is proposed to effectively overcome the stiffness of the physical spring while maintaining passivity at the end point. Compared with conventional load-side encoder-based impedance control schemes, the proposed control scheme presents comparable performance, as demonstrated by simulation and experimental results.

Outperformance of Mall-Receptionist Android as Inverse Reinforcement Learning is Transitioned to Reinforcement Learning

Robots can tackle human–robot interaction (HRI) tasks through inverse reinforcement learning (IRL). However, offline IRL agents' performance is upper-bounded by experts. Limited demonstration fails to provide an overall picture of the environment, especially in real-world applications. To further enhance IRL's performance, we implement a cross-modal inverse reinforcement learning to reinforcement learning (IRL-to-RL) transition framework for a real-world HRI interaction task, in which a mall receptionist android promotes sanitizer usage. During the 10-day experiment, the android develops a more proactive and effective strategy than the human expert. Furthermore, we explore four decay modes of prior knowledge supervision and suggest a preferable pattern for practical use. Our results demonstrate the feasibility of the framework to assist robots in switching to diverse modalities, learning incrementally with a sparse reward function, and eventually outperforming the human expert. We anticipate our framework to inspire more IRL-to-better learning paradigms, facilitating robots to outcompete teachers in more real-world HRI applications.

Hi-ROS: Open-source multi-camera sensor fusion for real-time people tracking

This paper presents Hi-ROS (Human Interaction in ROS), an open source framework focused on real-time accurate assessment of human motion. The system offers a series of tools to track multiple people in real-time by exploiting a calibrated camera network. No assumptions are made about the typology or number of cameras, nor about the body pose estimation algorithm used to extract the 3D poses of the people in the scene. The tools provided by Hi-ROS include a Skeleton Tracker to ensure temporal consistency of the detected poses, a Skeleton Merger to fuse the tracks from multiple cameras, thus limiting flickering phenomena, a Skeleton Optimizer to ensure limb length consistency, and a Skeleton Filter to perform real-time smoothing of the detected joint trajectories. Accuracy, tracking robustness, and real-time performance of the proposed system were evaluated on a public dataset, containing both single-person and multi-person sequences with up to 4 people interacting. The results obtained using different subsets of the proposed tools show how the complete Hi-ROS pipeline provides accurate and reliable estimates also in challenging scenarios, with a reduction of the RMSE of up to 27% with respect to a pure tracking approach. This work aims to push forward the development of unobtrusive human–robot interaction applications, multi-person automated posture analyses, rehabilitation performance assessments, and any possible application enabled by real-time accurate assessment of human motion via markerless motion capture.

A multidimensional taxonomy for human-robot interaction in construction

Despite the increased interest in construction robotics both in academia and the industry, insufficient attention has been given to aspects related to Human-Robot Interaction (HRI). Characterizing HRI for construction tasks can help researchers organize knowledge in a structured manner that allows for classifying construction robotics applications and comparing and benchmarking different studies. This paper builds upon existing taxonomies and empirical studies in HRI in various industries (e.g., construction, manufacturing, and military, among others) to propose a multidimensional taxonomy to characterize HRI applications in the construction industry. The taxonomy design followed a systematic literature review in which common themes were identified and grouped into 16 categories. The proposed taxonomy can be used as a foundation for systematic reviews and meta-analyses of HRI applications in construction and can benefit the construction industry by informing the design of collaborative tasks performed by human-robot teams.

SkinCell: A Modular Tactile Sensor Patch for Physical Human–Robot Interaction

In this article, we present the design, fabrication, integration, and performance evaluation of SkinCell sensors, a novel tactile array targeting human–robot interaction applications. A polyimide sheet is selected as the substrate of the sensor, and gold electrodes with circular interdigitated finger patterns are deposited through dc sputtering, creating a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula> 4 sensor array with 16 tactile elements. Sensing elements are completed with a thin layer of spin-coated poly (3,4-ethylenedioxythiophene) polystyrene sulfonate (PEDOT: PSS) organic polymer solution. Two individual sensor sheets are being laminated in a back-to-back fashion to create a half-Wheatstone-bridge configuration to reduce temperature coupling while improving sensitivity. The sensor array is then sandwiched between two layers of silicone elastomer with internal dimples and cavities to improve sensitivity, repeatability, and center of force detection resolution. A COMSOL simulation was performed to study the influence of adding the silicone encapsulation as well as design parameters to the sensor substrate. Our simulation results indicate an effective improvement of touch, i.e., voltage sensitivity by employing the featured encapsulation. Characterization experiments have been conducted to identify the sensitivity profile of each sensor on the array, which indicates an average sensitivity of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$26.5~\mu $ </tex-math></inline-formula> V/N. In subsequent experiments, the calibration profiles are used to identify the resultant force during pressing, as well as the location of the center of pressure (COP) on the SkinCell sensor. Finally, SkinCell sensors have been integrated into the OctoCan, a structural electronic device that can acquire squeezing pressure data during physical human–robot interaction (pHRI).

Integrating Virtual, Mixed, and Augmented Reality to Human–Robot Interaction Applications Using Game Engines: A Brief Review of Accessible Software Tools and Frameworks

This article identifies and summarizes software tools and frameworks proposed in the Human–Robot Interaction (HRI) literature for developing extended reality (XR) experiences using game engines. This review includes primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can control or interact with real robotic platforms using devices that extend the user’s reality. The objective of this article is not to present an extensive list of applications and tools. Instead, we present recent, relevant, common, and accessible frameworks and software tools implemented in research articles published in high-impact robotics conferences and journals. For this, we searched papers published during a seven-years period between 2015 and 2022 in relevant databases for robotics (Science Direct, IEEE Xplore, ACM digital library, Springer Link, and Web of Science). Additionally, we present and classify the application context of the reviewed articles in four groups: social robotics, programming of industrial robots, teleoperation of industrial robots, and Human–Robot collaboration (HRC).

Virtual reality in human-robot interaction: Challenges and benefits

&lt;abstract&gt; &lt;p&gt;Virtual reality (VR) technology has been increasingly employed in human-robot interaction (HRI) research to enhance the immersion and realism of the interaction. However, the integration of VR into HRI also introduces new challenges, such as latency, mismatch between virtual and real environments and potential adverse effects on human users. Despite these challenges, the use of VR in HRI has the potential to provide numerous benefits, including improved communication, increased safety and enhanced training and education. Yet, little research has been done by scholars to review the state of the art of VR applications in human-robot interaction. To bridge the gap, this paper provides an overview of the challenges and benefits of using VR in HRI, as well as current research in the field and future directions for development. It has been found that robots are getting more personalized, interactive and engaging than ever; and with the popularization of virtual reality innovations, we might be able to foresee the wide adoption of VR in controlling robots to fulfill various tasks of hospitals, schools and factories. Still, there are several challenges, such as the need for more advanced VR technologies to provide more realistic and immersive experiences, the development of more human-like robot models to improve social interactions and the need for better methods of evaluating the effectiveness of VR in human-robot interaction.&lt;/p&gt; &lt;/abstract&gt;

Sentiment Analysis in Turkish Question Answering Systems: An Application of Human-Robot Interaction

Sentiment Analysis in Turkish Question Answering Systems: An Application of Human-Robot Interaction

Iterative assist-as-needed control with skill learning for physical human-robot interaction

Collaborative robots referring to physical human-robot interaction applications have drawn considerable attention. There have been lots of methods based on skill learning to help robots mimic human collaborative behaviors. However, the stiffness profiles generated by these methods only rely on the quality of collected human demonstrations and cannot be adjusted according to detailed interaction tasks. To this end, this paper combines a skill learning method and an iterative assist-as-needed impedance controller. The skill learning method encodes and generates position and stiffness references. The iterative assist-as-needed controller enables the robot to adjust the generated stiffness profile actively according to the human intention. When the human tries to act as an active role in interaction tasks, the controller takes a lower stiffness parameter, which means it can tolerate more deviation from the desired trajectory, and vice versa. Compared to the previous results, on the one hand, this paper employs a skill learning method to generate an initial stiffness profile for the iterative assist-as-needed controller, which enables the robot to mimic human behaviors; on the other hand, the controller proposed in this paper allows the manual design of the damping and mass parameters while the previous controller accepts only the desired stiffness parameter. Finally, one simulation and one experiment are demonstrated to evaluate the performance of the designed controller.

Safe Learning-Based Control of Elastic Joint Robots via Control Barrier Functions

Ensuring safety is of paramount importance in physical human-robot interaction applications. This requires both adherence to safety constraints defined on the system state, as well as guaranteeing compliant behavior of the robot. If the underlying dynamical system is known exactly, the former can be addressed with the help of control barrier functions. The incorporation of elastic actuators in the robot's mechanical design can address the latter requirement. However, this elasticity can increase the complexity of the resulting system, leading to unmodeled dynamics, such that control barrier functions cannot directly ensure safety. In this paper, we mitigate this issue by learning the unknown dynamics using Gaussian process regression. By employing the model in a feedback linearizing control law, the safety conditions resulting from control barrier functions can be robustified to take into account model errors, while remaining feasible. In order to enforce them on-line, we formulate the derived safety conditions in the form of a second-order cone program. We demonstrate our proposed approach with simulations on a two-degree-of-freedom planar robot with elastic joints.

Enhancing social responses: effects of controlling language by a social robot in a decision-making game for human-robot interaction (HRI)

The rise of technology has induced the development of robots that engage with humans through social interaction. The robot is believed capable of assisting humans in their life. However, the current technology is still far from a fully autonomous robot as there are many limitations. Additionally, it is unclear whether the current social robot effectively influences social reactance in Human-Robot Interaction (HRI). The objective of the study is to investigate the influence of social cues used by the social robot on human social responses for HRI applications. Also, the study validates the reactance scale used in the questionnaire by correlating the measure with Galvanic Skin Response (GSR) readings. The study proposes Wizard of Oz (WoZ) approach to observe HRI through decision-making games. A social robot is programmed to persuade participants' in making choices. The participants' decisions made during the experiment and their GSR reading are recorded, and then they are asked to answer questionnaires. Statistical analyses are done on the collected data using the regression and MANOVA statistical tests. As results, there is a significant correlation between GSR reading and enjoyment. Regarding social cues, the participants feel more relaxed when the social robot exhibits social cues in High Controlling Language (HCL) conditions rather than Low Controlling Language (LCL) conditions. Furthermore, the Attitude trait of the social robots greatly influences human perceived social intelligence towards the robot.

Development of a Visual Perception System on a Dual-Arm Mobile Robot for Human-Robot Interaction

This paper presents the development of a visual-perception system on a dual-arm mobile robot for human-robot interaction. This visual system integrates three subsystems. Hand gesture recognition is utilized to trigger human-robot interaction. Engagement and intention of the participants are detected and quantified through a cognitive system. Visual servoing uses YOLO to identify the object to be tracked and hybrid, model-based tracking to follow the object's geometry. The proposed visual-perception system is implemented in the developed dual-arm mobile robot, and experiments are conducted to validate the proposed method's effects on human-robot interaction applications.

Fine-tuned RetinaNet models for Vision-based Human Presence Detection

Moving towards Industry 4.0, the idea of human-robot interaction (HRI) and human-robot collaboration (HRC) has been popularized. To introduce more robots into the industries, risk-correlated issues would be always on the hook as robots are not as flexible as human. In fact, although robots can replace human workers in some of the dangerous tasks, still human safety is always the top priority for all industries. The most common way to safeguard the human was to isolate the working space of human workers and robots. To realize the idea of Industry 4.0, it is postulated to have the robots and cobots out of the cage to maximize productivity and efficiency. Hence, studies have been conducted with the attempts to free the robots from the isolated working space while preserve the safety of human operators. The present study seeks to explore the feasibility of transfer learning strategy — fine-tuning to human presence detection tasks as the base of practicing safe HRI. A custom image dataset with 1463 images was collected and separated into train, validation, and test set with a ratio of 70:20:10. Three RetinaNet object detection models with different backbone networks were fine-tuned with the acquired dataset to transfer the knowledge learned from source domain to the target domain, which is the human presence detection tasks. The result has shown that the RetinaNet_ResNet152-V1-FPN has the highest test AP of 74.4% with an inference speed of 13.09 FPS, suggesting that it is the best fine-tuned RetinaNet models. This study has demonstrated the feasibility of using fine-tuning as the strategy to train the object detection models, which can possibly act as the base for improving HRI applications via a deep learning visual-based method. In summary, the research has signified the uses of deep learning models to perform human presence detections and can be further extended for HRI safety applications.

A robot for overground physical human-robot interaction experiments.

Many anticipated physical human-robot interaction (pHRI) applications in the near future are overground tasks such as walking assistance. For investigating the biomechanics of human movement during pHRI, this work presents Ophrie, a novel interactive robot dedicated for physical interaction tasks with a human in overground settings. Unique design requirements for pHRI were considered in implementing the one-arm mobile robot, such as the low output impedance and the ability to apply small interaction forces. The robot can measure the human arm stiffness, an important physical quantity that can reveal human biomechanics during overground pHRI, while the human walks alongside the robot. This robot is anticipated to enable novel pHRI experiments and advance our understanding of intuitive and effective overground pHRI.