Human-robot Interaction Applications Research Articles

Effective Acoustic Model-Based Beamforming Training for Static and Dynamic Hri Applications.

Human-robot collaboration will play an important role in the fourth industrial revolution in applications related to hostile environments, mining, industry, forestry, education, natural disaster and defense. Effective collaboration requires robots to understand human intentions and tasks, which involves advanced user profiling. Voice-based communication, rich in complex information, is key to this. Beamforming, a technology that enhances speech signals, can help robots extract semantic, emotional, or health-related information from speech. This paper describes the implementation of a system that provides substantially improved signal-to-noise ratio (SNR) and speech recognition accuracy to a moving robotic platform for use in human-robot interaction (HRI) applications in static and dynamic contexts. This study focuses on training deep learning-based beamformers using acoustic model-based multi-style training with measured room impulse responses (RIRs). The results show that this approach outperforms training with simulated RIRs or matched measured RIRs, especially in dynamic conditions involving robot motion. The findings suggest that training with a broad range of measured RIRs is sufficient for effective HRI in various environments, making additional data recording or augmentation unnecessary. This research demonstrates that deep learning-based beamforming can significantly improve HRI performance, particularly in challenging acoustic environments, surpassing traditional beamforming methods.

ExTraCT - Explainable trajectory corrections for language-based human-robot interaction using textual feature descriptions.

In human-robot interaction (HRI), understanding human intent is crucial for robots to perform tasks that align with user preferences. Traditional methods that aim to modify robot trajectories based on language corrections often require extensive training to generalize across diverse objects, initial trajectories, and scenarios. This work presents ExTraCT, a modular framework designed to modify robot trajectories (and behaviour) using natural language input. Unlike traditional end-to-end learning approaches, ExTraCT separates language understanding from trajectory modification, allowing robots to adapt language corrections to new tasks-including those with complex motions like scooping-as well as various initial trajectories and object configurations without additional end-to-end training. ExTraCT leverages Large Language Models (LLMs) to semantically match language corrections to predefined trajectory modification functions, allowing the robot to make necessary adjustments to its path. This modular approach overcomes the limitations of pre-trained datasets and offers versatility across various applications. Comprehensive user studies conducted in simulation and with a physical robot arm demonstrated that ExTraCT's trajectory corrections are more accurate and preferred by users in 80% of cases compared to the baseline. ExTraCT offers a more explainable approach to understanding language corrections, which could facilitate learning human preferences. We also demonstrated the adaptability and effectiveness of ExTraCT in a complex scenarios like assistive feeding, presenting it as a versatile solution across various HRI applications.

Janus Swarm Metamaterials for Information Display, Memory, and Encryption.

Metamaterials are emerging as an unconventional platform to perform computing abstractions in physical systems by processing environmental stimuli into information. While computation functions have been demonstrated in mechanical systems, they rely on compliant mechanisms to achieve predefined states, which impose inherent design restrictions that limit their miniaturization, deployment, reconfigurability, and functionality. Here, a metamaterial system is described based on responsive magnetoactive Janus particle (MAJP) swarms with multiple programmable functions. MAJPs are designed with tunable structure and properties in mind, that is, encoded swarming behavior and fully reversible switching mechanisms, to enable programmable dynamic display, non-volatile and semi-volatile memory, Boolean logic, and information encryption functions in soft, wearable devices. MAJPs and their unique swarming behavior open new functions for the design of multifunctional and reconfigurable display devices, and constitute a promising building block to develop the next generation of soft physical computing devices, with growing applications in security, defense, anti-counterfeiting, camouflage, soft robotics, and human-robot interaction.

A simulation‐assisted point cloud segmentation neural network for human–robot interaction applications

AbstractWith the advancement of industrial automation, the frequency of human–robot interaction (HRI) has significantly increased, necessitating a paramount focus on ensuring human safety throughout this process. This paper proposes a simulation‐assisted neural network for point cloud segmentation in HRI, specifically distinguishing humans from various surrounding objects. During HRI, readily accessible prior information, such as the positions of background objects and the robot's posture, can generate a simulated point cloud and assist in point cloud segmentation. The simulation‐assisted neural network utilizes simulated and actual point clouds as dual inputs. A simulation‐assisted edge convolution module in the network facilitates the combination of features from the actual and simulated point clouds, updating the features of the actual point cloud to incorporate simulation information. Experiments of point cloud segmentation in industrial environments verify the efficacy of the proposed method.

An emotion recognition system: bridging the gap between humans-machines interaction

Human emotion recognition has emerged as a vital research area in recent years due to its widespread applications in psychology, healthcare, education, entertainment, and human-robot interaction. This research article comprehensively analyzes a machine learning-based six-emotion classification algorithm, focusing on its development, evaluation, and potential applications. The study aims to assess the algorithm's performance, identify its limitations, and discuss the importance of selecting appropriate image descriptors for accurate emotion classification. The algorithm achieved an overall accuracy of 92.23\%, showcasing its potential in various fields. However, the classification of specific emotions, particularly ``excited'' and ``afraid'', demonstrated some limitations, suggesting further refinement. The study also highlights the significance of choosing suitable image descriptors, with the manual distance calculation used in the framework proving effective. This article offers insights into developing and evaluating a six-emotion classification algorithm using a machine learning framework, emphasizing its strengths, limitations, and possible applications in multiple domains. The findings contribute to ongoing efforts in creating robust, accurate, and versatile emotion recognition systems that cater to the diverse needs of various applications across psychology, healthcare, robotics, education, and entertainment.

Flexible Actuators Based on Conductive Polymer Ionogels and Their Electromechanical Modeling.

High-performance flexible actuators, integral components of soft robotics, hold promise for advancing applications in safe human-robot interactions, healthcare, and various other fields. Notable among these actuators are flexible electrochemical systems, recognized for their merits in low-voltage manipulation, rapid response speed, and cost-effectiveness. However, the optimization of output strain, response speed, and stability presents a significant challenge in this domain. Despite the application of diverse electrochemically active materials to enhance actuation performance, a critical need persists for corresponding electrical-mechanical models to comprehensively grasp actuation mechanisms. In this study, we introduce a novel electrochemical actuator that utilizes conductive polymer ionogel as active electrodes. This ionogel exhibits exceptional properties, including high conductivity, flexibility, and electrochemical activity. Our electrochemical actuators exhibit noteworthy bending strain capabilities and rapid response rates, achieving frequencies up to 10 Hz at a modest voltage of 1 V. An analytical model integrating ion migration and dynamic processes has been established to elucidate actuator behavior. Simulation results highlight that electrodes characterized by low resistance and high capacitance are optimal for simultaneous enhancement of bending strain and blocking force. However, the augmentation of Young's modulus, while increasing blocking force, compromises bending strain. Furthermore, a larger aspect ratio proves beneficial for unidirectional stress output, leading to increased bending strain, while actuator blocking force diminishes with greater length. These findings underscore the intricate interplay between material properties and dimensions in optimizing the performance of flexible electrochemical actuators. This work provides important practical and theoretical guidance for the manufacture of high-performance flexible actuators and the search for new smart materials.

Variable admittance control for safe physical human–robot interaction considering intuitive human intention

The trajectory planning of the arm is greatly facilitated by the physical direct teaching technology. Variable admittance control is a promising technology when a robot is interacting with a variable environment, such as a human whose stiffness might change during the interaction. Nevertheless, a canonical admittance controller with imperfect parameters may lead to robot oscillation, which brings more challenges to a variable admittance controller. In this paper, we propose an energy-based variable admittance controller with intrinsic oscillation suppression property. During the physical human–robot interaction (pHRI), the human intention is predicted based on the robot’s state and interaction force. The admittance parameters are tuned automatically to conform the robot’s motion to human intention. When the oscillation is detected online by the proposed wavelet module, our variable admittance model reveals oscillation suppression ability because of dissipating the energy generated by high-frequency oscillation. We compared the proposed variable admittance controller with other admittance controller approaches in both simulation and actual robot experiments. The proposed method shows significant improvement in oscillation suppression and human energy conservation in the human–robot interaction application.

Design, Fabrication, and Characterization of Inkjet-Printed Organic Piezoresistive Tactile Sensor on Flexible Substrate.

In this paper, we propose a novel tactile sensor with a "fingerprint" design, named due to its spiral shape and dimensions of 3.80 mm × 3.80 mm. The sensor is duplicated in a four-by-four array containing 16 tactile sensors to form a "SkinCell" pad of approximately 45 mm by 29 mm. The SkinCell was fabricated using a custom-built microfabrication platform called the NeXus which contains additive deposition tools and several robotic systems. We used the NeXus' six-degrees-of-freedom robotic platform with two different inkjet printers to deposit a conductive silver ink sensor electrode as well as the organic piezoresistive polymer PEDOT:PSS-Poly (3,4-ethylene dioxythiophene)-poly(styrene sulfonate) of our tactile sensor. Printing deposition profiles of 100-micron- and 250-micron-thick layers were measured using microscopy. The resulting structure was sintered in an oven and laminated. The lamination consisted of two different sensor sheets placed back-to-back to create a half-Wheatstone-bridge configuration, doubling the sensitivity and accomplishing temperature compensation. The resulting sensor array was then sandwiched between two layers of silicone elastomer that had protrusions and inner cavities to concentrate stresses and strains and increase the detection resolution. Furthermore, the tactile sensor was characterized under static and dynamic force loading. Over 180,000 cycles of indentation were conducted to establish its durability and repeatability. The results demonstrate that the SkinCell has an average spatial resolution of 0.827 mm, an average sensitivity of 0.328 mΩ/Ω/N, expressed as the change in resistance per force in Newtons, an average sensitivity of 1.795 µV/N at a loading pressure of 2.365 PSI, and a dynamic response time constant of 63 ms which make it suitable for both large area skins and fingertip human-robot interaction applications.

Antagonistic Magneto-Rheological Actuators with Inherent Output Boundedness: An Ideal Solution for High-Performance and Human-Safe Actuation

This paper studies the working principles of antagonistic magneto-rheological (MR) actuators, i.e., a combination of an electric motor and a pair of MR clutches in an antagonistic configuration, for compliant actuation in robotics. The study focuses on the unique boundedness property exhibited by MR actuators, which limits the output torques delivered to the load, independent of the received input torque and/or control commands. This inherent property is of significant importance for ensuring human safety in human–robot interaction applications. Through a comprehensive analysis, we provide analytical proof of the inherent output boundedness of antagonistic MR actuators and validate our findings through experimental results. Our research demonstrates that these actuators are well-suited for safe operations in robotic applications, eliminating the need for additional sensor measurements or complex control strategies. This promising capability enables the avoidance of trade-offs between actuator performance, complexity, and cost. The insights gained from this study contribute to advancing compliant actuation technology, paving the way for high-performance and human-safe robotic systems.

End-to-End Speaker Profiling Using 1D CNN Architectures and Filter Bank Initialization

The automatic estimation of speaker characteristics, such as height, age, and gender, has various applications in forensics, surveillance, customer service, and many human-robot interaction applications. These applications are often required to produce a response promptly. This work proposes a novel approach to speaker profiling by combining filter bank initializations, such as continuous wavelets and gammatone filter banks, with one-dimensional (1D) convolutional neural networks (CNN) and residual blocks. The proposed end-to-end model goes from the raw waveform to an estimated height, age, and gender of the speaker by learning speaker representation directly from the audio signal without relying on handcrafted and pre-computed acoustic features. The conducted experiments on the TIMIT dataset show that the proposed approach outperforms many previous studies on speaker profiling with a mean absolute error (MAE) of 5.18 and 4.91 cm in height estimation and MAE of 5.36 and 6.07 years in age estimation for males and females, respectively, and achieving an accuracy of 99.98% in gender prediction.

Safe High Stiffness Impedance Control for Series Elastic Actuators using Collocated Position Feedback

Physical human-robot interaction applications requiring safety and compliance characteristics usually employ mechanical devices such as series elastic actuators (SEAs). So far, impedance control schemes for SEAs have been investigated to overcome the limited displayable stiffness at the end point, represented by the stiffness of a physical spring, in a passive manner. SEAs usually mount two encoders on both the motor and load sides for position measurements. However, in several applications, the installation of a load-side encoder is not feasible from both cost and manufacturing perspectives. In this scenario, a Kalman Filter based on micro electro mechanical system accelerometers has been proposed for estimating the external force and load side quantities, without the need for load-side encoders. Herein, a novel impedance control scheme leveraging this approach is proposed to effectively overcome the stiffness of the physical spring while maintaining passivity at the end point. Compared with conventional load-side encoder-based impedance control schemes, the proposed control scheme presents comparable performance, as demonstrated by simulation and experimental results.

Outperformance of Mall-Receptionist Android as Inverse Reinforcement Learning is Transitioned to Reinforcement Learning

Robots can tackle human–robot interaction (HRI) tasks through inverse reinforcement learning (IRL). However, offline IRL agents' performance is upper-bounded by experts. Limited demonstration fails to provide an overall picture of the environment, especially in real-world applications. To further enhance IRL's performance, we implement a cross-modal inverse reinforcement learning to reinforcement learning (IRL-to-RL) transition framework for a real-world HRI interaction task, in which a mall receptionist android promotes sanitizer usage. During the 10-day experiment, the android develops a more proactive and effective strategy than the human expert. Furthermore, we explore four decay modes of prior knowledge supervision and suggest a preferable pattern for practical use. Our results demonstrate the feasibility of the framework to assist robots in switching to diverse modalities, learning incrementally with a sparse reward function, and eventually outperforming the human expert. We anticipate our framework to inspire more IRL-to-better learning paradigms, facilitating robots to outcompete teachers in more real-world HRI applications.

Hi-ROS: Open-source multi-camera sensor fusion for real-time people tracking

This paper presents Hi-ROS (Human Interaction in ROS), an open source framework focused on real-time accurate assessment of human motion. The system offers a series of tools to track multiple people in real-time by exploiting a calibrated camera network. No assumptions are made about the typology or number of cameras, nor about the body pose estimation algorithm used to extract the 3D poses of the people in the scene. The tools provided by Hi-ROS include a Skeleton Tracker to ensure temporal consistency of the detected poses, a Skeleton Merger to fuse the tracks from multiple cameras, thus limiting flickering phenomena, a Skeleton Optimizer to ensure limb length consistency, and a Skeleton Filter to perform real-time smoothing of the detected joint trajectories. Accuracy, tracking robustness, and real-time performance of the proposed system were evaluated on a public dataset, containing both single-person and multi-person sequences with up to 4 people interacting. The results obtained using different subsets of the proposed tools show how the complete Hi-ROS pipeline provides accurate and reliable estimates also in challenging scenarios, with a reduction of the RMSE of up to 27% with respect to a pure tracking approach. This work aims to push forward the development of unobtrusive human–robot interaction applications, multi-person automated posture analyses, rehabilitation performance assessments, and any possible application enabled by real-time accurate assessment of human motion via markerless motion capture.

A multidimensional taxonomy for human-robot interaction in construction

Despite the increased interest in construction robotics both in academia and the industry, insufficient attention has been given to aspects related to Human-Robot Interaction (HRI). Characterizing HRI for construction tasks can help researchers organize knowledge in a structured manner that allows for classifying construction robotics applications and comparing and benchmarking different studies. This paper builds upon existing taxonomies and empirical studies in HRI in various industries (e.g., construction, manufacturing, and military, among others) to propose a multidimensional taxonomy to characterize HRI applications in the construction industry. The taxonomy design followed a systematic literature review in which common themes were identified and grouped into 16 categories. The proposed taxonomy can be used as a foundation for systematic reviews and meta-analyses of HRI applications in construction and can benefit the construction industry by informing the design of collaborative tasks performed by human-robot teams.

SkinCell: A Modular Tactile Sensor Patch for Physical Human–Robot Interaction

In this article, we present the design, fabrication, integration, and performance evaluation of SkinCell sensors, a novel tactile array targeting human–robot interaction applications. A polyimide sheet is selected as the substrate of the sensor, and gold electrodes with circular interdigitated finger patterns are deposited through dc sputtering, creating a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula> 4 sensor array with 16 tactile elements. Sensing elements are completed with a thin layer of spin-coated poly (3,4-ethylenedioxythiophene) polystyrene sulfonate (PEDOT: PSS) organic polymer solution. Two individual sensor sheets are being laminated in a back-to-back fashion to create a half-Wheatstone-bridge configuration to reduce temperature coupling while improving sensitivity. The sensor array is then sandwiched between two layers of silicone elastomer with internal dimples and cavities to improve sensitivity, repeatability, and center of force detection resolution. A COMSOL simulation was performed to study the influence of adding the silicone encapsulation as well as design parameters to the sensor substrate. Our simulation results indicate an effective improvement of touch, i.e., voltage sensitivity by employing the featured encapsulation. Characterization experiments have been conducted to identify the sensitivity profile of each sensor on the array, which indicates an average sensitivity of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$26.5~\mu $ </tex-math></inline-formula> V/N. In subsequent experiments, the calibration profiles are used to identify the resultant force during pressing, as well as the location of the center of pressure (COP) on the SkinCell sensor. Finally, SkinCell sensors have been integrated into the OctoCan, a structural electronic device that can acquire squeezing pressure data during physical human–robot interaction (pHRI).

Integrating Virtual, Mixed, and Augmented Reality to Human–Robot Interaction Applications Using Game Engines: A Brief Review of Accessible Software Tools and Frameworks

This article identifies and summarizes software tools and frameworks proposed in the Human–Robot Interaction (HRI) literature for developing extended reality (XR) experiences using game engines. This review includes primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can control or interact with real robotic platforms using devices that extend the user’s reality. The objective of this article is not to present an extensive list of applications and tools. Instead, we present recent, relevant, common, and accessible frameworks and software tools implemented in research articles published in high-impact robotics conferences and journals. For this, we searched papers published during a seven-years period between 2015 and 2022 in relevant databases for robotics (Science Direct, IEEE Xplore, ACM digital library, Springer Link, and Web of Science). Additionally, we present and classify the application context of the reviewed articles in four groups: social robotics, programming of industrial robots, teleoperation of industrial robots, and Human–Robot collaboration (HRC).

Virtual reality in human-robot interaction: Challenges and benefits

&lt;abstract&gt; &lt;p&gt;Virtual reality (VR) technology has been increasingly employed in human-robot interaction (HRI) research to enhance the immersion and realism of the interaction. However, the integration of VR into HRI also introduces new challenges, such as latency, mismatch between virtual and real environments and potential adverse effects on human users. Despite these challenges, the use of VR in HRI has the potential to provide numerous benefits, including improved communication, increased safety and enhanced training and education. Yet, little research has been done by scholars to review the state of the art of VR applications in human-robot interaction. To bridge the gap, this paper provides an overview of the challenges and benefits of using VR in HRI, as well as current research in the field and future directions for development. It has been found that robots are getting more personalized, interactive and engaging than ever; and with the popularization of virtual reality innovations, we might be able to foresee the wide adoption of VR in controlling robots to fulfill various tasks of hospitals, schools and factories. Still, there are several challenges, such as the need for more advanced VR technologies to provide more realistic and immersive experiences, the development of more human-like robot models to improve social interactions and the need for better methods of evaluating the effectiveness of VR in human-robot interaction.&lt;/p&gt; &lt;/abstract&gt;

Sentiment Analysis in Turkish Question Answering Systems: An Application of Human-Robot Interaction

Sentiment Analysis in Turkish Question Answering Systems: An Application of Human-Robot Interaction

Iterative assist-as-needed control with skill learning for physical human-robot interaction

Collaborative robots referring to physical human-robot interaction applications have drawn considerable attention. There have been lots of methods based on skill learning to help robots mimic human collaborative behaviors. However, the stiffness profiles generated by these methods only rely on the quality of collected human demonstrations and cannot be adjusted according to detailed interaction tasks. To this end, this paper combines a skill learning method and an iterative assist-as-needed impedance controller. The skill learning method encodes and generates position and stiffness references. The iterative assist-as-needed controller enables the robot to adjust the generated stiffness profile actively according to the human intention. When the human tries to act as an active role in interaction tasks, the controller takes a lower stiffness parameter, which means it can tolerate more deviation from the desired trajectory, and vice versa. Compared to the previous results, on the one hand, this paper employs a skill learning method to generate an initial stiffness profile for the iterative assist-as-needed controller, which enables the robot to mimic human behaviors; on the other hand, the controller proposed in this paper allows the manual design of the damping and mass parameters while the previous controller accepts only the desired stiffness parameter. Finally, one simulation and one experiment are demonstrated to evaluate the performance of the designed controller.

Safe Learning-Based Control of Elastic Joint Robots via Control Barrier Functions

Ensuring safety is of paramount importance in physical human-robot interaction applications. This requires both adherence to safety constraints defined on the system state, as well as guaranteeing compliant behavior of the robot. If the underlying dynamical system is known exactly, the former can be addressed with the help of control barrier functions. The incorporation of elastic actuators in the robot's mechanical design can address the latter requirement. However, this elasticity can increase the complexity of the resulting system, leading to unmodeled dynamics, such that control barrier functions cannot directly ensure safety. In this paper, we mitigate this issue by learning the unknown dynamics using Gaussian process regression. By employing the model in a feedback linearizing control law, the safety conditions resulting from control barrier functions can be robustified to take into account model errors, while remaining feasible. In order to enforce them on-line, we formulate the derived safety conditions in the form of a second-order cone program. We demonstrate our proposed approach with simulations on a two-degree-of-freedom planar robot with elastic joints.