Human-robot Interface Research Articles

Competencies for Human-Robot Collaboration in the Construction Industry – Academia’s Perspective

ABSTRACT This paper investigates the perceptions of Construction Engineering and Management instructors regarding the competencies for Human-Robot Collaboration (HRC) in construction to determine knowledge areas, skills, and abilities prioritized for facilitating effective collaboration between humans and robots in the industry. A two-round Delphi study was employed to evaluate the perceptions of construction instructors regarding HRC competencies. This study’s findings revealed that human-robot collaboration knowledge areas prioritized by the instructors include HRC ethics and regulation, robot anatomy and technical specifications, construction robot applications, sensors, and task planning. The instructors prioritized skills such as task planning, application of machine learning algorithms, safety management, human-robot interface proficiency, and effective communication. HRC abilities prioritized include decision-making, continuous learning, critical thinking, attention to detail, analytical aptitude, and adaptability. This research established the competencies prioritized by the instructors for implementing HRC in the construction industry and recommended future research directions.

Electronic Skin as Human–Robot Interface for Human Grasp Recognition in Home-Care Robots: Introducing a Complete Set of Resources

Population aging has highlighted the significance of developing home-care robots to assist human life. However, traditional home-care robots work according to fixed procedures, which lack human–robot interaction to mimic human hand action. Human–robot interfaces provide an opportunity for interactions between robots and users, which play an important role in home-care robots. Traditional human–robot interfaces are based on rigid, cumbersome, and high-priced machines, which prevent their wide application in performing home-care tasks. An e-skin sensor with 11 stretchable resistors evenly distributed on the back of the human hand, which could be applied to monitor human finger motion and control robotic fingers to dexterously grasp diverse objects, is presented in this article. A 3D printer is employed to pattern high-performance wires into structurally soft and stretchable filamentary serpentines. The stretchable resistors are fabricated by a thin sensing film of mixed multiwalled carbon nanotubes (MWCNTs) and silicone elastomer. The e-skin sensor is applied in a hand-in-the-loop system for controlling a robotic hand to help grasp an object synchronously. A robot system is developed to connect the e-skin with the robotic hand to perform grasp actions with a success rate of more than 80%. By analyzing the experimental results of the e-skin sensor and robotic hand grasp, this work obtains insights showing that the skin sensor can help engineers design a home-care assistant plan for patients according to monitoring value or a human–robot interaction mode by robotic fingers mimicking human finger actions.

Considering Socially Scalable Human-Robot Interfaces

Collaborative robots are becoming increasingly present in everyday life, with applications ranging food and parcel delivery, security, and more. They can offer great value propositions for organizations and consumers. However, most people lack knowledge of how to interact with robots, and many robots themselves necessitate formal training which can be inaccessible to many and thus not societally scalable. Further, there is a lack of existing work investigating interfaces designs that can support non-dyadic interactions consisting of two or more individuals interacting with a robot sequentially and simultaneously; such interactions will be common in real-world usage. This research explores the efficacy of natural language interfaces for human-robot interaction through a human experiment and post-experiment survey. Results show that the natural language interface can afford teams enhanced capabilities to share robot control and avoid errors relative to other interfaces, while also increasing user perceptions towards overall interaction.

Ultrastretchable Segmented Sensors for Functional Human-Machine Interfaces.

The key feature that enables soft sensors to shorten the performance gap between robots and biological structures is their deformability, coupled with their capability to measure physical changes. Robots equipped with these sensors can interact safely and proprioceptively with their environments. This has sparked interest in developing novel sensors with high stretchability for application in human-robot interactions. This study presents a novel ultrasoft optoelectronic segmented sensor design capable of measuring strains exceeding 500%. The sensor features an ultrastretchable segment physically joined with an asymmetrically configured soft proprioceptive segment. This configuration enables it to measure high strain and to detect both the magnitude and direction of bending. Although the sensor cannot decouple these types of deformations, it can sense prescribed motions that combine stretching and bending. The proposed sensor was applied to a highly deformable scissor mechanism and a human-robot interface (HRI) device for a robotic arm, capable of quantifying parameters in complex interactions. The results from the experiments also demonstrate the potential of the proposed segmented sensor concept when used in tandem with machine learning, affording new dimensions of proprioception to robots during multilayered interactions with humans.

Assessment of Surgeons' Stress Levels with Digital Sensors during Robot-Assisted Surgery: An Experimental Study.

Robot-Assisted Minimally Invasive Surgery (RAMIS) marks a paradigm shift in surgical procedures, enhancing precision and ergonomics. Concurrently it introduces complex stress dynamics and ergonomic challenges regarding the human-robot interface and interaction. This study explores the stress-related aspects of RAMIS, using the da Vinci XI Surgical System and the Sea Spikes model as a standard skill training phantom to establish a link between technological advancement and human factors in RAMIS environments. By employing different physiological and kinematic sensors for heart rate variability, hand movement tracking, and posture analysis, this research aims to develop a framework for quantifying the stress and ergonomic loads applied to surgeons. Preliminary findings reveal significant correlations between stress levels and several of the skill-related metrics measured by external sensors or the SURG-TLX questionnaire. Furthermore, early analysis of this preliminary dataset suggests the potential benefits of applying machine learning for surgeon skill classification and stress analysis. This paper presents the initial findings, identified correlations, and the lessons learned from the clinical setup, aiming to lay down the cornerstones for wider studies in the fields of clinical situation awareness and attention computing.

Innovative Metaheuristic Optimization Approach with a Bi-Triad for Rehabilitation Exoskeletons.

The present work proposes a comprehensive metaheuristic methodology for the development of a medical robot for the upper limb rehabilitation, which includes the topological optimization of the device, kinematic models (5 DOF), human-robot interface, control and experimental tests. This methodology applies two cutting-edge triads: (1) the three points of view in engineering design (client, designer and community) and (2) the triad formed by three pillars of Industry 4.0 (autonomous machines and systems, additive manufacturing and simulation of virtual environments). By applying the proposed procedure, a robotic mechanism was obtained with a reduction of more than 40% of its initial weight and a human-robot interface with three modes of operation and a biomechanically viable kinematic model for humans. The digital twin instance and its evaluation through therapeutic routines with and without disturbances was assessed; the average RMSEs obtained were 0.08 rad and 0.11 rad, respectively. The proposed methodology is applicable to any medical robot, providing a versatile and effective solution for optimizing the design and development of healthcare devices. It adopts an innovative and scalable approach to enhance their processes.

Developing a Natural Language Understanding System for Dealing with the Sequencing Problem in Simulating Brain Damage

This paper is an attempt to show how a Human-Robot Interface (HRI) system in the Greek language can help people with brain damage in speech and its related perception issues. This proposal is not the product of research conducted on how to treat brain injuries. It is a conclusion stemming from research on intelligent Human-Robot interfaces, as a part of Artificial Intelligence and Natural Language Processing, which approaches the processing and understanding of natural language with specific methods. For the same reason, experiments on real patients have not been conducted. Thus, this paper does not propose a competing method, but a method for further study. Since it is referring to a very general and quite complex issue, an approach is presented here for the Sequencing problem. A person with such a problem cannot hierarchically organize the tasks needed to be performed. This Hierarchy has to do with both time and practicality. The particular problem here, as much as the innovation of our approach, lies not when there are explicit temporally defined instructions, but in the ability to derive these temporal values through the person’s perception from more vague temporal references. The present approach is developed based on our related previous works for deploying a robotic system that relies on Hole Semantics and the OMAS-III computational model as a grammatical formalism for its communication with humans.

SEMG-Based Robust Recognition of Grasping Postures with a Machine Learning Approach for Low-Cost Hand Control.

The design and control of artificial hands remains a challenge in engineering. Popular prostheses are bio-mechanically simple with restricted manipulation capabilities, as advanced devices are pricy or abandoned due to their difficult communication with the hand. For social robots, the interpretation of human intention is key for their integration in daily life. This can be achieved with machine learning (ML) algorithms, which are barely used for grasping posture recognition. This work proposes an ML approach to recognize nine hand postures, representing 90% of the activities of daily living in real time using an sEMG human-robot interface (HRI). Data from 20 subjects wearing a Myo armband (8 sEMG signals) were gathered from the NinaPro DS5 and from experimental tests with the YCB Object Set, and they were used jointly in the development of a simple multi-layer perceptron in MATLAB, with a global percentage success of 73% using only two features. GPU-based implementations were run to select the best architecture, with generalization capabilities, robustness-versus-electrode shift, low memory expense, and real-time performance. This architecture enables the implementation of grasping posture recognition in low-cost devices, aimed at the development of affordable functional prostheses and HRI for social robots.

A novel human-robot interface based on low-cost sEMG device designed for the collaborative wearable robot

Currently, the acquisition of surface electromyographic (sEMG) signals commonly requires powerful and expensive equipment or systems. The signal analysis also requires additional equipment to process independently, which is time-consuming and the results are difficult to understand and directly apply to wearable robots. Therefore, this paper developed a sEMG signal device with a simple structure and high-performance cost ratio. The hardware architecture consists of a two-stage small-size amplifier with a total gain of 1000 and a band-pass filter from 10 to 150 HZ. Since real measured sEMG signals are usually corrupted by various types of noises, this paper designed an envelope processing to obtain the sEMG envelope signal which has stronger anti-interference ability than the raw signal, and it is also easier to apply to wearable robots. The proposed system is evaluated quantitatively by the experimental test and electrode groups analysis. In addition, the feasibility and effectiveness of the system is performed by measuring the sEMG signal of the forearm muscle group of four volunteers during motion action and extract the time domain feature to control a wearable robot. The sEMG envelope successfully manipulates the wearable robot with four different actions, whose precision is as high as a conventional sEMG device. The proposed system is promising in providing the reliable control interface of a wearable robot based on the sEMG.

Development of a Human–Robot Interface for Cobot Trajectory Planning Using Mixed Reality

The growing demand for projects with collaborative robots, known as “cobots”, underlines the need to efficiently address the execution of tasks with speed and flexibility, without neglecting safety in human–robot interaction. In general terms, this practice requires knowledge of robotics programming and skill in the use of hardware. The proposed solution consists of a mixed reality (MR) application integrated into a mixed reality head-mounted device (HMD) that accelerates the process of programming the complex manoeuvres of a cobot. This advancement is achieved through voice and gesture recognition, in addition to the use of digital panels. This allows any user, regardless of his or her robotics experience, to work more efficiently. The Robot Operating System (ROS) platform monitors the cobot and manages the transfer of data between the two. The system uses QR (Quick Response) codes to establish a precise frame of reference. This solution has proven its applicability in industrial processes, by automating manoeuvres and receiving positive feedback from users who have evaluated its performance. This solution promises to revolutionize the programming and operation of cobots, and pave the way for efficient and accessible collaborative robotics.

Three‐dimensional mapping and immersive human–robot interfacing utilize Kinect‐style depth cameras and virtual reality for agricultural mobile robots

AbstractNavigation and autonomous operation in farmland or greenhouse environments present significant challenges for agricultural mobile robots on account of unstructured scene conditions and complex task requirements. The immersive interfaces based on human–robot interaction have been extensively developed, which bridge the high‐level decision of human beings and the precise sensing and high motion ability of robots. In this study, we propose a human–robot interaction framework based on three‐dimensional mapping and virtual reality (VR) visualization. On the construction client, dense three‐dimensional point‐cloud maps are created based on the simultaneous localization and mapping real‐time appearance‐based mapping algorithm with a Kinect‐style depth camera. Subsequently, point‐cloud filtering and mesh creation methods are introduced for postprocessing the map models. A data communication network is employed to transmit the optimized models to the VR‐based exploration client. Utilizing VR visualization, the operator can gain an intuitive and comprehensive understanding of the environments to be explored. The three‐dimensional agricultural scenes are mapped into VR models through the system, which can combine the physical worlds with the VR spaces in a better manner. Compared with the video streaming‐based method for three‐dimensional mapping, the map models with colors and textures are entirely displayed in the VR headset, thus providing robust and global human–robot interaction interfaces and enhancing the immersive exploration for human beings to a considerable extent. Experimental results indicate that the three‐dimensional map models exhibit relatively sufficient construction integrity and yield favorable optimization effects. Additionally, according to a user questionnaire survey, immersive VR‐based interfaces can be more effectively utilized for the three‐dimensional mapping of mobile robots. Research evidence and results demonstrate that our proposed system framework offers positive assistance and support for enhancing human–robot interaction and executing specific robotic tasks in unstructured field agricultural environments. Code implementation and related data sets are shared at the link https://github.com/WangDongBUAA/Mapping_Interfacing.

Neuromuscular Interfacing for Advancing Kinesthetic and Teleoperated Programming by Demonstration of Collaborative Robots.

This study addresses the challenges of Programming by Demonstration (PbD) in the context of collaborative robots, focusing on the need to provide additional degrees of programming without hindering the user's ability to demonstrate trajectories. PbD enables an intuitive programming of robots through demonstrations, allowing non-expert users to teach robot skills without coding. The two main PbD modalities, observational and kinesthetic, have limitations when it comes to programming the diverse functionalities offered by modern collaborative robots. To overcome these limitations, the study proposes the use of a wearable human-robot interface based on surface Electromyography (sEMG) to measure the forearm's muscle co-contraction level, enabling additional programming inputs through hand stiffening level modulations without interfering with voluntary movements. Vibrotactile feedback enhances the operator's understanding of the additional programming inputs during PbD tasks. The proposed approach is demonstrated through experiments involving a collaborative robot performing an industrial wiring task. The results showcase the effectiveness and intuitiveness of the interface, allowing simultaneous programming of robot compliance and gripper grasping. The framework, applicable to both teleoperation and kinesthetic teaching, demonstrated effectively in an industrial wiring task with a 100% success rate over the group of subjects. Furthermore, the presence of vibortactile feedback showed an average decrease of programming errors of 33%, and statistical analyses confirmed the subjects' ability to correctly modulate co-contraction levels. This innovative framework augments programming by demonstration by integrating neuromuscular interfacing and introducing structured programming logics, providing an intuitive human-robot interaction for programming both gripper and compliance in teleoperation and kinesthetic teaching.

Human-Robot Joint Misalignment, Physical Interaction, and Gait Kinematic Assessment in Ankle-Foot Orthoses.

Lower limb exoskeletons and orthoses have been increasingly used to assist the user during gait rehabilitation through torque transmission and motor stability. However, the physical human-robot interface (HRi) has not been properly addressed. Current orthoses lead to spurious forces at the HRi that cause adverse effects and high abandonment rates. This study aims to assess and compare, in a holistic approach, human-robot joint misalignment and gait kinematics in three fixation designs of ankle-foot orthoses (AFOs). These are AFOs with a frontal shin guard (F-AFO), lateral shin guard (L-AFO), and the ankle modulus of the H2 exoskeleton (H2-AFO). An experimental protocol was implemented to assess misalignment, fixation displacement, pressure interactions, user-perceived comfort, and gait kinematics during walking with the three AFOs. The F-AFO showed reduced vertical misalignment (peak of 1.37 ± 0.90 cm, p-value < 0.05), interactions (median pressures of 0.39-3.12 kPa), and higher user-perceived comfort (p-value < 0.05) when compared to H2-AFO (peak misalignment of 2.95 ± 0.64 and pressures ranging from 3.19 to 19.78 kPa). F-AFO also improves the L-AFO in pressure (median pressures ranging from 8.64 to 10.83 kPa) and comfort (p-value < 0.05). All AFOs significantly modified hip joint angle regarding control gait (p-value < 0.01), while the H2-AFO also affected knee joint angle (p-value < 0.01) and gait spatiotemporal parameters (p-value < 0.05). Overall, findings indicate that an AFO with a frontal shin guard and a sports shoe is effective at reducing misalignment and pressure at the HRI, increasing comfort with slight changes in gait kinematics.

Addressing the hazards of remote inspection techniques: a safety-net for vessel surveys

ABSTRACT Emerging technologies and technologies with emerging applications are stark catalysts of transformation in the maritime industry. As the industry progressively shifts to remote inspection techniques (RIT), there is a growing need to identify and assess the potential hazards that exist within the breadth and scope of vessel surveys and inspections. Such an assessment is essential bearing in mind the limitations, drawbacks and negative externalities pertaining to robotic platforms that are being integrated into the traditional manual-driven system. Moving the RIT-agenda forward entails optimisation of the human-robotic interface (free from seen and unforeseen impacts) that remains a prerequisite to the conduct of classification and statutory surveys. Drawing on findings from in-depth interviews with subject matter experts, this article makes recommendations for an RIT survey ‘safety net’ covering pre, ongoing and post operational aspects, and addressing data governance and psychosocial hazards as well as the hazards arising from the absence of a common legal liability framework.

A smart walker based on a hybrid motion model and machine learning method

Due to the degeneration of musculoskeletal structure and strength, the elderly population is facing significant mobility challenges. Walkers are widely used for mobility-impaired people. In this research, a smart walker has been developed by using a hybrid motion model (HMM) and a machine learning model based on the vertical interaction force to match the velocity between the walker and its user, which can reduce the horizontal interaction force thus to help lower the operational effort. It is equipped with only force sensors on the handlebars as a human-robot interface. The human motion is modeled by using the HMM in which an inverted pendulum model is used in the single support phase and a constant velocity model is used in the double support phase. A novel gait phase detector is built based on a machine learning method to correlate the force information and the walking gait phases. Experiments with three subjects are carried out to verify the effectiveness of the gait phase detection method and the hybrid motion model. The results demonstrate that the walker can accurately identify gait phases with more than 90 % accuracy and the horizontal interaction force is reduced to around 50 % of that under the constant velocity model and power-off condition when using the HMM during walking.

A Novel Physical Human–Robot Interface With Pressure Distribution Measurement Based on Electrical Impedance Tomography

A Novel Physical Human–Robot Interface With Pressure Distribution Measurement Based on Electrical Impedance Tomography

Highly Stretchable and Sensitive Strain Sensor Based on Porous Materials and Rhombic‐Mesh Structures for Robot Teleoperation

AbstractWearable sensors for human motion capture offer a promising human–robot interface to control robots in the teleoperation scenario, where robots could function as the second body of human operators to fulfill tasks remotely and accurately. In this paper, a novel strain sensor based on a soft polyurethane (PU) sponge and carbon nanotubes (CNT) is designed for motion capture of human joints. The unique 3D porous microstructure of the PU‐CNT sponge provides the sensor with high sensitivity. To bridge the gap between the high sensitivity and high stretchability of the strain sensor, a rhombic‐mesh structure with optimized geometric parameters, in conjunction with a pre‐compression design, is proposed for strain sensor prototyping, which endows the sensor with an extra elongation rate during the stretching process. The proposed PU‐CNT strain sensor manifests promising sensing performance with a stretchability of up to 300% and a maximum gauge factor of 3893, together with long‐term durability, low detection limit, and fast response capacity. Finally, the validation of the strain sensor is carried out by deploying the sensor on a human elbow to realize the teleoperation of a robot arm, which could be monitored through the digital twin model of the robot in a real‐time manner.

An Effective Head-Based HRI for 6D Robotic Grasping Using Mixed Reality

People with two-arm disabilities need to grasp various densely placed objects in their daily life. However, current arm-free human-robot interfaces (HRIs), such as language-based and gaze-based HRI, are difficult to effectively control the robotic arm to complete the above task. To achieve this task effectively, we innovatively propose an arm-free HRI based on Mixed Reality (MR) feedback and head control, which enables people to give full play to their intelligence in correcting robot perception errors and determining grasp decision-making. MR feedback makes 3D visualization of robot perception results and virtual grippers aligned with real grippers. Head control ensures these disabled people can flexibly and effectively control the gripper's 6D pose. In experiments, we densely place 15 objects of different sizes and shapes from daily life, including transparent and specular objects, such as a measuring tape, transparent bottles, and a charging head, which are grasped by 10 subjects using our HRI. Our experiment results and comparison results show that our HRI can more effectively complete the task, has the adaptability for unseen object grasping, and has high point cloud error tolerance. Presentation video is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://youtu.be/QO72XT8BYGs</uri> .

Inquiring Natural Language Processing Capabilities on Robotic Systems through Virtual Assistants: A Systemic Approach

This paper attempts to approach the interface of a robot from the perspective of virtual assistants. Virtual assistants can also be characterized as the mind of a robot, since they manage communication and action with the rest of the world they exist in. Therefore, virtual assistants can also be described as the brain of a robot and they include a Natural Language Processing (NLP) module for conducting communication in their human-robot interface. This work is focused on inquiring and enhancing the capabilities of this module. The problem is that nothing much is revealed about the nature of the human-robot interface of commercial virtual assistants. Therefore, any new attempt of developing such a capability has to start from scratch. Accordingly, to include corresponding capabilities to a developing NLP system of a virtual assistant, a method of systemic semantic modelling is proposed and applied. For this purpose, the paper briefly reviews the evolution of virtual assistants from the first assistant, in the form of a game, to the latest assistant that has significantly elevated their standards. Then there is a reference to the evolution of their services and their continued offerings, as well as future expectations. The paper presents their structure and the technologies used, according to the data provided by the development companies to the public, while an attempt is made to classify virtual assistants, based on their characteristics and capabilities. Consequently, a robotic NLP interface is being developed, based on the communicative power of a proposed systemic conceptual model that may enhance the NLP capabilities of virtual assistants, being tested through a small natural language dictionary in Greek.

Research on Artificial Intelligence based Fruit Disease Identification System (AI-FDIS) with the Internet of Things (IoT)

Monitoring fruit quality, volume, and development on the plantation are critical to ensuring that the fruits are harvested at the optimal time. Fruits are more susceptible to the disease while they are actively growing. It is possible to safeguard and enhance agricultural productivity by early detection of fruit diseases. A huge farm makes it tough to inspect each tree to learn about its fruit personally. There are several applications for image processing with the Internet of Things (IoT) in various fields. To safeguard the fruit trees from illness and weather conditions, it is difficult for the farmers and their workers to regularly examine these large areas. With the advent of Precision Farming, a new way of thinking about agriculture has emerged, incorporating cutting-edge technological innovations. One of the modern farmers’ biggest challenges is detecting fruit diseases in their early stages. If infections aren’t identified in time, farmers might see a drop in income. Hence this paper is about an Artificial Intelligence Based Fruit Disease Identification System (AI-FDIS) with a drone system featuring a high-accuracy camera, substantial computing capability, and connectivity for precision farming. As a result, it is possible to monitor large agricultural areas precisely, identify diseased plants, and decide on the chemical to spray and the precise dosage to use. It is connected to a cloud server that receives images and generates information from these images, including crop production projections. The farm base can interface with the system with a user-friendly Human-Robot Interface (HRI). It is possible to handle a vast area of farmland daily using this method. The agricultural drone is used to reduce environmental impact and boost crop productivity.