Safe Human-robot Interaction Research Articles

Attention-based hand pose estimation with voting and dual modalities

Hand pose estimation has recently emerged as a compelling topic in the robotic research community, because of its usefulness in learning from human demonstration or safe human–robot interaction. Although deep learning-based methods have been introduced for this task and have shown promise, it remains a challenging problem. To address this, we propose a novel end-to-end architecture for hand pose estimation using red-green-blue (RGB) and depth (D) data (RGB-D). Our approach processes the two data sources separately and utilizes a dense fusion network with an attention module to extract discriminative features. The features extracted include both spatial information and geometric constraints, which are fused to vote for the hand pose. We demonstrate that our voting mechanism in conjunction with the attention mechanism is particularly useful for solving the problem, especially when hands are heavily occluded by objects or are self-occluded. Our experimental results on benchmark datasets demonstrate that our approach outperforms state-of-the-art methods by a significant margin.

Bending and sensing performances of electro-ionic soft actuators based on carboxylated cellulose nanofibers reinforced with graphene nanoplatelets

Abstract Soft robots not only possess greater degrees of freedom and the capability for continuous transformation, but they also offer exceptionally high safety in human-robot interactions, avoiding harm to the human body. Soft actuators are essential for developing high-performance soft robots, offering significant bending deformation, rapid response times, and prolonged operational capabilities. Herein, we present an ionic electroactive soft actuator based on functional cellulose nanofibers, graphene nanoplatelets, and ionic liquid. The proposed actuator achieved a large displacement about ±8 mm under 2.0 V at 0.1 Hz, with long working stability (98% of initial peak displacement maintained after 1260 cycles of cycling). The human-robot interaction applications of this actuator were explored by simulating human fingers. More importantly, the static and dynamic sensing performances of the actuator were investigated, finding that it generated a sensing voltage of 0.37 V at a vibration displacement of only 1.75 mm. The designed actuator provides a promising approach for developing high-performance soft robots, soft actuators, flexible sensors, and flexible active devices.

Textile Fabric Defect Detection Using Enhanced Deep Convolutional Neural Network with Safe Human–Robot Collaborative Interaction

The emergence of modern robotic technology and artificial intelligence (AI) enables a transformation in the textile sector. Manual fabric defect inspection is time-consuming, error-prone, and labor-intensive. This offers a great possibility for applying more AI-trained automated processes with safe human–robot interaction (HRI) to reduce risks of work accidents and occupational illnesses and enhance the environmental sustainability of the processes. In this experimental study, we developed, implemented, and tested a novel algorithm that detects fabric defects by utilizing enhanced deep convolutional neural networks (DCNNs). The proposed method integrates advanced DCNN architectures to automatically classify and detect 13 different types of fabric defects, such as double-ends, holes, broken ends, etc., ensuring high accuracy and efficiency in the inspection process. The dataset is created through augmentation techniques and a model is fine-tuned on a large dataset of annotated images using transfer learning approaches. The experiment was performed using an anthropomorphic robot that was programmed to move above the fabric. The camera attached to the robot detected defects in the fabric and triggered an alarm. A photoelectric sensor was installed on the conveyor belt and linked to the robot to notify it about an impending fabric. The CNN model architecture was enhanced to increase performance. Experimental findings show that the presented system can detect fabric defects with a 97.49% mean Average Precision (mAP).

Operational space robust impedance control of the redundant surgical robot for minimally invasive surgery.

The motion accuracy, compliance, and control smoothness for the surgical robot are of great importance to improve the safety of human-robot interaction. However, the end effector that interacts with soft tissue during surgery affects the dynamics of the robot. The control performance of the controller may be decreased if the changing dynamics are not identified and updated in time. This paper proposes a robust impedance controller for the redundant remote center of motion manipulator influenced by external disturbances, including external torque, uncertainties, and unmodeled terms in the dynamics. To achieve the desired impedance, a continuously switching sliding manifold is proposed. When the sliding manifold is driven to zero, the motion error will converge to a bounded region. This can overcome the adverse effects of external disturbances while guaranteeing motion accuracy and compliance. Chattering of the sliding mode control is alleviated through the formulated continuously switching sliding manifold and integrated nonlinear disturbance observer. Simulations and experiments demonstrate that the proposed controller has excellent motion accuracy, compliance, and control smoothness. This provides potential application prospects for the redundant surgical robot to guarantee safe human-robot interaction.

Multipoint Variable parameter compliant control of redundant manipulator based on the equivalent twin model of flexible contact dynamics

To elucidate the dynamic coupling mechanism involved in multipoint/arbitrary point contact during human–robot interactions with redundant manipulators, we introduce a compliant control strategy that adapts to variable parameters. This strategy is based on the flexible contact dynamic equivalent twin system for human–robot interactions. Using the Virtual Elastic Element Representation model for arbitrary point contact dynamics and leveraging the skeleton principle for contact force conversion, we construct a flexible contact dynamic equivalent twin system tailored for redundant manipulators with multipoint/arbitrary point contact capabilities. Within this system, we implement real-time adjustments to the impedance stiffness coefficient in conjunction with the virtual end velocity of the manipulator. We optimize the joint angular velocity of the manipulator while considering the motion allocation of redundant degrees of freedom. This optimization enables multipoint contact variable parameter compliance control for safe human–robot interactions. To validate our approach, we analyze the multipoint contact dynamic model of the redundant manipulator's interaction with humans and assess the feasibility of the variable parameter compliance control method using robot operating system simulations. Ultimately, we verify the effectiveness and practicality of our proposed compliance control method through simulation and experimental results.

Active Inference-Based Planning for Safe Human-Robot Interaction: Concurrent Consideration of Human Characteristic and Rationality

Active Inference-Based Planning for Safe Human-Robot Interaction: Concurrent Consideration of Human Characteristic and Rationality

Design and evaluation of a four-DoF upper limb exoskeleton with gravity compensation

Gravity compensation (GC) mechanisms are commonly employed to either support limb gravity in passive exoskeletons or decrease motor power for safe human-robot interaction in active exoskeletons. This paper proposes a four-degree-of-freedom (DoF) upper limb exoskeleton with a theoretically perfect GC system, incorporating three DoFs in the shoulder and one DoF in the elbow. Paired with the anthropomorphic structure, the compact and adjustable GC system housed within the exoskeleton's linkages, reduces limitations on the limb's motion range and facilitates the integration of actuators for an active system. First, the GC system is designed through the analysis of potential energy equations, employing five cable-pulley-spring based units interconnected by parallel and differential mechanisms. Subsequently, the mechanical structure of the exoskeleton is developed, and the theoretically perfect GC in the quasi-static state is verified through numerical calculations. Then, the prototype is fabricated, and its actual performance is evaluated through experiments. The experimental results demonstrate the effectiveness of the exoskeleton with GC. Finally, a potential solution for the integration of actuators is demonstrated, and the limitation of the proposed system is also well discussed.

Robotic Assistance and Haptic Feedback in Arthroscopic Procedures: Design and Preliminary Evaluation of a Two-Arm System

Robot-assisted arthroscopic surgery has been receiving growing attention in the field of orthopedic surgery. Most of the existing robot-assisted surgical systems in orthopedics place more focus on open surgery than minimally invasive surgery (MIS). In traditional arthroscopic surgery, the surgeon needs to hold an arthroscope with one hand while performing the surgical operations with the other hand, which can restrict the dexterity of the surgical performance and increase the cognitive load. On the other hand, the surgeon heavily relies on the arthroscope view when conducting the surgery, whereas the arthroscope view is a largely localized view and lacks depth information. To assist the surgeon in both scenarios, in this work, we develop a two-arm robotic system. The left-arm robot is used as a robot-assisted arthroscope holder, and it can hold the arthroscope still at a designated pose and reject all other potential disturbances, while also allowing the operator to move it via a pedal switch whenever needed. The left-arm robot is implemented with an impedance controller and a gravity iterative learning (Git) scheme, where the former can provide compliant robot behavior, thus ensuring a safe human–robot interaction, while the latter can accurately learn for gravity compensation. The right-arm robot is used as a robot-assisted surgical tool, providing virtual fixture (VF) assistance and haptic feedback during the surgery. The right-arm robot is implemented with a point-based VF algorithm, which can generate VF directly from point clouds in any shape, render force feedback, and deliver it to the operator. Furthermore, the VF, the bone, and the surgical tool position are visualized in a 3D digital environment as additional visual feedback for the operator. A series of experiments are conducted to evaluate the effectiveness of the prototype. The results demonstrate that both arms can provide satisfactory assistance as designed.

Kinematic and Dynamic Modeling of 3DOF Variable Stiffness Links Manipulator with Experimental Validation

Variable stiffness link (VSL) manipulators are robotic arms that can adjust their link stiffness in real time to improve their adaptability and precision. They are particularly useful in industrial environments where safe collaboration with human workers is required. However, modeling and controlling these non-linear systems is a major challenge due to their complexity. This research paper presents a mathematical model for a 3DOF VSL manipulator, which is the first step towards optimizing performance, improving safety, and reducing costs. The accuracy and reliability of the model are demonstrated through verification experiments that strengthen confidence in its validity for engineering and scientific research. This study contributes to the understanding of the dynamics of VSL manipulators and provides insights for future advances in the use of such robots. By using the proposed model, the efficiency and precision of VSL manipulators can be improved while ensuring safe human–robot interaction in various industrial applications.

An effective nonlinear dynamic formulation to analyze grasping capability of soft pneumatic robotic gripper

ABSTRACT Soft pneumatic robotic grippers have found extensive applications across various engineering domains, which prompts active research due to their splendid compliance, high flexibility, and safe human-robot interaction over conventional stiff counterparts. Previously simplified rod-based models principally focused on clarifying overall large deformation and bending postures of soft grippers from static or quasi-static perspectives, whereas it is challenging to elaborate grasping characteristics of soft grippers without considering contact interaction and nonlinear large deformation behaviors. To address this, based on absolute nodal coordinate formulation (ANCF), comprehensively allowing for structural complexity, geometric, material and boundary nonlinearities, and incorporating Coulomb’ friction law with a multiple-point contact method, we put forward an effective nonlinear dynamic modeling approach for exploring grasping capability of soft gripper. Moreover, we solved the established dynamic equations using Generalized-α scheme, and conducted thorough numerical simulation analysis on a three-jaw soft pneumatic gripper (SPG) in terms of grasping configurations, displacements and contact forces. The proposed dynamic approach can accurately both describe complicated deformed configurations along with stress distribution and provide a feasible solution to simulate grasping targets, whose effectiveness and precision were analyzed theoretically and verified experimentally, which may shed new light on devising and optimizing other multifunctional SPGs.

A Bioinspired Robotic Finger for Multimodal Tactile Sensing Powered by Fiber Optic Sensors

The rapid advancement of soft robotic technology emphasizes the growing importance of tactile perception. Soft grippers, equipped with tactile sensing, can gather interactive information crucial for safe human–robot interaction, wearable devices, and dexterous manipulation. However, most soft grippers with tactile sensing abilities have limited modes of tactile perception, restricting their dexterity and safety. In addition, existing tactile systems are often complicated, leading to unstable perception signals. Inspired by various organisms, a novel multimodal tactile‐sensing soft robotic finger is proposed. This finger, based on a modified fin ray structure, integrates a distributed fiber optic sensing system as part of its tactile sensory neural system. It replicates human finger capabilities, discerning contact forces as low as 0.01 N with exceptional sensitivity (106.96 mN nm−1). Through training neural networks models, the finger achieves an accuracy exceeding 96% in recognizing roughness, material stiffness, and finger pad position. Assembled into two‐finger parallel gripper, it demonstrates precise manipulation capabilities for fragile items like strawberries and potato chips. Moreover, through synergistic interplay of multimodal tactile sensing, this finger can successfully grasp an underwater transparent sphere, mitigating limitations of visual perception. The developed soft finger holds promise in various scenarios including hazardous environment detection and specialized grasping tasks.

NP-MBO: A newton predictor-based momentum observer for interaction force estimation of legged robots

Swift perception of interaction forces is a crucial skill required for legged robots to ensure safe human–robot interaction and dynamic contact management. Proprioceptive-based interactive force is widely applied due to its outstanding cross-platform versatility. In this paper, we present a novel interactive force observer, which possesses superior dynamic tracking performance. We propose a dynamic cutoff frequency configuration method to replace the conventional fixed cutoff frequency setting in the traditional momentum-based observer (MBO). This method achieves a balance between rapid tracking and noise suppression. Moreover, to mitigate the phase lag introduced by the low-pass filtering, we cascaded a Newton Predictor (NP) after MBO, which features simple computation and adaptability. The precision analysis of this method has been presented. We conducted extensive experiments on the point-foot biped robot BRAVER to validate the performance of the proposed algorithm in both simulation and physical prototype.

Soft Robot Design, Manufacturing, and Operation Challenges: A Review

Advancements in smart manufacturing have embraced the adoption of soft robots for improved productivity, flexibility, and automation as well as safety in smart factories. Hence, soft robotics is seeing a significant surge in popularity by garnering considerable attention from researchers and practitioners. Bionic soft robots, which are composed of compliant materials like silicones, offer compelling solutions to manipulating delicate objects, operating in unstructured environments, and facilitating safe human–robot interactions. However, despite their numerous advantages, there are some fundamental challenges to overcome, which particularly concern motion precision and stiffness compliance in performing physical tasks that involve external forces. In this regard, enhancing the operation performance of soft robots necessitates intricate, complex structural designs, compliant multifunctional materials, and proper manufacturing methods. The objective of this literature review is to chronicle a comprehensive overview of soft robot design, manufacturing, and operation challenges in conjunction with recent advancements and future research directions for addressing these technical challenges.

Implementations of Digital Transformation and Digital Twins: Exploring the Factory of the Future

In the era of rapid technological advancement and evolving industrial landscapes, embracing the concept of the factory of the future (FoF) is crucial for companies seeking to optimize efficiency, enhance productivity, and stay sustainable. This case study explores the concept of the FoF and its role in driving the energy transition and digital transformation within the automotive sector. By embracing advancements in technology and innovation, these factories aim to establish a smart, sustainable, inclusive, and resilient growth framework. The shift towards hybrid and electric vehicles necessitates significant adjustments in vehicle components and production processes. To achieve this, the adoption of lighter materials becomes imperative, and new technologies such as additive manufacturing (AM) and artificial intelligence (AI) are being adopted, facilitating enhanced efficiency and innovation within the factory environment. An important aspect of this paradigm involves the development and utilization of a modular, affordable, safe human–robot interaction and highly performant intelligent robot. The introduction of this intelligent robot aims to foster a higher degree of automation and efficiency through collaborative human–robot environments on the factory floor and production lines, specifically tailored to the automotive sector. By combining the strengths of human and robotic capabilities, the future factory aims to revolutionize manufacturing processes, ultimately driving the automotive industry towards a more sustainable and technologically advanced future. This study explores the implementation of automation and the initial strides toward transitioning from Industry 4.0 to 5.0, focusing on three recognized, large, and automotive companies operating in the north of Portugal.

Toward Onboard Proportional Control of Multi-Chamber Soft Pneumatic Robots: A Magnetorheological Elastomer Valve Array.

Soft pneumatic actuators (SPAs) are commonly used in various applications because of their structural compliance, low cost, ease of manufacture, high adaptability, and safe human-robot interaction. The traditional approach for achieving proportional control of soft pneumatic robots requires the use of industrial proportional valves or syringe drivers, which are not only rigid and bulky but also hard to be integrated into the body of soft robots. In our previous research, we developed a Magnetorheological elastomer (MRE)-based soft valve that showed advantages for controlling SPAs due to its compliance, compactness, robustness, and compatibility for continuous pressure modulation. Modern soft robots with multiple chambers require more MRE valves onboard for their control. However, merely packing more MRE valves for soft robots can cause problems like magnetic interference, flow rate deviation, and overheating. Therefore, in this study, we proposed a two-dimensional MRE valve array design to solve issues of magnetic interference and overheating when expanding from a single MRE proportional valve into an integrated array. The magnetic interference and the overheating problem were investigated through multiphysics simulation, bringing the optimal choice of valve spacing (1.2 times the single valve diameter), magnetic coil pole arrangement (same pole), and the cooling system design (internal cooling chamber with flowing water). Physical experiments showed that our MRE valve array maintained its original flowrate performance with low magnetic interference (0.89 mT) and low coil temperature (under 73.9°C for 5 min).

A novel stiffness-controllable joint using antagonistic actuation principles

The inherent safety of collaborative robots is essential for enhancing human–robot interaction. The primary challenge in creating soft components for these robots is achieving sufficient force and stiffness. This paper presents a joint design for collaborative robots that addresses this challenge by incorporating an antagonistic actuation principle, allowing for adjustable stiffness. The novelty of our variable-stiffness joint lies in achieving a wide range of stiffness variation at any bending angle through antagonistic actuation. This bio-inspired principle results from the activation of two opposing actuation chambers. Compared to existing joints, our proposed joint is compact and utilises a high percentage of soft materials, enabling safer human–robot interaction. The paper outlines the joint’s design and fabrication process, highlighting the feasibility of our innovative concept. Kinematic and stiffness models are introduced to analyse the bending and stiffness characteristics, which are further validated through experimental testing. The stiffness experiments demonstrate significant stiffness changes achievable through the antagonistic actuation principle. Additionally, force experiments reveal our joint can generate 20 N force at a 1.5×105 Pa pressure. A constant force output experiment confirms the joint’s advantages in providing consistent force compared to motors. Finally, a case study showcases how our proposed joint can be embedded in serial robots with variable stiffness capabilities under loading and safe human–robot interaction.

Enhancing the Versatility and Performance of Soft Robotic Grippers, Hands, and Crawling Robots Through Three-Dimensional-Printed Multifunctional Buckling Joints.

Soft robotic grippers and hands offer adaptability, lightweight construction, and enhanced safety in human-robot interactions. In this study, we introduce vacuum-actuated soft robotic finger joints to overcome their limitations in stiffness, response, and load-carrying capability. Our design-optimized through parametric design and three-dimensional (3D) printing-achieves high stiffness using vacuum pressure and a buckling mechanism for large bending angles (>90°) and rapid response times (0.24 s). We develop a theoretical model and nonlinear finite-element simulations to validate the experimental results and provide valuable insights into the underlying mechanics and visualization of the deformation and stress field. We showcase versatile applications of the buckling joints: a three-finger gripper with a large lifting ratio (∼96), a five-finger robotic hand capable of replicating human gestures and adeptly grasping objects of various characteristics in static and dynamic scenarios, and a planar-crawling robot carrying loads 30 times its weight at 0.89 body length per second (BL/s). In addition, a jellyfish-inspired robot crawls in circular pipes at 0.47 BL/s. By enhancing soft robotic grippers' functionality and performance, our study expands their applications and paves the way for innovation through 3D-printed multifunctional buckling joints.

Design and control of a novel variable stiffness actuator based on antagonistic variable radius principle

Variable stiffness actuators (VSAs) are essential for ensuring safe human–robot interactions in robotic applications. This paper proposes a novel rotary VSA using an antagonistic Hoberman linkage mechanism (AHLM), which offers a large stiffness range and a compact structure. The VSA-AHLM consists of two sets of antagonistic-type quadratic springs based on spiral cams connected to the Hoberman linkage mechanism (HLM) through four cables. By simultaneously adjusting both the radius of the HLM and the spring preload, the stiffness of the VSA-AHLM can be varied within a large range. Furthermore, the position and stiffness of the VSA-AHLM can be controlled independently by two rotary motors. The geometric parameters of the spiral cam are determined to achieve the desired linear stiffness–elongation behavior of a quadratic spring, and detailed models of the actuator’s stiffness, elastic energy, and torque are established. An actuator prototype is fabricated to demonstrate the proposed variable stiffness approach. Experiments show that the developed actuator can achieve significant stiffness changes and exhibits excellent positioning and trajectory tracking performance.

Soft modularized robotic arm for safe human–robot interaction based on visual and proprioceptive feedback

This study proposes a modularized soft robotic arm with integrated sensing of human touches for physical human–robot interactions. The proposed robotic arm is constructed by connecting multiple soft manipulator modules, each of which consists of three bellow-type soft actuators, pneumatic valves, and an on-board sensing and control circuit. By employing stereolithography three-dimensional (3D) printing technique, the bellow actuator is capable of incorporating embedded organogel channels in the thin wall of its body that are used for detecting human touches. The organogel thus serves as a soft interface for recognizing the intentions of the human operators, enabling the robot to interact with them while generating desired motions of the manipulator. In addition to the touch sensors, each manipulator module has compact, soft string sensors for detecting the displacements of the bellow actuators. When combined with an inertial measurement unit (IMU), the manipulator module has a capability of estimating its own pose or orientation internally. We also propose a localization method that allows us to estimate the location of the manipulator module and to acquire the 3D information of the target point in an uncontrolled environment. The proposed method uses only a single depth camera combined with a deep learning model and is thus much simpler than those of conventional motion capture systems that usually require multiple cameras in a controlled environment. Using the feedback information from the internal sensors and camera, we implemented closed-loop control algorithms to carry out tasks of reaching and grasping objects. The manipulator module shows structural robustness and the performance reliability over 5,000 cycles of repeated actuation. It shows a steady-state error and a standard deviation of 0.8 mm and 0.3 mm, respectively, using the proposed localization method and the string sensor data. We demonstrate an application example of human–robot interaction that uses human touches as triggers to pick up and manipulate target objects. The proposed soft robotic arm can be easily installed in a variety of human workspaces, since it has the ability to interact safely with humans, eliminating the need for strict control of the environments for visual perception. We believe that the proposed system has the potential to integrate soft robots into our daily lives.

Encoded sewing soft textile robots.

Incorporating soft actuation with soft yet durable textiles could effectively endow the latter with active and flexible shape morphing and motion like mollusks and plants. However, creating highly programmable and customizable soft robots based on textiles faces a longstanding design and manufacturing challenge. Here, we report a methodology of encoded sewing constraints for efficiently constructing three-dimensional (3D) soft textile robots through a simple 2D sewing process. By encoding heterogeneous stretching properties into three spatial seams of the sewed 3D textile shells, nonlinear inflation of the inner bladder can be guided to follow the predefined spatial shape and actuation sequence, for example, tendril-like shape morphing, tentacle-like sequential manipulation, and bioinspired locomotion only controlled by single pressure source. Such flexible, efficient, scalable, and low-cost design and formation methodology will accelerate the development and iteration of soft robots and also open up more opportunities for safe human-robot interactions, tailored wearable devices, and health care.