Dexterous Robotic Manipulation Research Articles

Robust tube-based MPC with smooth computation for dexterous robot manipulation

Robust tube-based MPC with smooth computation for dexterous robot manipulation

Additively manufactured micro-lattice dielectrics for multiaxial capacitive sensors.

Soft sensors that can perceive multiaxial forces, such as normal and shear, are of interest for dexterous robotic manipulation and monitoring of human performance. Typical planar fabrication techniques have substantial design constraints that often prohibit the creation of functionally compelling and complex architectures. Moreover, they often require multiple-step operations for production. Here, we use an additive manufacturing process based on continuous liquid interface production to create high-resolution (30-micrometer) three-dimensional elastomeric polyurethane lattices for use as dielectric layers in capacitive sensors. We show that the capacitive responses and sensitivities are highly tunable through designs of lattice type, thickness, and material-void volume percentage. Microcomputed tomography and finite element simulation are used to elucidate the influence of lattice design on the deformation mechanism and concomitant sensing behavior. The advantage of three-dimensional printing is exhibited with examples of fully printed representative athletic equipment with integrated sensors.

Learning-Based Slip Detection for Dexterous Manipulation Using GelStereo Sensing.

Endowing the robot with tactile perception can effectively improve manipulation dexterity, along with various benefits of human-like touch. Using GelStereo (GS) tactile sensing, which gives high-resolution contact geometry information, including 2-D displacement field, and 3-D point cloud of the contact surface, we present a learning-based slip detection system in this study. The results reveal that the well-trained network achieves 95.79% accuracy on the never-seen testing dataset, which surpasses the current model-based and learning-based methods using visuotactile sensing. We also propose a general framework for slip feedback adaptive control for dexterous robot manipulation tasks. The experimental results show the effectiveness and efficiency of the proposed control framework using GS tactile feedback when deployed on real-world grasping and screwing manipulation tasks on various robot setups.

Triboelectric and iontronic dual-responsive bioinspired ionic skin for human–like dexterous robotic manipulation

Powerful tactile perception is keystone for robots to achieve human-like dexterous manipulation. However, endowing robots with tactile perception capabilities that approach or even surpass those of humans is a challenge. Organisms can perceive complex environments rapidly, and almost all perception is related to the directional transport of ions. Inspired by this mechanism, we design a flexible dual-responsive skin (FDRS) based on ions migration for proximity and tactile somatosensation, which is consisting of single-electrode triboelectric nanogenerator and iontronic sensor, achieving surpassing-skin capabilities. The single-electrode triboelectric nanogenerator can encode the proximity information of approaching targets into a series of voltage pulses with fast response time (100 μs). The iontronic tactile unit based on the hierarchical micro-hemispherical gel layer achieves high linearity (R2 = 0.998) over a broad range of 0–700 kPa. We demonstrate the applications of FDRS for three-dimensional object recognition and robotic control. These capabilities of the multilayer integrated FDRS provide new perspectives for the future development of human-like dexterous robotic manipulation.

Design, Fabrication, and Characterization of a Novel Optical Six-Axis Distributed Force and Displacement Tactile Sensor for Dexterous Robotic Manipulation.

Real-time multi-axis distributed tactile sensing is a critical capability if robots are to perform stable gripping and dexterous manipulation, as it provides crucial information about the sensor-object interface. In this paper, we present an optical-based six-axis tactile sensor designed in a fingertip shape for robotic dexterous manipulation. The distributed sensor can precisely estimate the local XYZ force and displacement at ten distinct locations and provide the global XYZ force and torque measurements. Its compact size, comparable to that of a human thumb, and minimal thickness allow seamless integration onto existing robotic fingers, eliminating the need for complex modifications to the gripper. The proposed sensor design uses a simple, low-cost fabrication method. Moreover, the optical transduction approach uses light angle and intensity sensing to infer force and displacement from deformations of the individual sensing units that form the overall sensor, providing distributed six-axis sensing. The local force precision at each sensing unit in the X, Y, and Z axes is 20.89 mN, 19.19 mN, and 43.22 mN, respectively, over a local force range of approximately ±1.5 N in X and Y and 0 to -2 N in Z. The local displacement precision in the X, Y, and Z axes is 56.70 μm, 50.18 μm, and 13.83 μm, respectively, over a local displacement range of ±2 mm in the XY directions and 0 to -1.5 mm in Z (i.e., compression). Additionally, the sensor can measure global torques, Tx, Ty, and Tz, with a precision of of 1.90 N-mm, 1.54 N-mm, and 1.26 N-mm, respectively. The fabricated design is showcased by integrating it with an OnRobot RG2 gripper and illustrating real-time measurements during in simple demonstration task, which generated changing global forces and torques.

Exploring Tactile Temporal Features for Object Pose Estimation during Robotic Manipulation.

Dexterous robotic manipulation tasks depend on estimating the state of in-hand objects, particularly their orientation. Although cameras have been traditionally used to estimate the object's pose, tactile sensors have recently been studied due to their robustness against occlusions. This paper explores tactile data's temporal information for estimating the orientation of grasped objects. The data from a compliant tactile sensor were collected using different time-window sample sizes and evaluated using neural networks with long short-term memory (LSTM) layers. Our results suggest that using a window of sensor readings improved angle estimation compared to previous works. The best window size of 40 samples achieved an average of 0.0375 for the mean absolute error (MAE) in radians, 0.0030 for the mean squared error (MSE), 0.9074 for the coefficient of determination (R2), and 0.9094 for the explained variance score (EXP), with no enhancement for larger window sizes. This work illustrates the benefits of temporal information for pose estimation and analyzes the performance behavior with varying window sizes, which can be a basis for future robotic tactile research. Moreover, it can complement underactuated designs and visual pose estimation methods.

Recent Progress of Tactile and Force Sensors for Human-Machine Interaction.

Human-Machine Interface (HMI) plays a key role in the interaction between people and machines, which allows people to easily and intuitively control the machine and immersively experience the virtual world of the meta-universe by virtual reality/augmented reality (VR/AR) technology. Currently, wearable skin-integrated tactile and force sensors are widely used in immersive human-machine interactions due to their ultra-thin, ultra-soft, conformal characteristics. In this paper, the recent progress of tactile and force sensors used in HMI are reviewed, including piezoresistive, capacitive, piezoelectric, triboelectric, and other sensors. Then, this paper discusses how to improve the performance of tactile and force sensors for HMI. Next, this paper summarizes the HMI for dexterous robotic manipulation and VR/AR applications. Finally, this paper summarizes and proposes the future development trend of HMI.

Detecting and Controlling Slip through Estimation and Control of the Sliding Velocity

Slipping detection and avoidance are key issues in dexterous robotic manipulation. The capability of robots to grasp and manipulate objects of common use can be greatly enhanced by endowing these robots with force/tactile sensors on their fingertips. Object slipping can be caused by both tangential and torsional loads when the grip force is too low. Contact force and moment measurements are required to counteract such loads and avoid slippage by controlling the grip force. In this paper, we use the SUNTouch force/tactile sensor, which provides the robotic control system with reliable measurements of both normal and tangential contact force components together with the torsional moment. By exploiting the limit surface concept and the LuGre friction model, we build a model of the object/fingertip planar sliding. This model is the basis of a nonlinear observer that estimates the sliding velocity and the friction state variable from the measured contact force and torsional moment. The slipping control system uses the estimated friction state to detect the slipping event and the estimated sliding velocity to control the grasp force. The control modality is twofold: the first one is aimed at avoiding object slip, while the second one allows the object to perform a controlled pivoting about the grasping axis. Experiments show that the robot is able to safely manipulate objects that require grasping forces in a large range, from 0.2 N to 10 N. This level of manipulation autonomy is attained by a suitably identified dynamic model that overcomes the limited generalization capability of existing learning-based approaches in the general roto-translational slip control.

Laser-Actuated Multi-Fingered Hand for Dexterous Manipulation of Micro-Objects

Dexterous robotic manipulation refers to the coordination of multiple fingers or manipulators to grasp and manipulate a target object. In past decades, various achievements have been established in dexterous robotic manipulation in the physical world. However, such dexterous ability is rarely found in nano/micro manipulation, which is mainly due to the difficulty in developing actuators and sensors at a nano/micro-scale, as well as the dependence on the physical properties of objects. As a result, only limited and relatively simple micro-manipulation tasks such as pushing and picking have been realized so far. Here, we demonstrate a platform to achieve dexterous manipulation in the micro-world by developing a micro multi-fingered hand that is actuated by optical traps. The system consists of several micro laser-actuated fingers which are coordinated to function as a robotic micro-hand. Each micro-finger is configured with three degrees of freedom in a 2-dimensional space, and a system composed of coordinated micro-fingers can then be utilized for grasping, rotating, moving, and even levering and rolling of objects in micron scale. Thus, the proposed approach in this paper establishes a foundation in achieving dexterous robotic manipulation of objects at micron scale through the coordination of multiple micro fingers.

Knot-inspired optical sensors for slip detection and friction measurement in dexterous robotic manipulation

Knot-inspired optical sensors for slip detection and friction measurement in dexterous robotic manipulation

Dexterous robotic manipulation using deep reinforcement learning and knowledge transfer for complex sparse reward‐based tasks

AbstractThis paper describes a deep reinforcement learning (DRL) approach that won Phase 1 of the Real Robot Challenge (RRC) 2021, and then extends this method to a more difficult manipulation task. The RRC consisted of using a TriFinger robot to manipulate a cube along a specified positional trajectory, but with no requirement for the cube to have any specific orientation. We used a relatively simple reward function, a combination of a goal‐based sparse reward and a distance reward, in conjunction with Hindsight Experience Replay (HER) to guide the learning of the DRL agent (Deep Deterministic Policy Gradient [DDPG]). Our approach allowed our agents to acquire dexterous robotic manipulation strategies in simulation. These strategies were then deployed on the real robot and outperformed all other competition submissions, including those using more traditional robotic control techniques, in the final evaluation stage of the RRC. Here we extend this method, by modifying the task of Phase 1 of the RRC to require the robot to maintain the cube in a particular orientation, while the cube is moved along the required positional trajectory. The requirement to also orient the cube makes the agent less able to learn the task through blind exploration due to increased problem complexity. To circumvent this issue, we make novel use of a Knowledge Transfer (KT) technique that allows the strategies learned by the agent in the original task (which was agnostic to cube orientation) to be transferred to this task (where orientation matters). KT allowed the agent to learn and perform the extended task in the simulator, which improved the average positional deviation from 0.134 to 0.02 m, and average orientation deviation from 142° to 76° during evaluation. This KT concept shows good generalization properties and could be applied to any actor‐critic learning algorithm.

Experimental assessment on the contact characteristics of 3D printed flexible poly lactic acid (PLA) soft fingertips

Abstract This purpose of this research work primarily focuses on assessing the contact characteristics of a novel 3D printed flexible poly lactic acid (PLA) fingertip exposed to a normal load ranging from 1–750 N. The 3D printed fingertip is pressed against three different target surfaces having concave, convex and flat profiles to facilitate a rational comparison. The growth of contact area is recorded for a wide range of applied normal force against the fingertip on logging sheets and the same is converted in vector format for facilitating digital measurements. From close examination of the results, it may be noted that the rate of growth of contact area follows the parametric relationship a = cN γ . A weighted least squares fit algorithm is used to formulate the parametric relationship based on experimental data. Further, the contact characteristics of the 3D printed fingertip follows the same pattern of soft neoprene fingertip, and it is well in line with the expected results. Hence, it is evident that 3D printed fingertips could be utilized for handling fragile to hard objects and capable of handling multi-profiled objects in dexterous robotic manipulation. Moreover, complex profiled fingertips can easily be manufactured by 3D printing, and it can be considered as a better alternative for conventionally manufactured anthropomorphic robotic grippers. Practical implications of this research will be highly useful for development of soft-fingered robotic grippers for dexterous robotic manipulations.

In-Hand Object Localization Using a Novel High-Resolution Visuotactile Sensor

In-hand object localization and manipulation has always been a challenging task in robotic community. In this article, we address this problem by vision-based tactile sensing with high-spatial resolution. Specifically, we design a novel tactile sensor based on stereo vision, named GelStereo, which can perceive tactile point cloud with high-spatial resolution ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&lt;\!1$</tex-math></inline-formula> mm). A tactile-based in-hand object localization pipeline composed of saliency detection and probabilistic point-set registration algorithms of the perceived contact point cloud is presented. Furthermore, extensive qualitative and quantitative analyses of perceived tactile point cloud and in-hand localization and insertion experiments of small parts are performed on our robot platform. The experimental results verify the accuracy and robustness of the tactile point cloud sensed by the novel GelStereo tactile sensor and the proposed in-hand object localization pipeline. This novel high-resolution visuotactile sensing technology has predictable application potential in the field of dexterous robotic manipulation.

Polymer-Based Optical Waveguide Triaxial Tactile Sensing for 3-Dimensional Curved Shell

To realize dexterous robotic manipulation and enhance the human-machine interaction, nowadays increasing efforts have been made towards multi-dimensional force sensing. However, there are still bottlenecks in integrating these sensors into robots because of the limitation on conformability to complex surfaces and miniaturization. To overcome the above challenges, we proposed a novel polymer-based waveguide tactile sensing method by embedding elastic optical waveguide channels into a 3D curved shell. By surrounding a soft tactel with four channels, the tactel is capable to measure normal and shear forces. We demonstrated that 3-axis force sensing is achieved on a shell structure with thickness of 2.1 mm. The average gauge factor in detecting the normal force from 0 to 1.1 N is −0.2207 N <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">−1</sup> , while the value is −0.1976 N <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">−1</sup> in the shear force sensing from −1 to 1 N. A force resolution of 0.1 mN has been achieved, and the proposed sensor has an average hysteresis of 22.23%. It performs well in dynamic force testing of 10 Hz and reaches average angle error 9.7357° and average amplitude error 0.1555 N between real force and predicted force which is derived by the calibration matrix. The experiment results prove that the proposed sensing method can provide triaxial force sensing on the complex 3D curved surface with good performance.

A review on sensory perception for dexterous robotic manipulation

Sensory perception for dexterous robotic hands is an active research area and recent progress in robotics. Effective dexterous manipulation requires robotic hands to accurately feedback their state or perceive the surrounding environment. This article reviews the state-of-the-art of sensory perception for dexterous robotic manipulation. Two types of sensors, such as intrinsic and extrinsic sensors, are introduced according to their function and layout in robotic hands. These sensors provide rich information to a robotic hand, which contains the posture, the contact information of objects, and the physical information of the environment. Then, a comprehensive analysis of perception methods including planning-level, control-level, and learning-level perceptions is presented. The information obtained from sensory perception can help robotic hands to make decisions effectively. Previously issued reviews mainly focus on the design of tactile senor, while we analyze and discuss the relationship among sensing, perception, and dexterous manipulation. Some potential research topics on sensory perception are also summarized and discussed.

Self-Supervised Contact Geometry Learning by GelStereo Visuotactile Sensing

Vision-based tactile sensors have recently shown promising contact information sensing capabilities in various fields, especially for dexterous robotic manipulation. However, dense contact geometry measurement is still a challenging problem. In this article, we update the design of our previous GelStereo tactile sensor and present a self-supervised contact geometry learning pipeline. Specifically, a self-supervised stereo-based depth estimation neural network (GS-DepthNet) is proposed to achieve real-time disparity estimation, and two specifically designed loss functions are proposed to accelerate the convergence of the network during the training process and improve the inference accuracy. Furthermore, extensive qualitative and quantitative experiments of perceived contact shape were performed on our GelStereo sensor. The experimental results verify the accuracy and robustness of the proposed contact geometry sensing pipeline. This updated GelStereo tactile sensor with dense contact geometric sensing capability has predictable application potential in the field of industrial and service robots.

Capacitive Flexible Three-Axis Force Tactile Sensor Based on Porous Composite Dielectric Layer for Dexterous Robotic Manipulation

Capacitive Flexible Three-Axis Force Tactile Sensor Based on Porous Composite Dielectric Layer for Dexterous Robotic Manipulation

Gaze-Based Dual Resolution Deep Imitation Learning for High-Precision Dexterous Robot Manipulation

A high-precision manipulation task, such as needle threading, is challenging. Physiological studies have proposed connecting low-resolution peripheral vision and fast movement to transport the hand into the vicinity of an object, and using high-resolution foveated vision to achieve the accurate homing of the hand to the object. The results of this study demonstrate that a deep imitation learning based method, inspired by the gaze-based dual resolution visuomotor control system in humans, can solve the needle threading task. First, we recorded the gaze movements of a human operator who was teleoperating a robot. Then, we used only a high-resolution image around the gaze to precisely control the thread position when it was close to the target. We used a low-resolution peripheral image to reach the vicinity of the target. The experimental results obtained in this study demonstrate that the proposed method enables precise manipulation tasks using a general-purpose robot manipulator and improves computational efficiency.

High Sensitivity and Wide Range Soft Magnetic Tactile Sensor Based on Electromagnetic Induction

Tactile sensors play a significant role in robotic systems to interact with the external world. In this article, a novel soft magnetic tactile sensor (SMTS) based on the law of electromagnetic induction is proposed. This new sensor uses the configuration of a transformer with planar coils and a Polydimethylsiloxane silicone elastomer with an inverted cone structure embedded in a copper sheet. The SMTS measures the induced voltage amplitude variation of the secondary coil that is caused by the changes in alternating current magnetic field between the coil and the conductive or ferromagnetic sheets. The principle of the new sensor is explained first and then the design methods are outlined. Prototypes are manufactured and a three-dimensional finite element model is built to analyze the mechanical property of the elastomer. Besides, a performance analysis platform and an electronic interface system with the excitation source, the signal processing and the data acquisition, are developed, in order to calibrate the sensor using the polynomial fitting method. The calibrated prototype shows competitive performances in comparison with other existing counterparts, which has a high sensitivity of 56.3 mV/N, a wide measurement range up to 15 N, and a low hysteresis of 2.02%. The new sensor is also of low cost, durable, and has potential applications in dexterous robotic manipulation.

Controlling Ocean One: Human–robot collaboration for deep‐sea manipulation

AbstractDeploying robots to explore venues that are inaccessible to humans, or simply inhospitable, has been a longstanding ambition of scientists, engineers, and explorers across numerous fields. The deep sea exemplifies an environment that is largely uncharted and denies human presence. Central to exploration is the capacity to deliver dexterous robotic manipulation to this unstructured environment. Unmanned underwater vehicles (UUVs) are successful in providing passive solutions for observation and mapping but currently are far from capable of delivering human‐level dexterity. The ones providing manipulation typically are UUVs coupled with position‐controlled hydraulic arms using disjoint controllers for navigation and manipulation that require expert operators. Ocean One is a humanoid underwater robot designed specifically for underwater manipulation. In this paper, we present Ocean One's control architecture that, through a collaboration between this humanoid robot and a human pilot, enables the deployment of dexterous robotic manipulation to the deep sea. We provide detailed descriptions of this architecture's two main components: first, a whole‐body controller that creates functional autonomy by coordinating manipulation, posture, and constraint tasks, and second, a set of haptic and visual human interfaces that enable intimate interaction while avoiding micromanagement. We test the presented methods in simulation and validate them in pool experiments and in two field deployments. On its maiden mission into the Mediterranean Sea, Ocean One explored the Lune, a French naval vessel that sank in 1664 off the coast of Toulon, France. In its second expedition, Ocean One assisted human divers in investigating underwater volcanic structures at Santorini, Greece.