Dexterous Manipulation Tasks Research Articles

Learning-Based Slip Detection for Dexterous Manipulation Using GelStereo Sensing.

Endowing the robot with tactile perception can effectively improve manipulation dexterity, along with various benefits of human-like touch. Using GelStereo (GS) tactile sensing, which gives high-resolution contact geometry information, including 2-D displacement field, and 3-D point cloud of the contact surface, we present a learning-based slip detection system in this study. The results reveal that the well-trained network achieves 95.79% accuracy on the never-seen testing dataset, which surpasses the current model-based and learning-based methods using visuotactile sensing. We also propose a general framework for slip feedback adaptive control for dexterous robot manipulation tasks. The experimental results show the effectiveness and efficiency of the proposed control framework using GS tactile feedback when deployed on real-world grasping and screwing manipulation tasks on various robot setups.

Dexterous manipulation: differential sensitivity of manipulation and grasp forces to task requirements.

How humans coordinate digit forces to perform dexterous manipulation is not well understood. This gap is due to the use of tasks devoid of dexterity requirements and/or the use of analytical techniques that cannot isolate the roles that digit forces play in preventing object slip and controlling object position and orientation (pose). In our recent work, we used a dexterous manipulation task and decomposed digit forces into FG, the internal force that prevents object slip, and FM, the force responsible for object pose control. Unlike FG, FM was modulated from object lift onset to hold, suggesting their different sensitivity to sensory feedback acquired during object lift. However, the extent to which FG and FM can be controlled independently remains to be determined. Importantly, how FG and FM change as a function of object property is mathematically indeterminate and therefore requires active modulation. To address this gap, we systematically changed either object mass or external torque. The FM normal component responsible for object orientation control was modulated to changes in object torque but not mass. In contrast, FG was distinctly modulated to changes in object mass and torque. These findings point to a differential sensitivity of FG and FM to task requirements and provide novel insights into the neural control of dexterous manipulation. Importantly, our results indicate that the proposed digit force decomposition has the potential to capture important differences in how sensory inputs are processed and integrated to simultaneously ensure grasp stability and dexterous object pose control.NEW & NOTEWORTHY Successful dexterous object manipulation requires simultaneous prevention of object slip and object pose control. How these two task goals are attained can be investigated by decomposing digit forces into grasp and manipulation forces, respectively. We found that these forces were characterized by differential sensitivity to changes in object properties (mass and torque). This finding suggests the involvement of distinct sensorimotor mechanisms that, combined, simultaneously ensure grasp stability and dexterous control of object pose.

Bi-DexHands: Towards Human-Level Bimanual Dexterous Manipulation.

Achieving human-level dexterity in robotics remains a critical open problem. Even simple dexterous manipulation tasks pose significant difficulties due to the high number of degrees of freedom and the need for cooperation among heterogeneous agents (e.g., finger joints). While some researchers have utilized reinforcement learning (RL) to control a single hand in manipulating objects, tasks that require coordinated bimanual cooperation are still under-explored due to the fewer suitable environments, which can result in difficulties and sub-optimal performance. To address these challenges, we introduce Bi-DexHands, a simulator with two dexterous hands featuring 20 bimanual manipulation tasks and thousands of target objects, designed to match various levels of human motor skills based on cognitive science research. We developed Bi-DexHands in Issac Gym, enabling highly efficient RL training at over 30,000 frames per second using a single NVIDIA RTX 3090. Based on Bi-DexHands, we present a comprehensive evaluation of popular RL algorithms in different settings, including single-agent/multi-agent RL, offline RL, multi-task RL, and meta RL. Our findings show that on-policy algorithms, such as PPO, can master simple manipulation tasks that correspond to those of 48-month-old babies, such as catching a flying object or opening a bottle. Furthermore, multi-agent RL can improve the ability to perform manipulations that require skilled bimanual cooperation, such as lifting a pot or stacking blocks. Despite achieving success in individual tasks, current RL algorithms struggle to learn multiple manipulation skills in most multi-task and few-shot learning scenarios. This highlights the need for further research and development within the RL community.

Tactile sensor-less fingertip contact detection and force estimation for stable grasping with an under-actuated hand

Detecting contact when fingers are approaching an object and estimating the magnitude of the force the fingers are exerting on the object after contact are important tasks for a multi-fingered robotic hand to stably grasp objects. However, for a linkage-based under-actuated robotic hand with a self-locking mechanism to realize stable grasping without using external sensors, such tasks are difficult to perform when only analyzing the robot model or only applying data-driven methods. Therefore, in this paper, a hybrid of previous approaches is used to find a solution for realizing stable grasping with an under-actuated hand. First, data from the internal sensors of a robotic hand are collected during its operation. Subsequently, using the robot model to analyze the collected data, the differences between the model and real data are explained. From the analysis, novel data-driven-based algorithms, which can overcome noted challenges to detect contact between a fingertip and the object and estimate the fingertip forces in real-time, are introduced. The proposed methods are finally used in a stable grasp controller to control a triple-fingered under-actuated robotic hand to perform stable grasping. The results of the experiments are analyzed to show that the proposed algorithms work well for this task and can be further developed to be used for other future dexterous manipulation tasks.

Elastic Tactile Sensor Glove for Dexterous Teaching by Demonstration.

We present a thin and elastic tactile sensor glove for teaching dexterous manipulation tasks to robots through human demonstration. The entire glove, including the sensor cells, base layer, and electrical connections, is made from soft and stretchable silicone rubber, adapting to deformations under bending and contact while preserving human dexterity. We develop a glove design with five fingers and a palm sensor, revise material formulations for reduced thickness, faster processing and lower cost, adapt manufacturing processes for reduced layer thickness, and design readout electronics for improved sensitivity and battery operation. We further address integration with a multi-camera system and motion reconstruction, wireless communication, and data processing to obtain multimodal reconstructions of human manipulation skills.

Visual dexterity: In-hand reorientation of novel and complex object shapes.

In-hand object reorientation is necessary for performing many dexterous manipulation tasks, such as tool use in less structured environments, which remain beyond the reach of current robots. Prior works built reorientation systems assuming one or many of the following conditions: reorienting only specific objects with simple shapes, limited range of reorientation, slow or quasi-static manipulation, simulation-only results, the need for specialized and costly sensor suites, and other constraints that make the system infeasible for real-world deployment. We present a general object reorientation controller that does not make these assumptions. It uses readings from a single commodity depth camera to dynamically reorient complex and new object shapes by any rotation in real time, with the median reorientation time being close to 7 seconds. The controller was trained using reinforcement learning in simulation and evaluated in the real world on new object shapes not used for training, including the most challenging scenario of reorienting objects held in the air by a downward-facing hand that must counteract gravity during reorientation. Our hardware platform only used open-source components that cost less than 5000 dollars. Although we demonstrate the ability to overcome assumptions in prior work, there is ample scope for improving absolute performance. For instance, the challenging duck-shaped object not used for training was dropped in 56% of the trials. When it was not dropped, our controller reoriented the object within 0.4 radians (23°) 75% of the time.

Pose optimization for mobile manipulator grasping based on hybrid manipulability

PurposeProper platform pose is important for the mobile manipulator to accomplish dexterous manipulation tasks efficiently and safely, and the evaluation criterion to qualify manipulation performance is critical to support the pose decision process. This paper aims to present a comprehensive index to evaluate the manipulator’s operation performance from various aspects.Design/methodology/approachIn this research, a criterion called hybrid manipulability (HM) is proposed to assess the performance of the manipulator’s operation, considering crucial factors such as joint limits, obstacle avoidance and stability. The determination of the optimal platform pose is achieved by selecting the pose that maximizes the HM within the feasible inverse reachability map associated with the target object.FindingsA self-built mobile manipulator is adopted as the experimental platform, and the feasibility of the proposed method is experimentally verified in the context of object-grasping tasks both in simulation and practice.Originality/valueThe proposed HM extends upon the conventional notion of manipulability by incorporating additional factors, including the manipulator’s joint limits, the obstacle avoidance situation during the operation and the manipulation stability when grasping the target object. The manipulator can achieve enhanced stability during grasping when positioned in the pose determined by the HM.

Ultra-lightweight anthropomorphic dual-arm rolling robot for dexterous manipulation tasks on linear infrastructures: A self-stabilizing system

Ultra-lightweight anthropomorphic dual-arm rolling robot for dexterous manipulation tasks on linear infrastructures: A self-stabilizing system

Distinct sensorimotor mechanisms underlie the control of grasp and manipulation forces for dexterous manipulation

Dexterous manipulation relies on the ability to simultaneously attain two goals: controlling object position and orientation (pose) and preventing object slip. Although object manipulation has been extensively studied, most previous work has focused only on the control of digit forces for slip prevention. Therefore, it remains underexplored how humans coordinate digit forces to prevent object slip and control object pose simultaneously. We developed a dexterous manipulation task requiring subjects to grasp and lift a sensorized object using different grasp configurations while preventing it from tilting. We decomposed digit forces into manipulation and grasp forces for pose control and slip prevention, respectively. By separating biomechanically-obligatory from non-obligatory effects of grasp configuration, we found that subjects prioritized grasp stability over efficiency in grasp force control. Furthermore, grasp force was controlled in an anticipatory fashion at object lift onset, whereas manipulation force was modulated following acquisition of somatosensory and visual feedback of object’s dynamics throughout object lift. Mathematical modeling of feasible manipulation forces further confirmed that subjects could not accurately anticipate the required manipulation force prior to acquisition of sensory feedback. Our experimental approach and findings open new research avenues for investigating neural mechanisms underlying dexterous manipulation and biomedical applications.

SURRL: Structural Unsupervised Representations for Robot Learning

Revolutionary advances have occurred in robot learning research, where a resurgence in reinforcement learning (RL) algorithms has fueled breakthroughs in acquiring complicated robotic skills without human intervention. Unfortunately, one limitation that has hampered reinforcement learning methods is that RL may be quite inefficient, that is, it may cost unrealistic learning time and prohibitively large numbers of trajectories to provide implausible models for achieving multi-task learning. In a broader perspective, the realization of robotics control by RL relies heavily on the availability of compact and expressive representations of the state spaces. In this paper, we propose to learn vivid general structural representations by utilizing structural prior knowledge of robots. Particularly, a novel framework called Structural Unsupervised Representations for Robot Learning (SURRL) is presented to enable multi-task learning. The task-agnostic sample trajectories are leveraged to learn the structural representations which are constructed by Graph Auto-Encoder in an unsupervised fashion. When learning a new task, the learned structural representations can be directly adopted for subsequently policy learning without training from scratch. Extensive experiments on continuous robotic environments, including dexterous manipulation tasks, characterize that our method’s effectiveness to learn optimal policies for handling multiple tasks learning, demonstrating significant improvements over other competitive methods.

Exploring Tactile Temporal Features for Object Pose Estimation during Robotic Manipulation.

Dexterous robotic manipulation tasks depend on estimating the state of in-hand objects, particularly their orientation. Although cameras have been traditionally used to estimate the object's pose, tactile sensors have recently been studied due to their robustness against occlusions. This paper explores tactile data's temporal information for estimating the orientation of grasped objects. The data from a compliant tactile sensor were collected using different time-window sample sizes and evaluated using neural networks with long short-term memory (LSTM) layers. Our results suggest that using a window of sensor readings improved angle estimation compared to previous works. The best window size of 40 samples achieved an average of 0.0375 for the mean absolute error (MAE) in radians, 0.0030 for the mean squared error (MSE), 0.9074 for the coefficient of determination (R2), and 0.9094 for the explained variance score (EXP), with no enhancement for larger window sizes. This work illustrates the benefits of temporal information for pose estimation and analyzes the performance behavior with varying window sizes, which can be a basis for future robotic tactile research. Moreover, it can complement underactuated designs and visual pose estimation methods.

An Adaptive Hand Exoskeleton for Teleoperation System

Teleoperation can assist people to complete various complex tasks in inaccessible or high-risk environments, in which a wearable hand exoskeleton is one of the key devices. Adequate adaptability would be available to enable the master hand exoskeleton to capture the motion of human fingers and reproduce the contact force between the slave hand and its object. This paper presents a novel finger exoskeleton based on the cascading four-link closed-loop kinematic chain. Each finger has an independent closed-loop kinematic chain, and the angle sensors are used to obtain the finger motion including the flexion/extension and the adduction/abduction. The cable tension is changed by the servo motor to transmit the contact force to the fingers in real time. Based on the finger exoskeleton, an adaptive hand exoskeleton is consequently developed. In addition, the hand exoskeleton is tested in a master–slave system. The experiment results show that the adaptive hand exoskeleton can be worn without any mechanical constraints, and the slave hand can follow the motions of each human finger. The accuracy and the real-time capability of the force reproduction are validated. The proposed adaptive hand exoskeleton can be employed as the master hand to remotely control the humanoid five-fingered dexterous slave hand, thus, enabling the teleoperation system to complete complex dexterous manipulation tasks.

Motion coordination and dexterous manipulation for underwater vehicle-manipulator systems

This paper addresses the redundancy resolution problem for underwater vehicle-manipulator systems (UVMSs). More specifically, a new redundancy resolution approach with motion coordination and dexterous manipulation is presented to guarantee the tracking accuracy of the end-effector while avoiding singular configurations of the manipulator. The task-priority redundancy resolution framework is adopted to achieve an effective coordinated motion of the vehicle and manipulator while exploiting the redundant degrees of freedom available. A dexterous manipulation task with low priority is proposed to reconfigure the UVMS into a more dexterous configuration by keeping the end-effector within the designed heuristic subset. An inequality-based singularity avoidance task is inserted into the redundancy resolution framework as a high-priority task to keep the measure of manipulability above a specified minimum value. Experimental results verify that the proposed approach significantly improves the dexterity and coordination of the UVMS.

Dexterous Object Manipulation with an Anthropomorphic Robot Hand via Natural Hand Pose Transformer and Deep Reinforcement Learning

Dexterous object manipulation using anthropomorphic robot hands is of great interest for natural object manipulations across the areas of healthcare, smart homes, and smart factories. Deep reinforcement learning (DRL) is a particularly promising approach to solving dexterous manipulation tasks with five-fingered robot hands. Yet, controlling an anthropomorphic robot hand via DRL in order to obtain natural, human-like object manipulation with high dexterity remains a challenging task in the current robotic field. Previous studies have utilized some predefined human hand poses to control the robot hand’s movements for successful object-grasping. However, the hand poses derived from these grasping taxonomies are limited to a partial range of adaptability that could be performed by the robot hand. In this work, we propose a combinatory approach of a deep transformer network which produces a wider range of natural hand poses to configure the robot hand’s movements, and an adaptive DRL to control the movements of an anthropomorphic robot hand according to these natural hand poses. The transformer network learns and infers the natural robot hand poses according to the object affordance. Then, DRL trains a policy using the transformer output to grasp and relocate the object to the designated target location. Our proposed transformer-based DRL (T-DRL) has been tested using various objects, such as an apple, a banana, a light bulb, a camera, a hammer, and a bottle. Additionally, its performance is compared with a baseline DRL model via natural policy gradient (NPG). The results demonstrate that our T-DRL achieved an average manipulation success rate of 90.1% for object manipulation and outperformed NPG by 24.8%.

Markovian policy network for efficient robot learning

What poses a challenge for robot learning is the fact that considerable training samples and time are required to find an optimal policy in the high-dimensional robot state space. To mitigate this problem of robot learning, there is an incentive in incorporating appropriate prior knowledge. In this paper, a novel framework named Markovian Policy Network (MPN) is presented to take full advantage of the structural prior knowledge for robots to achieve efficient robot learning. Specifically, we first make efforts to reveal the Markov Property of the robot, which characterizes the intrinsic prior knowledge for robotic physical structure. Next, the proposed framework is constructed by employing a modified graph neural network to incorporate the Markov Property of the robot into policy networks. By exploiting the structural prior knowledge of robots in our framework, the high-dimensional original state space could be decomposed into several low-dimensional state spaces with conditional independence to realize substantial reductions in the dimensionality of state space. Empirical results on diverse robotic systems including the challenging dexterous manipulation task, demonstrate not only the effectiveness but the robustness of the proposed MPN for robotic skill learning, delivering advancement substantially over traditional policy learning methods.

Multi-Phase Multi-Objective Dexterous Manipulation with Adaptive Hierarchical Curriculum

Dexterous manipulation tasks usually have multiple objectives. The priorities of these objectives may vary at different phases of a manipulation task. Current methods do not consider the objective priority and its change during the task, making a robot have a hard time or even fail to learn a good policy. In this work, we develop a novel Adaptive Hierarchical Curriculum to guide the robot to learn manipulation tasks with multiple prioritized objectives. Our method determines the objective priorities during the learning process and updates the learning sequence of the objectives to adapt to the changing priorities at different phases. A smooth transition function is developed to mitigate the effects on the learning stability when updating the learning sequence. The proposed method is validated in a multi-objective manipulation task with a JACO robot arm in which the robot needs to manipulate a target with obstacles surrounded. The simulation and physical experiment results show that the proposed method outperforms the baseline methods with a 92.5% success rate in 40 tests and on average takes 36.4% less time to finish the task.

Collective Behaviors of Magnetic Microparticle Swarms: From Dexterous Tentacles to Reconfigurable Carpets.

Microrobot swarms have promising prospects in biomedical applications ranging from targeted cargo delivery to minimally invasive surgery. However, such potential is constrained by the small output force and low efficiency of the current microrobot swarms. To address this challenge, we report a tentacle-like reconfigurable microrobot swarm by programming paramagnetic microparticles into reconfigurable carpets with numerous cilia. This wirelessly controlled microrobot swarm is constructed via a strong gradient magnetic field in combination with a programmable oscillating magnetic field. The gradient magnetic field is supplied by a permanent magnet, which enables fast formation of a microrobot swarm with powerful collective behaviors via cooperative physical structures within the swarm. The oscillating magnetic field is produced by a custom-built electromagnetic coil system, which is adopted as an actuation device for conducting dexterous manipulation via controllable oscillation motion. Using the proposed microrobot swarming strategy, a milligram-level magnetic carpet achieves a millinewton-level output force. By applying different types of magnetic fields, the magnetic carpet accomplishes dexterous manipulation tasks, lesion removal, and controllable drug diffusion with a high-efficiency response in microscale executions. The formation and control mechanisms of the microrobot swarm reported here provide a practical candidate for in vivo biomedical treatment.

Multi-Fingered In-Hand Manipulation With Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors

Multi-fingered hands could be used to achieve many dexterous manipulation tasks, similarly to humans, and tactile sensing could enhance the manipulation stability for a variety of objects. However, tactile sensors on multi-fingered hands have a variety of sizes and shapes. Convolutional neural networks (CNN) can be useful for processing tactile information, but the information from multi-fingered hands needs an arbitrary pre-processing, as CNNs require a rectangularly shaped input, which may lead to unstable results. Therefore, how to process such complex shaped tactile information and utilize it for achieving manipulation skills is still an open issue. This paper presents a control method based on a graph convolutional network (GCN) which extracts geodesical features from the tactile data with complicated sensor alignments. Moreover, object property labels are provided to the GCN to adjust in-hand manipulation motions. Distributed tri-axial tactile sensors are mounted on the fingertips, finger phalanges and palm of an Allegro hand, resulting in 1152 tactile measurements. Training data is collected with a data-glove to transfer human dexterous manipulation directly to the robot hand. The GCN achieved high success rates for in-hand manipulation. We also confirmed that fragile objects were deformed less when correct object labels were provided to the GCN. When visualizing the activation of the GCN with a PCA, we verified that the network acquired geodesical features. Our method achieved stable manipulation even when an experimenter pulled a grasped object and for untrained objects.

Submillimeter Lateral Displacement Enables Friction Sensing and Awareness of Surface Slipperiness.

Human tactile perception and motor control rely on the frictional estimates that stem from the deformation of the skin and slip events. However, it is not clear how exactly these mechanical events relate to the perception of friction. This study aims to quantify how minor lateral displacement and speed enables subjects to feel frictional differences. In a 2-alternative forced-choice protocol, an ultrasonic friction-reduction device was brought in contact perpendicular to the skin surface of an immobilized index finger; after reaching 1N normal force, the plate was moved laterally. A combination of four displacement magnitudes (0.2, 0.5, 1.2 and 2 mm), two levels of friction (high, low) and three displacement speeds (1, 5 and 10 mm/s) were tested. We found that the perception of frictional difference was enabled by submillimeter range lateral displacement. Friction discrimination thresholds were reached with lateral displacements ranging from 0.2 to 0.5 mm and surprisingly speed had only a marginal effect. These results demonstrate that partial slips are sufficient to cause awareness of surface slipperiness. These quantitative data are crucial for designing haptic devices that render slipperiness. The results also show the importance of subtle lateral finger movements present during dexterous manipulation tasks.

A Soft Robotic Gripper With an Active Palm and Reconfigurable Fingers for Fully Dexterous In-Hand Manipulation

This study is focused on developing a new dexterous soft robotic gripper with three fingers and an active palm capable of performing in-hand manipulation purposes. This innovative design meets all the dexterous manipulation requirements without any increase in mechanical complexity. In each finger, the bending position can be modified and controlled by moving a stiff rod inserted inside the center hole of the finger. In this way, the effective length of the manipulation can be changed. As a result, these reconfigurable fingers provide a more accessible workspace than conventional soft grippers. Besides, a large diversity of the finger's shape configurations results in more dexterity and in-hand manipulation capability. Workspace analysis is accomplished to characterize the advantages of the proposed design. The effectiveness of this soft robotic gripper is validated by different in-hand manipulation experimental tests, including rotation, regrasping, and rolling. The results suggest a promising solution to bridge the design gap between hard and soft robots for dexterous manipulation tasks. The hybrid design carries advantages of these two classes, such as reconfigurability, position, and shape control from hard robots, with large degrees of freedom (DOFs), complex deformations, and lightweight from soft robots. Like human manipulation, the palm plays a major role in stable grasping, especially for enhancing the in-hand manipulation capability. Therefore, we also investigate two types of vacuum palms (suction cup and granular particles) to guarantee a wide range of object manipulation tasks that cannot be completely performed by previously suggested soft grippers.