Achieving humanlike dexterity with anthropomorphic multifingered robotic hands requires precise finger coordination. However, dexterous manipulation remains highly challenging because of high-dimensional action-observation spaces, complex hand-object contact dynamics, and frequent occlusions. To address this, we drew inspiration from the human learning paradigm of observation and practice and propose a two-stage learning framework by learning visual-tactile integration representations via self-supervised learning from human demonstrations. We trained a unified multitask policy through reinforcement learning and online imitation learning. This decoupled learning enabled the robot to acquire generalizable manipulation skills using only monocular images and simple binary tactile signals. With the unified policy, we built a multifingered hand manipulation system that performs multiple complicated tasks with low-cost sensing. It achieved an 85% success rate across five complex tasks and 25 objects and further generalized to three unseen tasks that share similar hand-object coordination patterns with the training tasks.

The multi-finger robotic hand exhibits significant potential in grasping tasks owing to its high degrees of freedom (DoFs). Object grasping results in a closed-chain kinematic system between the hand and the object. This increases the dimensionality of trajectory tracking and substantially raises the computational complexity of traditional methods. Therefore, this study proposes the discrete finite-time convergent neurodynamics (DFTCN) algorithm to address the aforementioned issue. Specifically, a time-varying quadratic programming (TVQP) problem is formulated for each finger, incorporating joint angle and angular velocity constraints through log-sum-exp (LSE) functions. The TVQP problem is then transformed into a time-varying equation system (TVES) problem using the Karush–Kuhn–Tucker (KKT) conditions. A novel control law is designed, employing a three-step Taylor-type discretization for efficient implementation. Theoretical analysis verifies the algorithm’s stability and finite-time convergence property, with the maximum steady-state residual error being O(τ3). Numerical simulations illustrate the favorable convergence and high accuracy of the DFTCN algorithm compared with three existing dominant neurodynamic algorithms. The real-robot experiments further confirm its capability for precise grasping, even in the presence of camera noise and external disturbances.

A challenging and important problem for tendon-driven multi-fingered robotic hands is to ensure grasping adaptivity while minimizing the number of actuators needed to provide human-like functionality. Inspired by the Pisa/IIT SoftHand, this paper introduces a 3D-printed, highly underactuated, tactile-sensorized, five-finger robotic hand named the Tactile SoftHand-A, which features an antagonistic mechanism to actively open and close the hand. Our proposed dual-tendon design gives options that allow active control of specific (distal or proximal interphalangeal) joints; for example, to adjust from an enclosing to fingertip grasp or to manipulate an object with a fingertip. We also develop and integrate a new design of fully 3D-printed vision-based tactile sensor within the fingers that requires minimal hand assembly. A control scheme based on analytically extracting contact location and slip from the tactile images is used to coordinate the antagonistic tendon mechanism (using a marker displacement density map, suitable for TacTip-based sensors). We perform extensive testing of a single finger, the entire hand, and the tactile capabilities to show the improvements in reactivity, load-bearing, and manipulability in comparison to a SoftHand that lacks the antagonistic mechanism. We also demonstrate the hand’s reactivity to contact disturbances including slip, and how this enables teleoperated control from human hand gestures. Overall, this study points the way towards a class of low-cost, accessible, 3D-printable, tactile, underactuated human-like robotic hands, and we openly release the designs to facilitate others to build upon this work. The designs are open-sourced at github.com/HaoranLi-Data/Tactile_SoftHand_A.

Assistive technologies, particularly multi-fingered robotic hands (MFRHs), are critical for enhancing the quality of life for individuals with upper-limb disabilities. However, achieving precise and stable control of such systems remains a significant challenge. This study proposes an Improved Grey Wolf Optimization (IGWO)-tuned Linear Quadratic Regulator (LQR) to enhance the control performance of an MFRH. The MFRH was modeled using Denavit-Hartenberg kinematics and Euler-Lagrange dynamics, with micro-DC motors selected based on computed torque requirements. The LQR controller, optimized via IGWO to systematically determine weighting matrices, was benchmarked against PID and PID-PSO controllers under diverse input scenarios. For step input, the IGWO-LQR achieved a settling time of 0.018 s with zero overshoot for Joint 1, outperforming PID (settling time: 0.0721 s; overshoot: 6.58%) and PID-PSO (settling time: 0.042 s; overshoot: 2.1%). Similar improvements were observed across all joints, with Joint 3 recording an IAE of 0.001334 for IGWO-LQR versus 0.004695 for PID. Evaluations under square-wave, sine, and sigmoid inputs further validated the controller's robustness, with IGWO-LQR consistently delivering minimal tracking errors and rapid stabilization. These results demonstrate that the IGWO-LQR framework significantly enhances precision and dynamic response.

This study aims to create a motion to manipulate paper without creating folds using high-speed vision, to replicate the human handling of flexible objects with a multi-fingered robot hand. By leveraging the physical properties of the paper, we propose a handling strategy that allows the paper to be tensioned in any direction. By elucidating the state transitions of the shape of the paper during dynamic manipulation, we achieved dynamic bending of the paper without creating creases. Two operation patterns were proposed: one using two hands to manipulate the paper from both sides and the other utilizing the inertia of rapidly swinging down the arm with one hand. Additionally, by clarifying the state transitions of the shape that appear during dynamic paper manipulation, we propose a method for bending paper dynamically without creating folds. This task was completed using image processing with high-speed vision to respond to paper deformation during the operation. The experimental results show that, by using high-speed visual feedback to monitor the shape of a paper, we can determine the appropriate timing for action switches, allowing the same operation to be executed on papers of different sizes.

Recently, aerial manipulations are becoming more and more important for the practical applications of unmanned aerial vehicles (UAV) to choose, transport, and place objects in global space. In this paper, an aerial manipulation system consisting of a UAV, two onboard cameras, and a multi-fingered robotic hand with proximity sensors is developed. To achieve self-contained autonomous navigation to a targeted object, onboard tracking and depth cameras are used to detect the targeted object and to control the UAV to reach the target object, even in a Global Positioning System-denied environment. The robotic hand can perform proximity sensor-based grasping stably for an object that is within a position error tolerance (a circle with a radius of 50 mm) from the center of the hand. Therefore, to successfully grasp the object, a requirement for the position error of the hand (=UAV) during hovering after reaching the targeted object should be less than the tolerance. To meet this requirement, an object detection algorithm to support accurate target localization by combining information from both cameras was developed. In addition, camera mount orientation and UAV attitude sampling rate were determined by experiments, and it is confirmed that these implementations improved the UAV position error to within the grasping tolerance of the robot hand. Finally, the experiments on aerial manipulations using the developed system demonstrated the successful grasping of the targeted object.

In this study, we propose a multi-finger robotic hand with an iris mechanism that positions the blades of the hand, with fingers extending perpendicular to the blades attached to the blade tips, using a swing mechanism external to the iris mechanism. A single actuator drives the hand, which can concentrically grasp objects at multiple points while maintaining a regular polygonal shape around the entire circumference. The proposed structure can use the same mechanism to realize a hand with an arbitrary number of blades. Furthermore, the gears can be arranged in a planar configuration within the iris mechanism, thereby enabling adjustments to the grasping torque and speed by changing the gear ratio. This study examines the appropriate number of fingers for this iris multi-finger robotic hand, along with the cross-sectional shape of those fingers, according to the object to be grasped. We perform a geometric analysis of the relation between the rotational angle of the blades and the sizes of graspable objects, considering both the circumferential multi-point grasping of cylindrical objects and the multi-point pinching of rectangular objects. By comparing the analysis results of these different grasping methods, we consider how differences in the number of fingers and their cross-sectional shapes affect the size of graspable objects. The findings provide specific guidelines on the selection of an appropriate iris multi-finger robotic hand configuration for specific target objects.

Underactuated robotic hands are extensively used in remote manipulation due to their ability to adapt to various object sizes and shapes. Their structural simplicity and small number of actuators required for operation make them highly versatile and responsive, which is crucial for effective teleoperation. In addition to grasping performance, haptic feedback, which integrates force and tactile sensing, is essential for dexterous manipulation. This study proposes a solution using fiber-optic tendons embedded with fiber Bragg gratings (FBGs), combining sensing and actuation to simultaneously perform power transmission, along with force and tactile sensing. Each finger employs a fiber-optic tendon with three FBGs: one measures tendon tension, and the other two at the fingertip detect contact force and temperature. The tendon is placed on the volar side of the finger and routed to an actuation module with a servomotor at the wrist for power transmission. This tendon enables finger flexion, while a passive extension mechanism with linear springs on the dorsal side facilitates extension. Experimental results demonstrate the feasibility of this approach, showing the hand's multifunctional capabilities, including haptic feedback and power transmission, as well as its potential for teleoperation. This approach improves the robotic hand's ability to provide real-time feedback, improving dexterity in remote manipulation.

Industrial robot manipulators are controlled by motors installed in each joint. If the joints of a multi-fingered robotic hand are controlled by independent motors, a large number of motors would be required. As a result, the entire device, including the transmission system, becomes very large. In this study, we developed a wire-driven multi-fingered robot hand with electromagnetic clutches. Each electromagnetic clutch controls the connection between its associated winding drum and the motor shaft. Since multiple electromagnetic clutches can share the same motor shaft, the number of motors can be reduced. By using the voltage of the electromagnetic clutch as the control input of the feedback control system, the bending of each finger can be controlled independently.

Dexterous in-hand manipulation in robotics, particularly with multi-fingered robotic hands, poses significant challenges due to the intricate avoidance of collisions among fingers and the object being manipulated. Collision-free paths for all fingers must be generated in real-time, as the rapid changes in hand and finger positions necessitate instantaneous recalculations to prevent collisions and ensure undisturbed movement. This study introduces a real-time approach to motion planning in high-dimensional spaces. We first explicitly model the collisionfree space using neural networks that are retrievable in real time. Then, we combined the C-space representation with closed-loop control via dynamical system and sampling-based planning approaches. This integration enhances the efficiency and feasibility of path-finding, enabling dynamic obstacle avoidance, thereby advancing the capabilities of multi-fingered robotic hands for in-hand manipulation tasks.

Grasp planning for irregularly shaped objects using multifingered robotic hands is challenging due to the high dimensionality of the search space and a lack of proper modeling methods for object geometry. To address these issues, we propose a grasp planning approach based on Gaussian process implicit surfaces (GPIS). To explore the object geometry and identify feasible contact positions and normals, our method introduces several moving points called attractors along with a dynamical system. The dynamical system constrains and guides the attractors with the partial differentials of the GPIS, which can be conveniently obtained through the linear expression of a GP. The hand motion is also guided by the dynamical system. In addition, an inverse kinematics method, which considers finger joint limits, is developed to simultaneously adjust the palm pose and finger joint angles for a feasible grasp. The performance of our method is demonstrated using various robotic hands and objects, and real robot experiments are conducted to validate the planned grasp's effectiveness in reality. Experimental evaluation demonstrates that the method works for different robotic hands and objects of varying shapes, with a higher likelihood of generating grasps with better quality.

Multifingered robot hands can be extremely effective in physically exploring and recognizing objects, especially if they are extensively covered with distributed tactile sensors. Convolutional neural networks (CNNs) have been proven successful in processing high dimensional data, such as camera images, and are, therefore, very well suited to analyze distributed tactile information as well. However, a major challenge is to organize tactile inputs coming from different locations on the hand in a coherent structure that could leverage the computational properties of the CNN. Therefore, we introduce a morphology-specific CNN (MS-CNN), in which hierarchical convolutional layers are formed following the physical configuration of the tactile sensors on the robot. We equipped a four-fingered Allegro robot hand with several uSkin tactile sensors; overall, the hand is covered with 240 sensitive elements, each one measuring three-axis contact force. The MS-CNN layers process the tactile data hierarchically: at the level of small local clusters first, then each finger, and then the entire hand. We show experimentally that, after training, the robot hand can successfully recognize objects by a single touch, with a recognition rate of over 95%. Interestingly, the learned MS-CNN representation transfers well to novel tasks: by adding a limited amount of data about new objects, the network can recognize nine types of physical properties.

This article presents a study on the neurobiological control of voluntary movements for anthropomorphic robotic systems. A corticospinal neural network model has been developed to control joint trajectories in multi-fingered robotic hands. The proposed neural network simulates cortical and spinal areas, as well as the connectivity between them, during the execution of voluntary movements similar to those performed by humans or monkeys. Furthermore, this neural connection allows for the interpretation of functional roles in the motor areas of the brain. The proposed neural control system is tested on the fingers of a robotic hand, which is driven by agonist-antagonist tendons and actuators designed to accurately emulate complex muscular functionality. The experimental results show that the corticospinal controller produces key properties of biological movement control, such as bell-shaped asymmetric velocity profiles and the ability to compensate for disturbances. Movements are dynamically compensated for through sensory feedback. Based on the experimental results, it is concluded that the proposed biologically inspired adaptive neural control system is robust, reliable, and adaptable to robotic platforms with diverse biomechanics and degrees of freedom. The corticospinal network successfully integrates biological concepts with engineering control theory for the generation of functional movement. This research significantly contributes to improving our understanding of neuromotor control in both animals and humans, thus paving the way towards a new frontier in the field of neurobiological control of anthropomorphic robotic systems.

The intricate kinematics of the human hand enable simultaneous grasping and manipulation of multiple objects, essential for tasks, such as object transfer and in-hand manipulation. Despite its significance, the domain of robotic multi-object grasping is relatively unexplored and presents notable challenges in kinematics, dynamics, and object configurations. This letter introduces <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MultiGrasp</i> , a novel two-stage approach for multi-object grasping using a dexterous multi-fingered robotic hand on a tabletop. The process consists of (i) generating pre-grasp proposals and (ii) executing the grasp and lifting the objects. Our experimental focus is primarily on dual-object grasping, achieving a success rate of 44.13%, highlighting adaptability to new object configurations and tolerance for imprecise grasps. Additionally, the framework demonstrates the potential for grasping more than two objects at the cost of inference speed.

In the field of robotic hands, finger force coordination is usually achieved by complex mechanical structures and control systems. This study presents the design of a novel transmission system inspired from the physiological concept of force synergies, aiming to simplify the control of multifingered robotic hands. To this end, we collected human finger force data during six isometric grasping tasks, and force synergies (i.e. the synergy weightings and the corresponding activation coefficients) were extracted from the concatenated force data to explore their potential for force modulation. We then implemented two force synergies with a cable-driven transmission mechanism consisting of two spring-loaded sliders and five V-shaped bars. Specifically, we used fixed synergy weightings to determine the stiffness of the compression springs, and the displacements of sliders were determined by time-varying activation coefficients. The derived transmission system was then used to drive a five-finger robotic hand named SYN hand. We also designed a motion encoder to selectively activate desired fingers, making it possible for two motors to empower a variety of hand postures. Experiments on the prototype demonstrate successful grasp of a wide range of objects in everyday life, and the finger force distribution of SYN hand can approximate that of human hand during six typical tasks. To our best knowledge, this study shows the first attempt to mechanically implement force synergies for finger force modulation in a robotic hand. In comparison to state-of-the-art robotic hands with similar functionality, the proposed hand can distribute humanlike force ratios on the fingers by simple position control, rather than resorting to additional force sensors or complex control strategies. The outcome of this study may provide alternatives for the design of novel anthropomorphic robotic hands, and thus show application prospects in the field of hand prostheses and exoskeletons.

Detecting contact when fingers are approaching an object and estimating the magnitude of the force the fingers are exerting on the object after contact are important tasks for a multi-fingered robotic hand to stably grasp objects. However, for a linkage-based under-actuated robotic hand with a self-locking mechanism to realize stable grasping without using external sensors, such tasks are difficult to perform when only analyzing the robot model or only applying data-driven methods. Therefore, in this paper, a hybrid of previous approaches is used to find a solution for realizing stable grasping with an under-actuated hand. First, data from the internal sensors of a robotic hand are collected during its operation. Subsequently, using the robot model to analyze the collected data, the differences between the model and real data are explained. From the analysis, novel data-driven-based algorithms, which can overcome noted challenges to detect contact between a fingertip and the object and estimate the fingertip forces in real-time, are introduced. The proposed methods are finally used in a stable grasp controller to control a triple-fingered under-actuated robotic hand to perform stable grasping. The results of the experiments are analyzed to show that the proposed algorithms work well for this task and can be further developed to be used for other future dexterous manipulation tasks.

Markerless vision-based teleoperation that leverages innovations in computer vision offers the advantages of allowing natural and noninvasive finger motions for multifingered robot hands. However, current pose estimation methods still face inaccuracy issues due to the self-occlusion of the fingers. Herein, we develop a novel vision-based hand-arm teleoperation system that captures the human hands from the best viewpoint and at a suitable distance. This teleoperation system consists of an end-to-end hand pose regression network and a controlled active vision system. The end-to-end pose regression network (Transteleop), combined with an auxiliary reconstruction loss function, captures the human hand through a low-cost depth camera and predicts joint commands of the robot based on the image-to-image translation method. To obtain the optimal observation of the human hand, an active vision system is implemented by a robot arm at the local site that ensures the high accuracy of the proposed neural network. Human arm motions are simultaneously mapped to the slave robot arm under relative control. Quantitative network evaluation and a variety of complex manipulation tasks, for example, tower building, pouring, and multitable cup stacking, demonstrate the practicality and stability of the proposed teleoperation system.

Teleoperation enables a user to perform dangerous tasks (e.g., work in disaster zones or in chemical plants) from a remote location. Nevertheless, common approaches often provide cumbersome and unnatural usage. In this letter, we propose TeleFMG, an approach for teleoperation of a multi-finger robotic hand through natural motions of the user's hand. By using a lowcost wearable Force-Myography (FMG) device, musculoskeletal activities on the user's forearm are mapped to hand poses which, in turn, are mimicked by a robotic hand. The mapping is performed by a spatio-temporal data-based model based on the Temporal Convolutional Network. The model considers spatial positions of the sensors on the forearm along with temporal dependencies of the FMG signals. A set of experiments show the ability of a teleoperator to control a multi-finger hand through intuitive and natural finger motion. A robot is shown to successfully mimic the user's hand in object grasping and gestures. Furthermore, transfer to a new user is evaluated while showing that fine-tuning with a limited amount of new data significantly improves accuracy.

A multi-finger robotic hand with an iris mechanism that we previously developed was driven by a single actuator and could grasp an object by wrapping fingers completely around its circumference at multiple points. However, the blades used to grasp objects were within the robotic hand mechanism, so it could only grasp objects small enough to fit within the hollow disk comprising the outer surface of the device body. Furthermore, the hand could not grasp objects smaller than the thickness of the hollow disk. The multi-fingered robotic hand proposed in this study has a new mechanism in which the blades of the iris are placed outside of the hand mechanism, and fingers shaped as equilateral triangular prisms extend perpendicular to the disk of the robotic hand body and are attached to the blade tip. Placing the blade outside the mechanism and adjusting the gear ratios within allows adjustments to the gripping torque and speed. The vertically extended fingers can thus grasp small objects and objects longer than the blade diameter. In this study, we performed geometric and theoretical analyses of the proposed multi-fingered robotic hand. We then fabricated an actual robotic hand, verified the validity of the analyses.

Performing dexterous tasks with multi-fingered hand is still a challenge. Tactile sensors can provide touch states and the object features during in-hand manipulation. However, even with such rich data of touch states, achieving dexterous multi-fingered tasks is further complicated because of the underlying complexities. This paper presents a method for object property recognition, a Multi-Thread GCN (MT-GCN) architecture to process tactile and edge features and basically multi-modal data in a graph. The MT-GCN with tactile and edge features achieved high recognition rate, 86.08% for 6 classes of object property combinations from 8 objects. We could confirm that the graph edge features acquired by real robotic configuration and the MT-GCN architecture were effective for multi-fingered dexterous tasks.