Modelling and Control for Soft Finger Manipulation and Human-Robot Interaction

Biologically inspired robotic systems are becoming increasingly popular, especially in the field of medical robotics, in which building robotic devices able to replicate the human behavior guarantees obtaining motor recovery, functional substitution or human-robot interaction as human-like as possible. It is widely recognized that robotic rehabilitation devices improve the performance of the rehabilitation therapy performed by a human therapist in terms of action repetition and accurate tracking of the desired trajectory. Taking advantage from the plasticity of the neuro-muscular system, a human-inspired robotic rehabilitation therapy helps patients to re-learn movements. In the field of upper limb prosthetics, since the aim of a prosthetic hand is to replace a human hand, the robotic device has to be not only functional, but also as similar as possible to the human one both from the morphological point of view and as regards movement naturalness. On the other hand, since grasping is one of the human skills that robotic researchers mostly attempt at imitating, in the development of new robotic hands, the inspiration to the human hand behavior is increasing. From the analysis of the grasping action performed by human beings and from the study of the anatomy of the human hand and of its behavior during grasping, it is possible to obtain useful information for developing human-like grasping algorithms so as to acquire a better knowledge of the hand kinematics in order to design new human-like robotic hands and new rehabilitation devices. The definition of the kinematic structure of the hand and of the fingers is, in fact, the basis for designing new dexterous robotic hands and devices devoted to interact with the human hand (such as rehabilitation devices). Therefore this work is focused on the study of the hand kinematics, providing the basis for a further study regarding the hand dynamics. All the experiments done are in fact adaptable for a future study of the hand dynamics. In assistive robotics, as well as in the field of hand prostheses, the ability of performing smooth movements and obtaining a stable grasp is essential. Therefore, one of the aims of this thesis is to develop a bio-inspired approach for posture prediction and finger trajectory planning with a robotic hand. In order to do that, the human grasping action has been deeply analyzed. It has been decomposed in three main phases: reaching, pre-shaping and grasping. In order to reduce the complexity of planning dexterous hand grasps, it is useful to find the best hand preshape: therefore, this work is focused on this grasping phase. An accurate analysis of anatomy, surgery and rehabilitation literature has been done. In order to confirm the literature results and to cope with the lack of information, e.g. about thumb behavior, different methods for acquiring information about the human hand behavior have been used. Some important features about grasping have been collected from the analysis of the data obtained from two different devices for movement analysis: the Vicon system and a sensorized glove (the CyberGlove). The hand joint behavior during the grasping action has been analyzed asking different subjects to realize four different grasping tasks. The selected tasks guarantee that the subjects pose the hand in the most commonly used configurations. The experiments were performed asking subjects to wear the CyberGlove or attaching on their hands markers visible by the Vicon cameras. The obtained data have been analyzed using different hand kinematic human-inspired models. In order to overcome the drawbacks of the motion analysis devices listed before (as the not completely natural movements performed wearing a data glove, the impossibility to use the CyberGlove from people of different hand sizes and the high cost of the Vicon system), and to obtain information about the hand movements, the Kinect motion sensing device has also been used. For determining the finger joint positions and trajectories during hand movements, a finger tracking algorithm for the Kinect camera has been implemented. Blue markers have been placed on the hand joints following the same configuration used in the experiments performed with the Vicon cameras. A coloured blob detection algorithm and a multiple object tracking algorithm based on particle filters and extended Kalman filter has been implemented. When observing the human grasping behavior, thanks to the input devices listed before, it has been possible to notice some common characteristics among different subjects. The literature results about the dependence of grasping shape on object properties and grip types have been confirmed. The relationship between hand joints for each subject and among different subject has been investigated. One of the obtained results has been finding a constant value of the hand aperture angle (the angle between thumb and index finger). Also the curvature of the fingers is constant among different subjects (related to hand dimensions). Therefore, on the basis of neurological studies and of the analysis of the obtained data, a bio-inspired algorithm for predicting the power-grip posture and planning the finger trajectory of a robotic hand has been developed. The method estimates the best joint hand configuration during diagonal and transverse volar grasp minimizing a purposely defined objective function given by the sum of the joint distances from the object center of rotation (COR). The developed grasping algorithm calculates the position of the fingers for grasping, finding the best hand configuration that ensures a stable human-like grasp. The implementation of the algorithm on a real robotic platform has validated its effectiveness. From the above discussion, it is clear that the aim of this work is to find a way of exploiting the knowledge about a natural system, namely the human hand, in order to design a robotic system. After investigating and understanding in depth the human grasping action, the obtained results have multiple applications such as: overcoming the structural lack of the actual robotic hands (for instance, the non opposable thumb); developing new interfaces for rehabilitation (the finger tracking algorithm developed for the Kinect motion sensing device could be a new rehabilitation interface with potential application in the rehabilitation field); developing bio-inspired approaches for posture prediction and finger trajectory planning in order to perform a stable human-like grasp with a robotic hand.

The human hand is an amazing tool, demonstrated by its incredible motor capability and remarkable sense of touch. To enable robots to work in a human-centric environment, it is desirable to endow robotic hands with human-like capabilities for grasping and object manipulation. However, due to its inherent complexity and inevitable model uncertainty, robotic grasping and manipulation remains a challenge. This thesis focuses on grasp adaptation in the face of model and sensing uncertainties: Given an object whose properties are not known with certainty (e.g., shape, weight and external perturbation), and a multifingered robotic hand, we aim at determining where to put the fingers and how the fingers should adaptively interact with the object using tactile sensing, in order to achieve either a stable grasp or a desired dynamic behaviour. A central idea in this thesis is the object-centric dynamics: namely, that we express all control constraints into an object-centric representation. This simplifies computa- tion and makes the control versatile to the type of hands. This is an essential feature that distinguishes our work from other robust grasping work in the literature, where generating a static stable grasp for a given hand is usually the primary goal. In this thesis, grasp adaptation is a dynamic process that flexibly adapts the grasp to fit some purpose from the objectâ s perspective, in the presence of a variety of uncertainties and/or perturbations. When building a grasp adaptation for a given situation, there are two key problems that must be addressed: 1) the problem of choosing an initial grasp that is suitable for future adaptation, and more importantly 2) the problem of design- ing an adaptation strategy that can react adequately to achieve desired behaviour of the grasped object. To address challenge 1 (planning a grasp under shape uncertainty), we propose an approach to parameterizing the uncertainty in object shape using Gaussian Processes (GPs) and incorporate it as a constraint into contact-level grasp planning. To realize the planned contacts using different hands interchangeably, we further develop a prob- abilistic model to predict the feasible hand configurations, including hand pose and finger joints, given the desired contact points only. The model is built using the con- cept of Virtual Frame(VF), and it is independent from the choice of hand frame and object frame. The performance of the proposed approach is validated on two differ- ent robotic hands, an industrial gripper (4 DOF Barrett hand) and a humanoid hand (16 DOF Allegro hand) to manipulate objects of daily use with complex geometry and various texture (a spray bottle, a tea caddy, a jug and a bunny toy). In the second part of this thesis, we propose an approach to the design of adapta- tion strategy to ensure grasp stability in the presence of physical uncertainties of objects(object weight, friction at contacts and external perturbation). Based on an object-level impedance controller, we first design a grasp stability estimator in the object frame using the grasp experience and tactile sensing. Once a grasp is predicted to be unstable during online execution, the grasp adaptation strategy is triggered to improve the grasp stability, by either changing the stiffness at finger level or relocating the position of one fingertip to a better area.

The significant advances made in the design and construction of anthropomorphic robot hands, endow them with prehensile abilities reaching that of humans. However, using these powerful hands with the same level of expertise that humans display is a big challenge for robots. Traditional approaches use finger-tip (precision) or enveloping (power) methods to generate the best force closure grasps. However, this ignores the variety of prehensile postures available to the hand and also the larger context of arm action. This thesis explores a paradigm for grasp formation based on generating oppositional pressure within the hand, which has been proposed as a functional basis for grasping in humans (MacKenzie and Iberall, 1994). A set of opposition primitives encapsulates the hand's ability to generate oppositional forces. The oppositional intention encoded in a primitive serves as a guide to match the hand to the object, quantify its functional ability and relate this to the arm. In this thesis we leverage the properties of opposition primitives to both interpret grasps formed by humans and to construct grasps for a robot considering the larger context of arm action. In the first part of the thesis we examine the hypothesis that hand representation schemes based on opposition are correlated with hand function. We propose hand-parameters describing oppositional intention and compare these with commonly used methods such as joint angles, joint synergies and shape features. We expect that opposition-based parameterizations, which take an interaction-based perspective of a grasp, are able to discriminate between grasps that are similar in shape but different in functional intent. We test this hypothesis using qualitative assessment of precision and power capabilities found in existing grasp taxonomies. The next part of the thesis presents a general method to recognize oppositional intention manifested in human grasp demonstrations. A data glove instrumented with tactile sensors is used to provide the raw information regarding hand configuration and interaction force. For a grasp combining several cooperating oppositional intentions, hand surfaces can be simultaneously involved in multiple oppositional roles. We characterize the low-level interactions between different surfaces of the hand based on captured interaction force and reconstructed hand surface geometry. This is subsequently used to separate out and prioritize multiple and possibly overlapping oppositional intentions present in the demonstrated grasp. We evaluate our method on several human subjects across a wide range of hand functions. The last part of the thesis applies the properties encoded in opposition primitives to optimize task performance of the arm, for tasks where the arm assumes the dominant role. For these tasks, choosing the strongest power grasp available (from a force-closure sense) may constrain the arm to a sub-optimal configuration. Weaker grasp components impose fewer constraints on the hand, and can therefore explore a wider region of the object relative pose space. We take advantage of this to find the good arm configurations from a task perspective. The final hand-arm configuration is obtained by trading of overall robustness in the grasp with ability of the arm to perform the task. We validate our approach, using the tasks of cutting, hammering, screw-driving and opening a bottle-cap, for both human and robotic hand-arm systems.

Bidirectional Human-Robot Learning: Imitation and Skill Improvement

This paper describes the application of our theory for kinematic synthesis of articulated systems with contact task specifications to the simultaneous synthesis of mechanical fingers for coordinated movements. The contact direction and curvature constraints between the fingers and a body are transformed into conditions on the velocity and acceleration of certain points using the task geometry. The addition of requirements on the accelerations allows for a more accurate definition of the tasks in the vicinity of the specified positions, thus considering the local motions of the fingertips and a grasped object and accounts for the smoothness of motion on the design level. The position and higher order motion specifications provide position, velocity and acceleration synthesis equations, which can be solved in order for the fingers to obtain the desired coordinated task. It should be noted that the use of kinematic synthesis as a first step in the design of the multi-fingered robotic hands has been applied to individual fingers, however a technique for doing this simultaneously for multiple fingers does not exist.

This paper presents a methodology to perform a fine manipulation task with an anthropomorphic hand. We focus here on a three-finger precision grasp. The robotic hand used is the latest four finger fully actuated hand developed by the RoBioSS team of PPRIME institute. Different steps ranging from the generation of the contact points of the hand with the object to the experimental realization of the task are presented. The optimal grasp is found by implementing a genetic algorithm. The fingertips planning for the desired object motion is done by using the small displacement model. The proposed approach is illustrated by the practical experiment of screwing a light bulb with the hand.

The capability to manipulate small objects is one of the important requirements for producing assembly work robots. Moreover, a robot that exhibits human-like skills could be used to reduce the high labor cost for complex tasks. Therefore, we propose a seven-link dual-arm robot with three-fingered hands for cooperative tasks to manipulate small parts like nuts and bolts. As a continuity from a previous work in [1], we have proposed a method for calculating the 3D target pose in [2] to solve the problem of inclined target during grasping and screwing tasks. The study in [2] has presented a different approach to identify the geometric feature points of a target using vector calculations and the estimation of 6-DOF posture based on a method that is defined as CER (Confirm-Estimate-Rotate). In this paper, the implementation of this method on a 7-link robot arm in a visual servoing structure that focuses on an M10 nut as the starting point for a grasping task is shown. The dynamical system requires attention in many aspects including control, sensor integration, image processing and trajectory planning strategies for manipulation of small nuts. Here, the required kinematics adjustments for the motion control, grasping paths and configuration for robot fingers for attending the target in real-time operation is presented. Experimental results indicate the capability of our method to change the robot’s posture safely to match the posture of a target in inclined positions. Furthermore, a statistical grasping result has shown a moderate performance for various poses of nuts. Thus, this shows the improvement of robot for object manipulation and its applicability for more complex tasks in future.

The purpose of this paper is to present the expert massage robot using a multi-fingered robot hand. First, the fingertip forces applied by an expert human therapist was measured using sheet distrib...

The application of dexterous robotic hands out of research laboratories has been limited by the intrinsic complexity that these devices present. This is directly reflected as an economically unreasonable cost and a low overall reliability. Within the research reported in this thesis it is shown how the problem of complexity in the design of robotic hands can be tackled, taking advantage of modern technologies (i.e. rapid prototyping), leading to innovative concepts for the design of the mechanical structure, the actuation and sensory systems. The solutions adopted drastically reduce the prototyping and production costs and increase the reliability, reducing the number of parts required and averaging their single reliability factors. In order to get guidelines for the design process, the problem of robotic grasp and manipulation by a dual arm/hand system has been reviewed. In this way, the requirements that should be fulfilled at hardware level to guarantee successful execution of the task has been highlighted. The contribution of this research from the manipulation planning side focuses on the redundancy resolution that arise in the execution of the task in a dexterous arm/hand system. In literature the problem of coordination of arm and hand during manipulation of an object has been widely analyzed in theory but often experimentally demonstrated in simplified robotic setup. Our aim is to cover the lack in the study of this topic and experimentally evaluate it in a complex system as a anthropomorphic arm hand system.

The RoboFriend project is a student-based research project created to utilize elements of Human-Robot Interaction (HRI) in tackling the problem of motion planning in dynamic environment for robots with higher DOFs. By developing a humanoid robotic arm (9 DOFs), to be controlled using visual and/or audio input signals from users, and use these queues to learn to avoid similar obstacles in the future. This project was divided into 4 major sub-projects; development of the gripper (wrist to fingers), development of manipulator (shoulder to wrist), development of the electronics and controls, and finally development of the HRI interface. Each sub-project was given to an engineering student, to be completed as his/her final year engineering project. Being a student-based project, this work had a secondary objective; to investigate the effectiveness of incorporating robotics research in undergraduate engineering education. Challenges included the design/development of the humanoid gripper, due to its conflicting design parameters of flexibility and structural stability. At the end, the humanoid arm was made to response to human voice. Data also showed that research enhanced students’ learning experience as students were able to demonstrate their recently acquired knowledge in contributing to this work. Therefore, it can be concluded that incorporating research in undergraduate engineering studies is effective.

This paper presents a tele-control system constructed from a multi-fingered robot hand and operator. The angle of the robot hand is controlled by the angle of the operator’s finger, and the operator feels the environmental force, as detected by the robot hand, constituting so-called bilateral master/slave control.

Data glove is one of the most commonly used techniques in the robotic teleoperation systems. In this paper, we propose a robotic hand-arm teleoperation system with a novel data glove called YoBu, which can acquire human motions from both the arm and the hand simultaneously. The proposed data glove is designed to be stable, compact and portable. It is composed of eighteen low-cost inertial and magnetic measurement units, among which fifteen units are attached to the human operator's finger joints for robotic hand teleoperation and three units are attached to the palm, upper arm and forearm respectively for robotic arm teleoperation. In the robotic hand-arm teleoperation system, the operating commands generated by the data glove are transmitted to the robot via a Bluetooth wireless communication, which makes the whole robotic teleoperation system simple and user friendly. Finally, several experiments are implemented to verify the efficiency of the proposed robotic teleoperation system.

Enhancing Humans Trust in Robots through Explanations

Understanding the principles involved in visually-based coordinated motor control is one of the most fundamental and most intriguing research problems across a number of areas, including psychology, neuroscience, computer vision and robotics. Not very much is known regarding computational functions that the central nervous system performs in order to provide a set of requirements for visually-driven reaching and grasping. Additionally, in spite of several decades of advances in the field, the abilities of humanoids to perform similar tasks are by far modest when needed to operate in unstructured and dynamically changing environments. More specifically, our first focus is understanding the principles involved in human visuomotor coordination. Not many behavioral studies considered visuomotor coordination in natural, unrestricted, head-free movements in complex scenarios such as obstacle avoidance. To fill this gap, we provide an assessment of visuomotor coordination when humans perform prehensile tasks with obstacle avoidance, an issue that has received far less attention. Namely, we quantify the relationships between the gaze and arm-hand systems, so as to inform robotic models, and we investigate how the presence of an obstacle modulates this pattern of correlations. Second, to complement these observations, we provide a robotic model of visuomotor coordination, with and without the presence of obstacles in the workspace. The parameters of the controller are solely estimated by using the human motion capture data from our human study. This controller has a number of interesting properties. It provides an efficient way to control the gaze, arm and hand movements in a stable and coordinated manner. When facing perturbations while reaching and grasping, our controller adapts its behavior almost instantly, while preserving coordination between the gaze, arm, and hand. In the third part of the thesis, we study the neuroscientific literature of the primates. We here stress the view that the cerebellum uses the cortical reference frame representation. The cerebellum by taking into account this representation performs closed-loop programming of multi-joint movements and movement synchronization between the eye-head system, arm and hand. Based on this investigation, we propose a functional architecture of the cerebellar-cortical involvement. We derive a number of improvements of our visuomotor controller for obstacle-free reaching and grasping. Because this model is devised by carefully taking into account the neuroscientific evidence, we are able to provide a number of testable predictions about the functions of the central nervous system in visuomotor coordination. Finally, we tackle the flow of the visuomotor coordination in the direction from the arm-hand system to the visual system. We develop two models of motor-primed attention for humanoid robots. Motor-priming of attention is a mechanism that implements prioritizing of visual processing with respect to motor-relevant parts of the visual field. Recent studies in humans and monkeys have shown that visual attention supporting natural behavior is not exclusively defined in terms of visual saliency in color or texture cues, rather the reachable space and motor plans present the predominant source of this attentional modulation. Here, we show that motor-priming of visual attention can be used to efficiently distribute robot's computational resources devoted to visual processing.

Studies on grasping and manipulation by robotic multifingered hands and arm-hand systems