Manipulation Primitives Research Articles

Autogenerated manipulation primitives

The central theme in robotic manipulation is that of the robot interacting with the world through physical contact. We tend to describe that physical contact using specific words that capture the nature of the contact and the action, such as grasp, roll, pivot, push, pull, tilt, close, open etc. We refer to these situation-specific actions as manipulation primitives. Due to the nonlinear and nonsmooth nature of physical interaction, roboticists have devoted significant efforts towards studying individual manipulation primitives. However, studying individual primitives one by one is an inherently limited process, due engineering costs, overfitting to specific tasks, and lack of robustness to unforeseen variations. These limitations motivate the main contribution of this paper: a complete and general framework to autogenerate manipulation primitives. To do so, we develop the theory and computation of contact modes as a means to classify and enumerate manipulation primitives. The contact modes form a graph, specifically a lattice. Our algorithm to autogenerate manipulation primitives (AMP) performs graph-based optimization on the contact mode lattice and solves a linear program to generate each primitive. We designed several experiments to validate our approach. We benchmarked a wide range of contact scenarios and our pipeline’s runtime was consistently in the 10 s of milliseconds. In simulation, we planned manipulation sequences using AMP. In the real-world, we showcased the robustness of our approach to real-world modeling errors. We hope that our contributions will lead to more general and robust approaches for robotic manipulation.

Model Predictive Interaction Control for Robotic Manipulation Tasks

This article presents the concept of model predictive interaction control (MPIC) as a generic, flexible, and comprehensive approach for robotic manipulation tasks. MPIC is based on the repetitive solution of an optimal control problem that includes a robot model for motion prediction as well as an interaction model for force prediction. In order to handle both elastic and rigid contact situations, a cascaded approach with low-level PD control is adopted, which allows to combine the linear-elastic environment model and the limited controller stiffness. Due to its flexibility, MPIC can be favorably used for realizing the elementary manipulation primitives (MP) within a hierarchical task planning framework, where each MP corresponds to a particular parameterization of the cost function and the constraints. The control methodology and the manipulation approach are evaluated in simulations and experiments using a 7-degree-of-freedom industrial robot.

Robotic Cable Routing with Spatial Representation

Cable routing is a challenging task for robotic automation. To accomplish the task, it requires a high-level path planner to generate a sequence of cable configurations from the initial state to the target state and a low-level manipulation planner to plan the robot motion commands to transit between adjacent states. However, there are yet no proper representations to model the cable with the environment objects, impeding the design of both high-level path planning and low-level manipulation planning. In this letter, we propose a framework for cable routing with spatial representation. For high-level planning, by considering the spatial relations between the cable and the environment objects such as fixtures, the proposed method is able to plan a path from the initial state to the goal state in a graph. For low-level manipulation, multiple manipulation primitives are efficiently learned from human demonstration, to configure the cable to planned intermediate states leveraging the same spatial representation. We also implement a cable state estimator that robustly extracts the spatial representation from raw RGB-D images, thus completing the cable routing framework. We evaluate the proposed framework with various cables and fixture settings, and demonstrate that it outperforms some baselines in terms of reliability and generalizability.

A Grasping-Centered Analysis for Cloth Manipulation

Compliant and soft hands have gained a lot of attention in the past decade because of their ability to adapt to the shape of the objects, increasing their effectiveness for grasping. However, when it comes to grasping highly flexible objects such as textiles, we face the dual problem: it is the object that will adapt to the shape of the hand or gripper. In this context, the classic grasp analysis or grasping taxonomies are not suitable for describing textile objects grasps. This article proposes a novel definition of textile object grasps that abstracts from the robotic embodiment or hand shape and recovers concepts from the early neuroscience literature on hand prehension skills. This framework enables us to identify what grasps have been used in literature until now to perform robotic cloth manipulation, and allows for a precise definition of all the tasks that have been tackled in terms of manipulation primitives based on regrasps. In addition, we also review what grippers have been used. Our analysis shows how the vast majority of cloth manipulations have relied only on one type of grasp, and at the same time we identify several tasks that need more variety of grasp types to be executed successfully. Our framework is generic, provides a classification of cloth manipulation primitives and can inspire gripper design and benchmark construction for cloth manipulation.

Selective grasp in occluded space by all-around proximity perceptible finger

The goal of this research is to develop a “Selective Grasp” system with which robots can grasp and identify the target object even in occluded environments. In pursuit of this goal, we first develop a robot hand on which proximity sensors are mounted all around. In addition to the development, we propose a sensor model of the robot hand. By using the sensor model, robots can estimate the distance to the object and calibrate the sensors. With our robot hand, robots can accurately recognize their surroundings without touch. Secondly, we propose an approach in which robots can memorize spatial information of surroundings by building an environment map. The building map motion is generated by a combination of manipulation primitives based on proximity sensors. Thirdly, we propose a grasp planning method and an object shape classification method based on the environment map. By these methods, robots can grasp objects and classify shapes of the objects in occluded spaces. Lastly, we conduct real robot experiments, through which we validate the effectiveness of our proposed Selective Grasp system.

Object Isolation with Minimal Impact Towards the Object of Interest in a Complex Environment using Manipulation Primitives

It is common in the field of robotic manipulation to specifically target and precisely move, displace or manipulate the targeted object of interest. This however may not always be the best possible course of action as there are situations where it is not possible to manipulate the object of interest or is not in a condition to be manipulated. This research paper explores and subsequently proposes 3 object isolation technique for the purpose of isolating a targeted object of interest from the environment incontras to the standard utilization of object singulation technique. The goal was to develop an algorithm than can successfully isolate the object of interest from the environment via removing the environment without/with minimal impact towards the object of interest. Results from the experiment indicated that the proposed algorithms can successfully isolate the object of interest with minimal impact towards the object of interest scoring an average of 0.85cm/actuation for MSMAPPS, 0.75cm/actuation for MSMAPOS and finally 0.27cm/actuation for BSMAPOS. These results indicates a relatively small displacement per actuation at 4.35% displacement per actuation, 3.75% displacement per actuation, and 1.35% displacement per actuation relative to the workspace respectively.

Modeling and Evaluation of Robust Whole-Hand Caging Manipulation

Human in-hand dexterity can be highly fluid and unstructured, with multiple phalanxes breaking and re-establishing contact during any given task. In contrast, prevailing research in robotic manipulation has focused on highly structured well-controlled motions, where contact points are carefully characterized. Maintaining grasp stability by satisfying traditional closure conditions during complex within-hand manipulation motions can be difficult, even with highly articulated end effectors. However, simple grippers can still achieve an effective range of in-hand manipulation tasks without strict closure conditions, as long as the object can be bounded locally relative to the hand frame. The end effector can be considered as a tool to limit the range of possible object poses. In particular, the energy of the hand-object system can be used to determine an attractor region toward which the hand drives the object. This can be combined with a sparse sampling of the configuration space to find a set of manipulation primitives that can reliably constrain the object inside the hand workspace even without feedback, a strategy proposed as whole-hand caging manipulation. In this paper, experimental results with a planar underactuated gripper are presented to validate this manipulation strategy, and it is shown that even though contacts are regularly broken and reformed, the object can be reliably manipulated within the hand workspace without ejection, and challenging movements such as sliding and gaiting can be reliably performed.

Shallow-Depth Insertion: Peg in Shallow Hole Through Robotic In-Hand Manipulation

This letter presents a novel robotic manipulation technique suitable for shallow-depth insertion , which refers to the assembly of a relatively thin peg-like object into a hole with a shallow depth, as can be seen in a cell phone battery insertion for example. Our technique features dexterous manipulation actions that combine into a complete insertion operation, as also commonly demonstrated by humans performing the battery insertion task. The quasistatic stability of the presented insertion process is shown by constructing the space of force-closure grasps and manipulation primitives to navigate the space by changing the grasps. Our technique is directly applicable to a simple hardware setting with the conventional parallel-jaw gripper installed on an industrial robot arm. A set of experiments performed with our software demonstrates the effectiveness of our technique with a high success rate.

DeepDynamicHand: A Deep Neural Architecture for Labeling Hand Manipulation Strategies in Video Sources Exploiting Temporal Information.

Humans are capable of complex manipulation interactions with the environment, relying on the intrinsic adaptability and compliance of their hands. Recently, soft robotic manipulation has attempted to reproduce such an extraordinary behavior, through the design of deformable yet robust end-effectors. To this goal, the investigation of human behavior has become crucial to correctly inform technological developments of robotic hands that can successfully exploit environmental constraint as humans actually do. Among the different tools robotics can leverage on to achieve this objective, deep learning has emerged as a promising approach for the study and then the implementation of neuro-scientific observations on the artificial side. However, current approaches tend to neglect the dynamic nature of hand pose recognition problems, limiting the effectiveness of these techniques in identifying sequences of manipulation primitives underpinning action generation, e.g., during purposeful interaction with the environment. In this work, we propose a vision-based supervised Hand Pose Recognition method which, for the first time, takes into account temporal information to identify meaningful sequences of actions in grasping and manipulation tasks. More specifically, we apply Deep Neural Networks to automatically learn features from hand posture images that consist of frames extracted from grasping and manipulation task videos with objects and external environmental constraints. For training purposes, videos are divided into intervals, each associated to a specific action by a human supervisor. The proposed algorithm combines a Convolutional Neural Network to detect the hand within each video frame and a Recurrent Neural Network to predict the hand action in the current frame, while taking into consideration the history of actions performed in the previous frames. Experimental validation has been performed on two datasets of dynamic hand-centric strategies, where subjects regularly interact with objects and environment. Proposed architecture achieved a very good classification accuracy on both datasets, reaching performance up to 94%, and outperforming state of the art techniques. The outcomes of this study can be successfully applied to robotics, e.g., for planning and control of soft anthropomorphic manipulators.

FFRob: Leveraging symbolic planning for efficient task and motion planning

Mobile manipulation problems involving many objects are challenging to solve due to the high dimensionality and multi-modality of their hybrid configuration spaces. Planners that perform a purely geometric search are prohibitively slow for solving these problems because they are unable to factor the configuration space. Symbolic task planners can efficiently construct plans involving many variables but cannot represent the geometric and kinematic constraints required in manipulation. We present the FFRob algorithm for solving task and motion planning problems. First, we introduce extended action specification (EAS) as a general purpose planning representation that supports arbitrary predicates as conditions. We adapt existing heuristic search ideas for solving strips planning problems, particularly delete-relaxations, to solve EAS problem instances. We then apply the EAS representation and planners to manipulation problems resulting in FFRob. FFRob iteratively discretizes task and motion planning problems using batch sampling of manipulation primitives and a multi-query roadmap structure that can be conditionalized to evaluate reachability under different placements of movable objects. This structure enables the EAS planner to efficiently compute heuristics that incorporate geometric and kinematic planning constraints to give a tight estimate of the distance to the goal. Additionally, we show FFRob is probabilistically complete and has a finite expected runtime. Finally, we empirically demonstrate FFRob’s effectiveness on complex and diverse task and motion planning tasks including rearrangement planning and navigation among movable objects.

Genotype Specification Language.

We describe here the Genotype Specification Language (GSL), a language that facilitates the rapid design of large and complex DNA constructs used to engineer genomes. The GSL compiler implements a high-level language based on traditional genetic notation, as well as a set of low-level DNA manipulation primitives. The language allows facile incorporation of parts from a library of cloned DNA constructs and from the "natural" library of parts in fully sequenced and annotated genomes. GSL was designed to engage genetic engineers in their native language while providing a framework for higher level abstract tooling. To this end we define four language levels, Level 0 (literal DNA sequence) through Level 3, with increasing abstraction of part selection and construction paths. GSL targets an intermediate language based on DNA slices that translates efficiently into a wide range of final output formats, such as FASTA and GenBank, and includes formats that specify instructions and materials such as oligonucleotide primers to allow the physical construction of the GSL designs by individual strain engineers or an automated DNA assembly core facility.

Using Manipulation Primitives for Object Sorting in Cluttered Environments

This paper explores the idea of manipulation-aided perception and grasping in the context of sorting small objects on a cluttered tabletop. We present a robust pipeline that combines perception and manipulation to accurately sort objects by some property (e.g., color, size, shape, etc.). The pipeline uses two motion primitives to manipulate the scene in ways that help the robot to improve its perception and grasps. This results in the ability to sort cluttered object piles accurately. We also present an implementation on the PR2 robot that applies our algorithm to sort Duplo bricks by color and size, and compare our method to brick sorting without the aid of manipulation. The experimental results demonstrate the benefits of our approach, particularly in environments with a high degree of clutter.

Human Hand Motion Analysis With Multisensory Information

In order to study and analyze human hand motions that contain multimodal information, a generalized framework integrating multiple sensors is proposed and consists of modules of sensor integration, signal preprocessing, correlation study of sensory information, and motion identification. Three types of sensors are integrated to simultaneously capture the finger angle trajectories, the hand contact forces, and the forearm electromyography (EMG) signals. To facilitate the rapid acquisition of human hand tasks, methods to automatically synchronize and segment manipulation primitives are developed in the signal preprocessing module. Correlations of the sensory information are studied by using Empirical Copula and demonstrate that there exist significant relationships between muscle signals and finger trajectories and between muscle signals and contact forces. In addition, recognizing different hand grasps and manipulations based on the EMG signals is investigated by using Fuzzy Gaussian Mixture Models (FGMMs) and results of comparative experiments show FGMMs outperform Gaussian Mixture Models and support vector machine with a higher recognition rate. The proposed framework integrating the state-of-the-art sensor technology with the developed algorithms provides researchers a versatile and adaptable platform for human hand motion analysis and has potential applications especially in robotic hand or prosthetic hand control and human-computer interaction.

Manipulation primitives: A paradigm for abstraction and execution of grasping and manipulation tasks

Sensor-based reactive and hybrid approaches have proven a promising line of study to address imperfect knowledge in grasping and manipulation. However the reactive approaches are usually tightly coupled to a particular embodiment making transfer of knowledge difficult.This paper proposes a paradigm for modeling and execution of reactive manipulation actions, which makes knowledge transfer to different embodiments possible while retaining the reactive capabilities of the embodiments. The proposed approach extends the idea of control primitives coordinated by a state machine by introducing an embodiment independent layer of abstraction. Abstract manipulation primitives constitute a vocabulary of atomic, embodiment independent actions, which can be coordinated using state machines to describe complex actions. To obtain embodiment specific models, the abstract state machines are automatically translated to embodiment specific models, such that full capabilities of each platform can be utilized.The strength of the manipulation primitives paradigm is demonstrated by developing a set of corresponding embodiment specific primitives for object transport, including a complex reactive grasping primitive. The robustness of the approach is experimentally studied in emptying of a box filled with several unknown objects. The embodiment independence is studied by performing a manipulation task on two different platforms using the same abstract description.

Finally tagless observable recursion for an abstract grammar model

AbstractWe define a finally tagless, shallow embedding of a typed grammar language. In order to avoid the limitations of traditional parser combinator libraries (no bottom-up parsing, no full grammar analysis or transformation), we require object-language recursion to be observable in the meta-language. Since existing proposals for recursive constructs are not fully satisfactory, we propose new finally tagless primitive recursive constructs to solve the problem. To do this in a well-typed way, we require considerable infrastructure, for which we reuse techniques from the multirec generic programming library. Our infrastructure allows a precise model of the complex interaction between a grammar, a parsing algorithm and a set of semantic actions. On the flip side, our approach requires the grammar author to provide a type- and value-level encoding of the grammar's domain and we can provide only a limited form of constructs like many. We demonstrate five meta-language grammar algorithms exploiting our model, including a grammar pretty-printer, a reachability analysis, a translation of quantified recursive constructs to the standard one and an implementation of the left-corner grammar transform. The work we present forms the basis of the grammar-combinators parsing library1, which is the first to work with a precise, shallow model of abstract context-free grammars in a classical (not dependently typed) functional language and which supports a wide range of grammar manipulation primitives. From a more general point of view, our work shows a solution to the well-studied problem of observable sharing in shallowly embedded domain-specific languages and specifically in finally tagless domain-specific languages.

Processing and visualizing the data in tweets

Microblogs such as Twitter provide a valuable stream of diverse user-generated data. While the data extracted from Twitter is generally timely and accurate, the process by which developers extract structured data from the tweet stream is ad-hoc and requires reimplementation of common data manipulation primitives. In this paper, we present two systems for querying and extracting structure from Twitter-embedded data. The first, TweeQL, provides a streaming SQL-like interface to the Twitter API, making common tweet processing tasks simpler. The second, TwitInfo, shows how end-users can interact with and understand aggregated data from the tweet stream, in addition to showcasing the power of the TweeQL language. Together these systems show the richness of content that can be extracted from Twitter.

An optimal probabilistic solution for information confinement, privacy, and security in RFID systems

In this paper, we provide the following contributions to enhance the security of RFID based systems. First, we assume that among multiple servers storing the information related to the tags some of them can be compromised. For this new threat scenario, we devise a technique to make RFID identification server dependent, providing a different unique secret key shared by a tag and a server. The solution proposed requires the tag to store just a single key, thus fitting the constraints on tag's memory. Second, we provide a probabilistic tag identification scheme that requires the server to perform just bitwise operations and simple list manipulation primitives, thus speeding up the identification process. The tag identification protocol assures privacy, security and resilience to DoS attacks thanks to its stateless nature. Moreover, we extend the tag identification protocol to achieve mutual authentication and resilience to replay attacks. The proposed identification protocol, unlike other probabilistic protocols, never rejects a legitimate tag. Furthermore, the identification protocol requires the reader to access the local database (DB) of tags’ keys O( n) times—where n is the number of tags in the system—while it has been shown in the literature that a privacy preserving identification protocol requires a reader to access Θ ( n ) times this DB. In this sense, our protocol is optimal. Finally, the three features suggested in this paper, namely, reader-dependent key management, tag identification, and mutual authentication, can be independently adopted to build alternative solutions.

Recognition of in-hand manipulation using contact state transition for multifingered robot hand control

This paper proposes a method for recognizing in-hand manipulation of the operator by observing a contact state transition between an object and the human hand. An instrumented object with a pressure distribution sensor and a position/orientation sensor has been developed. By processing information from the sensors, the contact regions on the operator’s palm surface are detected. A contact state transition diagram is created by taking practical contact states into account. A recognition algorithm based on Dynamic Programming (DP) is proposed to recognize the type of in-hand manipulation by comparing the similarity of the contact state transition between an input sequence and template manipulation primitives. The validity of the proposed method is confirmed by experiments.

Manipulation of Convex Objects via Two-agent Point-contact Push

This paper explores a sensorless manipulation method for orienting and translating convex objects in the plane. The manipulation task is performed by a two-agent point-contact push. During the manipulation, each agent makes a point contact with the object, and both agents push together along a straight-line. One advantage of the two-agent point-contact push over the physical fence based push is that the two-agent point-contact push can manipulate non-polygonal parts, and reduce the position and orientation uncertainties simultaneously. First, two manipulation primitives are identified, equilibrium and non-equilibrium pushes, and the motion of the object characterized under these two pushing actions. Then, a controllability analysis is conducted for this class of manipulation using the theory of positive bases. After the analysis, the planning problem is studied in the framework of a switched system, and an analytical solution to the planning problem is developed. Finally, manipulation examples and experiments are provided to demonstrate the proposed manipulation method.

Executing assembly tasks specified by manipulation primitive nets

Numerous scientific publications in the open literature show approaches for automatic assembly planning, automated robot programming, notations for the task frame formalism, robot control architectures for hybrid control methods, and respective experimental results in these areas. But there are still significant gaps between these individual fields. Considering the whole chain, from assembly planning via autonomous robot programming to the execution of complex robot tasks, the latter part of it is discussed in this paper: manipulation of primitive nets as output of task planning systems are decomposed into single manipulation primitives, which are subsequently used to generate parameters for hybrid control. A hybrid controller computes set-points for a joint position controller. Our aim is to define versatile interfaces between the mentioned disciplines, in order to close the gaps between them. Derived from the task-frame formalism, manipulation primitives constitute the base interface in this sense. After its description, the composition of manipulation primitive nets is described. Regarding the control architecture, the interpretation of manipulation primitives as atomic commands and the setting of unambiguous low level control parameters is discussed. Subsequently, the software architecture necessary to realize the complex control structure for compliant motion, is introduced. To highlight the meaning for practical implementations, several experimental results of sample assembly tasks are shown.