3D Deformable Objects Research Articles

AGAR - Attention Graph-RNN for Adaptative Motion Prediction of Point Clouds of Deformable Objects

This article focuses on motion prediction for point cloud sequences in the challenging case of deformable 3D objects, such as human body motion. First, we investigate the challenges caused by deformable shapes and complex motions present in this type of representation, with the ultimate goal of understanding the technical limitations of state-of-the-art models. From this understanding, we propose an improved architecture for point cloud prediction of deformable 3D objects. Specifically, to handle deformable shapes, we propose a graph-based approach that learns and exploits the spatial structure of point clouds to extract more representative features. Then, we propose a module able to combine the learned features in a adaptative manner according to the point cloud movements. The proposed adaptative module controls the composition of local and global motions for each point, enabling the network to model complex motions in deformable 3D objects more effectively. We tested the proposed method on the following datasets: MNIST moving digits, the Mixamo human bodies motions [ 15 ], JPEG [ 5 ] and CWIPC-SXR [ 32 ] real-world dynamic bodies. Simulation results demonstrate that our method outperforms the current baseline methods given its improved ability to model complex movements as well as preserve point cloud shape. Furthermore, we demonstrate the generalizability of the proposed framework for dynamic feature learning by testing the framework for action recognition on the MSRAction3D dataset [ 19 ] and achieving results on par with state-of-the-art methods.

DOVE: Learning Deformable 3D Objects by Watching Videos

Learning deformable 3D objects from 2D images is often an ill-posed problem. Existing methods rely on explicit supervision to establish multi-view correspondences, such as template shape models and keypoint annotations, which restricts their applicability on objects “in the wild”. A more natural way of establishing correspondences is by watching videos of objects moving around. In this paper, we present DOVE, a method that learns textured 3D models of deformable object categories from monocular videos available online, without keypoint, viewpoint or template shape supervision. By resolving symmetry-induced pose ambiguities and leveraging temporal correspondences in videos, the model automatically learns to factor out 3D shape, articulated pose and texture from each individual RGB frame, and is ready for single-image inference at test time. In the experiments, we show that existing methods fail to learn sensible 3D shapes without additional keypoint or template supervision, whereas our method produces temporally consistent 3D models, which can be animated and rendered from arbitrary viewpoints. Project page: https://dove3d.github.io/.

HMDO : Markerless multi-view hand manipulation capture with deformable objects

We construct the first markerless deformable interaction dataset recording interactive motions of the hands and deformable objects, called HMDO (Hand Manipulation with Deformable Objects). With our built multi-view capture system, it captures the deformable interactions with multiple perspectives, various object shapes, and diverse interactive forms. Our motivation is the current lack of hand and deformable object interaction datasets, as 3D hand and deformable object reconstruction is challenging. Mainly due to mutual occlusion, the interaction area is difficult to observe, the visual features between the hand and the object are entangled, and the reconstruction of the interaction area deformation is difficult. To tackle this challenge, we propose a method to annotate our captured data. Our key idea is to collaborate with estimated hand features to guide the object global pose estimation, and then optimize the deformation process of the object by analyzing the relationship between the hand and the object. Through comprehensive evaluation, the proposed method can reconstruct interactive motions of hands and deformable objects with high quality. HMDO currently consists of 21600 frames over 12 sequences. In the future, this dataset could boost the research of learning-based reconstruction of deformable interaction scenes.

Real-Time Surface-Based Volume Constraints on Mass-Spring Model in Unity3D

This paper describes a parallel method to simulate real-time 3D deformable objects using volume preservation constraints on a mass-spring model (MSM) to achieve plausible results in real-time performance. Instead of considering a volumetric mesh which is mostly used to simulate deformable objects, we purely take the surface of the 3D object into account to reduce the time complexity and obtain a high-quality deformable. In the conventional MSM, we can simply control the shape of the deformable object through the stiffness and damping coefficients which is beneficial for our volume constraint. We use the divergence theorem and implicit constraint enforcement scheme to maintain the volume of the object and deform it freely. The surface-based volume constraint is applied to correct the force of the spring network. The proposed algorithm was designed on compute shader in Unity3D and runs outside the normal rendering pipeline in the graphics processing unit (GPU) in order to utilize the massive parallel process to accelerate the performance of the simulation. The performance of the simulation can be accelerated by using the parallel processing method on the GPU with an average speedup factor of 4.27 using the conventional mass-spring method, and an average speedup factor of 2.54 for the volume preservation constraint method. We present several scenes which demonstrate the volume-preserving deformations using the conventional mass-spring method and volume-preservation constraint method and the volume loss of deformable objects for all 3D models is compared. The volume preservation method obtains a volume loss significantly lower than the conventional MSM for all experimental tests.

DefGraspSim: Physics-Based Simulation of Grasp Outcomes for 3D Deformable Objects

Robotic grasping of 3D deformable objects (e.g., fruits/vegetables, internal organs, bottles/boxes) is critical for real-world applications such as food processing, robotic surgery, and household automation. However, developing grasp strategies for such objects is uniquely challenging. Unlike rigid objects, deformable objects have infinite degrees of freedom and require field quantities (e.g., deformation, stress) to fully define their state. As these quantities are not easily accessible in the real world, we propose studying interaction with deformable objects through physics-based simulation. As such, we simulate grasps on a wide range of 3D deformable objects using a GPU-based implementation of the corotational finite element method (FEM). To facilitate future research, we open-source our simulated dataset (34 objects, 1e5 Pa elasticity range, 6800 grasp evaluations, 1.1M grasp measurements), as well as a code repository that allows researchers to run our full FEM-based grasp evaluation pipeline on arbitrary 3D object models of their choice. Finally, we demonstrate good correspondence between grasp outcomes on simulated objects and their real counterparts.

Volumetric Object Modeling Using Internal Shape Preserving Constraint in Unity 3D

In real-time contents, such as games and interactive simulators, it is very important to reduce the amount of simulation computation of 3D deformable objects. Although position-based dynamics has been proposed to reduce the amount of computation, the number of nodes for the tetrahedral model to represent a volumetric deformable object has to be increased, which makes the real-time simulation difficult. Therefore, this paper proposes an Internal shape preserving constraint(ISPC) generation algorithm integrated into the position-based dynamics to represent the physical properties of the 3D volumetric deformable object, while reducing the number of nodes filling the interior of the object. The proposed algorithm not only provides motion behavior similar to the tetrahedral model by using a surface model, but also enables real-time simulation by reducing the number of nodes constituting the 3D virtual object. It showed high FPS with reduced computation time compared to the tetrahedral model, and the volume maintenance and physical properties of model were expressed similarly to the tetrahedral model.

Real-time Volume Preserving Constraints for Volumetric Model on GPU

This paper presents a parallel method for simulating real-time 3D deformable objects using the volume preservation mass-spring system method on tetrahedron meshes. In general, the conventional mass-spring system is manipulated as a force-driven method because it is fast, simple to implement, and the parameters can be controlled. However, the springs in traditional mass-spring system can be excessively elongated which cause severe stability and robustness issues that lead to shape restoring, simulation blow-up, and huge volume loss of the deformable object. In addition, traditional method that uses a serial process of the central processing unit (CPU) to solve the system in every frame cannot handle the complex structure of deformable object in real-time. Therefore, the first order implicit constraint enforcement for a mass-spring model is utilized to achieve accurate visual realism of deformable objects with tough constraint error. In this paper, we applied the distance constraint and volume conservation constraints for each tetrahedron element to improve the stability of deformable object simulation using the mass-spring system and behave the same as its real-world counterparts. To reduce the computational complexity while ensuring stable simulation, we applied a method that utilizes OpenGL compute shader, a part of OpenGL Shading Language (GLSL) that executes on the graphic processing unit (GPU) to solve the numerical problems effectively. We applied the proposed methods to experimental volumetric models, and volume percentages of all objects are compared. The average volume percentages of all models during the simulation using the mass-spring system, distance constraint, and the volume constraint method were 68.21%, 89.64%, and 98.70%, respectively. The proposed approaches are successfully applied to improve the stability of mass-spring system and the performance comparison from our experimental tests also shows that the GPU-based method is faster than CPU-based implementation for all cases.

Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (Invited Paper).

We propose a method to learn 3D deformable object categories from raw single-view images, without external supervision. The method is based on an autoencoder that factors each input image into depth, albedo, viewpoint and illumination. In order to disentangle these components without supervision, we use the fact that many object categories have, at least approximately, a symmetric structure. We show that reasoning about illumination allows us to exploit the underlying object symmetry even if the appearance is not symmetric due to shading. Furthermore, we model objects that are probably, but not certainly, symmetric by predicting a symmetry probability map, learned end-to-end with the other components of the model. Our experiments show that this method can recover very accurately the 3D shape of human faces, cat faces and cars from single-view images, without any supervision or a prior shape model. On benchmarks, we demonstrate superior accuracy compared to another method that uses supervision at the level of 2D image correspondences.

Grasp Planning Pipeline for Robust Manipulation of 3D Deformable Objects with Industrial Robotic Hand + Arm Systems

In the grasping and manipulation of 3D deformable objects by robotic hands, the physical contact constraints between the fingers and the object have to be considered in order to validate the robustness of the task. Nevertheless, previous works rarely establish contact interaction models based on these constraints that enable the precise control of forces and deformations during the grasping process. This paper considers all steps of the grasping process of deformable objects in order to implement a complete grasp planning pipeline by computing the initial contact points (pregrasp strategy), and later, the contact forces and local deformations of the contact regions while the fingers close over the grasped object (grasp strategy). The deformable object behavior is modeled using a nonlinear isotropic mass-spring system, which is able to produce potential deformation. By combining both models (the contact interaction and the object deformation) in a simulation process, a new grasp planning method is proposed in order to guarantee the stability of the 3D grasped deformable object. Experimental grasping experiments of several 3D deformable objects with a Barrett hand (3-fingered) and a 6-DOF industrial robotic arm are executed. Not only will the final stable grasp configuration of the hand + object system be obtained, but an arm + hand approaching strategy (pregrasp) will also be computed.

Depth Estimation of a Deformable Object via a Monocular Camera

The depth estimation of the 3D deformable object has become increasingly crucial to various intelligent applications. In this paper, we propose a feature-based approach for accurate depth estimation of a deformable 3D object with a single camera, which reduces the problem of depth estimation to a pose estimation problem. The proposed method needs to reconstruct the target object at the very beginning. With the 3D reconstruction as an a priori model, only one monocular image is required afterwards to estimate the target object’s depth accurately, regardless of pose changes or deformability of the object. Experiments are taken on an NAO robot and a human to evaluate the depth estimation accuracy by the proposed method.

New graph distance for deformable 3D objects recognition based on triangle-stars decomposition

We address the problem of comparing deformable 3D objects represented by graphs such as triangular tessellations. We propose a new graph matching technique to measure the distance between these graphs. The proposed approach is based on a new decomposition of triangular tessellations into triangle-stars. The algorithm ensures a minimum number of disjoint triangle-stars, provides improved dissimilarity by covering larger neighbors and allows the creation of descriptors that are invariant or at least oblivious under the most common deformations. The present approach is based on an approximation of the Graph Edit Distance, which is fault-tolerant to noise and distortion, thus making our technique particularly suitable for the comparison of deformable objects. Classification is performed with supervised machine learning techniques. Our approach defines a metric space using graph embedding and graph kernel techniques. It is proved that the proposed distance is a pseudo-metric. Its time complexity is determined and the method is evaluated against benchmark databases. Our experimental results confirm the performances and the accuracy of our system.

Elastic Alignment of Triangular Surface Meshes

A novel region-based approach is proposed to find a thin plate spline map between a pair of deformable 3D objects represented by triangular surface meshes. The proposed method works without landmark extraction and feature correspondences. The aligning transformation is simply found by solving a system of integral equations. Each equation is generated by integrating a non-linear function over the object domains. We derive recursive formulas for the efficient computation of these integrals for open and closed surface meshes. Based on a series of comparative tests on a large synthetic dataset, our triangular mesh-based algorithm outperforms state of the art methods both in terms of computing time and accuracy. The applicability of the proposed approach has been demonstrated on the registration of 3D lung CT volumes, brain surfaces and 3D human faces.

Robust shape estimation for 3D deformable object manipulation

Existing shape estimation methods for deformable object manipulation suffer from the drawbacks of being off-line, model dependent, noise-sensitive or occlusion-sensitive, and thus are not appropriate for manipulation tasks requiring high precision. In this paper, we present a real-time shape estimation approach for autonomous robotic manipulation of 3D deformable objects. Our method fulfills all the requirements necessary for the high-quality deformable object manipulation in terms of being real-time, model-free and robust to noise and occlusion. These advantages are accomplished using a joint tracking and reconstruction framework, in which we track the object deformation by aligning a reference shape model with the stream input from the RGB-D camera, and simultaneously upgrade the reference shape model according to the newly captured RGB-D data. We have evaluated the quality and robustness of our real-time shape estimation pipeline on a set of deformable manipulation tasks implemented on physical robots. Videos are available at https://lifeisfantastic.github.io/DeformShapeEst/

A 3D deformable model-based framework for the retrieval of near-isometric flattenable objects using Bag-of-Visual-Words

We introduce a 3D deformable model-based framework for the retrieval of near-isometric flattenable objects using keypoints and BoVW (Bag-of-Visual-Words). By 3D deformable model we mean a texturemapped 3D shape which may deform isometrically. We assume that such a model is available for each object in the database. We exploit the 3D deformable models at the training and the retrieval phases. For our first contribution, we exploit the possibility of generating synthetic data from the 3D deformable models to define a new BoVW model for the database object representation. Our model chooses an optimal per-object representation by maximizing each object’s mean average precision. The maximization is done over multiple candidate representations which are generated using the criteria of keypoint repeatability, weight discriminance and stability. Our second contribution is the use of SfT (Shape-from-Template) to facilitate geometric verification at the retrieval phase, for a few objects hypothesized using the new BoVW model. Existing methods use a rigid model, such as the fundamental matrix, or a simple deformable model based on semi-local constraints. SfT however is a physics-based method which uses an object’s 3D deformable model to reconstruct its isometric 3D deformation from a single input image. The output of SfT thus directly provides a geometric verification score. A byproduct of our work is to extend the scope of SfT. The proposed object retrieval framework is used to provide SfT with a few object hypotheses which may be quickly tested for the 3D deformable object selection. Performance evaluation on synthetic and real images reveals the benefits of our retrieval framework using a database with size varying between 20 and 1000 objects. The use of the new BoVW model and SfT versus the BoVW baseline and a rigid model improves the retrieval performance by 4.2% and 11.3% with p-values of 5×10−6 and 7×10−30 respectively.

Model-based strategy for grasping [formula omitted] deformable objects using a multi-fingered robotic hand

This paper presents a model-based strategy for 3D deformable object grasping using a multi-fingered robotic hand. The developed contact model is based on two force components (normal force and tangential friction force, including slipping and sticking effects) and uses a non-linear mass–spring system to describe the object deformations due the mechanical load applied by the fingers of the robotic hand. The object–finger interaction is simulated in order to compute the required contact forces and deformations to robustly grasp objects with large deformations. Our approach is able to achieve this by using a non-linear model that outperforms current techniques that are limited to using linear models. After the contact forces computed by the simulation of the contact model guarantee the equilibrium of the grasp, they will be used as set-points for force-controlling the closing of the real fingers, and thus, the proposed grasping strategy is implemented. Two different objects (cube and sphere) made from two soft materials (foam and rubber) are tested in order to verify that the proposed model can represent their non-linear deformations and that the proposed grasp strategy can implement a robust grasp of them with a multi-fingered robotic hand equipped with tactile sensors. Thereby, both the grasping strategy and the proposed contact model are validated experimentally.

Acquisition and Neural Network Prediction of 3D Deformable Object Shape Using a Kinect and a Force-Torque Sensor.

The realistic representation of deformations is still an active area of research, especially for deformable objects whose behavior cannot be simply described in terms of elasticity parameters. This paper proposes a data-driven neural-network-based approach for capturing implicitly and predicting the deformations of an object subject to external forces. Visual data, in the form of 3D point clouds gathered by a Kinect sensor, is collected over an object while forces are exerted by means of the probing tip of a force-torque sensor. A novel approach based on neural gas fitting is proposed to describe the particularities of a deformation over the selectively simplified 3D surface of the object, without requiring knowledge of the object material. An alignment procedure, a distance-based clustering, and inspiration from stratified sampling support this process. The resulting representation is denser in the region of the deformation (an average of 96.6% perceptual similarity with the collected data in the deformed area), while still preserving the object’s overall shape (86% similarity over the entire surface) and only using on average of 40% of the number of vertices in the mesh. A series of feedforward neural networks is then trained to predict the mapping between the force parameters characterizing the interaction with the object and the change in the object shape, as captured by the fitted neural gas nodes. This series of networks allows for the prediction of the deformation of an object when subject to unknown interactions.

Robust deformable shape reconstruction from monocular video with manifold forests

Existing approaches to recover structure of 3D deformable objects and camera motion parameters from an uncalibrated images assume the object’s shape could be modelled well by a linear subspace. These methods have been proven effective and well suited when the deformations are relatively small, but fail to reconstruct the objects with relatively large deformations. This paper describes a novel approach for 3D non-rigid shape reconstruction, based on manifold decision forest technique. The use of this technique can be justified by noting that a specific type of shape variations might be governed by only a small number of parameters, and therefore can be well represented in a low-dimensional manifold. The key contributions of this work are the use of random decision forests for the shape manifold learning and robust metric for calculation of the re-projection error. The learned manifold defines constraints imposed on the reconstructed shapes. Due to a nonlinear structure of the learned manifold, this approach is more suitable to deal with large and complex object deformations when compared to the linear constraints. The robust metric is applied to reduce the effect of measurement outliers on the quality of the reconstruction. In many practical applications outliers cannot be completely removed and therefore the use of robust techniques is of particular practical interest. The proposed method is validated on 2D points sequences projected from the 3D motion capture data for ground truth comparison and also on real 2D video sequences. Experiments show that the newly proposed method provides better performance compared to previously proposed ones, including the robustness with respect to measurement noise, missing measurements and outliers present in the data.

Stable haptic interaction based on adaptive hierarchical shape matching

In this paper, we present a framework allowing users to interact with geometrically complex 3D deformable objects using (multiple) haptic devices based on an extended shape matching approach. There are two major challenges for haptic-enabled interaction using the shape matching method. The first is how to obtain a rapid deformation propagation when a large number of shape matching clusters exist. The second is how to robustly handle the collision response when the haptic interaction point hits the particle-sampled deformable volume. Our framework extends existing multi-resolution shape matching methods, providing an improved energy convergence rate. This is achieved by using adaptive integration strategies to avoid insignificant shape matching iterations during the simulation. Furthermore, we present a new mechanism called stable constraint particle coupling which ensures consistent deformable behavior during haptic interaction. As demonstrated in our experimental results, the proposed method provides natural and smooth haptic rendering as well as efficient yet stable deformable simulation of complex models in real time.

Picking up a soft 3D object by “feeling” the grip

This paper describes a strategy for a robotic hand to pick up deformable 3D objects on a table. Inspired by human hand behavior, the robotic hand employs two rigid fingers to first squeeze such an object until it “feels” the object to be liftable. Such “feeling” is provided by a (virtual) liftability test that is repeatedly conducted during the squeeze. Passing of the test then triggers a lifting action. Throughout the manipulation the object’s deformation and its state of contact with the fingers and the table are being tracked based on contact events. Deformable modeling uses the finite element method (FEM) while slip computation employs the homotopy continuation method to determine the contact displacements induced by finger movements. The experiment was conducted for everyday items ranging from vegetables to a toy. A simulation-based comparison between deformable grasping and rigid body grasping reveals why soft objects are easier to pick up than hard ones, and demonstrates how a rigid body grasping strategy may fail on soft objects in certain situations.

A fast modal space transform for robust nonrigid shape retrieval

Nonrigid or deformable 3D objects are common in many application domains. Retrieval of such objects in large databases based on shape similarity is still a challenging problem. In this paper, we take advantages of functional operators as characterizations of shape deformation, and further propose a framework to design novel shape signatures for encoding nonrigid geometries. Our approach constructs a context-aware integral kernel operator on a manifold, then applies modal analysis to map this operator into a low-frequency functional representation, called fast functional transform, and finally computes its spectrum as the shape signature. In a nutshell, our method is fast, isometry-invariant, discriminative, smooth and numerically stable with respect to multiple types of perturbations. Experimental results demonstrate that our new shape signature for nonrigid objects can outperform all methods participating in the nonrigid track of the SHREC'11 contest. It is also the second best performing method in the real human model track of SHREC'14.