3D Keypoints Research Articles

VGF-Net: Visual-Geometric fusion learning for simultaneous drone navigation and height mapping

The drone navigation requires the comprehensive understanding of both visual and geometric information in the 3D world. In this paper, we present a Visual-Geometric Fusion Network (VGF-Net), a deep network for the fusion analysis of visual/geometric data and the construction of 2.5D height maps for simultaneous drone navigation in novel environments. Given an initial rough height map and a sequence of RGB images, our VGF-Net extracts the visual information of the scene, along with a sparse set of 3D keypoints that capture the geometric relationship between objects in the scene. Driven by the data, VGF-Net adaptively fuses visual and geometric information, forming a unified Visual-Geometric Representation. This representation is fed to a new Directional Attention Model (DAM), which helps enhance the visual-geometric object relationship and propagates the informative data to dynamically refine the height map and the corresponding keypoints. An entire end-to-end information fusion and mapping system is formed, demonstrating remarkable robustness and high accuracy on the autonomous drone navigation across complex indoor and large-scale outdoor scenes.

ASPset: An outdoor sports pose video dataset with 3D keypoint annotations

Recent advances in deep learning approaches to computer vision problems have led to renewed interest in the task of predicting 3D human joint locations from raw image data, with application areas including sports analysis, human-computer interaction, and physical rehabilitation. Although supervised learning of deep neural networks has proven to be effective for pose estimation, it requires a wealth of varied data to generalise well to previously unseen examples. Consequently, progress in 3D human pose estimation has been slowed by the fact that 3D keypoint annotations are notoriously difficult to obtain, traditionally requiring a large array of cameras and/or the use of wearable markers/sensors. In this paper we describe a methodology for obtaining 3D human pose annotations using only three video cameras and without any wearables. We apply this methodology to construct ASPset-510 (Australian Sports Pose Dataset), a large collection of natural sports-related video with 3D pose annotations. Using ASPset-510 as an additional source of training examples, we found that we could improve pose model generalisation on the established MPI-INF-3DHP benchmark. We make ASPset-510 publicly available, and provide strong baseline results for future work to compare against.

ASPP-DF-PVNet: Atrous Spatial Pyramid Pooling and Distance-Filtered PVNet for occlusion resistant 6D object pose estimation

Detecting objects and estimating their 6D poses from a single RGB image is quite challenging under severe occlusions. Recently, vector-field based methods have shown certain robustness to occlusion and truncation. Based on the vector-field representation, applying voting strategy to localize 2D keypoints can further reduce the influence of outliers. To improve the effectiveness of vector-field based deep network and voting scheme, we propose Atrous Spatial Pyramid Pooling and Distance-Filtered PVNet (ASPP-DF-PVNet), an occlusion resistant framework for 6D object pose estimation. ASPP-DF-PVNet utilizes the effective Atrous Spatial Pyramid Pooling (ASPP) module of Deeplabv3 to capture multi-scale features and encode global context information, which improves the accuracy of segmentation and vector-field prediction comparing to the original PVNet, especially under severe occlusions. Considering that the distances between pixels and keypoint hypotheses will affect the voting deviations, we then present a distance-filtered voting scheme which takes the voting distances into consideration to filter out the votes with large deviations. Experiments demonstrate that our method outperforms the state-of-the-art methods by a considerable margin without using pose refinement, and obtains competitive results against the methods with refinement on the LINEMOD and Occlusion LINEMOD datasets.

Pointless Pose: Part Affinity Field-Based 3D Pose Estimation without Detecting Keypoints

Human pose estimation finds its application in an extremely wide domain and is therefore never pointless. We propose in this paper a new approach that, unlike any prior one that we are aware of, bypasses the 2D keypoint detection step based on which the 3D pose is estimated, and is thus pointless. Our motivation is rather straightforward: 2D keypoint detection is vulnerable to occlusions and out-of-image absences, in which case the 2D errors propagate to 3D recovery and deteriorate the results. To this end, we resort to explicitly estimating the human body regions of interest (ROI) and their 3D orientations. Even if a portion of the human body, like the lower arm, is partially absent, the predicted orientation vector pointing from the upper arm will take advantage of the local image evidence and recover the 3D pose. This is achieved, specifically, by deforming a skeleton-shaped puppet template to fit the estimated orientation vectors. Despite its simple nature, the proposed approach yields truly robust and state-of-the-art results on several benchmarks and in-the-wild data.

Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection.

In this paper, a manipulation planning method for object re-orientation based on semantic segmentation keypoint detection is proposed for robot manipulator which is able to detect and re-orientate the randomly placed objects to a specified position and pose. There are two main parts: (1) 3D keypoint detection system; and (2) manipulation planning system for object re-orientation. In the 3D keypoint detection system, an RGB-D camera is used to obtain the information of the environment and can generate 3D keypoints of the target object as inputs to represent its corresponding position and pose. This process simplifies the 3D model representation so that the manipulation planning for object re-orientation can be executed in a category-level manner by adding various training data of the object in the training phase. In addition, 3D suction points in both the object’s current and expected poses are also generated as the inputs of the next operation stage. During the next stage, Mask Region-Convolutional Neural Network (Mask R-CNN) algorithm is used for preliminary object detection and object image. The highest confidence index image is selected as the input of the semantic segmentation system in order to classify each pixel in the picture for the corresponding pack unit of the object. In addition, after using a convolutional neural network for semantic segmentation, the Conditional Random Fields (CRFs) method is used to perform several iterations to obtain a more accurate result of object recognition. When the target object is segmented into the pack units of image process, the center position of each pack unit can be obtained. Then, a normal vector of each pack unit’s center points is generated by the depth image information and pose of the object, which can be obtained by connecting the center points of each pack unit. In the manipulation planning system for object re-orientation, the pose of the object and the normal vector of each pack unit are first converted into the working coordinate system of the robot manipulator. Then, according to the current and expected pose of the object, the spherical linear interpolation (Slerp) algorithm is used to generate a series of movements in the workspace for object re-orientation on the robot manipulator. In addition, the pose of the object is adjusted on the z-axis of the object’s geodetic coordinate system based on the image features on the surface of the object, so that the pose of the placed object can approach the desired pose. Finally, a robot manipulator and a vacuum suction cup made by the laboratory are used to verify that the proposed system can indeed complete the planned task of object re-orientation.

Head Pose Estimation through Keypoints Matching between Reconstructed 3D Face Model and 2D Image.

Mainstream methods treat head pose estimation as a supervised classification/regression problem, whose performance heavily depends on the accuracy of ground-truth labels of training data. However, it is rather difficult to obtain accurate head pose labels in practice, due to the lack of effective equipment and reasonable approaches for head pose labeling. In this paper, we propose a method which does not need to be trained with head pose labels, but matches the keypoints between a reconstructed 3D face model and the 2D input image, for head pose estimation. The proposed head pose estimation method consists of two components: the 3D face reconstruction and the 3D–2D matching keypoints. At the 3D face reconstruction phase, a personalized 3D face model is reconstructed from the input head image using convolutional neural networks, which are jointly optimized by an asymmetric Euclidean loss and a keypoint loss. At the 3D–2D keypoints matching phase, an iterative optimization algorithm is proposed to match the keypoints between the reconstructed 3D face model and the 2D input image efficiently under the constraint of perspective transformation. The proposed method is extensively evaluated on five widely used head pose estimation datasets, including Pointing’04, BIWI, AFLW2000, Multi-PIE, and Pandora. The experimental results demonstrate that the proposed method achieves excellent cross-dataset performance and surpasses most of the existing state-of-the-art approaches, with average MAEs of on Pointing’04, on BIWI, on AFLW2000, on Multi-PIE, and on Pandora, although the model of the proposed method is not trained on any of these five datasets.

Triangulate geometric constraint combined with visual-flow fusion network for accurate 6DoF pose estimation

Estimating the 6D object pose based on a monocular RGB image is a challenging task in computer vision, which produces false positives under the influence of occlusion or cluttered environments. In addition, the prediction of translation is affected by changes of the image size. In this work, we present a novel two-stage method TGCPose6D for robust 6DoF object pose estimation which is composed of 2D keypoint detection and translation refinement. In the first stage, the 2D keypoint regression space is constrained by triangulate geometric feature vectors, and the low-quality prediction is suppressed by the center-heatmap weighted loss function, thereby the performance of keypoint detection is significantly improved. In the second stage, the Visual-Flow Fusion network (VFFNet) is used to extract the visual feature and optical flow feature of the rendered image and the observed image, and to predict the relative translation based on the difference of features. Specifically, the VFFNet is trained iteratively to gain the ability to predict the relative translation deviation. Extensive experiments are conducted to demonstrate the effectiveness of the proposed TGCPose6D method. Our overall pose estimation pipeline outperforms state-of-the-art object pose estimation methods on several benchmarks.

Anatomy-Aware 3D Human Pose Estimation With Bone-Based Pose Decomposition

In this work, we propose a new solution to 3D human pose estimation in videos. Instead of directly regressing the 3D joint locations, we draw inspiration from the human skeleton anatomy and decompose the task into bone direction prediction and bone length prediction, from which the 3D joint locations can be completely derived. Our motivation is the fact that the bone lengths of a human skeleton remain consistent across time. This promotes us to develop effective techniques to utilize global information across <i>all</i> the frames in a video for high-accuracy bone length prediction. Moreover, for the bone direction prediction network, we propose a fully-convolutional propagating architecture with long skip connections. Essentially, it predicts the directions of different bones hierarchically without using any time-consuming memory units (<i>e.g.</i> LSTM). A novel joint shift loss is further introduced to bridge the training of the bone length and bone direction prediction networks. Finally, we employ an implicit attention mechanism to feed the 2D keypoint visibility scores into the model as extra guidance, which significantly mitigates the depth ambiguity in many challenging poses. Our full model outperforms the previous best results on Human3.6M and MPI-INF-3DHP datasets, where comprehensive evaluation validates the effectiveness of our model.

Detecting Object Surface Keypoints From a Single RGB Image via Deep Learning Network for 6-DoF Pose Estimation

Estimating the 6-DoF (Degree of Freedom) object pose from a single RGB image is one of the challenging tasks in the field of computer vision. Before the pose which is defined as the translation and rotation parameters can be derived by the traditional PnP algorithm, 2D image projections of a set of 3D object keypoints must be accurately detected. In this paper, we present techniques for defining 3D object surface keypoints and predicting their corresponding 2D counterparts via deep-learning network architectures. The main technique to designate 3D object keypoints is to employ quadratic fitting scheme for calculating the principal surface curvatures as the weights and then select from all surface points the ones mostly distributive with larger curvatures to describe the object shape as possible. However, the 2D projected keypoints are not directly regressed from the network, but encoded as the unit vector fields pointing to them, so that the voting scheme to recover back those 2D keypoints can be performed. Moreover, an effective loss function with the regularization term is adopted in training ResNet for predicting image projections of object keypoints by focusing on small-scale errors. Experimental results show that our proposed technique outperforms state-of-the-art approaches in both “2D projection” and “3D transformation” metrics.

RGB-D Videos-Based Early Prediction of Infant Cerebral Palsy via General Movements Complexity

Early detection and intervention of cerebral palsy can promote neural remodeling in the process of brain development, thus reducing the negative effects of cerebral palsy. In this paper, we proposed a novel method for early prediction of infant cerebral palsy based on General Movements Assessment (GMA) theory with RGB-D videos. Firstly, we explored the human pose recognition in supine position based on RGB-D videos. Then we further apply it to auto-GMA. Specifically, we employ current pose estimation method on RGB images to achieve the infant full body 2D key points. By combining the depth information, the 3D movement of infant in supine position can be obtained. Then the infant's movement complexity index is achieved by extracting the infant's whole-body movement characteristic. In order to verify the effectiveness of the method, we did some experiments on a public dataset consisting 12 real recorded infants' movement RGB-D videos, with 4 of the samples were diagnosed as abnormal infants by a GMA expert. We use expert GMA ratings of these recorded movements as the gold standard. Our method achieved state-of-the-art with sensitivity of 100%, specificity of 87.5%, and accuracy of 91.7%. The results show that the method has great potential in assisting doctors in diagnose infant cerebral palsy.

A Practical O(N2) Outlier Removal Method for Correspondence-Based Point Cloud Registration.

Point cloud registration (PCR) is an important and fundamental problem in 3D computer vision, whose goal is to seek an optimal rigid model to register a point cloud pair. Correspondence-based PCR techniques do not require initial guesses and gain more attentions. However, 3D keypoint techniques are much more difficult than their 2D counterparts, which results in extremely high outlier rates. Current robust techniques suffer from very high computational cost. In this paper, we propose a polynomial time ( O(N2), where N is the number of correspondences.) outlier removal method. Its basic idea is to reduce the input set into a smaller one with a lower outlier rate based on bound principle. To seek tight lower and upper bounds, we originally define two concepts, i.e., correspondence matrix (CM) and augmented correspondence matrix (ACM). We propose a cost function to minimize the determinant of CM or ACM, where the cost of CM rises to a tight lower bound and the cost of ACM leads to a tight upper bound. Then, we propose a scale-adaptive Cauchy estimator (SA-Cauchy) for further optimization. Extensive experiments on simulated and real PCR datasets demonstrate that the proposed method is robust at outlier rates above 99 percent and 1 ∼2 orders faster than its competitors. The source code will be made publicly available in https://ljy-rs.github.io/web/.

Real-Time and Efficient 6-D Pose Estimation From a Single RGB Image

6D pose estimation is an important branch in the field of vision measurement, and is widely used in the fields of robotics, autonomous driving and reality augmentation. The latest research trend in 6D pose estimation is to train a deep neural network to directly predict the 2D projection position of the 3D keypoint from the image, establish the corresponding relationship, and finally use the perspective-n-point (PnP) algorithm to perform pose estimation. The current challenge of pose estimation is that when objects are textureless, occluded or scene-clutterd, the detection accuracy is reduced, and most of the existing algorithm models are large and cannot accommodate real-time requirements. In this paper, we introduce a densely connected feature pyramid network (DFPN) that can efficiently integrate and utilize features. We combine the cross-stage partial network (CSPNet) with DFPN to design a new network for 6D pose estimation, DFPN-6D, a new approach for 6D object pose estimation. DFPN-6D can efficiently and accurately handle objects with textureless, occluded and scene clutter and estimate their full 6D poses in a single shot. Furthermore, we propose a new confidence calculation method and loss function for object pose estimation, which can fully consider spatial information. Finally, we propose a novel augmentation method for direct 6D pose estimation approaches to improve performance and generalization ability in the case of occlusion, which is called 6D augmentation. Our approach achieves a new state-of-the-art accuracy of 98.06 and 87.09 in terms of the ADD(-S) metric on the Linemod dataset and Occluded-Linemod dataset, and our method also achieves the best result in terms of the different metric on the MULT-I dataset, BIN-P dataset and T-LESS dataset, respectively, while still running end-to-end at over 65 FPS. The experimental results demonstrate that our algorithm is robust to textureless materials and occlusion while running more efficiently than other methods. We also deploy our proposed method to a real robot to grasp and manipulate objects based on the estimated pose.

Understanding Pixel-Level 2D Image Semantics With 3D Keypoint Knowledge Engine.

Pixel-level 2D object semantic understanding is an important topic in computer vision and could help machine deeply understand objects (e.g., functionality and affordance) in our daily life. However, most previous methods directly train on correspondences in 2D images, which is end-to-end but loses plenty of information in 3D spaces. In this paper, we propose a new method on predicting image corresponding semantics in 3D domain and then projecting them back onto 2D images to achieve pixel-level understanding. In order to obtain reliable 3D semantic labels that are absent in current image datasets, we build a large scale keypoint knowledge engine called KeypointNet, which contains 103,450 keypoints and 8,234 3D models from 16 object categories. Our method leverages the advantages in 3D vision and can explicitly reason about objects self-occlusion and visibility. We show that our method gives comparative and even superior results on standard semantic benchmarks.

Attitude Mounting Misalignment Estimation Method for the Calibration of UAV LiDAR System by using a TIN-based Corresponding Model

In traditional attitude mounting misalignment estimation methods for the calibration of unmanned autonomous vehicle (UAV) based light detection and ranging (LiDAR) system, signalized targets and iterative corresponding models are required, which makes it highly cost and computationally time-consuming. This paper presents an attitude mounting misalignment estimation (AMME) method for the calibration of UAV LiDAR system. The proposed method is divided into the coarse registration of LiDAR strips and the estimation of the attitude mounting misalignment. Firstly, 3D keypoints are extracted in the point clouds using the scale-invariant feature transform (SIFT) algorithm. Afterwards, the point feature transform (PFH) descriptor is used for 3D keypoint matching. Then, the coarse registration is executed. In the second part of the contribution, the systematic errors in the attitude mounting misalignment are estimated by incorporating the proposed triangular irregular network (TIN) corresponding model into the calibration modelling. Using the TIN-based corresponding model saves time and cost for AMME method. Furthermore, it provides two important effects: practical and computational, as no designed calibration boards, segmentation and iterative matching are needed. The performance of the proposed method is demonstrated under an UAV LiDAR data onboarded with lightweight navigation sensors. The experimental results show the efficacy of the method in comparison with a state-of-the-art method.

Attitude Mounting Misalignment Estimation Method for the Calibration of UAV LiDAR System by using a TIN-based Corresponding Model

In traditional attitude mounting misalignment estimation methods for the calibration of unmanned autonomous vehicle (UAV) based light detection and ranging (LiDAR) system, signalized targets and iterative corresponding models are required, which makes it highly cost and computationally time-consuming. This paper presents an attitude mounting misalignment estimation (AMME) method for the calibration of UAV LiDAR system. The proposed method is divided into the coarse registration of LiDAR strips and the estimation of the attitude mounting misalignment. Firstly, 3D keypoints are extracted in the point clouds using the scale-invariant feature transform (SIFT) algorithm. Afterwards, the point feature transform (PFH) descriptor is used for 3D keypoint matching. Then, the coarse registration is executed. In the second part of the contribution, the systematic errors in the attitude mounting misalignment are estimated by incorporating the proposed triangular irregular network (TIN) corresponding model into the calibration modelling. Using the TIN-based corresponding model saves time and cost for AMME method. Furthermore, it provides two important effects: practical and computational, as no designed calibration boards, segmentation and iterative matching are needed. The performance of the proposed method is demonstrated under an UAV LiDAR data onboarded with lightweight navigation sensors. The experimental results show the efficacy of the method in comparison with a state-of-the-art method.

RGB2Hands

Tracking and reconstructing the 3D pose and geometry of two hands in interaction is a challenging problem that has a high relevance for several human-computer interaction applications, including AR/VR, robotics, or sign language recognition. Existing works are either limited to simpler tracking settings ( e.g. , considering only a single hand or two spatially separated hands), or rely on less ubiquitous sensors, such as depth cameras. In contrast, in this work we present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera that explicitly considers close interactions. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN that regresses multiple complementary pieces of information, including segmentation, dense matchings to a 3D hand model, and 2D keypoint positions, together with newly proposed intra-hand relative depth and inter-hand distance maps. These predictions are subsequently used in a generative model fitting framework in order to estimate pose and shape parameters of a 3D hand model for both hands. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline through an extensive ablation study. Moreover, we demonstrate that our approach offers previously unseen two-hand tracking performance from RGB, and quantitatively and qualitatively outperforms existing RGB-based methods that were not explicitly designed for two-hand interactions. Moreover, our method even performs on-par with depth-based real-time methods.

New multi‐view human motion capture framework

Estimating human pose and shape without markers is a challenging problem. This study proposes a multiple-view markerless human motion capture framework. Firstly, a multi-view camera system is built for capturing real-time images of moving humans on multiple views. Secondly, by employing the OpenPose method, the authors calculate robust 3D key points from 2D key points of the human body, which are estimated from the multi-view images. And dense 3D point cloud is reconstructed from images. Thirdly, they propose a novel SMPL-based method to represent human motion by fitting the SMPL model to 3D key points and 3D point clouds. In order to achieve a more accurate human pose, a penalty term is utilised to solve the problem of error accumulation in the process of human motion capture. In addition, they present a dense mesh template-based SMPL that can be deformed to point cloud to recover a real human body shape. Finally, they map multi-view colour images onto the human mesh model to acquire rendered mesh. The experimental results show that the proposed method improves the accuracy of human pose and realises the 3D human body model more realistic.

Learning Semantic Keypoint Representations for Door Opening Manipulation

We consider a scenario where a robot is capable of autonomously opening previously unseen doors. Prior works either use model-based methods that rely strongly on accurate kinematic models, or learn a policy from scratch through trial-and-error, which cannot generalize to large variations in shape, and location of doors. In this letter, we propose a novel method for opening unseen doors with no prior knowledge of door model, which leverages semantic 3D keypoints as door handle representations to generate the end-effector trajectory from a motion planner. The keypoint representations are predicted from raw visual input by a deep neural network, which can provide a concise, and semantic description of the handle to determine the grasp pose, and subsequent motion planning. In contrast to existing works that require known object models or significant manual effort on data collection, we present a data augmentation technique to automatically generate large amounts of realistic-looking synthetic data with almost no human labeling effort. An augmented dataset, consisting of large amounts of synthetic data, and small amounts of real data, is used to train the network. Qualitative results show that our proposed method outperforms the state-of-the-art pose-based methods on real test sets in terms of perception metrics. Hardware experiments demonstrate that our proposed method can achieve 94.2% success rate on opening 6 previously unseen doors with significant shape variations under different environments, and conditions.

A multi-task Faster R-CNN method for 3D vehicle detection based on a single image

Vehicle detection is an important part of robot environmental perception. In this paper, a 3D vehicle detection method using a single image is proposed to generate the 3D space coordinate information of the object using monocular vision for autonomous driving. The proposed method works under the multi-task framework and integrates 2D object detection, 3D object detection, orientation estimation and key point detection into one unified deep convolution neural network (DCNN) which could be trained by end-to-end learning. Besides, our proposed method is built by modifying Fast R-CNN using multi-task learning, and thus our proposed method is named multi-task Faster R-CNN (MT-Faster R-CNN). The experiments on KITTI dataset are conducted to evaluate our proposed method and the other 3D vehicle detection methods. The experimental results demonstrate that our proposed method is competitive and could significantly assist autonomous driving.

Human 3D pose estimation in a lying position by RGB-D images for medical diagnosis and rehabilitation.

Posture recognition in the human lying position is of great significance for the rehabilitation evaluation of lying patients and the diagnosis of infants with early cerebral palsy. In this paper, we proposed a novel method for human 3D pose estimation in a lying position with the RGB image and corresponding depth information. Firstly, we employ current pose estimation method on RGB images to achieve the human full body 2D keypoints. By combining the depth information and coordinate transformation, the 3D movement of human in lying position can be obtained. We validate our method with two public datasets. The results show that the accuracy can reach the state-of-the-art.