Semantic Keypoints Research Articles

High-speed aerial grasping using a soft drone with onboard perception

Contrary to the stunning feats observed in birds of prey, aerial manipulation and grasping with flying robots still lack versatility and agility. Conventional approaches using rigid manipulators require precise positioning and are subject to large reaction forces at grasp, which limit performance at high speeds. The few reported examples of high-speed aerial grasping rely on motion capture systems, or fail to generalize across environments and grasp targets. We describe the first example of a soft aerial manipulator equipped with a fully onboard perception pipeline, capable of robustly localizing and grasping visually and morphologically varied objects. The proposed system features a novel passively closed tendon-actuated soft gripper that enables fast closure at grasp, while compensating for position errors, complying to the target-object morphology, and dampening reaction forces. The system includes an onboard perception pipeline that combines a neural-network-based semantic keypoint detector, a state-of-the-art robust 3D object pose estimator, and a fixed-lag smoother to estimate the pose of known objects. The resulting pose estimate is passed to a minimum-snap trajectory planner, tracked by an adaptive controller that fully compensates for the added mass of the grasped object. Finally, a finite-element-based controller determines optimal gripper configurations for grasping. Experiments on three different targets confirm that our approach enables dynamic, high-speed, and versatile grasping, all of which are necessary capabilities for tasks such as rapid package delivery or emergency relief. We demonstrate fully onboard vision-based grasps of a variety of objects, in both indoor and outdoor environments, and up to speeds of 2.0 m/s—the fastest vision-based grasp reported in the literature. Finally, we take a major step in expanding the utility of our platform beyond stationary targets, by demonstrating motion-capture-based grasps of targets moving up to 0.3 m/s, with relative speeds up to 1.5 m/s.

PointNu-Net: Keypoint-Assisted Convolutional Neural Network for Simultaneous Multi-Tissue Histology Nuclei Segmentation and Classification

Automatic nuclei segmentation and classification play a vital role in digital pathology. However, previous works are mostly built on data with limited diversity and small sizes, making the results questionable or misleading in actual downstream tasks. In this article, we aim to build a reliable and robust method capable of dealing with data from the ‘the clinical wild’. Specifically, we study and design a new method to simultaneously detect, segment, and classify nuclei from Haematoxylin and Eosin (H&E) stained histopathology data, and evaluate our approach using the recent largest dataset: PanNuke. We address the detection and classification of each nuclei as a novel semantic keypoint estimation problem to determine the center point of each nuclei. Next, the corresponding class-agnostic masks for nuclei center points are obtained using dynamic instance segmentation. Meanwhile, we proposed a novel Joint Pyramid Fusion Module (JPFM) to model the cross-scale dependencies, thus enhancing the local feature for better nuclei detection and classification. By decoupling two simultaneous challenging tasks and taking advantage of JPFM, our method can benefit from class-aware detection and class-agnostic segmentation, thus leading to a significant performance boost. We demonstrate the superior performance of our proposed approach for nuclei segmentation and classification across 19 different tissue types, delivering new benchmark results.

SmokePose: End-to-End Smoke Keypoint Detection

Smoke detection has been a research focus due to its application value in fire and toxic gas leakage alarms. Here we formulate a novel research paradigm for in-depth smoke analysis: modeling the smoke plume in an image with two semantic keypoints, i.e., the start-point (where the smoke comes from) and the end-point (where the smoke spreads), and localizing the keypoints through a heatmap-based detection method. A specialized dataset is developed for smoke keypoint detection, collecting images online and manually annotating the keypoints. Based on the dataset, we propose a Transformer-based model called SmokePose that employs a hierarchical Transformer encoder and a pure Transformer decoder to detect smoke keypoints in an end-to-end manner. We demonstrated the performance of the proposed SmokePose with comparative experiments and ablation studies. A further discussion on the visualization of the attention maps helps to understand the mechanism of SmokePose and to reveal essential image clues for smoke keypoint detection.

Optimal and Robust Category-Level Perception: Object Pose and Shape Estimation From 2-D and 3-D Semantic Keypoints

Optimal and Robust Category-Level Perception: Object Pose and Shape Estimation From 2-D and 3-D Semantic Keypoints

Real-Time Localization for Closed-Loop Control of Assistive Furniture

For people with limited mobility, navigating in cluttered indoor environment is challenging. In this work, we propose a mobile assistive furniture suite that is designed to ease the life of people with special needs in indoor movement. To enable intelligent coordination of this system, a key component is the localization of each mobile furniture. The challenge is to assess the state of an arbitrary living scenario so that the estimation can be used as a real-time feedback signal for autonomous closed-loop control of mobile furniture. We propose a perception pipeline that addresses these challenges. A machine learning model is designed and trained to jointly achieve multi-object semantic keypoint detection and classification in camera images. The synthetic data generation is employed to augment the training set and boost the model performance. A robust point cloud registration uses the detected semantic keypoints and depth information to estimate poses of the furniture. Tracking is applied to achieve smooth estimation. A high-performance accelerator that optimizes the efficiency of using heterogeneous devices is applied to achieve real-time performance. This visual perception pipeline is used in closed-loop control to steer the mobile furniture from initial to a desired location demonstrated in experiments on real hardware.

Universe Points Representation Learning for Partial Multi-Graph Matching

Many challenges from natural world can be formulated as a graph matching problem. Previous deep learning-based methods mainly consider a full two-graph matching setting. In this work, we study the more general partial matching problem with multi-graph cycle consistency guarantees. Building on a recent progress in deep learning on graphs, we propose a novel data-driven method (URL) for partial multi-graph matching, which uses an object-to-universe formulation and learns latent representations of abstract universe points. The proposed approach advances the state of the art in semantic keypoint matching problem, evaluated on Pascal VOC, CUB, and Willow datasets. Moreover, the set of controlled experiments on a synthetic graph matching dataset demonstrates the scalability of our method to graphs with large number of nodes and its robustness to high partiality.

Fully automatic tracking of native glenohumeral kinematics from stereo-radiography

The current work introduces a system for fully automatic tracking of native glenohumeral kinematics in stereo-radiography sequences. The proposed method first applies convolutional neural networks to obtain segmentation and semantic key point predictions in biplanar radiograph frames. Preliminary bone pose estimates are computed by solving a non-convex optimization problem with semidefinite relaxations to register digitized bone landmarks to semantic key points. Initial poses are then refined by registering computed tomography-based digitally reconstructed radiographs to captured scenes, which are masked by segmentation maps to isolate the shoulder joint. A particular neural net architecture which exploits subject-specific geometry is also introduced to improve segmentation predictions and increase robustness of subsequent pose estimates. The method is evaluated by comparing predicted glenohumeral kinematics to manually tracked values from 17 trials capturing 4 dynamic activities. Median orientation differences between predicted and ground truth poses were 1.7∘ and 8.6∘ for the scapula and humerus, respectively. Joint-level kinematics differences were less than 2∘ in 65%, 13%, and 63% of frames for XYZ orientation DoFs based on Euler angle decompositions. Automation of kinematic tracking can increase scalability of tracking workflows in research, clinical, or surgical applications.

Detection of the Bolt Loosening Angle through Semantic Key Point Extraction Detection by Using an Hourglass Network

Damage in bolts, which are used as connecting fasteners in steel structures, affects structural safety. Sophisticated machine vision methods have been formulated for the detection of loose bolts, but their accuracy remains an area for improvement. In this paper, a method based on a stacked hourglass network is proposed for automatically extracting the key points of a bolt and for obtaining the bolt loosening angle by comparing the rotations of the key points before and after the bolt is damaged. A data set containing 100 images of key bolt loosening points was collected, and rotation was performed as data augmentation to yield 1800 images. Moreover, a method was designed for automatically annotating the augmented image data set. In this study, 70%, 10%, and 20% of the annotated image data set were used for training, validation, and testing, respectively. Subsequently, a neural network model based on a stacked hourglass network was established to train the annotated image data set. The detection results were evaluated in terms of normalized errors (NEs), percentage of correct key points (PCK), detection speed, and training time. In testing, the proposed method accurately and efficiently identified the bolt loosening angle, with a PCK value as high as 99.3%. The accuracy of the proposed method was also highly robust to different shooting distances, viewing angles, and illumination levels.

Deep learning for 6D pose estimation of objects — A case study for autonomous driving

Nowadays, the potential benefits and implementation of autonomous driving have attracted widespread attention from both industry and academia. This study will solve view-invariant object detection and semantic key-point pose assumptions from a single RGB image. A machine learning method for estimating the absolute pose of an on-road vehicle for autonomous driving from monocular vision alone without the help of additional sensors is a complex task. The main purpose of this work is to identify other vehicles on the road and estimate their exact angular position from a single image with improved accuracy. The focus of the study is to create a new algorithm by applying a potentially deep convoluted neural network followed by a repetitive neural structure for more accurate 6D pose inference. A 6D pose hypothesis is presented in this study, based on a deep hybrid architecture for individual vehicles of an end-to-end approach to a task consisting of a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN). In this work, we will use a large-scale dataset consistent with the understanding of a 3D car instance called ApolloCar3D. The data set contains 5,277 real-life street scenes with examples of about 60K cars. By comparison, the ApolloCar3D is twenty times larger than the PASCAL3D+ and KITTI datasets. Ultimately, the idea is to efficiently eliminate motionless cars and predict the next pose given in the speed context, allowing a comprehensive evaluation, and passing the output through LSTM (long short-term memory) with an additional filter layer. The new filter added to the LSTM will efficiently filter and isolate stationary or parking vehicles and focus on on-road vehicles. Since the LSTM has a non-linear high-dimensional hidden memory state, it can preserve the past continuity of each generation’s data history and pay more attention to those road vehicles rather than parked or stationary vehicles to act accordingly. So for each new vehicle, the pose estimator classifier can use LSTM memory and compare the historical pose with the newly filtered data. The successful implementation of this innovative concept will lead to significant improvements in the real-life traffic situation in the field of computer vision and autonomous driving.

Learning Semantic Keypoints for Object Detection in Aerial Images

Object detection in aerial images has achieved remarkable progress with the advent of deep convolutional neural networks (CNNs). It is, however, still a challenging task since the objects in aerial images are arbitrarily oriented and often densely packed. In this letter, we propose a novel method for oriented object detection in aerial images that represents objects as rotation equivariant semantic keypoints. Unlike conventional methods that represent object rotation according to angles from each axis in the Cartesian coordinate system, we represent object using a canonical orientation to ensure rotation equivariance. We accomplish this by representing an object as semantic keypoints, where each keypoint of the object consistently corresponds to the semantic part, regardless of rotation variation. To this end, we define the “head” point of the object as the canonical orientation and the remaining bounding box vectors as semantic keypoints in clockwise order. To discriminate visual attributes between different categories, we further use category-specific semantic keypoints, so that object classification and localization can be jointly solved in a cooperative manner. Our experiments demonstrate the effectiveness of rotation equivariant semantic keypoints on oriented object detection.

Shift Pose: A Lightweight Transformer-like Neural Network for Human Pose Estimation.

High-performing, real-time pose detection and tracking in real-time will enable computers to develop a finer-grained and more natural understanding of human behavior. However, the implementation of real-time human pose estimation remains a challenge. On the one hand, the performance of semantic keypoint tracking in live video footage requires high computational resources and large parameters, which limiting the accuracy of pose estimation. On the other hand, some transformer-based models were proposed recently with outstanding performance and much fewer parameters and FLOPs. However, the self-attention module in the transformer is not computationally friendly, which makes it difficult to apply these excellent models to real-time jobs. To overcome the above problems, we propose a transformer-like model, named ShiftPose, which is regression-based approach. The ShiftPose does not contain any self-attention module. Instead, we replace the self-attention module with a non-parameter operation called the shift operator. Meanwhile, we adapt the bridge–branch connection, instead of a fully-branched connection, such as HRNet, as our multi-resolution integration scheme. Specifically, the bottom half of our model adds the previous output, as well as the output from the top half of our model, corresponding to its resolution. Finally, the simple, yet promising, disentangled representation (SimDR) was used in our study to make the training process more stable. The experimental results on the MPII datasets were 86.4 PCKH, 29.1PCKH@0.1. On the COCO dataset, the results were 72.2 mAP and 91.5 AP50, 255 fps on GPU, with 10.2M parameters, and 1.6 GFLOPs. In addition, we tested our model for single-stage 3D human pose estimation and draw several useful and exploratory conclusions. The above results show good performance, and this paper provides a new method for high-performance, real-time attitude detection and tracking.

OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association

Many image-based perception tasks can be formulated as detecting, associating and tracking semantic keypoints, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , human body pose estimation and tracking. In this work, we present a general framework that jointly detects and forms spatio-temporal keypoint associations in a single stage, making this the first real-time pose detection and tracking algorithm. We present a generic neural network architecture that uses Composite Fields to detect and construct a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">spatio-temporal pose</i> which is a single, connected graph whose nodes are the semantic keypoints ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., a person’s body joints) in multiple frames. For the temporal associations, we introduce the Temporal Composite Association Field (TCAF) which requires an extended network architecture and training method beyond previous Composite Fields. Our experiments show competitive accuracy while being an order of magnitude faster on multiple publicly available datasets such as COCO, CrowdPose and the PoseTrack 2017 and 2018 datasets. We also show that our method generalizes to any class of semantic keypoints such as car and animal parts to provide a holistic perception framework that is well suited for urban mobility such as self-driving cars and delivery robots.

Object aspect classification and 6DoF pose estimation

• We introduce a simple aspect-representation for 3D objects for 6DoF pose estimation. • This representation is used to cluster appearances of an object as aspects. • Proposed model predicts both the aspect of a given object and the semantic-key points. • Multi-tasking to high confidence key-point vectors hence more accurate results. • Proposed method is significantly faster due to small backbone and limited PnP Space. We present a novel approach to the problem of estimating a given object's 6DoF pose from a single RGB image. Recent works focus on a multi-stage approach, which first detects key-points followed by perspective -n-point pose estimation algorithm and a pose refinement procedure. We show that adding a classifier block estimating the predefined aspects of the objects improves the multi-stage process. This is due to the fact that the additional classifier acts as a constraint simplifying the required neural network and at the same time yielding better key-point selection. We reduce the search space for the key-point selection and exclude false-positives by mapping the appearance of an object to an aspect. The simplified neural network allows faster inference and a smaller footprint. Our experiments show that our hypothesis performs similar to the state-of-the-art on three different datasets. We also show that an off-the-shelf refinement process can further improve our results to surpass state-of-the-art on several objects. Another advantage is, the proposed pipeline can run efficiently on real-time due to the smaller neural network backbone used. The code to replicate this research will be publicly available at https://github.com/greymad/6DoFPoseAspects

Texture-Less Shiny Objects Grasping in a Single RGB Image Using Synthetic Training Data

In the industrial domain, estimating the pose of texture-less shiny parts is challenging but worthwhile. In this study, it is impractical to utilize texture information to obtain the pose because the features are likely to be affected by the surrounding objects. In addition, the colors of the metal parts are similar, making object segmentation challenging. This study proposes dividing the entire process into three steps: object detection, feature extraction, and pose estimation. We use the Mask-RCNN to detect objects and HRNet to extract the corresponding features. For metal parts of different shapes, different keypoints were chosen accordingly. Conventional contour-based methods are inapplicable to parts containing planar surfaces because the objects occlude each other in clustered environments. In this case, we used dense discrete points along the edges as semantic keypoints for metal parts containing planar elements. We chose skeleton points as semantic keypoints for parts containing cylindrical components. Subsequently, we combined the localization of semantic keypoints and the corresponding CAD model information to estimate the 6D pose of an individual object in sight. The implementation of deep learning approaches requires massive training datasets and intensive labeling. Thus, we propose a method to generate training datasets and automatically label them. Experiments show that the algorithm based on synthetic data performs well in a natural environment, despite not utilizing real scenario images for training.

Monocular Relative Pose Estimation Pipeline for Uncooperative Resident Space Objects

This paper aims to present a deep learning-based pipeline for estimating the pose of an uncooperative target spacecraft from a single grayscale monocular image. The possibility of enabling autonomous vision-based relative navigation in close proximity to a noncooperative resident space object would be especially appealing for mission scenarios such as on-orbit servicing and active debris removal. The relative pose estimation pipeline proposed in this work leverages state-of-the-art convolutional neural network (CNN) architectures to detect the features of the target spacecraft using monocular vision. Specifically, the overall pipeline is composed of three main subsystems. The input image is first processed using an object detection CNN that localizes the bounding box enclosing our target. This is followed by a second CNN that regresses the location of semantic key points of the spacecraft. Eventually, a geometric optimization algorithm exploits the detected key-point locations to solve for the final relative pose. The proposed pipeline demonstrated centimeter-/degree-level pose accuracy on the spacecraft pose estimation dataset (SPEED), along with considerable robustness to changes in illumination and background conditions. In addition, the architecture showed to generalize well on real images, despite having exclusively exploited synthetic data from the SPEED to train the CNNs.

SKP: Semantic 3D Keypoint Detection for Category-Level Robotic Manipulation

As human-assisting robots are starting to play an important role in the food service and hospitality field, service vision for robot manipulation on intra-category objects, which have variations in shape, size, and appearance, is an inevitable and valuable task. Previous works typically rely on estimating the pose of the objects, such methods are not sufficient to generalize well to category-level. Hence, we propose Semantic Keypoint Detection Network (SKP) for category-level robotic manipulation tasks, that 1) uses semantic keypoint representation to expand the perception module to category level and drive the entire grasping pipeline to handle shape variations and dynamic environment, 2) leverages the underlying geometric constraints via jointly optimize instance-aware features with the self-selection mechanism, 3) utilizes multi-scale multi-layer features for 3D keypoint position regression and keypoint association and 4) train with synthetic data only with domain randomization. Extensive experiments on synthetic and real-world datasets show that our work improves accuracy for category-level keypoint detection and demonstrates the generalizability and practicality in robotics grasping tasks.

Semantic keypoint-based pose estimation from single RGB frames

This paper presents an approach to estimating the continuous 6-DoF pose of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. Unlike prior investigators, we are agnostic to whether the object is textured or textureless, as the convnet learns the optimal representation from the available training-image data. Furthermore, the approach can be applied to instanceand class-based pose recovery. Additionally, we accompany our main pipeline with a technique for semi-automatic data generation from unlabeled videos. This procedure allows us to train the learnable components of our method with minimal manual intervention in the labeling process. Empirically, we show that our approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios even against a cluttered background. We apply our approach both to several, existing, large-scale datasets - including PASCAL3D+, LineMOD-Occluded, YCB-Video, and TUD-Light - and, using our labeling pipeline, to a new dataset with novel object classes that we introduce here. Extensive empirical evaluations show that our approach is able to provide pose estimation results comparable to the state of the art.

Removing Atmospheric Turbulence via Deep Adversarial Learning

Restoring images degraded due to atmospheric turbulence is challenging as it consists of several distortions. Several deep learning methods have been proposed to minimize atmospheric distortions that consist of a single-stage deep network. However, we find that a single-stage deep network is insufficient to remove the mixture of distortions caused by atmospheric turbulence. We propose a two-stage deep adversarial network that minimizes atmospheric turbulence to mitigate this. The first stage reduces the geometrical distortion and the second stage minimizes the image blur. We improve our network by adding channel attention and a proposed sub-pixel mechanism, which utilizes the information between the channels and further reduces the atmospheric turbulence at the finer level. Unlike previous methods, our approach neither uses any prior knowledge about atmospheric turbulence conditions at inference time nor requires the fusion of multiple images to get a single restored image. Our final restoration models DT-GAN+ and DTD-GAN+ outperform the general state-of-the-art image-to-image translation models and baseline restoration models. We synthesize turbulent image datasets to train the restoration models. Additionally, we also curate a natural turbulent dataset from YouTube to show the generalisability of the proposed model. We perform extensive experiments on restored images by utilizing them for downstream tasks such as classification, pose estimation, semantic keypoint estimation, and depth estimation. We observe that our restored images outperform turbulent images in downstream tasks by a significant margin demonstrating the restoration model's applicability in real-world problems.

Stereo CenterNet-based 3D object detection for autonomous driving

Recently, three-dimensional (3D) detection based on stereo images has progressed remarkably; however, most advanced methods adopt anchor-based two-dimensional (2D) detection or depth estimation to address this problem. Nevertheless, high computational cost inhibits these methods from achieving real-time performance. In this study, we propose a 3D object detection method, Stereo CenterNet (SC), using geometric information in stereo imagery. SC predicts the four semantic key points of the 3D bounding box of the object in space and utilizes 2D left and right boxes, 3D dimension, orientation, and key points to restore the bounding box of the object in the 3D space. Subsequently, we adopt an improved photometric alignment module to further optimize the position of the 3D bounding box. Experiments conducted on the KITTI dataset indicate that the proposed SC exhibits the best speed-accuracy trade-off among advanced methods without using extra data.

Efficient interactions for reconstructing complex buildings via joint photometric and geometric saliency segmentation

The reconstruction of level-of-detail 2 (LOD-2) buildings has drawn considerable attention over the past two decades. Since completely automatic reconstruction approaches still face many difficulties in industry solutions, efficient and robust interactive modeling tools are of huge demand. This paper proposes an interactive LOD-2 modeling approach that can generate boundary-precise and topology-correct building models efficiently from Digital Surface Model (DSM) and Digital Orthophoto Map (DOM) data. Click-based saliency segmentation is first adopted to extract the outline of building primitives, where both photometric and geometric information are unified. Those outlines decompose the complex-shaped buildings into predefined primitive types. The primitives are then further enriched with other semantic key points, which are essential for inferring the primitives’ shapes and positions. Finally, the models can be built based on the constructive solid geometry and assembly of partial primitives into complex buildings. The order of the primitives can be irrelevant, and the sketches and semantic key points do not necessarily be perfect. Users only need very few and simple operators to generate the integral models, mostly several clicks on a 2D platform. Experimental results on various dataset demonstrate the proposed approach can minimize the amount of interaction while maintaining the ability to deal with various roof types. The integral model quality indices at pixel and object level achieve the precision of 92.3% and 82.5%, accordingly.