Pose Regression Network Research Articles

A Dexterous Hand-Arm Teleoperation System Based on Hand Pose Estimation and Active Vision.

Markerless vision-based teleoperation that leverages innovations in computer vision offers the advantages of allowing natural and noninvasive finger motions for multifingered robot hands. However, current pose estimation methods still face inaccuracy issues due to the self-occlusion of the fingers. Herein, we develop a novel vision-based hand-arm teleoperation system that captures the human hands from the best viewpoint and at a suitable distance. This teleoperation system consists of an end-to-end hand pose regression network and a controlled active vision system. The end-to-end pose regression network (Transteleop), combined with an auxiliary reconstruction loss function, captures the human hand through a low-cost depth camera and predicts joint commands of the robot based on the image-to-image translation method. To obtain the optimal observation of the human hand, an active vision system is implemented by a robot arm at the local site that ensures the high accuracy of the proposed neural network. Human arm motions are simultaneously mapped to the slave robot arm under relative control. Quantitative network evaluation and a variety of complex manipulation tasks, for example, tower building, pouring, and multitable cup stacking, demonstrate the practicality and stability of the proposed teleoperation system.

FilterformerPose: Satellite Pose Estimation Using Filterformer.

Satellite pose estimation plays a crucial role within the aerospace field, impacting satellite positioning, navigation, control, orbit design, on-orbit maintenance (OOM), and collision avoidance. However, the accuracy of vision-based pose estimation is severely constrained by the complex spatial environment, including variable solar illumination and the diffuse reflection of the Earth's background. To overcome these problems, we introduce a novel satellite pose estimation network, FilterformerPose, which uses a convolutional neural network (CNN) backbone for feature learning and extracts feature maps at various CNN layers. Subsequently, these maps are fed into distinct translation and orientation regression networks, effectively decoupling object translation and orientation information. Within the pose regression network, we have devised a filter-based transformer encoder model, named filterformer, and constructed a hypernetwork-like design based on the filter self-attention mechanism to effectively remove noise and generate adaptive weight information. The related experiments were conducted using the Unreal Rendered Spacecraft On-Orbit (URSO) dataset, yielding superior results compared to alternative methods. We also achieved better results in the camera pose localization task, indicating that FilterformerPose can be adapted to other computer vision downstream tasks.

Structure-guided camera localization for indoor environments

Camera-based indoor localization is a fundamental aspect of indoor navigation, virtual reality, and location-based services. Deep learning methods have exhibited remarkable performance with low storage requirements and high efficiency. However, existing methods mainly derive features implicitly for pose regression without considering explicit structure information from images. This paper proposes that incorporating such information can improve the localization performance of learning-based approaches. We extract structure information from RGB images in the form of depth maps and edge maps and design two modules for depth-weighted and edge-weighted feature fusion. These modules are integrated into the pose regression network to enhance pose prediction. Furthermore, we employ a self-attention module for high-level feature extraction to augment the network capacity. Extensive experiments are conducted on the publicly available 7Scenes and 12Scenes datasets, and the results demonstrate that the proposed method achieves high localization performance, with an average positional error of 0.19m and 0.16m, respectively. The code for this work is available at https://github.com/lqing900205/structureLoc.

A Compact and Powerful Single-Stage Network for Multi-Person Pose Estimation

Multi-person pose estimation generally follows top-down and bottom-up paradigms. The top-down paradigm detects all human boxes and then performs single-person pose estimation on each ROI. The bottom-up paradigm locates identity-free keypoints and then groups them into individuals. Both of them use an extra stage to build the relationship between human instance and corresponding keypoints (e.g., human detection in a top-down manner or a grouping process in a bottom-up manner). The extra stage leads to a high computation cost and a redundant two-stage pipeline. To address the above issue, we introduce a fine-grained body representation method. Concretely, the human body is divided into several local parts and each part is represented by an adaptive point. The novel body representation is able to sufficiently encode the diverse pose information and effectively model the relationship between human instance and corresponding keypoints in a single-forward pass. With the proposed body representation, we further introduce a compact single-stage multi-person pose regression network, called AdaptivePose++, which is the extended version of AAAI-22 paper AdaptivePose. During inference, our proposed network only needs a single-step decode operation to estimate the multi-person pose without complex post-processes and refinements. Without any bells and whistles, we achieve the most competitive performance on representative 2D pose estimation benchmarks MS COCO and CrowdPose in terms of accuracy and speed. In particular, AdaptivePose++ outperforms the state-of-the-art SWAHR-W48 and CenterGroup-W48 by 3.2 AP and 1.4 AP on COCO mini-val with faster inference speed. Furthermore, the outstanding performance on 3D pose estimation datasets MuCo-3DHP and MuPoTS-3D further demonstrates its effectiveness and generalizability on 3D scenes.

DiveNet: Dive Action Localization and Physical Pose Parameter Extraction for High Performance Training

The tremendous progress of deep convolution neural networks has shown promising results on the classification of various sports activities. However, the accurate localization of a particular sports event or activity in a continuous video stream is still a challenging problem. The accurate detection of sports actions enables the comparison of different performances, objectively. In this work, we propose the DiveNet action localization module to detect the springboard diving sports action in an unconstrained environment. We used Temporal Convolution Network (TCN) over a backbone feature extractor to localize diving actions, with low latency. We estimate the divers center of mass (COM) trajectory and the peak dive height using the temporal demarcations provided by the action localization step via the projectile motion formula. In addition, we train a DiveNet pose regression network, which extends the Unipose architecture with direct physical parameter estimation, i.e COM and 2D joint keypoints. We propose a new homography computation method between the diving motion plane and the image-view for each dive. This enables the representation of physical parameters in metric scale, without any calibration. We release the first publicly available diving sports video dataset, recorded at 60 Hz with a static camera setup for different springboard heights. DiveNet action localization achieves an accuracy of 95% with a single frame latency (< 25 ms). The DiveNet pose regression model shows competitive results around 70% PCK on different diving pose datasets. We achieve COM accuracy of 6 pixels, dive peak height sensitivity of 20 cm and mean joint angle errors around 10 degrees.

PPR-Net++: Accurate 6-D Pose Estimation in Stacked Scenarios

Most supervised learning-based pose estimation methods for stacked scenes are trained on massive synthetic datasets. In most cases, the challenge is that the learned network on the training dataset is no longer optimal on the testing dataset. To address this problem, we propose a pose regression network PPR-Net++. It transforms each scene point into a point in the centroid space, followed by a clustering process and a voting process. In the training phase, a mapping function between the network’s critical parameter (i.e., the bandwidth of the clustering algorithm) and the compactness of the centroid distributions is obtained. This function is used to adapt the bandwidth between centroid distributions of two different domains. In addition, to further improve the pose estimation accuracy, the network also predicts the confidence of each point, based on its visibility and pose error. Only the points with high confidence have the right to vote for the final object pose. In experiments, our method is trained on the IPA synthetic dataset and compared with the state-of-the-art algorithm. When tested with the public synthetic Siléane dataset, our method is better in all eight objects, where five of them are improved by more than 5% in average precision (AP). On IPA real dataset, our method outperforms a large margin by 20%. This lays a solid foundation for robot grasping in industrial scenarios. Note to Practitioners—Our work is motivated by industrial product assembly based on robot grasping. The industrial parts are usually manufactured by numerical machines and piled in bins. Our method can estimate the poses of visible parts accurately. A pose of a part includes its centroid and spatial orientations. Combined with a depth camera, this algorithm allows an industrial robot to understand complex stacked scenes. We improve the pose estimation accuracy in order to assemble parts with robot grasping, without an additional pose adjuster. Our network can learn from a synthetic dataset and apply it to real-world data, without a significant accuracy drop. The synthetic dataset can be obtained easily by computer simulation programs, so the training data are sufficient. Experiments demonstrate that our method outperforms the state-of-the-art pose estimation approaches.

LiDAR-based localization using universal encoding and memory-aware regression

Visual localization is critical to many robotics and computer vision applications. Absolute pose regression performs localization by encoding scene features followed by pose regression, which has achieved impressive results in localization. It recovers 6-DoF poses from captured scene data alone. However, current methods suffer from being retrained with specific source data whenever the scene changes, resulting in expensive computational costs, data privacy disclosure, and unreliable localization caused by the inability to memorize all data. In this paper, we propose a novel LiDAR-based absolute pose regression network with universal encoding to avoid redundant retraining and the loss of data privacy. Specifically, we propose using universal feature encoding for different scenes. Only the regressor needs to be retrained to achieve higher efficiency, and the training is performed using the encoded features without source data, which preserves data privacy. Then, we propose a memory regressor for memory-aware regression, where the hidden unit numbers in the regressor determine the memorization capacity. It can be used to derive and improve the upper bound of the capacity to enable more reliable localization. Then, it is possible to modify the regressor structure to adapt different memorization capacity requirements for different scene sizes. Extensive experiments on outdoor and indoor datasets validated the above analyses and demonstrated the effectiveness of the proposed method.

Improving synthetic 3D model-aided indoor image localization via domain adaptation

Although the deep learning-based indoor image localization has made significant improvement in terms of accuracy, efficiency, and storage requirement of large indoor scenes, the need for collecting huge labeled training data severely limits its practical application. Recently, the synthetic images rendered from widely available 3D models have shown promising potential to relieve the data collection problem. However, due to the dramatic differences between the synthetic and real images, the localization accuracy of approaches trained on synthetic images is not comparable to the methods trained on real images. In this paper, we propose a domain adaptation-based approach to address this issue. Specifically, the proposed approach mainly contains a model consisting of a multi-level constrained pose regression network and a feature-level discriminator network. The discriminator network forces the pose regression network to generate domain-invariant features from real and synthetic images by adversarial learning and thus reduces the performance gaps. In addition, the multi-level constraints further enhance the localization accuracy of pose regression. We perform extensive experiments on open-source rendering images in different settings. The results show that the proposed method significantly improves the performance. The code for the proposed work is available at https://github.com/lqing900205/BIM_domainadaptation.

Adversarial Pose Regression Network for Pose-Invariant Face Recognitions

Face recognition has achieved significant progress in recent years. However, the large pose variation between face images remains a challenge in face recognition. We observe that the pose variation in the hidden feature maps is one of the most critical factors to hinder the representations from being pose-invariant. Based on the observation, we propose an Adversarial Pose Regression Network (APRN) to extract pose-invariant identity representations by disentangling their pose variation in hidden feature maps. To model the pose discriminator in APRN as a regression task in its 3D space, we also propose an Adversarial Regression Loss Function and extend the adversarial learning from classification problems to regression problems in this paper. Our APRN is a plug-and-play structure that can be embedded in other state-of-the-art face recognition algorithms to improve their performance additionally. The experiments show that the proposed APRN consistently and significantly boosts the performance of baseline networks without extra computational costs in the inference phase. APRN achieves comparable or even superior to the state-of-the-art on CFP, Multi-PIE, IJB-A and MegaFace datasets. The code will be released, hoping to nourish our proposals to other computer vision fields

INDOOR LIDAR RELOCALIZATION BASED ON DEEP LEARNING USING A 3D MODEL

Abstract. Indoor localization, navigation and mapping systems highly rely on the initial sensor pose information to achieve a high accuracy. Most existing indoor mapping and navigation systems cannot initialize the sensor poses automatically and consequently these systems cannot perform relocalization and recover from a pose estimation failure. For most indoor environments, a map or a 3D model is often available, and can provide useful information for relocalization. This paper presents a novel relocalization method for lidar sensors in indoor environments to estimate the initial lidar pose using a CNN pose regression network trained using a 3D model. A set of synthetic lidar frames are generated from the 3D model with known poses. Each lidar range image is a one-channel range image, used to train the CNN pose regression network from scratch to predict the initial sensor location and orientation. The CNN regression network trained by synthetic range images is used to estimate the poses of the lidar using real range images captured in the indoor environment. The results show that the proposed CNN regression network can learn from synthetic lidar data and estimate the pose of real lidar data with an accuracy of 1.9 m and 8.7 degrees.

VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometry

Semantic understanding and localization are fundamental enablers of robot autonomy that have for the most part been tackled as disjoint problems. While deep learning has enabled recent breakthroughs across a wide spectrum of scene understanding tasks, its applicability to state estimation tasks has been limited due to the direct formulation that renders it incapable of encoding scene-specific constrains. In this work, we propose the VLocNet++ architecture that employs a multitask learning approach to exploit the inter-task relationship between learning semantics, regressing 6-DoF global pose and odometry, for the mutual benefit of each of these tasks. Our network overcomes the aforementioned limitation by simultaneously embedding geometric and semantic knowledge of the world into the pose regression network. We propose a novel adaptive weighted fusion layer to aggregate motion-specific temporal information and to fuse semantic features into the localization stream based on region activations. Furthermore, we propose a self-supervised warping technique that uses the relative motion to warp intermediate network representations in the segmentation stream for learning consistent semantics. Finally, we introduce a first-of-a-kind urban outdoor localization dataset with pixel-level semantic labels and multiple loops for training deep networks. Extensive experiments on the challenging Microsoft 7-Scenes benchmark and our DeepLoc dataset demonstrate that our approach exceeds the state-of-the-art outperforming local feature-based methods while simultaneously performing multiple tasks and exhibiting substantial robustness in challenging scenarios.

SHPR-Net: Deep Semantic Hand Pose Regression From Point Clouds

3-D hand pose estimation is an essential problem for human–computer interaction. Most of the existing depth-based hand pose estimation methods consume 2-D depth map or 3-D volume via 2-D/3-D convolutional neural networks. In this paper, we propose a deep semantic hand pose regression network (SHPR-Net) for hand pose estimation from point sets, which consists of two subnetworks: a semantic segmentation subnetwork and a hand pose regression subnetwork. The semantic segmentation network assigns semantic labels for each point in the point set. The pose regression network integrates the semantic priors with both input and late fusion strategy and regresses the final hand pose. Two transformation matrices are learned from the point set and applied to transform the input point cloud and inversely transform the output pose, respectively, which makes the SHPR-Net more robust to geometric transformations. Experiments on NYU, ICVL, and MSRA hand pose data sets demonstrate that our SHPR-Net achieves high performance on par with the start-of-the-art methods. We also show that our method can be naturally extended to hand pose estimation from the multi-view depth data and achieves further improvement on the NYU data set.