Pose Estimation Of Spacecraft Research Articles

PVSPE: A pyramid vision multitask transformer network for spacecraft pose estimation

Spacecraft pose estimation (SPE) plays a vital role in the relative navigation system for on-orbit servicing and active debris removal. Current deep learning-based methods have made great achievements on object pose estimation. However, towards the challenging onboard SPE missions, most existing Convolutional Neural Network (CNN) methods failed to capture remote vision attention, leading to the reduction of accuracy and robustness. In this paper, we presented an end-to-end multi-task Pyramid Transformer SPE network (PVSPE) consisting of two novel feature extraction modules: EnhancedPVT (EnPVT) and SlimGFPN. The EnPVT module is designed to combine global spatial and channel attention, while the Slim GFPN module can fuse features more effectively. Matrix Fisher and multivariate Gaussian distributions are further employed to model the uncertainty of pose regression to increase its accuracy. Extensive experiments are carried out on challenging SPEED + and SHIRT datasets, to validate the performances on pose estimation and vision-based navigation, respectively. The results show that the proposed PVSPE model achieved high accuracy for SPE on the SPEED + dataset even under different scales and severe illumination, demonstrating its robustness and high generalization. Leveraging the insightful uncertainty model of PVSPE, the vision-based navigation pipeline, combined with Kalman filters, accurately estimated the satellite pose under challenging rendezvous scenarios on the SHIRT dataset, with degree-level attitude errors and centimeter-level translation accuracy at steady-state.

Transformer Network-Aided Relative Pose Estimation for Non-cooperative Spacecraft Using Vision Sensor

AbstractThe objective of the proposed work is to perform monocular vision-based relative 6-DOF pose estimation of the non-cooperative target spacecraft relative to the chaser satellite in rendezvous operations. In this work, the convolutional neural network (CNN) is replaced by the high-resolution transformer network to predict the feature points of the target satellite. The self-attention mechanism inside the transformer provides the advantage of overcoming the inadequacies of the translation equivariance, 2D neighborhood awareness, and long-range dependencies in CNN. First, the 3D model of the target satellite is reconstructed using the inverse direct linear transform (IDLT) method. Then, the pose estimation pipeline is developed with a learning-based image-processing subsystem and geometric optimization of the pose solver. The image-processing subsystem performs target localization using CNN-based architecture. Then, the key points detection network performs regression to predict 2D key points using the transformer-based network. Afterward, the predicted key points based on their confidence scores are projected onto the corresponding 3D points, and the pose value is computed using the efficient perspective-n-point method. The pose is refined using the non-linear iterative Gauss–Newton method. The proposed architecture is trained and tested on the spacecraft pose estimation dataset and it shows superior accuracy both in translation and rotation values. The architecture has shown robustness against the drastically changing clutter background and light conditions in the space images due to the self-attention mechanism. Moreover, this method consumes less computation resources by using fewer floating-point operations and trainable parameters with low input image resolution.

Spacecraft Homography Pose Estimation with Single-Stage Deep Convolutional Neural Network.

Spacecraft pose estimation using computer vision has garnered increasing attention in research areas such as automation system theory, control theory, sensors and instruments, robot technology, and automation software. Confronted with the extreme environment of space, existing spacecraft pose estimation methods are predominantly multi-stage networks with complex operations. In this study, we propose an approach for spacecraft homography pose estimation with a single-stage deep convolutional neural network for the first time. We formulated a homomorphic geometric constraint equation for spacecraft with planar features. Additionally, we employed a single-stage 2D keypoint regression network to obtain homography 2D keypoint coordinates for spacecraft. After decomposition to obtain the rough spacecraft pose based on the homography matrix constructed according to the geometric constraint equation, a loss function based on pixel errors was employed to refine the spacecraft pose. We conducted extensive experiments using widely used spacecraft pose estimation datasets and compared our method with state-of-the-art techniques in the field to demonstrate its effectiveness.

Feature-aided pose estimation approach based on variational auto-encoder structure for spacecrafts

Real-time 6 Degree-of-Freedom (DoF) pose estimation is of paramount importance for various on-orbit tasks. Benefiting from the development of deep learning, Convolutional Neural Networks (CNNs) in feature extraction has yielded impressive achievements for spacecraft pose estimation. To improve the robustness and interpretability of CNNs, this paper proposes a Pose Estimation approach based on Variational Auto-Encoder structure (PE-VAE) and a Feature- Aided pose estimation approach based on Variational Auto-Encoder structure (FA-VAE), which aim to accurately estimate the 6 DoF pose of a target spacecraft. Both methods treat the pose vector as latent variables, employing an encoder-decoder network with a Variational Auto-Encoder (VAE) structure. To enhance the precision of pose estimation, PE-VAE uses the VAE structure to introduce reconstruction mechanism with the whole image. Furthermore, FA-VAE enforces feature shape constraints by exclusively reconstructing the segment of the target spacecraft with the desired shape. Comparative evaluation against leading methods on public datasets reveals similar accuracy with a threefold improvement in processing speed, showcasing the significant contribution of VAE structures to accuracy enhancement, and the additional benefit of incorporating global shape prior features.

Spacecraft Pose Estimation: Robust 2D and 3D-Structural Losses and Unsupervised Domain Adaptation by Inter-Model Consensus

Spacecraft Pose Estimation: Robust 2D and 3D-Structural Losses and Unsupervised Domain Adaptation by Inter-Model Consensus

SESPnet: a lightweight network with attention mechanism for spacecraft pose estimation

SESPnet: a lightweight network with attention mechanism for spacecraft pose estimation

Spacecraft Pose Estimation Based on Different Camera Models

Spacecraft pose estimation is an important technology to maintain or change the spacecraft orientation in space. For spacecraft pose estimation, when two spacecraft are relatively distant, the depth information of the space point is less than that of the measuring distance, so the camera model can be seen as a weak perspective projection model. In this paper, a spacecraft pose estimation algorithm based on four symmetrical points of the spacecraft outline is proposed. The analytical solution of the spacecraft pose is obtained by solving the weak perspective projection model, which can satisfy the requirements of the measurement model when the measurement distance is long. The optimal solution is obtained from the weak perspective projection model to the perspective projection model, which can meet the measurement requirements when the measuring distance is small. The simulation results show that the proposed algorithm can obtain better results, even though the noise is large.

Leveraging neural network uncertainty in adaptive unscented Kalman Filter for spacecraft pose estimation

This paper introduces an adaptive Convolutional Neural Network (CNN)-based Unscented Kalman Filter for the pose estimation of uncooperative spacecraft. The validation is carried out at Stanford’s robotic Testbed for Rendezvous and Optical Navigation on the Satellite Hardware-In-the-loop Rendezvous Trajectories (SHIRT) dataset, which simulates vision-based rendezvous trajectories of a servicer spacecraft to PRISMA’s Tango spacecraft. The proposed navigation system is stress-tested on synthetic as well as realistic lab imagery by simulating space-like illumination conditions on-ground. The validation is performed at different levels of the navigation system by first training and testing the adopted CNN on SPEED+, Stanford’s spacecraft pose estimation dataset with specific emphasis on domain shift between a synthetic domain and an Hardware-In-the-Loop domain. A novel data augmentation scheme based on light randomization is proposed to improve the CNN robustness under adverse viewing conditions, reaching centimeter-level and 10 degree-level pose errors in 80% of the SPEED+ lab images. Next, the entire navigation system is tested on the SHIRT dataset. Results indicate that the inclusion of a new scheme to adaptively scale the heatmaps-based measurement error covariance based on filter innovations improves filter robustness by returning centimeter-level position errors and moderate attitude accuracies, suggesting that a proper representation of the measurements uncertainty combined with an adaptive measurement error covariance is key in improving the navigation robustness.

Robust multi-task learning and online refinement for spacecraft pose estimation across domain gap

This work presents Spacecraft Pose Network v2 (SPNv2), a Convolutional Neural Network (CNN) for pose estimation of noncooperative spacecraft across domain gap. SPNv2 is a multi-scale, multi-task CNN which consists of a shared multi-scale feature encoder and multiple prediction heads that perform different tasks on a shared feature output. These tasks are all related to detection and pose estimation of a target spacecraft from an image, such as prediction of pre-defined satellite keypoints, direct pose regression, and binary segmentation of the satellite foreground. It is shown that by jointly training on different yet related tasks with extensive data augmentations on synthetic images only, the shared encoder learns features that are common across image domains that have fundamentally different visual characteristics compared to synthetic images. This work also introduces Online Domain Refinement (ODR) which refines the parameters of the normalization layers of SPNv2 on the target domain images online at deployment. Specifically, ODR performs self-supervised entropy minimization of the predicted satellite foreground, thereby improving the CNN’s performance on the target domain images without their pose labels and with minimal computational efforts. The GitHub repository for SPNv2 is available at https://github.com/tpark94/spnv2.

Position Awareness Network for Noncooperative Spacecraft Pose Estimation Based on Point Cloud

Spacecraft pose estimation plays a vital role in many on-orbit space missions, such as rendezvous and docking, debris removal, and on-orbit maintenance. At present, the mainstream descriptors-based pose estimation methods ignore the fact that satellite point cloud contains many similar structures, generating numerous mismatched correspondence pairs, and leading to low pose estimation accuracy. This article proposes a position awareness network (PANet) for spacecraft pose estimation. Specifically, the point cloud is first fed into a hierarchical embedding network to extract the key points and construct local structural descriptors. We also build the relative position features by encoding the relative position between key points and reference points. The matching matrix between point clouds is then calculated by comprehensively considering the local structure descriptors and relative location features. In this way, the problem of ambiguous matching caused by similar local structures is avoided. Finally, weighted singular value decomposition (SVD) is utilized to solve the pose between the point clouds based on the correspondence pairs generated by the matching matrix. Besides, a large-scale satellite point cloud dataset is also constructed for training and testing pose estimation algorithms. Empirical experiments on the dataset demonstrate the effectiveness of the proposed PANet, which achieves 1.18$^\circ$ rotation error and 0.136 m translate error, surpassing state-of-the-art methods by a large margin.

Satellite Pose Estimation Competition 2021: Results and Analyses

This article concerns the second Satellite Pose Estimation Competition (SPEC2021), an international competition designed to address the challenging problem of domain gap in developing spaceborne computer vision algorithms. Due to the lack of representative spaceborne images of the target object at the development stage, most algorithms are typically developed and trained on computer-generated synthetic images of the target, which causes the algorithm to overfit the features specific to the synthetic imagery. In order to promote new ideas to tackle this important problem and benchmark the robustness of different algorithms across domain gaps, SPEC2021 utilizes the next-generation spacecraft pose estimation dataset comprising images from labeled images from computer graphics and real-life sources. This article covers the dataset and competition design and analyzes the results of the competition which received submissions from 36 participating teams. Specifically, through the solutions developed by the two winning teams, this article sheds light on key components that can contribute to successfully bridging the domain gap in spaceborne computer vision applications.

Spacecraft depth completion based on the gray image and the sparse depth map

Perceiving the three-dimensional (3D) structure of the spacecraft is a prerequisite for successfully executing many on-orbit space missions, and it can provide critical input for many downstream vision algorithms. In this paper, we propose to sense the 3D structure of spacecraft using light detection and ranging sensor (LIDAR) and a monocular camera. To this end, Spacecraft Depth Completion Network (SDCNet) is proposed to recover the dense depth map based on gray image and sparse depth map. Specifically, SDCNet decomposes the object-level spacecraft depth completion task into foreground segmentation subtask and foreground depth completion subtask, which segments the spacecraft region first and then performs depth completion on the segmented foreground area. In this way, the background interference to foreground spacecraft depth completion is effectively avoided. Moreover, an attention-based feature fusion module is also proposed to aggregate the complementary information between different inputs, which deduces the correlation between different features along the channel and the spatial dimension sequentially. Besides, four metrics are also proposed to evaluate object-level depth completion performance, which can more intuitively reflect the quality of spacecraft depth completion results. Finally, a large-scale satellite depth completion dataset is constructed for training and testing spacecraft depth completion algorithms. Empirical experiments on the dataset demonstrate the effectiveness of the proposed SDCNet, which achieves 0.225m mean absolute error of interest and 0.778m mean absolute truncation error, surpassing state-of-the-art methods by a large margin. The spacecraft pose estimation experiment is also conducted based on the depth completion results, and the experimental results indicate that the predicted dense depth map could meet the needs of downstream vision tasks.

Monocular-Vision-Based Relative Pose Estimation of Noncooperative Spacecraft Using Multicircular Features

Accurate relative pose (position and orientation) estimation for noncooperative target spacecraft is the basic module for on-orbit services, such as capture and repair. This article technically proposes a nonlinear optimization method based on monocular vision to estimate the relative pose for the solid-of-revolution-shaped noncooperative spacecraft. Specifically, considering the multicircular features on the target, the parameters of the ellipses derived from the circles perspective projection are first obtained by the propounded ellipse detection algorithm. Moreover, the constraint that the center of each circle falls on the center axis (normal direction) of the spacecraft is utilized to optimally solve the center position and the normal of the multicircles. In particular, the roll angle around the center axis is recovered through the geometric constraints of the solar panels, and the six-degree-of-freedom spacecraft pose estimation is then accomplished. Consequently, it is analyzed that the proposed optimization algorithm is able to provide a closed-form solution to ensure the accuracy of the relative pose and only requires limited computational resources. Finally, experimental results on the synthetic images and the physical scenes are conducted to evaluate the efficiency of the proposed approach.

Monocular Relative Pose Estimation Pipeline for Uncooperative Resident Space Objects

This paper aims to present a deep learning-based pipeline for estimating the pose of an uncooperative target spacecraft from a single grayscale monocular image. The possibility of enabling autonomous vision-based relative navigation in close proximity to a noncooperative resident space object would be especially appealing for mission scenarios such as on-orbit servicing and active debris removal. The relative pose estimation pipeline proposed in this work leverages state-of-the-art convolutional neural network (CNN) architectures to detect the features of the target spacecraft using monocular vision. Specifically, the overall pipeline is composed of three main subsystems. The input image is first processed using an object detection CNN that localizes the bounding box enclosing our target. This is followed by a second CNN that regresses the location of semantic key points of the spacecraft. Eventually, a geometric optimization algorithm exploits the detected key-point locations to solve for the final relative pose. The proposed pipeline demonstrated centimeter-/degree-level pose accuracy on the spacecraft pose estimation dataset (SPEED), along with considerable robustness to changes in illumination and background conditions. In addition, the architecture showed to generalize well on real images, despite having exclusively exploited synthetic data from the SPEED to train the CNNs.

Influence factors of the accuracy of monocular vision pose estimation for spacecraft based on neural network

Accurate acquisition of spacecraft pose information is a key step in tasks such as space debris removal, rendezvous and docking, and on-orbit maintenance. The application of neural network has deepened for aerospace measurement and control. Studying the parameters’ influence and degree of imaging and neural network on the accuracy of spacecraft pose estimation is necessary and urgent, but there is still little research in related fields. Approximate Bayesian inference in the depth Gaussian process is utilized in this paper to explore above problems, and systematic verification experiments are carried based on the general neural network and spacecraft pose estimation data set. The verification results depict that the estimation errors of position and attitude can be both effectively reduced by selecting a camera whose radial distortion error is less than 240%. Selecting a camera with focal ratio error less than 2.0% can effectively reduce the position estimation error. Selecting a camera whose principal point error is less than 1.0% can effectively improve the accuracy of attitude estimation. The suppression of tangential distortion error does not significantly improve the estimation accuracy of both position and attitude. Parameters about weighted attenuation factor, the number of network layers and characteristic channels of neural network are all sensitive to the accuracy of pose estimation of spacecraft within a specific range. An excellent balance can be achieved between the estimation accuracy and computational overhead when the weight attenuation factor is set between 0.01 and 0.1, and the number of network layers and characteristic channels are set separately between 20 ∼ 40 and 70 ∼ 90. It can reach the accuracy of the traditional pose estimation algorithm, and has better processing time stability in real ground test system by using the network architecture of ResNet34 with weight attenuation factor of 0.01, characteristic channels of 90.

Uncooperative Spacecraft Pose Estimation Using Monocular Monochromatic Images

Imaging cameras are cost-effective sensors for spacecraft navigation. Image-driven techniques to extract the target spacecraft from its background are efficient and do not require pretraining. In this paper, we introduce several image-driven foreground extraction methods, including combining the difference of Gaussian-based scene detection and graph manifold ranking-based foreground saliency generation. We successfully apply our foreground extraction method on infrared images from the STS-135 flight mission captured by the space shuttle’s Triangulation and LIDAR Automated Rendezvous and Docking System (TriDAR) thermal camera. Our saliency approach demonstrates state-of-the-art performance and provides an order of magnitude reduction in processing speed from the traditional methods. Furthermore, we develop a new uncooperative spacecraft pose estimation method by combining our foreground extraction technique with the level-set region-based pose estimation with novel initialization and gradient descent enhancements. Our method is validated using synthetically generated Envisat, Radarsat model, and International Space Station motion sequences. The proposed process is also validated with real rendezvous flight images of the International Space Station.

Neural Network-Based Pose Estimation for Noncooperative Spacecraft Rendezvous

This article presents the Spacecraft Pose Network (SPN), the first neural network-based method for on-board estimation of the pose, i.e., the relative position and attitude, of a known noncooperative spacecraft using monocular vision. In contrast to other state-of-the-art pose estimation approaches for spaceborne applications, the SPN method does not require the formulation of hand-engineered features and only requires a single grayscale image to determine the pose of the spacecraft relative to the camera. The SPN method uses a convolutional neural network (CNN) with three branches to solve the problem of relative attitude estimation. The first branch of the CNN bootstraps a state-of-the-art object detection algorithm to detect a 2-D bounding box around the target spacecraft in the input image. The region inside the 2-D bounding box is then used by the other two branches of the CNN to determine the relative attitude by initially classifying the input region into discrete coarse attitude labels before regressing to a finer estimate. The SPN method then estimates the relative position by using the constraints imposed by the detected 2-D bounding box and the estimated relative attitude. Further, with the detection of 2-D bounding boxes of subcomponents of the target spacecraft, the SPN method is easily generalizable to estimate the pose of multiple target geometries. Finally, to facilitate integration with navigation filters and perform continuous pose tracking, the SPN method estimates the uncertainty associated with the estimated pose. The secondary contribution of this article is the generation of the Spacecraft PosE Estimation Dataset (SPEED), which is used to train and evaluate the performance of the SPN method. SPEED consists of synthetic as well as actual camera images of a mock-up of the Tango spacecraft from the PRISMA mission. The synthetic images are created by fusing OpenGL-based renderings of the spacecraft's 3-D model with actual images of the Earth captured by the Himawari-8 meteorological satellite. The actual camera images are created using a seven degrees-of-freedom robotic arm, which positions and orients a vision-based sensor with respect to a full-scale mock-up of the Tango spacecraft with submillimeter and submillidegree accuracy. The SPN method, trained only on synthetic images, produces degree-level relative attitude error and cm-level relative position errors when evaluated on the actual camera images with a different distribution not used during training.

Utilization of FPGA for Onboard Inference of Landmark Localization in CNN-Based Spacecraft Pose Estimation

In the recent past, research on the utilization of deep learning algorithms for space applications has been widespread. One of the areas where such algorithms are gaining attention is in spacecraft pose estimation, which is a fundamental requirement in many spacecraft rendezvous and navigation operations. Nevertheless, the application of such algorithms in space operations faces unique challenges compared to application in terrestrial operations. In the latter, they are facilitated by powerful computers, servers, and shared resources, such as cloud services. However, these resources are limited in space environment and spacecrafts. Hence, to take advantage of these algorithms, an on-board inferencing that is power- and cost-effective is required. This paper investigates the use of a hybrid Field Programmable Gate Array (FPGA) and Systems-on-Chip (SoC) device for efficient onboard inferencing of the Convolutional Neural Network (CNN) part of such pose estimation methods. In this study, Xilinx’s Zynq UltraScale+ MPSoC device is used and proposed as an effective onboard-inferencing solution. The performance of the onboard and computer inferencing is compared, and the effectiveness of the hybrid FPGA-CPU architecture is verified. The FPGA-based inference has comparable accuracy to the PC-based inference with an average RMS error difference of less than 0.55. Two CNN models that are based on encoder-decoder architecture have been investigated in this study and three approaches demonstrated for landmarks localization.

Fast and Accurate Spacecraft Pose Estimation From Single Shot Space Imagery Using Box Reliability and Keypoints Existence Judgments

Real-time 6DOF (6 Degree of Freedom) pose estimation of an uncooperative spacecraft is an important part of proximity operations, e.g., space debris removal, spacecraft rendezvous and docking, on-orbit servicing, etc. In this article, a novel efficient deep learning based approach is proposed to estimate the 6DOF pose of uncooperative spacecraft using monocular-vision measurement. Firstly, we introduce a new lightweight YOLO-liked CNN to detect spacecraft and predict 2D locations of the projected keypoints of a prior reconstructed 3D model in real-time. Then, we design two novel models for predicting the bounding box (bbox) reliability scores and the probability of keypoints existence. The two models not only significantly reduce the false positive, but also speed up convergence. Finally, the 6DOF pose is estimated and refined using Perspective-n-Point and geometric optimizer. Results demonstrate that the proposed approach achieves 73.2% average precision and 77.6% average recall for spacecraft detection on the SPEED dataset after only 200 training epochs. For the pose estimation task, the mean rotational error is 0.6812°, and the mean translation error is 0.0320m. The proposed approach achieves competitive pose estimation performance and extreme lightweight ( $\sim ~0.89$ million learnable weights in total) on the SPEED dataset while being efficient for real-time applications.

Rectangular-structure-based pose estimation method for non-cooperative rendezvous

Pose estimation for spacecraft is widely recognized as an important technology for space applications. Many space missions require accurate relative pose between the chaser and the target spacecraft. Stereovision is a usual mean to estimate the pose of non-cooperative targets during proximity operations. In this paper, an accurate algorithm for pose estimation is proposed by taking advantage of the geometric structure of the object. With stereo cameras, our approach employs the corners of the solar panel as features and uses bundle adjustment to improve the accuracy. Simulation results demonstrate that our method improves the precision and robustness for non-cooperative spacecraft pose estimation. Our laboratory experiments further validate the approach. The results show that the rotation angle error of our method is within 0.8°, and the measurement error in translation is less than 0.4% at the distance from 3m to 1.2m.