Design of relative motion and attitude profiles for three-dimensional resident space object imaging with a laser rangefinder

This paper discusses the application of a single beam laser rangefinder (LRF) to point cloud generation, shape detection, and shape reconstruction for a space-based space situational awareness (SSA) mission. The LRF is part of the payload of a chaser satellite tasked to image a resident space object (RSO). The one-dimensional (1D) nature of LRF returns significantly increases the complexity of the imaging task. To maximize coverage, a method to autonomously detect and fill gaps in sparse point cloud coverage using a narrow field of view (NFOV) camera in conjunction with the LRF is presented. First, relative orbital motion and scanning attitude motion are combined to generate a baseline 3D point cloud of the RSO. The effectiveness of pregenerated command profiles is analyzed by using a weighted edge reconstruction metric that scores how well a point cloud characterizes RSO shape. The design and characterization of multiple relative motion and attitude maneuver profiles, as well as the time and propellant cost of each profile, are presented with the assumption that the entire metrology chain is error free. Next, a three-part algorithm is used that 1) creates a 3D panoramic map from stitched NFOV camera images, 2) correlates the areas of sparse LRF coverage to the map, and 3) generates attitude commands to close the coverage. This provides a consistent and reliable method to register positions of strike points relative to each other and to the NFOV image of the RSO with a priori knowledge of the RSO attitude. Gaps and sparse areas in LRF coverage are covered with strike points; the result is a point cloud of significantly higher resolution than that obtained with preprogrammed attitude profiles. The analysis bears particular relevance to power-constrained nanosatellite missions for space-based SSA for whom a multibeam LRF payload is not feasible. Maneuvers can now be designed on-line in real time; results presented validate the utility of a single-beam LRF as a tool for 3D imaging of RSO shapes.

This paper expands on previous studies by the authors into 3D imaging with a single-beam laser rangefinder (LRF) by implementing real-time attitude maneuvers of a chaser satellite flying in relative orbit around a resident space object (RSO). Point clouds generated with an LRF are much sparser than those generated with an imaging LIDAR, making it difficult to autonomously distinguish between gaps in coverage and truly empty space. Furthermore, if both the attitude and the shape of the target RSO are unknown, it is particularly difficult to register a collection of LRF strike points together and detect gaps in strike point coverage in realtime. This paper presents the incorporation of a narrow field of-view (NFOV) camera that detects the strike point on the RSO and supplements LRF distance measurements with image data. This data is used to generate attitude command profiles that efficiently fill LRF coverage gaps and generate high density point clouds, thus maximizing coverage of an unknown RSO. Results obtained so far point the way to a real-time implementation of the algorithm. A method to detect and close gaps in LRF strike point coverage is presented first. Coverage gap detection is achieved using Voronoi diagrams, where Voronoi cells are centered at the LRF strike points. A three-part algorithm is used that 1) creates a 3D panoramic map from “stitched” NFOV camera images; 2) correlates the areas of sparse LRF coverage to the map; and 3) generates attitude commands to close the coverage gaps. The map provides a consistent and reliable method to register positions of strike points relative to each other and to the NFOV image of the RSO without a priori knowledge of the RSO attitude. Using this algorithm, gaps and sparse areas in LRF coverage are covered with strike points, allowing for the generation of a higher-resolution point cloud than that obtained with preprogrammed attitude profiles. Attitude maneuvers can now be designed on-line in real-time such that they satisfy the constraints of the chaser spacecraft attitude determination and control system. Finally, the effectiveness of the camera-aided generation of attitude profiles is analyzed by using a weighted edge reconstruction metric, and comparing results to those generated with pre-programmed attitude maneuvers. The effect of on-line maneuver generation on the overall decrease of time and propellant expenditure to generate an adequate point cloud is also discussed. The analysis bears particular relevance to low-budget, nano-satellite demonstration missions for space-based space situational awareness (SSA).

Point clouds obtained from 3D sensors are usually sparse. Existing methods mainly focus on upsampling sparse point clouds in a supervised manner by using dense ground truth point clouds. In this paper, we propose a self-supervised point cloud upsampling network (SSPU-Net) to generate dense point clouds without using ground truth. To achieve this, we exploit the consistency between the input sparse point cloud and generated dense point cloud for the shapes and rendered images. Specifically, we first propose a neighbor expansion unit (NEU) to upsample the sparse point clouds, where the local geometric structures of the sparse point clouds are exploited to learn weights for point interpolation. Then, we develop a differentiable point cloud rendering unit (DRU) as an end-to-end module in our network to render the point cloud into multi-view images. Finally, we formulate a shape-consistent loss and an image-consistent loss to train the network so that the shapes of the sparse and dense point clouds are as consistent as possible. Extensive results on the CAD and scanned datasets demonstrate that our method can achieve impressive results in a self-supervised manner.

Three-dimensional (3D) object recognition is crucial for intelligent autonomous agents such as autonomous vehicles and robots alike to operate effectively in unstructured environments. Most state-of-art approaches rely on relatively dense point clouds and performance drops significantly for sparse point clouds. Unsupervised domain adaption allows to minimise the discrepancy between dense and sparse point clouds with minimal unlabelled sparse point clouds, thereby saving additional sparse data collection, annotation and retraining costs. In this work, we propose a novel method for point cloud based object recognition with competitive performance with state-of-art methods on dense and sparse point clouds while being trained only with dense point clouds.

In this study, we propose a high-throughput and low-cost automatic detection method based on deep learning to replace the inefficient manual counting of rapeseed siliques. First, a video is captured with a smartphone around the rapeseed plants in the silique stage. Feature point detection and matching based on SIFT operators are applied to the extracted video frames, and sparse point clouds are recovered using epipolar geometry and triangulation principles. The depth map is obtained by calculating the disparity of the matched images, and the dense point cloud is fused. The plant model of the whole rapeseed plant in the silique stage is reconstructed based on the structure-from-motion (SfM) algorithm, and the background is removed by using the passthrough filter. The downsampled 3D point cloud data is processed by the DGCNN network, and the point cloud is divided into two categories: sparse rapeseed canopy siliques and rapeseed stems. The sparse canopy siliques are then segmented from the original whole rapeseed siliques point cloud using the sparse-dense point cloud mapping method, which can effectively save running time and improve efficiency. Finally, Euclidean clustering segmentation is performed on the rapeseed canopy siliques, and the RANSAC algorithm is used to perform line segmentation on the connected siliques after clustering, obtaining the three-dimensional spatial position of each silique and counting the number of siliques. The proposed method was applied to identify 1457 siliques from 12 rapeseed plants, and the experimental results showed a recognition accuracy greater than 97.80%. The proposed method achieved good results in rapeseed silique recognition and provided a useful example for the application of deep learning networks in dense 3D point cloud segmentation.

Super resolution of laser range data based on image-guided fusion and dense matching

Abstract. Photogrammetric processing algorithms can suffer problems due to either the initial image quality (noise, low radiometric quality, shadows and so on) or to certain surface materials (shiny or textureless objects). This can result in noisy point clouds and/or difficulties in feature extraction. Specifically, dense point clouds which are generated with photogrammetric method using a lightweight thermal camera, are more noisy and sparse than the point clouds of high-resolution digital camera images. In this paper, new method which produces more reliable and dense thermal point cloud using the sparse thermal point cloud and high resolution digital point cloud was considered. Both thermal and digital images were obtained with UAS (Unmanned Aerial System) based lightweight Optris PI 450 and Canon EOS 605D camera images. Thermal and digital point clouds, and orthophotos were produced using photogrammetric methods. Problematic thermal point cloud was transformed to a high density thermal point cloud using image processing methods such as rasterizing, registering, interpolation and filling. The results showed that the obtained thermal point cloud - up to chosen processing parameters - was 87% more densify than the original point cloud. The second improvement was gained at the height accuracy of the thermal point cloud. New densified point cloud has more consistent elevation model while the original thermal point cloud shows serious deviations from the expected surface model.

Photogrammetric processing algorithms can suffer problems due to either the initial image quality (noise, low radiometric quality, shadows and so on) or to certain surface materials (shiny or textureless objects). This can result in noisy point clouds and/or difficulties in feature extraction. Specifically, dense point clouds which are generated with photogrammetric method using a lightweight thermal camera, are more noisy and sparse than the point clouds of high-resolution digital camera images. In this paper, new method which produces more reliable and dense thermal point cloud using the sparse thermal point cloud and high resolution digital point cloud was considered. Both thermal and digital images were obtained with UAS (Unmanned Aerial System) based lightweight Optris PI 450 and Canon EOS 605D camera images. Thermal and digital point clouds, and orthophotos were produced using photogrammetric methods. Problematic thermal point cloud was transformed to a high density thermal point cloud using image processing methods such as rasterizing, registering, interpolation and filling. The results showed that the obtained thermal point cloud - up to chosen processing parameters - was 87% more densify than the original point cloud. The second improvement was gained at the height accuracy of the thermal point cloud. New densified point cloud has more consistent elevation model while the original thermal point cloud shows serious deviations from the expected surface model.

In order to solve the problems of large memory consumption and large reconstruction error in the process of dense point cloud 3D reconstruction, an optimal reconstruction method of dense point cloud sampling and filtering is proposed. First of all, the feature points are extracted and matched, the mismatching is removed, the camera pose is calculated, and the sparse point cloud is generated. Then, the sparse point cloud is densified, and the improved sampling algorithm and filtering algorithm are used to reduce the memory and optimize the dense point cloud. Finally, the optimized dense point cloud is reconstructed and smoothed. The experimental results show that after the sampling algorithm, the computational memory required in the reconstruction process decreases from 5.3GB to 3.2GB, and the relative error of the optimized point cloud reconstruction is less than 4%, which is significantly lower than that before optimization. By comparison, it is found that the relative error is reduced by 1% to 3%. Experimental results verify the effectiveness of the method and show that the dense point cloud reconstruction with higher accuracy can be achieved with lower computational memory.

The fusion of laser point cloud and visual image depends on the point cloud density and the target framing effect, the traditional laser point cloud processing for sparse point cloud clustering effect is poor, it is difficult to frame small objects as well as medium and long distance objects. Then the subsequent sensor fusion is easy to miss the recognition of obstacles. In this paper, we improve the frame selection method for sparse point clouds, firstly build a deep learning framework pointpillar, use pointpillar to frame the sparse laser point clouds, then spatially calibrate the lidar coordinate system and camera coordinate system, project the lidar point clouds to the camera image, improve the late fusion method, effectively use the detection results of single sensor, and finally The late-fusion is performed with the target detection results of the camera image to output the exact distance as well as the category of the target. Experiments show that compared with the recognition effect of the traditional fusion algorithm, the number of frames is increased by 6 and the missed recognition rate is reduced from 31.41% to 12.31%.

In the study of Simultaneous Localization and Mapping (SLAM), the existence of dynamic obstacles will have a great impact on it, and when there are many dynamic obstacles, it will lead to great challenges in mapping. Therefore, segmenting dynamic objects in the environment is particularly important. The common data format in the field of autonomous robots is point clouds. How to use point clouds to segment dynamic objects is the focus of this study. The existing point clouds instance segmentation methods are mostly based on dense point clouds. In our application scenario, we use 16-line LiDAR (sparse point clouds) and propose a sparse point clouds instance segmentation method based on spatio-temporal encoding and decoding for autonomous robots in dynamic environments. Compared with other point clouds instance segmentation methods, the proposed algorithm has significantly improved average percision and average recall on instance segmentation of our point clouds dataset. In addition, the annotation of point clouds is time-consuming and laborious, and the existing dataset for point clouds instance segmentation is also very limited. Thus, we propose an autonomous point clouds annotation algorithm that integrates object tracking, segmentation, and point clouds to 2D mapping methods, the resulting data can then be used for training robust model.

A CCURATEmodeling of the differential translation and rotation between two spacecraft is essential for cooperative distributed space systems, spacecraft formation flying (SFF), rendezvous, and docking. High-fidelity relative motion modeling, as opposed to absolute motion modeling, is particularly important for autonomous missions [1]. Point-mass models for relative spacecraft translational motion have been extensively studied over the past 50 years, since Clohessy and Wiltshire (CW) presented a rendezvous model for a circular reference orbit and a spherical Earth [2]. Following the work of Clohessy and Wiltshire, variants on the point-mass model were developed, such as generalizations to elliptic reference orbits [3–5] and an oblate Earth [6,7]. The growing interest in SFF motivated the research of relative spacecraft motion modeling, yielding more accurate and complete equations and solutions for perturbed relative motion [8–10]. However, most of the works focused on point-mass, 3 degrees-offreedom (DOF) spacecraft. Obviously, performing a space mission that consists of several cooperative space vehicles requires modeling the relative rotational motion in addition to the relative translation, that is, 6-DOF models. Models for the relative motion of 6-DOF spacecraft have gained attention in the literature only in recent years. Among the first to suggest treating the spacecraft relative angular velocity in an SFF control problem were Pan and Kapila [11], who addressed the coupled translational and rotational dynamics of two spacecraft. By defining two body-fixed reference frames, one attached to the leader and the other attached to the follower, it was proposed [11] to use a two-part relative motion model: one that accounts for the relative translational dynamics of the body-fixed coordinate frame origins, and another that captures the relative attitude dynamics of the two body-fixed frames. A similar modeling approach was used for relative motion estimation [1]. In addition, tensorial equations of motion for a formation consisting ofN spacecraft, each modeled as a rigid body, were derived [12]. However, only the absolute equations of motion were developed [12]; a relative version of these equations was not given. Moreover, a clear mathematical relationship between the developed models and the traditional nonlinear point-mass relative motion and CW models was not provided. The coupling between the translational and rotational motion in the aforementioned models [1,11] was induced by gravity torques. The kinematic coupling, which is essentially a projection of the rotational motion about the center of mass (c.m.) onto the relative translational configuration space, was neglected. It is this kinematic coupling that the current paper is concerned with. In general, rigid-body dynamics can be represented as translation of the c.m. and rotation about the c.m. [13]. Thus, spacecraft relative motion must be composed by combining the relative translational and rotational dynamics of arbitrary points on the spacecraft. Whenever one of these points does not coincide with the spacecraft’s c.m., a kinematic coupling between the rotational and translational dynamics of these points is obtained. The purpose of this paper is to quantify the kinematic coupling effect and to show that this effect is key for high-precision modeling of tight SFF, rendezvous, and docking. This effect is also important in vision-based relative attitude and position control, where arbitrary feature points on a target vehicle are to be tracked. Given two rigidbody spacecraft, the model presented herein is formulated in a general manner that describes the motion between any two arbitrary points on the spacecraft. The relative translational motion is then generated by both the spacecraft orbitalmotion and the rotation about the c.m. In addition, this paper provides a CW-like approximation of the relative motion that includes the kinematic coupling. This new approximation is aimed at alleviating an apparent contradiction in linearized relative motion theories: to obtain linear equations of motion, the spacecraft are assumed to operate in close proximity. However, if the spacecraft are close to each other, then they can no longer be treated as point masses, because the spacecraft shape and size affects the relative translation between off-c.m. points. This effect is accentuated as the distances between spacecraft decrease. The remainder of this paper is organized as follows. First, a background on the relative position and attitude dynamics is given. Then, a new coupled relative spacecraft motion model is presented. The newly developed model is then examined in a simulation.

The recent popularization of airborne lidar scanners has provided a steady source of point cloud datasets containing the altitudes of bare earth surface and vegetation features as well as man-made structures. In contrast to terrestrial lidar, which produces dense point clouds of small areas, airborne laser sensors usually deliver sparse datasets that cover large municipalities. The latter are very useful in constructing digital representations of cities; however, reconstructing 3D building shapes from a sparse point cloud is a time-consuming process because automatic shape reconstruction methods work best with dense point clouds and usually cannot be applied for this purpose. Moreover, existing methods dedicated to reconstructing simplified 3D buildings from sparse point clouds are optimized for detecting simple building shapes, and they exhibit problems when dealing with more complex structures such as towers, spires, and large ornamental features, which are commonly found e.g., in buildings from the renaissance era. In the above context, this paper proposes a novel method of reconstructing 3D building shapes from sparse point clouds. The proposed algorithm has been optimized to work with incomplete point cloud data in order to provide a cost-effective way of generating representative 3D city models. The algorithm has been tested on lidar point clouds representing buildings in the city of Gdansk, Poland.

Three-dimensional Reconstruction has drawn much attention in computer vision. Generating a dense point cloud from a single image is a more challenging task. However, generating dense point clouds directly costs expensively in calculation and memory and may cause the network hard to train. In this work, we propose a two-stage training dense point cloud generation network. We first train our attention-based sparse point cloud generation network to generate a sparse point cloud from a single image. Then we train our dense point cloud generation network to densify the generated sparse point cloud. After combining the two stages and finetuning, we obtain an end-to-end network that generates a dense point cloud from a single image. Through evaluation of both synthetic and real-world datasets, we demonstrate that our approach outperforms state of the art works in dense point cloud generation. Our source code is available at https://github.com/VIM-Lab/AttentionDPCR .

Recently, point cloud technology has been applied in the ship engineering field. However, the dense point cloud acquired by terrestrial laser scanning (TLS) technology in ship engineering applications brings an obstacle to some powerful and advanced point-based deep learning point cloud processing methods. This paper presents a deep learning pre-procession module to ensure the feasibility of processing dense point clouds on commonly available computer devices. The pre-procession module is designed according to the traditional point cloud processing methods and the PointNet++ paradigm, and is evaluated on two ship structure datasets and two popular point cloud datasets. Experimental results illustrate that (i) the proposed module improves the performance of point-based deep learning semantic segmentation networks, and (ii) the proposed module empowers the existing point-based deep learning networks with the capability to process dense input point clouds. The proposed module may provide a useful semantic segmentation tool for realistic dense point clouds in various industrial applications.