3D Point Cloud Reconstruction Research Articles

Weakly-Supervised Single-view Dense 3D Point Cloud Reconstruction via Differentiable Renderer

In recent years, addressing ill-posed problems by leveraging prior knowledge contained in databases on learning techniques has gained much attention. In this paper, we focus on complete three-dimensional (3D) point cloud reconstruction based on a single red-green-blue (RGB) image, a task that cannot be approached using classical reconstruction techniques. For this purpose, we used an encoder-decoder framework to encode the RGB information in latent space, and to predict the 3D structure of the considered object from different viewpoints. The individual predictions are combined to yield a common representation that is used in a module combining camera pose estimation and rendering, thereby achieving differentiability with respect to imaging process and the camera pose, and optimization of the two-dimensional prediction error of novel viewpoints. Thus, our method allows end-to-end training and does not require supervision based on additional ground-truth (GT) mask annotations or ground-truth camera pose annotations. Our evaluation of synthetic and real-world data demonstrates the robustness of our approach to appearance changes and self-occlusions, through outperformance of current state-of-the-art methods in terms of accuracy, density, and model completeness.

Geometric Modeling of Rosa roxburghii Fruit Based on Three-Dimensional Point Cloud Reconstruction

Fruit three-dimensional (3D) model is crucial to estimating its geometrical and mechanical properties and improving the level of fruit mechanical processing. Considering the complex geometrical features and the required model accuracy, this paper proposed a 3D point cloud reconstruction method for the Rosa roxburghii fruit based on a three-dimensional laser scanner, including 3D point cloud generation, point cloud registration, fruit thorns segmentation, and 3D reconstruction. The 3D laser scanner was used to obtain the original 3D point cloud data of the Rosa roxburghii fruit, and then the fruit thorns data were removed by the segmentation algorithm combining the statistical outlier removal and radius outlier removal. By analyzing the effects of five-point cloud simplification methods, the optimal simplification method was determined. The Poisson reconstruction algorithm, the screened Poisson reconstruction algorithm, the greedy projection triangulation algorithm, and the Delaunay triangulation algorithm were utilized to reconstruct the fruit model. The number of model vertices, the number of facets, and the relative volume error were used to determine the best reconstruction algorithm. The results indicated that this model can better reconstruct the actual surface of Rosa roxburghii fruit. The method provides a reference for the related application.

3D Point Cloud Reconstruction Using Inversely Mapping and Voting from Single Pass CSAR Images

3D reconstruction has raised much interest in the field of CSAR. However, three dimensional imaging results with single pass CSAR data reveals that the 3D resolution of the system is poor for anisotropic scatterers. According to the imaging mechanism of CSAR, different targets located on the same iso-range line in the zero doppler plane fall into the same cell while for the same target point, imaging point will fall into the different positions at different aspect angles. In this paper, we proposed a method for 3D point cloud reconstruction using projections on 2D sub-aperture images. The target and background in the sub-aperture images are separated and binarized. For a projection point of target, given a series of offsets, the projection point will be mapped inversely to the 3D mesh along the iso-range line. We can obtain candidate points of the target. The intersection of iso-range lines can be regarded as voting process. For a candidate, the more times of intersection, the higher the number of votes, and the candidate point will be reserved. This fully excavates the information contained in the angle dimension of CSAR. The proposed approach is verified by the Gotcha Volumetric SAR Data Set.

3D-ARNet: An accurate 3D point cloud reconstruction network from a single-image

3D-ARNet: An accurate 3D point cloud reconstruction network from a single-image

Unmanned aerial vehicle and artificial intelligence revolutionizing efficient and precision sustainable forest management

The ecological value of tropical forests in water conservation district has been of great interest because of their rich vegetation types and higher biomass density than any other land cover types, it is urgent to evaluate the ecological value of tropical forests in water conservation district. However, the monitoring of tropical forests in water conservation district is faced with many problems, such as high forest density, complexity and diversity of the forest structure, complex topography and climate conditions, and the difficulty of access for investigators. In order to solve the above difficulties, this study combined 3D point cloud reconstruction based on Unmanned Aerial Vehicle - Structure from Motion (UAV-SfM) technology with forest type classification based on the Convolutional Neural Network (CNN) method, combined with a small amount of forest permanent sample plot survey data, to accurately evaluate the forest biomass distribution and forest biodiversity in water conservation district. The results show that the overall classification accuracy of the 20 forest types in water conservation district based on the CNN method is 0.61, the overall Kappa coefficient is 0.59, and the conditional Kappa coefficient is concentrated in the range of 0.43–0.85. The Root Mean Square Error (RMSE) of the plane measurement of UAV-SfM technology is 0.432 m, and the RMSE of the elevation measurement is 0.989 m, the effect of this UAV technology in tropical forest monitoring is superior. Using the techniques mentioned above, this study can effectively and accurately monitor and evaluate the biomass distribution and biodiversity of tropical forests in the water conservation district. Based on the precision forest ecological monitoring data, this study can develop a scientific and reasonable sustainable forest management plan for the water conservation district according to the distribution of forest biomass and biodiversity. The combination of UAV-SfM technology and the CNN method is an innovative attempt, and the integration of UAV and artificial intelligence technology solves practical problems faced by sustainable forest management. UAV and artificial intelligence will also provide an important foundation for forest ecological environment sustainability assessment research.

Recovering dense 3D point clouds from single endoscopic image

Background and objective: Recovering high-quality 3D point clouds from monocular endoscopic images is a challenging task. This paper proposes a novel deep learning-based computational framework for 3D point cloud reconstruction from single monocular endoscopic images.Methods: An unsupervised mono-depth learning network is used to generate depth information from monocular images. Given a single mono endoscopic image, the network is capable of depicting a depth map. The depth map is then used to recover a dense 3D point cloud. A generative Endo-AE network based on an auto-encoder is trained to repair defects of the dense point cloud by generating the best representation from the incomplete data. The performance of the proposed framework is evaluated against state-of-the-art learning-based methods. The results are also compared with non-learning based stereo 3D reconstruction algorithms.Results: Our proposed methods outperform both the state-of-the-art learning-based and non-learning based methods for 3D point cloud reconstruction. The Endo-AE model for point cloud completion can generate high-quality, dense 3D endoscopic point clouds from incomplete point clouds with holes. Our framework is able to recover complete 3D point clouds with the missing rate of information up to 60%. Five large medical in-vivo databases of 3D point clouds of real endoscopic scenes have been generated and two synthetic 3D medical datasets are created. We have made these datasets publicly available for researchers free of charge.Conclusions: The proposed computational framework can produce high-quality and dense 3D point clouds from single mono-endoscopy images for augmented reality, virtual reality and other computer-mediated medical applications.

Fusion Key Frame Image Confidence Assessment of the Medical Service Robot Whole Scene Reconstruction

Facing the precise service and emergency rescue needs of medical service robots in irregular scene, in order to achieve better navigation and path planning for robots in service scenarios, for the whole reconstruction of the absolute scale service scenario, this article proposes a frame of whole scene three-dimensional (3D) point cloud reconstruction based on the fusion of scene depth estimation, confidence assessment, and pose tracking with monocular camera. The algorithm first collects the scene focus stack images under an initial viewing angle through the robot mobile terminal of camera. The absolute depth information of the scene is estimated on the server side, and the confidence level of the reconstructed image of the point cloud is evaluated, and non- uniform sampling is performed to reduce the influence of the error estimation. Based on the sparse key frame position information defined by monocular SLAM, the 3D reconstruction of the whole scene in absolute scale is realized through multi-perspective point cloud pose matching. It provides information of cloud reconstruction of scenic spots for target recognition and navigation of a medical service robot.

Visual Reconstruction and Localization-Based Robust Robotic 6-DoF Grasping in the Wild

The intelligent grasping expects that the manipulator has the ability to grasp objects with high degree of freedom in a wild (unstructured) environment. Due to low perception ability in handing targets and environments, most industrial robots are limited to top-down 4-DoF grasping. In this work, we propose a novel low-cost coarse to fine robotic grasping framework. First, we design a global localization based environment perception method, which enables the manipulator to roughly and automatically locate work space. Then, constrained by the above initial localization, a 3D point cloud reconstruction based 6-DoF pose estimation method is proposed for the manipulator further fine locating grasping target. Finally, our framework realizes full function of visual 6DoF robotic grasping, which includes two different visual servoing and grasp planning strategies for different objects grasping. Meanwhile, it also can integrate various state-of-arts 6DoF pose estimation algorithms to facilitate various practical grasping applications or researches. Experimental results show that our method achieves autonomous robotic grasping with high degree of freedom in an unknown environment. Especially for objects with occlusion, singular shape or small scale, our method can still maintain robust grasping.

Model-Based Joint Bit Allocation Between Geometry and Color for Video-Based 3D Point Cloud Compression

Rate distortion optimization plays a very important role in image/video coding. But for 3D point cloud, this problem has not been investigated. In this paper, the rate and distortion characteristics of 3D point cloud are investigated in detail, and a typical and challenging rate distortion optimization problem is solved for 3D point cloud. Specifically, since the quality of the reconstructed 3D point cloud depends on both the geometry and color distortions, we first propose analytical rate and distortion models for the geometry and color information in video-based 3D point cloud compression platform, and then solve the joint bit allocation problem for geometry and color based on the derived models. To maximize the reconstructed quality of 3D point cloud, the bit allocation problem is formulated as a constrained optimization problem and solved by an interior point method. Experimental results show that the rate-distortion performance of the proposed solution is close to that obtained with exhaustive search but at only 0.68% of its time complexity. Moreover, the proposed rate and distortion models can also be used for the other rate-distortion optimization problems (such as prediction mode decision) and rate control technologies for 3D point cloud coding in the future.

A Method for 3D Reconstruction of the Ming and Qing Official-Style Roof Using a Decorative Components Template Library

The ancient roof decorative components of the official-style architectures from the Ming and Qing dynasties in China hold both physical and symbolic significance. These roof structures are the essential objects in three-dimensional (3D) modeling of ancient architectures for traditional Chinese cultural preservation. Although ancient architectures can be surveyed by a 3D laser scanner, the complex geometry and diverse pattern of their roof decorative components make the 3D point cloud reconstruction challenging, or at some points, nearly impossible in a fully automated manner. In this paper, we propose a method to ensure that the 3D shape of each roof decorative component is accurately modeled. First, we establish a decorative components template library (or “template library” in short hereafter), which is the first of its kind for the roofs of Ming and Qing official-style architectures. The process of establishing the decorative components template library begins with a remote collection of survey data using a terrestrial laser scanner and digital camera. The next stage involves the design and construction of different 3D decorative components in the template library with reference to the manuscripts written in the Ming and Qing dynasties’ architectural pattern books. With the point cloud data collected on any Ming and Qing official-style architecture, we further propose a geo-registration mechanism to search for an optimal fitting of the decorative components from the template library on the collected point cloud automatically. Based on the experimental results, the accuracy of point cloud registration yields less than 0.02 m, which meets the accuracy of the 3D model at LoD 300 level. Time consumption is less than 5s and stable, for large volume computing capacity has good robustness. The proposed strategy provides a new way for the 3D modeling of large and clustered historical architectures, particularly with complex structures.

Body2vec: 3D Point Cloud Reconstruction for Precise Anthropometry with Handheld Devices.

Current point cloud extraction methods based on photogrammetry generate large amounts of spurious detections that hamper useful 3D mesh reconstructions or, even worse, the possibility of adequate measurements. Moreover, noise removal methods for point clouds are complex, slow and incapable to cope with semantic noise. In this work, we present body2vec, a model-based body segmentation tool that uses a specifically trained Neural Network architecture. Body2vec is capable to perform human body point cloud reconstruction from videos taken on hand-held devices (smartphones or tablets), achieving high quality anthropometric measurements. The main contribution of the proposed workflow is to perform a background removal step, thus avoiding the spurious points generation that is usual in photogrammetric reconstruction. A group of 60 persons were taped with a smartphone, and the corresponding point clouds were obtained automatically with standard photogrammetric methods. We used as a 3D silver standard the clean meshes obtained at the same time with LiDAR sensors post-processed and noise-filtered by expert anthropological biologists. Finally, we used as gold standard anthropometric measurements of the waist and hip of the same people, taken by expert anthropometrists. Applying our method to the raw videos significantly enhanced the quality of the results of the point cloud as compared with the LiDAR-based mesh, and of the anthropometric measurements as compared with the actual hip and waist perimeter measured by the anthropometrists. In both contexts, the resulting quality of body2vec is equivalent to the LiDAR reconstruction.

An automatic electronic instrument for accurate measurements of food volume and density.

Accurate measurements of food volume and density are often required as 'gold standards' for calibration of image-based dietary assessment and food database development. Currently, there is no specialised laboratory instrument for these measurements. We present the design of a new volume of density (VD) meter to bridge this technological gap. Our design consists of a turntable, a load sensor, a set of cameras and lights installed on an arc-shaped stationary support, and a microcomputer. It acquires an array of food images, reconstructs a 3D volumetric model, weighs the food and calculates both food volume and density, all in an automatic process controlled by the microcomputer. To adapt to the complex shapes of foods, a new food surface model, derived from the electric field of charged particles, is developed for 3D point cloud reconstruction of either convex or concave food surfaces. We conducted two experiments to evaluate the VD meter. The first experiment utilised computer-synthesised 3D objects with prescribed convex and concave surfaces of known volumes to investigate different food surface types. The second experiment was based on actual foods with different shapes, colours and textures. Our results indicated that, for synthesised objects, the measurement error of the electric field-based method was <1 %, significantly lower compared with traditional methods. For real-world foods, the measurement error depended on the types of food volumes (detailed discussion included). The largest error was approximately 5 %. The VD meter provides a new electronic instrument to support advanced research in nutrition science.

A morphing-Based 3D point cloud reconstruction framework for medical image processing

Background and Objective: In the virtual surgery simulation system, the reconstruction of a highly precise soft tissue 3D model is an effective method to improve the user’s visual telepresence. However, the traditional point cloud generation method based on subdivision and filling is unsatisfactory due to its low accuracy and slow speed.Methods: To address this problem, we present a novel 3D point cloud reconstructing model based on Morphing. The 3D surface model of soft tissue (live) is obtained from a series of 2D CT images using Mimics. The 3D voxel model of soft tissue is reconstructed through a sequential change of the 3D surface model by utilizing Morphing. A nonlinear interpolation method is used to fit the irregular shape of the model and improve simulation accuracy.Results: The point cloud model builds from discrete points, avoiding the problems of instability and computational complexity, which are inherent in both the surface and volume models for soft tissue. Compared with the volumetric subdividing and voxel filling method, the simulation results show that the 3D cloud model reconstructed based on Morphing is more fast, accurate and consistent with the real soft tissue.Conclusions: The simulating experiment of soft tissue deformation using 3D point cloud model which reconstructed using moprhing proved our method is effective and correct.

Scene Target 3D Point Cloud Reconstruction Technology Combining Monocular Focus Stack and Deep Learning

In order to obtain the depth information of the target in the scene and realize three-dimensional (3D) reconstruction, in this paper, a target reconstruction method combining monocular focus stack image and deep neural network is proposed. This method makes full use of the advantages of light field imaging technology and can generate the all focus image. The method first collects multiple frames of continuous images at different focal lengths of the scene, using a divide and conquer algorithm strategy, uplink uses YOLO neural network to identify the target in 3D space and track the position information; the downlink reconstructs the four-dimensional (4D) light field data based on the focus stack image frequency domain back projection, and then uses light field imaging technology to invert the scene parallax; subsequently, achieve scene depth estimation and reconstruction of all focus image; finally, the uplink and downlink are merged to realize the reconstruction of the 3D point cloud of the space target. Experimental results on real scenes show the effectiveness of the proposed algorithm.

Towards Intelligent Retail: Automated on-Shelf Availability Estimation Using a Depth Camera

Efficient management of on-shelf availability and inventory is a key issue to achieve customer satisfaction and reduce the risk of profit loss for both retailers and manufacturers. Conventional store audits based on physical inspection of shelves are labor-intensive and do not provide reliable assessment. This paper describes a novel framework for automated shelf monitoring, using a consumer-grade depth sensor. The aim is to develop a low-cost embedded system for early detection of out-of-stock situations with particular regard to perishable goods stored in countertop shelves, refrigerated counters, baskets or crates. The proposed solution exploits 3D point cloud reconstruction and modelling techniques, including surface fitting and occupancy grids, to estimate product availability, based on the comparison between a reference model of the shelf and its current status. No a priori knowledge about the product type is required, while the shelf reference model is automatically learnt based on an initial training stage. The output of the system can be used to generate alerts for store managers, as well as to continuously update product availability estimates for automated stock ordering and replenishment and for e-commerce apps. Experimental tests performed in a real retail environment show that the proposed system is able to estimate the on-shelf availability percentage of different fresh products with a maximum average discrepancy with respect to the actual one of about 5.0%.

Deep Unsupervised Learning of 3D Point Clouds via Graph Topology Inference and Filtering.

We propose a deep autoencoder with graph topology inference and filtering to achieve compact representations of unorganized 3D point clouds in an unsupervised manner. Many previous works discretize 3D points to voxels and then use lattice-based methods to process and learn 3D spatial information; however, this leads to inevitable discretization errors. In this work, we try to handle raw 3D points without such compromise. The proposed networks follow the autoencoder framework with a focus on designing the decoder. The encoder of the proposed networks adopts similar architectures as in PointNet, which is a well-acknowledged method for supervised learning of 3D point clouds. The decoder of the proposed networks involves three novel modules: the folding module, the graph-topology-inference module, and the graph-filtering module. The folding module folds a canonical 2D lattice to the underlying surface of a 3D point cloud, achieving coarse reconstruction; the graph-topology-inference module learns a graph topology to represent pairwise relationships between 3D points, pushing the latent code to preserve both coordinates and pairwise relationships of points in 3D point clouds; and the graph-filtering module couples the above two modules, refining the coarse reconstruction through a learnt graph topology to obtain the final reconstruction. The proposed decoder leverages a learnable graph topology to push the codeword to preserve representative features and further improve the unsupervised-learning performance. We further provide theoretical analyses of the proposed architecture. We provide an upper bound for the reconstruction loss and further show the superiority of graph smoothness over spatial smoothness as a prior to model 3D point clouds. In the experiments, we validate the proposed networks in three tasks, including 3D point cloud reconstruction, visualization, and transfer classification. The experimental results show that (1) the proposed networks outperform the state-of-the-art methods in various tasks, including reconstruction and transfer classification; (2) a graph topology can be inferred as auxiliary information without specific supervision on graph topology inference; (3) graph filtering refines the reconstruction, leading to better performances; and (4) designing a powerful decoder could improve the unsupervised-learning performance, just like a powerful encoder.

Three-Dimensional Morphological Measurement Method for a Fruit Tree Canopy Based on Kinect Sensor Self-Calibration

Perception of the fruit tree canopy is a vital technology for the intelligent control of a modern standardized orchard. Due to the complex three-dimensional (3D) structure of the fruit tree canopy, morphological parameters extracted from two-dimensional (2D) or single-perspective 3D images are not comprehensive enough. Three-dimensional information from different perspectives must be combined in order to perceive the canopy information efficiently and accurately in complex orchard field environment. The algorithms used for the registration and fusion of data from different perspectives and the subsequent extraction of fruit tree canopy related parameters are the keys to the problem. This study proposed a 3D morphological measurement method for a fruit tree canopy based on Kinect sensor self-calibration, including 3D point cloud generation, point cloud registration and canopy information extraction of apple tree canopy. Using 32 apple trees (Yanfu 3 variety) morphological parameters of the height (H), maximum canopy width (W) and canopy thickness (D) were calculated. The accuracy and applicability of this method for extraction of morphological parameters were statistically analyzed. The results showed that, on both sides of the fruit trees, the average relative error (ARE) values of the morphological parameters including the fruit tree height (H), maximum tree width (W) and canopy thickness (D) between the calculated values and measured values were 3.8%, 12.7% and 5.0%, respectively, under the V1 mode; the ARE values under the V2 mode were 3.3%, 9.5% and 4.9%, respectively; and the ARE values under the V1 and V2 merged mode were 2.5%, 3.6% and 3.2%, respectively. The measurement accuracy of the tree width (W) under the double visual angle mode had a significant advantage over that under the single visual angle mode. The 3D point cloud reconstruction method based on Kinect self-calibration proposed in this study has high precision and stable performance, and the auxiliary calibration objects are readily portable and easy to install. It can be applied to different experimental scenes to extract 3D information of fruit tree canopies and has important implications to achieve the intelligent control of standardized orchards.

3D Point Cloud Reconstruction Using Structure from Motion with Sequence Images in Hadoop

3D Point Cloud Reconstruction Using Structure from Motion with Sequence Images in Hadoop

Three-Dimensional Point Cloud Reconstruction and Morphology Measurement Method for Greenhouse Plants Based on the Kinect Sensor Self-Calibration

Plant morphological data are an important basis for precision agriculture and plant phenomics. The three-dimensional (3D) geometric shape of plants is complex, and the 3D morphology of a plant changes relatively significantly during the full growth cycle. In order to make high-throughput measurements of the 3D morphological data of greenhouse plants, it is necessary to frequently adjust the relative position between the sensor and the plant. Therefore, it is necessary to frequently adjust the Kinect sensor position and consequently recalibrate the Kinect sensor during the full growth cycle of the plant, which significantly increases the tedium of the multiview 3D point cloud reconstruction process. A high-throughput 3D rapid greenhouse plant point cloud reconstruction method based on autonomous Kinect v2 sensor position calibration is proposed for 3D phenotyping greenhouse plants. Two red–green–blue–depth (RGB-D) images of the turntable surface are acquired by the Kinect v2 sensor. The central point and normal vector of the axis of rotation of the turntable are calculated automatically. The coordinate systems of RGB-D images captured at various view angles are unified based on the central point and normal vector of the axis of the turntable to achieve coarse registration. Then, the iterative closest point algorithm is used to perform multiview point cloud precise registration, thereby achieving rapid 3D point cloud reconstruction of the greenhouse plant. The greenhouse tomato plants were selected as measurement objects in this study. Research results show that the proposed 3D point cloud reconstruction method was highly accurate and stable in performance, and can be used to reconstruct 3D point clouds for high-throughput plant phenotyping analysis and to extract the morphological parameters of plants.

CAPNet: Continuous Approximation Projection for 3D Point Cloud Reconstruction Using 2D Supervision

Knowledge of 3D properties of objects is a necessity in order to build effective computer vision systems. However, lack of large scale 3D datasets can be a major constraint for datadriven approaches in learning such properties. We consider the task of single image 3D point cloud reconstruction, and aim to utilize multiple foreground masks as our supervisory data to alleviate the need for large scale 3D datasets. A novel differentiable projection module, called ‘CAPNet’, is introduced to obtain such 2D masks from a predicted 3D point cloud. The key idea is to model the projections as a continuous approximation of the points in the point cloud. To overcome the challenges of sparse projection maps, we propose a loss formulation termed ‘affinity loss’ to generate outlierfree reconstructions. We significantly outperform the existing projection based approaches on a large-scale synthetic dataset. We show the utility and generalizability of such a 2D supervised approach through experiments on a real-world dataset, where lack of 3D data can be a serious concern. To further enhance the reconstructions, we also propose a test stage optimization procedure to obtain reconstructions that display high correspondence with the observed input image.