Point Cloud Reconstruction Research Articles

Surface Defect Detection of Cabbage Based on Curvature Features of 3D Point Cloud

The dents and cracks of cabbage caused by mechanical damage during transportation have a direct impact on both commercial value and storage time. In this study, a method for surface defect detection of cabbage is proposed based on the curvature feature of the 3D point cloud. First, the red-green-blue (RGB) images and depth images are collected using a RealSense-D455 depth camera for 3D point cloud reconstruction. Then, the region of interest (ROI) is extracted by statistical filtering and Euclidean clustering segmentation algorithm, and the 3D point cloud of cabbage is segmented from background noise. Then, the curvature features of the 3D point cloud are calculated using the estimated normal vector based on the least square plane fitting method. Finally, the curvature threshold is determined according to the curvature characteristic parameters, and the surface defect type and area can be detected. The flat-headed cabbage and round-headed cabbage are selected to test the surface damage of dents and cracks. The test results show that the average detection accuracy of this proposed method is 96.25%, in which, the average detection accuracy of dents is 93.3% and the average detection accuracy of cracks is 96.67%, suggesting high detection accuracy and good adaptability for various cabbages. This study provides important technical support for automatic and non-destructive detection of cabbage surface defects.

Linear Laser Scanning Measurement Method Tracking by a Binocular Vision

The 3D scanning of a freeform structure relies on the laser probe and the localization system. The localization system, determining the effect of the point cloud reconstruction, will generate positioning errors when the laser probe works in complex paths with a fast speed. To reduce the errors, in this paper, a linear laser scanning measurement method is proposed based on binocular vision calibration. A simple and effective eight-point positioning marker attached to the scanner is proposed to complete the positioning and tracking procedure. Based on this, the method of marked point detection based on image moment and the principle of global coordinate system calibration are introduced in detail. According to the invariance principle of space distance, the corresponding points matching method between different coordinate systems is designed. The experimental results show that the binocular vision system can complete localization under different light intensities and complex environments, and that the repeated translation error of the binocular vision system is less than 0.22 mm, while the rotation error is less than 0.15°. The repeated error of the measurement system is less than 0.36 mm, which can meet the requirements of the 3D shape measurement of the complex workpiece.

Multi-view real-time acquisition and 3D reconstruction of point clouds for beef cattle

Body size, weight, and body condition score parameters are key indicators for monitoring cattle growth and they can be utilized to predict beef cattle yield and evaluate economic traits. However, it is easy to lay intense stress on cattle while measuring livestock’s body size manually, also along with giving negative effects on their feeding and weight gain. To resolve this problem, we design a real-time point cloud collection system for beef cattle with five depth cameras on a gantry structure. We developed point cloud preprocessing, registration, and 3D reconstruction algorithms, and quantitatively estimated the influence of light intensity during point cloud collection. The algorithms perform point cloud filtering, registration, segmentation, down-sampling, 3D reconstruction of the global point cloud, and target recognition. The maximum uncertainty of the calculated body width and length is 20 mm, and the acquisition time is within 0.08 s. We established a real-time system for 3D cattle point cloud- collection, which involves no stress on cattle when measuring. The point cloud collected by the system can provide technical support for the automatic extraction of key features during livestock body measurements.

Estimating Tree Defects with Point Clouds Developed from Active and Passive Sensors

Traditional inventories require large investments of resources and a trained workforce to measure tree sizes and characteristics that affect wood quality and value, such as the presence of defects and damages. Handheld light detection and ranging (LiDAR) and photogrammetric point clouds developed using Structure from Motion (SfM) algorithms achieved promising results in tree detection and dimensional measurements. However, few studies have utilized handheld LiDAR or SfM to assess tree defects or damages. We used a Samsung Galaxy S7 smartphone camera to photograph trees and create digital models using SfM, and a handheld GeoSLAM Zeb Horizon to create LiDAR point cloud models of some of the main tree species from the Pacific Northwest. We compared measurements of damage count and damage length obtained from handheld LiDAR, SfM photogrammetry, and traditional field methods using linear mixed-effects models. The field method recorded nearly twice as many damages per tree as the handheld LiDAR and SfM methods, but there was no evidence that damage length measurements varied between the three survey methods. Lower damage counts derived from LiDAR and SfM were likely driven by the limited point cloud reconstructions of the upper stems, as usable tree heights were achieved, on average, at 13.6 m for LiDAR and 9.3 m for SfM, even though mean field-measured tree heights was 31.2 m. Our results suggest that handheld LiDAR and SfM approaches show potential for detection and measurement of tree damages, at least on the lower stem.

Cross-view SLAM solver: Global pose estimation of monocular ground-level video frames for 3D reconstruction using a reference 3D model from satellite images

Accurate pose estimation of monocular ground-level images with respect to a satellite/aerial photogrammetric dataset is an extremely challenging task. Existing solutions often perform an offline post-registration on 3D results from both sources, which however, suffers from non-rigid geometric distortions of the 3D monocular reconstruction and the lack of overlaps between the air to ground content. This paper provides an online solution that performs accurate pose estimation of the ground images with respect to a 3D model derived from satellite images, followed by a dense 3D reconstruction. Our solution takes advantage of the simultaneous localization and mapping (SLAM) paradigm to dynamically incorporate reference observations from the satellite 3D model during the incremental pose estimation, called a cross-view SLAM solver, which leverages both ground-to-satellite error and image-level reprojection errors at the frame level to yield image poses that are well-registered to the satellite 3D model for facade point cloud reconstruction. This process also has the advantage of correcting non-rigid distortions and trajectory drifts that are often presented in monocular SLAM systems. In addition, our solution leverages both the geometric and semantic information from the satellite model and ground images to perform a per-frame correction for frame-level pose initialization, in which a novel scheme called pose buffer is introduced to initialize the pose of each keyframe through robust visual hull alignment of ground objects. The proposed approach has been experimented using four trajectories of monocular videos collections (around 7,000 frames per trajectory on average) and a 3D semantic model from multi-view satellite images to estimate the poses of the video frames and yield point clouds consistent with the satellite 3D models, evaluated by using LiDAR ground-truth. Both qualitative and quantitative experiments demonstrate that our solution yields accurate, drift-free poses and point clouds consistent with the satellite data and visually much more pleasing 3D models with facade information. Compared to the LiDAR ground-truth, the derived 3D models with ground-level images have achieved a mean absolute error of 1.78 m (improved from 3.15 m achieved using SLAM without utilizing satellite 3D models) (A testing program will be made available through https://github.com/GDAOSU/Cross-View-SLAM).

Hierarchical Point Cloud Encoding and Decoding With Lightweight Self-Attention Based Model

In this letter we present SA-CNN, a hierarchical and lightweight self-attention based encoding and decoding architecture for representation learning of point cloud data. The proposed SA-CNN introduces convolution and transposed convolution stacks to capture and generate contextual information among unordered 3D points. Following conventional hierarchical pipeline, the encoding process extracts feature in local-to-global manner, while the decoding process generates feature and point cloud in coarse-to-fine, multi-resolution stages. We demonstrate that SA-CNN is capable of a wide range of applications, namely classification, part segmentation, reconstruction, shape retrieval, and unsupervised classification. While achieving the state-of-the-art or comparable performance in the benchmarks, SA-CNN maintains its model complexity several order of magnitude lower than the others. In term of qualitative results, we visualize the multi-stage point cloud reconstructions and latent walks on rigid objects as well as deformable non-rigid human and robot models.

CAD-Aided 3D Reconstruction of Intelligent Manufacturing Image Based on Time Series

To improve the three-dimensional (3D) reconstruction effect of intelligent manufacturing image and reduce the reconstruction time, a new CAD-aided 3D reconstruction of intelligent manufacturing image based on time series was proposed. Kinect sensor is used to collect depth image data and convert it into 3D point cloud coordinates. The collected point cloud data are divided into regions, and different point cloud denoising algorithms are used to filter and denoise the divided regions. With the help of CAD, FLANN matching algorithm is used to extract feature points of time-series images and complete image matching. Three-dimensional reconstruction of sparse point cloud and dense point cloud is carried out to complete 3D reconstruction of intelligent manufacturing images. The experimental results show that the image PSNR of this method is always above 52 dB, and the maximum reconstruction time is 4.9 s. The 3D reconstruction effect of intelligent manufacturing image is better, and it has higher practical application value.

Assessment of Canopy Health with Drone-Based Orthoimagery in a Southern Appalachian Red Spruce Forest

Consumer-grade drone-produced digital orthoimagery is a valuable tool for conservation management and enables the low-cost monitoring of remote ecosystems. This study demonstrates the applicability of RGB orthoimagery for the assessment of forest health at the scale of individual trees in a 46-hectare plot of rare southern Appalachian red spruce forest on Whitetop Mountain, Virginia. We used photogrammetric Structure from Motion software Pix4Dmapper with drone-collected imagery to generate a mosaic for point cloud reconstruction and orthoimagery of the plot. Using 3-band RBG digital orthoimagery, we visually classified 9402 red spruce individuals, finding 8700 healthy (92.5%), 251 declining/dying (2.6%), and 451 dead (4.8%). We mapped individual spruce trees in each class and produced kernel density maps of health classes (live, dead, and dying). Our approach provided a nearly gap-free assessment of the red spruce canopy in our study site, versus a much more time-intensive field survey. Our maps provided useful information on stand mortality patterns and canopy gaps that could be used by managers to identify optimal locations for selective thinning to facilitate understory sapling regeneration. This approach, dependent mainly on an off-the-shelf drone system and visual interpretation of orthoimagery, could be applied by land managers to measure forest health in other spruce, or possibly spruce-fir, communities in the Appalachians. Our study highlights the usefulness of drone-produced orthoimagery for conservation monitoring, presenting a valid and accessible protocol for the monitoring and assessment of forest health in remote spruce, and possibly other conifer, populations. Adoption of drone-based monitoring may be especially useful in light of climate change and the possible displacement of southern Appalachian red spruce (and spruce-fir) ecosystems by the upslope migration of deciduous trees.

Traffic Sign Three‐Dimensional Reconstruction Based on Point Clouds and Panoramic Images

AbstractTraffic signs are a very important source of information for drivers and pilotless automobiles. With the advance of Mobile LiDAR System (MLS), massive point clouds have been applied in three‐dimensional digital city modelling. However, traffic signs in MLS point clouds are low density, colourless and incomplete. This paper presents a new method for the reconstruction of vertical rectangle traffic sign point clouds based on panoramic images. In this method, traffic sign point clouds are extracted based on arc feature and spatial semantic features analysis. Traffic signs in images are detected by colour and shape features and a convolutional neural network. Traffic sign point cloud and images are registered based on outline features. Finally, traffic sign points match traffic sign pixels to reconstruct the traffic sign point cloud. Experimental results have demonstrated that this proposed method can effectively obtain colourful and complete traffic sign point clouds with high resolution.

LFPNet: Lightweight network on real point sets for fruit classification and segmentation

• A lightweight fruit phenotypic processing network is proposed to solve the complex calculation of the fruit features recognition in real scenes. • By combining 3D-SPP and global feature feedback, a novel fruit segmentation network is designed to improve the segmentation integrity. • Based on preprocessing and 3D STN, LFPNet is proved to be generalization in directly processing the disordered fruit scenes without ideal labeled. • Our LFPNet has only 1.17 M parameters and improves 2.15% on mIoU, 3.175% on OA, 3.45% on mAcc. 3D point cloud reconstruction, as the key technology to obtain high-throughput fruit phenotypic data, has solved the problems caused by complex environments, high fruit similarity, and the lack of public datasets suitable for fruit characterization. However, in the process of identifying and segmenting fruit data from point cloud, the existing network architectures lead to problems such as classification error, incomplete segmentation and low efficiency. In this paper, we introduce LFPNet, a novel and efficient lightweight neural network that directly consumes fruit point clouds in the real scene. Our network mainly has the following three advantages: 1) The introduction of voxel-filter based down-sampling preprocessing can help to avoid classification error caused by invalid noise interference. 2) A 3D STN is designed to solve the lack of spatial invariance in convolutional neural network (CNN) when calculating and analyzing fruit point clouds. 3) By introducing spatial pyramid pooling and combining local and global features, a fruit segmentation network is built to improve the integrity of segmentation in fruit scenes. Experimental results show that our LFPNet performs as well as or better than most of its peers in terms of classification accuracy and segmentation integrity.

Efficient Lightweight Surface Reconstruction Method from Rock-Mass Point Clouds

As for rock numerical calculation and stability analysis, it is essential to build a numerical model of rock mass with concise and accurate structure information through the three-dimensional surface reconstruction of rock-mass point clouds. However, the current research on lightweight surface reconstruction of non-artificial objects is very limited. In this paper, an efficient lightweight surface reconstruction method for rock-mass point clouds is proposed. Firstly, the input point cloud is segmented to obtain the plane primitives. In this process, the recognition of texture information and the complete over-segmentation of effective information play a vital role in the high-precision segmentation of rock surfaces. Secondly, the boundaries of all planes are reorganized according to the obvious connectivity in the segmentation results, so as to realize the assembly of the model, while solving all collision problems. Finally, an integer programming model is constructed to screen the best closure scheme of each plane, thus ensuring the best outcome of the reconstruction. In this study, seven groups of natural rock-mass point clouds are used to validate the proposed method. As suggested by the experimental results, this algorithm is effective in compressing the point cloud data of rock mass, to generate a watertight numerical model that can be directly used for simulation calculation. In addition, this method has strong robustness to noise and can effectively deal with highly corrupted rock-mass point cloud data.

Research on 3D Defect Information Management of Drainage Pipeline Based on BIM

With the age of pipeline and increase in the volume of urban sewage, the pipeline has different degrees of defects, which can cause safety problems such as road collapse and urban flooding. The service life of drainage pipes is closely related to daily maintenance and inspection, so it is very important to inspect the defects and monitor the operation of drainage pipes regularly. However, the existing research lacks quantitative detection and intelligent management of pipeline defect information. Therefore, the depth camera is used as the sensor to quantitatively detect the volume and area of the pit on the concrete pipe, and a defect information management platform is constructed in this paper. Firstly, combined BIM model with 3D point cloud, this paper proposes a 3D defect information management platform of drainage pipeline. Then, the depth camera is used to collect the damage data and preprocess the data, and a method for calculating the damage volume and surface area of drainage pipeline based on 3D mesh reconstruction of the defect point cloud is proposed. The verification experiment results show that the error between the quantized volume and the real volume is mostly within 10%, and the maximum error is 17.54%, indicating high accuracy. The drainage pipeline information model is created. Finally, the data is uploaded to the information management platform to realize the visualization and informatization of pipeline defects and the later operation and maintenance requirements of the pipeline.

Part-Wise AtlasNet for 3D point cloud reconstruction from a single image

Learning to generate three dimensional (3D) point clouds from a single image remains a challenging task. Numerous approaches with encoder–decoder architectures have been proposed. However, these methods are hard to realize structured reconstructions and usually lack constraints on the local structures of 3D objects. AtlasNet as a representative model of 3D reconstruction consists of many branches, and each branch is a neural network used to reconstruct one local patch of a 3D object. However, the neural networks in AtlasNet and the patches of 3D objects are not in one-to-one correspondence before training. This case is not conducive to adding some reconstruction constraints to the local structures of 3D objects. Based on the architecture of AtlasNet, we propose Part-Wise AtlasNet in which each neural network is only responsible for reconstructing one specific part of a 3D object. This kind of restriction facilitates imposition of several local constraints on the final reconstruction loss, hence better recovering 3D objects with finer local structures. Both the qualitative results and quantitative analysis show that the variants of the proposed method with the local reconstruction losses generate structured point clouds with a higher visual quality and achieve better performance than other methods in 3D point cloud generation from a single image.

Quadruple tripatch-wise modular architecture-based real-time structure from motion

Structure from motion (SFM), a research hotspot in the intelligent transportation field (including autonomous driving, environmental perception and augmented reality (AR) for artificial intelligence terminals), can automatically recover ego-motion state estimations and 3D scene reconstructions from multiple images or video sequences. Most existing vision methods operate offline in indoor scenes, and their reconstruction accuracies greatly depend on the tracking lifetimes and accuracies of feature points. Reprojection matrix and redundant regression noise computations are exponential disasters for the calculation and reconstruction drift of large-scale scenes. This paper proposes a quadruple tripatch-wise modular architecture (QTMA) for autonomous vehicle stereo image sequences that decomposes rigid scenes into nonrigid motion-segmented pieces for reconstruction. An advanced energy function for salient image features is established by combining multiple feature types with weighted finite-element mesh. Closed quadruple annular matching and relocation are performed via multiresolution pyramid images. The proposed incremental integral-mapping calculation method and unique tree-like stacked storage containers prolong the tracking lifetimes of consecutive frames and ensure the spatiotemporal consistency and robustness of the homonymous image features in different subsequences. Experimental results verify the effectiveness of this architecture for different transportation scenes; the frame rate processing speed reaches 30 fps, the calculation accuracy regarding the path distance difference reaches 99.49%, and the estimation results regarding the maximum speeds of motion are closer to the ground truth. The translation error of the motion pose is 0.0136%, and the rotation error is 0.0035 [deg/m], which has more yaw stability than the existing state-of-the-art methods. Furthermore, in the reconstructed point cloud quality demonstration, the mean value of roughness is reduced by 40.127%, the mean value of density is improved by 27.701%, and the accuracy reaches 91.149% within a certain distance tolerance. This paper has significant theoretical research value and application potential for positioning, path tracking, and navigation in adaptive cruise control (ACC) and advanced driver assistance systems (ADAS).

3D Texture Reconstruction of Abdominal Cavity Based on Monocular Vision SLAM for Minimally Invasive Surgery

The depth information of abdominal tissue surface and the position of laparoscope are very important for accurate surgical navigation in computer-aided surgery. It is difficult to determine the lesion location by empirically matching the laparoscopic visual field with the preoperative image, which is easy to cause intraoperative errors. Aiming at the complex abdominal environment, this paper constructs an improved monocular simultaneous localization and mapping (SLAM) system model, which can more accurately and truly reflect the abdominal cavity structure and spatial relationship. Firstly, in order to enhance the contrast between blood vessels and background, the contrast limited adaptive histogram equalization (CLAHE) algorithm is introduced to preprocess abdominal images. Secondly, combined with AKAZE algorithm, the Oriented FAST and Rotated BRIEF(ORB) algorithm is improved to extract the features of abdominal image, which improves the accuracy of extracted symmetry feature points pair and uses the RANSAC algorithm to quickly eliminate the majority of mis-matched pairs. The medical bag-of-words model is used to replace the traditional bag-of-words model to facilitate the comparison of similarity between abdominal images, which has stronger similarity calculation ability and reduces the matching time between the current abdominal image frame and the historical abdominal image frame. Finally, Poisson surface reconstruction is used to transform the point cloud into a triangular mesh surface, and the abdominal cavity texture image is superimposed on the 3D surface described by the mesh to generate the abdominal cavity inner wall texture. The surface of the abdominal cavity 3D model is smooth and has a strong sense of reality. The experimental results show that the improved SLAM system increases the registration accuracy of feature points and the densification, and the visual effect of dense point cloud reconstruction is more realistic for Hamlyn dataset. The 3D reconstruction technology creates a realistic model to identify the blood vessels, nerves and other tissues in the patient’s focal area, enabling three-dimensional visualization of the focal area, facilitating the surgeon’s observation and diagnosis, and digital simulation of the surgical operation to optimize the surgical plan.

Enhancement of 3D Point Cloud Contents Using 2D Image Super Resolution Network

Media technology has been developed to give users a sense of immersion. Recent media using 3D spatial data, such as augmented reality and virtual reality, has attracted attention. A point cloud is a data format that consists of a number of points, and thus can express 3D media using coordinates and color information for each point. Since a point cloud has a larger capacity than 2D images, a technology to compress the point cloud is required, i.e., standardized in the international standard organization MPEG as a video-based point cloud compression (V-PCC). V-PCC decomposes 3D point cloud data into 2D patches along orthogonal directions, and those patches are placed into a 2D image sequence, and then compressed using existing 2D video codecs. However, data loss may occur while converting a 3D point cloud into a 2D image sequence and encoding this sequence using a legacy video codec. This data loss can cause deterioration in the quality of a reconstructed point cloud. This paper proposed a method of enhancing a reconstructed point cloud by applying a super resolution network to the 2D patch image sequence of a 3D point cloud.

NURBS-Diff: A Differentiable Programming Module for NURBS

Boundary representations (B-reps) using Non-Uniform Rational B-splines (NURBS) are the de facto standard used in CAD, but their utility in deep learning-based approaches is not well researched. We propose a differentiable NURBS module to integrate NURBS representations of CAD models with deep learning methods. We mathematically define the derivatives of the NURBS curves or surfaces with respect to the input parameters (control points, weights, and the knot vector). These derivatives are used to define an approximate Jacobian used for performing the “backward” evaluation to train the deep learning models. We have implemented our NURBS module using GPU-accelerated algorithms and integrated it with PyTorch, a popular deep learning framework. We demonstrate the efficacy of our NURBS module in performing CAD operations such as curve or surface fitting and surface offsetting. Further, we show its utility in deep learning for unsupervised point cloud reconstruction and enforce analysis constraints. These examples show that our module performs better for certain deep learning frameworks and can be directly integrated with any deep-learning framework requiring NURBS.

ADR-MVSNet: A cascade network for 3D point cloud reconstruction with pixel occlusion

3D point cloud reconstruction is an urgent task in computer vision for environment perception. Nevertheless, the reconstructed scene is inaccurate and incomplete, because the visibility of pixels is not taken into account by existing methods. In this paper, a cascaded network with a multiple cost volume aggregation module named ADR-MVSNet is proposed. Three improvements are presented in ADR-MVSNet. First, to improve the reconstruction accuracy and reduce the time complexity, an adaptive depth reduction module, which adaptively adjusts the depth range of the pixel through the confidence interval, is proposed. Second, to more accurately estimate the depth of occluded pixels in multiview images, a multiple cost volume aggregation module, in which Gini impurity is introduced to measure the confidence of pixel depth prediction, is proposed. Third, a multiscale photometric consistency filter module is proposed, which considers the information in multiple confidence maps at the same time and filters out outliers accurately to remove pixels with low confidence. Therefore, the accuracy of point cloud reconstruction is improved. The experimental results on the DTU and Tanks and Temple datasets demonstrate that ADR-MVSNet achieves highly accurate and highly complete reconstruction compared with state-of-the-art benchmarks.

Oriented and Directional Chamfer Distance Losses for 3D Object Reconstruction From a Single Image

The application of deep learning in the field of 3D reconstruction has greatly improved the quality of 3D object reconstruction. For methods that take the point cloud as supervision information, previous research has mainly focused on the network architecture while setting Chamfer Distance (CD) loss as the default loss function. However, CD only contains distance information while ignoring directional information. In this paper, we introduce novel CD losses considering directions that can be used in a 3D reconstruction network. These CD losses consider both direction and distance information, and have two specific variants, Oriented Chamfer Distance (OCD) and Directional Chamfer Distance (DCD). Numerous experiments conducted on the deformable patch and point cloud reconstruction, show that some classic neural networks for 3D reconstruction with OCD or DCD loss can achieve better reconstruction results than those with CD loss.

Research on Object Panoramic 3D Point Cloud Reconstruction System Based on Structure From Motion

3D reconstruction is the transformation of real objects into mathematical models. By using 3D models, we can observe the shape and measure the parameters, and help us to analyze the properties of objects. For the problems of incompleteness and inefficiency in the reconstruction of object 3D point clouds, a fast and automated system for panoramic 3D point cloud reconstruction of objects was proposed. First, we designed an automatic platform, which could acquire RGB image sequences of objects in two directions. Then we adopted the Structure From Motion (SFM) algorithm to generate point clouds. For the problem of different scales of point clouds, we obtained the scaling by calculating the length ratio of the axes of the oriented bounding box, and scaled the point clouds to a uniform scale. In addition, markers were placed around the object and used to acquire the rotation matrix of the object point cloud in two directions. Finally, we verified the point cloud models of different objects generated by the system, and found that the relative error didn’t exceed 6.67%. According to the results, the system proposed could reconstruct the panoramic 3D point cloud of the object better and provide a reference for related research.