Point Cloud Reconstruction Research Articles

ADR-MVSNet: A cascade network for 3D point cloud reconstruction with pixel occlusion

3D point cloud reconstruction is an urgent task in computer vision for environment perception. Nevertheless, the reconstructed scene is inaccurate and incomplete, because the visibility of pixels is not taken into account by existing methods. In this paper, a cascaded network with a multiple cost volume aggregation module named ADR-MVSNet is proposed. Three improvements are presented in ADR-MVSNet. First, to improve the reconstruction accuracy and reduce the time complexity, an adaptive depth reduction module, which adaptively adjusts the depth range of the pixel through the confidence interval, is proposed. Second, to more accurately estimate the depth of occluded pixels in multiview images, a multiple cost volume aggregation module, in which Gini impurity is introduced to measure the confidence of pixel depth prediction, is proposed. Third, a multiscale photometric consistency filter module is proposed, which considers the information in multiple confidence maps at the same time and filters out outliers accurately to remove pixels with low confidence. Therefore, the accuracy of point cloud reconstruction is improved. The experimental results on the DTU and Tanks and Temple datasets demonstrate that ADR-MVSNet achieves highly accurate and highly complete reconstruction compared with state-of-the-art benchmarks.

Oriented and Directional Chamfer Distance Losses for 3D Object Reconstruction From a Single Image

The application of deep learning in the field of 3D reconstruction has greatly improved the quality of 3D object reconstruction. For methods that take the point cloud as supervision information, previous research has mainly focused on the network architecture while setting Chamfer Distance (CD) loss as the default loss function. However, CD only contains distance information while ignoring directional information. In this paper, we introduce novel CD losses considering directions that can be used in a 3D reconstruction network. These CD losses consider both direction and distance information, and have two specific variants, Oriented Chamfer Distance (OCD) and Directional Chamfer Distance (DCD). Numerous experiments conducted on the deformable patch and point cloud reconstruction, show that some classic neural networks for 3D reconstruction with OCD or DCD loss can achieve better reconstruction results than those with CD loss.

Research on Object Panoramic 3D Point Cloud Reconstruction System Based on Structure From Motion

3D reconstruction is the transformation of real objects into mathematical models. By using 3D models, we can observe the shape and measure the parameters, and help us to analyze the properties of objects. For the problems of incompleteness and inefficiency in the reconstruction of object 3D point clouds, a fast and automated system for panoramic 3D point cloud reconstruction of objects was proposed. First, we designed an automatic platform, which could acquire RGB image sequences of objects in two directions. Then we adopted the Structure From Motion (SFM) algorithm to generate point clouds. For the problem of different scales of point clouds, we obtained the scaling by calculating the length ratio of the axes of the oriented bounding box, and scaled the point clouds to a uniform scale. In addition, markers were placed around the object and used to acquire the rotation matrix of the object point cloud in two directions. Finally, we verified the point cloud models of different objects generated by the system, and found that the relative error didn’t exceed 6.67%. According to the results, the system proposed could reconstruct the panoramic 3D point cloud of the object better and provide a reference for related research.

Distance Measurement Based on Three-Dimensional Surface Reconstruction of Spatial Point Cloud

Distance Measurement Based on Three-Dimensional Surface Reconstruction of Spatial Point Cloud

Dual-View 3D Reconstruction via Learning Correspondence and Dependency of Point Cloud Regions.

Multi-view 3D reconstruction generally adopts the feature fusion strategy to guide the generation of 3D shape for objects with different views. Empirically, the correspondence learning of object regions across different views enables better feature fusion. However, such idea has not been fully exploited in existing methods. Furthermore, current methods fail to explore the intrinsic dependency among regions within a 3D shape, leading to a rough reconstruction result. To address the above issues, we propose a Dual-View 3D Point Cloud reconstruction architecture named DVPC, which takes two views images as inputs, and progressively generates a refined 3D point cloud. First, a point cloud generation network is assigned to generate a coarse point cloud for each input view. Second, a dual-view point clouds synthesis network is presented in DVPC. It constructs a regional attention mechanism to learn a high-quality correspondence among regions across two coarse point clouds in different views, so that our DVPC can achieve feature fusion accurately. And then it develops a point cloud deformation module to produce a relatively-precise point cloud via establishing the communication between the coarse point cloud and the fused feature. Lastly, a point-region transformer network is devised to model the dependency among regions within the relatively-precise point cloud. With the dependency, the relatively-precise point cloud is refined into a desirable 3D point cloud with rich details. Qualitative and quantitative experiments on the ShapeNet and Pix3D datasets demonstrate that the proposed DVPC outperforms the state-of-the-art methods in terms of reconstruction quality.

Mobile Mapping System for Automatic Extraction of Geodetic Coordinates for Traffic Signs Based on Enhanced Point Cloud Reconstruction

Lidar sensors are commonly equipped on a mobile mapping system (MMS) to establish point clouds for HD map creation. However, the point clouds themselves do not contain object attributes. Therefore, human operators have to manually obtain objects’ position to assign attributes for further HD map conversion, inevitably resulting in time-consuming processes and significant labor costs. To solve the above problems, in this paper, we present an MMS equipped with non-survey grade Lidar, commercial grade camera, and entry level GNSS/INS, which incorporates ground control points (GCPs) with a Normal Distribution Transform Simultaneously Localization and Mapping (NDT SLAM) refinement and fluctuation adjustment to secure both absolute position accuracy and relative position accuracy of the reconstructed point cloud. Meanwhile, a deep neural network for image detection is employed to obtain the bounding box of traffic signs from each image frame. By applying the translation and rotation transformation between Lidar points and camera pixels, intersection of the detected object in the image and Lidar scan points can be found. By accumulating extracted Lidar points of the traffic sign in several detection frames, we can then obtain an accurate 3D geodetic coordinate of the traffic signs. Experimental results show that point clouds can be reconstructed with an average 3D RMSE of only 8.6cm, and center geodetic coordinates of traffic signs can be further extracted in sub-meter accuracy to significantly reduce labor work in HD map creation.

Research on 3D Modeling of Wupaolong based on Sparse Point Cloud Reconstruction of SFM Algorithm

As a classic project for the inheritance of Chinese folk culture, Wupaolong has the characteristics of unique material selection and complex production technology. However, as the finished products of folk handicrafts may not comply with production standards, a common problem in the production of handicrafts. Focusing on the problems of poor visual effects such as uneven surface and jagged edges in the craft paper binding model constructed by point cloud, and the need to enhance the sense of reality, this paper proposes an optimization method for 3D model and triangulation based on 3D line features. This method first preprocesses the feature points and performs sparse matching and dense matching on the processed feature points to generate a dense point cloud. The dense point cloud is then processed by an adaptive weighted median filtering method. This is followed by application of surface modeling method based on Delaunay algorithm to realize the reconstruction of object triangulation. Based on the triangular patch, the reference plane of the craft paper binding surface is fitted, and the model surface is corrected and optimized based on the reference plane. With the aid of three-dimensional line segments, the triangulation at the edge of the craft paper binding is corrected and optimized. Finally, an optimized three-dimensional model of craft paper binding is obtained. Experimental results show that the proposed algorithm can improve the unevenness of the 3D model plane and edges, and maintain the characteristics of the 3D model plane and edges. Compared with the existing surface reconstruction algorithms based on point cloud features, the optimized triangulation and model have higher accuracy and quality, which can greatly improve the visual effect of the model and improve the accuracy of the three-dimensional model. The research results of this paper can realize 3D modeling development for the production process of Wupaolong, effectively improve the elevation accuracy of the model, realize the production standard of digital Wupaolong model, and provide reference for folk production process.

Scanning Path Planning of the Robot for Breast Ultrasound Examination Based on Binocular Vision and NURBS

Ultrasonic scanning is the most commonly used method for detecting breast cancer. Traditional breast ultrasound mainly relies on manual examination, which requires doctors with rich clinical experience. The scanning path planning of the robot for breast ultrasound examination based on binocular vision and non-uniform rational B-Splines (NURBS) is proposed in this paper to solve the limitations of traditional examination methods. First, breast images are acquired by the binocular camera. The depth information of the images is obtained through stereo matching, and the 3D point cloud reconstruction of the breast model is completed. Second, the spatial index algorithm based on the K-dimension tree (Kd-tree) and uniform sampling method are selected to denoise and down-sample the point cloud model to complete the preliminary optimization of the model. The point cloud information for the scanning path is extracted according to the method of point cloud extraction and the principle of normal vector calculation. Then, the interpolation optimization of the scanning path based on the NURBS curve is completed. The optimized scanning path is transformed into the robot motion path. Finally, the robot motion is simulated and analyzed, and the experimental results show that the rationality of robot operation and the smoothness of robot motion meet the requirements of path planning defined in this paper. The error analysis experiment shows that the path planned by this method can ensure the efficiency and accuracy of ultrasound examination.

Video-Based Point Cloud Compression Artifact Removal

Photo-realistic point cloud capture and transmission are the fundamental enablers for immersive visual communication. The coding process of dynamic point clouds, especially video-based point cloud compression (V-PCC) developed by the MPEG standardization group, is now delivering state-of-the-art performance in compression efficiency. V-PCC is based on the projection of the point cloud patches to 2D planes and encoding the sequence as 2D texture and geometry patch sequences. However, the resulting quantization errors from coding can introduce compression artifacts, which can be very unpleasant for the quality of experience (QoE). In this work, we developed a novel out-of-the-loop point cloud geometry artifact removal solution that can significantly improve reconstruction quality without additional bandwidth cost. Our novel framework consists of a point cloud sampling scheme, an artifact removal network, and an aggregation scheme. The point cloud sampling scheme employs a cube-based neighborhood patch extraction to divide the point cloud into patches. The geometry artifact removal network then processes these patches to obtain artifact-removed patches. The artifact-removed patches are then merged together using an aggregation scheme to obtain the final artifact-removed point cloud. We employ 3D deep convolutional feature learning for geometry artifact removal that jointly recovers both the quantization direction and the quantization noise level by exploiting projection and quantization prior. The simulation results demonstrate that the proposed method is highly effective and can considerably improve the quality of the reconstructed point cloud.

RTSfM: Real-Time Structure From Motion for Mosaicing and DSM Mapping of Sequential Aerial Images With Low Overlap

Inspired by simultaneous localization and mapping (SLAM) style workflow, this article presented an online sequential structure from motion (SfM) solution for high-frequency video and large baseline high-resolution aerial images with high efficiency and novel precision. First, as traditional SLAM systems are not good in processing low overlap images, based on our novel hierarchical feature matching paradigm with multihomography and BoW, we proposed a robust tracking method where the relative pose and its scale are estimated separately followed by a joint optimization by considering both perspective-n-point (PnP) and epipolar constraints. Second, to further optimize the camera poses for the sparse map and dense pointcloud reconstruction, we provided a graph-based optimization with reprojection and GPS constraints, which make the camera trajectory and map georeferenced. We also incrementally generated the dense point cloud in real time from keyframes after local mapping optimization. Finally, we use a publicly available aerial image dataset with sequences of different environments, to evaluate the effectiveness of the proposed method, meanwhile, the robust performance of our solution is demonstrated with applications of high-quality aerial images mosaic and digital surface model (DSM) reconstruction in real time. Compared with the state-of-the-art SLAM and traditional SfM methods, the presented system can output large-scale high-quality ortho-mosaic and DSM in real time with the low computational cost.

Vegetation Removal on 3D Point Cloud Reconstruction of Cut-Slopes Using U-Net

The 3D point cloud reconstruction from photos taken by an unmanned aerial vehicle (UAV) is a promising tool for monitoring and managing risks of cut-slopes. However, surface changes on cut-slopes are likely to be hidden by seasonal vegetation variations on the cut-slopes. This paper proposes a vegetation removal method for 3D reconstructed point clouds using (1) a 2D image segmentation deep learning model and (2) projection matrices available from photogrammetry. For a given point cloud, each 3D point of it is reprojected into the image coordinates by the projection matrices to determine if it belongs to vegetation or not using the 2D image segmentation model. The 3D points belonging to vegetation in the 2D images are deleted from the point cloud. The effort to build a 2D image segmentation model was significantly reduced by using U-Net with the dataset prepared by the colour index method complemented by manual trimming. The proposed method was applied to a cut-slope in Doam Dam in South Korea, and showed that vegetation from the two point clouds of the cut-slope at winter and summer was removed successfully. The M3C2 distance between the two vegetation-removed point clouds showed a feasibility of the proposed method as a tool to reveal actual change of cut-slopes without the effect of vegetation.

OC-SLAM: Steadily Tracking and Mapping in Dynamic Environments

Visual Simultaneous Localization and Mapping (SLAM) system is mainly used in real-time localization and mapping tasks of robots in various complex environments, while traditional monocular vision algorithms are struggling to cope with weak texture and dynamic scenes. To solve these problems, this work presents an object detection and clustering assisted SLAM algorithm (OC-SLAM), which adopts a faster object detection algorithm to add semantic information to the image and conducts geometrical constraint on the dynamic keypoints in the prediction box to optimize the camera pose. It also uses RGB-D camera to perform dense point cloud reconstruction with the dynamic objects rejected, and facilitates European clustering of dense point clouds to jointly eliminate dynamic features combining with object detection algorithm. Experiments in the TUM dataset indicate that OC-SLAM enhances the localization accuracy of the SLAM system in the dynamic environments compared with original algorithm and it has shown impressive performance in the localizition and can build a more precise dense point cloud map in dynamic scenes.

3D Reconstruction of Unstructured Objects Using Information From Multiple Sensors

The Structure-from-Motion (SfM) algorithm is widely used for point cloud reconstruction. However, one drawback of conventional SfM based methods is that the obtained final point sets may contain holes and noise, which could degrade the estimation of reconstructed objects especially for smooth surfaces with few features. The other drawback is the accuracy and speed of SfM based methods depend on the uncertain number of images. To overcome these limitations, this paper proposes a novel 3D reconstruction method for unstructured objects based on the structure from motion in combination with the structured light, in which the point sets of structured light and the point sets of structure from motion can come from different target objects. Since the two point sets coming from multiple sensors do not scale well for register, making it difficult to find corresponding points, a scaled principal component analysis algorithm is proposed for the registration to overcome the impact due to large scale variance. With a large scale factor, a recalculated registration center is proposed via feature region segmentation to achieve point cloud registration again. The two point sets are matched using the proposed optimization method to complete 3D reconstruction. Surface reconstruction is performed using the Poisson algorithm to obtain a smooth surface. The proposed method is tested on some simple structured objects and real-life data of complex unstructured objects collected using range sensors. Compared with several state-of-the-art algorithms, experimental results confirm its potential for surface reconstruction from depth data calculated from the two sets.

3D MODELS BASED ON UAV IMAGES AND GNSS TECHNOLOGY IN THE FIELDS OF ARCHITECTURE, CIVIL ENGINEERING, SPATIAL PLANNING, AND ENERGETICS

<jats:p>The paper presents projects in the fields of architecture, spatial planning and energetics, in order to analyze the quality of 3D terrain models obtained by combining photogrammetry using drone images (UAV) and terrestrial measurements obtained by GNSS technology. In terms of geometric accuracy, the geometry of the reconstructed point clouds was controlled in three different projects. The achieved geometric accuracy met the criteria of required accuracy defined by clients. Further, the paper provides an overview of the advantages and disadvantages of such a methodology, as well as a review of the legislation that accompanies UAV systems in Bosnia and Herzegovina / Republika Srpska.</jats:p>

Finite element simulation of mechanical properties of Rosa roxburghii under compression loading

AbstractIt is very necessary to understand the mechanical properties of Rosa roxburghii fruit for harvesting, packaging, processing, transportation, storage, and related equipment design and improvement. This research was to determine the mechanical properties of R. roxburghii fruits through compressive experiments and finite element simulations and to verify the accuracy of the finite element model. In order to achieve finite element analysis, an accurate three‐dimensional (3D) geometric model was reconstructed by applying 3D laser technology and point cloud reconstruction algorithm. The elastic modulus and Poisson's ratio of R. roxburghii fruits were determined to be 4.0 MPa and 0.35 through experiments. The finite element simulation results provided valuable numerical data and visualized stress distribution. The simulation results showed that R. roxburghii fruits were damaged under the loads of 156.2 N (vertical) or 148.3 N (horizontal), with a maximum deformation of 5.0 mm (vertical) or 6.93 mm (horizontal), respectively. In the vertical and horizontal compression positions, the maximum relative errors between the experimental results and the finite element model are 24.71 and 13.99%, respectively. The results of the compressive strain energy of the fruit showed that the maximum relative error between the experimentally measured strain energy and the strain energy calculated by FEM simulation was 9.20 and 2.09% in the vertical and horizontal directions, respectively.Practical ApplicationsAs there is no relevant literature on the mechanical analysis of Rosa roxburghii fruit, the finite element method has been successfully applied to analyze the mechanical properties of R. roxburghii fruit under compressive load. At the same time, the simulation results have also been verified by experimental data, and the finite element technology has certain application prospects in food processing engineering. This study proposes a method based on 3D scanning technology and point cloud reconstruction algorithm to reconstruct an accurate geometric model of R. roxburghii fruit for finite element simulation, and conducts an in‐depth analysis on the stress, strain, and strain energy of the fruit. The purpose of this article is to propose a reliable method to predict the mechanical properties of the fruits under compressive load. The finite element simulation results can not only provide the basic data needed for fruit packaging and protection but also can be used for the design of automated fruit picking manipulators and the improvement of fruit processing equipment.

Truck body type classification using a deep representation learning ensemble on 3D point sets

Understanding the spatiotemporal distribution of commercial vehicles is essential for facilitating strategic pavement design, freight planning, and policy making. Hence, transportation agencies have been increasingly interested in collecting truck body configuration data due to its strong association with industries and freight commodities, to better understand their distinct operational characteristics and impacts on infrastructure and the environment. The rapid advancement of Light Detection and Ranging (LiDAR) technology has facilitated the development of non-intrusive detection solutions that are able to accurately classify truck body types in detail. This paper proposes a new truck classification method using a LiDAR sensor oriented to provide a wide field-of-view of roadways. In order to enrich the sparse point cloud obtained from the sensor, point clouds originating from the same truck across consecutive frames were grouped and combined using a two-stage vehicle reconstruction framework to generate a dense three-dimensional (3D) point cloud representation of each truck. Subsequently, PointNet – a deep representation learning algorithm – was adopted to train the classification model from reconstructed point clouds. The model utilizes low-level features extracted from the 3D point clouds and detects key features associated with each truck class. Finally, model ensemble techniques were explored to reduce the generalization error by averaging the results of seven PointNet models and further enhancing the overall model performance. The optimal number of models in the ensemble was determined through a comprehensive sensitivity analysis with the consideration of the average correct classification rate (CCR), the variability of the prediction results, and the computation efficiency. The developed model is capable of distinguishing passenger vehicles and 29 different truck body configurations with an average CCR of 83 percent. The average correct classification rate of the developed method on the test dataset was 90 percent for trucks pulling a large trailer(s).

Unsupervised recursive deep fitting of 3D primitives to points

The reconstruction of 3D point clouds with high-level geometric primitives is highly desirable due to the compactness and effectiveness of this representation. Nevertheless, correctly fitting 3D point sets with matching primitives is challenging due to the high combinatorial nature of this problem. We introduce a recursive neural network architecture that learns to fit 3D points with geometric primitives in an unsupervised manner. Our core idea is to divide-and-conquer the combinatorial complexity of primitive fitting utilizing recursion layers in our neural network architecture. Through recursion, network layers focus on different shape scales and fit primitives in a coarse-to-fine manner. I.e., early recursion layers solve for coarse regions, fitting large primitives while later layers solve for remaining fine-scale regions, fitting smaller primitives. The network model guarantees a global solution as recursion layers collaborate with each other to minimize an accumulative global loss. We present experiments that validate our approach in terms of accuracy, robustness and performance. We also compare to state-of-the-art to demonstrate our method’s advantages.

Avatars for Teleconsultation: Effects of Avatar Embodiment Techniques on User Perception in 3D Asymmetric Telepresence.

A 3D Telepresence system allows users to interact with each other in a virtual, mixed, or augmented reality (VR, MR, AR) environment, creating a shared space for collaboration and communication. There are two main methods for representing users within these 3D environments. Users can be represented either as point cloud reconstruction-based avatars that resemble a physical user or as virtual character-based avatars controlled by tracking the users' body motion. This work compares both techniques to identify the differences between user representations and their fit in the reconstructed environments regarding the perceived presence, uncanny valley factors, and behavior impression. Our study uses an asymmetric VR/AR teleconsultation system that allows a remote user to join a local scene using VR. The local user observes the remote user with an AR head-mounted display, leading to facial occlusions in the 3D reconstruction. Participants perform a warm-up interaction task followed by a goal-directed collaborative puzzle task, pursuing a common goal. The local user was represented either as a point cloud reconstruction or as a virtual character-based avatar, in which case the point cloud reconstruction of the local user was masked. Our results show that the point cloud reconstruction-based avatar was superior to the virtual character avatar regarding perceived co-presence, social presence, behavioral impression, and humanness. Further, we found that the task type partly affected the perception. The point cloud reconstruction-based approach led to higher usability ratings, while objective performance measures showed no significant difference. We conclude that despite partly missing facial information, the point cloud-based reconstruction resulted in better conveyance of the user behavior and a more coherent fit into the simulation context.

An iterative refinement method of point cloud for binocular vision 3D reconstruction

This paper introduces a new binocular stereo deep learning network based on point cloud, which can realize higher precision point cloud reconstruction through continuous iteration of the network. Our method directly carries out point cloud processing on the target, calculates the difference between the current depth map and the real depth, estimates the loss according to the predicted point cloud and the information of the dual view input image, and then uses the appropriate loss function to iteratively process the point cloud. In addition, we can customize the number of iterations to achieve higher precision point cloud effect. The proposed network basically achieves good results on KITTI data set.

A Point Cloud Fusion Method for Space Target 3D Laser Point Cloud and Visible Light Image Reconstruction Point Cloud

In this paper, a high-precision spatial target lidar for 3D reconstruction of point cloud and visible light image 3D reconstruction point cloud fusion method is proposed. This fusion method uses the solved 3D reconstruction to invert the camera pose and 3D point cloud model centroid position optimization initial value selection. The registration accuracy and registration efficiency of the ICP algorithm are improved. At the same time, according to the characteristics of the two sets of point clouds, the Euclidean distance threshold is used to delete the noise points of the 3D point cloud edge, and the high-precision 3D reconstruction point with the scale information fusion is obtained. cloud. The simulation experiment of the spatial target simulation model shows that the fusion method can effectively improve the point cloud density, fill the reconstruction vulnerability, and improve the point cloud accuracy of the spatial target 3D reconstruction.