Three-dimensional Point Cloud Research Articles

Semantic Segmentation for Digital Archives of Borobudur Reliefs Based on Soft-Edge Enhanced Deep Learning

Segmentation and visualization of three-dimensional digital cultural heritage are important analytical tools for the intuitive understanding of content. In this paper, we propose a semantic segmentation and visualization framework that automatically classifies carved items (people, buildings, plants, etc.) in cultural heritage reliefs. We also apply our method to the bas-reliefs of Borobudur Temple, a UNESCO World Heritage Site in Indonesia. The difficulty in relief segmentation lies in the fact that the boundaries of each carved item are formed by indistinct soft edges, i.e., edges with low curvature. This unfavorable relief feature leads the conventional methods to fail to extract soft edges, whether they are three-dimensional methods classifying a three-dimensional scanned point cloud or two-dimensional methods classifying pixels in a drawn image. To solve this problem, we propose a deep-learning-based soft edge enhanced network to extract the semantic labels of each carved item from multichannel images that are projected from the three-dimensional point clouds of the reliefs. The soft edges in the reliefs can be clearly extracted using our novel opacity-based edge highlighting method. By mapping the extracted semantic labels into three-dimensional points of the relief data, the proposed method provides comprehensive three-dimensional semantic segmentation results of the Borobudur reliefs.

A Novel Type of Boundary Extraction Method and Its Statistical Improvement for Unorganized Point Clouds Based on Concurrent Delaunay Triangular Meshes.

Currently, three-dimensional (3D) laser-scanned point clouds have been broadly applied in many important fields, such as non-contact measurements and reverse engineering. However, it is a huge challenge to efficiently and precisely extract the boundary features of unorganized point cloud data with strong randomness and distinct uncertainty. Therefore, a novel type of boundary extraction method will be developed based on concurrent Delaunay triangular meshes (CDTMs), which adds the vertex-angles of all CDTMs around a common data point together as an evaluation index to judge whether this targeted point will appear at boundary regions. Based on the statistical analyses on the CDTM numbers of every data point, another new type of CDTM-based boundary extraction method will be further improved by filtering out most of potential non-edge points in advance. Then these two CDTM-based methods and popular α-shape method will be employed in conducting boundary extractions on several point cloud datasets for comparatively analyzing and discussing their extraction accuracies and time consumptions in detail. Finally, all obtained results can strongly demonstrate that both these two CDTM-based methods present superior accuracies and strong robustness in extracting the boundary features of various unorganized point clouds, but the statistically improved version can greatly reduce time consumption.

Non-Cooperative Target Attitude Estimation Method Based on Deep Learning of Ground and Space Access Scene Radar Images

Determining the attitude of a non-cooperative target in space is an important frontier issue in the aerospace field, and has important application value in the fields of malfunctioning satellite state assessment and non-cooperative target detection in space. This paper proposes a non-cooperative target attitude estimation method based on the deep learning of ground and space access (GSA) scene radar images to solve this problem. In GSA scenes, the observed target satellite can be imaged not only by inverse synthetic-aperture radar (ISAR), but also by space-based optical satellites, with space-based optical images providing more accurate attitude estimates for the target. The spatial orientation of the intersection of the orbital planes of the target and observation satellites can be changed by fine tuning the orbit of the observation satellite. The intersection of the orbital planes is controlled to ensure that it is collinear with the position vector of the target satellite when it is accessible to the radar. Thus, a series of GSA scenes are generated. In these GSA scenes, the high-precision attitude values of the target satellite can be estimated from the space-based optical images obtained by the observation satellite. Thus, the corresponding relationship between a series of ISAR images and the attitude estimation of the target at this moment can be obtained. Because the target attitude can be accurately estimated from the GSA scenes obtained by a space-based optical telescope, these attitude estimation values can be used as training datasets of ISAR images, and deep learning training can be performed on ISAR images of GSA scenes. This paper proposes an instantaneous attitude estimation method based on a deep network, which can achieve robust attitude estimation under different signal-to-noise ratio conditions. First, ISAR observation and imaging models were created, and the theoretical projection relationship from the three-dimensional point cloud to the ISAR imaging plane was constructed based on the radar line of sight. Under the premise that the ISAR imaging plane was fixed, the ISAR imaging results, theoretical projection map, and target attitude were in a one-to-one correspondence, which meant that the mapping relationship could be learned using a deep network. Specifically, in order to suppress noise interference, a UNet++ network with strong feature extraction ability was used to learn the mapping relationship between the ISAR imaging results and the theoretical projection map to achieve ISAR image enhancement. The shifted window (swin) transformer was then used to learn the mapping relationship between the enhanced ISAR images and target attitude to achieve instantaneous attitude estimation. Finally, the effectiveness of the proposed method was verified using electromagnetic simulation data, and it was found that the average attitude estimation error of the proposed method was less than 1°.

COPRNet: correspondence confidence and overlap score guided network for indoor partial point cloud registration

ABSTRACT Three-dimensional (3D) point cloud registration is a critical topic in 3D computer vision and remote sensing. Several algorithms based on deep learning techniques have recently tried to deal with indoor partial-to-partial point cloud registration by searching the correspondences between input point clouds. However, existing correspondence-based methods are vulnerable to noise and do not adequately exploit geometric information to extract features, resulting in incorrect correspondences. In this work, we develop a novel network using correspondence confidence and overlap scores to address these challenges. Specifically, we first introduce a feature interaction module that combines spatial structure information to encode unique geometric embedding, greatly enhancing the feature perception. Furthermore, we design a corresponding point matching module, which includes a two-stage point filtering strategy. This method effectively improves the ability to identify embedded inliers from outliers and accurately remove spurious matches, thus allowing the network to focus more on the accurate correspondences of overlapping regions. Extensive experiments on different benchmark datasets indicate that our network shows superior performance of indoor point cloud registration, especially in low overlap registration, with significant improvement over state-of-the-art (SOTA) methods. Our code can be found at https://github.com/tranceok/COPRNet.

Global Reconstruction Method of Maize Population at Seedling Stage Based on Kinect Sensor

Automatic plant phenotype measurement technology based on the rapid and accurate reconstruction of maize structures at the seedling stage is essential for the early variety selection, cultivation, and scientific management of maize. Manual measurement is time-consuming, laborious, and error-prone. The lack of mobility of large equipment in the field make the high-throughput detection of maize plant phenotypes challenging. Therefore, a global 3D reconstruction algorithm was proposed for the high-throughput detection of maize phenotypic traits. First, a self-propelled mobile platform was used to automatically collect three-dimensional point clouds of maize seedling populations from multiple measurement points and perspectives. Second, the Harris corner detection algorithm and singular value decomposition (SVD) were used for the pre-calibration single measurement point multi-view alignment matrix. Finally, the multi-view registration algorithm and iterative nearest point algorithm (ICP) were used for the global 3D reconstruction of the maize seedling population. The results showed that the R2 of the plant height and maximum width measured by the global 3D reconstruction of the seedling maize population were 0.98 and 0.99 with RMSE of 1.39 cm and 1.45 cm and mean absolute percentage errors (MAPEs) of 1.92% and 2.29%, respectively. For the standard sphere, the percentage of the Hausdorff distance set of reconstruction point clouds less than 0.5 cm was 55.26%, and the percentage was 76.88% for those less than 0.8 cm. The method proposed in this study provides a reference for the global reconstruction and phenotypic measurement of crop populations at the seedling stage, which aids in the early management of maize with precision and intelligence.

Three-dimensional point cloud registration algorithm based on the correlation coefficient

Point cloud registration is an important research problem in machine vision. One of the most widely used methods for point set registration is the iterative closest point (ICP) algorithm. However, ICP is known to be susceptible to local minima because it is based on the local iterative optimization technique and its performance critically relies on the quality of the initialization. To reduce the possibility of the algorithm falling into local minima, we propose an iterative algorithm based on geometric distance and local feature weighting named weighted correlation coefficient iterative closest point algorithm (WCC_ICP). The algorithm first establishes the corresponding relationship of points using linear representation and constructs an optimization model with constraints. Subsequently, the constrained optimization model is converted into an unconstrained optimization model by Lagrangian multiplication. Finally, an iterative technique is used to solve the optimal rigid body transformation of the optimized model. In addition, in each iteration, the identification ability of the local feature operator of points is used to repair the corresponding relation of points, which reduces the risk of the iterative algorithm falling into the local minima. Numerical experiments show that the proposed WCC_ICP algorithm reduces the risk of the iterative algorithm falling into local minima.

Three-Dimensional Point Cloud Semantic Segmentation for Cultural Heritage: A Comprehensive Review

In the cultural heritage field, point clouds, as important raw data of geomatics, are not only three-dimensional (3D) spatial presentations of 3D objects but they also have the potential to gradually advance towards an intelligent data structure with scene understanding, autonomous cognition, and a decision-making ability. The approach of point cloud semantic segmentation as a preliminary stage can help to realize this advancement. With the demand for semantic comprehensibility of point cloud data and the widespread application of machine learning and deep learning approaches in point cloud semantic segmentation, there is a need for a comprehensive literature review covering the topics from the point cloud data acquisition to semantic segmentation algorithms with application strategies in cultural heritage. This paper first reviews the current trends of acquiring point cloud data of cultural heritage from a single platform with multiple sensors and multi-platform collaborative data fusion. Then, the point cloud semantic segmentation algorithms are discussed with their advantages, disadvantages, and specific applications in the cultural heritage field. These algorithms include region growing, model fitting, unsupervised clustering, supervised machine learning, and deep learning. In addition, we summarized the public benchmark point cloud datasets related to cultural heritage. Finally, the problems and constructive development trends of 3D point cloud semantic segmentation in the cultural heritage field are presented.

Human-vision-inspired cluster identification for single-molecule localization microscopy.

Single-molecule localization microscopy has enabled scientists to visualize cellular structures at the nanometer scale. However, researchers are facing great challenges in analyzing images presented by point clouds. Existing algorithms for cluster identification are coordinate-based analyses requiring users to input cutoff thresholds based on the distance or density of the point cloud. These thresholds are often one's best guess with repeated visual inspections, making the cluster assignment user-dependent. Here, we present a cluster identification algorithm mimicking the modulation transfer function of human vision. This approach does not require any input parameters and produces visually satisfactory cluster assignments. We tested this algorithm by identifying the clusters of the fusion proteins of the Nipah virus on its host cells. This algorithm was further extended to analyze three-dimensional point clouds using virus-like particles as an example.

EFFECT OF KEYFRAMES EXTRACTION FROM THERMAL INFRARED VIDEO STREAM TO GENERATE DENSE POINT CLOUD OF THE BUILDING'S FACADE

Abstract. Keyframes extraction is required and effective for the 3D reconstruction of objects from a thermal video sequence to increase geometric accuracy, reduce the volume of aerial triangulation calculations, and generate the dense point cloud. The primary goal and focus of this paper are to assess the effect of keyframes extraction from the thermal infrared video sequence on the geometric accuracy of the dense point cloud generated. The method of keyframes extraction of thermal infrared video presented in this paper consists of three basic steps. (A) The ability to identify and remove blur frames from non-blur frames in a sequence of recorded frames. (B) The ability to apply the standard baseline condition between sequence frames to establish the overlap condition and prevent the creation of degeneracy conditions. (C) Evaluating degeneracy conditions and keyframes extraction using Geometric Robust Information Criteria (GRIC). The performance evaluation criteria for keyframes extraction in the generation of the thermal infrared dense point cloud in this paper are to assess the increase in density of the generated three-dimensional point cloud and reduce reprojection error. Based on the results and assessments presented in this paper, using keyframes increases the density of the thermal infrared dense point cloud by about 0.03% to 0.10% of points per square meter. It reduces the reprojection error by about 0.005% of pixels (2 times).

Density estimation method of mature wheat based on point cloud segmentation and clustering

The sustainable development of agriculture needs to rely on precision agricultural technology. The premise and key to realizing precision agriculture is the research on the characteristics of crops in the field. For wheat plants, the ear is the flower or fruit part at the top of the wheat stem, which is one of the important components of yield, and its density can be said to be one of the most important traits. Its number is arguably-one of the most important phenotypic traits. The traditional method of density measurement is manual, which is time-consuming and laborious. Therefore, an efficient and convenient wheat density estimation scheme is needed to provide data support for crop yield monitoring, to achieve a better production management system. Stereo vision is a 3D imaging method that allows rapid measurement of plant structures, and point cloud segmentation is the key to studying the 3D spatial characteristics of plants. In this paper, the three-dimensional point cloud of wheat reconstructed by stereo vision technology is used for segmentation and clustering, and a method for clustering dense wheat rows is proposed. Firstly, a binocular camera was used to record video to reconstruct the wheat point cloud; then, point cloud pre-processing was used to remove noise; then, the octree splitting algorithm and voxel mesh merging algorithm were used to divide the dense wheat, and then clustering algorithm was used to get the point cloud of wheat ears; finally, the relationship model between the number of wheat ears point clouds and the number of wheat plants was established by linear regression analysis with R2 of 0.97. To verify the effectiveness of the algorithm, the actual field measurements and the predicted values of the algorithm were compared, with R2 of 0.93. The density estimation method provides a new method for the study of phenotypic information of crop population information and also provides a reference for nondestructive crop measurements.

Apparent Deterioration Law and Shear Failure Mode of Rock-Mortar Interface Based on Topography-Sensing Technology.

As an advanced spatial technology, topography-sensing technology is comprehensive, macroscopic, and intuitive. It shows unique advantages for rock structure interpretation and has important guiding significance for the research of the shear performances of rock-mortar interface under cyclic load in rock mass engineering. In this paper, cyclic shearing tests combined with the shear surface topography-sensing technology are employed to investigate the evolution characteristics of the interface morphology and the strength deterioration of the rock-mortar interface. Primarily, mortar and three types of rocks are used to prepare different rock-mortar interfaces, which are then applied to cyclic shear loading under two constant normal stresses. Subsequently, the shear strength degradation and dilatancy characteristics of rock-mortar interfaces with varying shear times are discussed. In addition, on the basis of the non-contact three-dimensional topography-sensing technology, the apparent three-dimensional point-cloud coordinate information of rock-mortar interface before and after each shear loading is obtained, and the apparent three-dimensional topography parameters of rock-mortar interface are calculated, according to which the influences of normal stress and lithology on the topography of interface subjected to cyclic shearing loading are analyzed.

Monitoring Mining Surface Subsidence with Multi-Temporal Three-Dimensional Unmanned Aerial Vehicle Point Cloud

Long-term and high-intensity coal mining has led to the increasingly serious surface subsidence and environmental problems. Surface subsidence monitoring plays an important role in protecting the ecological environment of the mining area and the sustainable development of modern coal mines. The development of surveying technology has promoted the acquisition of high-resolution terrain data. The combination of an unmanned aerial vehicle (UAV) point cloud and the structure from motion (SfM) method has shown the potential of collecting multi-temporal high-resolution terrain data in complex or inaccessible environments. The difference of the DEM (DoD) is the main method to obtain the surface subsidence in mining areas. However, the obtained digital elevation model (DEM) needs to interpolate the point cloud into the grid, and this process may introduce errors in complex natural topographic environments. Therefore, a complete three-dimensional change analysis is required to quantify the surface change in complex natural terrain. In this study, we propose a quantitative analysis method of ground subsidence based on three-dimensional point cloud. Firstly, the Monte Carlo simulation statistical analysis was adopted to indirectly evaluate the performance of direct georeferencing photogrammetric products. After that, the operation of co-registration was carried out to register the multi-temporal UAV dense matching point cloud. Finally, the model-to-model cloud comparison (M3C2) algorithm was used to quantify the surface change and reveal the spatio-temporal characteristics of surface subsidence. In order to evaluate the proposed method, four periods of multi-temporal UAV photogrammetric data and a period of airborne LiDAR point cloud data were collected in the Yangquan mining area, China, from 2020 to 2022. The 3D precision map of a sparse point cloud generated by Monte Carlo simulation shows that the average precision in X, Y and Z directions is 44.80 mm, 45.22 and 63.60 mm, respectively. The standard deviation range of the M3C2 distance calculated by multi-temporal data in the stable area is 0.13–0.19, indicating the consistency of multi-temporal photogrammetric data of UAV. Compared with DoD, the dynamic moving basin obtained by the M3C2 algorithm based on the 3D point cloud obtained more real surface deformation distribution. This method has high potential in monitoring terrain change in remote areas, and can provide a reference for monitoring similar objects such as landslides.

Assessing reforestation after a fire using UAVs

In 2022, an aerial photograph of a 16-year-old burnt area was carried out along the northwestern border of the Botanical Garden of Petrozavodsk State University (Republic of Karelia, Russia). In addition to this, a field survey of the survey sites was carried out. The work was carried out to assess the state of natural forest regeneration after the fire. The results of combined studies showed that after the fire, viable young growth of mixed composition was formed with a predominance of Pinus sylvestris (71% of the total number of tree species), Betula pendula (22%), Populus tremula and Alnus incana (3.5% each). When comparing plant heights on three-dimensional point clouds with field measurements, it was only possible to reliably determine plant heights for a large generally accepted category (> 1.5 m), while the dependence of determining tree heights obtained by different methods turned out to be very significant (R2 = 0.95). The maximum height of P. sylvestris trees in the study area was 6.2 m, the average values varied from 3.5 m to 4.5 m, which indicates the success of reforestation after a fire on rocky outcrops.

Geometric Dimension Measurement Method for Bolster Spring Based on Three-Dimensional Laser Point Clouds

Geometric Dimension Measurement Method for Bolster Spring Based on Three-Dimensional Laser Point Clouds

Deep Neural Networks-Enabled Vehicle Detection Using High-Resolution Automotive Radar Imaging

Advanced driver assistance systems (ADASs) and autonomous vehicles rely on different types of sensors, such as camera, radar, ultrasonic, and LiDAR to sense the surrounding environment. Compared with the other types of sensors, millimeter-wave automotive radar has advantages in terms of cost and reliability under bad weather conditions (e.g., snow, rain, fog) and doesn't suffer from light condition variations (e.g., darkness). Typical radar devices used in today's commercial vehicles with ADAS features produce sparse point clouds in low angular resolution with a limited number of antennas. In this paper, we present a machine learning-aided signal processing chain to suppress the radar imaging blur effect introduced by the phase migration in time-division multiplexing multi-input multi-output (MIMO) radar, to generate low-level high-resolution radar bird's eye view (BEV) spectra with rich object's features. Compared with radar point clouds, there is no information loss in radar BEV spectra. We then propose a Temporal-fusion, Distance tolerant single stage object detection Network, termed as, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TDRadarNet</i> , and an enhanced version, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TDRadarNet+</i> to robustly detect vehicles in both long and short ranges on radar BEVs. We introduce a first-of-its-kind multi-model dataset, containing 14,800 frames of high-resolution low-level radar BEV spectra with synchronized stereo cameras RGB images and three-dimensional (3D) LiDAR point clouds. Our dataset achieves 0.39 m range resolution and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1.2^\circ$</tex-math></inline-formula> degree azimuth angular resolution with 100 meters maximum detectable range. Moreover, we create a sub-dataset, the Doppler Unfolding dataset, containing 244,140 beam vectors extracted from the 3D radar data cube. With extensive testing and evaluation, we demonstrate that our Doppler unfolding network achieves 93.46% Doppler unfolding accuracy. Compared to YOLOv7, a state-of-the-art image-based object detection network, TDRadarNet achieves a 70.3% Average Precision (AP) for vehicle detection, demonstrating a 21.0% improvement; TDRadarNet+ achieves a 73.9% AP, showing a 24.6% improvement in performance.

Deep Learning-Based Autonomous Excavation: A Bucket-Trajectory Planning Algorithm

The increased risk to the safety of excavator personnel and difficulty in training them, combined with a manpower shortage, have led to an increased demand for machine automation. This study applies a long short-term memory algorithm for automating a bucket-tip trajectory planning AI system. Unlike other autonomous excavation techniques, the proposed approach in this study performs the bucket-trajectory planning of the excavator without prior knowledge of nonlinear bucket–soil interaction dynamics during excavation, which requires precise adjustment of parameters with the heuristic analysis of correlation between them. Based on data acquisition from excavation of excavator experts, this method uses the three-dimensional point cloud of terrain and bucket motion data in excavation process for training and application of the AI. Especially, we transform the point cloud, which comprises massive number of points and increases the computation complexity, into the much smaller number of values, which are sufficient for representing the three-dimensional shape of target terrain. To ensure safety against collision with underground obstacles, a collision avoidance algorithm is applied to prevent crashes during excavations along the given path, based on continuous monitoring of the pressure in the excavator’s hydraulic cylinder. Comparative experiments reveal that the bucket-tip trajectory planning AI system with the collision avoidance algorithm generates a traceable trajectory for the machine controller, equipped in the excavator, and yields the desired excavation volume and lead time without collision, regardless of the topographic change caused through successive excavations.

Three-Dimensional Point Cloud Segmentation Based on Context Feature for Sheet Metal Part Boundary Recognition

Three-Dimensional Point Cloud Segmentation Based on Context Feature for Sheet Metal Part Boundary Recognition

Measurement Algorithm for Anti-Stress Cone Parameters of Cross-Linked Polyethylene Cable Joint Based on Three-Dimensional Point Cloud Processing

Measurement Algorithm for Anti-Stress Cone Parameters of Cross-Linked Polyethylene Cable Joint Based on Three-Dimensional Point Cloud Processing

Fusion and Visualization of Three-Dimensional Point Cloud and Optical Images

Fusion and Visualization of Three-Dimensional Point Cloud and Optical Images

Investigation on the fractal characteristic of asphalt pavement texture roughness incorporating 3D reconstruction technology

&lt;abstract&gt; &lt;p&gt;The textural roughness of asphalt pavement surface is an important indicator to characterize pavement skid resistance. In this paper, multi-visual technology was applied to capture the surface image of asphalt pavement which was transformed into a visualized 3D point cloud model. Then, based on the principle of the digital elevation model (DEM), the disordered 3D point cloud is rasterized and projected into a 2D matrix which contains generalized point cloud elevation information. Meanwhile, the 2D matrix is transformed into grayscale to build the equivalent grayscale image. Furthermore, the fractal dimensions were calculated in terms of one-dimensional pavement section profile, two-dimensional grayscale, and equivalent grayscale to characterize the pavement roughness. The results showed that the fractal dimensions are positively correlated with the mean texture depth (MTD), and the fractal dimension of equivalent grayscale has the best correlation with MTD. It should be highlighted that the equivalent grayscale image is directly transformed by the reconstruction of the three-dimensional point cloud, and the grayscale value of each point can represent the elevation of different pavement surfaces. Therefore, the equivalent grayscale image can better reflect the real roughness of the pavement surface. Meanwhile, the proposed method in this paper can effectively reduce the influence of some factors (e.g., light and color, etc..) on the texture detection of the pavement surface.&lt;/p&gt; &lt;/abstract&gt;