3D Point Cloud Research Articles

3D Reconstruction From a Single Sketch via View-Dependent Depth Sampling.

Reconstructing a 3D shape based on a single sketch image is challenging due to the inherent sparsity and ambiguity present in sketches. Existing methods lose fine details when extracting features to predict 3D objects from sketches. Upon analyzing the 3D-to-2D projection process, we observe that the density map, characterizing the distribution of 2D point clouds, can serve as a proxy to facilitate the reconstruction process. In this work, we propose a novel sketch-based 3D reconstruction model named SketchSampler. It initiates the process by translating a sketch through an image translation network into a more informative 2D representation, which is then used to generate a density map. Subsequently, a two-stage probabilistic sampling process is employed to reconstruct a 3D point cloud: first, recovering the 2D points (i.e., the x and y coordinates) by sampling the density map; and second, predicting the depth (i.e., the z coordinate) by sampling the depth values along the ray determined by each 2D point. Additionally, we convert the reconstructed point cloud into a 3D mesh for wider applications. To reduce ambiguity, we incorporate hidden lines in sketches. Experimental results demonstrate that our proposed approach significantly outperforms other baseline methods.

3D scene generation for zero-shot learning using ChatGPT guided language prompts

Zero-shot learning in the realm of 3D point cloud data remains relatively unexplored compared to its 2D image counterpart. This domain introduces fresh challenges due to the absence of robust pre-trained feature extraction models. To tackle this, we introduce a prompt-guided method for 3D scene generation and supervision, enhancing the network’s ability to comprehend the intricate relationships between seen and unseen objects. Initially, we utilize basic prompts resembling scene annotations generated from one or two point cloud objects. Recognizing the limited diversity of basic prompts, we employ ChatGPT to expand them, enriching the contextual information within the descriptions. Subsequently, leveraging these descriptions, we arrange point cloud objects’ coordinates to fabricate augmented 3D scenes. Lastly, employing contrastive learning, we train our proposed architecture end-to-end, utilizing pairs of 3D scenes and prompt-based captions. We posit that 3D scenes facilitate more efficient object relationships than individual objects, as demonstrated by the effectiveness of language models like BERT in contextual understanding. Our prompt-guided scene generation method amalgamates data augmentation and prompt-based annotation, thereby enhancing 3D ZSL performance. We present ZSL and generalized ZSL results on both synthetic (ModelNet40, ModelNet10, and ShapeNet) and real-scanned (ScanOjbectNN) 3D object datasets. Furthermore, we challenge the model by training with synthetic data and testing with real-scanned data, achieving state-of-the-art performance compared to existing 2D and 3D ZSL methods in the literature. Codes and models are available at: https://github.com/saharahmadisohraviyeh/ChatGPT_ZSL_3D.

Selective weighted least square and piecewise bilinear transformation for accurate satellite DSM generation

One of the main products of multi-view stereo (MVS) high-resolution satellite (HRS) images in photogrammetry and remote sensing is digital surface model (DSM). Producing DSMs from MVS HRS images still faces serious challenges due to various reasons such as complexity of imaging geometry and exterior orientation model in HRS, as well as large dimensions and various geometric and illumination variations. The main motivation for conducting this research is to provide a novel and efficient method that enhances the accuracy and completeness of extracting DSM from HRS images compared to existing recent methods. The proposed method called Sat-DSM, consists of five main stages. Initially, a very dense set of tie-points is extracted from the images using a tile-based matching method, phase congruency-based feature detectors and descriptors, and a local geometric consistency correspondence method. Then, the process of Rational Polynomial Coefficients (RPC) block adjustment is performed to compensate the RPC bias errors. After that, a dense matching process is performed to generate 3D point clouds for each pair of input HRS images using a new geometric transformation called PWB (pricewise bilinear) and an accurate area-based matching method called SWLSM (selective weighted least square matching). The key innovations of this research include the introduction of SWLSM and PWB methods for an accurate dense matching process. The PWB is a novel and simple piecewise geometric transformation model based on superpixel over-segmentation that has been proposed for accurate registration of each pair of HRS images. The SWLSM matching method is based on phase congruency measure and a selection strategy to improve the well-known LSM (least square matching) performance. After dense matching process, the final stage is spatial intersection to generate 3D point clouds, followed by elevation interpolation to produce DSM. To evaluate the Sat-DSM method, 12 sets of MVS-HRS data from IRS-P5, ZY3-1, ZY3-2, and Worldview-3 sensors were selected from areas with different landscapes such as urban, mountainous, and agricultural areas The results indicate the superiority of the proposed Sat-DSM method over four other methods CATALYST, SGM (Semi-global matching), SS-DSM (structural similarity based DSM extraction), and Sat-MVSF in terms of completeness, RMSE, and MEE. The demo code is available at https://www.researchgate.net/publication/377721674_SatDSM.

3D point cloud regularization method for uniform mesh generation of mining excavations

Mine excavation systems are usually dozens of kilometers long with varying geometry on a small scale (roughness and shape of the walls) and on a large scale (varying widths of the tunnels, turns, and crossings). In this article, the authors address the problem of analyzing laser scanning data from large mining structures that can be used for various purposes, with focus on ventilation simulations. Together with the quality of the measurement data (diverse point-cloud density, missing samples, holes induced by obstructions in the field of view, measurement noise), this creates problems that require multi-stage processing of the obtained data. The authors propose a robust methodology to process a single segmented section of the mining system. The presented approach focuses on obtaining a point cloud ready for application in the computational fluid dynamics (CFD) analysis of airflow with minimal need for additional manual corrections on the generated mesh model. This requires the point cloud to have evenly distributed points and reduced noise (together with removal of objects inside) while keeping the unique geometrical properties and shape of the scanned tunnels. Proposed methodology uses trajectory of the excavation either obtained during the measurements or by skeletonization process explained in the article. Cross-sections obtained on planes perpendicular to the trajectory are processed towards the equalization of point distribution, removing measurement noise, holes in the point cloud and objects inside the excavation. The effects of the proposed algorithm are validated by comparing the processed cloud with the original cloud and testing within the CFD environment. The algorithm proved high effectiveness in improving skewness rate of the obtained mesh and geometry mapping accuracy (standard deviation below 5 centimeters in cloud-to-mesh comparison).

Improved 3D laser point cloud reconstruction for autonomous mobile robot applications by using SVM-R technique

PurposeIn autonomous mobile robots, high-level accuracy and precision in 3D perception are required for object detection, shape estimation and obstacle distance measurement. However, the existing methods suffer from limitations like inaccurate point clouds, noise in sensor data and synchronization problems between 2D LiDAR and servomotor. These factors can lead to the wrong perception and also introduce noise during sensor registration. Thus, the purpose of this study is to address these limitations and enhance the perception in autonomous mobile robots.Design/methodology/approachA new sensor mounting structure is developed for 3D mapping by using a 2D LiDAR and servomotor. The proposed method uses a support vector machine regression (SVM-R) technique to optimize the waypoints of the servomotor for the point cloud reconstruction process and to obtain a highly accurate and detailed representation of the environment.FindingsThe study includes an analysis of the SVM-R model, including Linear, radial basis function (RBF) and Polynomial kernel. Results show that the Linear kernel performs better with the lowest 3.67 mean absolute error (MAE), 26.24 mean squared error (MSE) and 5.12 root mean squared error (RMSE) values than the RBF and Polynomial kernels. The 2D to 3D point cloud reconstruction shows that the proposed method with a new sensor mounting structure performs better in perception accuracy and achieves an error of 0.45% in measuring the height of the target objects whereas in previous techniques the error was very large.Originality/valueThe study shows the effectiveness of SVM-R in the 3D point cloud reconstruction process and exhibits remarkable performance for object height measurement. Further, the proposed technique is applicable for future advanced visual applications and has a superior performance over the other conventional methods.

Upsampling Algorithm for V-PCC-Coded 3D Point Clouds

Point cloud (PC) compression is crucial to immersive visual applications such as autonomous vehicles to classify objects on the roads. The Motion Picture Experts Group (MPEG) standardization group has achieved a notable compression efficiency, called video-based PC compression (V-PCC), which consists of an encoder-decoder. The V-PCC encoder takes original 3D PC data and projects them onto multiple 2D planes to generate several 2D feature images. These images are then compressed using the well-established High-Efficiency Video Coding (HEVC) method. The V-PCC decoder uses compressed information and decoding techniques to reconstruct the 3D PC. However, the PCs produced by V-PCC are often sparse, non-uniform, and contain artifacts. In many practical applications, it is necessary to recover complete PCs from partial ones in real time. This article presents a method for enhancing decoded PCs as a post-processing step in the V-PCC with reduced computational time. Our approach involves a 2D upsampling for the V-PCC occupancy image, which increases the density of the PC, and a 2D high-resolution auxiliary information modification algorithm for the 2D-3D conversion of high-resolution 3D PCs, which improves the uniformity and reduces the noise in the PC. The 3D high-resolution PC has been further enhanced using the developed 3D outlier removal and point regeneration algorithm. Our proposed work can significantly simplify the state-of-the-art super resolution methods for PCs and reduce the time complexity of 61–75% while maintaining a high level of quality in PCs.

Robust 3D point clouds classification based on declarative defenders

Robust 3D point clouds classification based on declarative defenders

Study of the application of UAV and LiDAR to topographic survey of mountainous forest areas – Case study of Win 3 Wind Power Plant Project, Huong Hoa District, Quang Tri Province, Vietnam

Abstract Unmanned Aerial Vehicle (UAV) technology in conjunction with aerial photography and aerial Light Detection and Ranging (LiDAR) was used for topographic surveying of the Win 3 Wind Power Project area, a complex mountainous terrain situated in the Huong Hoa district of Quang Tri province, Vietnam with the primary objective to produce a map with a scale of 1:1,000 with a contour interval of 1.0 m, accompanied by 3D point clouds, a Digital Elevation Model (DEM), and an orthomosaic. Employing the UAV LiDAR system of DJI Matrice 300 RTK, we conduct ed survey flights, achieving a high point density laser scanning of 229 points/m2 and capturing ultra-high-resolution imagery at 4.22 cm/pixel. The integration of aerial LiDAR and photographic data results in a rich and detailed information repository regarding the surveyed terrain demonstrating that UAV LiDAR technology is a robust tool for geographic data collection, particularly in challenging terrain conditions in Vietnam. We also evaluated the accuracy of the created topographic map, achieving results that surpass the technical requirements and quality standards of the project. This underscores the potential and effectiveness of UAV LiDAR technology in the field of large-scale geographic data collection and mapping in Vietnam.

3D Point Cloud Fusion Method Based on EMD Auto-Evolution and Local Parametric Network

Although the development of high-resolution remote sensing satellite technology has made it possible to reconstruct the 3D structure of object-level features using satellite imagery, the results from a single reconstruction are often insufficient to comprehensively describe the 3D structure of the target. Therefore, developing an effective 3D point cloud fusion method can fully utilize information from multiple observations to improve the accuracy of 3D reconstruction. To this end, this paper addresses the problems of shape distortion and sparse point cloud density in existing 3D point cloud fusion methods by proposing a 3D point cloud fusion method based on Earth mover’s distance (EMD) auto-evolution and local parameterization network. Our method is divided into two stages. In the first stage, EMD is introduced as a key metric for evaluating the fusion results, and a point cloud fusion method based on EMD auto-evolution is constructed. The method uses an alternating iterative technique to sequentially update the variables and produce an initial fusion result. The second stage focuses on point cloud optimization by constructing a local parameterization network for the point cloud, mapping the upsampled point cloud in the 2D parameter domain back to the 3D space to complete the optimization. Through these two steps, the method achieves the fusion of two sets of non-uniform point cloud data obtained from satellite stereo images into a single, denser 3D point cloud that more closely resembles the true target shape. Experimental results demonstrate that our fusion method outperforms other classical comparison algorithms for targets such as buildings, planes, and ships, and achieves a fused RMSE of approximately 2 m and an EMD accuracy better than 0.5.

Influence of the Inclusion of Off-Nadir Images on UAV-Photogrammetry Projects from Nadir Images and AGL (Above Ground Level) or AMSL (Above Mean Sea Level) Flights

UAV-SfM techniques are in constant development to address the challenges of accurate and precise mapping in terrains with complex morphologies. In contrast with the traditional photogrammetric processes, where only nadir images were considered, the combination of those with oblique imagery, also called off-nadir, has emerged as an optimal solution to achieve higher accuracy in these kinds of landscapes. UAV flights at a constant height above ground level (AGL) have also been considered a possible alternative to improve the resulting 3D point clouds compared to those obtained from constant height above mean sea level (AMSL) flights. The aim of this study is to evaluate the effect of incorporating oblique images as well as the type of flight on the accuracy and precision of the point clouds generated through UAV-SfM workflows for terrains with complex geometries. For that purpose, 58 scenarios with different camera angles and flight patterns for the oblique images were considered, 29 for each type of flight (AMSL and AGL). The 3D point cloud derived from each of the 58 scenarios was compared with a reference 3D point cloud acquired with a terrestrial laser scanner (TLS). The results obtained confirmed that both incorporating oblique images and using AGL flight mode have a positive effect on the mapping. Combination of nadir image blocks, obtained from an AGL crosshatch flight plan, with supplemental oblique images collected with a camera angle of between 20° and 35° yielded the best accuracy and precision records.

Enhancing environmental modeling and 3D point cloud construction for unmanned vehicles using laser SLAM and stereo vision with Convolutional Neural Networks

Abstract. This research presents an innovative SLAM algorithm that integrates Convolutional Neural Networks (CNNs) with LIDAR and stereo vision to significantly enhance the accuracy of environmental modeling and the construction of dense 3D point cloud maps in complex and dynamic surroundings. By processing pre-recorded video data and employing advanced image segmentation techniques, this study achieves a deep fusion of visual and geometric data, resulting in highly detailed and precise 3D representations of the environment. The experimental results demonstrate that this approach effectively detects and excludes dynamic objects, thereby significantly improving the overall quality, robustness, and reliability of the constructed maps. This work represents a substantial advancement in SLAM technology, particularly in its capability to model and capture intricate environmental details under varying conditions, making it a powerful tool for precise environmental mapping.

Remote Sensing and GIS in Natural Resource Management: Comparing Tools and Emphasizing the Importance of In-Situ Data

Remote sensing (RS) and Geographic Information Systems (GISs) provide significant opportunities for monitoring and managing natural resources across various temporal, spectral, and spatial resolutions. There is a critical need for natural resource managers to understand the expanding capabilities of image sources, analysis techniques, and in situ validation methods. This article reviews key image analysis tools in natural resource management, highlighting their unique strengths across diverse applications such as agriculture, forestry, water resources, soil management, and natural hazard monitoring. Google Earth Engine (GEE), a cloud-based platform introduced in 2010, stands out for its vast geospatial data catalog and scalability, making it ideal for global-scale analysis and algorithm development. ENVI, known for advanced multi- and hyperspectral image processing, excels in vegetation monitoring, environmental analysis, and feature extraction. ERDAS IMAGINE specializes in radar data analysis and LiDAR processing, offering robust classification and terrain analysis capabilities. Global Mapper is recognized for its versatility, supporting over 300 data formats and excelling in 3D visualization and point cloud processing, especially in UAV applications. eCognition leverages object-based image analysis (OBIA) to enhance classification accuracy by grouping pixels into meaningful objects, making it effective in environmental monitoring and urban planning. Lastly, QGIS integrates these remote sensing tools with powerful spatial analysis functions, supporting decision-making in sustainable resource management. Together, these tools when paired with in situ data provide comprehensive solutions for managing and analyzing natural resources across scales.

Enhancing Fluvial Modeling and Flood Prediction Accuracy through the Fusion of 3D Point Clouds, Multispectral, and RGB Data

Enhancing Fluvial Modeling and Flood Prediction Accuracy through the Fusion of 3D Point Clouds, Multispectral, and RGB Data

GETr: A Geometric Equivariant Transformer for Point Cloud Registration

AbstractAs a fundamental problem in computer vision, 3D point cloud registration (PCR) aims to seek the optimal transformation to align point cloud pairs. Meanwhile, the equivariance lies at the core of matching point clouds at arbitrary pose. In this paper, we propose GETr, a geometric equivariant transformer for PCR. By learning the point‐wise orientations, we decouple the coordinate to the pose of the point clouds, which is the key to achieve equivariance in our framework. Then we utilize attention mechanism to learn the geometric features for superpoints matching, the proposed novel self‐attention mechanism encodes the geometric information of point clouds. Finally, the coarse‐to‐fine manner is used to obtain high‐quality correspondence for registration. Extensive experiments on both indoor and outdoor benchmarks demonstrate that our method outperforms various existing state‐of‐the‐art methods.

Empirical Evaluation and Simulation of the Impact of Global Navigation Satellite System Solutions on Uncrewed Aircraft System–Structure from Motion for Shoreline Mapping and Charting

Uncrewed aircraft systems (UASs) and structure-from-motion/multi-view stereo (SfM/MVS) photogrammetry are efficient methods for mapping terrain at local geographic scales. Traditionally, indirect georeferencing using ground control points (GCPs) is used to georeference the UAS image locations before further processing in SfM software. However, this is a tedious practice and unsuitable for surveying remote or inaccessible areas. Direct georeferencing is a plausible alternative that requires no GCPs. It relies on global navigation satellite system (GNSS) technology to georeference the UAS image locations. This research combined field experiments and simulation to investigate GNSS-based post-processed kinematic (PPK) as a means to eliminate or reduce reliance on GCPs for shoreline mapping and charting. The study also conducted a brief comparison of real-time network (RTN) and precise point positioning (PPP) performances for the same purpose. Ancillary experiments evaluated the effects of PPK base station distance and GNSS sample rate on the accuracy of derived 3D point clouds and digital elevation models (DEMs). Vertical root mean square errors (RMSEz), scaled to the 95% confidence interval using an assumption of normally-distributed errors, were desired to be within 0.5 m to satisfy National Oceanic and Atmospheric Administration (NOAA) requirements for nautical charting. Simulations used a Monte Carlo approach and empirical tests to examine the influence of GNSS performance on the quality of derived 3D point clouds. RTN and PPK results consistently yielded RMSEz values within 10 cm, thus satisfying NOAA requirements for nautical charting. PPP did not meet the accuracy requirements but showed promising results that prompt further investigation. PPK experiments using higher GNSS sample rates did not always provide the best accuracies. GNSS performance and model accuracies were enhanced when using base stations located within 30 km of the survey site. Results without using GCPs observed a direct relationship between point cloud accuracy and GNSS performance, with R2 values reaching up to 0.97.

Is it possible to generate accurate 3D point clouds with UAS-LIDAR and UAS-RGB photogrammetry without GCPs? A case study on a beach and rocky cliff

ContextRecently, Unoccupied Aerial Systems (UAS) with photographic or Light Detection and Ranging (LIDAR) sensors have incorporated on-board survey-grade Global Navigation Satellite Systems that allow the direct georeferencing of the resulting datasets without Ground Control Points either in Real-Time (RTK) or Post-Processing Kinematic (PPK) modes. These approaches can be useful in hard-to-reach or hazardous areas. However, the resulting 3D models have not been widely tested, as previous studies tend to evaluate only a few points and conclude that systematic errors can be found.ObjectivesWe test the absolute positional accuracy of point clouds produced using UAS with direct-georeferencing systems.MethodsWe test the accuracy and characteristics of point clouds produced using a UAS-LIDAR (with PPK) and a UAS-RGB (Structure-from-Motion or SfM photogrammetry with RTK and PPK) in a challenging environment: a coastline with a composite beach and cliff. The resulting models of each processing were tested using as a benchmark a point cloud surveyed simultaneously by a Terrestrial Laser Scanner.ResultsThe UAS-LIDAR produced the most accurate point cloud, with homogeneous cover and no noise. The systematic bias previously observed in the UAS-RGB RTK approaches are minimized using oblique images. The accuracy observed across the different surveyed landforms varied significantly.ConclusionsThe UAS-LIDAR and UAS-RGB with PPK produced unbiased point clouds, being the latter the most cost-effective method. For the other direct georeferencing systems/approaches, the acquisition of GCP or the co-registration of the resulting point cloud is still necessary.

Attention Mechanisms in PointNet++ for Effective Object Classification in 3D Point Clouds

Attention Mechanisms in PointNet++ for Effective Object Classification in 3D Point Clouds

A novel method: YOLO-CE and 3D point cloud-based feature extraction for welding seams of tower bases

Abstract Robotic automated welding of non-standard steel structures presents significant challenges, particularly for electric power tower bases. This study introduces a novel approach that integrates the You Only Look Once—Compact Invert Block and Efficient Local Attention (YOLO-CE) model, an enhanced version of YOLOV8 for 2D image segmentation, with 3D point cloud technology. The YOLO-CE model is used to accurately extract point cloud data from the target area, which is then processed using the MSAC algorithm for efficient plane segmentation. Weld lines are identified through plane equations, allowing for initial weld point cloud extraction. To further refine accuracy, an optimized evaluation equation is developed that accounts for both the distance between the weld point cloud and the fitted plane, and the angle between their normal vectors. This enables precise classification of the weld point cloud. From this classification, key weld feature points are identified, and their exact positions are determined by calculating the distances between these points and their intersections with three planes. The reliability of the proposed method was validated using a robot for precise measurements, with a total error margin of less than 1.5084 mm, demonstrating high accuracy and stability. Post-operation inspections confirmed that the welds were filled and free from defects, meeting all process requirements. The YOLO-CE model achieved a mIoU of 96.38% and a precision of 99.8%, highlighting its effectiveness. This method provides an efficient and precise solution for the automated welding of non-standard steel structural components and has promising application potential.

3D morphometry of Martian craters from HRSC DEMs using a multi-scale semantic segmentation network and morphological analysis

The morphology of impact craters reveals the structure and composition of the Martian surface, especially the subsurface conditions and Martian geological history, which have increasing importance in Mars exploration missions. This work presents a 3D morphometric method for detecting and delineating Martian craters, and a 3D morphological analysis was conducted. Specifically, this work first focused on the segmentation of Martian craters. Based on the segmentation results, clustering of crater instances was then carried out. Finally, with the individual craters that were extracted, a morphological analysis involving the measurements of their diameter, depth, area, RMS height, rim height, circularity, and the statistics thereof was performed. Unlike previous studies, which have mainly used optical images and object detection approaches, this work regards crater extraction as a semantic segmentation task instead of an object detection task to better delineate the precise shape and boundary information. Digital elevation model (DEM) was utilized as primary data to directly obtain 3D information, which was converted into 3D point cloud format and fed to a multi-scale semantic segmentation network. The semantic segmentation results achieved an overall accuracy of 0.932 and mIOU of 0.871 on the test data. We automatically counted 63 craters in Noachis Terra and 40 craters in Terra Cimmeria. The 3D morphological measurements showed that 66% of the impact craters in the first region were larger than 10 km in diameter, while 50% of the impact craters in the second region were larger than 10 km. In both areas, craters could reach a maximum depth of 2000 m. With the proposed method, we can automatically conduct 3D morphological measurements of Martian craters with high efficiency that is improved by 15 times compared with that of manual crater analysis tools. The achieved 3D morphometric results can provide a reference and support for future research on Martian landforms.

3D dataset generation using virtual reality for forest biodiversity

ABSTRACT While forest biodiversity faces a concerning decline, modern technology presents promising avenues for mitigation. However, a critical gap persists in reconciling ecological knowledge with the technical expertise required to use state-of-the-art technologies in 3D data classification. Currently, one main issue is the scarcity of 3D datasets for biodiversity, particularly within the context of machine learning applications. Unlike the straightforward classification of human-made structures, forest environments are uniquely intricate and nuanced due to its inherently complex nature. This study addresses this challenge by introducing a fully automated pipeline for tree stem 3D point cloud segmentation, focussing on a biodiversity indicator: tree-related microhabitats (TreMs). Furthermore, our research advances the field by demonstrating that machine learning models trained with labels generated by our proposed virtual reality (VR) method, Labelling Flora, yield predictions statistically similar to the traditional desktop-based labelling methods. This implies that existing 3D datasets could be augmented using the more rapid approach of VR labelling. Additionally, the findings of this paper demonstrate the potential integration of VR and immersive technology into the 3D labelling workflow, facilitating a quicker and more intuitive labelling process. This could empower users, who are non-familiar with 3D modelling, to contribute their expertise to the segmentation process.