3D Point Cloud Research Articles

Automatic detection of color markings and numbers on trees in point clouds from Personal Laser Scanning (PLS) and Terrestrial Laser Scanning (TLS)

In forestry, colored spray paint is used for different purposes, such as marking trees that are chosen for the next harvest or that will be promoted by future silvicultural activities, or for the demarcation of skid/cable roads. Numerous studies have demonstrated the application of Terrestrial Laser Scanners (TLS) and Personal Laser Scanners (PLS) for the acquisition of tree parameters (diameters, heights, and tree positions) in a forest inventory context. However, no studies have included classification based on spectral information by applying colored markings to trees. When a laser scanner is combined with a camera, the color information can be mapped onto the laser point cloud, allowing for additional analysis of spectral information. In this study, we have tested the analysis of joint information from imagery and 3D point clouds in two different settings: (i) PLS point cloud data from a GeoSLAM ZEB Horizon were complemented with spectral information from images collected with a NCTech iSTAR Pulsar 360-degree camera, (ii) TLS point cloud data from a RIEGL VZ-400i were combined with imagery from a Nikon D850 digital single lens reflex (DSLR) camera with 14 mm fisheye lens. The major objective was to automatically classify trees that were marked with colored sprays automatically identify the numbers sprayed on the trees. The marked trees could be classified correctly with a balanced accuracy of 77.26% for PLS and 98.01% for the data collected with the TLS system using a support vector machine (SVM) model. The numbers written on the trees were classified from the TLS data using a YOLOv8 model. The accuracy of the digit classification reached up to 97.30%. In conclusion, it is possible to automatically detect marked trees and to automatically recognize the numbers spayed on these trees from colored point cloud data. However, accuracy strongly depends on the selected combination of scanner and camera. The findings of future research on this subject might differ with the ongoing development of PLS systems and could be enhanced by creating a large dataset of handwritten numbers collected using laser scanning systems.

Fuzzy preference matroids rough sets for approximate guided representation in transformer

Recently, the transformer has exhibited remarkable performance across various applications, primarily owing to its exceptional capability in capturing global information through the attention mechanism. Nevertheless, the dot-product within attention results in inaccurate and confusing guidance in the presence of uncertain features, subsequently affecting robustness. Hence, introducing an approximate guided method to handle uncertain features is essential for achieving enhanced objectivity. In this study, we employed fuzzy preference relations to construct a matroids structure, introducing it into rough sets to form the fuzzy preference matroids rough sets. Detailed mathematical properties and axiomatic proofs are presented. The closed sets within the matroids form a subspace of the lower approximation proving that the proposed satisfy the approximation ability of rough sets. Further, an approximate guided representation method with a lower approximation of the matroids has been developed. It is integrated into a plug-and-play transformer block that can be flexibly deployed across various tasks. Experimental validation has demonstrated the outstanding performance of this method across 3D point cloud classification, named entity recognition, multimodal emotion recognition, and image classification. Specifically, on the Weibo NER and the IEMOCAP datasets, attain state-of-the-art (SOTA) F1-scores. The resource code is validated at https://github.com/WinnieSunning/FPMRST.

Language-Level Semantics-Conditioned 3D Point Cloud Segmentation

In this work, a language-level Semantics-Conditioned framework for 3D Point cloud segmentation, called SeCondPoint, is proposed, where language-level semantics are introduced to condition the modeling of the point feature distribution, as well as the pseudo-feature generation, and a feature–geometry-based Mixup approach is further proposed to facilitate the distribution learning. Since a large number of point features could be generated from the learned distribution thanks to the semantics-conditioned modeling, any existing segmentation network could be embedded into the proposed framework to boost its performance. In addition, the proposed framework has the inherent advantage of dealing with novel classes, which seems an impossible feat for the current segmentation networks. Extensive experimental results on two public datasets demonstrate that three typical segmentation networks could achieve significant improvements over their original performances after enhancement by the proposed framework in the conventional 3D segmentation task. Two benchmarks are also introduced for a newly introduced zero-shot 3D segmentation task, and the results also validate the proposed framework.

Adaptive Weighted Data Fusion for Line Structured Light and Photometric Stereo Measurement System.

Line structured light (LSL) measurement systems can obtain high accuracy profiles, but the overall clarity relies greatly on the sampling interval of the scanning process. Photometric stereo (PS), on the other hand, is sensitive to tiny features but has poor geometrical accuracy. Cooperative measurement with these two methods is an effective way to ensure precision and clarity results. In this paper, an LSL-PS cooperative measurement system is brought out. The calibration methods used in the LSL and PS measurement system are given. Then, a data fusion algorithm with adaptive weights is proposed, where an error function that contains the 3D point cloud matching error and normal vector error is established. The weights, which are based on the angles of adjacent normal vectors, are also added to the error function. Afterward, the fusion results can be obtained by solving linear equations. From the experimental results, it can be seen that the proposed method has the advantages of both the LSL and PS methods. The 3D reconstruction results have the merits of high accuracy and high clarity.

Improving Mineral Classification Using Multimodal Hyperspectral Point Cloud Data and Multi-Stream Neural Network

In this paper, we leverage multimodal data to classify minerals using a multi-stream neural network. In a previous study on the Tinto dataset, which consisted of a 3D hyperspectral point cloud from the open-pit mine Corta Atalaya in Spain, we successfully identified mineral classes by employing various deep learning models. However, this prior work solely relied on hyperspectral data as input for the deep learning models. In this study, we aim to enhance accuracy by incorporating multimodal data, which includes hyperspectral images, RGB images, and a 3D point cloud. To achieve this, we have adopted a graph-based neural network, known for its efficiency in aggregating local information, based on our past observations where it consistently performed well across different hyperspectral sensors. Subsequently, we constructed a multi-stream neural network tailored to handle multimodality. Additionally, we employed a channel attention module on the hyperspectral stream to fully exploit the spectral information within the hyperspectral data. Through the integration of multimodal data and a multi-stream neural network, we achieved a notable improvement in mineral classification accuracy: 19.2%, 4.4%, and 5.6% on the LWIR, SWIR, and VNIR datasets, respectively.

Bijective upsampling and learned embedding for point clouds correspondences

In this paper, we present a novel pipeline to compute and refine a data-driven solution for estimating the correspondence between 3D point clouds. Our method is compatible with the functional map framework, so it relies on a functional representation of the correspondence, but, differently from other similar approaches, this method is specifically designed to exploit this functional scenario for point cloud matching. Our new method merges a data-driven approach to compute functional basis and descriptors on the shape’s surface and a new refinement method designed for the learned basis. This refinement algorithm arises from a different way of converting functional operators into point-to-point correspondence, which we prove to promote bijectivity between maps, exploiting a theoretical result. Iterating this procedure and performing basis upsampling in the same way as other similar methods, ours increases the accuracy of the correspondence, leading to more bijective correspondences. Different from other approaches, our method allows us to train a functional basis, considering the refinement stage. Combining our new pipeline with an improved feature extractor, our solution outperforms previous methods in various evaluations and settings. We test our method over different datasets, comprising near-isometric and non-isometric pairs.

Accurately identifying the defects of bubbles and foreign objects under the protective films of electric vehicle batteries by using 3D point clouds

Abstract For the defects of bubbles and foreign objects under the protective film of electric vehicle batteries, it is difficult to accurately identify them over traditional 2D optical images. In this paper, we first propose a supervoxel-based region growing algorithm for pre-segmentation of point clouds. Secondly, we utilize radial basis function interpolation and threshold segmentation methods to accurately segment defect point clouds from the entire point cloud. Finally, we develop a feature descriptor and combine it with support vector machine to classify bubbles and foreign objects under the film. This paper achieves the identification of bubbles and foreign objects under the film through two steps: point cloud segmentation and point cloud classification. Experimental results demonstrate that the proposed point cloud segmentation method exhibits high robustness to noise and the intrinsic curvature of the workpiece. Additionally, in the classification scenario presented in this paper, the proposed feature descriptor outperforms classical feature descriptors. Compared to image-based deep learning methods, the defect recognition algorithm proposed in this paper has clear principles and superior performance, with precision and recall of 95.63% and 96.95%, and an intersection over union metric of 0.926.

Designing and evaluation of a mixed reality system for crime scene investigation training: a hybrid approach

Police investigation in real-life crime scenes is an essential aspect of forensic science education. However, the practicality of bringing young investigators to actual crime scenes is often hindered by the costs and challenges involved. In order to overcome these obstacles, new technologies such as mixed reality (MR) are being explored as potential solutions. MR technology offers an interactive and cost-effective way to simulate real-life crime scenes, providing a valuable training experience for young investigators. This paper presents a novel design of a MR system using Microsoft HoloLens 2.0, which is tailored to work in a spatial 3D scanned and reconstructed crime scene using FARO point cloud 3D scanner X130 blended with photogrammetry techniques. The system was developed through the lens of Experiential Learning Theory and designed using a participatory approach, providing a cost-effective solution to help trained Kuwaiti police officers enhance their investigative skills. In order to evaluate the system’s user experience and user interaction, the Questionnaire of User Interaction Satisfaction and User Experience Questionnaire were utilised. Forty-four young police officers evaluated the system. Police students showed positive levels of satisfaction with user interaction and overall user experience with minimal negative feedback. Female students showed higher satisfaction with the overall impression compared to male students. Based on the positive feedback regarding the system expansion, the system will be taken into the commercialisation stage in the future to be provided as an essential tool for crime scene education and investigation practices.

FeatSync: 3D point cloud multiview registration with attention feature-based refinement

Current studies often decompose multiview registration into several individual tasks, ignoring the correlation between each stage and making certain assumptions on noise distribution without knowledge from previous stages. These issues bring difficulties in generalization to real cases. In this paper, we propose an end-to-end feature-based multiview registration model that takes a set of raw 3D point cloud fragments as input and outputs the global transformation. Unlike previous works, our method allows the exchange of information between stages. We firstly estimate pairwise registration by a attention-based model to assist feature learning. In the next stage, we utilize iteratively reweighted least squares (IRLS) algorithm to refine and obtain the global transformation. In each iteration, instead of making assumptions on noises, we directly construct a model to infer the outliers from pairwise registration so that such an inference can help synchronization produce more reliable results. To follow the process in IRLS algorithm, we propose a simple yet effective refinement module to boost feature-based pairwise estimations in an iterative manner, which can be seamlessly integrated into the IRLS procedure. Extensive experiments conducted on benchmark datasets show that the results of our proposed method outperformed existing methods.

Beyond local patches: Preserving global–local interactions by enhancing self-attention via 3D point cloud tokenization

Transformer-based architectures have recently shown impressive performance on various point cloud understanding tasks such as 3D object shape classification and semantic segmentation. Particularly, this can be attributed to their self-attention mechanism, which has the ability to capture long-range dependencies. However, current methods have constrained it to operate in local patches due to its quadratic memory constraints. This hinders their generalization ability and scaling capacity due to the loss of non-locality in early layers. To tackle this issue, we propose a window-based transformer architecture that captures long-range dependencies while aggregating information in the local patches. We do this by interacting each window with a set of global point cloud tokens — a representative subset of the entire scene — and augmenting the local geometry through a 3D Histogram of Oriented Gradients (HOG) descriptor. Through a series of experiments on segmentation and classification tasks, we show that our model exceeds the state-of-the-art on S3DIS semantic segmentation (+1.67% mIoU), ShapeNetPart part segmentation (+1.03% instance mIoU) and performs competitively on ScanObjectNN 3D object classification.11The code and trained models shall be made publicly available.

Modelowanie cyfrowe w celu przewidywania zagrożenia osuwania się gruntu na wybranym obszarze; Przypadek testowy Ponzano (Włochy)

Creating a digital model is one of the aims of the geotechnical engineers, to predict the land sliding hazard which occur in different regions in the world. In February 2017, an extensive landslide occurred in the hamlet of Ponzano as a sloped area in the Abruzzo region in Italy. In this regard, predicting the land sliding hazard is one of the important issues to prevent hazard to the civilizations. In this project we created a model in the 3D dimension through the Plaxis 3D numerical solution software from the data based of the region Abruzzo in 2007 to evaluate the land sliding hazard before happened and then compare the results with the data from the drone data surveyed recently in the 2022. In this regard, the data from the Abruzzo resources from 2007 imported into the QGIS as the open-source cross-platform software to analysis the geospatial data and then imported into the Recap software to work on the point cloud data and then imported into the Civil 3D software to create a solid surface from the TIN surface and finally since the solid surface contains a large number of irrelative details, they were imported into the Rhinoceros software to create a NURBS surface to be smoothed for better performance in the analysis. The NURBS surface imported into the Plaxis and all of the geometry and geotechnical engineering parameters by considering the investigated geotechnical survey that was conducted in parallel in the area, defined for the model. The “Hardening Soil” model considered for the 1st layer as the “clay and lime” and the “Hoek-Brown” model defined for the 2nd layer as the “marl-flysch”. A fine mesh elements distribution assessed also for the model. The phases defined as the “gravity” to define the unit weight of the soil layers, the “plastic” phase to calculate the instant deformations and the “consolidation” to analyze the plastic deformations in the sloped area of the model. In parallel, the drone data achieved in 2022, were imported into the CloudCompare as the 3D point cloud processing software and different methods such as the “segment”, “statistical outlier Filter”, “CSF filter”, “noise filter”, “cross section” etc. were performed to clean the data and then imported into the Recap software to work on data and then imported into the Civil 3D software to create solid surface of the current data after the land sliding. In this regard, to evaluate the displacement occurred from the year 2007 toward the 2022, a TIN volume surface as the colored map created through the Civil 3D software to show the displacements in the z direction and all of the results were compared with the Plaxis 3D numerical solution software. The results showed that the colored map with the displacement in the positive and negative direction of the z is the same of the analyzed model and the values match each other’s and we created a digital model of the selected area to predict the land sliding hazard in the region in the following.

Proficient Calibration Methodologies for Fixed Photogrammetric Monitoring Systems

This work focuses on investigating the accuracy of 3D reconstructions from fixed stereo-photogrammetric monitoring systems through different camera calibration procedures. New reliable and effective calibration methodologies that require minimal effort and resources are presented. A full-format camera equipped with fixed 50 and 85 mm focal length optics is considered, but the methodologies are general and can be applied to other systems. Four different calibration strategies are considered: (i) full-field calibration (FF); (ii) multi-image on-the-job calibration (MI); (iii) point cloud-based calibration (PC); and (iv) self (on-the-job) calibration (SC). To evaluate the calibration strategies and assess their actual performance and practicality, two test sites are used. The full-field calibration, while very reliable, demands significant effort if it needs to be repeated. The multi-image strategy emerges as a favourable compromise, offering good results with minimal effort for its realisation. The point cloud-based method stands out as the optimal choice, balancing ease of implementation with quality results; however, it requires a reference 3D point cloud model. On-the-job calibration with monitoring images is the simplest but least reliable option, prone to uncertainty and potential inaccuracies, and should hence be avoided. Ultimately, prioritising result reliability over absolute accuracy is paramount in continuous monitoring systems.

Intracranial aneurysm detection based on 3D point cloud object detection method

Intracranial aneurysm detection based on 3D point cloud object detection method

Distributed Batch Learning of Growing Neural Gas for Quick and Efficient Clustering

Growing neural gas (GNG) has been widely used in topological mapping, clustering and unsupervised tasks. It starts from two random nodes and grows until it forms a topological network covering all data. The time required for growth depends on the total amount of data and the current network nodes. To accelerate growth, we introduce a novel distributed batch processing method to extract the rough distribution called Distributed Batch Learning Growing Neural Gas (DBL-GNG). First, instead of using a for loop in standard GNG, we adopt a batch learning approach to accelerate learning. To do this, we replace most of the standard equations with matrix calculations. Next, instead of starting with two random nodes, we start with multiple nodes in different distribution areas. Furthermore, we also propose to add multiple nodes to the network instead of adding them one by one. Finally, we introduce an edge cutting method to reduce unimportant links between nodes to obtain a better cluster network. We demonstrate DBL-GNG on multiple benchmark datasets. From the results, DBL-GNG performs faster than other GNG methods by at least 10 times. We also demonstrate the scalability of DBL-GNG by implementing a multi-scale batch learning process in it, named MS-DBL-GNG, which successfully obtains fast convergence results. In addition, we also demonstrate the dynamic data adaptation of DBL-GNG to 3D point cloud data. It is capable of processing and mapping topological nodes on point cloud objects in real time.

Development of large-scale synthetic 3D point cloud datasets for vision-based bridge structural condition assessment

Visual recognition of 3D point cloud data of bridge inspection scenes is a key step in automating the visual inspection process, which is currently largely manual and inefficient. To alleviate the lack of large-scale annotated point cloud datasets for training such 3D visual recognition algorithms, this research investigates an approach for developing large-scale synthetic point cloud datasets. The proposed approach proceeds in four steps: (1) random generation of different types of bridges in computer graphics environments; (2) sampling of camera trajectories that represent data collection scenarios during bridge inspection; (3) 3D reconstruction using Structure from Motion (SfM) applied to rendered synthetic images; (4) automated annotation of the reconstructed point cloud using ground truth masks obtained with synthetic images. Besides, this research proposes to store point uncertainty information defined by the error between the ground truth depth and the depth calculated from the SfM results. Prior to training, thresholds can be applied to this uncertainty information to control the levels of outliers in the dataset. This research demonstrates the proposed approach by generating point cloud datasets for two data collection scenarios. The effectiveness of the generated datasets is investigated by training 3D semantic segmentation algorithms and evaluating the performance on real and synthetic point cloud data. The proposed approach for point cloud dataset generation will facilitate the development of generalizable and high level-of-detail 3D recognition algorithms toward autonomous bridge inspection.

ModelNet-O: A large-scale synthetic dataset for occlusion-aware point cloud classification

Recently, 3D point cloud classification has made significant progress with the help of many datasets. However, these datasets do not reflect the incomplete nature of real-world point clouds caused by occlusion, which limits the practical application of current methods. To bridge this gap, we propose ModelNet-O, a large-scale synthetic dataset of 123,041 samples that emulates real-world point clouds with self-occlusion caused by scanning from monocular cameras. ModelNet-O is 10 times larger than existing datasets and offers more challenging cases to evaluate the robustness of existing methods. Our observation on ModelNet-O reveals that well-designed sparse structures can preserve structural information of point clouds under occlusion, motivating us to propose a robust point cloud processing method that leverages a critical point sampling (CPS) strategy in a multi-level manner. We term our method PointMLS. Through extensive experiments, we demonstrate that our PointMLS achieves state-of-the-art results on ModelNet-O and competitive results on regular datasets such as ModelNet40 and ScanObjectNN, and we also demonstrate its robustness and effectiveness. Code available: https://github.com/fanglaosi/ModelNet-O_PointMLS.

RailPC: A large‐scale railway point cloud semantic segmentation dataset

AbstractSemantic segmentation in the context of 3D point clouds for the railway environment holds a significant economic value, but its development is severely hindered by the lack of suitable and specific datasets. Additionally, the models trained on existing urban road point cloud datasets demonstrate poor generalisation on railway data due to a large domain gap caused by non‐overlapping special/rare categories, for example, rail track, track bed etc. To harness the potential of supervised learning methods in the domain of 3D railway semantic segmentation, we introduce RailPC, a new point cloud benchmark. RailPC provides a large‐scale dataset with rich annotations for semantic segmentation in the railway environment. Notably, RailPC contains twice the number of annotated points compared to the largest available mobile laser scanning (MLS) point cloud dataset and is the first railway‐specific 3D dataset for semantic segmentation. It covers a total of nearly 25 km railway in two different scenes (urban and mountain), with 3 billion points that are finely labelled as 16 most typical classes with respect to railway, and the data acquisition process is completed in China by MLS systems. Through extensive experimentation, we evaluate the performance of advanced scene understanding methods on the annotated dataset and present a synthetic analysis of semantic segmentation results. Based on our findings, we establish some critical challenges towards railway‐scale point cloud semantic segmentation. The dataset is available at https://github.com/NNU‐GISA/GISA‐RailPC, and we will continuously update it based on community feedback.

Distributed collaborative inspections through smart infrastructure metaverse

We propose a real-time distributed collaborative system where on- and off-site inspectors perform synchronous structural inspections; called Smart Infrastructure Metaverse. Mixed Reality (MR) headsets are equipped with a holographic display that allows on-site inspectors to visualize digitized information overlaid on the structure. On the other hand, Virtual Reality (VR) headsets allow remote inspectors to visualize and interact with a pre-built 3D point cloud map. The MR inspectors execute inspection tasks such as adding annotations or taking measurements via MR interfaces, while the VR inspectors also perform the same tasks in a virtual environment, while the scene of the structure is viewed through 360°panoramic images aligned with the pre-built map using a proposed multi-shot localization algorithm. An experimental study has demonstrated the feasibility of SIM on a railroad bridge where two on-site MR inspectors and one remote VR inspector perform a visual inspection.

A clearance control and interference repair method for ship rib plate assembly based on reverse engineering

Due to the influence of manufacturing errors, welding deformation of ship parts and other factors, interference often occurs due to out-of-tolerance assembly clearance for ship rib plate assembly using the pull-in method. To solve the problem, a clearance control and interference repair method for ship rib plate assembly is proposed. Firstly, the factors causing ship rib assembly errors were analyzed, and an accurate model of the ship rib with manufacturing errors was built based on a 3D scanning point cloud. Secondly, the fit clearances between multiple components during the motion process were calculated based on simulation. Finally, the assembly clearances were redistributed through pose coordination technology to autonomously control the assembly clearance, allocate excess clearances to interference areas and quantitatively complete the subsequent interference processing. The case study results show that the proposed method can achieve quantitative prediction of assembly clearances and generate a precise repair scheme. The interference for ship rib plate assembly is reduced by 50%, and the efficiency is increased by 24%.

RCFI-Net: A reliable correspondences evaluation and feature interaction network for fast and accurate point cloud registration

3D point cloud registration is a crucial task in computer vision, robotics, and safeguarding cultural artifacts in the digital realm. However, maintaining a balance between efficiency and accuracy during the 3D point cloud registration process continues to be a challenge. To solve this issue, a Reliable Correspondences Evaluation and Feature Interaction Network called RCFI-Net has been proposed for fast and precise registration. Our model use a two-stage approach to improve registration efficiency. In the first stage, a Transformer with position encoding network is employed to reinforce point features. Overlapping masks are learned based on the attention mechanism to identify overlapping areas, followed by the sampling of interest points that have high scores in these areas to accelerate registration. In the second stage, non-distinguishing points are eliminated, and a triangulated descriptor is introduced to further differentiate inliers and outliers and find reliable correspondences. Our proposed method has been evaluated on both public datasets and real Terracotta Warriors data, with results showing that it outperforms traditional and feature-learning methods in terms of accuracy, efficiency, and robustness.