Color Point Cloud Research Articles

Automated Defect Detection on Dry-Hanging Stone Curtain Walls through Colored Point Clouds

Stone curtain walls are widely used in contemporary architectures; however, their regular inspection is always labor-intensive, time-consuming, and hazardous due to the complex and enclosed spatial structure of these high-rise building enclosures. To address this issue, this study proposes an automated and novel inspection method, which is composed of the following three steps: First, we utilize 3D laser scanning technology to capture colored point cloud data of the stone curtain wall system; subsequently, by extracting and processing the integration of color and depth information, the stone panels and end sealants are precisely segmented; finally, various defects, such as cracks, unevenness, and irregularities, are automatically identified through artificial intelligence algorithms in a timely manner. To validate the proposed method, an on-site experiment was carried out to demonstrate the effectiveness in detecting multiple defects concurrently on stone curtain walls. The experimental results showed that our proposed method could provide a non-contact and automated inspection alternative for all the stone curtain walls with a high accuracy of anomaly detection, facilitating rational maintenance plans and strategies to ensure the safety and performance of these modern building enclosures.

Automatic detection of color markings and numbers on trees in point clouds from Personal Laser Scanning (PLS) and Terrestrial Laser Scanning (TLS)

In forestry, colored spray paint is used for different purposes, such as marking trees that are chosen for the next harvest or that will be promoted by future silvicultural activities, or for the demarcation of skid/cable roads. Numerous studies have demonstrated the application of Terrestrial Laser Scanners (TLS) and Personal Laser Scanners (PLS) for the acquisition of tree parameters (diameters, heights, and tree positions) in a forest inventory context. However, no studies have included classification based on spectral information by applying colored markings to trees. When a laser scanner is combined with a camera, the color information can be mapped onto the laser point cloud, allowing for additional analysis of spectral information. In this study, we have tested the analysis of joint information from imagery and 3D point clouds in two different settings: (i) PLS point cloud data from a GeoSLAM ZEB Horizon were complemented with spectral information from images collected with a NCTech iSTAR Pulsar 360-degree camera, (ii) TLS point cloud data from a RIEGL VZ-400i were combined with imagery from a Nikon D850 digital single lens reflex (DSLR) camera with 14 mm fisheye lens. The major objective was to automatically classify trees that were marked with colored sprays automatically identify the numbers sprayed on the trees. The marked trees could be classified correctly with a balanced accuracy of 77.26% for PLS and 98.01% for the data collected with the TLS system using a support vector machine (SVM) model. The numbers written on the trees were classified from the TLS data using a YOLOv8 model. The accuracy of the digit classification reached up to 97.30%. In conclusion, it is possible to automatically detect marked trees and to automatically recognize the numbers spayed on these trees from colored point cloud data. However, accuracy strongly depends on the selected combination of scanner and camera. The findings of future research on this subject might differ with the ongoing development of PLS systems and could be enhanced by creating a large dataset of handwritten numbers collected using laser scanning systems.

Probability-Based LIDAR-Camera Calibration Considering Target Positions and Parameter Evaluation Using a Data Fusion Map.

The data fusion of a 3-D light detection and ranging (LIDAR) point cloud and a camera image during the creation of a 3-D map is important because it enables more efficient object classification by autonomous mobile robots and facilitates the construction of a fine 3-D model. The principle behind data fusion is the accurate estimation of the LIDAR-camera's external parameters through extrinsic calibration. Although several studies have proposed the use of multiple calibration targets or poses for precise extrinsic calibration, no study has clearly defined the relationship between the target positions and the data fusion accuracy. Here, we strictly investigated the effects of the deployment of calibration targets on data fusion and proposed the key factors to consider in the deployment of the targets in extrinsic calibration. Thereafter, we applied a probability method to perform a global and robust sampling of the camera external parameters. Subsequently, we proposed an evaluation method for the parameters, which utilizes the color ratio of the 3-D colored point cloud map. The derived probability density confirmed the good performance of the deployment method in estimating the camera external parameters. Additionally, the evaluation quantitatively confirmed the effectiveness of our deployments of the calibration targets in achieving high-accuracy data fusion compared with the results obtained using the previous methods.

Robust colored point cloud alignment based on L*a*b* guided and Cauchy kernel

AbstractPrecision agriculture benefits from point set registration, which can monitor plant health and growth in real time, promote the precise application of fertilizers and pesticides, and provide technical support for achieving sustainable development of agriculture. In this work, we propose a robust point set registration method for precision agriculture based on L*a*b* color guidance, bidirectional search and Cauchy distribution. First, the L*a*b* color guidance is applied to establish accurate correspondences between agricultural RGB‐D data. Second, the bidirectional nearest neighbor search strategy between point sets improves the reliability of establishing correspondences and broadens the convergence domain of the algorithm. Third, Cauchy distribution is utilized as an energy function for noise suppression, which further improves the robustness of the algorithm in dealing with complex vegetation scenes. Finally, results of ablation and simulation experiments indicate that the proposed registration algorithm can achieve more accurate and robust alignment results than other classic and state‐of‐the‐art point cloud registration algorithms to achieve monitoring and comparison of plant growth.

Point Cloud Quality Assessment Using a One-Dimensional Model Based on the Convolutional Neural Network.

Recent advancements in 3D modeling have revolutionized various fields, including virtual reality, computer-aided diagnosis, and architectural design, emphasizing the importance of accurate quality assessment for 3D point clouds. As these models undergo operations such as simplification and compression, introducing distortions can significantly impact their visual quality. There is a growing need for reliable and efficient objective quality evaluation methods to address this challenge. In this context, this paper introduces a novel methodology to assess the quality of 3D point clouds using a deep learning-based no-reference (NR) method. First, it extracts geometric and perceptual attributes from distorted point clouds and represent them as a set of 1D vectors. Then, transfer learning is applied to obtain high-level features using a 1D convolutional neural network (1D CNN) adapted from 2D CNN models through weight conversion from ImageNet. Finally, quality scores are predicted through regression utilizing fully connected layers. The effectiveness of the proposed approach is evaluated across diverse datasets, including the Colored Point Cloud Quality Assessment Database (SJTU_PCQA), the Waterloo Point Cloud Assessment Database (WPC), and the Colored Point Cloud Quality Assessment Database featured at ICIP2020. The outcomes reveal superior performance compared to several competing methodologies, as evidenced by enhanced correlation with average opinion scores.

3D colored object reconstruction from a single view image through diffusion

In this paper we propose a novel 3D colored object reconstruction method from a single view image. Given a reference image, a conditional diffusion probabilistic model is built to reconstruct both a 3D point cloud shape and the corresponding color features at each point, and then images from arbitrary views can be synthesized using a volume rendering technique. The approach involves several sequential steps. First, the reference RGB image is encoded into separate shape and color latent variables. Then, a shape prediction module predicts reverse geometric noise based on the shape latent variable within the diffusion model. Next, a color prediction module predicts color features for each 3D point using information from the color latent variable. Finally, a volume rendering module transforms the generated colored point cloud into 2D image space, facilitating training based solely on a reference image. Experimental results demonstrate that the proposed method achieves competitive performance on colored 3D shape reconstruction and novel view image synthesis.

3D-PCGR: Colored Point Cloud Generation and Reconstruction with Surface and Scale Constraints

In the field of 3D point cloud data, the 3D representation of objects is often affected by factors such as lighting, occlusion, and noise, leading to issues of information loss and incompleteness in the collected point cloud data. Point cloud completion algorithms aim to generate complete object point cloud data using partial or local point cloud data as input. Despite promising results achieved by existing methods, current point cloud completion approaches often lack smooth and structural consistency, resulting in a messy overall structure. To address these shortcomings in point cloud completion, we propose a point cloud generative method based on surface consistency and scale rendering. In addition, to solve the limitation of existing methods that mainly focus on geometric features in 3D point cloud completion and do not make full use of color information, we introduce an object reconstruction method based on texture and geometric features. Extensive experiments demonstrate that our proposed methods exhibit superior performance in terms of local details and overall object structure.

3D Question Answering.

Visual question answering (VQA) has experienced tremendous progress in recent years. However, most efforts have only focused on 2D image question-answering tasks. In this article, we extend VQA to its 3D counterpart, 3D question answering (3DQA), which can facilitate a machine's perception of 3D real-world scenarios. Unlike 2D image VQA, 3DQA takes the color point cloud as input and requires both appearance and 3D geometrical comprehension to answer the 3D-related questions. To this end, we propose a novel transformer-based 3DQA framework "3DQA-TR", which consists of two encoders to exploit the appearance and geometry information, respectively. Finally, the multi-modal information about the appearance, geometry, and linguistic question can attend to each other via a 3D-linguistic Bert to predict the target answers. To verify the effectiveness of our proposed 3DQA framework, we further develop the first 3DQA dataset "ScanQA", which builds on the ScanNet dataset and contains over 10 K question-answer pairs for 806 scenes. To the best of our knowledge, ScanQA is the first large-scale dataset with natural-language questions and free-form answers in 3D environments that is fully human-annotated. We also use several visualizations and experiments to investigate the astonishing diversity of the collected questions and the significant differences between this task from 2D VQA and 3D captioning. Extensive experiments on this dataset demonstrate the obvious superiority of our proposed 3DQA framework over state-of-the-art VQA frameworks and the effectiveness of our major designs. Our code and dataset will be made publicly available to facilitate research in this direction. The code and data are available at http://shuquanye.com/3DQA_website/.

PCQD‐AR: Subjective quality assessment of compressed point clouds with head‐mounted augmented reality

AbstractThis letter fully studies the coloured point cloud quality assessment in augmented reality (AR) environment through subjective test. Firstly, a point cloud dataset, named point cloud quality dataset‐AR, including ten reference point clouds and their 90 distorted versions is presented, which were generated using the reference software of video‐based point cloud compression across various combinations of geometry and texture quantization parameters. Then, the impact of geometry and texture distortions on perceived quality of point clouds in AR environment was discussed in detail. Moreover, the performance of existing objective point cloud quality assessment metrics on the proposed dataset is evaluated. The subjective dataset including the values of mean opinion score were released to public.

A Point Cloud Registration Framework with Color Information Integration

Point cloud registration serves as a critical tool for constructing 3D environmental maps. Both geometric and color information are instrumental in differentiating diverse point features. Specifically, when points appear similar based solely on geometric features, rendering them challenging to distinguish, the color information embedded in the point cloud carries significantly important features. In this study, the colored point cloud is utilized in the FCGCF algorithm, a refined version of the FCGF algorithm, incorporating color information. Moreover, we introduce the PointDSCC method, which amalgamates color consistency from the PointDSC method for outlier removal, thus enhancing registration performance when synergized with other pipeline stages. Comprehensive experiments across diverse datasets reveal that the integration of color information into the registration pipeline markedly surpasses the majority of existing methodologies and demonstrates robust generalizability.

Multi-View Projection Based Joint Geometry and Color Hole Repairing Method for G-PCC Trisoup Encoded Color Point Cloud

Multi-View Projection Based Joint Geometry and Color Hole Repairing Method for G-PCC Trisoup Encoded Color Point Cloud

UAV-Based High-Throughput Phenotyping to Segment Individual Apple Tree Row Based on Geometrical Features of Poles and Colored Point Cloud

Highlights Integrating RGB values and 3D coordinates of fruit trees provided phenotype-related data. The use of point clouds projection in tree rows segmentation has been confirmed. Using geometric features increased the accuracy of removing poles in an orchard. Segmenting fruit trees after removing poles on a single tree row was more accurate than traditional segmentation method. UAV equipped with LiDAR and a camera is promising for orchard high-throughput phenotyping. Abstract. High-throughput phenotyping (HTP) of fruit trees is important for providing crop geometrical information to evaluate their high yield genotypes. Unmanned aerial vehicle (UAV) is suitable for HTP by obtaining remote sensing data of large modern apple orchards, where each tree row needs to be segmented before segmenting a single tree. This study aims to develop a method for segmenting each row without noise (ERWON) of apple trees based on integrating RGB values and three-dimensional coordinates by UAV. A robust, real-time, RGB-colored, and LiDAR-inertial-visual tightly-coupled state estimation network was used to form a dense map of the orchard, which provided datasets of colored point clouds. Supporting poles were removed from the point clouds based on the consistent number of half upper parts and lower parts. Random sampling and an effective local feature aggregator were trained to segment ERWON after pole segmentation. Results showed that a precision of 0.971, a recall of 0.984, and an intersection-over-union of 0.817 for ERWON segmentation were achieved. This method proposed a potential solution for addressing the challenge of accurately and efficiently segmenting ERWON in large orchards. It is expected to be helpful for obtaining general parameters, such as geometric, morphological, and textural characteristics, as well as more specific parameters relevant to a particular phenotyping task. Keywords: Apple trees, Detection, Point cloud, RGB-colored, Segmentation.

Colored Point Cloud Quality Assessment Using Complementary Features in 3D and 2D Spaces

Colored Point Cloud Quality Assessment Using Complementary Features in 3D and 2D Spaces

Omnidirectional-Sensor-System-Based Texture Noise Correction in Large-Scale 3D Reconstruction.

The evolution of cameras and LiDAR has propelled the techniques and applications of three-dimensional (3D) reconstruction. However, due to inherent sensor limitations and environmental interference, the reconstruction process often entails significant texture noise, such as specular highlight, color inconsistency, and object occlusion. Traditional methodologies grapple to mitigate such noise, particularly in large-scale scenes, due to the voluminous data produced by imaging sensors. In response, this paper introduces an omnidirectional-sensor-system-based texture noise correction framework for large-scale scenes, which consists of three parts. Initially, we obtain a colored point cloud with luminance value through LiDAR points and RGB images organization. Next, we apply a voxel hashing algorithm during the geometry reconstruction to accelerate the computation speed and save the computer memory. Finally, we propose the key innovation of our paper, the frame-voting rendering and the neighbor-aided rendering mechanisms, which effectively eliminates the aforementioned texture noise. From the experimental results, the processing rate of one million points per second shows its real-time applicability, and the output figures of texture optimization exhibit a significant reduction in texture noise. These results indicate that our framework has advanced performance in correcting multiple texture noise in large-scale 3D reconstruction.

Dataset Creation for Semantic Segmentation Using Colored Point Clouds Considering Shadows on Traversable Area

Semantic segmentation, which provides pixel-wise class labels for an input image, is expected to improve the movement performance of autonomous robots significantly. However, it is difficult to train a good classifier for target applications; public large-scale datasets are often unsuitable. Actually, a classifier trained using Cityscapes is not enough accurate for the Tsukuba Challenge. To generate an appropriate dataset for the target environment, we attempt to construct a semi-automatic method using a colored point cloud obtained with a 3D scanner. Although some degree of accuracy is achieved, it is not practical. Hence, we propose a novel method that creates images with shadows by rendering them in the 3D space to improve the classification accuracy of actual images with shadows, for which existing methods do not output appropriate results. Experimental results using datasets captured around the Tsukuba City Hall demonstrate that the proposed method was superior when appropriate constraints were applied for shadow generation; the mIoU was improved from 0.358 to 0.491 when testing images were obtained at different locations.

Three-Dimensional Quantification and Visualization of Leaf Chlorophyll Content in Poplar Saplings under Drought Using SFM-MVS

As global temperatures warm, drought reduces plant yields and is one of the most serious abiotic stresses causing plant losses. The early identification of plant drought is of great significance for making improvement decisions in advance. Chlorophyll is closely related to plant photosynthesis and nutritional status. By tracking the changes in chlorophyll between plant strains, we can identify the impact of drought on a plant’s physiological status, efficiently adjust the plant’s ecosystem adaptability, and achieve optimization of planting management strategies and resource utilization efficiency. Plant three-dimensional reconstruction and three-dimensional character description are current research hot spots in the development of phenomics, which can three-dimensionally reveal the impact of drought on plant structure and physiological phenotypes. This article obtains visible light multi-view images of four poplar varieties before and after drought. Machine learning algorithms were used to establish the regression models between color vegetation indices and chlorophyll content. The model, based on the partial least squares regression (PLSR), reached the best performance, with an R2 of 0.711. The SFM-MVS algorithm was used to reconstruct the plant’s three-dimensional point cloud and perform color correction, point cloud noise reduction, and morphological calibration. The trained PLSR chlorophyll prediction model was combined with the point cloud color information, and the point cloud color was re-rendered to achieve three-dimensional digitization of plant chlorophyll content. Experimental research found that under natural growth conditions, the chlorophyll content of poplar trees showed a gradient distribution state with gradually increasing values from top to bottom; after being given a short period of mild drought stress, the chlorophyll content accumulated. Compared with the value before stress, it has improved, but no longer presents a gradient distribution state. At the same time, after severe drought stress, the chlorophyll value decreased as a whole, and the lower leaves began to turn yellow, wilt and fall off; when the stress intensity was consistent with the duration, the effect of drought on the chlorophyll value was 895 < SY-1 < 110 < 3804. This research provides an effective tool for in-depth understanding of the mechanisms and physiological responses of plants to environmental stress. It is of great significance for improving agricultural and forestry production and protecting the ecological environment. It also provides decision-making for solving plant drought problems caused by global climate change.

ASSESSING THE ALIGNMENT BETWEEN GEOMETRY AND COLORS IN TLS COLORED POINT CLOUDS

Abstract. The integration of the color information from RGB cameras with the point cloud geometry is used in numerous applications. However, little attention has been paid on errors that occur when aligning colors to points in terrestrial laser scanning (TLS) point clouds. Such errors may impact the performance of algorithms that utilize colored point clouds. Herein, we propose a procedure for assessing the alignment between the TLS point cloud geometry and colors. The procedure is based upon identifying artificial targets observed in both LiDAR-based point cloud intensity data and camera-based RGB data, and quantifying the quality of the alignment using differences between the target center coordinates estimated separately from these two data sources. Experimental results with eight scanners show that the quality of the alignment depends on the scanner, the software used for colorizing the point clouds, and may change with changing environmental conditions. While we found the effects of misalignment to be negligible for some scanners, they exhibited clearly systematic patterns exceeding the beam divergence, image and scan resolution for four of the scanners. The maximum deviations were about 2 mrad perpendicular to the line-of-sight when colorizing the point clouds with the respective manufacturer’s software or scanner in-built functions, while they were up to about 5 mrad when using a different software. Testing the alignment quality, e.g., using the approach presented herein, is thus important for applications requiring accurate alignment of the RGB colors with the point cloud geometry.

3D Semantic Segmentation of Aerial Photogrammetry Models Based on Orthographic Projection

Semantic segmentation of 3D scenes is one of the most important tasks in the field of computer vision and has attracted much attention. In this paper, we propose a novel framework for 3D semantic segmentation of aerial photogrammetry models, which uses orthographic projection to improve efficiency while still ensuring high precision, and can also be applied to multiple types of models (i.e., textured mesh or colored point cloud). In our pipeline, we first obtain RGB images and elevation images from the 3D scene through orthographic projection, then use the image semantic segmentation network to segment these images to obtain pixel-wise semantic predictions, and finally back-project the segmentation results to the 3D model for fusion. Specifically, for the image semantic segmentation model, we design a cross-modality feature aggregation module and a context guidance module based on category features, which assist the network in learning more discriminative features between different objects. For the 2D-3D semantic fusion, we combine the segmentation results of the 2D images with the geometric consistency of the 3D models for joint optimization, which further improves the accuracy of the 3D semantic segmentation. Extensive experiments on two large-scale urban scenes demonstrate the efficiency and feasibility of our algorithm and surpass the current mainstream 3D deep learning methods.

Class3Dp: A supervised classifier of vegetation species from point clouds

Recognizing the species composition of an ecosystem is essential for conservation and land management. This study presents the software Class3Dp, a supervised classifier of vegetation species for coloured point clouds. Class3Dp is run through a graphical user interface (GUI) that allows for the selection of training samples from RGB or MS (multispectral) clouds and their classification based on geometric, spectral and neighbourhood features, along with different machine learning methods, obtaining the point cloud classified according to the classes (species) introduced. A case study is shown where a classification of ground and vegetation is carried out, obtaining an overall accuracy (OA) of 0.94 in the RGB classification and 0.95 in the MS. Points classified as vegetation were re-classified in the species Anthyllis cytisoides L., Chamaerops humilis L., Cistus monspeliensis L., Pistacia lentiscus L. and Quercus coccifera L., obtaining an OA of 0.86 in the RGB classification and 0.87 in the MS.

Multi-Object Tracking Based on a Novel Feature Image With Multi-Modal Information

Multi-object tracking technology plays a crucial role in many applications, such as autonomous vehicles and security monitoring. This paper proposes a multi-object tracking framework based on the multi-modal information of 3D point clouds and color images. At each sampling instant, the 3D point cloud and image acquired by a LiDAR and a camera are fused into a color point cloud, where objects are detected by the Point-GNN method. And, a novel height-intensity-density (HID) image is constructed from the bird's eye view. The HID image truly reflects the shapes and materials of objects and effectively avoids the influence of object occlusion, which is helpful to object tracking. In two sequential HID images, a new rotation kernel correlation filter is proposed to predict the objects. Furthermore, an object retention module and an object re-recognition module are developed to overcome the object matching failure in the in-between frames. The proposed method takes full advantage of the multi-modal data and effectively achieves the information complementation to improve the accuracy of multi-object tracking. The experiments with the KITTI dataset show that the proposed method has the best performance among the existing traditional multi-object tracking methods.