Color Point Cloud Research Articles

Perceptual Quality Assessment of Colored 3D Point Clouds.

3D point clouds have found a wide variety of applications in multimedia processing, remote sensing, and scientific computing. Although most point cloud processing systems are developed to improve viewer experiences, little work has been dedicated to perceptual quality assessment of 3D point clouds. In this work, we build a new 3D point cloud database, namely the Waterloo Point Cloud (WPC) database. In contrast to existing datasets consisting of small-scale and low-quality source content of constrained viewing angles, the WPC database contains 20 high quality, realistic, and omni-directional source point clouds and 740 diversely distorted point clouds. We carry out a subjective quality assessment experiment over the database in a controlled lab environment. Our statistical analysis suggests that existing objective point cloud quality assessment (PCQA) models only achieve limited success in predicting subjective quality ratings. We propose a novel objective PCQA model based on an attention mechanism and a variant of information content-weighted structural similarity, which significantly outperforms existing PCQA models. The database has been made publicly available at https://github.com/qdushl/Waterloo-Point-Cloud-Database.

A general mobile manipulator automation framework for flexible tasks in controlled environments

To enable a mobile manipulator to perform human tasks from a single teaching demonstration is vital to flexible automation. We call our proposed method MMPA (Mobile Manipulator Process Automation with One-shot Teaching). Currently, there is no effective and robust MMPA framework which is not influenced by the mobile base’s parking precision. The proposed MMPA framework consists of two stages: collecting data (mobile base’s location, environment information, end-effector’s path) in the teaching stage for robot learning; letting the end-effector repeat the nearly same path as the reference path in the world frame to reproduce the work in the automation stage. More specifically, in the automation stage, the robot navigates to the specified location without the need of a precise parking. Then, based on colored point cloud registration, the proposed IPE (Iterative Pose Estimation by Eye & Hand) algorithm could estimate the accurate 6D relative parking pose of the robot arm base without the need of any marker. Finally, the robot could learn the error compensation from the parking pose’s bias to modify the end-effector’s path to make it repeat a nearly same path in the world coordinate system as recorded in the teaching stage. Hundreds of trials have been conducted with a real mobile manipulator to show the superior robustness of the system and the accuracy of the process automation regardless of the parking precision. For the released code, please contact AI@amigaga.com.

Contrastive Predictive Autoencoders for Dynamic Point Cloud Self-Supervised Learning

We present a new self-supervised paradigm on point cloud sequence understanding. Inspired by the discriminative and generative self-supervised methods, we design two tasks, namely point cloud sequence based Contrastive Prediction and Reconstruction (CPR), to collaboratively learn more comprehensive spatiotemporal representations. Specifically, dense point cloud segments are first input into an encoder to extract embeddings. All but the last ones are then aggregated by a context-aware autoregressor to make predictions for the last target segment. Towards the goal of modeling multi-granularity structures, local and global contrastive learning are performed between predictions and targets. To further improve the generalization of representations, the predictions are also utilized to reconstruct raw point cloud sequences by a decoder, where point cloud colorization is employed to discriminate against different frames. By combining classic contrast and reconstruction paradigms, it makes the learned representations with both global discrimination and local perception. We conduct experiments on four point cloud sequence benchmarks, and report the results on action recognition and gesture recognition under multiple experimental settings. The performances are comparable with supervised methods and show powerful transferability.

Filtering Green Vegetation Out from Colored Point Clouds of Rocky Terrains Based on Various Vegetation Indices: Comparison of Simple Statistical Methods, Support Vector Machine, and Neural Network

Filtering out vegetation from a point cloud based on color is only rarely used, largely due to the lack of knowledge of the suitability of input information (color, vegetation indices) and the thresholding methods. We have evaluated multiple vegetation indices (ExG, ExR, ExB, ExGr, GRVI, MGRVI, RGBVI, IKAW, VARI, CIVE, GLI, and VEG) and combined them with 10 methods of threshold determination based on training set selection (including machine learning methods) and the renowned Otsu’s method. All these combinations were applied to four clouds representing vegetated rocky terrain, and the results were compared. The ExG and GLI indices were generally the most suitable for this purpose, with the best F-scores of 97.7 and 95.4, respectively, and the best-balanced accuracies for the same combination of the method/vegetation index of 98.9 and 98.3%, respectively. Surprisingly, these best results were achieved using the simplest method of threshold determination, considering only a single class (vegetation) with a normal distribution. This algorithm outperformed all other methods, including those based on a support vector machine and a deep neural network. Thanks to its simplicity and ease of use (only several patches representing vegetation must be manually selected as a training set), this method can be recommended for vegetation removal from rocky and anthropogenic surfaces.

Automatic Point Cloud Colorization of Ground-Based LiDAR Data Using Video Imagery without Position and Orientation System

With the continuous development of three-dimensional city modeling, traditional close-range photogrammetry is limited by complex processing procedures and incomplete 3D depth information, making it unable to meet high-precision modeling requirements. In contrast, the integration of light detection and ranging and cameras in mobile measurement systems provides a new and highly effective solution. Currently, integrated mobile measurement systems commonly require cameras, lasers, position and orientation system and inertial measurement units; thus, the hardware cost is relatively expensive, and the system integration is complex. Therefore, in this paper, we propose a ground mobile measurement system only composed of a LiDAR and a GoPro camera, providing a more convenient and reliable way to automatically obtain 3D point cloud data with spectral information. The automatic point cloud coloring based on video images mainly includes four aspects: (1) Establishing models for radial distortion and tangential distortion to correct video images. (2) Establishing a registration method based on normalized Zernike moments to obtain the exterior orientation elements. The error of the result is only 0.5–1 pixel, which is far higher than registration based on a collinearity equation. (3) Establishing relative orientation based on essential matrix decomposition and nonlinear optimization. This involves uniformly using the speeded-up robust features algorithm with distance restriction and random sample consensus to select corresponding points. The vertical parallax of the stereo image pair model is less than one pixel, indicating that the accuracy is high. (4) A point cloud coloring method based on Gaussian distribution with central region restriction is adopted. Only pixels within the central region are considered valid for coloring. Then, the point cloud is colored based on the mean of the Gaussian distribution of the color set. In the colored point cloud, the textures of the buildings are clear, and targets such as windows, grass, trees, and vehicles can be clearly distinguished. Overall, the result meets the accuracy requirements of applications such as tunnel detection, street-view modeling and 3D urban modeling.

Exploring an Innovative Moral Education Cultivation Model in Higher Education through Neural Network Perspective: A Preliminary Study

ABSTRACT This paper aims to enhance the effectiveness of moral education teaching through an innovative approach. By integrating neural network technology, a data-model-driven three-dimensional initial model construction algorithm was proposed to identify the characteristics of moral education classroom teaching. The 3D space base point and initial value of shape parameters were obtained based on color point cloud data, and the 3D space coordinates of all the vertices of the model were calculated using the Euclidean distance algorithm, resulting in the construction of an initial 3D model. Furthermore, a model of Innovative Moral Education (IME) in higher education was developed based on neural network, and a framework analysis was conducted. Results from simulation teaching research indicate that the IME teaching mode based on the proposed neural network model has a significant practical effect. Overall, this study offers insights into an innovative approach to moral education teaching and contributes to the development of effective pedagogical practices.

Fast vehicle detection based on colored point cloud with bird’s eye view representation

RGB cameras and LiDAR are crucial sensors for autonomous vehicles that provide complementary information for accurate detection. Recent early-level fusion-based approaches, flourishing LiDAR data with camera features, may not accomplish promising performance ascribable to the immense difference between two modalities. This paper presents a simple and effective vehicle detection method based on an early-fusion strategy, unified 2D BEV grids, and feature fusion. The proposed method first eliminates many null point clouds through cor-calibration. It augments point cloud data by color information to generate 7D colored point cloud, and unifies augmented data into 2D BEV grids. The colored BEV maps can then be fed to any 2D convolution network. A peculiar Feature Fusion (2F) detection module is utilized to extract multiple scale features from BEV images. Experiments on the KITTI public benchmark and Nuscenes dataset show that fusing RGB image with point cloud rather than raw point cloud can lead to better detection accuracy. Besides, the inference time of the proposed method reaches 0.05 s/frame thanks to its simple and compact architecture.

The artistic aesthetic and cultural value orientation of 3D printing

Abstract For the current artwork, 3D printing design simulation technology has the disadvantage of low clarity and poor effect, which cannot effectively describe the designer's work. To this end, this paper investigates the artistic aesthetic and cultural value orientation of 3D printing. First, the 3D digitization work of the oil painting was carried out by using the multi-baseline photogrammetry method in photogrammetry technology. Then the non-measurement digital camera was checked for internal orientation elements, the control panel was photographed, and the internal orientation elements of the camera were measured using the spatial backward rendezvous solution method. Next, the digital image acquisition of the oil painting is carried out, and finally, the photos are sequentially introduced into the Lens photo multi-baseline photogrammetry system to generate the color point cloud model of the oil painting after performing the steps of homonymous point selection null-three matching and beam leveling. In order to enhance the visualization effect, the color point cloud model was used to generate a digital model of the TIN irregular triangular network to realize the 3D digitization of oil painting art. The results of the study show that the thickness of each layer obtained by UV inkjet printing of five types of inks: yellow, green, blue, black, and white is about 11.19 μm, 12.71 μm, 7.91 μm, 9.47 μm, and 13.12 μm, respectively. 3D printing technology for positive impact on art design needs to be recognized and respected, and the better development of related technologies can make the advantages and advancement of the technology itself more fully embodied.

Robot indoor navigation point cloud map generation algorithm based on visual sensing

Abstract At present, low-cost Red Green Blue Depth (RGB-D) sensors are mainly used in indoor robot environment perception, but the depth information obtained by RGB-D cameras has problems such as poor accuracy and high noise, and the generated 3D color point cloud map has low accuracy. In order to solve these problems, this article proposes a vision sensor-based point cloud map generation algorithm for robot indoor navigation. The aim is to obtain a more accurate point cloud map through visual SLAM and Kalman filtering visual-inertial navigation attitude fusion algorithm. The results show that in the positioning speed test data of the fusion algorithm in this study, the average time-consuming of camera tracking is 23.4 ms, which can meet the processing speed requirement of 42 frames per second. The yaw angle error of the fusion algorithm is the smallest, and the ATE test values of the algorithm are smaller than those of the Inertial measurement unit and Simultaneous-Localization-and-Mapping algorithms. This research algorithm can make the mapping process more stable and robust. It can use visual sensors to make more accurate route planning, and this algorithm improves the indoor positioning accuracy of the robot. In addition, the research algorithm can also obtain a dense point cloud map in real time, which provides a more comprehensive idea for the research of robot indoor navigation point cloud map generation.

Morphological Operations on Unorganized Point Clouds Using Octree Graphs

Point clouds resulting from digital scanning are increasingly being used in the heritage field to create knowledge-based models, such as building information models (BIM) . Nevertheless, the use of digital image processing techniques in point cloud unorganized data is not well explored yet because of the lack of information about point adjacency and corresponding difficulty in structuring and manipulating data. In this study we propose an approach to deal with image processing-like mathematical morphological operations in unorganized point cloud using the octree graph. Our method consists of three main steps, namely: (i) computing the best distance value between points in the cloud; (ii) constructing an octree from point cloud; and (iii) producing a graph using rectangular layout. We exemplify the use of the proposed methodology performing morphological operations in grayscale and color point clouds of historical buildings. Results prove the effectiveness of the octree representation, which can efficiently generate an adjacency map, demonstrated by the resulting images, and has the potential to be applied at different areas of 3D data analysis.

CRBeDaSet: A Benchmark Dataset for High Accuracy Close Range 3D Object Reconstruction

This paper presents the CRBeDaSet—a new benchmark dataset designed for evaluating close range, image-based 3D modeling and reconstruction techniques, and the first empirical experiences of its use. The test object is a medium-sized building. Diverse textures characterize the surface of elevations. The dataset contains: the geodetic spatial control network (12 stabilized ground points determined using iterative multi-observation parametric adjustment) and the photogrammetric network (32 artificial signalized and 18 defined natural control points), measured using Leica TS30 total station and 36 terrestrial, mainly convergent photos, acquired from elevated camera standpoints with non-metric digital single-lens reflex Nikon D5100 camera (ground sample distance approx. 3 mm), the complex results of the bundle block adjustment with simultaneous camera calibration performed in the Pictran software package, and the colored point clouds (ca. 250 million points) from terrestrial laser scanning acquired using the Leica ScanStation C10 and post-processed in the Leica Cyclone™ SCAN software (ver. 2022.1.1) which were denoized, filtered, and classified using LoD3 standard (ca. 62 million points). The existing datasets and benchmarks were also described and evaluated in the paper. The proposed photogrammetric dataset was experimentally tested in the open-source application GRAPHOS and the commercial suites ContextCapture, Metashape, PhotoScan, Pix4Dmapper, and RealityCapture. As the first experience in its evaluation, the difficulties and errors that occurred in the software used during dataset digital processing were shown and discussed. The proposed CRBeDaSet benchmark dataset allows obtaining high accuracy (“mm” range) of the photogrammetric 3D object reconstruction in close range, based on a multi-image view uncalibrated imagery, dense image matching techniques, and generated dense point clouds.

Indoor Radio Map Construction via Ray Tracing With RGB-D Sensor-Based 3D Reconstruction: Concept and Experiments in WLAN Systems

This paper proposes an RGB-D sensor-aided ray-tracing simulation framework for indoor radio map construction that models indoor information, such as walls and obstacles, as a set of cuboids. Radio maps can accelerate indoor wireless systems, including resource optimization and fingerprint-based localization. We can construct an accurate radio map using a ray-tracing algorithm with precise information about indoor structures and obstacles. However, it is difficult to acquire this indoor information because manual observations, including the placement of obstacles, are expensive. To mitigate this critical drawback, we propose the acquisition of colored point clouds using an RGB-D sensor. After the point clouds are labeled via a three-dimensional semantic segmentation task, we model the indoor information as a set of cuboids that can be immediately used for the ray-tracing simulation. Indoor experiments using wireless LANs over 5180 and 2452 MHz clarified that the proposed method accurately automates the radio-map construction process from indoor information acquisition to radio propagation simulation.

GQE-Net: A Graph-Based Quality Enhancement Network for Point Cloud Color Attribute.

In recent years, point clouds have become increasingly popular for representing three-dimensional (3D) visual objects and scenes. To efficiently store and transmit point clouds, compression methods have been developed, but they often result in a degradation of quality. To reduce color distortion in point clouds, we propose a graph-based quality enhancement network (GQE-Net) that uses geometry information as an auxiliary input and graph convolution blocks to extract local features efficiently. Specifically, we use a parallel-serial graph attention module with a multi-head graph attention mechanism to focus on important points or features and help them fuse together. Additionally, we design a feature refinement module that takes into account the normals and geometry distance between points. To work within the limitations of GPU memory capacity, the distorted point cloud is divided into overlap-allowed 3D patches, which are sent to GQE-Net for quality enhancement. To account for differences in data distribution among different color components, three models are trained for the three color components. Experimental results show that our method achieves state-of-the-art performance. For example, when implementing GQE-Net on a recent test model of the geometry-based point cloud compression (G-PCC) standard, 0.43 dB, 0.25 dB and 0.36 dB Bjφntegaard delta (BD)-peak-signal-to-noise ratio (PSNR), corresponding to 14.0%, 9.3% and 14.5% BD-rate savings were achieved on dense point clouds for the Y, Cb, and Cr components, respectively. The source code of our method is available at https://github.com/xjr998/GQE-Net.

High-Resolution Terrain Reconstruction of Slot Canyon Using Backpack Mobile Laser Scanning and UAV Photogrammetry

Accurate terrain models are critical for studying the formation and development of slot canyons. However, for slot canyon landforms, it is challenging to generate comprehensive and high-resolution morphological data by individual observation due to the inaccessibility of steep walls on either side and the complexity of the field observation environment, such as variable-slope terrain, partial vegetation cover, and lack of satellite signal. Off-the-shelf surveying techniques, including Unmanned Aerial Vehicles (UAV) photogrammetry and Backpack Mobile Laser Scanning (BMLS), facilitate slot canyon surveys and provide better observations. This paper proposes an integrated scheme to generate comprehensive and centimeter-resolution slot canyon terrain datasets (e.g., color point clouds, Digital Elevation Models (DEM), and 3D mesh) using BMLS and fine UAV photogrammetry. The results show that the fine flight of UAVs based on a rough model can avoid collision with obstacles or flying into restricted areas, allowing users to perform tasks faster and safer. Data integration of BMLS and UAV photogrammetry can obtain accurate terrain datasets with a Root Mean Squared Error (RMSE) of point cloud registration of 0.028 m. Such high-resolution integration terrain datasets reduce local data shadows produced solely by individual datasets, providing a starting point to revealing morphological evolution and genesis of slot canyons.

Fast color point cloud registration based on virtual viewpoint image

With the increase of point cloud scale, the time required by traditional ICP-related point cloud registration methods increases dramatically, which cannot meet the registration requirements of large-scale point clouds. In this paper, a fast registration technique for large scale point clouds based on virtual viewpoint image generation is studied. Firstly, the projection image of color point cloud is generated by virtual viewpoint. Then, the feature is extracted based on ORB and the rotation and translation matrix is calculated. The experimental results show that the registration time of the proposed method is about 1s when the size of the point cloud is from 300,000 to 2 million, which is improved by 17–258 times compared with the traditional ICP registration method, and the registration error is reduced by 80% from ICP 5.0 to 1.0. This paper provides a new idea and method for large-scale color point cloud registration.

No-Reference Quality Assessment for 3D Colored Point Cloud and Mesh Models

To improve the viewer's Quality of Experience (QoE) and optimize computer graphics applications, 3D model quality assessment (3D-QA) has become an important task in the multimedia area. Point cloud and mesh are the two most widely used digital representation formats of 3D models, the visual quality of which is quite sensitive to lossy operations like simplification and compression. Therefore, many related studies such as point cloud quality assessment (PCQA) and mesh quality assessment (MQA) have been carried out to measure the visual quality degradations of 3D models. However, a large part of previous studies utilize full-reference (FR) metrics, which indicates they can not predict the quality level with the absence of the reference 3D model. Furthermore, few 3D-QA metrics consider color information, which significantly restricts their effectiveness and scope of application. In this paper, we propose a no-reference (NR) quality assessment metric for colored 3D models represented by both point cloud and mesh. First, we project the 3D models from 3D space into quality-related geometry and color feature domains. Then, the 3D natural scene statistics (3D-NSS) and entropy are utilized to extract quality-aware features. Finally, machine learning is employed to regress the quality-aware features into visual quality scores. Our method is validated on the colored point cloud quality assessment database (SJTU-PCQA), the Waterloo point cloud assessment database (WPC), and the colored mesh quality assessment database (CMDM). The experimental results show that the proposed method outperforms most compared NR 3D-QA metrics with competitive computational resources and greatly reduces the performance gap with the state-of-the-art FR 3D-QA metrics. The code of the proposed model is publicly available now to facilitate further research.

Assessment of Using UAV Imagery over Featureless Surfaces for Topographic Applications

The growth and development of Unmanned Aerial Vehicles (UAVs) as a photogrammetric platform, concurrently with the advances in Computer Vision (CV) and image processing algorithms have resulted using UAV Photogrammetry in several topographic applications. CV software algorithms rely on extracting, describing, and matching tie points from the sequences overlapping images to generate 3D colored point clouds. One of the biggest problems obstructing the automated processing of UAV imagery is the featureless of the covered surface. This paper has provided the ability, results, and accuracy of processing images captured by UAVs over non-textured sandy surfaces by providing four aligning and geo-referencing techniques. These four methods, IG/blind matching, IG/reference matching, DG/blind matching, and DG/reference matching, have been presented and tested for 630 aerial images with 80 % overlap and 80 % side lap covered approximately 1 km2 at altitude 178 m above ground level (AGL). The results showed that the captured images could be used to extract the photogrammetric topographical measurements with reliable accuracy. The four techniques' geometric accuracy has ranged between (0.043 m to 0.076 m) & (0.047 m to 0.074) for generated point clouds and linear exterior orientation (EO) parameters, respectively. The indirect geo-referencing with reference matching (IG/reference) recorded the highest-level accuracy of point clouds with 0.043m RMSE compared to the direct geo-referencing with reference matching (DG/reference) which gave the highest geometric accuracy of the linear EO parameters with 0.047m

Banana Pseudostem Width Detection Based on Kinect V2 Depth Sensor.

This study used Kinect V2 sensor to collect the three-dimensional point cloud data of banana pseudostem and developed an automatic measurement method of banana pseudostem width. The banana plant was selected as the research object in a banana plantation in Fusui, Guangxi. The mobile measurement of banana pseudostem was carried out at a distance of 1 m from the banana plant using the field operation platform with Kinect V2 as the collection equipment. To eliminate the background data and improve the processing speed, a cascade classifier was used to recognize banana pseudostems from the depth image, extract the region of interest (ROI), and transform the ROI into a color point cloud combined with the color image; secondly, the point cloud was sparse by down-sampling; then, the point cloud noise was removed according to the classification of large-scale and small-scale noise; finally, the stem point cloud was segmented along the y-axis, and the difference between the maximum and minimum values in the x-axis direction of each segment was calculated as its horizontal width. The center point of each segment point cloud was used to fit the slope of the stem centerline, and the average horizontal width was corrected to the stem diameter. The test results show that the average measurement error is only 2.7 mm, the average relative error was 1.34%, and the measurement time is only about 300 ms. It could provide an effective solution for the automatic and rapid measurement of stem width of banana plants and other similar plants.

Three-dimensional reconstruction and phenotype measurement of maize seedlings based on multi-view image sequences.

As an important method for crop phenotype quantification, three-dimensional (3D) reconstruction is of critical importance for exploring the phenotypic characteristics of crops. In this study, maize seedlings were subjected to 3D reconstruction based on the imaging technology, and their phenotypic characters were analyzed. In the first stage, a multi-view image sequence was acquired via an RGB camera and video frame extraction method, followed by 3D reconstruction of maize based on structure from motion algorithm. Next, the original point cloud data of maize were preprocessed through Euclidean clustering algorithm, color filtering algorithm and point cloud voxel filtering algorithm to obtain a point cloud model of maize. In the second stage, the phenotypic parameters in the development process of maize seedlings were analyzed, and the maize plant height, leaf length, relative leaf area and leaf width measured through point cloud were compared with the corresponding manually measured values, and the two were highly correlated, with the coefficient of determination (R2) of 0.991, 0.989, 0.926 and 0.963, respectively. In addition, the errors generated between the two were also analyzed, and results reflected that the proposed method was capable of rapid, accurate and nondestructive extraction. In the third stage, maize stem leaves were segmented and identified through the region growing segmentation algorithm, and the expected segmentation effect was achieved. In general, the proposed method could accurately construct the 3D morphology of maize plants, segment maize leaves, and nondestructively and accurately extract the phenotypic parameters of maize plants, thus providing a data support for the research on maize phenotypes.

PolyU-BPCoMa: A dataset and benchmark towards mobile colorized mapping using a backpack multisensorial system

Constructing colorized point clouds from mobile laser scanning and images is a fundamental work in surveying and mapping. It is also an essential prerequisite for building digital twins for smart cities. However, existing public datasets are either in relatively small scales or lack accurate geometrical and color ground truth. This paper documents a multisensorial dataset named PolyU-BPCoMA which is distinctively positioned towards mobile colorized mapping. The dataset incorporates resources of 3D LiDAR, spherical imaging, GNSS and IMU on a backpack platform. Color checker boards are pasted in each surveyed area as targets and ground truth data are collected by an advanced terrestrial laser scanner (TLS). 3D geometrical and color information can be recovered in the colorized point clouds produced by the backpack system and the TLS, respectively. Accordingly, we provide an opportunity to benchmark the mapping and colorization accuracy simultaneously for a mobile multisensorial system. The dataset is approximately 800 GB in size covering both indoor and outdoor environments. The dataset and development kits are available at https://github.com/chenpengxin/PolyU-BPCoMa.git . • A dataset and benchmark towards mobile colorized mapping using a backpack multisensorial system. • An opportunity to benchmark the mapping and colorization accuracy simultaneously for MLS. • A development kit and a baseline result for performance comparison are available. • Other usages include LiDAR/visual odometry and terrestrial point cloud registration.