ParallelNN: A Parallel Octree-based Nearest Neighbor Search Accelerator for 3D Point Clouds

Segmentation and analysis of the environment using 3D data (point clouds) in real time are dynamically developing the area. Falling prices of depth sensors based on technologies: LIDAR, ToF, RADAR, increasing computing power, growing interest in autonomous vehicles and robots, favor this trend. This paper presents test studies of loader crane working area monitoring system based on the Velodyne VLP-16 LIDAR scanner. Developed system use ground plane estimation and surroundings segmentation algorithms. The ground points filtering algorithm is based on the dot product of vectors as well as interpolation using the RANSAC method. Segmentation algorithm uses angle threshold between points and breadth-first search (BFS) method for determine neighborhood. The proposed system was adapted to operate with sparse LIDAR data in real time. Described system allows for detects human bodies, environmental elements, and monitors changes in the loader crane work area. The effectiveness of developed algorithms was tested on data obtained from loader crane test bench. An experiment showed that segmentation and monitoring loader crane working area in real time even with sparse data is possible. Moreover, the authors discuss other methods used to segmentation sparse point cloud in real time, describe the Velodyne VLP-16 scanner, and presents an outline of current research into HMI interfaces for loader cranes.

Light Detection and Ranging (LiDAR) has been widely adopted in modern self-driving vehicles and mobile robotics, providing 3D information of the scene and surrounding objects. However, LiDAR systems suffer from many kinds of noise, and its noisy point clouds degrade downstream tasks. Existing LiDAR point cloud de-noising methods are time-consuming or cannot deal with the noise caused by occlusions or penetrating transparent surfaces. In this paper, we introduce a depth-prior-based LiDAR point clouds de-noising method to deal with all types of noise in LiDAR point clouds in real time. The depth prior is derived from the fundamental principles of range-gated imaging and divides the depth of field into three parts, which can provide an effective depth signal. The LiDAR point cloud, which is acquired in synchronization with gated images, is projected into a depth map, and points whose depth is inconsistent with the depth prior can be regarded as noise and can be removed, finally. We conduct an ablation study and compare the proposed method with existing de-noising methods using the Gated2Depth dataset, which is to our knowledge the first long-range-gated dataset specifically designed for 3D detection in adverse weather conditions and includes all the necessary data. The results demonstrated that the proposed method achieves superior performance across all metrics.

This study proposes a privacy-preserving Visual SLAM framework for estimating camera poses and performing bundle adjustment with mixed line and point clouds in real time. Previous studies have proposed localization methods to estimate a camera pose using a line-cloud map for a single image or a reconstructed point cloud. These methods offer a scene privacy protection against the inversion attacks by converting a point cloud to a line cloud, which reconstruct the scene images from the point cloud. However, they are not directly applicable to a video sequence because they do not address computational efficiency. This is a critical issue to solve for estimating camera poses and performing bundle adjustment with mixed line and point clouds in real time. Moreover, there has been no study on a method to optimize a line-cloud map of a server with a point cloud reconstructed from a client video because any observation points on the image coordinates are not available to prevent the inversion attacks, namely the reversibility of the 3D lines. The experimental results with synthetic and real data show that our Visual SLAM framework achieves the intended privacy-preserving formation and real-time performance using a line-cloud map.KeywordsVisual SLAMPrivacyLine cloudPoint cloud

Many place recognition approaches, which identify previously visited places or locations by matching current sensory data, such as 2D RGB images and 3D point clouds, have been proposed to achieve accurate and robust localization and loop closure detection in global positioning system (GPS)-denied environments. Since visual place recognition (VPR) methods that rely on images captured by camera sensors are highly sensitive to variations in appearance, including changes in lighting, surface color, and shadows, they can lead to poor place recognition accuracy. In contrast, light detection and ranging (LiDAR)-based place recognition (LPR) approaches based on 3D point cloud data that captures the shape and geometric structure of the environment are robust to changes in place appearance and can therefore provide more reliable place recognition results than VPR methods. This work presents an indoor LPR method called PointNetVLAD-based indoor pedestrian localization (PIPL). PIPL is a deep network model that uses PointNetVLAD to learn to extract global descriptors from 3D LiDAR point cloud data. PIPL can recognize places previously visited by a pedestrian using point clouds captured by a low-cost LiDAR sensor on a smartphone in small-scale indoor environments, while PointNetVLAD performs place recognition for vehicles using high-cost LiDAR, GPS, and inertial measurement unit (IMU) sensors in large-scale outdoor areas. For place recognition on 3D point cloud reference maps generated from LiDAR scans, PointNetVLAD exploits the universal transverse mercator (UTM) coordinate system based on GPS and IMU measurements, whereas PIPL uses a virtual coordinate system designed in this study due to the unavailability of GPS indoors. In experiments conducted in campus buildings, PIPL shows significant advantages over NetVLAD (known as a convolutional neural network (CNN)-based VPR method). Particularly in indoor environments with repetitive scenes where geometric structures are preserved and image-based appearance features are sparse or unclear, PIPL achieved 39% higher top-1 accuracy and 10% higher top-3 accuracy compared to NetVLAD. Furthermore, PIPL achieved place recognition accuracy comparable to NetVLAD even with a small number of points in a 3D point cloud and outperformed NetVLAD even with a smaller model training dataset. The experimental results also indicate that PIPL requires over 76% less place retrieval time than NetVLAD while maintaining robust place classification performance.

Technological innovations in the hardware of RGB-D sensors have allowed the acquisition of 3D point clouds in real time. Consequently, various applications have arisen related to the 3D world, which are receiving increasing attention from researchers. Nevertheless, one of the main problems that remains is the demand for computationally intensive processing that required optimized approaches to deal with 3D vision modeling, especially when it is necessary to perform tasks in real time. A previously proposed multi-resolution 3D model known as foveated point clouds can be a possible solution to this problem. Nevertheless, this is a model limited to a single foveated structure with context dependent mobility. In this work, we propose a new solution for data reduction and feature detection using multifoveation in the point cloud. Nonetheless, the application of several foveated structures results in a considerable increase of processing since there are intersections between regions of distinct structures, which are processed multiple times. Towards solving this problem, the current proposal brings an approach that avoids the processing of redundant regions, which results in even more reduced processing time. Such approach can be used to identify objects in 3D point clouds, one of the key tasks for real-time applications as robotics vision, with efficient synchronization allowing the validation of the model and verification of its applicability in the context of computer vision. Experimental results demonstrate a performance gain of at least 27.21% in processing time while retaining the main features of the original, and maintaining the recognition quality rate in comparison with state-of-the-art 3D object recognition methods.

Abstract. Forest digitisation is one of the next major challenges to be tackled in the forestry domain. As a consequence of tremendous advances in 3D scanning technologies, broad areas of forest can be mapped in 3D dramatically faster than 20 years ago. Consequently, capturing 3D forest point clouds with the use of 3D sensing technologies – such as lidar – is becoming predominant in the field of forestry. However, the processing of 3D point clouds to bring semantics to the 3D forestry data – e.g. by linking them with ecological values – has not seen similar advancements. Therefore, in this paper we consider a novel approach based on the use of VR (Virtual reality) as a potential solution for deriving biodiversity from 3D point clouds acquired in the field. That is, we developed a VR labelling application to visualise forest point clouds and to perform the segmentation of several biodiversity components on tree stems e.g., mosses, lichens and bark pockets. Furthermore, the VR segmented point cloud was analysed with standard accuracy and precision metrics. Namely, the proposed VR application managed to achieve an IoU (Intersection over Union) rate value of 98.74% for the segmentation of bark pockets and resp. 93.71% for the moss and lichen classes. These encouraging results reinforce the potential for the proposed VR labelling method for other purposes in the future, for example for AI (Artificial Intelligence) training dataset creation.

With the rapid development of web technologies and the popularity of smartphones, users are uploading and sharing a large number of images every day. Therefore, it is a very important issue nowadays to enable users to discover exactly the information they need in the vast amount of data and to make it possible to integrate their large amount of image material efficiently. However, traditional content-based image retrieval techniques are based on images, and there is a “semantic gap” between this and people's understanding of images. To address this “semantic gap,” a keyframe image processing method for 3D point clouds is proposed, and based on this, a U-Net-based binary data stream semantic segmentation network is established for keyframe image processing of 3D point clouds in combination with deep learning techniques.

Three dimension (3D) point cloud data in fog-filled environments were measured using light detection and ranging (LIDAR). Disaster response robots cannot easily navigate through such environments because this data contain false data and distance errors caused by fog. We propose a method for recognizing and removing fog based on 3D point cloud features and a distance correction method for reducing measurement errors. Laser intensity and geometrical features are used to recognize false data. However, these features are not sufficient to measure a 3D point cloud in fog-filled environments with 6 and 2 m visibility, as misjudgments occur. To reduce misjudgment, laser beam penetration features were added. Support vector machine (SVM) and K-nearest neighbor (KNN) are used to classify point cloud data into ‘fog’ and ‘objects.’ We evaluated our method in heavy fog (6 and 2 m visibility). SVM has a better F-measure than KNN; it is higher than 90% in heavy fog (6 and 2 m visibility). The distance error correction method reduces distance errors in 3D point cloud data by a maximum of 4.6%. A 3D point cloud was successfully measured using LIDAR in a fog-filled environment. Our method’s recall (90.1%) and F-measure (79.4%) confirmed its robustness.

LiDAR (Light Detection and Ranging) point clouds are measurements of irregularly distributed points on scanned object surfaces acquired with airborne or terrestrial LiDAR systems. Feature extraction is the key to transform LiDAR data into spatial information. Surface features are dominant in most LiDAR data corresponding to scanned object surfaces. This paper proposes a general method to segment co-surface points. An incremental segmentation strategy is developed for the implementation, which comprises several algorithms and employs various criteria to gradually segment LiDAR point clouds into several levels. There are four operation steps. First, the proximity of point clouds is established as spatial indices defined in an octree-structured voxel space. Second, a connected-component labeling algorithm for voxels is applied for segmenting neighboring points. Third, coplanar points then can be segmented with the octree-based split-and-merge algorithm as plane features. Finally, combining neighboring plane features forms surface features. With respect to each step, processed LiDAR point clouds are segmented into organized points, neighboring point groups, coplanar point groups, and co-surface point groups. The proposed method enables an incremental retrieval and analysis of a large LiDAR dataset. Experiment results demonstrate the effectiveness of the segmentation algorithm in handling both airborne and terrestrial LiDAR data. The end results as well as the intermediate results of the segmentation may be useful for object modeling of different purposes using LiDAR data.

Abstract. With their high recording rate of hundreds of thousands of points acquired per second, speed of execution and a remote acquisition mode, SLAM based mobile mapping systems (MMS) are a very powerful solution for capturing 3D point clouds in real time, simply by walking in the area of interest. Regarding indoor surveys, these MMS have been integrated in handheld or backpack solutions and become fast scanning sensors. Despite their advantages, the geometric accuracy of 3D point clouds guaranteed with these sensors is lower than the one reachable with static TLS. In this paper the effectiveness of two recent mobile mapping systems namely the GeoSLAM ZEB-REVO RT and the more recent GreenValley LiBackPack C50 is investigated for indoor surveys. In order to perform a reliable assessment study, several datasets produced with each sensor are compared to the high-cost georeferenced point cloud obtained with static laser scanning target-based technique.

Generation of hyperspectral point clouds: Mapping, compression and rendering

In this research, an attention mechanism is integrated to enhance object classification in the processing of 3D point clouds. Point clouds obtained from LiDAR sensors are crucial for robotics and autonomous driving, as they provide detailed spatial data. However, large data volumes and noise often challenge traditional processing methods. To address this, the improved PointNet++ neural network is employed with an embedded attention mechanism, allowing it to focus on the most relevant portions of the input for object classification. PointNet++'s hierarchical structure, combined with attention layers, enables effective classification of complex scenes by prioritizing key features in the point cloud data. Tests on the KITTI dataset demonstrate that the attention-based approach boosts classification accuracy and reduces processing time. This method shows promise for building more reliable and efficient perception systems for self-driving vehicles and other 3D data analysis applications. By leveraging attention mechanisms within PointNet++, this study underscores their potential to enhance processing speed and accuracy, addressing critical challenges in the management of large-scale 3D point cloud data and supporting the development of faster, more accurate neural network-based systems for real-world applications.

3D point clouds are a de facto standard for 3D documentation and modelling. The advances in laser scanning technology broadens the usability and access to 3D measurement systems. 3D point clouds are used in many disciplines such as robotics, 3D modelling, archeology and surveying. Scanners are able to acquire up to a million of points per second to represent the environment with a dense point cloud. This represents the captured environment with a very high degree of detail. The combination of laser scanning technology with photography adds color information to the point clouds. Thus the environment is represented more realistically. Full 3D models of environments, without any occlusion, require multiple scans. Merging point clouds is a challenging process. This thesis presents methods for point cloud registration based on the panorama images generated from the scans. Image representation of point clouds introduces 2D image processing methods to 3D point clouds. Several projection methods for the generation of panorama maps of point clouds are presented in this thesis. Additionally, methods for point cloud reduction and compression based on the panorama maps are proposed. Due to the large amounts of data generated from the 3D measurement systems these methods are necessary to improve the point cloud processing, transmission and archiving. This thesis introduces point cloud processing methods as a novel framework for the digitisation of archeological excavations. The framework replaces the conventional documentation methods for excavation sites. It employs point clouds for the generation of the digital documentation of an excavation with the help of an archeologist on-site. The 3D point cloud is used not only for data representation but also for analysis and knowledge generation. Finally, this thesis presents an autonomous indoor mobile mapping system. The mapping system focuses on the sensor placement planning method. Capturing a complete environment requires several scans. The sensor placement planning method solves for the minimum required scans to digitise large environments. Combining this method with a navigation system on a mobile robot platform enables it to acquire data fully autonomously. This thesis introduces a novel hole detection method for point clouds to detect obscured parts of a captured environment. The sensor placement planning method selects the next scan position with the most coverage of the obscured environment. This reduces the required number of scans. The navigation system on the robot platform consist of path planning, path following and obstacle avoidance. This guarantees the safe navigation of the mobile robot platform between the scan positions. The sensor placement planning method is designed as a stand alone process that could be used with a mobile robot platform for autonomous mapping of an environment or as an assistant tool for the surveyor on scanning projects.

To provide a realistic environment for remote sensing applications, point clouds are used to realize a three-dimensional (3D) digital world for the user. Motion recognition of objects, e.g., humans, is required to provide realistic experiences in the 3D digital world. To recognize a user’s motions, 3D landmarks are provided by analyzing a 3D point cloud collected through a light detection and ranging (LiDAR) system or a red green blue (RGB) image collected visually. However, manual supervision is required to extract 3D landmarks as to whether they originate from the RGB image or the 3D point cloud. Thus, there is a need for a method for extracting 3D landmarks without manual supervision. Herein, an RGB image and a 3D point cloud are used to extract 3D landmarks. The 3D point cloud is utilized as the relative distance between a LiDAR and a user. Because it cannot contain all information the user’s entire body due to disparities, it cannot generate a dense depth image that provides the boundary of user’s body. Therefore, up-sampling is performed to increase the density of the depth image generated based on the 3D point cloud; the density depends on the 3D point cloud. This paper proposes a system for extracting 3D landmarks using 3D point clouds and RGB images without manual supervision. A depth image provides the boundary of a user’s motion and is generated by using 3D point cloud and RGB image collected by a LiDAR and an RGB camera, respectively. To extract 3D landmarks automatically, an encoder–decoder model is trained with the generated depth images, and the RGB images and 3D landmarks are extracted from these images with the trained encoder model. The method of extracting 3D landmarks using RGB depth (RGBD) images was verified experimentally, and 3D landmarks were extracted to evaluate the user’s motions with RGBD images. In this manner, landmarks could be extracted according to the user’s motions, rather than by extracting them using the RGB images. The depth images generated by the proposed method were 1.832 times denser than the up-sampling-based depth images generated with bilateral filtering.

Crop 3D modeling allows site-specific management at different crop stages. In recent years, light detection and ranging (LiDAR) sensors have been widely used for gathering information about plant architecture to extract biophysical parameters for decision-making programs. The study reconstructed vineyard crops using light detection and ranging (LiDAR) technology. Its accuracy and performance were assessed for vineyard crop characterization using distance measurements, aiming to obtain a 3D reconstruction. A LiDAR sensor was installed on-board a mobile platform equipped with an RTK-GNSS receiver for crop 2D scanning. The LiDAR system consisted of a 2D time-of-flight sensor, a gimbal connecting the device to the structure, and an RTK-GPS to record the sensor data position. The LiDAR sensor was facing downwards installed on-board an electric platform. It scans in planes perpendicular to the travel direction. Measurements of distance between the LiDAR and the vineyards had a high spatial resolution, providing high-density 3D point clouds. The 3D point cloud was obtained containing all the points where the laser beam impacted. The fusion of LiDAR impacts and the positions of each associated to the RTK-GPS allowed the creation of the 3D structure. Although point clouds were already filtered, discarding points out of the study area, the branch volume cannot be directly calculated, since it turns into a 3D solid cluster that encloses a volume. To obtain the 3D object surface, and therefore to be able to calculate the volume enclosed by this surface, a suitable alpha shape was generated as an outline that envelops the outer points of the point cloud. The 3D scenes were obtained during the winter season when only branches were present and defoliated. The models were used to extract information related to height and branch volume. These models might be used for automatic pruning or relating this parameter to evaluate the future yield at each location. The 3D map was correlated with ground truth, which was manually determined, pruning the remaining weight. The number of scans by LiDAR influenced the relationship with the actual biomass measurements and had a significant effect on the treatments. A positive linear fit was obtained for the comparison between actual dry biomass and LiDAR volume. The influence of individual treatments was of low significance. The results showed strong correlations with actual values of biomass and volume with R2 = 0.75, and when comparing LiDAR scans with weight, the R2 rose up to 0.85. The obtained values show that this LiDAR technique is also valid for branch reconstruction with great advantages over other types of non-contact ranging sensors, regarding a high sampling resolution and high sampling rates. Even narrow branches were properly detected, which demonstrates the accuracy of the system working on difficult scenarios such as defoliated crops.