3D Point Cloud Processing Research Articles

Keyframe image processing of semantic 3D point clouds based on deep learning.

With the rapid development of web technologies and the popularity of smartphones, users are uploading and sharing a large number of images every day. Therefore, it is a very important issue nowadays to enable users to discover exactly the information they need in the vast amount of data and to make it possible to integrate their large amount of image material efficiently. However, traditional content-based image retrieval techniques are based on images, and there is a "semantic gap" between this and people's understanding of images. To address this "semantic gap," a keyframe image processing method for 3D point clouds is proposed, and based on this, a U-Net-based binary data stream semantic segmentation network is established for keyframe image processing of 3D point clouds in combination with deep learning techniques.

Automatic Recognition and Digital Documentation of Cultural Heritage Hemispherical Domes using Images

Recent advancements in optical metrology have enabled continuous documentation of dense 3-dimensional (3D) point clouds of construction projects, including cultural heritage preservation projects. These point clouds must then be further processed to generate semantic digital models, which is integral to the lifecycle management of heritage sites. For large-scale and continuous digital documentation, processing of dense 3D point clouds is computationally cumbersome, and consequentially requires additional hardware for data management and analysis, increasing the time, cost, and complexity of the project. Fast and reliable solutions for generating the geometric digital models is, hence, eminently desirable. This article presents an original approach to generate reliable semantic digital models of heritage hemispherical domes using only two images. New closed formulations were derived to establish the relationships between a sphere and its projected ellipse onto an image. These formulations were then utilised to create new methods for: (i) selecting the best pair of images from an image network; (ii) detecting ellipses corresponding to projection of spheres in images; (iii) matching of the detected ellipses between images; and (iv) generating the sphere's geometric digital models. The effectiveness of the proposed method was evaluated under both laboratory and real-world datasets. Laboratory experiments revealed that the proposed process using the best pair of images provided results as accurate as that achieved using eight randomly selected images, while improving computation time by a factor of 50. The results of the two real-world datasets showed that the digital model of a hemispherical dome was generated with 6.2 mm accuracy, while improving the total computation time of current best practice by a factor of 7. Real-world experimentation also showed that the proposed method can provide metric-scale definition for photogrammetric point clouds with 3 mm accuracy using spherical targets. The results suggest that the proposed method was successful in automatically generating fast and accurate geometric digital models of hemispherical domes.

FEASIBILITY STUDY OF USING VIRTUAL REALITY FOR INTERACTIVE AND IMMERSIVE SEMANTIC SEGMENTATION OF SINGLE TREE STEMS

Abstract. Forest digitisation is one of the next major challenges to be tackled in the forestry domain. As a consequence of tremendous advances in 3D scanning technologies, broad areas of forest can be mapped in 3D dramatically faster than 20 years ago. Consequently, capturing 3D forest point clouds with the use of 3D sensing technologies – such as lidar – is becoming predominant in the field of forestry. However, the processing of 3D point clouds to bring semantics to the 3D forestry data – e.g. by linking them with ecological values – has not seen similar advancements. Therefore, in this paper we consider a novel approach based on the use of VR (Virtual reality) as a potential solution for deriving biodiversity from 3D point clouds acquired in the field. That is, we developed a VR labelling application to visualise forest point clouds and to perform the segmentation of several biodiversity components on tree stems e.g., mosses, lichens and bark pockets. Furthermore, the VR segmented point cloud was analysed with standard accuracy and precision metrics. Namely, the proposed VR application managed to achieve an IoU (Intersection over Union) rate value of 98.74% for the segmentation of bark pockets and resp. 93.71% for the moss and lichen classes. These encouraging results reinforce the potential for the proposed VR labelling method for other purposes in the future, for example for AI (Artificial Intelligence) training dataset creation.

MFOC-CliqueNet: A CliqueNet-Based Optimal Combination of Multidimensional Features Classification Method for Large-Scale Laser Point Clouds

As large-scale laser 3D point clouds data contains massive and complex data, it faces great challenges in the automatic intelligent processing and classification of large-scale 3D point clouds. Aiming at the problem that 3D point clouds in complex scenes are self-occluded or occluded, which could reduce the object classification accuracy, we propose a multidimension feature optimal combination classification method named MFOC-CliqueNet based on CliqueNet for large-scale laser point clouds. The optimal combination matrix of multidimension features is constructed by extracting the three-dimensional features and multidirectional two-dimension features of 3D point cloud. This is the first time that multidimensional optimal combination features are introduced into cyclic convolutional networks CliqueNet. It is important for large-scale 3D point cloud classification. The experimental results show that the MFOC-CliqueNet framework can realize the latest level with fewer parameters. The experiments on the Large-Scale Scene Point Cloud Oakland dataset show that the classification accuracy of our method is 98.9%, which is better than other classification algorithms mentioned in this paper.

Safe trajectory design and pose estimation for target monitoring in GEO

This paper lies in the framework of mission scenarios, such as Active Debris Removal and On-Orbit Servicing, which require an active spacecraft (chaser) to orbit in close-proximity with respect to a space target. Specifically, these activities involve relative orbital maneuvers, such as monitoring, rendezvous and docking, in which the target-chaser distance ranges from a few tens of meters (depending on the target size) up to contact (in the case of docking). A critical challenge related to the realization of these maneuvers is the need to minimize the risk of collision, considering that the target is a non-cooperative object which may be characterized by uncontrolled rotational dynamics. This goal can be achieved by designing relative trajectories which satisfy specific constraints in terms of safety and stability, on one side, as well as by exploiting relative navigation technologies and algorithms which provide highly accurate estimates of the target-chaser relative motion parameters thus allowing to relax the control requirements. Both these aspects are addressed by this paper with focus on Geostationary Earth Orbits since they represent a particularly crowded orbital region in which the possibility to remove large debris and to extend the operative life of spacecraft, such as telecommunication ones, may have a significant scientific and economic benefit. Hence, an original method is presented to design safety ellipses for target monitoring around GEO targets, which, simultaneously, can provide optimal relative observation geometry for relative navigation (pose determination) using Electro-Optical sensors. The design approach is formulated in mean orbit parameters and it is based on a relative motion model relevant to two-satellite formations which includes the non-Keplerian perturbations due to secular Earth oblateness, as well as the possibility of considering targets moving along a small-eccentricity orbit. An example of trajectory design is shown considering a GEO target as test case. Given this trajectory, pose determination performance is also evaluated within a numerical simulation environment capable of realistically reproducing target-chaser relative dynamics, the operation of a scanning LIDAR selected on board the chaser as relative navigation sensor, and pose estimation algorithms based on the processing of 3D point clouds.

Concepts and techniques for web-based visualization and processing of massive 3D point clouds with semantics

3D point cloud technology facilitates the automated and highly detailed acquisition of real-world environments such as assets, sites, and countries. We present a web-based system for the interactive exploration and inspection of arbitrary large 3D point clouds. Our approach is able to render 3D point clouds with billions of points using spatial data structures and level-of-detail representations. Point-based rendering techniques and post-processing effects are provided to enable task-specific and data-specific filtering, e.g., based on semantics. A set of interaction techniques allows users to collaboratively work with the data (e.g., measuring distances and annotating). Additional value is provided by the system’s ability to display additional, context-providing geodata alongside 3D point clouds and to integrate processing and analysis operations. We have evaluated the presented techniques and in case studies and with different data sets from aerial, mobile, and terrestrial acquisition with up to 120 billion points to show their practicality and feasibility.

MODELLING LANDSCAPE MORPHODYNAMICS BY TERRESTRIAL PHOTOGRAMMETRY: AN APPLICATION TO BEACH AND FLUVIAL SYSTEMS

Beach and fluvial systems are highly dynamic environments, being constantly modified by the action of different natural and anthropic phenomena. To understand their behaviour and to support a sustainable management of these fragile environments, it is very important to have access to cost-effective tools. These methods should be supported on cutting-edge technologies that allow monitoring the dynamics of the natural systems with high periodicity and repeatability at different temporal and spatial scales instead the tedious and expensive field-work that has been carried out up to date. The work herein presented analyses the potential of terrestrial photogrammetry to describe beach morphology. Data processing and generation of high resolution 3D point clouds and derived DEMs is supported by the commercial Agisoft PhotoScan. Model validation is done by comparison of the differences in the elevation among the photogrammetric point cloud and the GPS data along different beach profiles. Results obtained denote the potential that the photogrammetry 3D modelling has to monitor morphological changes and natural events getting differences between 6 and 25 cm. Furthermore, the usefulness of these techniques to control the layout of a fluvial system is tested by the performance of some modeling essays in a hydraulic pilot channel.

MODELLING LANDSCAPE MORPHODYNAMICS BY TERRESTRIAL PHOTOGRAMMETRY: AN APPLICATION TO BEACH AND FLUVIAL SYSTEMS

Abstract. Beach and fluvial systems are highly dynamic environments, being constantly modified by the action of different natural and anthropic phenomena. To understand their behaviour and to support a sustainable management of these fragile environments, it is very important to have access to cost-effective tools. These methods should be supported on cutting-edge technologies that allow monitoring the dynamics of the natural systems with high periodicity and repeatability at different temporal and spatial scales instead the tedious and expensive field-work that has been carried out up to date. The work herein presented analyses the potential of terrestrial photogrammetry to describe beach morphology. Data processing and generation of high resolution 3D point clouds and derived DEMs is supported by the commercial Agisoft PhotoScan. Model validation is done by comparison of the differences in the elevation among the photogrammetric point cloud and the GPS data along different beach profiles. Results obtained denote the potential that the photogrammetry 3D modelling has to monitor morphological changes and natural events getting differences between 6 and 25 cm. Furthermore, the usefulness of these techniques to control the layout of a fluvial system is tested by the performance of some modeling essays in a hydraulic pilot channel.

3D CV Descriptor on Parallel Heterogeneous Platforms

Embedded three-dimensional (3D) Computer Vision (CV) is considered a technology enabler for future consumer applications, attracting a wide interest in academia and industry. However, 3D CV processing is a computation-intensive task. Its high computational cost is directly related to the processing of 3D point clouds, with the 3D descriptor computation representing one of the main bottlenecks. Understanding the main computational challenges of 3D CV applications, as well as the key characteristics, enabling features, and limitations of current computing platforms, is clearly strategic to identify the directions of evolution for future embedded processing systems targeting 3D CV. In this work, an innovative and complex 3D descriptor (called SHOT) has been ported on a high-end and an embedded computing platform. The high-end system is composed by a high-performance Intel CPU coupled with a Nvidia GPU. The embedded platform is, instead, composed by an ARM-based processor, coupled with the STHORM accelerator. STHORM is a many-core low-power accelerator developed by ST Microelectronics, featuring up to 64 computational units. The SHOT descriptor has been parallelized using the OpenCL programming model for both platforms. Finally, we have performed an in-depth performance comparison and analysis between general-purpose processors and accelerators in both high-end and embedded domains, discussing and highlighting the main differences in the Hardware/Software (HW/SW) design methodologies and approaches between high-end and embedded systems targeting 3D CV applications.

Object class segmentation of massive 3D point clouds of urban areas using point cloud topology

A large number of remote-sensing techniques and image-based photogrammetric approaches allow an efficient generation of massive 3D point clouds of our physical environment. The efficient processing, analysis, exploration, and visualization of massive 3D point clouds constitute challenging tasks for applications, systems, and workflows in disciplines such as urban planning, environmental monitoring, disaster management, and homeland security. We present an approach to segment massive 3D point clouds according to object classes of virtual urban environments including terrain, building, vegetation, water, and infrastructure. The classification relies on analysing the point cloud topology; it does not require per-point attributes or representative training data. The approach is based on an iterative multi-pass processing scheme, where each pass focuses on different topological features and considers already detected object classes from previous passes. To cope with the massive amount of data, out-of-core spatial data structures and graphics processing unit (GPU)-accelerated algorithms are utilized. Classification results are discussed based on a massive 3D point cloud with almost 5 billion points of a city. The results indicate that object-class-enriched 3D point clouds can substantially improve analysis algorithms and applications as well as enhance visualization techniques.

Automatic Detection and Classi cation of Objects in Point Clouds using multi-stage Semantics

Due to the increasing availability of large unstructured point clouds from lasers scanning and photogrammetry, there is a growing demand for automatic evaluation methods. Given the complexity of the underlying problems, several new methods resort to using semantic knowledge in particular for object detection and classification support. In this paper, we present a novel approach, which makes use of advanced algorithms, and benefits from intelligent knowledge management strategies for the processing of 3D point clouds and object classification in a scanned scene. In particular, our method extends the use of semantic knowledge to all stages of the processing, including the guidance of the 3D processing algorithms. The complete solution consists of a multi-stage, iterative, concept based on three factors: the modeled knowledge, the package of algorithms, and the classification engine.

Automatic Detection and Classi cation of Objects in Point Clouds using multi-stage Semantics

Due to the increasing availability of large unstructured point clouds from lasers scanning and photogrammetry, there is a growing demand for automatic evaluation methods. Given the complexity of the underlying problems, several new methods resort to using semantic knowledge in particular for object detection and classification support. In this paper, we present a novel approach, which makes use of advanced algorithms, and benefits from intelligent knowledge management strategies for the processing of 3D point clouds and object classification in a scanned scene. In particular, our method extends the use of semantic knowledge to all stages of the processing, including the guidance of the 3D processing algorithms. The complete solution consists of a multi-stage, iterative, concept based on three factors: the modeled knowledge, the package of algorithms, and the classification engine.

Horizontal plane detection from 3D point clouds of buildings

The detection of the horizontal plane or its normal direction is important in the processing of 3D point clouds of buildings. Generally, the horizontal plane is detected from clues of vanishing points of visual images or from the border detection of 3D point data. However, these methods are not fit for most building structures of various kinds. Presented is a novel detection method to extract directly the horizontal plane from 3D point clouds of buildings. Experimental results demonstrate that the method is simple and robust.