Dense 3D Point Clouds Research Articles

Time-Lapse Landform Monitoring in the Pisciarelli (Campi Flegrei-Italy) Fumarole Field Using UAV Photogrammetry

The utility of new imaging technologies to better understand hazardous geological environments cannot be overstated. The combined use of unmanned aerial vehicles (UAV) and digital photogrammetry (DP) represents a rapidly evolving technique that permits geoscientists to obtain detailed spatial data. This can aid in rapid mapping and analyses of dynamic processes that are modifying contemporary landscapes, particularly through the creation of a time series of digital data to help monitor the geomorphological evolution of volcanic structures. Our study comprises a short-term (in geological terms) monitoring program of the dynamic and diffuse Pisciarelli degassing structure caused by the interplay between intensive rainfall and hydrothermal activity. This area, an unstable fumarole field located several hundred meters east of the Solfatara Crater of the Campi Flegrei caldera (southern Italy), is characterized by consistent soil degassing, fluid emission from ephemeral vents, and hot mud pools. This degassing activity is episodically accompanied by seismic swarms and macroscopic morphology changes such as the appearance of vigorously degassing vents, collapsing landslides, and bubbling mud. In late-2019 and 2020, we performed repeated photogrammetric UAV surveys using the Structure from Motion (SfM) technique. This approach allowed us to create dense 3D point clouds and digital orthophotos spanning one year of surveys. The results highlight the benefits of photogrammetry data using UAV for the accurate remote monitoring and mapping of active volcanoes and craters in harsh and dangerous environments.

Local Feature Performance Evaluation for Structure-From-Motion and Multi-View Stereo Using Simulated City-Scale Aerial Imagery

Ubiquitous low cost multi-rotor and fixed wing drones or unmanned aerial vehicles (UAVs) have accelerated the need for reliable, robust, and scalable Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipelines suitable for a variety of flightpath trajectories especially in degraded environments. Feature tracking being a core part of SfM and MVS, is essential for multiview scene modeling and perception, but difficult to evaluate in large scale datasets due to the lack of sufficient ground-truth. For large-scale aerial imagery, accurate camera orientation and dense 3D point cloud accuracy can be used to assess the impact of accurate feature localization and track length. We propose a novel view simulation (or synthesis) framework which generates visually realistic new unseen camera views for feature detection using known high fidelity camera poses for modeling. Seven state-of-the-art local handcrafted and learning-based features are quantitatively evaluated for robustness and matchability within the SfM and MVS pipelines using the open source COLMAP software. Our experimental results provide performance rankings of each feature, using twelve different evaluation metrics across three synthetic city-wide aerial image sequences. We show that recent learned features, SuperPoint and LF-Net, have not only reached the quality of the best handcrafted features like SIFT and SURF, but now outperform them in terms of more accurate 3D camera pose estimates and longer feature tracks. SuperPoint produces 1.51 meter average position error and 0.03° average angular error, while SIFT remains competitive (second best for pose and overall) with 1.78 meter and 0.11° errors respectively.

Toward real-time optical estimation of ocean waves’ space-time fields

Stereo 3D reconstruction is continuously increasing its popularity in the study of mid-to small-scale sea waves. In the recent past, different approaches have been proposed to reconstruct the space-time sea surface elevation field from synchronized stereo frames. Usually, the reconstruction is performed by first recovering a dense and sparse 3D point cloud from stereo pairs and then by interpolating it into a regular grid. Even considering state-of-the-art methods, with typical image resolutions, it's unlikely to perform the first step in less than dozens of seconds (without paralellization), that can easily doubled if we include the subsequent interpolation step. This will limit the applicability of stereo based wave analysis for all the approaches (like monitoring or data assimilation) in which the time between the raw acquisition and the output of processed data is critical.In this paper, we propose a new method to directly estimate the sea surface spectrum over time from a sequence of stereo frames. We exploit the frequency dispersion relation of gravity waves and the (Non-uniform) Fast Fourier Transform to continuously update the 3D surface and 2D wavenumber spectrum estimations given a sparse set of matching features between the two frames. This effectively combines the two aforementioned steps with a performance gain of more than an order of magnitude while keeping a reconstruction accuracy that is extremely close to the current state-of-the-art approaches. Code is available at https://gitlab.com/fibe/wassfast.

Image enhancement comparison to improve underwater cultural heritage survey

This work aims at presenting an underwater image application to obtain an improved 3D model of cultural assets. In 2016, more than 500 images were acquired by a GoPro Camera with a low resolution of 72 dpi and focal length of 3 mm, without flash and are now used to reconstruct the 3D model of some amphoras of a Roman shipwreck found in Albenga (Italy). We have applied state-of-art image enhancement techniques, such as ACE, CLAHE, LAB and SP algorithms, to improve the quality of underwater images affected by low contrast, poor visibility conditions, not uniform lighting, colour variations, noise and blur effect. The visual quality has been evaluated through quantitative metrics, like average luminance, information entropy, average gradient of image, UCIQE and UIQM. Then, our efforts have been devoted to the dense 3D point cloud generation using a SfM-MVS software. In particular, the 3D reconstruction results are in line with the metric evaluation: in fact, the more accurate 3D objects are obtained from that enhanced dataset with the highest measured image quality.

Surface Reconstruction Assessment in Photogrammetric Applications

The image-based 3D reconstruction pipeline aims to generate complete digital representations of the recorded scene, often in the form of 3D surfaces. These surfaces or mesh models are required to be highly detailed as well as accurate enough, especially for metric applications. Surface generation can be considered as a problem integrated in the complete 3D reconstruction workflow and thus visibility information (pixel similarity and image orientation) is leveraged in the meshing procedure contributing to an optimal photo-consistent mesh. Other methods tackle the problem as an independent and subsequent step, generating a mesh model starting from a dense 3D point cloud or even using depth maps, discarding input image information. Out of the vast number of approaches for 3D surface generation, in this study, we considered three state of the art methods. Experiments were performed on benchmark and proprietary datasets of varying nature, scale, shape, image resolution and network designs. Several evaluation metrics were introduced and considered to present qualitative and quantitative assessment of the results.

Fast Reconstruction of 3D Point Cloud Model Using Visual SLAM on Embedded UAV Development Platform

In recent years, the rapid development of unmanned aerial vehicle (UAV) technologies has made data acquisition increasingly convenient, and three-dimensional (3D) reconstruction has emerged as a popular subject of research in this context. These 3D models have many advantages, such as the ability to represent realistic scenes and a large amount of information. However, traditional 3D reconstruction methods are expensive, and require long and complex processing. As a result, they cannot rapidly respond when used in time-sensitive applications, e.g., those for such natural disasters as earthquakes, debris flow, etc. Computer vision-based simultaneous localization and mapping (SLAM) along with hardware development based on embedded systems, can provide a solution to this problem. Based on an analysis of the principle and implementation of the visual SLAM algorithm, this study proposes a fast method to quickly reconstruct a dense 3D point cloud model on a UAV platform combined with an embedded graphics processing unit (GPU). The main contributions are as follows: (1) to resolve the contradiction between the resource limitations and the computational complexity of visual SLAM on UAV platforms, the technologies needed to compute resource allocation, communication between nodes, and data transmission and visualization in an embedded environment were investigated to achieve real-time data acquisition and processing. Visual monitoring to this end is also designed and implemented. (2) To solve the problem of time-consuming algorithmic processing, a corresponding parallel algorithm was designed and implemented based on the parallel programming framework of the compute unified device architecture (CUDA). (3) The visual odometer and methods of 3D “map” reconstruction were designed using under a monocular vision sensor to implement the prototype of the fast 3D reconstruction system. Based on preliminary results of the 3D modeling, the following was noted: (1) the proposed method was feasible. By combining UAV, SLAM, and parallel computing, a simple and efficient 3D reconstruction model of an unknown area was obtained for specific applications. (2) The parallel SLAM algorithm used in this method improved the efficiency of the SLAM algorithm. On the one hand, the SLAM algorithm required 1/6 of the time taken by the structure-from-motion algorithm. On the other hand, the speedup obtained using the parallel SLAM algorithm based on the embedded GPU on our test platform was 7.55 × that of the serial algorithm. (3) The depth map results show that the effective pixel with an error less than 15cm is close to 60%.

New multi‐view human motion capture framework

Estimating human pose and shape without markers is a challenging problem. This study proposes a multiple-view markerless human motion capture framework. Firstly, a multi-view camera system is built for capturing real-time images of moving humans on multiple views. Secondly, by employing the OpenPose method, the authors calculate robust 3D key points from 2D key points of the human body, which are estimated from the multi-view images. And dense 3D point cloud is reconstructed from images. Thirdly, they propose a novel SMPL-based method to represent human motion by fitting the SMPL model to 3D key points and 3D point clouds. In order to achieve a more accurate human pose, a penalty term is utilised to solve the problem of error accumulation in the process of human motion capture. In addition, they present a dense mesh template-based SMPL that can be deformed to point cloud to recover a real human body shape. Finally, they map multi-view colour images onto the human mesh model to acquire rendered mesh. The experimental results show that the proposed method improves the accuracy of human pose and realises the 3D human body model more realistic.

An improved semi-global matching method with optimized matching aggregation constraint

Stereo dense image matching (DIM) is a key technique in generating dense 3D point clouds at low cost, among which semi-global matching (SGM) is one of the best compromise between the matching accuracy and the time cost. Most commercial or open-source DIM software packages therefore adopt SGM as the core algorithm for the 3D point generation, which computes matching results in 2D image space by simply aggregating the matching results of multi-directional 1D paths. However, such aggregations of SGM did not consider the disparity consistency between adjacent pixels in 2D image space, which will finally decrease the matching accuracy. To achieve higher-accuracy while keep the high time efficiency of SGM, this paper proposes an improved SGM with a novel matching aggregation optimization constraint. The core algorithm formulates the matching aggregation as the optimization of a global energy function, and a local solution of the energy function is utilized to impose the disparity consistency between adjacent pixels, which is capable of removing noises in the matching aggregation results and increasing the final matching accuracy at low time cost. Experiments on aerial image dataset show that the proposed method outperformed the traditional SGM method and another improved SGM method. Compared with the traditional SGM, our proposed method can increase the average matching accuracy by at most 11%. Therefore, our proposed method can applied in some smart 3D applications, e.g. 3D change detection, city-scale reconstruction, and global survey mapping.

LARGE SCALE 3D POINT CLOUD MODELING FROM CAD DATABASE IN COMPLEX INDUSTRIAL ENVIRONMENTS

Abstract. In this paper, we present a method to build Computer Aided Design (CAD) representations of dense 3D point cloud scenes by queries in a large CAD model database. This method is applied to real world industrial scenes for infrastructure modeling. The proposed method firstly relies on a region growing algorithm based on novel edge detection method. This algorithm is able to produce geometrically coherent regions which can be agglomerated in order to extract the objects of interest of an industrial environment. Each segment is then processed to compute relevant keypoints and multi-scale features in order to be compared to all CAD models from the database. The best fitting model is estimated together with the rigid six degree of freedom (6 DOF) transformation for positioning the CAD model on the 3D scene. The proposed novel keypoints extractor achieves robust and repeatable results that captures both thin geometrical details and global shape of objects. Our new multi-scale descriptor stacks geometrical information around each keypoint at short and long range, allowing non-ambiguous matching for object recognition and positioning. We illustrate the efficiency of our method in a real-world application on 3D segmentation and modeling of electrical substations.

3D Point Cloud Simplification Based on k-Nearest Neighbor and Clustering

While the reconstruction of 3D objects is increasingly used today, the simplification of 3D point cloud, however, becomes a substantial phase in this process of reconstruction. This is due to the huge amounts of dense 3D point cloud produced by 3D scanning devices. In this paper, a new approach is proposed to simplify 3D point cloud based on k-nearest neighbor (k-NN) and clustering algorithm. Initially, 3D point cloud is divided into clusters using k-means algorithm. Then, an entropy estimation is performed for each cluster to remove the ones that have minimal entropy. In this paper, MATLAB is used to carry out the simulation, and the performance of our method is testified by test dataset. Numerous experiments demonstrate the effectiveness of the proposed simplification method of 3D point cloud.

Cyber-enhanced canine suit with wide-view angle for three-dimensional LiDAR SLAM for indoor environments

Understanding the topographical information of the disaster site can enhance the efficiency and safety of search and rescue missions. This study describes the development of a cyber-enhanced canine suit for three-dimensional (3D) simultaneous localization and mapping within indoor environments. The suit’s weight was approximately 3.0 kg and was compatible with large dogs (>30.0 kg). It could collect dense, wide view-angle, and long-range 3D point clouds. The collected point clouds were processed offline to create 3D maps. The suit was equipped with a protective cover that could prevent physical damage to the light imaging detection and ranging (LiDAR) sensor from collision. The protective cover was designed to (a) eliminate any effects on the performance of the 3D LiDAR and (b) sustain the stress from the impact at the velocity of 20.0 m/s with no physical damage. Additionally, the canine suit was also modified for better fitting and stability of the LiDAR on a wider range of dog bodies. The performance of the suit was demonstrated on real dogs in four different scenarios in areas with different space sizes, dog gait patterns, and ground layouts. The space size and position errors observed did not exceed 0.80% and 1.13%, respectively.

Enhancing Image-Based Multiscale Heritage Recording with Near-Infrared Data

Passive sensors, operating in the visible (VIS) spectrum, have widely been used towards the trans-disciplinary documentation, understanding, and protection of tangible cultural heritage (CH). Although, many heritage science fields benefit significantly from additional information that can be acquired in the near-infrared (NIR) spectrum. NIR imagery, captured for heritage applications, has been mostly investigated with two-dimensional (2D) approaches or by 2D-to-three-dimensional (3D) integrations following complicated techniques, including expensive imaging sensors and setups. The availability of high-resolution digital modified cameras and software implementations of Structure-from-Motion (SfM) and Multiple-View-Stereo (MVS) algorithms, has made the production of models with spectral textures more feasible than ever. In this research, a short review of image-based 3D modeling with NIR data is attempted. The authors aim to investigate the use of near-infrared imagery from relatively low-cost modified sensors for heritage digitization, alongside the usefulness of spectral textures produced, oriented towards heritage science. Therefore, thorough experimentation and assessment with different software are conducted and presented, utilizing NIR imagery and SfM/MVS methods. Dense 3D point clouds and textured meshes have been produced and evaluated for their metric validity and radiometric quality, comparing to results produced from VIS imagery. The datasets employed come from heritage assets of different dimensions, from an archaeological site to a medium-sized artwork, to evaluate implementation on different levels of accuracy and specifications of texture resolution.

A Real-Time 3D Laparoscopic Imaging System: Design, Method, and Validation.

This paper aims to propose a 3D laparoscopic imaging system that can realize dense 3D reconstruction in real time. Based on the active stereo technique which yields high-density, accurate and robust 3D reconstruction by combining structured light and stereo vision, we design a laparoscopic system consisting of two image feedback channels and one pattern projection channel. Remote high-speed image acquisition and pattern generation lay the foundation for the real-time dense 3D surface reconstruction and enable the miniaturization of the laparoscopic probe. To enhance the reconstruction efficiency and accuracy, we propose a novel active stereo method by which the dense 3D point cloud is obtained using only five patterns, while most existing multiple-shot structured light techniques require [Formula: see text] patterns. In our method, dual-frequency phase-shifting fringes are utilized to uniquely encode the pixels of the measured targets, and a dual-codeword matching scheme is developed to simplify the matching procedure and achieve high-precision reconstruction. Compared with the existing structured light techniques, the proposed method shows better real-time efficiency and accuracy in both quantitative and qualitative ways. Ex-vivo experiments demonstrate the robustness of the proposed method to different biological organs and the effectiveness to lesions and deformations of the organs. Feasibility of the proposed system for real-time dense 3D reconstruction is verified in dynamic experiments. According to the experimental results, the system acquires 3D point clouds with a speed of 12 frames per second. Each frame contains more than 40,000 points, and the average errors tested on standard objects are less than 0.2mm. This paper provides a new real-time dense 3D reconstruction method for 3D laparoscopic imaging. The established prototype system has shown good performance in reconstructing surface of biological tissues.

Classification of 3D Point Clouds Using Color Vegetation Indices for Precision Viticulture and Digitizing Applications

Remote sensing applied in the digital transformation of agriculture and, more particularly, in precision viticulture offers methods to map field spatial variability to support site-specific management strategies; these can be based on crop canopy characteristics such as the row height or vegetation cover fraction, requiring accurate three-dimensional (3D) information. To derive canopy information, a set of dense 3D point clouds was generated using photogrammetric techniques on images acquired by an RGB sensor onboard an unmanned aerial vehicle (UAV) in two testing vineyards on two different dates. In addition to the geometry, each point also stores information from the RGB color model, which was used to discriminate between vegetation and bare soil. To the best of our knowledge, the new methodology herein presented consisting of linking point clouds with their spectral information had not previously been applied to automatically estimate vine height. Therefore, the novelty of this work is based on the application of color vegetation indices in point clouds for the automatic detection and classification of points representing vegetation and the later ability to determine the height of vines using as a reference the heights of the points classified as soil. Results from on-ground measurements of the heights of individual grapevines were compared with the estimated heights from the UAV point cloud, showing high determination coefficients (R² > 0.87) and low root-mean-square error (0.070 m). This methodology offers new capabilities for the use of RGB sensors onboard UAV platforms as a tool for precision viticulture and digitizing applications.

Geometric Accuracy Assessments of Orthophoto Production from UAV Aerial Images

Orthophoto mosaic is assembled from aerial perspective images through a process called orthorectification, which eliminate photographic tilts and terrain relief effects. These orthorectified images have been resampled from the original ones that may have been prepared from a DTM which does not accurately model the surface. Meanwhile, some proprietary software such as Agisoft utilizes spatially dense 3D point clouds that are generated from a so called Structure from Motion technique to generate the orthophoto. The software provides a black-box method to regard these clouds as DSM, and it utilizes this surface model to project pixels from the original images. This paper investigates geometric accuracy of the produced orthophoto mosaic according to the American Society of Photogrammetry and Remote Sensing (ASPRS) standards. To minimize scale differences among images, a 35mm fixed-lens camera is mounted on a fixed-wing UAV platform. Flight missions are carried out at around 250m flying height controlled by a navigational grade sensor on board to provide spatial resolution of about 27mm. A number of orthophoto mosaics are produced by varying numbers of GCPs, flight paths configuration and terrain relief differences. The geometric accuracies are assessed through a provision of ICPs on each test field area. Coordinates deviations between the ICP and the corresponding orthophotos are framed into a RMSE figures. Within a 95% confidence level, it is revealed that a suitable orthophoto map scale is up to around 1:500. It is recommended that a cross flight configuration to achieve better results.

Optimized LOAM Using Ground Plane Constraints and SegMatch-Based Loop Detection

Reducing the cumulative error in the process of simultaneous localization and mapping (SLAM) has always been a hot issue. In this paper, in order to improve the localization and mapping accuracy of ground vehicles, we proposed a novel optimized lidar odometry and mapping method using ground plane constraints and SegMatch-based loop detection. We only used the lidar point cloud to estimate the pose between consecutive frames, without any other sensors, such as Global Positioning System (GPS) and Inertial Measurement Unit (IMU). Firstly, the ground plane constraints were used to reduce matching errors. Then, based on more accurate lidar odometry obtained from lidar odometry and mapping (LOAM), SegMatch completed segmentation matching and loop detection to optimize the global pose. The neighborhood search was also used to accomplish the loop detection task in case of failure. Finally, the proposed method was evaluated and compared with the existing 3D lidar SLAM methods. Experiment results showed that the proposed method could realize low drift localization and dense 3D point cloud map construction.

Leaf Area Index evaluation in vineyards using 3D point clouds from UAV imagery

The Leaf Area Index (LAI) is an ecophysiology key parameter characterising the canopy-atmosphere interface where most of the energy fluxes are exchanged. However, producing maps for managing the spatial and temporal variability of LAI in large croplands with traditional techniques is typically laborious and expensive. The objective of this paper is to evaluate the reliability of LAI estimation by processing dense 3D point clouds as a cost-effective alternative to traditional LAI assessments. This would allow for high resolution, extensive and fast mapping of the index, even in hilly and not easily accessible regions. In this setting, the 3D point clouds were generated from UAV-based multispectral imagery and processed by using an innovative methodology presented here. The LAI was estimated by a multivariate linear regression model using crop canopy descriptors derived from the 3D point cloud, which account for canopy thickness, height and leaf density distribution along the wall. For the validation of the estimated LAI, an experiment was conducted in a vineyard in Piedmont: the leaf area of 704 vines was manually measured by the inclined point quadrant approach and six UAV flights were contextually performed to acquire the aerial images. The vineyard LAI estimated by the proposed methodology showed to be correlated with the ones obtained by the traditional manual method. Indeed, the obtained R2 value of 0.82 can be considered fully adequate, compatible to the accuracy of the reference LAI manual measurement.

Reliability of Terrestrial Laser Scanner Measurement in Slope Monitoring

Dense 3D point clouds provided by terrestrial laser scanner (TLS) has demonstrated significant reliability of TLS in landslide monitoring. However, existence of errors in measurement is inevitable which eventually has decreased the quality of TLS data. To concretely measure the capability of TLS in landslide monitoring, this study has performed two epoch measurements using tacheometry (for benchmarking) and TLS (Topcon GLS-2000) at Kulim Techno City, Kedah, Malaysia. Sixteen (16) artificial targets were well-distributed on the slope to determine the accuracy of the employed TLS. Results obtained revealed that Topcon GLS-2000 provides 0.006m of accuracy. However, the presence of high incidence angles in TLS measurement has limited the capability to identify the significant displacement of the targets.

Automatic 3D reconstruction of SEM images based on Nano-robotic manipulation and epipolar plane images

This paper reports a new and general 3D reconstruction algorithm by using the light field reconstruction theory to effectively construct 3D SEM images in a large range. Firstly, a nano-robotic system was employed to automatically capture a group of SEM images along a linear path with a fixed step size, which allowed the 3D SEM images to be reconstructed beyond the field of view (FOV) of SEM. Then, the epipolar-plane images (EPI) were generated, and the depth image was reconstructed based on the specific linear structures emerging in EPI and the automatic depth estimation algorithm. After that, the depth image was stitched and the dense 3D point cloud was obtained by using the delaunay technology. Depth reconstruction with the proposed algorithm does not depend on the matching corresponding points technology. This means nearly all kinds of SEM samples, even those with a simple texture structure or an almost flat surface, can be reconstructed. In addition, the proposed method allows constructing the 3D images out of the FOV of SEM with the assistance of nanorobot. The performance of the proposed algorithm was tested using our self-built database with several microscopic samples. The results demonstrate that the proposed algorithm is general and effective and it is particularly suitable for reconstructing highly complex micro surfaces with a flat surface in a large range.

3D MODELING FOR ROCK ART DOCUMENTATION USING LIGHTWEIGHT MULTISPECTRAL CAMERA

Abstract. This paper discusses the use of the lightweight multispectral camera to acquire three-dimensional data for rock art documentation application. The camera consists of five discrete bands, used for taking the motifs of the rock art paintings on a big structure of a cave based on the close-range photogrammetry technique. The captured images then processed using commercial structure-from-motion photogrammetry software, which automatically extracts the tie point. The extracted tie points were then used as input to generate a dense point cloud based on the multi-view stereo (MVS) and produced the multispectral 3D model, and orthophotos in a different wavelength. For comparison, the paintings and the wall surface also observed by using terrestrial laser scanner which capable of recording thousands of points in a short period of time with high accuracy. The cloud-to-cloud comparison between multispectral and TLS 3D point cloud show a sub-cm discrepancy, considering the used of the natural features as control target during 3D construction. Nevertheless, the processing also provides photorealistic orthophoto, indicates the advantages of the multispectral camera in generating dense 3D point cloud as TLS, photorealistic 3D model as RGB optic camera, and also with the multiwavelength output.