ResNeRF-PCAC: Super Resolving Residual Learning NeRF for High Efficiency Point Cloud Attributes Coding

Abstract. Permafrost landscapes in the Arctic are highly vulnerable to warming, with rapid changes underway. High-resolution remote sensing, especially aerial datasets, offers valuable insights into current permafrost characteristics and thaw dynamics. Here, we present a new dataset of very high resolution orthomosaics, point clouds, and digital surface models that we acquired over permafrost landscapes in northwestern Canada and northern and northwestern Alaska for the purpose of better understanding the impacts of climate change on permafrost landscapes. The imagery was collected with the Modular Aerial Camera System (MACS) during aerial campaigns conducted by the Alfred Wegener Institute in the summers of 2018, 2019, and 2021. The MACS was specifically developed by the German Aerospace Center (DLR) for operation under challenging light conditions in polar environments. It features cameras in the optical and the near-infrared wavelengths with up to a 16 MP resolution. We processed the images to four-band (blue–green–red–near-infrared) orthomosaics and digital surface models with spatial resolutions of 7 to 20 cm as well as 3D point clouds with point densities of up to 41 points m−2. The dataset collection features 102 subprojects from 35 target regions (1.4–161.1 km2 in size). Project sizes range from 4.8 to 336 GB. In total, 3.17 TB were published. The horizontal precision of the datasets is in the range of 1–2 px and vertical precision is better than 0.10 m. The datasets are not radiometrically calibrated. Overall, these very high resolution images and point clouds provide significant opportunities for mapping permafrost landforms and generating detailed training datasets for machine learning, can serve as a baseline for change detection for thermokarst and thermo-erosion processes, and help with upscaling of field measurements to lower-resolution satellite observations. The dataset is available on the PANGAEA repository at https://doi.org/10.1594/PANGAEA.961577 (Rettelbach et al., 2024).

Multiview 3D reconstruction techniques enable digital reconstruction of 3D objects from the real world by fusing different viewpoints of the same object into a single 3D representation. This process is by no means trivial and the acquisition of high quality point cloud representations of dynamic 3D objects is still an open problem. In this paper, an approach for high fidelity 3D point cloud generation using low cost 3D sensing hardware is presented. The proposed approach runs in an efficient low-cost hardware setting based on several Kinect v2 scanners connected to a single PC. It performs autocalibration and runs in real-time exploiting an efficient composition of several filtering methods including Radius Outlier Removal (ROR), Weighted Median filter (WM) and Weighted Inter-Frame Average filtering (WIFA). The performance of the proposed method has been demonstrated through efficient acquisition of dense 3D point clouds of moving objects.

High resolution 3d point cloud data obtained by 3d laser scanning system has become a research hotspot and difficulty in recent years due to its large data volume, irregular data and high scene complexity. Target detection is the basis of scene analysis and understanding, which provides the underlying object and analysis basis for high-level scene understanding. Based on high resolution three-dimensional point cloud data of target recognition and tracking problem both in theory and application is facing great challenge, is a new research topic in this paper, according to the laser point cloud data processing as the research object, analyses the characteristics of lidar point cloud data and data processing of train of thought, analysis of lidar point cloud data storage and retrieval strategy, on the basis of the target recognition based on the laser point cloud data. The lidar data are distributed discretely in form. The discretization here refers to the irregular distribution of the positions and intervals of exponential data points in the three-dimensional space, namely the irregular distribution of data. In recent years, with the rise of deep learning and the large-scale application of deep learning in image detection, speech recognition, text processing and other related fields, it has become one of the current important research topics to use the method of deep learning for target recognition of three-dimensional point cloud data. Its main idea is to learn hierarchical feature expression through supervised way and describe the object from the bottom to the top. This method can effectively improve the ability of object feature representation and the performance of object recognition. Deep learning is also widely used in object recognition, object detection, scene segmentation and other image processing. Therefore, this paper adopts the method of deep learning to classify and identify 3d objects.

As Light Detection And Ranging (LiDAR) increasingly becomes an essential component in robotic navigation and autonomous driving, the processing of high throughput 3D point clouds in real time is widely required. This work considers the point cloud k-Nearest Neighbor (kNN) search, which is an important 3D processing kernel. Although applying fine-grained parallelism optimization on internal processing, e.g., using multiple workers, has demonstrated high efficiency, previous accelerators with DDR external memory are fundamentally limited by the external bandwidth bottleneck. To break this bottleneck, this work proposes a highly parallel architecture, namely ParallelNN, for highly efficient kNN search processing of high throughput point clouds. First, we optimize the multichannel cache based on High Bandwidth Memory (HBM) and on-chip memory to provide large external bandwidth. Then, a novel parallel depth-first octree construction algorithm is proposed and mapped onto multiple construction branches with trace-coded construction queues, which can regularize random accesses and perform multi-branch octree construction efficiently. Furthermore, in the search stage, we present algorithm-architecture co-optimization, including parallel keyframe-based scheduling and multi-branch flexible search engines, to provide conflict-free access and maximum reuse opportunities for reference points, which achieves more than 27.0× speedup compared with baseline architectures. We prototype ParallelNN on Virtex HBM FPGA and perform extensive benchmarking on the KITTI dataset. The results demonstrate that ParallelNN achieves up to 107.7× and 12.1× speedup over CPU and GPU implementations, while being more energy efficient, e.g., outperforming CPU and GPU implementations by 73.6× and 31.1×, respectively. Besides, with the proposed algorithm-architecture co-optimization, ParallelNN achieves 11.4× speedup over state-of-the-art architecture. Moreover, ParallelNN is configurable and can be easily generalized to similar octree-based applications.

Photogrammetric processing algorithms can suffer problems due to either the initial image quality (noise, low radiometric quality, shadows and so on) or to certain surface materials (shiny or textureless objects). This can result in noisy point clouds and/or difficulties in feature extraction. Specifically, dense point clouds which are generated with photogrammetric method using a lightweight thermal camera, are more noisy and sparse than the point clouds of high-resolution digital camera images. In this paper, new method which produces more reliable and dense thermal point cloud using the sparse thermal point cloud and high resolution digital point cloud was considered. Both thermal and digital images were obtained with UAS (Unmanned Aerial System) based lightweight Optris PI 450 and Canon EOS 605D camera images. Thermal and digital point clouds, and orthophotos were produced using photogrammetric methods. Problematic thermal point cloud was transformed to a high density thermal point cloud using image processing methods such as rasterizing, registering, interpolation and filling. The results showed that the obtained thermal point cloud - up to chosen processing parameters - was 87% more densify than the original point cloud. The second improvement was gained at the height accuracy of the thermal point cloud. New densified point cloud has more consistent elevation model while the original thermal point cloud shows serious deviations from the expected surface model.

Abstract. Photogrammetric processing algorithms can suffer problems due to either the initial image quality (noise, low radiometric quality, shadows and so on) or to certain surface materials (shiny or textureless objects). This can result in noisy point clouds and/or difficulties in feature extraction. Specifically, dense point clouds which are generated with photogrammetric method using a lightweight thermal camera, are more noisy and sparse than the point clouds of high-resolution digital camera images. In this paper, new method which produces more reliable and dense thermal point cloud using the sparse thermal point cloud and high resolution digital point cloud was considered. Both thermal and digital images were obtained with UAS (Unmanned Aerial System) based lightweight Optris PI 450 and Canon EOS 605D camera images. Thermal and digital point clouds, and orthophotos were produced using photogrammetric methods. Problematic thermal point cloud was transformed to a high density thermal point cloud using image processing methods such as rasterizing, registering, interpolation and filling. The results showed that the obtained thermal point cloud - up to chosen processing parameters - was 87% more densify than the original point cloud. The second improvement was gained at the height accuracy of the thermal point cloud. New densified point cloud has more consistent elevation model while the original thermal point cloud shows serious deviations from the expected surface model.

This paper expands on previous studies by the authors into 3D imaging with a single-beam laser rangefinder (LRF) by implementing real-time attitude maneuvers of a chaser satellite flying in relative orbit around a resident space object (RSO). Point clouds generated with an LRF are much sparser than those generated with an imaging LIDAR, making it difficult to autonomously distinguish between gaps in coverage and truly empty space. Furthermore, if both the attitude and the shape of the target RSO are unknown, it is particularly difficult to register a collection of LRF strike points together and detect gaps in strike point coverage in realtime. This paper presents the incorporation of a narrow field of-view (NFOV) camera that detects the strike point on the RSO and supplements LRF distance measurements with image data. This data is used to generate attitude command profiles that efficiently fill LRF coverage gaps and generate high density point clouds, thus maximizing coverage of an unknown RSO. Results obtained so far point the way to a real-time implementation of the algorithm. A method to detect and close gaps in LRF strike point coverage is presented first. Coverage gap detection is achieved using Voronoi diagrams, where Voronoi cells are centered at the LRF strike points. A three-part algorithm is used that 1) creates a 3D panoramic map from “stitched” NFOV camera images; 2) correlates the areas of sparse LRF coverage to the map; and 3) generates attitude commands to close the coverage gaps. The map provides a consistent and reliable method to register positions of strike points relative to each other and to the NFOV image of the RSO without a priori knowledge of the RSO attitude. Using this algorithm, gaps and sparse areas in LRF coverage are covered with strike points, allowing for the generation of a higher-resolution point cloud than that obtained with preprogrammed attitude profiles. Attitude maneuvers can now be designed on-line in real-time such that they satisfy the constraints of the chaser spacecraft attitude determination and control system. Finally, the effectiveness of the camera-aided generation of attitude profiles is analyzed by using a weighted edge reconstruction metric, and comparing results to those generated with pre-programmed attitude maneuvers. The effect of on-line maneuver generation on the overall decrease of time and propellant expenditure to generate an adequate point cloud is also discussed. The analysis bears particular relevance to low-budget, nano-satellite demonstration missions for space-based space situational awareness (SSA).

Terrestrial laserscanning (TLS), also called ground-based LiDAR (Light Detection And Ranging) is a relatively new method which revolutionised geomorphological research in many domains. However, detailed studies of tidal flats by TLS have not been described in the literature yet. This study aims to fill this methodological gap by the application of TLS at two different locations on the coast of Jiangsu Province, Eastern China, and an assessment of the usability of this method for geomorphological research in such environments. The acquired point clouds are first processed to remove erroneous and noisy points. Subsequently, point clouds are computed to produce polygonal meshes and grid-based digital terrain model (DTM) more commonly used by the scientific community. The accuracy of the measurements is assessed by an analysis of elevation deviations for flat and horizontal concrete blocks. High quality point clouds with point densities of up to 4,000 points/m2 were acquired for a distance of up to 200 m. The data allowed for the detection of small landforms such as tidal channels, creeks and ripples in centimetre and decimetre scale. The point clouds had an average error of approximately 3 mm, however for some few points errors of up to 1.8 cm were detected. Based on the results it can be concluded that TLS can be a useful additional method for geomorphological research on tidal flats due to its ability to describe the landforms from high density point clouds. Repeated scanning could therefore provide data to quantitatively and qualitatively describe geomorphological changes over wider areas and thereby improve the understanding of sedimentation and erosion on tidal flats.

High resolution satellite imagery and 3D laser point cloud data provide precise geometry, rich spectral information and clear texture of feature. The segmentation of high resolution remote sensing images and 3D laser point cloud is the basis of object-oriented remote sensing image analysis, for the segmentation results will directly influence the accuracy of subsequent analysis and discrimination. Currently, there still lacks a common segmentation theory to support these algorithms. So when we face a specific problem, we should determine applicability of the segmentation method through segmentation accuracy assessment, and then determine an optimal segmentation. To today, the most common method for evaluating the effectiveness of a segmentation method is subjective evaluation and supervised evaluation. For providing a more objective evaluation result, we have carried out following work. Analysis and comparison previous proposed image segmentation accuracy evaluation methods, which are area-based metrics, location-based metrics and combinations metrics. 3D point cloud data, which was gathered by Reigl VZ1000, was used to make two-dimensional transformation of point cloud data. The object-oriented segmentation result of aquaculture farm, building and farmland polygons were used as test object and adopted to evaluate segmentation accuracy.

Semi-automatic extraction of rock mass structural data from high resolution LIDAR point clouds

Abstract. Fogo in the Cabo Verde archipelago off western Africa is one of the most prominent and active ocean island volcanoes on Earth, posing an important hazard both to local populations and at a regional level. The last eruption took place between 23 November 2014 and 8 February 2015 in the Chã das Caldeiras area at an elevation close to 1800 ma.s.l. The eruptive episode gave origin to extensive lava flows that almost fully destroyed the settlements of Bangaeira, Portela and Ilhéu de Losna. During December 2016 a survey of the Chã das Caldeiras area was conducted using a fixed-wing unmanned aerial vehicle (UAV) and real-time kinematic (RTK) global navigation satellite system (GNSS), with the objective of improving the terrain models and visible imagery derived from satellite platforms, from metric to decimetric resolution and accuracy. The main result is a very high resolution and quality 3D point cloud with a root mean square error of 0.08 m in X, 0.11 m in Y and 0.12 m in Z, which fully covers the most recent lava flows. The survey comprises an area of 23.9 km2 and used 2909 calibrated images with an average ground sampling distance of 7.2 cm. The dense point cloud, digital surface models and orthomosaics with 25 and 10 cm resolutions, a 50 cm spaced elevation contour shapefile, and a 3D texture mesh, as well as the full aerial survey dataset are provided. The delineation of the 2014/15 lava flows covers an area of 4.53 km2, which is smaller but more accurate than the previous estimates from 4.8 to 4.97 km2. The difference in the calculated area, when compared to previously reported values, is due to a more detailed mapping of the flow geometry and to the exclusion of the areas corresponding to kīpukas (outcrops surrounded by lava flows). Our study provides a very high resolution dataset of the areas affected by Fogo's latest eruption and is a case study supporting the advantageous use of UAV aerial photography surveys in disaster-prone areas. This dataset provides accurate baseline data for future eruptions, allowing for different applications in Earth system sciences, such as hydrology, ecology and spatial modelling, as well as to planning. The dataset is available for download at https://doi.org/10.5281/zenodo.4718520 (Vieira et al., 2021).

In this paper we present a novel sensing system, robust Near-infrared Structured Light Scanning (NIRSL) for three-dimensional human model scanning application. Human model scanning due to its nature of various hair and dress appearance and body motion has long been a challenging task. Previous structured light scanning methods typically emitted visible coded light patterns onto static and opaque objects to establish correspondence between a projector and a camera for triangulation. In the success of these methods rely on scanning objects with proper reflective surface for visible light, such as plaster, light colored cloth. Whereas for human model scanning application, conventional methods suffer from low signal to noise ratio caused by low contrast of visible light over the human body. The proposed robust NIRSL, as implemented with the near infrared light, is capable of recovering those dark surfaces, such as hair, dark jeans and black shoes under visible illumination. Moreover, successful structured light scan relies on the assumption that the subject is static during scanning. Due to the nature of body motion, it is very time sensitive to keep this assumption in the case of human model scan. The proposed sensing system, by utilizing the new near-infrared capable high speed LightCrafter DLP projector, is robust to motion, provides accurate and high resolution three-dimensional point cloud, making our system more efficient and robust for human model reconstruction. Experimental results demonstrate that our system is effective and efficient to scan real human models with various dark hair, jeans and shoes, robust to human body motion and produces accurate and high resolution 3D point cloud.

Abstract. With the rapid developments of the sensor technology, high spatial resolution imagery and airborne Lidar point clouds can be captured nowadays, which make classification, extraction, evaluation and analysis of a broad range of object features available. High resolution imagery, Lidar dataset and parcel map can be widely used for classification as information carriers. Therefore, refinement of objects classification is made possible for the urban land cover. The paper presents an approach to object based image analysis (OBIA) combing high spatial resolution imagery and airborne Lidar point clouds. The advanced workflow for urban land cover is designed with four components. Firstly, colour-infrared TrueOrtho photo and laser point clouds were pre-processed to derive the parcel map of water bodies and nDSM respectively. Secondly, image objects are created via multi-resolution image segmentation integrating scale parameter, the colour and shape properties with compactness criterion. Image can be subdivided into separate object regions. Thirdly, image objects classification is performed on the basis of segmentation and a rule set of knowledge decision tree. These objects imagery are classified into six classes such as water bodies, low vegetation/grass, tree, low building, high building and road. Finally, in order to assess the validity of the classification results for six classes, accuracy assessment is performed through comparing randomly distributed reference points of TrueOrtho imagery with the classification results, forming the confusion matrix and calculating overall accuracy and Kappa coefficient. The study area focuses on test site Vaihingen/Enz and a patch of test datasets comes from the benchmark of ISPRS WG III/4 test project. The classification results show higher overall accuracy for most types of urban land cover. Overall accuracy is 89.5% and Kappa coefficient equals to 0.865. The OBIA approach provides an effective and convenient way to combine high resolution imagery and Lidar ancillary data for classification of urban land cover.

With the rapid developments of the sensor technology, high spatial resolution imagery and airborne Lidar point clouds can be captured nowadays, which make classification, extraction, evaluation and analysis of a broad range of object features available. High resolution imagery, Lidar dataset and parcel map can be widely used for classification as information carriers. Therefore, refinement of objects classification is made possible for the urban land cover. The paper presents an approach to object based image analysis (OBIA) combing high spatial resolution imagery and airborne Lidar point clouds. The advanced workflow for urban land cover is designed with four components. Firstly, colour-infrared TrueOrtho photo and laser point clouds were pre-processed to derive the parcel map of water bodies and nDSM respectively. Secondly, image objects are created via multi-resolution image segmentation integrating scale parameter, the colour and shape properties with compactness criterion. Image can be subdivided into separate object regions. Thirdly, image objects classification is performed on the basis of segmentation and a rule set of knowledge decision tree. These objects imagery are classified into six classes such as water bodies, low vegetation/grass, tree, low building, high building and road. Finally, in order to assess the validity of the classification results for six classes, accuracy assessment is performed through comparing randomly distributed reference points of TrueOrtho imagery with the classification results, forming the confusion matrix and calculating overall accuracy and Kappa coefficient. The study area focuses on test site Vaihingen/Enz and a patch of test datasets comes from the benchmark of ISPRS WG III/4 test project. The classification results show higher overall accuracy for most types of urban land cover. Overall accuracy is 89.5% and Kappa coefficient equals to 0.865. The OBIA approach provides an effective and convenient way to combine high resolution imagery and Lidar ancillary data for classification of urban land cover.

In this paper we present a novel sensing system, robust Near-infrared Structured Light Scanning (NIRSL) for three-dimensional human model scanning application. Human model scanning due to its nature of various hair and dress appearance and body motion has long been a challenging task. Previous structured light scanning methods typically emitted visible coded light patterns onto static and opaque objects to establish correspondence between a projector and a camera for triangulation. In the success of these methods rely on scanning objects with proper reflective surface for visible light, such as plaster, light colored cloth. Whereas for human model scanning application, conventional methods suffer from low signal to noise ratio caused by low contrast of visible light over the human body. The proposed robust NIRSL, as implemented with the near infrared light, is capable of recovering those dark surfaces, such as hair, dark jeans and black shoes under visible illumination. Moreover, successful structured light scan relies on the assumption that the subject is static during scanning. Due to the nature of body motion, it is very time sensitive to keep this assumption in the case of human model scan. The proposed sensing system, by utilizing the new near-infrared capable high speed LightCrafter DLP projector, is robust to motion, provides accurate and high resolution three-dimensional point cloud, making our system more efficient and robust for human model reconstruction. Experimental results demonstrate that our system is effective and efficient to scan real human models with various dark hair, jeans and shoes, robust to human body motion and produces accurate and high resolution 3D point cloud.