Point Cloud Image Research Articles

An Algorithm for Building Exterior Facade Corner Point Extraction Based on UAV Images and Point Clouds

The high-precision building exterior facade corner point (BEFCP) is an essential element in topographic and cadastral surveys. However, current extraction methods rely on the interactions of humans with the 3D real-scene models produced by unmanned aerial vehicle (UAV) oblique photogrammetry, which have a high workload, low efficiency, poor precision, and cannot satisfy the requirements of automation. The dense point cloud contains discrete 3D building structure information. Still, it is challenging to accurately filter out the partial point cloud characterizing the building structure from it in order to achieve BEFCP extraction. The BEFCPs are always located on the plumb line of the building’s exterior wall. Thus, this paper back-calculated the plumb line from the image and designed a photographic ray corresponding to the image point and point cloud intersection point calculation algorithm to recover its approximate spatial position in order to successfully extract the accurate point cloud in the building structure neighborhood. It then utilized the high signal-to-noise ratio property of the point cloud as a base to eliminate the noise points and, finally, accurately located the building exterior façade corner points by recovering the building structure through segmental linear fitting of the point cloud. The proposed algorithm conducted automated building exterior facade corner point extraction via both of planar-to-stereo and rough-to-precise strategies, reached a 92.06% correctness rate and ±4.5 cm point mean square location error in the experiment, and was able to extract and distinguish the building exterior facade corner points under eaves obstruction and extreme proximity. It is suitable for all high-precision surveying and mapping tasks in building areas based on oblique photogrammetry, which can effectively improve the automation of mapping production.

A workflow to visualize vertebrate eyes in 3D.

To establish a workflow to visualize the surgical anatomy in 3D based on histological data of eyes of experimental animals for improving the planning of complex surgical procedures. Four C57BL/6J wild-type(wt) mouse eyes, three Brown Norway rat eyes and four Chinchilla Bastard rabbit eyes were enucleated and processed for standard histology with serial sections and hematoxylin and eosin staining. Image stacks were processed to obtain a representation of the eye anatomy in 3D. In addition, virtual image stacks and 3D point clouds were generated by processing sagittal sections of eyes with stepwise 180° rotation and projection around the eye axis to construct a rotationally symmetric 3D model from one single sagittal section. Serial sections of whole eyes of mice, rats and rabbits showed significant artifacts interfering with a practical image stack generation and straightforward 3D reconstruction despite the application of image registration techniques. A workflow was established to obtain a 3D image of the eye based on virtual image stacks and point cloud generation by rotation of a single sagittal section of the eye around the symmetry axis. By analyzing the tissue shrinkage during histological processing true biometric reconstructions of the eyes were feasible making the resulting model usable for 3D modeling and simulation, e.g. for planning of complex surgical procedures in different species. Because serial sections of the eye with standard histological protocols yielded too many artifacts for a straightforward 3D visualization we reconstructed a pseudorealistic 3D model based on virtual image stacks and point cloud generation calculated from a single sagittal section of the eye. Such a model detailing microscopic structures of the whole eye will allow for a specific planning of surgical procedures in small animal eyes in order to prevent surgical complications in a very early stage of an experiment and it will support the design and development of complex intraocular implants. It will therefore be helpful in surgical teaching and improve laboratory animal welfare by an expected reduction of experimental animal numbers. Further processing including integration of mechanical tissue properties is needed to convert these 3D models into a practical virtual reality teaching and simulation platform for eyes of several species.

ProxyMix: Proxy-based Mixup training with label refinery for source-free domain adaptation

Due to privacy concerns and data transmission issues, Source-free Unsupervised Domain Adaptation (SFDA) has gained popularity. It exploits pre-trained source models, rather than raw source data for target learning, to transfer knowledge from a labeled source domain to an unlabeled target domain. Existing methods solve this problem typically with additional parameters or noisy pseudo labels, and we propose an effective method named Proxy-based Mixup training with label refinery (ProxyMix) to avoid these drawbacks. To avoid additional parameters and leverages information in the source model, ProxyMix defines classifier weights as class prototypes and creates a class-balanced proxy source domain using nearest neighbors of the prototypes. To improve the reliability of pseudo labels, we further propose the frequency-weighted aggregation strategy to generate soft pseudo labels for unlabeled target data. Our strategy utilizes target features’ internal structure, increases weights of low-frequency class samples, and aligns the proxy and target domains using inter- and intra-domain mixup regularization. This mitigates the negative impact of noisy labels. Experiments on three 2D image and 3D point cloud object recognition benchmarks demonstrate that ProxyMix yields state-of-the-art performance for source-free UDA tasks.

Determination of extensibility and toughness of wheat-flour dough based on bubbles blown by sheeted dough and airflow-3D imaging technique

The resistance strength (toughness) and extensibility of dough are closely related to the quality of the dough product. In this study, the toughness and extensibility of dough were detected using an airflow-structured light 3D imaging technique combine with bubbles blown out of sheeted dough. The spatial deformation characteristics and point cloud features of each point cloud image were determined using the point cloud data processing algorithms. The frame, height, volume, surface area, projected area of three planes, and three point cloud features corresponding to the maximum height of the bubble were taken as the extracted features. A total of eight features were selected, which were consequently concentrated in the first three principal components (PCs). The first three PCs were fed into three different prediction models. The ELM model achieved a relatively higher prediction accuracy than the other two models. For toughness and extensibility, Rp were 9092 and 0.8950, respectively.

Cultural relic reconstruction device based on ORB-SLAM

ORB-SLAM, an advanced visual SLAM technology, has excellent positioning and mapping capabilities [1]. In the reconstruction of cultural relics, ORB-SLAM technology can achieve autonomous perception and modeling of the surrounding environment of cultural relics by fusing and analyzing the images of monocular depth cameras, thereby reconstructing the 3D model of cultural relics, providing new possibilities for the protection and inheritance of cultural relics. Point cloud images are a digital technology used for 3D modeling, which can save point data on object surfaces and their relationships as 3D point clouds [2]. In terms of cultural relic reconstruction, point cloud images can restore and reconstruct the shape, size, texture, and other features of cultural relics by processing the collected point data on the surface of cultural relics into a 3D model. The purpose of this device is to reconstruct cultural relics by generating point cloud images based on ORB-SLAM technology. Combined with the designed handheld pan tilt device, it can achieve multi-angle point cloud mapping of cultural relics scenes in complex terrain. This paper will introduce the handheld gimbal design of the device, the realization of point cloud mapping based on the ORB-SLAM3 algorithm [3], and the application of point cloud scenes. Through the research in this article, it is expected to provide a new and efficient method for cultural relic reconstruction in the field of cultural relic protection and make contributions to the protection and inheritance of cultural heritage.

UAV hyperspectral combined with LiDAR to estimate chlorophyll content at the stand and individual tree scales.

Chlorophyll is an important indicator of vegetation health status, accurate estimation of which is important for evaluating forest carbon sink. In this study, we estimated the chlorophyll content of coniferous forests, broad-leaved forests and mixed forest stands at stand and individual tree level by unmanned air vehicle (UAV) hyperspectral data combined with light detection and ranging (LiDAR) point clouds, which improved the non-destructive estimation accuracy of forest chlorophyll. We further comprehensively analyzed the spatial distribution of chlorophyll content at different scales. A total of 36 spectral characteristic variables related to chlorophyll content were screened by correlation analysis based on the fusion of UAV hyperspectral data and LiDAR point clouds combining with the empirical data from ground plots. We constructed multiple models for chlorophyll estimation by using statistical model, including multiple stepwise regression, BP neural network, BP neural network optimized by firefly algorithm, random forest and hybrid data-driven PROSPECT mechanism model. The optimal model was selected to estimate the chlorophyll content. The horizontal and vertical distribution of chlorophyll content at the stand level and individual tree level were analyzed. The results showed that the random forest model was superior to the models constructed by multiple stepwise regression, BP neural network and BP neural network optimized by firefly algorithm for chlorophyll estimation with R2 and RMSE of 0.59-0.64 and 3.79-5.83 μg·cm-2, respectively. The accuracy of the mechanism model was the highest, with R2 and RMSE of 0.97 and 3.40 μg·cm-2. The chlorophyll contents differed across stand types, with that in broad-leaved forest (25.25-31.60 μg·cm-2) being higher than mixed forest (13.52-23.93 μg·cm-2) and coniferous forest (6.40-13.71 μg·cm-2). There were significant differences in chlorophyll contents the in vertical direction among different stands. For individual tree of different species, the chlorophyll content inside the canopy was lower than that outside the canopy in the horizontal direction. In the vertical direction, there was no difference in chlorophyll content among different layers of Pinus sylvestris var. mongolica canopy. However, significant differences were observed among the upper, middle, and lower layers of Juglans mandshurica canopy. Using the fusion of hyperspectral image and LiDAR point cloud data, the mechanism model driven by hybrid data could effectively improve the accuracy and stability of chlorophyll content estimation at different scales.

Ore extraction and analysis from RGB image and 3D Point Cloud

Ore extraction and analysis from RGB image and 3D Point Cloud

Terahertz 3D point cloud imaging for complex targets.

The reconstruction of complex targets using terahertz technology is often hindered by diffraction and interference of electromagnetic waves, leading to the loss of fine target details. In this research article, we have introduced a terahertz synthetic aperture radar (SAR) imaging method that integrates an iterative closest point (ICP) algorithm, referred to as SAR-ICP, to achieve accurate reconstruction of intricate target structures. To accomplish this, multiple sets of point cloud data are acquired by varying the illumination viewpoint. The ICP algorithm is then employed to align and fuse these datasets, resulting in the generation of high-quality three-dimensional (3D) images. The experimental results validate the effectiveness of the proposed SAR-ICP method. The information entropy of the reconstructed 3D image using the SAR-ICP is approximately 0.05 times that of the conventional SAR method, indicating a superior image quality. In the future, we anticipate the widespread application of this method in areas such as security inspection, non-destructive testing, and other complex scenarios.

A Specialized Database for Autonomous Vehicles Based on the KITTI Vision Benchmark

Autonomous driving systems have emerged with the promise of preventing accidents. The first critical aspect of these systems is perception, where the regular practice is the use of top-view point clouds as the input; however, the existing databases in this area only present scenes with 3D point clouds and their respective labels. This generates an opportunity, and the objective of this work is to present a database with scenes directly in the top-view and their labels in the respective plane, as well as adding a segmentation map for each scene as a label for segmentation work. The method used during the creation of the proposed database is presented; this covers how to transform 3D to 2D top-view image point clouds, how the detection labels in the plane are generated, and how to implement a neural network for the generated segmentation maps of each scene. Using this method, a database was developed with 7481 scenes, each with its corresponding top-view image, label file, and segmentation map, where the road segmentation metrics are as follows: F1, 95.77; AP, 92.54; ACC, 97.53; PRE, 94.34; and REC, 97.25. This article presents the development of a database for segmentation and detection assignments, highlighting its particular use for environmental perception works.

Rapid extraction of pavement aggregate gradation based on point clouds using deep learning networks

Usage of asphalt mixture with poor gradation will most likely lead to pavement deficiency. There is a growing need for rapid and non-destructive methods to extract pavement aggregate gradation. In this study, a deep learning-based method that utilizes point clouds data for gradation extraction was proposed. Firstly, a data enhancement algorithm along with three data format conversion methods (aligned point cloud, voxel, and depth image) were proposed to preprocess the original collected point clouds. Subsequently, different neural network models were designed for each data format to extract gradation. Finally, a multi-feature fusion network was developed, which using extraction network as the backbone and additional auxiliary information. In the case study, the MAE loss of multi-feature fusion networks with PointNet, Vox-ResNet34 and GoogLeNet-v4 as the backbone respectively achieved 0.202, 0.142 and 0.046 on the test set, which means an estimation accuracy of more than 95% for the pavement aggregate gradation.

Pavement crack detection based on point cloud data and data fusion.

The three-dimensional detection in point cloud data for pavement cracks has drawn the attention of many researchers recently. In the field of pavement surface point cloud detection, the key tasks include the identification of pavement cracks and the extraction of the location and size information of pavement cracks. Based on the point cloud data of pavement surface, we developed two methods to directly extract and detect cracks, respectively. The first method is based on the improved sliding window algorithm by combining the random sample consensus (RANSAC) technique to directly extract the crack information from point clouds. The second method is developed based on YOLOv5 to process the two-dimensional images transformed from point cloud data for automatic pavement crack detection. We also attempted to fuse the point cloud images with greyscale images as input for the YOLOv5. Analysis results show that the improved sliding window algorithm efficiently extracts pavement cracks with less noise, and the YOLOv5-based method obtains a good detection of pavement cracks. This article is part of the theme issue 'Artificial intelligence in failure analysis of transportation infrastructure and materials'.

Visual Quality Assessment of Point Clouds Compared to Natural Reference Images

This paper proposes a point cloud (PC) visual quality assessment (VQA) framework that reflects the human visual system (HVS). The proposed framework compares natural images acquired using a digital camera and PC images generated via 2D projection in terms of appropriate objective quality evaluation metrics. Humans primarily consume natural images; thus, human knowledge is typically formed from natural images. Thus, natural images can be more reliable reference data than PC data. The proposed framework performs an image alignment process based on feature matching and image warping to use the natural images as a reference which enhances the similarities of the acquired natural and corresponding PC images. The framework facilitates identifying which objective VQA metrics can be used to reflect the HVS effectively. We constructed a database of natural images and three PC image qualities, and objective and subjective VQAs were conducted. The experimental result demonstrates that the acceptable consistency among different PC qualities appears in the metrics that compare the global structural similarity of images. We found that the SSIM, MAD, and GMSD achieved remarkable Spearman rank-order correlation coefficient scores of 0.882, 0.871, and 0.930, respectively. Thus, the proposed framework can reflect the HVS by comparing the global structural similarity between PC and natural reference images.

Photometric LiDAR and RGB-D Bundle Adjustment

The joint optimization of the sensor trajectory and 3D map is a crucial characteristic of Simultaneous Localization and Mapping SLAM systems. To achieve this, the gold standard is Bundle Adjustment (BA). Modern 3D LiDARs now retain higher resolutions that enable the creation of point cloud images resembling those taken by conventional cameras. Nevertheless, the typical effective global refinement techniques employed for RGB-D sensors are not widely applied to LiDARs. This paper presents a novel BA photometric strategy that accounts for both RGB-Dand LiDAR in the same way. Our work can be used on top of any SLAM/GNSS estimate to improve and refine the initial trajectory. We conducted different experiments using these two depth sensors on public benchmarks. Our results show that our system performs on par or better compared to other state-of-the-art ad-hoc SLAM/BA strategies, free from data association and without making assumptions about the environment. In addition, we present the benefit of jointly using RGB-D and LiDAR within our unified method. We finally release an open-source CUDA/C++ implementation at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/digiamm/ba_md_slam</uri> .

MGIC: Multigrid-in-Channels Neural Network Architectures

We present a multigrid-in-channels (MGIC) approach that tackles the quadratic growth of the number of parameters with respect to the number of channels in standard convolutional neural networks (CNNs). Thereby our approach addresses the redundancy in CNNs that is also exposed by the recent success of lightweight CNNs. Lightweight CNNs can achieve comparable accuracy to standard CNNs with fewer parameters; however, the number of weights still scales quadratically with the CNN’s width. Our MGIC architectures replace each CNN block with an MGIC counterpart that utilizes a hierarchy of nested grouped convolutions of small group size to address this. Hence, our proposed architectures scale linearly with respect to the network’s width while retaining full coupling of the channels as in standard CNNs. Our extensive experiments on image classification, segmentation, and point cloud classification show that applying this strategy to different architectures like ResNet and MobileNetV3 reduces the number of parameters while obtaining similar or better accuracy.

CALIP: Zero-Shot Enhancement of CLIP with Parameter-Free Attention

Contrastive Language-Image Pre-training (CLIP) has been shown to learn visual representations with promising zero-shot performance. To further improve its downstream accuracy, existing works propose additional learnable modules upon CLIP and fine-tune them by few-shot training sets. However, the resulting extra training cost and data requirement severely hinder the efficiency for model deployment and knowledge transfer. In this paper, we introduce a free-lunch enhancement method, CALIP, to boost CLIP's zero-shot performance via a parameter-free attention module. Specifically, we guide visual and textual representations to interact with each other and explore cross-modal informative features via attention. As the pre-training has largely reduced the embedding distances between two modalities, we discard all learnable parameters in the attention and bidirectionally update the multi-modal features, enabling the whole process to be parameter-free and training-free. In this way, the images are blended with textual-aware signals and the text representations become visual-guided for better adaptive zero-shot alignment. We evaluate CALIP on various benchmarks of 14 datasets for both 2D image and 3D point cloud few-shot classification, showing consistent zero-shot performance improvement over CLIP. Based on that, we further insert a small number of linear layers in CALIP's attention module and verify our robustness under the few-shot settings, which also achieves leading performance compared to existing methods. Those extensive experiments demonstrate the superiority of our approach for efficient enhancement of CLIP. Code is available at https://github.com/ZiyuGuo99/CALIP.

Posture identification for stall-housed sows around estrus using a robotic imaging system

Posture patterns of sows or gilts could reflect their welfare, estrus, and health status. However, due to the large animal-to-staff ratio at sow farms, there is not yet a solution to closely monitor individual sows’ posture patterns to determine their estrus status and make other managemental decisions. This study aimed to evaluate the performance of the convolutional neural network (CNN) models in recognizing sows’ posture patterns using images of a LiDAR camera. The camera equipped on a wall-mounted robotic platform was used to take images of 21 sows from the back view at a 5-minute interval. Four types of images (IR: infrared image, DI: depth image, PCI: point cloud image, PCI_IR: point cloud image with IR as the alpha channel) were used to identify the postures of individual sows. Five CNN models were evaluated in this study, including SP_CNN (5 layers), VGG16, MobileNet, Xception, and DenseNet121. Results showed that the image types had no significant impact on the validation accuracy of posture recognition (99.8 ± 0.2%, p > 0.1). Statistical results indicate that VGG16 required significantly more time (t = 2,218 ms) than the other four models in processing single images without significantly improving identification accuracy. The SP_CNN model had a significantly lower validation accuracy (99.6%) and used significantly less image processing time (t = 91 ms) than the rest of the models. It was found that for SP_CNN, DI images outperformed the other three types of images. Although no significant difference was found in validation accuracy (99.91% ± 0.04%) among MobileNet, Xception, and Densnet121. The MobileNet required significantly less time to process single images (t = 210 ms). The Xception model with PCI_IR images showed the highest test accuracy (98.6%). The results also show that both hourly and daily activity levels (standing or sitting) were significantly higher on the day of the onset of estrus than the values on the previous days. Meanwhile, the daily idle level (lateral lying) decreased significantly from two days prior to the onset of estrus. The decrease in daily idle level was drastic on the day of onset of estrus (12.9 ± 7.1%) and before (15.0 ± 12.0%). The study also found that monitoring intervals should not exceed 15 min for accurately quantifying sows’ activity levels. The study concluded that posture patterns derived from the robotic imaging system could help detect sows’ estrus status.

(LC): LiDAR-Camera Loop Constraints for Cross-Modal Place Recognition

Localization has been a challenging task for autonomous navigation. A loop detection algorithm must overcome environmental changes for the place recognition and re-localization of robots. Therefore, deep learning has been extensively studied for the consistent transformation of measurements into localization descriptors. Street view images are easily accessible; however, images are vulnerable to appearance changes. LiDAR can robustly provide precise structural information. However, constructing a point cloud database is expensive, and point clouds exist only in limited places. Different from previous works that train networks to produce shared embedding directly between the 2D image and 3D point cloud, we transform both data into 2.5D depth images for matching. In this work, we propose a novel cross-matching method, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(LC)<inline-formula><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula></i> , for achieving LiDAR localization without a prior point cloud map. To this end, LiDAR measurements are expressed in the form of range images before matching them to reduce the modality discrepancy. Subsequently, the network is trained to extract localization descriptors from disparity and range images. Next, the best matches are employed as a loop factor in a pose graph. Using public datasets that include multiple sessions in significantly different lighting conditions, we demonstrated that LiDAR-based navigation systems could be optimized from image databases and vice versa.

Hyperspectral image-aided LiDAR point cloud labeling via spatio-spectral feature representation learning

Urban scene-level 3D point cloud labeling is a very laborious and expensive task compared to images. Conversely however, image processing techniques, deep learning or otherwise are more established and mature. Thus, in a multi-source data environment, the labeling of a point cloud scene via an automated image process as an initial step, followed by a manual human verification process is an effective way to save man hours and cost. With the above as the goal, this study presents a simple but robust spatio-spectral feature representation approach. In this approach, a class-aware band selection and reduction (CBSR) technique is developed for optimal hyperspectral feature representation. A double-branched convolutional Gaussian Bernoulli deep belief network (CGBDBN) is then used for hierarchical spatial feature extraction from LiDAR-derived data and the CBSR data. Using stacked ensemble learning, spatio-spectral features are generated from the two feature streams via a fusion rule and then classified — the results of which are used in labeling a raw 3D LiDAR point cloud through projection. To evaluate this study, extensive experiments were conducted on the IEEE 2018 Houston dataset — the only publicly available dataset with both hyperspectral image (HSI) and 3D point cloud covering the same area — for urban scene classification. The results indicate that the developed CBSR attained comparatively competitive results with state-of-the-art approaches, thus making it a robust spectral feature representation technique. Also, the weight-sharing property, probabilistic modeling, and hierarchical nature of CGBDBN gives our approach the ability to capture high-level contextual features. Furthermore, compared to the spatial- or spectral-only features, the generated spatio-spectral features are more discriminative and significantly aided in improving the proposed model’s efficacy. Overall, the proposed approach, based on the evaluation metrics, is a robust and effective approach for both coarse- and fine-grained raw LiDAR point cloud labeling tasks.

Individual Pig Identification Using Back Surface Point Clouds in 3D Vision.

The individual identification of pigs is the basis for precision livestock farming (PLF), which can provide prerequisites for personalized feeding, disease monitoring, growth condition monitoring and behavior identification. Pig face recognition has the problem that pig face samples are difficult to collect and images are easily affected by the environment and body dirt. Due to this problem, we proposed a method for individual pig identification using three-dimension (3D) point clouds of the pig's back surface. Firstly, a point cloud segmentation model based on the PointNet++ algorithm is established to segment the pig's back point clouds from the complex background and use it as the input for individual recognition. Then, an individual pig recognition model based on the improved PointNet++LGG algorithm was constructed by increasing the adaptive global sampling radius, deepening the network structure and increasing the number of features to extract higher-dimensional features for accurate recognition of different individuals with similar body sizes. In total, 10,574 3D point cloud images of ten pigs were collected to construct the dataset. The experimental results showed that the accuracy of the individual pig identification model based on the PointNet++LGG algorithm reached 95.26%, which was 2.18%, 16.76% and 17.19% higher compared with the PointNet model, PointNet++SSG model and MSG model, respectively. Individual pig identification based on 3D point clouds of the back surface is effective. This approach is easy to integrate with functions such as body condition assessment and behavior recognition, and is conducive to the development of precision livestock farming.

FusionPillars: A 3D Object Detection Network with Cross-Fusion and Self-Fusion

In the field of unmanned systems, cameras and LiDAR are important sensors that provide complementary information. However, the question of how to effectively fuse data from two different modalities has always been a great challenge. In this paper, inspired by the idea of deep fusion, we propose a one-stage end-to-end network named FusionPillars to fuse multisensor data (namely LiDAR point cloud and camera images). It includes three branches: a point-based branch, a voxel-based branch, and an image-based branch. We design two modules to enhance the voxel-wise features in the pseudo-image: the Set Abstraction Self (SAS) fusion module and the Pseudo View Cross (PVC) fusion module. For the data from a single sensor, by considering the relationship between the point-wise and voxel-wise features, the SAS fusion module self-fuses the point-based branch and the voxel-based branch to enhance the spatial information of the pseudo-image. For the data from two sensors, through the transformation of the images’ view, the PVC fusion module introduces the RGB information as auxiliary information and cross-fuses the pseudo-image and RGB image of different scales to supplement the color information of the pseudo-image. Experimental results revealed that, compared to existing current one-stage fusion networks, FusionPillars yield superior performance, with a considerable improvement in the detection precision for small objects.