Point Cloud Image Research Articles

The Importance of Ground Control Points In A Photogrammetric Workflow

Abstract This study investigates the optimal distribution and pattern of ground control points (GCPs) in aerial photogrammetric projects. Aerial triangulation (AT), also known as bundle adjustment, is the fundamental step in refining 3D reconstruction models and camera positions, thereby minimizing reprojection errors. The study utilizes data from a national project in Romania, employing high-resolution aerial images acquisition using photogrammetric sensors. The project has rigorous requirements of ground control points (GCP) placement and field measurements using GNSS and geometric leveling techniques. The study employs various scenarios, manipulating the number and distribution of GCPs, to assess their influence on planimetric and altimetric accuracy. Results indicate that the configuration and number of GCPs significantly affect the accuracy of photogrammetric products, such as dense image point clouds, digital surface models, and orthophotos. Moreover, the study underscores the importance of precise GCP determination methods, especially in regions lacking a precise gravimetric geoid model. In scenarios with inadequate GCP coverage the outcomes have inferior quality, emphasizing the critical role of GCPs in ensuring the quality of photogrammetric products. Overall, the research gives a clear view on the best placement patterns of GCPs and their influence on AT process evaluation performed in check points (CHKs).

A region feature fusion network for point cloud and image to detect 3D object

AbstractSensor fusion is very important for collaborative intelligent systems. A regional feature fusion network called ReFuNet for detecting 3D Object is proposed. It is difficult to detect distant or small objects accurately for the sparsity of LiDAR point cloud. The LiDAR point cloud and camera image information to solve the problem of point cloud sparsity is used, which can integrate image‐rich semantic information to enhance point cloud features. Also, the authors’ ReFuNet method segments the possible areas of objects by the results of 2D image detection. A cross‐attention mechanism adaptively fuses image and point cloud features within the areas. Then, the authors’ ReFuNet uses fused features to predict the 3D bounding boxes of objects. Experiments on the KITTI 3D object detection dataset showed that the authors’ proposed fusion method effectively improved the performance of 3D object detection.

An image segmentation and point cloud registration combined scheme for sensing of obscured tree branches

Automated robots are emerging as a solution for labor-intensive fruit orchard management. Three-dimensional (3D) reconstruction of tree branches is a fundamental requirement for robots to perform tasks like pruning and fruit harvesting. Current branch sensing methods often rely on planar segmentation with limited 3D information or computationally expensive point cloud segmentation, which may not be suitable for natural orchards with obscured tree branches. This study proposes a novel scheme that reconstructs occluded branches from RGB-D (Red-Green-Blue-Depth) images by integrating the point clouds converted from planar segmentation masks and depth images. The proposed approach extends the existing 2D branch sensing techniques to 3D, leveraging multi-view information. The deep learning models DeeplabV3+ and Pix2pix are employed to generate the segmentation masks, separately. And the Fast Global Registration (FGR) is used to register the multi-view point clouds. The results demonstrate that the output point clouds have at least a 24 % increase in the number of corresponding points after FGR. Furthermore, the time cost per hundred corresponding points is reduced by 85 % and 69 % when using the DeepLabV3 + and Pix2pix-based schemes, respectively, compared to the PointNet++ approach. These findings indicate that the proposed scheme significantly improves the sensing of occluded branches in terms of output richness and computational efficiency, making it applicable to natural orchard working spaces.

A subway tunnel image stitching method based on point cloud mapping relationships and high-resolution image

The paper presents an intelligent tunnel 3D laser image acquisition system. The system can quickly acquire the 3D point cloud and tunnel surface image of the whole tunnel section. The paper proposes the subway tunnel image stitching method based on point cloud mapping relationships and high-resolution image. Firstly, the method extracts the homonymous points from the mapped image generated from the 3D point cloud and the camera image based on the maximum information coefficient method. Secondly, the 3D point cloud positions corresponding to the homonymous points are determined. The optimization of camera parameters is achieved based on the gradient descent method. Then, the tunnel point cloud is expanded to generate a gray scale image based on spherical projection. Based on the optimized parameters, the point cloud is projected to the camera image to determine the depth value of each camera image pixel point. Finally, the pixel values of the camera image are projected based on the spherical projection to obtain the stitching result images. This method stitches individual images together to form a more complete tunnel surface image. The larger tunnel surface image helps in identifying tunnel defects.

Instance-level 6D pose estimation based on multi-task parameter sharing for robotic grasping

Six-dimensional pose estimation task predicts its 3D rotation matrix and 3D translation matrix in the world coordinate system by inputting the color image or depth image of the target object. Existing methods usually use deep neural networks to directly predict or regress object poses based on keypoint methods. The prediction results usually have deviations depending on whether the surface shape of the object is prominent or not and the size of the object. To solve this problem, we propose the six-dimensional pose estimation based on multi-task parameter sharing (PMP) framework to incorporate object category information into the pose estimation network through the form of an object classification auxiliary task. First, we extract the image features and point cloud features of the target object separately, and fuse them point by point; then, we share the confidence of each keypoint in pose estimation task and the knowledge of the classification task, get the key points with higher confidence, and predict the object pose; finally, the obtained object pose is passed through an iterative optimization network to obtain the final pose. The experimental results on the LineMOD dataset show that the proposed method can improve the accuracy of pose estimation and narrow the gap in the prediction accuracy of objects with different shapes. We also tested on a new dataset of small-scale objects, which contains object RGBD images and accurate 3D point cloud information. The proposed method is applied to the grasping experiment on the UR5 robotic arm, which satisfies the real-time pose estimation results during the grasping process.

Three-dimensional reconstruction of polarized ambient light separation in complex illumination

In current research, it is still a hot topic for 3D reconstruction under complex illumination. This paper uses a polarization camera combined with a coding technique to propose a new 3D reconstruction method for polarized ambient light separation. Based on the polarization camera, a specific separation model is established to analyze the relationship between the polarization characteristics of polarized and natural light. Specular reflections were filtered first and then analyzed based on the stocks vector and muller matrix. A specific calculation process was used to calculate different polarization azimuths according to the polarization characteristics, and finally, the polarized light and ambient light were separated. The experimental results show that the use of this polarization camera approach reduces the number of steps required to rotate the polarizer multiple times. This not only reduces the shooting time but also improves the efficiency. Moreover, after separating the ambient light, polarization imaging suppresses the interference of the ambient light, which helps to highlight the complete point cloud image more clearly in the 3D reconstruction. The standard deviation of 3D reconstruction was improved to 0.1675 mm by using this method in indoor and outdoor experiments.

Estimating the timber value of a forest property using geographically balanced samples and unoccupied aerial vehicle data

Abstract A common task in forestry is to determine the value of a forest property, and timber is the most valuable component of that property. Remotely sensed data collected by an unoccupied aerial vehicle (UAV) are suited for this purpose as most forest properties are of a size that permits the efficient collection of UAV data. These UAV data, when linked to a probability sample of field plots, enable the model-assisted (MA) estimation of the timber value and its associated uncertainty. Our objective was to estimate the value of timber (€/ha) in a 40-ha forest property in Finland. We used a systematic sample of field plots (n = 160) and 3D image point cloud data collected by an UAV. First, we studied the effects of spatial autocorrelation on the variance estimates associated with the timber value estimates produced using a field data-based simple expansion (EXP) estimator. The variance estimators compared were simple random sampling, Matérn, and a variant of the Grafström–Schelin estimator. Second, we compared the efficiencies of the EXP and MA estimators under different sampling intensities. The sampling intensity was varied by subsampling the systematic sample of 160 field plots. In the case of the EXP estimator, the simple random sampling variance estimator produced the largest variance estimates, whereas the Matérn estimator produced smaller variance estimates than the Grafström–Schelin estimator. The MA estimator was more efficient than the EXP estimator, which suggested that the reduction of sampling intensity from 160 to 60 plots is possible without deterioration in precision. The results suggest that the use of UAV data improves the precision of timber value estimates compared to the use of field data only. In practice, the proposed application improves the cost-efficiency of the design-based appraisal of a forest property because expensive field workload can be reduced by means of UAV data.

Enhancing angular resolution in single-photon LIDAR for daytime long-range 3D imaging

Single photon LIDAR exhibits exceptional detection sensitivity and finds extensive applications in long-distance measurement and three-dimensional imaging domains. However, it is susceptible to background noise interference, leading to a trade-off between scanning imaging efficiency and angular resolution, thereby limiting its overall performance. In this paper, we propose a novel ‘compression-resampling’ high-resolution imaging method for single-photon scanning LIDAR based on the spatial correlation of the target scene. This approach significantly enhances the system's angular resolution without compromising imaging efficiency. By employing this method, we successfully conducted 3D imaging over distances exceeding 26 km even under challenging conditions with high levels of background noise during daylight hours. Within a target field of view measuring 1° × 1°, we reconstructed a high-resolution 3D point cloud image comprising 1000 × 1410 pixels. Notably, this achievement yielded an impressive angular resolution of 17.4 urad while maintaining an acceptable imaging time of approximately 20 minutes. The results unequivocally demonstrate the effectiveness of our proposed method in improving the quality of three-dimensional imaging and enhancing angular resolution while simultaneously reducing imaging time requirements, thus expanding potential application scenarios for single-photon LIDAR technology.

Point2Real: Bridging the Gap between Point Cloud and Realistic Image for Open-World 3D Recognition

Recognition in open-world scenarios is an important and challenging field, where Vision-Language Pre-training paradigms have greatly impacted the 2D domain. This inspires a growing interest in introducing 2D pre-trained models, such as CLIP, into the 3D domain to enhance the ability of point cloud understanding. Considering the difference between discrete 3D point clouds and real-world 2D images, reducing the domain gap is crucial. Some recent works project point clouds onto a 2D plane to enable 3D zero-shot capabilities without training. However, this simplistic approach leads to an unclear or even distorted geometric structure, limiting the potential of 2D pre-trained models in 3D. To address the domain gap, we propose Point2Real, a training-free framework based on the realistic rendering technique to automate the transformation of the 3D point cloud domain into the Vision-Language domain. Specifically, Point2Real leverages a shape recovery module that devises an iterative ball-pivoting algorithm to convert point clouds into meshes, narrowing the gap in shape at first. To simulate photo-realistic images, a set of refined textures as candidates is applied for rendering, where the CLIP confidence is utilized to select the suitable one. Moreover, to tackle the viewpoint challenge, a heuristic multi-view adapter is implemented for feature aggregation, which exploits the depth surface as an effective indicator of view-specific discriminability for recognition. We conduct experiments on ModelNet10, ModelNet40, and ScanObjectNN datasets, and the results demonstrate that Point2Real outperforms other approaches in zero-shot and few-shot tasks by a large margin.

Frozen CLIP Transformer Is an Efficient Point Cloud Encoder

The pretrain-finetune paradigm has achieved great success in NLP and 2D image fields because of the high-quality representation ability and transferability of their pretrained models. However, pretraining such a strong model is difficult in the 3D point cloud field due to the limited amount of point cloud sequences. This paper introduces Efficient Point Cloud Learning (EPCL), an effective and efficient point cloud learner for directly training high-quality point cloud models with a frozen CLIP transformer. Our EPCL connects the 2D and 3D modalities by semantically aligning the image features and point cloud features without paired 2D-3D data. Specifically, the input point cloud is divided into a series of local patches, which are converted to token embeddings by the designed point cloud tokenizer. These token embeddings are concatenated with a task token and fed into the frozen CLIP transformer to learn point cloud representation. The intuition is that the proposed point cloud tokenizer projects the input point cloud into a unified token space that is similar to the 2D images. Comprehensive experiments on 3D detection, semantic segmentation, classification and few-shot learning demonstrate that the CLIP transformer can serve as an efficient point cloud encoder and our method achieves promising performance on both indoor and outdoor benchmarks. In particular, performance gains brought by our EPCL are 19.7 AP50 on ScanNet V2 detection, 4.4 mIoU on S3DIS segmentation and 1.2 mIoU on SemanticKITTI segmentation compared to contemporary pretrained models. Code is available at \url{https://github.com/XiaoshuiHuang/EPCL}.

Construction and forward kinematics of a novel reconfigurable parallel mechanism

In this paper, a kind of hybrid kinematics chain with variable constraints is constructed by using a 6R spatial bifurcated chain and 4-DOF serial limbs, by using them, a 3-DOF Reconfigurable Parallel Mechanisms (RPM) was constructed, which has four motion modes: 3T, 2T1R, 2R1T, and 3R motion modes. Using screw theory, the change of wrench system in the hybrid kinematics chain caused by the reconfiguration of the 6R bifurcated chain is analyzed, which affects the wrench system of the moving platform (MP), and thus causes the reconfiguration of the parallel mechanism (PM). The triangular pyramid fixed to the MP was found as auxiliary geometry and used to intuitively describe the constraint relationship between joint axes in limbs. The latter was used to establish a unified forward kinematics model of the RPM in four motion modes. The general forward kinematics solution procedure for this type of RPM is given. The overall Jacobian matrix was obtained by screw theory for singularity analysis. The condition number of the Jacobian matrix measures the degree to which the mechanism is close to the singular configuration. Based on the Monte Carlo simulation, point cloud images representing the workspace of RPM were obtained.

FFLDGA-Net: Image retrieval method based on Feature Fusion Learnable Descriptor Graph Attention Network

Image retrieval aims to retrieve and return the image in the database that is most similar to the query image. However, the performance of image retrieval models is often hindered by the limited dimensionality of images, which lacks depth information about objects. To address this issue, we propose a novel image retrieval model called FFLDGA-Net (Feature Fusion-based Learnable Descriptor Graph Attention Network). This model aims to overcome the absence of depth information in images by fusing feature information from both image data and point cloud data. First, we introduce the LDGA-Net, which effectively improved the model’s ability to mine hard samples and negative samples. Then, we combine the multi-scale route convolution module with a one-dimensional path aggregation network to fuse point clouds and image features at multiple scales, and establish a high-dimensional relationship between features and low-dimensional features. To mitigate training noise, we incorporate a soft label strategy tailored to the dataset’s characteristics. Our experimental results on two benchmark datasets demonstrate the significant improvements achieved by FFLDGA-Net in image retrieval performance.

A Coarse-to-Fine Transformer-Based Network for 3D Reconstruction from Non-Overlapping Multi-View Images

Reconstructing 3D structures from non-overlapping multi-view images is a crucial task in the field of 3D computer vision, since it is difficult to establish feature correspondences and infer depth from overlapping parts of views. Previous methods, whether generating the surface mesh or volume of an object, face challenges in simultaneously ensuring the accuracy of detailed topology and the integrity of the overall structure. In this paper, we introduce a novel coarse-to-fine Transformer-based reconstruction network to generate precise point clouds from multiple input images at sparse and non-overlapping viewpoints. Specifically, we firstly employ a general point cloud generation architecture enhanced by the concept of adaptive centroid constraint for the coarse point cloud corresponding to the object. Subsequently, a Transformer-based refinement module applies deformation to each point. We design an attention-based encoder to encode both image projection features and point cloud geometric features, along with a decoder to calculate deformation residuals. Experiments on ShapeNet demonstrate that our proposed method outperforms other competing methods.

Intelligent extraction of road cracks based on vehicle laser point cloud and panoramic sequence images

In light of the limited efficacy of conventional methods for identifying pavement cracks and the absence of comprehensive depth and location data in two-dimensional photographs, this study presents an intelligent strategy for extracting road cracks. This methodology involves the integration of laser point cloud data obtained from a vehicle-mounted system and a panoramic sequence of images. The study employs a vehicle-mounted LiDAR measurement system to acquire laser point cloud and panoramic sequence image data simultaneously. A convolutional neural network is utilized to extract cracks from the panoramic sequence image. The extracted sequence image is then aligned with the laser point cloud, enabling the assignment of RGB information to the vehicle-mounted three dimensional (3D) point cloud and location information to the two dimensional (2D) panoramic image. Additionally, a threshold value is set based on the crack elevation change to extract the aligned roadway point cloud. The three-dimensional data pertaining to the cracks can be acquired. The experimental findings demonstrate that the use of convolutional neural networks has yielded noteworthy outcomes in the extraction of road cracks. The utilization of point cloud and image alignment techniques enables the extraction of precise location data pertaining to road cracks. This approach exhibits superior accuracy when compared to conventional methods. Moreover, it facilitates rapid and accurate identification and localization of road cracks, thereby playing a crucial role in ensuring road maintenance and traffic safety. Consequently, this technique finds extensive application in the domains of intelligent transportation and urbanization development. The technology exhibits significant promise for use in the domains of intelligent transportation and city development.

Fusion of airborne multimodal point clouds for vegetation parameter correction extraction in burned areas.

Most experimental studies use unimodal data for processing, the RGB image point cloud cannot separate the shrub and tree layers according to the visible vegetation index, and the airborne laser point cloud is difficult to distinguish between the ground and grass ranges, to address the above problems, a multi-band information image fusing the LiDAR point cloud and the RGB image point cloud is constructed. In this study, data collected from UAV platforms, including RGB image point clouds and laser point clouds, were used to construct a fine canopy height model (using laser point cloud data) and high-definition digital orthophotos (using image point cloud data), and the orthophotos were fused with a canopy height model (CHM) by selecting the Difference Enhancement Vegetation Index (DEVI) and Normalised Green-Blue Discrepancy Index (NGBDI) after comparing the accuracy of different indices. Morphological reconstruction of CHM + DEVI/NGBDI fusion image, remove unreasonable values; construct training samples, using classification regression tree algorithm, segmentation of the range of the burned areas and adaptive extraction of vegetation as trees, shrubs and grasslands, tree areas as foreground markers using the local maximum algorithm to detect the tree apexes, the non-tree areas are assigned to be the background markers, and the Watershed Transform is performed to obtain the segmentation contour; the original laser point cloud is divided into chunks according to the segmented single-tree contour, and the highest point is traversed to search for the highest point, and corrected for the height of the single-tree elevations one by one. Accuracy analysis of the vegetation information extracted by the method with the measured data showed that the improved method increased the overall recall rate by 4.1%, the overall precision rate by 3.7%, the overall accuracy F1 score by 3.9%, and the tree height accuracy by 8.8%, 1.4%, 1.7%, 6.4%, 1.8%, and 0.3%, respectively, in the six sampling plots. The effectiveness of the improved method is verified, while the higher the degree of vegetation mixing in the region the better the extraction effect of the improved algorithm.

Room temperature operation of germanium-silicon single-photon avalanche diode.

The ability to detect single photons has led to the advancement of numerous research fields1-11. Although various types of single-photon detector have been developed12, because of two main factors-that is, (1) the need for operating at cryogenic temperature13,14 and (2) the incompatibility with complementary metal-oxide-semiconductor (CMOS)fabrication processes15,16-so far, to our knowledge, only Si-based single-photon avalanche diode (SPAD)17,18 has gained mainstream success and has been used in consumer electronics. With the growing demand to shift the operation wavelength from near-infrared to short-wavelength infrared (SWIR)for better safety and performance19-21, an alternative solution is required because Si has negligible optical absorption for wavelengths beyond 1 µm. Here we report a CMOS-compatible, high-performing germanium-silicon SPAD operated at room temperature, featuring a noise-equivalent power improvement over the previous Ge-based SPADs22-28 by 2-3.5 orders of magnitude. Key parameters such as dark count rate, single-photon detection probability at 1,310 nm, timing jitter, after-pulsing characteristic time and after-pulsing probability are, respectively, measured as 19 kHz µm-2, 12%, 188 ps, ~90 ns and <1%, with a low breakdown voltage of 10.26 V and a small excess bias of 0.75 V. Three-dimensional point-cloud images are captured with direct time-of-flight technique as proof of concept. This work paves the way towards using single-photon-sensitive SWIR sensors, imagers and photonic integrated circuits in everyday life.

COMBINING IMAGE AND POINT CLOUD SEGMENTATION TO IMPROVE HERITAGE UNDERSTANDING

Abstract. Current 2D and 3D semantic segmentation frameworks are developed and trained on specific benchmark datasets, often rich of synthetic data, and when they are applied to complex and real-world heritage scenarios they offer much lower accuracy than expected. In this work, we present and demonstrate an early and late fusion of methods for semantic segmentation in cultural heritage applications. We rely on image datasets, point clouds and BIM models. The early fusion utilizes multi-view rendering to generate RGBD imagery of the scene. In contrast, the late fusion approach merges image-based segmentation with a Point Transformer applied to point clouds. Two scenarios are considered and inference results show that predictions are primarily influenced by whether the scene has a predominantly geometric or texture-based signature, underscoring the necessity of fusion methods.

Design and Implementation of Nursing-Secure-Care System with mmWave Radar by YOLO-v4 Computing Methods

In computer vision and image processing, the shift from traditional cameras to emerging sensing tools, such as gesture recognition and object detection, addresses privacy concerns. This study navigates the Integrated Sensing and Communication (ISAC) era, using millimeter-wave signals as radar via a Convolutional Neural Network (CNN) model for event sensing. Our focus is on leveraging deep learning to detect security-critical gestures, converting millimeter-wave parameters into point cloud images, and enhancing recognition accuracy. CNNs present complexity challenges in deep learning. To address this, we developed flexible quantization methods, simplifying You Only Look Once (YOLO)-v4 operations with an 8-bit fixed-point number representation. Cross-simulation validation showed that CPU-based quantization improves speed by 300% with minimal accuracy loss, even doubling the YOLO-tiny model’s speed in a GPU environment. We established a Raspberry Pi 4-based system, combining simplified deep learning with Message Queuing Telemetry Transport (MQTT) Internet of Things (IoT) technology for nursing care. Our quantification method significantly boosted identification speed by nearly 2.9 times, enabling millimeter-wave sensing in embedded systems. Additionally, we implemented hardware-based quantization, directly quantifying data from images or weight files, leading to circuit synthesis and chip design. This work integrates AI with mmWave sensors in the domain of nursing security and hardware implementation to enhance recognition accuracy and computational efficiency. Employing millimeter-wave radar in medical institutions or homes offers a strong solution to privacy concerns compared to conventional cameras that capture and analyze the appearance of patients or residents.

Fall Detection System Based on Point Cloud Enhancement Model for 24 GHz FMCW Radar.

Automatic fall detection plays a significant role in monitoring the health of senior citizens. In particular, millimeter-wave radar sensors are relevant for human pose recognition in an indoor environment due to their advantages of privacy protection, low hardware cost, and wide range of working conditions. However, low-quality point clouds from 4D radar diminish the reliability of fall detection. To improve the detection accuracy, conventional methods utilize more costly hardware. In this study, we propose a model that can provide high-quality three-dimensional point cloud images of the human body at a low cost. To improve the accuracy and effectiveness of fall detection, a system that extracts distribution features through small radar antenna arrays is developed. The proposed system achieved 99.1% and 98.9% accuracy on test datasets pertaining to new subjects and new environments, respectively.

Enhancement of low-cost UAV-based photogrammetric point cloud using MMS point cloud and oblique images for 3D urban reconstruction

The accurate and dense reconstruction of high-quality 3D spatial information is essential for digital twin-based smart cities. Unmanned aerial vehicle (UAV)-based photogrammetry allows the 3D modeling of urban environments in the shortest possible time. Accurate georeferencing is a prerequisite for utilizing geospatial information. The geolocation accuracy of real-time kinematic (RTK)/post-processing kinematic (PPK) UAV-based photogrammetry is significantly high; however, RTK/PPK UAVs are costly. In contrast, the geolocation accuracy of low-cost UAV-based photogrammetry is generally low. It can be improved by indirect georeferencing using ground control points (GCPs) obtained in situ; however, this requires significant amounts of human resources and time. Therefore, this study analyzes the suitability of utilizing the mobile mapping system (MMS) point cloud as GCPs for low-cost UAV-based photogrammetry. We checked the significance of the vertical distribution of GCPs on the geolocation accuracy of low-cost UAV-based photogrammetry using the feature points of building facades (captured via vehicle-based MMS along roads) as GCPs. In addition, typical UAV-based photogrammetry only uses nadir images, which limits the detailed 3D reconstruction of building facades. In this study, a detailed reconstruction and an improved vertical geolocation accuracy were achieved using oblique images. Experiments demonstrated that the geolocation accuracy of the low-cost UAV-based photogrammetric point cloud improved to within 16 cm in the X-, Y-, and Z-directions. It was at its highest when the GCPs were diversely distributed in the vertical direction. Finally, we generated an enhanced point cloud by merging the low-cost UAV-based photogrammetric point cloud and the MMS point cloud.