Point Cloud Image Research Articles

Perception Methods for Adverse Weather Based on Vehicle Infrastructure Cooperation System: A Review.

Environment perception plays a crucial role in autonomous driving technology. However, various factors such as adverse weather conditions and limitations in sensing equipment contribute to low perception accuracy and a restricted field of view. As a result, intelligent connected vehicles (ICVs) are currently only capable of achieving autonomous driving in specific scenarios. This paper conducts an analysis of the current studies on image or point cloud processing and cooperative perception, and summarizes three key aspects: data pre-processing methods, multi-sensor data fusion methods, and vehicle-infrastructure cooperative perception methods. Data pre-processing methods summarize the processing of point cloud data and image data in snow, rain and fog. Multi-sensor data fusion methods analyze the studies on image fusion, point cloud fusion and image-point cloud fusion. Because communication channel resources are limited, the vehicle-infrastructure cooperative perception methods discuss the fusion and sharing strategies for cooperative perception information to expand the range of perception for ICVs and achieve an optimal distribution of perception information. Finally, according to the analysis of the existing studies, the paper proposes future research directions for cooperative perception in adverse weather conditions.

XLPE cable joint defects measurement method based on point cloud remapping

For the problems of low detection accuracy caused by weak texture and strong reflection on the surface of Cross-linked polyethylene (XLPE) high-voltage cable joint, this paper proposed a joint point cloud remapping and image segmentation method for the cable joint defects measurement, which improved the defects measurement accuracy from the correlation of the disordered point cloud and the salience of defects. Firstly, the discrete noise in the joint point cloud is removed using a modified radius filtering algorithm. Secondly, for the problems of disorderliness and weak association of the cable joint point cloud, we utilized Rodrigues' rotation formula (RRF) to correct the position of the XLPE cable joint point cloud. Then, for the demand of improving feature expression of non-significant defects, a point cloud remapping method is proposed, which remapping insignificant defects point cloud into saliency image for more effective detection. Lastly, an image segmentation method based on 8-neighborhood growth is applied for explicit defects detection, and the detected defects are quantified by the proposed measurement approach. Extensive experiments are conducted on the collected cable joints' point cloud and simulated cable joints' point cloud. With the help of the proposed method, we can achieve the best effects in DE, FR quantitative comparison, and the measurement inaccuracies are less than 4.5%. Experimental results demonstrate that the proposed method enjoys state-of-the-art performance in cable joint defects measurement.

Instance Segmentation Frustum–PointPillars: A Lightweight Fusion Algorithm for Camera–LiDAR Perception in Autonomous Driving

The fusion of camera and LiDAR perception has become a research focal point in the autonomous driving field. Existing image–point cloud fusion algorithms are overly complex, and processing large amounts of 3D LiDAR point cloud data requires high computational power, which poses challenges for practical applications. To overcome the above problems, herein, we propose an Instance Segmentation Frustum (ISF)–PointPillars method. Within the framework of our method, input data are derived from both a camera and LiDAR. RGB images are processed using an enhanced 2D object detection network based on YOLOv8, thereby yielding rectangular bounding boxes and edge contours of the objects present within the scenes. Subsequently, the rectangular boxes are extended into 3D space as frustums, and the 3D points located outside them are removed. Afterward, the 2D edge contours are also extended to frustums to filter the remaining points from the preceding stage. Finally, the retained points are sent to our improved 3D object detection network based on PointPillars, and this network infers crucial information, such as object category, scale, and spatial position. In pursuit of a lightweight model, we incorporate attention modules into the 2D detector, thereby refining the focus on essential features, minimizing redundant computations, and enhancing model accuracy and efficiency. Moreover, the point filtering algorithm substantially diminishes the volume of point cloud data while concurrently reducing their dimensionality, thereby ultimately achieving lightweight 3D data. Through comparative experiments on the KITTI dataset, our method outperforms traditional approaches, achieving an average precision (AP) of 88.94% and bird’s-eye view (BEV) accuracy of 90.89% in car detection.

A robot-system for picking parts from unstructured bins – A practical Approach

A bin picking task involves multiple steps, including object detection, 3D pose estimation, and path planning. Dealing with unstructured and cluttered bins presents challenges due to potential overlaps and misidentification of parts. We propose a purely mathematical approach for achieving highly precise 3D pose estimation at the millimeter level. Our method involves capturing a point-cloud image of the parts within the bin and applying a best fit algorithm to smooth the surface. Subsequently, we employ the Iterative Closest Point Method (ICP) to match a CAD model of the desired part with the smoothed surface point-cloud. Finally, we identify gripping points to enable the robot to pick up the part. The concept is validated in a use-case involving Lego sorting. It is compared with an industrial tool for performance evaluation purposes.

High-Resolution Point-Cloud Imaging With Doppler Division MIMO Radar Based on the 2-D Hybrid Sparse Array

High-Resolution Point-Cloud Imaging With Doppler Division MIMO Radar Based on the 2-D Hybrid Sparse Array

Laser Point Cloud-Image Fusion Technology for Intelligent Driving Vehicles Based on Semi-Supervised Learning Algorithm

Laser Point Cloud-Image Fusion Technology for Intelligent Driving Vehicles Based on Semi-Supervised Learning Algorithm

Adaptive learning point cloud and image diversity feature fusion network for 3D object detection

3D object detection is a critical task in the fields of virtual reality and autonomous driving. Given that each sensor has its own strengths and limitations, multi-sensor-based 3D object detection has gained popularity. However, most existing methods extract high-level image semantic features and fuse them with point cloud features, focusing solely on consistent information from both sensors while ignoring their complementary information. In this paper, we present a novel two-stage multi-sensor deep neural network, called the adaptive learning point cloud and image diversity feature fusion network (APIDFF-Net), for 3D object detection. Our approach employs the fine-grained image information to complement the point cloud information by combining low-level image features with high-level point cloud features. Specifically, we design a shallow image feature extraction module to learn fine-grained information from images, instead of relying on deep layer features with coarse-grained information. Furthermore, we design a diversity feature fusion (DFF) module that transforms low-level image features into point-wise image features and explores their complementary features through an attention mechanism, ensuring an effective combination of fine-grained image features and point cloud features. Experiments on the KITTI benchmark show that the proposed method outperforms state-of-the-art methods.

SEMANTIC ENRICHMENT OF 3D POINT CLOUDS USING 2D IMAGE SEGMENTATION

Abstract. 3D point cloud segmentation is computationally intensive due to the lack of inherent structural information and the unstructured nature of the point cloud data, which hinders the identification and connection of neighboring points. Understanding the structure of the point cloud data plays a crucial role in obtaining a meaningful and accurate representation of the underlying 3D environment. In this paper, we propose an algorithm that builds on existing state-of-the-art techniques of 2D image segmentation and point cloud registration to enrich point clouds with semantic information. DeepLab2 with ResNet50 as backbone architecture trained on the COCO dataset is used for indoor scene semantic segmentation into several classes like wall, floor, ceiling, doors, and windows. Semantic information from 2D images is propagated along with other input data, i.e., RGB images, depth images, and sensor information to generate 3D point clouds with semantic information. Iterative Closest Point (ICP) algorithm is used for the pair-wise registration of consecutive point clouds and finally, optimization is applied using the pose graph optimization on the whole set of point clouds to generate the combined point cloud of the whole scene. 3D point cloud of the whole scene contains pseudo-color information which denotes the semantic class to which each point belongs. The proposed methodology use an off-the-shelf 2D semantic segmentation deep learning model to semantically segment 3D point clouds collected using handheld mobile LiDAR sensor. We demonstrate a comparison of the accuracy achieved compared to a manually segmented point cloud on an in-house dataset as well as a 2D3DS benchmark dataset.

REAL-TIME UAV 3D IMAGE POINT CLOUDS MAPPING

Abstract. This paper proposes a real-time 3D image point clouds mapping algorithm for UAVs that is capable of mapping effectively in weak GNSS environments. And a UAV mapping system is integrated with a RGB camera, an inertial measurement unit (IMU), a GNSS receiver, data transmission devices, and a DJI M300 flight platform. To achieve real-time and robust mapping, the system utilizes a visual-inertial odometry (VIO) that tightly couples GNSS, RGB image, and IMU, which provides stable state estimation information for mapping. Subsequently, a dense matching algorithm based on key frames is adopted to recover 3D mapping information with low-computational cost. Extensive experiments are conducted on our test site, demonstrating the system's ability to build maps stably, even under the effect of wind. The results compared with the trajectory reconstructed by Pix4D show that the system achieves competitive accuracy of pose estimation and is capable of real-time mapping.

Trust your Good Friends: Source-free Domain Adaptation by Reciprocal Neighborhood Clustering.

Domain adaptation (DA) aims to alleviate the domain shift between source domain and target domain. Most DA methods require access to the source data, but often that is not possible (e.g. due to data privacy or intellectual property). In this paper, we address the challenging source-free domain adaptation (SFDA) problem, where the source pretrained model is adapted to the target domain in the absence of source data. Our method is based on the observation that target data, which might not align with the source domain classifier, still forms clear clusters. We capture this intrinsic structure by defining local affinity of the target data, and encourage label consistency among data with high local affinity. We observe that higher affinity should be assigned to reciprocal neighbors. To aggregate information with more context, we consider expanded neighborhoods with small affinity values. Furthermore, we consider the density around each target sample, which can alleviate the negative impact of potential outliers. In the experimental results we verify that the inherent structure of the target features is an important source of information for domain adaptation. We demonstrate that this local structure can be efficiently captured by considering the local neighbors, the reciprocal neighbors, and the expanded neighborhood. Finally, we achieve state-of-the-art performance on several 2D image and 3D point cloud recognition datasets.

End-effector path tracking of a 14 DOF Rover Manipulator System in CG-Space framework

AbstractIn order to resolve redundancy and path planning of a high DOF mobile manipulator using conventional approaches like Jacobian and a pseudoinverse method, researchers face the limitation of computational load and delay in response. If such kind of mobile manipulator is traversing the rough terrain, then conventional methods become too costly to implement due to the handling of redundant joints, obstacles, and wheel-terrain interaction. A few optimization-based redundancy resolution approaches try incorporating wheel-terrain interaction but fail in real-time response. This paper describes a 14 DOF Rover Manipulator System’s end-effector path-tracking approach using CG-Space framework to incorporate wheel-terrain interaction. CG-Space means the center of gravity (CG) locus of the Rover. The Rover’s CG is calculated while traversing over 3D terrain using a multivariable optimization method and a 3D point cloud image of the actual terrain and stored as CG-Space over the given terrain. First of all, we decide which part of the system moves to track the path, that is, arm or Rover, depending upon the manipulator’s work volume and manipulability measure restrictions. The next task is to obtain the Rover pose according to the end-effector path using a simple arm’s inverse kinematic solution between the CG-Space and end-effector task space without resolving redundancy. Meanwhile, obstacles and non-traversable regions are avoided in CG-Space. On diverse 3D terrains, simulation and experimental results support the suggested CG-Space framework for end-effector tracking.

High-precision surface profilometry on a micron-groove based on dual-comb electronically controlled optical sampling.

We demonstrate an optical method for 3D profilometry of micro-nano devices with large step structures. The measurement principle is based on a dual-comb direct time-of-flight detection. An electronically controlled optical sampling (ECOPS) approach is used to improve the acquisition rate. In a proof-of-principle distance measurement experiment, the measurement precision reaches 15nm at 4000-times averages. The method has been used to characterize the profile of a large aspect-ratio rectangular micron-groove with 10µm width and 62.3µm depth. By point-by-point scanning, a 3D point cloud image is obtained, and the 3D profile of the micro-structure is quantitatively reconstructed with sub-micrometer precision. The proposed high-precision, high-speed surface 3D profile measurement technology could be applied to profilometry and inspection of complex microelectronics devices in the future.

Three-Dimensional Reconstruction of Railway Bridges Based on Unmanned Aerial Vehicle–Terrestrial Laser Scanner Point Cloud Fusion

To address the incomplete image data collection of close-to-ground structures, such as bridge piers and local features like the suspension cables in bridges, obtained from single unmanned aerial vehicle (UAV) oblique photography and the difficulty in acquiring point cloud data for the top structures of bridges using single terrestrial laser scanners (TLSs), as well as the lack of textural information in TLS point clouds, this study aims to establish a high-precision, complete, and realistic bridge model by integrating UAV image data and TLS point cloud data. Using a particular large-scale dual-track bridge as a case study, the methodology involves aerial surveys using a DJI Phantom 4 RTK for comprehensive image capture. We obtain 564 images circling the bridge arches, 508 images for orthorectification, and 491 images of close-range side views. Subsequently, all images, POS data, and ground control point information are imported into Context Capture 2023 software for aerial triangulation and multi-view image dense matching to generate dense point clouds of the bridge. Additionally, ground LiDAR scanning, involving the placement of six scanning stations both on and beneath the bridge, was conducted and the point cloud data from each station are registered in Trimble Business Center 5.5.2 software based on identical feature points. Noise point clouds are then removed using statistical filtering techniques. The integration of UAV image point clouds with TLS point clouds is achieved using the iterative closest point (ICP) algorithm, followed by the creation of a TIN model and texture mapping using Context Capture 2023 software. The effectiveness of the integrated modeling is verified by comparing the geometric accuracy and completeness of the images with those obtained from a single UAV image-based model. The integrated model is used to generate cross-sectional profiles of the dual-track bridge, with detailed annotations of boundary dimensions. Structural inspections reveal honeycomb surfaces and seepage in the bridge piers, as well as painted rust and cracks in the arch ribs. The geometric accuracy of the integrated model in the X, Y, and Z directions is 1.2 cm, 0.8 cm, and 0.9 cm, respectively, while the overall 3D model accuracy is 1.70 cm. This method provides technical reference for the reconstruction of three-dimensional point cloud bridge models. Through 3D reconstruction, railway operators can better monitor and assess the condition of bridge structures, promptly identifying potential defects and damages, thus enabling the adoption of necessary maintenance and repair measures to ensure the structural safety of the bridges.

RETRACTED ARTICLE: Unmanned Vehicle Fusion Positioning Technology Based on “5G + Beidou” and 3D Point Cloud Image

Unmanned vehicles need to know their location and direction information accurately to plan and navigate their paths. However, the positioning system is susceptible to interference from a variety of factors, which leads to increased positioning errors, thereby affecting the accuracy of unmanned vehicle positioning. An unmanned vehicle fusion positioning technology based on the "5G + Beidou" integrated positioning system was proposed. While using the "5G + Beidou" base station for positioning, the 3D point cloud image was fused, and the high-precision real-time positioning was carried out through the vehicle's autonomous navigation algorithm. This paper first analyzed the current situation and characteristics of GNSS technology and studied the key technologies and principles of the "5G + Beidou" integrated positioning system. Then, aiming at the difficulty of 5G base station deployment, the GNSS system parameter optimization scheme based on a multidimensional fusion structure was designed. Finally, in the experiment, it was verified that the fusion system could achieve higher precision positioning results compared with traditional single-dimensional GNSS and multi-dimensional GNSS. The technical advantages of "5G + Beidou" were used for data fusion processing of unmanned vehicles, and a positioning method based on the combination of 3D point cloud image and high-precision map was proposed. Through some experiments, it was concluded that the fusion location method could control the error below 0.1, which showed the accuracy of the fusion location.

Light efficacy estimation for fruit trees based on LIDAR point clouds: A case study on pear trees

It is important to evaluate the effect of fruit-tree pruning based on canopy-structure characteristics so as to keep appropriate illumination for tree growth. However, current pruning and evaluation mainly rely on manual experience due to lacking a convenient and automatic approach to obtain such features. This paper proposes a method for estimating light efficacy based on layered canopy point-clouds, where leaf-area was selected as the canopy characteristic. LIDAR was utilised to scan pear trees in three dimensions to acquire initial point clouds, and preprocess was conducted to obtain the point-cloud images of different canopy layers. Then, an optimised algorithm based on Freeman Chain was developed to accurately draw the outlines of point-cloud clusters in the images, and pixel proportion was used to calculate the real leaf-area of each layer. Finally, based on Deep Neural Network, an estimation model was established between different layer leaf-areas and their average illumination acquired in tests. The proposed method was applied to estimate the illumination of a random pear tree to evaluate the variation of light intensity. The results showed that: 1) the average relative error of leaf-area calculation of the optimised Freeman Chain method was about 6.74 %, although the running duration was longer than the conventional one, 2) the overall correlation coefficient, R, of the model was more than 0.95, while that of validation and test sets were more than 0.94, and 3) the illumination estimation was accurate and precise with an average relative error of 1.42 % and a standard deviation of relative errors of 1.19. The validation indicated that this method could achieve the prediction of illumination based on different leaf-area characteristics. The study is expected to give a technical solution for the illumination evaluation before and after tree pruning.

Detection and characterization of spike architecture based on deep learning and X-ray computed tomography in barley

BackgroundSpike is the grain-bearing organ in cereal crops, which is a key proxy indicator determining the grain yield and quality. Machine learning methods for image analysis of spike-related phenotypic traits not only hold the promise for high-throughput estimating grain production and quality, but also lay the foundation for better dissection of the genetic basis for spike development. Barley (Hordeum vulgare L.) is one of the most important crops globally, ranking as the fourth largest cereal crop in terms of cultivated area and total yield. However, image analysis of spike-related traits in barley, especially based on CT-scanning, remains elusive at present.ResultsIn this study, we developed a non-invasive, high-throughput approach to quantitatively measuring the multitude of spike architectural traits in barley through combining X-ray computed tomography (CT) and a deep learning model (UNet). Firstly, the spikes of 11 barley accessions, including 2 wild barley, 3 landraces and 6 cultivars were used for X-ray CT scanning to obtain the tomographic images. And then, an optimized 3D image processing method was used to point cloud data to generate the 3D point cloud images of spike, namely ‘virtual’ spike, which is then used to investigate internal structures and morphological traits of barley spikes. Furthermore, the virtual spike-related traits, such as spike length, grain number per spike, grain volume, grain surface area, grain length and grain width as well as grain thickness were efficiently and non-destructively quantified. The virtual values of these traits were highly consistent with the actual value using manual measurement, demonstrating the accuracy and reliability of the developed model. The reconstruction process took 15 min approximately, 10 min for CT scanning and 5 min for imaging and features extraction, respectively.ConclusionsThis study provides an efficient, non-invasive and useful tool for dissecting barley spike architecture, which will contribute to high-throughput phenotyping and breeding for high yield in barley and other crops.

UAMD-Net: A Unified Adaptive Multimodal Neural Network for Dense Depth Completion

Depth estimation is a critical problem in robotics applications especially autonomous driving. Currently, depth prediction based on binocular stereo matching and depth completion based on fusion of monocular image and laser point cloud are two mainstream methods. However, the former usually suffers from lack of constraint while building cost volume, and the latter could not be trained in self-supervised way and haven’t utilized the geometric constraint of stereo matching, which we think will further improve the performance. Therefore, we propose a novel multimodal neural network, namely UAMD-Net, for dense depth completion based on fusion of binocular stereo matching and the weak constraint from the sparse point clouds. Specifically, the sparse point clouds are converted to sparse depth map and filled to the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multimodal feature encoder</i> (MFE) with binocular image, constructing a cross-modal cost volume. Then, it will be further processed by the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multimodal feature aggregator</i> (MFA) and the depth regression layer. Furthermore, since previous multimodal depth estimation methods ignore the problem of modality dependence, we propose a new training strategy called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">random modality dropout</i> (RMD) which enables the network to be adaptively trained with multiple modality inputs and inference with specific modality inputs. Benefiting from the flexible network structure and adaptive training method, our proposed network can realize unified training under various modality input conditions. Comprehensive experiments conducted on KITTI and DrivingStereo depth completion datasets demonstrate that our method produces robust results and outperforms other state-of-the-art methods.

Transferring Vision-Language Models for Visual Recognition: A Classifier Perspective

Transferring knowledge from pre-trained deep models for downstream tasks, particularly with limited labeled samples, is a fundamental problem in computer vision research. Recent advances in large-scale, task-agnostic vision-language pre-trained models, which are learned with billions of samples, have shed new light on this problem. In this study, we investigate how to efficiently transfer aligned visual and textual knowledge for downstream visual recognition tasks. We first revisit the role of the linear classifier in the vanilla transfer learning framework, and then propose a new paradigm where the parameters of the classifier are initialized with semantic targets from the textual encoder and remain fixed during optimization. To provide a comparison, we also initialize the classifier with knowledge from various resources. In the empirical study, we demonstrate that our paradigm improves the performance and training speed of transfer learning tasks. With only minor modifications, our approach proves effective across 17 visual datasets that span three different data domains: image, video, and 3D point cloud.

Cooperative Perception With V2V Communication for Autonomous Vehicles

Occlusion is a critical problem in the Autonomous Driving System. Solving this problem requires robust collaboration among autonomous vehicles traveling on the same roads. However, transferring the entirety of raw sensors' data among autonomous vehicles is expensive and can cause a delay in communication. This paper proposes a method called Realtime Collaborative Vehicular Communication based on Bird's-Eye-View (BEV) map. The BEV map holds the accurate depth information from the point cloud image while its 2D representation enables the method to use a novel and well-trained image-based backbone network. Most importantly, we encode the object detection results into the BEV representation to reduce the volume of data transmission and make real-time collaboration between autonomous vehicles possible. The output of this process, the BEV map, can also be used as direct input to most route planning modules. Numerical results show that this novel method can increase the accuracy of object detection by cross-verifying the results from multiple points of view. Thus, in the process, this new method also reduces the object detection challenges that stem from occlusion and partial occlusion. Additionally, different from many existing methods, this new method significantly reduces the data needed for transfer between vehicles, achieving a speed of 21.92 Hz for both the object detection process and the data transmission process, which is sufficiently fast for a real-time system.

Development of 3‐D point cloud imaging and 2‐D image processing methods for high‐throughput phenotyping of perennial ryegrass spikes

AbstractAdvances in technology have increased the adoption of high‐throughput phenotyping (HTP) methodologies, potentially replacing laborious and time‐consuming measurements and data recording. One promising HTP tool for fine‐featured and small‐sized characteristics is the three‐dimensional (3‐D) scanning and imaging system, but the utility of present two‐dimensional (2‐D) technology has not been fully explored for this purpose. The objective of this work was to reduce assessment and field evaluation time by developing 2‐D photogrammetric and 3‐D point cloud imaging methods for HTP of spike characteristics in perennial ryegrass (Lolium perenne L.), with special attention to traits that might be associated with seed retention. These HTP imaging systems were compared with direct data capture by hand on spikes of 21 diverse global accessions of perennial ryegrass. The Fiji (ImageJ) open‐source imaging software was used for photogrammetric analysis of spike structure, including spike length, spikelet number, internode length, and 2‐D curvature of the spike. An original approach to nondestructive HTP for crop characterization was developed using the commercially available Artec Space Spider. This 3‐D optical sensor was used to generate dense 3‐D point clouds to measure spike length, spikelet number, internode length, spikelet length, spikelet angle, and 3‐D curvature of the spike. Both methods were found to accurately characterize the subject; the 3‐D method was slower than 2‐D but was more precise (p ≤ 0.01) than 2‐D image analysis with a linear measurement deviation of only 0.17%. Fiji was effectively used for post‐processing image analysis, and the Space Spider can be used directly in the field to support HTP data collection. This non‐destructive field measurement system facilitates HTP in perennial ryegrass spikes and likely in other applications.