3D Range Images Research Articles

Time-slicing high dynamic range 3D imaging.

Fringe projection profilometry (FPP) has been widely used in 3D measurement due to its high precision and non-contact properties. Nevertheless, it still faces great challenges in measuring scenes with complex reflectivity, in which the dynamic range of the reflected light field of the scene is significantly higher than that of the image detector. In this paper, we propose a time-slicing strategy for high dynamics range 3D imaging by projecting a series of sinusoidal fringe patterns with short and equal length exposure time and performing the fusion of different numbers of short exposure images according to the local gray-value distribution of the images. Moreover, to further improve the measurement efficiency, we realize phase unwrapping using complementary Gray code patterns, which are binary and insensitive to the image sensor's nonlinear response to the reflected light from the scene under test. Experiments are conducted to demonstrate the feasibility and efficiency of the proposed method.

3D observation instrument for pathology tissues

Pathology telemedicine uses digital data of pathological samples to acquire technical support from sophisticated remote experts, aiding cancer diagnosis in less developed regions. In this paper, we propose a color 3D observation system composed of three cameras and a digital projector, achieving complete range 3D imaging of pathology tissues. A new phase mapping method for the projector calibration is proposed by locating the geometric centroid of the checkboard’s white squares, avoiding the center positioning error of traditional circular markers introduced by perspective distortion. A denoising method successively using amplitude modulation judge, phase monotonicity check, and phase abruptness constraint is proposed, which facilitates obtaining clean 3D point clouds.

Long range 3D imaging through atmospheric obscurants using array-based single-photon LiDAR.

Single-photon light detection and ranging (LiDAR) has emerged as a strong candidate technology for active imaging applications. In particular, the single-photon sensitivity and picosecond timing resolution permits high-precision three-dimensional (3D) imaging capability through atmospheric obscurants including fog, haze and smoke. Here we demonstrate an array-based single-photon LiDAR system, which is capable of performing 3D imaging in atmospheric obscurant over long ranges. By adopting the optical optimization of system and the photon-efficient imaging algorithm, we acquire depth and intensity images through dense fog equivalent to 2.74 attenuation lengths at distances of 13.4 km and 20.0 km. Furthermore, we demonstrate real-time 3D imaging for moving targets at 20 frames per second in mist weather conditions over 10.5 km. The results indicate great potential for practical applications of vehicle navigation and target recognition in challenging weather.

Research on infrared image guided GM-APD range image recovery algorithm under limited detections

Geiger-mode avalanche photodiode (GM-APD) focal plane array lidar has a high photon-sensitive detection ability and can acquire three-dimensional(3D) distance information. However, high noise intensity and limited detections can easily lead to the loss of target pixels in the range image, and excellent signal extraction and noise suppression technology become crucial. To solve this problem, we propose a guided image recovery (GIR) method which first uses the infrared image information to recover the GM-APD range image. Unlike image fusion technology, which sacrifices the true range value or the objectively existing 3D range image to get the complete image, it cannot only recover the complete image but also guarantee the 3D image is true. The main innovation of the proposed algorithm is to search the missing pixels of the range image based on the boundary information of infrared images and extracts the range values from their own original measurement data according to the adjacent pixels’ information. According to Monte Carlo simulation and experimental data analysis, the proposed method has excellent target recovery, noise suppression, and fidelity performance under limited detections and low SBRs conditions, which provides an important reference for GM-APD lidar real-time detection. It also provides a new research direction for GM-APD lidar range image reconstruction.

Up-Sampling Method for Low-Resolution LiDAR Point Cloud to Enhance 3D Object Detection in an Autonomous Driving Environment.

Automobile datasets for 3D object detection are typically obtained using expensive high-resolution rotating LiDAR with 64 or more channels (Chs). However, the research budget may be limited such that only a low-resolution LiDAR of 32-Ch or lower can be used. The lower the resolution of the point cloud, the lower the detection accuracy. This study proposes a simple and effective method to up-sample low-resolution point cloud input that enhances the 3D object detection output by reconstructing objects in the sparse point cloud data to produce more dense data. First, the 3D point cloud dataset is converted into a 2D range image with four channels: x, y, z, and intensity. The interpolation on the empty space is calculated based on both the pixel distance and range values of six neighbor points to conserve the shapes of the original object during the reconstruction process. This method solves the over-smoothing problem faced by the conventional interpolation methods, and improves the operational speed and object detection performance when compared to the recent deep-learning-based super-resolution methods. Furthermore, the effectiveness of the up-sampling method on the 3D detection was validated by applying it to baseline 32-Ch point cloud data, which were then selected as the input to a point-pillar detection model. The 3D object detection result on the KITTI dataset demonstrates that the proposed method could increase the mAP (mean average precision) of pedestrians, cyclists, and cars by 9.2%p, 6.3%p, and 5.9%p, respectively, when compared to the baseline of the low-resolution 32-Ch LiDAR input. In future works, various dataset environments apart from autonomous driving will be analyzed.

Meta-RangeSeg: LiDAR Sequence Semantic Segmentation Using Multiple Feature Aggregation

LiDAR sensor is essential to the perception system in autonomous vehicles and intelligent robots. To fulfill the real-time requirements in real-world applications, it is necessary to efficiently segment the LiDAR scans. Most of previous approaches directly project 3D point cloud onto the 2D spherical range image so that they can make use of the efficient 2D convolutional operations for image segmentation. Although having achieved the encouraging results, the neighborhood information is not well-preserved in the spherical projection. Moreover, the temporal information is not taken into consideration in the single scan segmentation task. To tackle these problems, we propose a novel approach to semantic segmentation for LiDAR sequences named Meta-RangeSeg, where a new range residual image representation is introduced to capture the spatial-temporal information. Specifically, Meta-Kernel is employed to extract the meta features, which reduces the inconsistency between the 2D range image coordinates input and 3D Cartesian coordinates output. An efficient U-Net backbone is used to obtain the multi-scale features. Furthermore, Feature Aggregation Module (FAM) strengthens the role of range channel and aggregates features at different levels. We have conducted extensive experiments for performance evaluation on SemanticKITTI and SemanticPOSS. The promising results show that our proposed Meta-RangeSeg method is more efficient and effective than the existing approaches.

An Efficient Ensemble Deep Learning Approach for Semantic Point Cloud Segmentation Based on 3D Geometric Features and Range Images.

Mobile light detection and ranging (LiDAR) sensor point clouds are used in many fields such as road network management, architecture and urban planning, and 3D High Definition (HD) city maps for autonomous vehicles. Semantic segmentation of mobile point clouds is critical for these tasks. In this study, we present a robust and effective deep learning-based point cloud semantic segmentation method. Semantic segmentation is applied to range images produced from point cloud with spherical projection. Irregular 3D mobile point clouds are transformed into regular form by projecting the clouds onto the plane to generate 2D representation of the point cloud. This representation is fed to the proposed network that produces semantic segmentation. The local geometric feature vector is calculated for each point. Optimum parameter experiments were also performed to obtain the best results for semantic segmentation. The proposed technique, called SegUNet3D, is an ensemble approach based on the combination of U-Net and SegNet algorithms. SegUNet3D algorithm has been compared with five different segmentation algorithms on two challenging datasets. SemanticPOSS dataset includes the urban area, whereas RELLIS-3D includes the off-road environment. As a result of the study, it was demonstrated that the proposed approach is superior to other methods in terms of mean Intersection over Union (mIoU) in both datasets. The proposed method was able to improve the mIoU metric by up to 15.9% in the SemanticPOSS dataset and up to 5.4% in the RELLIS-3D dataset.

Real-Time Scene-Aware LiDAR Point Cloud Compression Using Semantic Prior Representation

Existing LiDAR point cloud compression (PCC) methods tend to treat compression as a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fidelity</i> issue, without sufficiently addressing its <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">machine perception</i> aspect. The latter issue is often encountered by the decoder agents that might aim to conduct scene-understanding related tasks only, such as computing the localization information. For tackling this challenge, a novel LiDAR PCC system is proposed to compress the point cloud geometry, which contains a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">back channel</i> for allowing the decoder to initiate such request to the encoder. The key success of our PCC method lies in our proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">semantic prior representation</i> (SPR) and its lossy encoding algorithm with variable precision to generate the final bitstream; the entire process is fast and achieves real-time performance. Note that our SPR is a compact and effective representation of three-dimensional (3D) input point clouds, and it consists of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">labels, predictions</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">residuals</i> . These information can be generated by first exploiting a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scene-aware object segmentation</i> to a set of 2D range images (frames) individually, which were generated from the 3D point clouds via a projection process. Based on the generated labels, the pixels associated with those moving objects are considered as noisy information and should be removed for not only saving bit budget on transmission but also, most importantly, improving the accuracy of localization computed at the decoder. Experimental results conducted on the commonly-used test dataset have shown that our proposed system outperforms the MPEG’s G-PCC (TMC13-v14.0) in a large bitrate range. In fact, the performance gap will become even larger when more and/or large moving objects are involved in the input point clouds.

Point cloud registration and change detection in urban environment using an onboard Lidar sensor and MLS reference data

This paper presents a new method for urban scene analysis, which comprises 3D point cloud registration and change detection through fusing Lidar point clouds with significantly different density characteristics. The introduced method is able to extract dynamic scene segments (traffic participants or urban renewal regions) and seasonal changes (vegetation regions) from instant 3D (i3D) measurements captured by a Rotating Multi-beam (RMB) Lidar sensor mounted onto the top of a moving vehicle. As reference data, we rely on a dense point cloud-based environment model provided by Mobile Laser Scanning (MLS) systems. The proposed approach is composed of new solutions for two main subtasks. First, a novel multimodal point cloud registration algorithm is introduced, which can improve the alignment of the sparse i3D measurements to the dense MLS data, where conventional point level registration or keypoint/segment matching strategies fail. Second, an efficient Markov Random Field-based change extraction step is implemented between the registered point clouds, which exploits that due to geometric considerations of mapping with the given sensor configuration, the essence of the problem can be solved quickly in the 2D range image domain without information loss. Experimental evaluation is conducted on a new Benchmark set that contains three different heavy traffic road sections in city center areas covering in total nearly 1 km long pathway sections. Test data consists of relevant industrial measurements provided by a state-of-the-art RMB scanner (with a point density of around 50-500 points/m2) and an up-to-date MLS system (more than 5000 points/m2). The clear advantages of the new method are quantitatively demonstrated against various reference techniques. In comparison to six different point cloud registration methods, the median value of point level distances is decreased by 1–2 orders of magnitude by the proposed approach. Regarding change detection, the new method outperforms the reference models either in F1-scores (by around 10–25%) or in computational complexity (10–1000 times faster).

A Lightweight Feature Map Creation Method for Intelligent Vehicle Localization in Urban Road Environments

Accurate and reliable localization in urban environments is critical for high-level autonomous driving systems. In comparison to other localization strategies, map-based methods are more reliable and have higher localization accuracy. However, due to higher storage and update costs, dense map-based localization methods are inefficient for widespread use. In urban road environments, structural features such as road curbs that limit the road passable area and pole-like features such as streetlights, tree trunks, and traffic lights exhibit exceptional long-term stability and extraction consistency. The maps based on road curbs and pole-like features are unaffected by dynamic obstacles such as vehicles and pedestrians, making them ideal for vehicle localization in urban environments. In this paper, we design a lightweight feature map for lidar localization in urban environments based on road curbs and pole-like features and propose a novel, fast and accurate extraction strategy for these features, implementing the entire pipeline of feature extraction, map construction, and localization.We combine the orderliness of 2D range images with the spatial properties of 3D point clouds to extract road curbs and poles quickly and accurately. The proposed feature extraction methods and localization system are evaluated on datasets collected in multiple environments and times. The experimental results demonstrate that the method proposed in this paper is more efficient and robust at features extraction and vehicle localization compared with other state-of-the-art approaches.

Two-channel 3D range geometry compression with primitive depth modification

Modern computing and imaging technologies have allowed for many recent advances to be made in the field of 3D range imaging: range data can now be acquired at speeds much faster than real-time, with sub-millimeter precision. However, these benefits come at the cost of an increased quantity of data being generated by 3D range imaging systems, potentially limiting the number of applications that can take advantage of this technology. One common approach to the compression of 3D range data is to encode it within the three color channels of a traditional 24-bit RGB image. This paper presents a novel method for the modification and compression of 3D range data such that the original depth information can be stored within, and recovered from, only two channels of a traditional 2D RGB image. Storage within a traditional image format allows for further compression to be realized via lossless or lossy image compression techniques. For example, when JPEG 80 is used to store the encoded output image, this method achieves an 18.2% reduction in file size when compared to a similar three-channel, image-base compression method, with only a corresponding 0.17% reduction in global reconstruction accuracy.

Solid-state block-based pulsed laser illuminator for single-photon avalanche diode detection-based time-of-flight 3D range imaging

A laser diode illuminator for single-photon avalanche diode detection-based pulsed time-of-flight 3D range imaging is presented. The illuminator supports a block-based illumination scheme and consists of a 16-element custom-designed common anode quantum-well laser diode bar working in the enhanced gain switching regime and lasing at ∼810 nm. The laser diode elements are separately addressable and driven with gallium nitride drivers, which produce current pulses with a width of ∼2 ns; the current pulse amplitude was estimated from the supply voltage (90 V) as 5 to 10 A. Cylindrical optics are used to produce a total illumination field-of-view of 40 × 10 deg2 (full width at half maximum) in 16 separately addressable blocks. With a laser pulsing frequency of 256 kHz and laser pulse energy of ∼8.5 nJ, the average optical illumination power of the transmitter is 2.2 mW.

3D Object Detection and Instance Segmentation from 3D Range and 2D Color Images.

Instance segmentation and object detection are significant problems in the fields of computer vision and robotics. We address those problems by proposing a novel object segmentation and detection system. First, we detect 2D objects based on RGB, depth only, or RGB-D images. A 3D convolutional-based system, named Frustum VoxNet, is proposed. This system generates frustums from 2D detection results, proposes 3D candidate voxelized images for each frustum, and uses a 3D convolutional neural network (CNN) based on these candidates voxelized images to perform the 3D instance segmentation and object detection. Results on the SUN RGB-D dataset show that our RGB-D-based system’s 3D inference is much faster than state-of-the-art methods, without a significant loss of accuracy. At the same time, we can provide segmentation and detection results using depth only images, with accuracy comparable to RGB-D-based systems. This is important since our methods can also work well in low lighting conditions, or with sensors that do not acquire RGB images. Finally, the use of segmentation as part of our pipeline increases detection accuracy, while providing at the same time 3D instance segmentation.

Two-Channel 3D Range Geometry Compression with Virtual Plane Encoding

Modern computing and imaging technologies have allowed for many recent advances to be made in the field of 3D range imaging: range data can now be acquired at speeds much faster than real-time, with sub-millimeter precision. However, these benefits come at the cost of an increased quantity of data being generated by 3D range imaging systems, potentially limiting the number of applications that can take advantage of this technology. One common approach to the compression of 3D range data is to encode it within the three color channels of a traditional 24-bit RGB image. This paper presents a novel method for the modification and compression of 3D range data such that the original depth information can be stored within, and recovered from, only two channels of a traditional 2D RGB image. Storage within a traditional image format allows for further compression to be realized via lossless or lossy image compression techniques. For example, when JPEG 80 was used to store the encoded output image, this method was able to achieve an 18.2% reduction in file size when compared to a similar three-channel, image-base compression method, with only a corresponding 0.17% reduction in global reconstruction accuracy.

Deep learning-based roadway crack classification using laser-scanned range images: A comparative study on hyperparameter selection

In recent years, deep learning-based crack detection methods have been widely explored and applied due to their high versatility and adaptability. In civil engineering applications, recent research on crack detection through deep convolutional neural network (DCNN) includes road pavement crack detection, bridge inspection, defects detection in shield tunnel lining, etc. Despite the increasing popularity of DCNN on crack detection, many challenges have yet to be properly addressed. For crack detection using three-dimensional (3D) range (i.e., elevation) images, disturbances such as surface variation can negatively affect the detection performance. Besides, some typical non-crack patterns such as grooves can be easily misidentified as cracks, i.e., false positives. Another issue lies in the selection of hyperparameters related with the design of a DCNN architecture. For example, the hyperparameters which are related with network structure (e.g., kernel size, network depth and width) and training (e.g., mini-batch size and learning rate) can impact the network performance to a significant extent. Therefore, they need to be properly determined for optimal performance. However, for deep learning-based roadway crack classification using laser-scanned range images, a comprehensive discussion on the hyperparameter selection/tuning has not been thoroughly performed. This study develops a hyperparameter selection process involving a series of experiments on laser-scanned range images with high diversities, investigating the optimal joint hyperparameter configuration on network structure and training for DCNN-based roadway crack classification. In a comparative study, 36 DCNN architectures with varying layouts are developed for crack classification. These architecture candidates differ in kernel sizes (e.g., 3 × 3, 7 × 7, and 11 × 11), network depths (from 5 to 8 weight layers), and widths (from 16 to 96 kernels in each convolutional layer). The 7-layer DCNN with constant 7 × 7 kernels and increasing network widths yields the highest classification performance among the proposed 36 DCNN classifiers, which may be because it can best reflect the complexity of the acquired laser-scanned roadway range images. Once the optimal architecture layout is determined, further discussion on the selection of min-batch sizes, learning rates, dropout factor and leaky rectified linear unit (LReLU) factor is performed. Experimental results show the optimal architecture with associated training configuration can achieve consistent and accurate performance, under the contamination of surface variations and grooved patterns in laser-scanned range images. Discussion on the hyperparameter selection can provide insights for the development of DCNN in similar applications using laser-scanned range images.

Photon counting polarization imaging strategy for target classification under photon-starved environments

Considering the requirements of long-range detection, eye safety and anti-reconnaissance, the detection of weak signal has become a research hotspot in the field of lidar. Gm-APD(Geiger-Mode Avalanche Photodiode) has extremely high response sensitivity and extremely high timing accuracy. At present, Gm-APD photon counting lidar system can obtain the 3D (Dimension) range image of target in the case of weak echo signals. However, considering the detection of obscured target, the 3D range image cannot distinguish which pixel point echoes come from the target and which come from the shelter, which will cause the target recognition to be blurred and uncertain. In this paper, a Gm-APD photon counting polarization imaging lidar system is proposed, which can simultaneously obtain the range and polarization information of the target under the condition of the extremely weak echo optical signals (the number of photons per pulse echo is much less than 1). Finally, the experimental system was built to image a model airplane shaded by tree branches, with an average of 0.01 photons per echo pulse, a clear image of model airplane is separated from the shelter according to polarization information.

Snapshot Compressive ToF+Spectral Imaging via Optimized Color-Coded Apertures.

Compressive multispectral imaging systems comprise a new generation of spectral imagers that capture coded projections of a scene where spectral data cubes are reconstructed computationally. Separately, time-of-flight (ToF) cameras obtain 2D range images where each pixel records the distance from the camera sensor to the target surface. The demand for these imaging modalities is rapidly increasing, and thus, there is strong interest in developing new image sensors that can simultaneously acquire multispectral-color-and-depth imagery (MS+D) using a single aperture. Work in this path has been mainly developed via RGB+D imaging. However, in RGB+D, the multispectral image is limited to three spectral channels, and the imaging system often relies on two image sensors. We recently proposed a compressive MS+D imaging device that used a digital-micromirror-device, requiring a bulky double imaging-and-relay path. To overcome the bulkiness and other difficulties of our previous imaging system, this work presents a more-compact MS+D imaging device with snapshot capabilities. It provides better spectral sensing, relying on a static color-coded-aperture (CCA) and a ToF sensor. To guarantee good quality in the recovery, we develop an optimization method for CCA based-on blue-noise-multitoning, solved via the direct-binary-search algorithm. A testbed-setup is reported along with simulated and real experiments that demonstrate the MS+D capabilities of the proposed system over static and dynamic scenes.

A range-gated CMOS SPAD array for real-time 3D range imaging

A range image of a scene is produced with a solid-state time of- flight system that uses active illumination and a time-gated single photon avalanche diode (SPAD) array. The distance to a target from the imager is measured by delaying the time gate in small steps and counting the photons of the pixels in each delay step in successive measurements. To achieve a high frame rate, the number of delay steps needed is minimized by limiting the scanning of the depth only to the range of interest. To be able to measure scenes with objects in different ranges, the array has been divided into groups of pixels with independently controlled time gating. This paper demonstrates an algorithm that can be used to control the time gating of the pixel groups in the sensor array to achieve depth maps of the scene with the time-gated SPAD array in real time and at a 70 Hz frame rate.

Hybrid lidar system using combination of streak tube imaging and time-correlated single-photon counting

A hybrid lidar system design using the complementarity streak tube imaging lidar (STIL) and time-correlated single-photon counting (TCSPC) lidar is proposed. The new scheme is proposed for the issue that the low sensitivity of STIL system when detecting weak echo signals, and the range walk error of TCSPC lidar system when detecting strong echo signals. The hybrid system is validated by an indoor experiment. The 3D range image of the target made by a 3D printer is obtained, and the makers on the target with a height of 1 cm is clearly identified. The range data have a root mean square error of 2.9 mm. The results show that the applicability for dynamic range of echo signal intensity is improved by the hybrid lidar system.

A resample strategy and artificial bee colony optimization-based 3d range imaging registration

For two point clouds with low overlapping rate, the registration process is more difficult and the registration speed is slow. In this paper, we reduce the number of sampling points to simplify the calculation, and then propose a new idea based on equal interval method called resample strategy, it can effectively avoid ambiguity during the registration process by improving the ergodicity and utilization of the sampling point set. In addition, we also introduce a new solution search equation that has more exploitation performance to alternatively search with an enhanced artificial bee colony algorithm. The computation time has been effectively reduced by using these two proposed strategies. The registration experimental results aiming to a variety of point cloud models show that our 3d image registration algorithm is better than many other algorithms based on classical or improved bionic intelligence optimization methods.