Scene Points Research Articles

CSPC-Dataset: New LiDAR Point Cloud Dataset and Benchmark for Large-Scale Scene Semantic Segmentation

Large-scale point clouds scanned by light detection and ranging (lidar) sensors provide detailed geometric characteristics of scenes due to the provision of 3D structural data. The semantic segmentation of large-scale point clouds is a crucial step for an in-depth understanding of complex scenes. Of late, although a large number of point cloud semantic segmentation algorithms have been proposed, semantic segmentation methods are still far from being satisfactory in terms of precision and accuracy of large-scale point clouds. For machine learning (ML) and deep learning (DL) methodologies, the semantic segmentation is largely influenced by the quality of training sets and methods themselves. Therefore, we construct a new point cloud dataset, namely CSPC-Dataset (Complex Scene Point Cloud Dataset) for large-scale scene semantic segmentation. CSPC-Dataset point clouds are acquired by a wearable laser mobile mapping robot. It covers five complex urban and rural scenes and mainly includes six types of objects, i.e., ground, car, building, vegetation, bridge, and pole. It provides large-scale outdoor scenes with color information, which has advantages such as the scene more complete, point density relatively uniform, diversity and complexity of objects and the high discrepancy between different scenes. Based on the CSPC-Dataset, we construct a new benchmark, which includes approximately 68 million points with explicit semantic labels. To extend the dataset into a wide range of applications, this paper provides the semantic segmentation results and comparative analysis of 7 baseline methods based on CSPC-Dataset. In the experiment part, three groups of experiments are conducted for benchmarking, which offers an effective way to make comparisons with different point-labeling algorithms. The labeling results have shown that the highest Intersection over Union (IoU) of pole, ground, building, car, vegetation, and bridge for all benchmarks is 36.0%, 97.8%, 93.7%, 65.6%, 92.0%, and 69.6%.

A spatio-temporal multi-scale binary descriptor.

Binary descriptors are widely used for multi-view matching and robotic navigation. However, their matching performance decreases considerably under severe scale and viewpoint changes in non-planar scenes. To overcome this problem, we propose to encode the varying appearance of selected 3D scene points tracked by a moving camera with compact spatio-temporal descriptors. To this end, we first track interest points and capture their temporal variations at multiple scales. Then, we validate feature tracks through 3D reconstruction and compress the temporal sequence of descriptors by encoding the most frequent and stable binary values. Finally, we determine multiscale correspondences across views with a matching strategy that handles severe scale differences. The proposed spatio-temporal multi-scale approach is generic and can be used with a variety of binary descriptors. We show the effectiveness of the joint multiscale extraction and temporal reduction through comparisons of different temporal reduction strategies and the application to several binary descriptors.

Noise Reduction on Bracketed Images for High Dynamic Range Imaging

The quality of a high dynamic range (HDR) image produced from bracketed images taken at different camera exposure times can be degraded by noise contained in bracketed images. In this paper, we propose a noise reduction method on bracketed images based on exposure time ratio. First, for each pixel pair of a same scene point lying on two different images, the ratio of their intensity values is compared with the ratio of exposure times of the images on which the pixels are lying. If the compared ratios are close, these two pixels are included in noise-free pixels set. The complement of noise-free pixels set is defined as noisy pixels set. Then, the intensity value of each pixel in noisy pixels set is corrected by its expected value computed from noise-free pixel of the same scene point lying on another image. Experimental results show that all the noisy intensity values can be correctly restored from noise-free pixels except the saturated pixels. Denoising results by PSNR show that the proposed method outperforms the other recent denoising methods such as based-on pixel density filter (BPDF), noise adaptive fuzzy switching median filter (NAFSMF), and adaptive Riesz mean filter (ARmF).

Catadioptric images compression using an adapted neighborhood and the shape-adaptive DCT

In the catadioptric images, the neighborhood topology is usually modified by the mirror shape which consists of significant radial distortions and low non-uniform resolution because of its convexity geometry. Consequently, conventional compression approaches (i.e. H.264 intra-frame coding or JPEG standard) based on rectangular block partitioning cannot be applied directly to catadioptric images because they inevitably produce significant block effects. In this investigation, we propose a new coding scheme based on an adaptive block partitioning. Particularly, to reduce the blocking artifacts, we propose the shape-adaptive discrete cosine transform (SA-DCT) and adapted neighborhood. The adapted neighborhood is obtained from the equivalence between the catadioptric image and the scene point’s projection on the unit sphere. The adapted blocks shape correctly follows the geometric context and varies according to the resolution of the catadioptric images. Such changes improve considerably the compression efficiency using the block partitioning in catadioptric images. Experimental results figured out that our proposal scheme outperforms the JPEG standard and the Intra-frame coder H264. Moreover the proposed coding scheme based on SA-DCT transform and the suitable partitioning for catadioptric images provides high PSNR and a considerable reduction of block effects compared to conventional approaches, especially in the case of low bit rate.

Point Set Multi-Level Aggregation Feature Extraction Based on Multi-Scale Max Pooling and LDA for Point Cloud Classification

Accurate and effective classification of lidar point clouds with discriminative features expression is a challenging task for scene understanding. In order to improve the accuracy and the robustness of point cloud classification based on single point features, we propose a novel point set multi-level aggregation features extraction and fusion method based on multi-scale max pooling and latent Dirichlet allocation (LDA). To this end, in the hierarchical point set feature extraction, point sets of different levels and sizes are first adaptively generated through multi-level clustering. Then, more effective sparse representation is implemented by locality-constrained linear coding (LLC) based on single point features, which contributes to the extraction of discriminative individual point set features. Next, the local point set features are extracted by combining the max pooling method and the multi-scale pyramid structure constructed by the point’s coordinates within each point set. The global and the local features of the point sets are effectively expressed by the fusion of multi-scale max pooling features and global features constructed by the point set LLC-LDA model. The point clouds are classified by using the point set multi-level aggregation features. Our experiments on two scenes of airborne laser scanning (ALS) point clouds—a mobile laser scanning (MLS) scene point cloud and a terrestrial laser scanning (TLS) scene point cloud—demonstrate the effectiveness of the proposed point set multi-level aggregation features for point cloud classification, and the proposed method outperforms other related and compared algorithms.

Automatic identification and autonomous sorting of cylindrical parts in cluttered scene based on monocular vision 3D reconstruction

Purpose This paper aims to propose an identification method based on monocular vision for cylindrical parts in cluttered scene, which solves the issue that iterative closest point (ICP) algorithm fails to obtain global optimal solution, as the deviation from scene point cloud to target CAD model is huge in nature. Design/methodology/approach The images of the parts are captured at three locations by a camera amounted on a robotic end effector to reconstruct initial scene point cloud. Color signatures of histogram of orientations (C-SHOT) local feature descriptors are extracted from the model and scene point cloud. Random sample consensus (RANSAC) algorithm is used to perform the first initial matching of point sets. Then, the second initial matching is conducted by proposed remote closest point (RCP) algorithm to make the model get close to the scene point cloud. Levenberg Marquardt (LM)-ICP is used to complete fine registration to obtain accurate pose estimation. Findings The experimental results in bolt-cluttered scene demonstrate that the accuracy of pose estimation obtained by the proposed method is higher than that obtained by two other methods. The position error is less than 0.92 mm and the orientation error is less than 0.86°. The average recognition rate is 96.67 per cent and the identification time of the single bolt does not exceed 3.5 s. Practical implications The presented approach can be applied or integrated into automatic sorting production lines in the factories. Originality/value The proposed method improves the efficiency and accuracy of the identification and classification of cylindrical parts using a robotic arm.

Design of multipath error correction algorithm of coarse and fine sparse decomposition based on compressed sensing in time-of-flight cameras

ABSTRACTA single pixel in time of flight cameras receives multiple reflected light from different scene points, resulting in erroneous depth information. In this paper, based on the proposed sparse decomposition, a coarse and fine sparse decomposition based on compressed sensing is applied to multipath separation. The applied method uses a linear combination of multiple frequency signals to modulate the source. The measured vector obtained through finite random measurements is subjected to two sparse decompositions – rough separation and detailed positioning, and finally the minimum direct path depth is accurately recovered. Under the premise of the same number of measurements, calculation amount, and storage space, the accuracy of the coarse and fine sparse decomposition based on compressed sensing is improved by nearly an order of magnitude compared to the sparse decomposition without compressed sensing. Moreover, our method can basically achieve multi-path separation accuracy to the sub-millimeter level.

3D shape measurement of translucent objects based on Fourier single-pixel imaging in projector-camera system.

3D shape measurement by structured light is a popular technique for recovering object surfaces. However, structured light technique assumes that scene points are directly illuminated by the light source(s). Consequently, global illumination effects, such as subsurface scattering in translucent objects, may cause measurement errors in recovered 3D shapes. In this research, we propose a 3D shape measurement method of translucent objects based on Fourier single-pixel imaging (FSI) technique. The 3D shapes of the translucent objects are reconstructed through stereo matching of direct illumination light, which is separated from the surface. Experimental results show that the proposed method can separate the direct illumination light and the subsurface scattering light. The feasibility and accuracy of the method are analyzed, and the qualitative and quantitative results of the method are provided.

Combined Edge- and Stixel-based Object Detection in 3D Point Cloud.

Environment perception is critical for feasible path planning and safe driving for autonomous vehicles. Perception devices, such as camera, LiDAR (Light Detection and Ranging), IMU (Inertial Measurement Unit), etc., only provide raw sensing data with no identification of vital objects, which is insufficient for autonomous vehicles to perform safe and efficient self-driving operations. This study proposes an improved edge-oriented segmentation-based method to detect the objects from the sensed three-dimensional (3D) point cloud. The improved edge-oriented segmentation-based method consists of three main steps: First, the bounding areas of objects are identified by edge detection and stixel estimation in corresponding two-dimensional (2D) images taken by a stereo camera. Second, 3D sparse point clouds of objects are reconstructed in bounding areas. Finally, the dense point clouds of objects are segmented by matching the 3D sparse point clouds of objects with the whole scene point cloud. After comparison with the existing methods of segmentation, the experimental results demonstrate that the proposed edge-oriented segmentation method improves the precision of 3D point cloud segmentation, and that the objects can be segmented accurately. Meanwhile, the visualization of output data in advanced driving assistance systems (ADAS) can be greatly facilitated due to the decrease in computational time and the decrease in the number of points in the object’s point cloud.

Appearance Flow Completion for Novel View Synthesis

AbstractNovel view synthesis from sparse and unstructured input views faces challenges like the difficulty with dense 3D reconstruction and large occlusion. This paper addresses these problems by estimating proper appearance flows from the target to input views to warp and blend the input views. Our method first estimates a sparse set 3D scene points using an off‐the‐shelf 3D reconstruction method and calculates sparse flows from the target to input views. Our method then performs appearance flow completion to estimate the dense flows from the corresponding sparse ones. Specifically, we design a deep fully convolutional neural network that takes sparse flows and input views as input and outputs the dense flows. Furthermore, we estimate the optical flows between input views as references to guide the estimation of dense flows between the target view and input views. Besides the dense flows, our network also estimates the masks to blend multiple warped inputs to render the target view. Experiments on the KITTI benchmark show that our method can generate high quality novel views from sparse and unstructured input views.

Evolution of New Integrated Haze Removal Algorithm Based on Haze Line

Haze is a standout amongst the most essential factor that debases the outside pictures. Picture corruption relies upon the separation of the Object from the camera as the separation increment from the camera of the scene point haze additionally increases. This paper presents a haze removal technique which is relies on the Haze-line prior that is recently introduced. This prior is based on the observation that the pixel values of a hazy image can be modeled as lines in RGB space that intersect at the air- light proposed approach is known as haze line. In this paper we proposed Post processing CBMG (Contrast, Brightness, Midtone and Gama adjustment) followed by four stages: clustering the pixel into haze line, Estimation of Transmission map, Regularization, Dehazing, This method is implement by observation of two basic things first pixels in a given group are regularly non local . Second pictures with improved perceivability (or sunny morning picture) have more differentiation than the pictures stopped by terrible climate. Experimental result demonstrate that our method remove haze layer and provide qualitative result.

Model-Based 3D Pose Estimation of a Single RGB Image Using a Deep Viewpoint Classification Neural Network

This paper presents a model-based approach for 3D pose estimation of a single RGB image to keep the 3D scene model up-to-date using a low-cost camera. A prelearned image model of the target scene is first reconstructed using a training RGB-D video. Next, the model is analyzed using the proposed multiple principal analysis to label the viewpoint class of each training RGB image and construct a training dataset for training a deep learning viewpoint classification neural network (DVCNN). For all training images in a viewpoint class, the DVCNN estimates their membership probabilities and defines the template of the class as the one of the highest probability. To achieve the goal of scene reconstruction in a 3D space using a camera, using the information of templates, a pose estimation algorithm follows to estimate the pose parameters and depth map of a single RGB image captured by navigating the camera to a specific viewpoint. Obviously, the pose estimation algorithm is the key to success for updating the status of the 3D scene. To compare with conventional pose estimation algorithms which use sparse features for pose estimation, our approach enhances the quality of reconstructing the 3D scene point cloud using the template-to-frame registration. Finally, we verify the ability of the established reconstruction system on publicly available benchmark datasets and compare it with the state-of-the-art pose estimation algorithms. The results indicate that our approach outperforms the compared methods in terms of the accuracy of pose estimation.

Semi-automated Cleaning of Laser Scanning Campaigns with Machine Learning

Terrestrial laser scanning campaigns provide an important means to document the 3D structure of historical sites. Unfortunately, the process of converting the 3D point clouds acquired by the laser scanner into a coherent and accurate 3D model has many stages and is not generally automated. In particular, the initial cleaning stage of the pipeline—in which undesired scene points are deleted—remains largely manual and is usually labour intensive. In this article, we introduce a semi-automated cleaning approach that incrementally trains a random forest (RF) classifier on an initial keep/discard point labelling generated by the user when cleaning the first scan(s). The classifier is then used to predict the labelling of the next scan in the sequence. Before this classification is presented to the user, a denoising post-process, based on the 2D range map representation of the laser scan, is applied. This significantly reduces small isolated point clusters that the user would otherwise have to fix. The user then selects the remaining incorrectly labelled points and these are weighted, based on a confidence estimate, and fed back into the classifier to retrain it for the next scan. Our experiments, across 8 scanning campaigns, show that when the scan campaign is coherent, i.e., it does not contain widely disparate or contradictory data, the classifier yields a keep/discard labelling that typically ranges between 95% and 99%. This is somewhat surprising, given that the data in each class can represent many object types, such as a tree, person, wall, and so on, and that no further effort beyond the point labeling of keep/discard is required of the user. We conducted an informal timing experiment over a 15-scan campaign, which compared the processing time required by our software, without user interaction (point label correction) time, against the time taken by an expert user to completely clean all scans. The expert user required 95mins to complete all cleaning. The average time required by the expert to clean a single scan was 6.3mins. Even with current unoptimized code, our system was able to generate keep/discard labels for all scans, with 98% (average) accuracy, in 75mins. This leaves as much as 20mins for the user input required to relabel the 2% of mispredicted points across the set of scans before the full system time would match the expert’s cleaning time.

Three-dimensional reconstruction of a point immersed in water and application to a single rising bubble

The ability to reconstruct an object submerged in water is instrumental in scenarios such as the study of a rising bubble trajectory, and subsequently, its motion and force balance. In this paper, we propose a method that can reconstruct the three-dimensional position of a scene point immersed in water, while taking into account the refraction-induced distortion at the air–water interface between the point and camera. The scene point is captured by two orthogonally placed high-speed cameras, providing a pair of images through which the point’s image coordinates are obtained. With pre-calibrated camera matrices, the world coordinates of the point’s position can be correctly calculated using a triangulation method, provided the scene point is in air. However, because of refraction, triangulating the image pairs of the point submerged in water results in erroneous world coordinates. Thus, we propose a method to correct the point’s image coordinates to account for refraction induced distortion during reconstruction. The method was first verified by reconstructing the coordinates of square corners on a checkerboard, which produced results that deviated from the real value by 0.18%, as opposed to the 5.15% false enlargement prior to correction. Then, we applied the method to the reconstruction of the three-dimensional trajectories of single rising spherical bubbles, whose results were in favorable agreement with previous studies.

Video Extrapolation Using Neighboring Frames

With the popularity of immersive display systems that fill the viewer’s field of view (FOV) entirely, demand for wide FOV content has increased. A video extrapolation technique based on reuse of existing videos is one of the most efficient ways to produce wide FOV content. Extrapolating a video poses a great challenge, however, due to the insufficient amount of cues and information that can be leveraged for the estimation of the extended region. This article introduces a novel framework that allows the extrapolation of an input video and consequently converts a conventional content into one with wide FOV. The key idea of the proposed approach is to integrate the information from all frames in the input video into each frame. Utilizing the information from all frames is crucial because it is very difficult to achieve the goal with a two-dimensional transformation based approach when parallax caused by camera motion is apparent. Warping guided by three-dimensnional scene points matches the viewpoints between the different frames. The matched frames are blended to create extended views. Various experiments demonstrate that the results of the proposed method are more visually plausible than those produced using state-of-the-art techniques.

Aligning 2.5D Scene Fragments With Distinctive Local Geometric Features and Voting-Based Correspondences

Aligning 2.5D views has been extensively explored in the past decades, where most prior works have concentrated on object data with complex structures. This paper presents a method to align real-word scene scans with challenging features such as noise, poor geometric information, and highly repeatable patterns. Our method consists of two modules: pairwise and multiview alignments. Key to the proposed pairwise alignment method is the rotational contour signature geometric feature and voting-based correspondence selection algorithm. The former promises strong discriminative power for 2.5D scene data, while the latter affords high-quality correspondences via a voting process for all raw feature matches using $L_{2}$ distance and point pair affinity constraints. For the multiview alignment method, we first use a connected graph algorithm to establish the connections of all 2.5D views for coarse merging; then, we propose a shape-growing iterative closest point algorithm for further refinement. Experiments are conducted on scene point cloud datasets addressing both the indoor and outdoor scenarios, whereby we demonstrate that the proposed pairwise alignment method clearly outperforms the state of the art. Moreover, the proposed multiview alignment method manages to put multiple unordered 2.5D scene fragments into a unified coordinate system automatically, accurately, and efficiently.

Top-Down Approach to the Automatic Extraction of Individual Trees from Scanned Scene Point Cloud Data

Urban trees are essential elements in outdoor scenes recorded via terrestrial laser scanning. Although considerable interest has been centered on tree detection and reconstruction in r ...

Spherical-Model-Based SLAM on Full-View Images for Indoor Environments

As we know, SLAM (Simultaneous Localization and Mapping) relies on surroundings. A full-view image provides more benefits to SLAM than a limited-view image. In this paper, we present a spherical-model-based SLAM on full-view images for indoor environments. Unlike traditional limited-view images, the full-view image has its own specific imaging principle (which is nonlinear), and is accompanied by distortions. Thus, specific techniques are needed for processing a full-view image. In the proposed method, we first use a spherical model to express the full-view image. Then, the algorithms are implemented based on the spherical model, including feature points extraction, feature points matching, 2D-3D connection, and projection and back-projection of scene points. Thanks to the full field of view, the experiments show that the proposed method effectively handles sparse-feature or partially non-feature environments, and also achieves high accuracy in localization and mapping. An experiment is conducted to prove that the accuracy is affected by the view field.

Deep Video Dehazing with Semantic Segmentation.

Recent research have shown the potential of using convolutional neural networks (CNNs) to accomplish single image dehazing. In this work, we take one step further to explore the possibility of exploiting a network to perform haze removal for videos. Unlike single image dehazing, video based approaches can take advantage of the abundant information that exists across neighboring frames. In this work, assuming that a scene point yields highly correlated transmission values between adjacent video frames, we develop a deep learning solution for video dehazing, where a CNN is trained end-to-end to learn how to accumulate information across frames for transmission estimation. The estimated transmission map is subsequently used to recover a haze-free frame via atmospheric scattering model. In addition, as the semantic information of a scene provides a strong prior for image restoration, we propose to incorporate global semantic priors as input to regularize the transmission maps so that the estimated maps can be smooth in the regions of the same object and only discontinuous across the boundaries of different objects. To train this network, we generate a dataset consisted of synthetic hazy and haze-free videos for supervision based on the NYU depth dataset. We show that the features learned from this dataset are capable of removing haze that arises in outdoor scenes in a wide range of videos. Extensive experiments demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods on both synthetic and real-world videos.

Real-time layer-based computer-generated hologram calculation for the Fourier transform optical system.

With the growing interest for augmented reality devices, holography is often considered as a promising technology to overcome the focus issues of conventional stereoscopic displays. To enlarge the field of view of holographic head-mounted displays, a Fourier transform optical system (FTOS) has been proposed. However, since the scene geometry is distorted by the FTOS, it is necessary to compensate the position of each scene point during the hologram computation, resulting in long calculation times. In this paper, we propose a real-time computer-generated hologram calculation method for the FTOS. Whereas previously proposed methods used a ray-tracing approach to compensate the distortion induced by the FTOS, our proposed method relies on a layer-based approach. Experimental results show that our method is able to compute holograms of resolution (3840×2160) in real time at 24 frames per second, enabling its use in augmented reality applications.