Point Cloud Compression Research Articles

Phase-only hologram video compression using a deep neural network for up-scaling and restoration.

This paper proposes a coding method for compressing a phase-only hologram video (PoHV), which can be directly displayed in a commercial phase-only spatial light modulator. Recently, there has been active research to use a standard codec as an anchor to develop a new video coding for 3D data such as MPEG point cloud compression. The main merit of this approach is that if a new video codec is developed, the performance of relative coding methods can be increased simultaneously. Furthermore, compatibility is increased by the capability to use various anchor codecs, and the developing time is decreased. This paper uses a currently used video codec as an anchor codec and develops a coding method including progressive scaling and a deep neural network to overcome low temporal correlation between frames of a PoHV. Since it is difficult to temporally predict a correlation between frames of a PoHV, this paper adopts a scaling function and a neural network in the encoding and decoding process, not adding complexity to an anchor itself to predict temporal correlation. The proposed coding method shows an enhanced coding gain of an average of 22%, compared with an anchor in all coding conditions. When observing numerical and optical reconstructions, the result images by the proposed show clearer objects and less juddering than the result by the anchor.

3QNet

Since the development of 3D applications, the point cloud, as a spatial description easily acquired by sensors, has been widely used in multiple areas such as SLAM and 3D reconstruction. Point Cloud Compression (PCC) has also attracted more attention as a primary step before point cloud transferring and saving, where the geometry compression is an important component of PCC to compress the points geometrical structures. However, existing non-learning-based geometry compression methods are often limited by manually pre-defined compression rules. Though learning-based compression methods can significantly improve the algorithm performances by learning compression rules from data, they still have some defects. Voxel-based compression networks introduce precision errors due to the voxelized operations, while point-based methods may have relatively weak robustness and are mainly designed for sparse point clouds. In this work, we propose a novel learning-based point cloud compression framework named 3D Point Cloud Geometry Quantiation Compression Network (3QNet), which overcomes the robustness limitation of existing point-based methods and can handle dense points. By learning a codebook including common structural features from simple and sparse shapes, 3QNet can efficiently deal with multiple kinds of point clouds. According to experiments on object models, indoor scenes, and outdoor scans, 3QNet can achieve better compression performances than many representative methods.

Point Cloud Geometry Compression Based on Multi-Layer Residual Structure.

Point cloud data are extensively used in various applications, such as autonomous driving and augmented reality since it can provide both detailed and realistic depictions of 3D scenes or objects. Meanwhile, 3D point clouds generally occupy a large amount of storage space that is a big burden for efficient communication. However, it is difficult to efficiently compress such sparse, disordered, non-uniform and high dimensional data. Therefore, this work proposes a novel deep-learning framework for point cloud geometric compression based on an autoencoder architecture. Specifically, a multi-layer residual module is designed on a sparse convolution-based autoencoders that progressively down-samples the input point clouds and reconstructs the point clouds in a hierarchically way. It effectively constrains the accuracy of the sampling process at the encoder side, which significantly preserves the feature information with a decrease in the data volume. Compared with the state-of-the-art geometry-based point cloud compression (G-PCC) schemes, our approach obtains more than 70-90% BD-Rate gain on an object point cloud dataset and achieves a better point cloud reconstruction quality. Additionally, compared to the state-of-the-art PCGCv2, we achieve an average gain of about 10% in BD-Rate.

Rate-Distortion Optimal Transform Coefficient Selection for Unoccupied Regions in Video-Based Point Cloud Compression

This paper presents a novel method to determine rate-distortion optimized transform coefficients for efficient compression of videos generated from point clouds. The method exploits a generalized frequency selective extrapolation approach that iteratively determines rate-distortion-optimized coefficients for all basis functions of two-dimensional discrete cosine and sine transforms. The method is applied to blocks containing both occupied and unoccupied pixels in video based point cloud compression for HEVC encoding. In the proposed algorithm, only the values of the transform coefficients are changed such that resulting bit streams are compliant to the V-PCC standard. For all-intra coded point clouds, bitrate savings of more than 4% for geometry and more than 6% for texture error metrics with respect to standard encoding can be observed. These savings are more than twice as high as savings obtained using competing methods from literature. In the randomaccess case, our proposed method outperforms competing V-PCC methods by more than 0.5%.

Real-Time LiDAR Point Cloud Compression Using Bi-Directional Prediction and Range-Adaptive Floating-Point Coding

Due to the large amount of data involved in the three-dimensional (3D) LiDAR point clouds, point cloud compression (PCC) becomes indispensable to many real-time applications. In autonomous driving of connected vehicles for example, point clouds are constantly acquired along the time and subjected to be compressed. Among the existing PCC methods, very few of them have effectively removed the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">temporal</i> redundancy inherited in the point clouds. To address this issue, a novel lossy LiDAR PCC system is proposed in this paper, which consists of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">inter</i> -frame coding and the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">intra</i> -frame coding. For the former, a deep-learning approach is proposed to conduct bi-directional frame prediction using an asymmetric residual module and 3D space-time convolutions; the proposed network is called the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">bi-directional prediction network</i> (BPNet). For the latter, a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">range-adaptive floating-point coding</i> (RAFC) algorithm is proposed for encoding the reference frames and the B-frame prediction residuals in the 32-bit floating-point precision. Since the pixel-value distribution of these two types of data are quite different, various encoding modes are designed for providing adaptive selection. Extensive simulation experiments have been conducted using multiple point cloud datasets, and the results clearly show that our proposed PCC system consistently outperforms the state-of-the-art MPEG G-PCC in terms of data fidelity and localization, while delivering real-time performance.

Point Cloud Compression: Impact on Object Detection in Outdoor Contexts.

Increasing demand for more reliable and safe autonomous driving means that data involved in the various aspects of perception, such as object detection, will become more granular as the number and resolution of sensors progress. Using these data for on-the-fly object detection causes problems related to the computational complexity of onboard processing in autonomous vehicles, leading to a desire to offload computation to roadside infrastructure using vehicle-to-infrastructure communication links. The need to transmit sensor data also arises in the context of vehicle fleets exchanging sensor data, over vehicle-to-vehicle communication links. Some types of sensor data modalities, such as Light Detection and Ranging (LiDAR) point clouds, are so voluminous that their transmission is impractical without data compression. With most emerging autonomous driving implementations being anchored on point cloud data, we propose to evaluate the impact of point cloud compression on object detection. To that end, two different object detection architectures are evaluated using point clouds from the KITTI object dataset: raw point clouds and point clouds compressed with a state-of-the-art encoder and three different compression levels. The analysis is extended to the impact of compression on depth maps generated from images projected from the point clouds, with two conversion methods tested. Results show that low-to-medium levels of compression do not have a major impact on object detection performance, especially for larger objects. Results also show that the impact of point cloud compression is lower when detecting objects using depth maps, placing this particular method of point cloud data representation on a competitive footing compared to raw point cloud data.

Real-Time Scene-Aware LiDAR Point Cloud Compression Using Semantic Prior Representation

Existing LiDAR point cloud compression (PCC) methods tend to treat compression as a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fidelity</i> issue, without sufficiently addressing its <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">machine perception</i> aspect. The latter issue is often encountered by the decoder agents that might aim to conduct scene-understanding related tasks only, such as computing the localization information. For tackling this challenge, a novel LiDAR PCC system is proposed to compress the point cloud geometry, which contains a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">back channel</i> for allowing the decoder to initiate such request to the encoder. The key success of our PCC method lies in our proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">semantic prior representation</i> (SPR) and its lossy encoding algorithm with variable precision to generate the final bitstream; the entire process is fast and achieves real-time performance. Note that our SPR is a compact and effective representation of three-dimensional (3D) input point clouds, and it consists of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">labels, predictions</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">residuals</i> . These information can be generated by first exploiting a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scene-aware object segmentation</i> to a set of 2D range images (frames) individually, which were generated from the 3D point clouds via a projection process. Based on the generated labels, the pixels associated with those moving objects are considered as noisy information and should be removed for not only saving bit budget on transmission but also, most importantly, improving the accuracy of localization computed at the decoder. Experimental results conducted on the commonly-used test dataset have shown that our proposed system outperforms the MPEG’s G-PCC (TMC13-v14.0) in a large bitrate range. In fact, the performance gap will become even larger when more and/or large moving objects are involved in the input point clouds.

OctAttention: Octree-Based Large-Scale Contexts Model for Point Cloud Compression

In point cloud compression, sufficient contexts are significant for modeling the point cloud distribution. However, the contexts gathered by the previous voxel-based methods decrease when handling sparse point clouds. To address this problem, we propose a multiple-contexts deep learning framework called OctAttention employing the octree structure, a memory-efficient representation for point clouds. Our approach encodes octree symbol sequences in a lossless way by gathering the information of sibling and ancestor nodes. Expressly, we first represent point clouds with octree to reduce spatial redundancy, which is robust for point clouds with different resolutions. We then design a conditional entropy model with a large receptive field that models the sibling and ancestor contexts to exploit the strong dependency among the neighboring nodes and employ an attention mechanism to emphasize the correlated nodes in the context. Furthermore, we introduce a mask operation during training and testing to make a trade-off between encoding time and performance. Compared to the previous state-of-the-art works, our approach obtains a 10%-35% BD-Rate gain on the LiDAR benchmark (e.g. SemanticKITTI) and object point cloud dataset (e.g. MPEG 8i, MVUB), and saves 95% coding time compared to the voxel-based baseline. The code is available at https://github.com/zb12138/OctAttention.

Virtual structured-light three-dimensional point cloud compression with geometric reshaping

Converting three-dimensional (3D) point cloud data to two-dimensional (2D) image based on virtual structured-light system is a popular method in 3D data compression because the image format is easier to process by the exiting storing and transmitting methods. When this method is used in the 3D point cloud compression, the quantization error is introduced during the coordinate mapping. To solve this problem, a virtual structured-light 3D point cloud compression algorithm based on geometric reshaping is presented. And a least-square system parameter optimization method is proposed to further improve the data decoding accuracy. In the proposed method, the 3D spatial coordinates are reshaped to a 2D matrix first, and then the 2D matrix is stored as a “Holoimage” by using the parameter optimized virtual structured-light system. The quantization error introduced by coordinate mapping between X and Y coordinates of the 3D point cloud and the 2D image pixel coordinates is suppressed, so the decoding accuracy is improved. In addition, the geometric information of the 3D point cloud is hidden synchronously when the 3D point cloud is compressed, which is of great significance in the copyright protection of the point cloud data. Experiments verify the effectiveness of the proposed algorithm. The decoding root mean square error (RMSE) of the proposed method is decreased by 79.86% on average compared with the traditional one under the same compression ratio, and the compression ratio of the proposed method is 1.96 times bigger than the traditional one under the similar decoding accuracy.

Survey on Deep Learning-Based Point Cloud Compression

Point clouds are becoming essential in key applications with advances in capture technologies leading to large volumes of data. Compression is thus essential for storage and transmission. In this work, the state of the art for geometry and attribute compression methods with a focus on deep learning based approaches is reviewed. The challenges faced when compressing geometry and attributes are considered, with an analysis of the current approaches to address them, their limitations and the relations between deep learning and traditional ones. Current open questions in point cloud compression, existing solutions and perspectives are identified and discussed. Finally, the link between existing point cloud compression research and research problems to relevant areas of adjacent fields, such as rendering in computer graphics, mesh compression and point cloud quality assessment, is highlighted.

Depth camera based remote three-dimensional reconstruction using incremental point cloud compression

Real-time three-dimensional reconstruction is a fundamental technology applied by robots in unknown environments, especially in remote operating systems. This paper proposes a remote indoor environment reconstruction method based on incremental point cloud compression, which improves the effectiveness of teleoperation-based 3D mapping in the context of hazardous area work environments. First, due to the influence of data bandwidth in remote map building, this paper adds point cloud compression to the keyframe-based building method to form a new reconstruction technique using incremental point cloud compression, so as to realize the mapping of the remote environment with small data transmission. Second, a teleoperation system that integrates a remote robot platform and a local operation platform is constructed. Remote robot equipped with a depth camera can better complete the environmental reconstruction under the guidance of the local operator. Finally, we build the actual scene for 3D construction and perform task planning in the reconstruction model. Experimental results and data prove the practicality of our system.

A Method Based on Curvature and Hierarchical Strategy for Dynamic Point Cloud Compression in Augmented and Virtual Reality System.

As a kind of information-intensive 3D representation, point cloud rapidly develops in immersive applications, which has also sparked new attention in point cloud compression. The most popular dynamic methods ignore the characteristics of point clouds and use an exhaustive neighborhood search, which seriously impacts the encoder’s runtime. Therefore, we propose an improved compression means for dynamic point cloud based on curvature estimation and hierarchical strategy to meet the demands in real-world scenarios. This method includes initial segmentation derived from the similarity between normals, curvature-based hierarchical refining process for iterating, and image generation and video compression technology based on de-redundancy without performance loss. The curvature-based hierarchical refining module divides the voxel point cloud into high-curvature points and low-curvature points and optimizes the initial clusters hierarchically. The experimental results show that our method achieved improved compression performance and faster runtime than traditional video-based dynamic point cloud compression.

Vehicular Cooperative Perception Through Action Branching and Federated Reinforcement Learning

Cooperative perception plays a vital role in extending a vehicle's sensing range beyond its line-of-sight. However, exchanging raw sensory data under limited communication resources is infeasible. Towards enabling an efficient cooperative perception, vehicles need to address the following fundamental question: What sensory data needs to be shared?, at which resolution?, and with which vehicles? To answer this question, in this paper, a novel framework is proposed to allow reinforcement learning (RL)-based vehicular association, resource block (RB) allocation, and content selection of cooperative perception messages (CPMs) by utilizing a quadtree-based point cloud compression mechanism. Furthermore, a federated RL approach is introduced in order to speed up the training process across vehicles. Simulation results show the ability of the RL agents to efficiently learn the vehicles' association, RB allocation, and message content selection while maximizing vehicles' satisfaction in terms of the received sensory information. The results also show that federated RL improves the training process, where better policies can be achieved within the same amount of time compared to the non-federated approach.

Enhancement of 3D Point Cloud Contents Using 2D Image Super Resolution Network

Media technology has been developed to give users a sense of immersion. Recent media using 3D spatial data, such as augmented reality and virtual reality, has attracted attention. A point cloud is a data format that consists of a number of points, and thus can express 3D media using coordinates and color information for each point. Since a point cloud has a larger capacity than 2D images, a technology to compress the point cloud is required, i.e., standardized in the international standard organization MPEG as a video-based point cloud compression (V-PCC). V-PCC decomposes 3D point cloud data into 2D patches along orthogonal directions, and those patches are placed into a 2D image sequence, and then compressed using existing 2D video codecs. However, data loss may occur while converting a 3D point cloud into a 2D image sequence and encoding this sequence using a legacy video codec. This data loss can cause deterioration in the quality of a reconstructed point cloud. This paper proposed a method of enhancing a reconstructed point cloud by applying a super resolution network to the 2D patch image sequence of a 3D point cloud.

Patch Re-Segmentation and Packing for Dynamic Point Cloud Compression via Back-and-Forth Structure

The dynamic point cloud is widely needed in 3D vision related applications such as virtual reality and telepresence. Due to the huge amount of data, a key technology before the effective application is the dynamic point cloud compression. The state-of-the-art dynamic point cloud compression scheme, video-based point cloud compression (V-PCC), generates 2D videos with some uncorrelation due to the patch segmentation and packing process, which will affect the compression efficiency. In this paper, we propose a Packing with Patch Correlation Improvement (PPCI) algorithm to adaptively remove the uncorrelated parts between matched patches in packing for the sake of inter-prediction performance. We first propose a basic unidirectional patch re-segmentation operator to remove the uncorrelated parts of the patches in the current point cloud relative to the patches in its reference point cloud. The removed parts will be formed as new patches and added to the patch collection of the current point cloud. Then we propose a back-and-forth structure, which is a combination of several basic patch re-segmentation operators, to bilaterally remove the uncorrelated parts of matched patches in a back-and-forth (BF) unit. Furthermore, we propose a framework to adaptively decide the best length of each BF unit in a point cloud sequence. Experimental results show that our method achieves noticeable bitrate savings compared with the existing V-PCC packing methods, particularly for sequences with small motion.

LiDAR Point Cloud Compression by Vertically Placed Objects Based on Global Motion Prediction

A point cloud acquired through a Light Detection And Ranging (LiDAR) sensor can be illustrated as a continuous frame with a time axis. Since the frame-by-frame point cloud has a high correlation between frames, a higher compression efficiency can be obtained by using an inter-prediction scheme, and for this purpose, Geometry-based Point Cloud Compression (G-PCC) in the Moving Picture Expert Group (MPEG) opened Inter-Exploratory Model (Inter-EM) which experiments on continuous LiDAR based point cloud frames compression through inter-prediction. The points of the LiDAR based point cloud have two different types of motion: global motion brought about by a vehicle with a LiDAR sensor and local motion generated by an object e.g., a walking person. Thus, Inter-EM consists of a compression structure in terms of both global and local motion, and the Inter-EM’s global motion compensation technology increases the compression efficiency via a single matrix describing the global motion of points. However, this is difficult to predict with a single matrix, which causes imprecise global motion estimation since the objects in a LiDAR-based point cloud show different global motion estimates according to object characteristics such as shape and position. Therefore, this paper proposes a global motion prediction and compensation scheme that considers the characteristics of objects for efficient compression of LiDAR-based point cloud frames. The proposed global motion prediction and compensation scheme achieved maximum gain of −22.0% and average of −9.4% in terms of the Bjontegaard-Delta-rate (BD-rate), and effectively compressed the LiDAR-based sparse point cloud.

Differential Transform for Video-Based Plenoptic Point Cloud Coding.

Point cloud compression has been studied in standard bodies and we are here concerned with the Moving Picture Experts Group video-based point cloud compression (V-PCC) solution. Plenoptic point clouds (PPC) is a novel volumetric data representation wherein points are associated with colors in all viewing directions to improve realism. It is sampled as a number ( Nc ) of attribute colors per point. We propose a new method for the efficient video-based compression of PPC that is backwards compatible with the existing single-color V-PCC decoder. V-PCC generates three image atlases which are encoded using an image/video encoder. We assume there may be a reference color which is to be encoded as the main payload. We generate Nc+3 atlases and we produce Nc differential images against the reference color image. Those difference images are pixel-wise transformed using an Nc -point discrete cosine transform, generating Nc transformed atlases which are encoded, forming the secondary payload. Such secondary information is the plenoptic enhancement to the point cloud. If there is no reference attribute, we skip the differences and use the lowest frequency of the transformed atlases as the main payload. Results are presented that show an unrivaled performance of the proposed method.

Video-Based Point Cloud Compression Artifact Removal

Photo-realistic point cloud capture and transmission are the fundamental enablers for immersive visual communication. The coding process of dynamic point clouds, especially video-based point cloud compression (V-PCC) developed by the MPEG standardization group, is now delivering state-of-the-art performance in compression efficiency. V-PCC is based on the projection of the point cloud patches to 2D planes and encoding the sequence as 2D texture and geometry patch sequences. However, the resulting quantization errors from coding can introduce compression artifacts, which can be very unpleasant for the quality of experience (QoE). In this work, we developed a novel out-of-the-loop point cloud geometry artifact removal solution that can significantly improve reconstruction quality without additional bandwidth cost. Our novel framework consists of a point cloud sampling scheme, an artifact removal network, and an aggregation scheme. The point cloud sampling scheme employs a cube-based neighborhood patch extraction to divide the point cloud into patches. The geometry artifact removal network then processes these patches to obtain artifact-removed patches. The artifact-removed patches are then merged together using an aggregation scheme to obtain the final artifact-removed point cloud. We employ 3D deep convolutional feature learning for geometry artifact removal that jointly recovers both the quantization direction and the quantization noise level by exploiting projection and quantization prior. The simulation results demonstrate that the proposed method is highly effective and can considerably improve the quality of the reconstructed point cloud.

Attribute Artifacts Removal for Geometry-Based Point Cloud Compression.

Geometry-based point cloud compression (G-PCC) can achieve remarkable compression efficiency for point clouds. However, it still leads to serious attribute compression artifacts, especially under low bitrate scenarios. In this paper, we propose a Multi-Scale Graph Attention Network (MS-GAT) to remove the artifacts of point cloud attributes compressed by G-PCC. We first construct a graph based on point cloud geometry coordinates and then use the Chebyshev graph convolutions to extract features of point cloud attributes. Considering that one point may be correlated with points both near and far away from it, we propose a multi-scale scheme to capture the short- and long-range correlations between the current point and its neighboring and distant points. To address the problem that various points may have different degrees of artifacts caused by adaptive quantization, we introduce the quantization step per point as an extra input to the proposed network. We also incorporate a weighted graph attentional layer into the network to pay special attention to the points with more attribute artifacts. To the best of our knowledge, this is the first attribute artifacts removal method for G-PCC. We validate the effectiveness of our method over various point clouds. Objective comparison results show that our proposed method achieves an average of 9.74% BD-rate reduction compared with Predlift and 10.13% BD-rate reduction compared with RAHT. Subjective comparison results present that visual artifacts such as color shifting, blurring, and quantization noise are reduced.

Learning-Based Rate Control for Video-Based Point Cloud Compression.

Due to limited transmission resources and storage capacity, efficient rate control is important in Video-based Point Cloud Compression (V-PCC). In this paper, we propose a learning-based rate control method to improve the rate-distortion (RD) performance of V-PCC. A low-latency synchronous rate control structure is designed to reduce the overhead of pre-coding. The basic unit (BU) parameters are predicted accurately based on our proposed CNN-LSTM neural network, instead of the online updating approach, which can be inaccurate due to low consistency between adjacent 2D frames in V-PCC. When determining the quantization parameters for the BU, a patch-based clipping method is proposed to avoid unnecessary clipping. This approach is able to improve the RD performance and subjective dynamic point cloud quality. Experiments show that our proposed rate control method outperforms present approaches.