Video-based Point Cloud Research Articles

Occupancy Map Guided Attributes Artifacts Removal for Video-based Point Cloud Compression

Point clouds offer realistic three-dimensional (3-D) representations of objects and scenes at the expense of large data volumes. To represent such data compactly in real-world applications, Video-based Point Cloud Compression (V-PCC) converts their texture into two-dimensional (2-D) attributes and occupancy maps before applying lossy video compression. Unfortunately, the coding artifacts introduced in the decoded attribute maps eventually degrade the quality of the reconstructed point cloud, thereby influencing its immersive experience. This paper proposes a deep learning-based attribute map enhancement method that fully leverages the occupancy map's guidance. The design philosophy is that the cross-modality guidance from occupancy can be leveraged as critical information to enhance the attribute. Therefore, instead of treating attribute and occupancy as two separate sources of signals, occupancy serves as an indispensable auxiliary, such that the proposed framework explicitly provides the model with abundant clues by conducting local feature modification and global dependencies aggregation. In particular, the proposed framework is compatible with existing V-PCC bitstreams and can be feasibly incorporated into the standardized decoder pipeline. Extensive evaluations show the effectiveness of the proposed framework in attribute enhancement, with equivalently 6.0% Bjontegaard Delta-rate (BD-rate) savings obtained.

Leveraging occupancy map to accelerate video-based point cloud compression

Video-based Point Cloud Compression enables point cloud streaming over the internet by converting dynamic 3D point clouds to 2D geometry and attribute videos, which are then compressed using 2D video codecs like H.266/VVC. However, the complex encoding process of H.266/VVC, such as the quadtree with nested multi-type tree (QTMT) partition, greatly hinders the practical application of V-PCC. To address this issue, we propose a fast CU partition method dedicated to V-PCC to accelerate the coding process. Specifically, we classify coding units (CUs) of projected images into three categories based on the occupancy map of a point cloud: unoccupied, partially occupied, and fully occupied. Subsequently, we employ either statistic-based rules or machine-learning models to manage the partition of each category. For unoccupied CUs, we terminate the partition directly; for partially occupied CUs with explicit directions, we selectively skip certain partition candidates; for the remaining CUs (partially occupied CUs with complex directions and fully occupied CUs), we train an edge-driven LightGBM model to predict the partition probability of each partition candidate automatically. Only partitions with high probabilities are retained for further Rate–Distortion (R–D) decisions. Comprehensive experiments demonstrate the superior performance of our proposed method: under the V-PCC common test conditions, our method reduces encoding time by 52% and 44% in geometry and attribute, respectively, while incurring only 0.68% (0.66%) BD-Rate loss in D1 (D2) measurements and 0.79% (luma) BD-Rate loss in attribute, significantly surpassing state-of-the-art works.

CC-SMC: Chain coding-based segmentation map lossless compression

A segmentation map, either static or dynamic, refers to a two-dimensional picture that may vary with time and indicates the segmentation label per pixel. Both the semantic map and the occupancy map in video-based point cloud compression (V-PCC) belong to the segmentation map we referred to. The semantic map can work for many machine vision tasks like tracking and has been used as a layer of image representation in some image compression methods. The occupancy map constitutes a part of the point cloud coding bitstream. Since segmentation maps are widely used, how to efficiently compress them is of interest. We propose a segmentation map lossless compression scheme namely CC-SMC, exploiting the nature of segmentation maps that usually contain limited colors and sharp edges. Specifically, we design a chain coding-based scheme combined with quadtree-based block partitioning. For intraframe coding, one block is partitioned recursively with a quadtree structure, until the block contains only one color, is smaller than a threshold, or satisfies the defined chain coding condition. We revise the three-orthogonal chain coding method to incorporate contextual information and design effective intraframe prediction methods. For interframe coding, one block may find a reference block; the chain difference between the current and the reference blocks is coded. We implement the proposed scheme and test it on several different kinds of segmentation maps. Compared with advanced lossless image compression techniques, our proposed scheme obtains more than 10% bits reduction as well as more than 20% decoding time-saving. The code is available at https://github.com/Yang-Runyu/CC-SMC.

Optimized quantization parameter selection for video-based point cloud compression

Point clouds are sets of points used to visualize three-dimensional (3D) objects. Point clouds can be static or dynamic. Each point is characterized by its 3D geometry coordinates and attributes such as color. High-quality visualizations often require millions of points, resulting in large storage and transmission costs, especially for dynamic point clouds. To address this problem, the moving picture experts group has recently developed a compression standard for dynamic point clouds called video-based point cloud compression (V-PCC). The standard generates two-dimensional videos from the geometry and color information of the point cloud sequence. Each video is then compressed with a video coder, which converts each frame into frequency coefficients and quantizes them using a quantization parameter (QP). Traditionally, the QPs are severely constrained. For example, in the low-delay configuration of the V-PCC reference software, the quantization parameter values of all the frames in a group of pictures are set to be equal. We show that the rate-distortion performance can be improved by relaxing this constraint and treating the QP selection problem as a multi-variable constrained combinatorial optimization problem, where the variables are the QPs. To solve the optimization problem, we propose a variant of the differential evolution (DE) algorithm. Differential evolution is an evolutionary algorithm that has been successfully applied to various optimization problems. In DE, an initial population of randomly generated candidate solutions is iteratively improved. At each iteration, mutants are generated from the population. Crossover between a mutant and a parent produces offspring. If the performance of the offspring is better than that of the parent, the offspring replaces the parent. While DE was initially introduced for continuous unconstrained optimization problems, we adapt it for our constrained combinatorial optimization problem. Also, unlike standard DE, we apply individual mutation to each variable. Furthermore, we use a variable crossover rate to balance exploration and exploitation. Experimental results for the low-delay configuration of the V-PCC reference software show that our method can reduce the average bitrate by up to 43% compared to a method that uses the same QP values for all frames and selects them according to an interior point method.

Improved Video-Based Point Cloud Compression via Segmentation.

A point cloud is a representation of objects or scenes utilising unordered points comprising 3D positions and attributes. The ability of point clouds to mimic natural forms has gained significant attention from diverse applied fields, such as virtual reality and augmented reality. However, the point cloud, especially those representing dynamic scenes or objects in motion, must be compressed efficiently due to its huge data volume. The latest video-based point cloud compression (V-PCC) standard for dynamic point clouds divides the 3D point cloud into many patches using computationally expensive normal estimation, segmentation, and refinement. The patches are projected onto a 2D plane to apply existing video coding techniques. This process often results in losing proximity information and some original points. This loss induces artefacts that adversely affect user perception. The proposed method segments dynamic point clouds based on shape similarity and occlusion before patch generation. This segmentation strategy helps maintain the points' proximity and retain more original points by exploiting the density and occlusion of the points. The experimental results establish that the proposed method significantly outperforms the V-PCC standard and other relevant methods regarding rate-distortion performance and subjective quality testing for both geometric and texture data of several benchmark video sequences.

Occupancy-Assisted Attribute Artifact Reduction for Video-Based Point Cloud Compression

Occupancy-Assisted Attribute Artifact Reduction for Video-Based Point Cloud Compression

Fast Video-Based Point Cloud Compression Based on Early Termination and Transformer Model

Fast Video-Based Point Cloud Compression Based on Early Termination and Transformer Model

Cracks-suppression perceptual geometry coding for dynamic point clouds

Dynamic point clouds can effectively describe 3D objects and natural scenes, providing users with an immersive visual experience, but their huge amount of data requires efficient compression tools. To this end, the video-based point cloud compression (V-PCC) standard was developed, however, it may lead to cracks in the compressed point cloud at low bitrates. To solve the problem, this paper proposes a cracks-suppression perceptual geometry coding method for dynamic point cloud. Firstly, considering that the edge regions of 2D geometry patches in V-PCC are prone to cracks, a prior knowledge based crack region detection and preprocessing scheme is designed to reduce cracks. Secondly, considering the unique perceptual geometry characteristics of point cloud, a projected geometry curvature based structural similarity (PGC-SSIM) is proposed to evaluate geometry quality of point clouds, which contains the geometric perception information of point cloud, and is more consistent with the human visual perception. Finally, based on the PGC-SSIM, an adaptive quantization parameter adjustment strategy is designed for rate-distortion optimization in geometry coding of dynamic point clouds. The experimental results show that the proposed method can effectively reduce cracks in the compressed point cloud without reducing the coding performance compared to the V-PCC anchor. Moreover, the presented PGC-SSIM can be used to improve the visual quality of the compressed point cloud.

Video-based point cloud compression artifact removal based on the geometry video enhancement

Video-based point cloud compression artifact removal based on the geometry video enhancement

Occupancy map-based low complexity motion prediction for video-based point cloud compression

This paper proposes an occupancy map-based low complexity motion prediction method for video-based point cloud compression (V-PCC). Firstly, we propose to utilize the occupancy map, direction gradient, and regional dispersion to divide the attribute maps into static, complex, and common blocks. Then, we propose an early termination method for static blocks, an adaptive motion search range method for complex blocks, and an early inter prediction mode decision algorithm for affine motion regions in common blocks. Experiment results show that, in comparison to the test model category2 (TMC2) v15.0, called the anchor method, the average bitrate savings of Y, U, and V components of the proposed method achieve 24.27%, 32.64%, and 31.23% on 8i voxelized full bodies version 2 (8iVFBv2) dataset, respectively. Further, the time savings is 41.97% for attribute maps. Similarly, the proposed method also achieves consistent performance on Microsoft voxelized upper bodies (MVUB) dataset.

Improved Patch Packing and Refining Segmentation for the V-PCC Standard

High-performance coding solutions are urgently needed for the storage and transmission of 3D point clouds due to the development of 3D data acquisition facilities and the increasing scale of acquired point clouds. Video-based point cloud compression (V-PCC) is the most advanced international standard for compressing dynamic point clouds. However, it still has serious issues of time consumption and the large size of the occupancy map. Considering the aforementioned issues, based on V-PCC, we propose the Voxel Selection-based Refining Segmentation (VS-RS), which is used to accelerate the refining segmentation process of the point cloud. Furthermore, the data-adaptive patch packing (DAPP) is proposed to reduce the size of the occupancy map. In order to specify the effect of the improvement, we also designed novel evaluation indicators. Experimental results show that the proposed method achieves a Bjøntegaard Delta rate (BD-rate) gain of −1.58% in the V-PCC benchmark. Additionally, it reduces encoding time by up to 31.86% and reduces the size of the occupancy map by up to 20.14%.

Improved grid refine segmentation for 3D point cloud in video-based point cloud compression (V-PCC)

Improved grid refine segmentation for 3D point cloud in video-based point cloud compression (V-PCC)

Simple Grid-Based Refinement Segmentation Algorithm for MPEG Video-Based Point Cloud Compression

Simple Grid-Based Refinement Segmentation Algorithm for MPEG Video-Based Point Cloud Compression

Enhanced Temporal Consistency for Global Patch Allocation in Video-Based Point Cloud Compression

Enhanced Temporal Consistency for Global Patch Allocation in Video-Based Point Cloud Compression

Low Complexity Coding Unit Decision for Video-Based Point Cloud Compression.

With growing demand for point cloud coding, Video-based Point Cloud Compression (V-PCC) is released for dynamic point clouds, relying on mature 2D video coding techniques. However, the huge computational complexity of 2D video codec is inherited by V-PCC, thereby resulting in a notably time-consuming encoding process for the projection videos. To accelerate the compression, this paper proposes a low complexity coding unit decision algorithm for V-PCC intra coding. First, the 2D sequences (occupancy, geometry, and attribute sequences) are projected from same 3D point could frames in V-PCC. By exploring the strong correlations among them, the cross-projection information is creatively proposed for improving the predication performance of CU partition. Second, considering the disparate coding losses generated by incorrect partitioning decisions of different CUs, we develop a rate-distortion-oriented learning approach aimed at increasing the decision accuracy of the CUs, severely affecting coding performance. Third, to accommodate the particular coding architecture of V-PCC intra configuration, we further devise an overall framework, including targeted feature extraction and partitioning decision for intra and inter coding of geometry and attribute sequences. The final experimental results strongly demonstrate the effectiveness of our proposed algorithm. The time consumption of total projection sequence compression can be reduced by 57.80%, while the coding losses on Geom.BD-TotalRate (D1 and D2) and Attr.BD-TotalRate Luma component are only 0.08%, 0.33%, and 0.16%, respectively, which can be negligible. To the best of our knowledge, the proposed algorithm achieves state-of-the-art performance for accelerating the projection sequence compression in V-PCC All-Intra configuration.

Inter-Frame Compression for Dynamic Point Cloud Geometry Coding.

Efficient point cloud compression is essential for applications like virtual and mixed reality, autonomous driving, and cultural heritage. This paper proposes a deep learning-based inter-frame encoding scheme for dynamic point cloud geometry compression. We propose a lossy geometry compression scheme that predicts the latent representation of the current frame using the previous frame by employing a novel feature space inter-prediction network. The proposed network utilizes sparse convolutions with hierarchical multiscale 3D feature learning to encode the current frame using the previous frame. The proposed method introduces a novel predictor network for motion compensation in the feature domain to map the latent representation of the previous frame to the coordinates of the current frame to predict the current frame's feature embedding. The framework transmits the residual of the predicted features and the actual features by compressing them using a learned probabilistic factorized entropy model. At the receiver, the decoder hierarchically reconstructs the current frame by progressively rescaling the feature embedding. The proposed framework is compared to the state-of-the-art Video-based Point Cloud Compression (V-PCC) and Geometry-based Point Cloud Compression (G-PCC) schemes standardized by the Moving Picture Experts Group (MPEG). The proposed method achieves more than 88% BD-Rate (Bjøntegaard Delta Rate) reduction against G-PCCv20 Octree, more than 56% BD-Rate savings against G-PCCv20 Trisoup, more than 62% BD-Rate reduction against V-PCC intra-frame encoding mode, and more than 52% BD-Rate savings against V-PCC P-frame-based inter-frame encoding mode using HEVC. These significant performance gains are cross-checked and verified in the MPEG working group.

Lightweight fully connected network-based fast CU size decision for video-based point cloud compression

Video-based point cloud compression (V-PCC) utilizes high efficiency video coding (HEVC) to compress geometry and attribute videos generated from dynamic point cloud projection. However, the HEVC exhaustive coding unit (CU) size decision process is complex and hinders the real-time application of V-PCC. To reduce the coding complexity of V-PCC, this paper proposes a method that combined hand-crafted features and lightweight neural network to accurately predict the best CU partition in advance. First, we extract hand-crafted features, including direct features (DFs) and indirect feature (IF), as mixed features. DFs are simple and require no additional calculation, while IF is obtained indirectly by transforming the global and local distortions of the CU extracted before size decision determination. Second, we propose a lightweight fully connected network (LFCN) as the backbone network, two feature types are used as inputs to the LFCN to predict whether the CU should be split into sub-CUs, and the LFCN can be fully integrated into the encoder with only about 1.58KB of additional parameters. Experimental results show that the proposed method reduces coding complexity by an average of 51.2% while Luma’s BD-TotalRate only increases by 0.1% on average under the All Intra (AI) configuration.

Compression Performance Analysis of Experimental Holographic Data Coding Systems.

It is challenging to find a proper way to compress computer-generated holography (CGH) data owing to their huge data requirements and characteristics. This study proposes CGH data coding systems with high-efficiency video coding (HEVC), three-dimensional extensions of HEVC (3D-HEVC), and video-based point cloud compression (V-PCC) codecs. In the proposed system, we implemented a procedure for codec usage and format conversion and evaluated the objective and subjective results to analyze the performance of the three coding systems. We discuss the relative advantages and disadvantages of the three coding systems with respect to their coding efficiency and reconstruction results. Our analysis concluded that 3D-HEVC and V-PCC are potential solutions for compressing red, green, blue, and depth (RGBD)-sourced CGH data.

Subjective Quality Assessment of V-PCC-Compressed Dynamic Point Clouds Degraded by Packet Losses.

This article describes an empirical exploration on the effect of information loss affecting compressed representations of dynamic point clouds on the subjective quality of the reconstructed point clouds. The study involved compressing a set of test dynamic point clouds using the MPEG V-PCC (Video-based Point Cloud Compression) codec at 5 different levels of compression and applying simulated packet losses with three packet loss rates (0.5%, 1% and 2%) to the V-PCC sub-bitstreams prior to decoding and reconstructing the dynamic point clouds. The recovered dynamic point clouds qualities were then assessed by human observers in experiments conducted at two research laboratories in Croatia and Portugal, to collect MOS (Mean Opinion Score) values. These scores were subject to a set of statistical analyses to measure the degree of correlation of the data from the two laboratories, as well as the degree of correlation between the MOS values and a selection of objective quality measures, while taking into account compression level and packet loss rates. The subjective quality measures considered, all of the full-reference type, included point cloud specific measures, as well as others adapted from image and video quality measures. In the case of image-based quality measures, FSIM (Feature Similarity index), MSE (Mean Squared Error), and SSIM (Structural Similarity index) yielded the highest correlation with subjective scores in both laboratories, while PCQM (Point Cloud Quality Metric) showed the highest correlation among all point cloud-specific objective measures. The study showed that even 0.5% packet loss rates reduce the decoded point clouds subjective quality by more than 1 to 1.5 MOS scale units, pointing out the need to adequately protect the bitstreams against losses. The results also showed that the degradations in V-PCC occupancy and geometry sub-bitstreams have significantly higher (negative) impact on decoded point cloud subjective quality than degradations of the attribute sub-bitstream.

V-PCC Projection Based Blind Point Cloud Quality Assessment for Compression Distortion

In immersive media communication with point cloud (PC), PC compression will inevitably cause distortion, which may affect quality of users’ visual experience. To monitor the visual quality of PCs, point cloud quality assessment (PCQA) metrics are highly desired. Inspired by the video-based point cloud compression (V-PCC) standard to project a PC to planes and quantify the corresponding projection maps, a new blind PCQA metric based on V-PCC texture projection map and geometry projection map of PC is proposed for evaluating PC with compression distortion. Specifically, considering the combination of texture and geometry features of the projection maps to characterize distorted PC, a dual-stream convolutional network is designed from the perspective of global and local feature description to extract texture and geometry features of the distorted PC. The network mainly consists of two modules, that is, global texture and geometry fusion (GTGF) module and local geometry weighted local texture (LGWLT) module. The GTGF module is designed to realize the fusion of global texture and geometry features, which can describe PC's combination effect of texture and geometry distortions. Aiming at the relationship between texture and geometry distortions, the LGWLT module is designed to extract the features of local texture distortion caused by local geometry distortion. Furthermore, visual perception is combined to take distortions of the PC's salient regions into account in the process of PCQA. The experimental results on three PC databases show that the proposed PCQA metric is more consistent with visual perception, thus has greater potential than the representative PCQA metrics.