Quantization After Inter Prediction in Displacement Coding of Dynamic Meshes

This article presents a new compression scheme for 3D dynamic meshes, referred to as Video and Subdivision based Mesh Coding (VSMC). The VSMC approach combines a displaced subdivision surface model with video-based coding in order to achieve efficient compression performance and real-time, low-power decoding and playback. In addition, VSMC supports a rich set of functionalities including scalability (spatial, temporal, and quality) and progressive transmission. The proposed scheme [1] was shown to outperform the anchor for the MPEG Call for Proposals on Dynamic Mesh coding [2] and was recently selected by the ISO MPEG 3D Graphics Coding group as the basis for the upcoming Video-based Dynamic Mesh Coding standard.

Dynamic point clouds and meshes are used in a wide variety of applications such as gaming, visualization, medicine, and more recently AR/VR/MR. This paper presents two extensions of MPEG-I Video-based Point Cloud Compression (V-PCC) standard to support mesh coding. The extensions are based on Edgebreaker and TFAN mesh connectivity coding algorithms implemented in the Google Draco software and the MPEG SC3DMC software for mesh coding, respectively. Lossless results for the proposed frameworks on top of version 8.0 of the MPEG-I V-PCC test model (TMC2) are presented and compared with Draco for dense meshes.

ISO/IEC JTC1 SC29, also called MPEG, has been working on a compression standard for dynamic meshes since couple of year now and it has released a Call for Proposals (CfP) for Dynamic Mesh Coding in October 2021. One of the goals of the future standard is to utilize the Visual Volumetric Video-based Coding (V3C) framework, defined in ISO/IEC 23090-5, that is already used for dynamic point cloud compression and volumetric video. In this paper, the authors described their vision of how dynamic mesh compression could be achieved, which corresponds to their technical response to the CfP. The presented objective and subjective results will show that the proposed solution outperforms the anchor in terms of objective metrics and subjective perceived visual quality for low bit rate use cases.

Laplacian mesh compression, also known as high‐pass mesh coding, is a popular technique for efficiently storing both static and dynamic triangle meshes that gained further recognition with the advent of perceptual mesh distortion evaluation metrics. Currently, the usual rule of thumb that drives the decision for a mesh compression algorithm is whether or not accuracy in absolute scale is required: Laplacian mesh encoding is chosen when perceptual quality is the main objective, while other techniques provide better results in terms of mechanistic error measures such as mean squared error.In this work, we present a modification of the Laplacian mesh encoding algorithm that preserves its benefits while it substantially reduces the resulting absolute error. Our approach is based on analyzing the reconstruction stage and modifying the quantization of differential coordinates, so that the decoded result stays close to the input even in areas that are distant from anchor points. In our approach, we avoid solving an overdetermined system of linear equations and thus reduce data redundancy, improve conditioning and achieve faster processing. Our approach can be directly applied to both static and dynamic mesh compression and we provide quantitative results comparing our approach with the state of the art methods.

A Survey on Coding of Static and Dynamic 3D Meshes

In this paper, we propose a Multiple Description Coding (MDC) method for reliable transmission of compressed time consistent 3D dynamic meshes. It trades off reconstruction quality for error resilience to provide the best expected reconstruction of 3D mesh sequence at the decoder side. The method is based on partitioning the mesh frames into two sets by temporal subsampling and encoding each set independently by a 3D dynamic mesh coder. The encoded independent bitstreams or so-called descriptions are transmitted independently. The 3D dynamic mesh coder is based on predictive coding with spatial and temporal layered decomposition. In addition, the proposed method allows for different redundancy allocations by including a number of encoded spatial layers of the frames in the other set. The algorithm is evaluated with redundancy-rate-distortion curves and it is shown that, when one of the descriptions is lost, acceptable quality can be achieved with around 50% redundancy.

The Video-based Dynamic Mesh Coding (V-DMC) standard exploits the temporal correlation by tracing the motions of the vertices, which applies only to tracked frames with one-to-one vertex correspondence. For non-tracked frames, only intra mode is applied. This paper proposes an embedded graph representation method that can efficiently represent inter-frame differences for both tracked and non-tracked frames. First, we construct an embedded graph by simplifying the given mesh. Then, we compute a set of affine transformations on graph nodes and use their linear combinations to represent the inter-frame difference. Finally, we apply our implementation to increase the number of predicted frames (P-frames) and thus improve the resulting coding performance. Evaluations on Moving Picture Experts Group (MPEG) test sequences demonstrate the significant rate distortion improvements achieved by our method over V-DMC. The proposed method is highly compliant with V-DMC, and a part of it has been adopted into the V-DMC reference software during the MPEG-3DGH 144th meeting.

Dynamic meshes reasonably represent time-varying 3D objects, but compression is required due to the large amount of data involved. One efficient framework decomposes a dynamic mesh into a base mesh and displacements using decimation and subdivision. The displacements are converted to levels by wavelet transforms and quantization, and they are coded by arithmetic coding. The levels of the current frame are predicted from the reference frame, and only the residuals are coded. However, the residual tends to be large since the coefficients of each frame are quantized before performing inter prediction. In this paper, we propose a method of quantizing the residuals obtained after applying inter prediction in order to reduce the amount of required data. The experimental results show that the proposed method improves coding efficiency (BD-Rate: -0.3 %) and that the reconstructed mesh has no quality degradations.

In this paper, we propose a Multiple Description Coding (MDC) method for reliable transmission of compressed time consistent 3D dynamic meshes. It trades off reconstruction quality for error resilience to provide the best expected reconstruction of 3D mesh sequence at the decoder side. The method is based on partitioning the mesh vertices into two sets and encoding each set independently by a 3D dynamic mesh coder. The encoded independent bitstreams or socalled descriptions are transmitted independently. The 3D dynamic mesh coder is based on predictive coding with spatial and temporal layered decomposition. In addition, the proposed method allows for different redundancy allocations by duplicating a number of encoded spatial layers in both sets. The algorithm is evaluated with redundancy-rate-distortion curves and flexible trade-off between redundancy and side distortions can be achieved.

Rate-distortion-optimized predictive compression of dynamic 3D mesh sequences

In this paper, we present a wavelet-based progressive compression method for 3-D dynamic meshes. Our method exploits the spatial and temporal redundancy. We encode the geometry of base mesh, the wavelet coefficients and the connectivity of each resolution level in order to reduce the spatial redundancy of intra meshes. For inter mesh coding, we encode the differences of geometry of base meshes and of their wavelet coefficients between adjacent frames to reduce the temporal redundancy. Our proposal is based on the wavelet-based multiresolution analysis which uses a perfect reconstruction filter bank and therefore it enables not only progressive representation but also lossless compression. The simulation results demonstrate that the proposed method is applicable to lossy and lossless compression of 3-D dynamic meshes.

We propose a complete system to enable progressive coding with quality scalability of the mesh geometry, in MPEG's state-of-the-art Video-based Dynamic Mesh Coding (V-DMC) framework. In particular, we propose an alternative method for encoding the subdivision wavelet coefficients in V-DMC, using a zerotree coding approach that works directly in the native 3D mesh space. This allows us to identify parent-child relationships amongst the wavelet coefficients across different subdivision levels, which can be used to achieve an efficient and versatile coding mechanism. We demonstrate that, given a starting base mesh, a target subdivision surface and a desired maximum number of zerotree passes, our system produces an elegant and visually attractive lossy-to-lossless mesh geometry reconstruction with no further user intervention. Moreover, lossless coefficient encoding with our approach requires nearly the same bitrate as the default displacement coding methods in V-DMC. Yet, our approach provides several quality resolution levels embedded in the same bitstream, while the current V-DMC solutions encode a single quality level only. To the best of our knowledge, this is the first time that a zerotree-based method has been proposed and demonstrated to work for the compression of dynamic time-varying meshes, and the first time that an embedded quality-scalable approach has been used in the V-DMC framework.

In this paper we propose a novel method to create dynamic mesh sequences with fixed connectivity from multiple camera video streams. Fixed connectivity is useful for dynamic mesh coding as mesh connectivity has to be transmitted only once and not frame-wise. The proposed method runs automatically. It deploys mesh parametrizations and Voronoi diagrams from the computer graphics realm as well as 2D optical and 3D scene flow from computer vision. Assuming that a given dynamic mesh sequence does not necessarily have constant connectivity, motion estimation is performed by applying a 3D motion field to a static reconstruction in one frame. Afterwards, mesh connectivity is transferred patch-wise from one frame to the next one using remeshing techniques. The entire method is independent of static mesh resolution, hence supplying a basis for dynamic mesh coding and simplification.

Recent developments in the compression of dynamic meshes or mesh sequences have shown that the statistical dependencies within a mesh sequence can be exploited well by predictive coding approaches. Coders introduced so far use experimentally determined or heuristic thresholds for tuning the algorithms. In video coding rate-distortion (RD) optimization is often used to avoid fixing of thresholds and to select a coding mode. We applied these ideas and present here an RD-optimized mesh coder. It includes different prediction modes as well as an RD cost computation that controls the mode selection across all possible spatial partitions of a mesh to find the clustering structure together with the associated prediction modes. The structure of the RD-optimized D3DMC coder is presented, followed by comparative results with mesh sequences at different resolutions.

Impact of vertex clustering on registration-based 3D dynamic mesh coding