Video and Subdivision based Mesh Coding

ISO/IEC JTC1 SC29, also called MPEG, has been working on a compression standard for dynamic meshes since couple of year now and it has released a Call for Proposals (CfP) for Dynamic Mesh Coding in October 2021. One of the goals of the future standard is to utilize the Visual Volumetric Video-based Coding (V3C) framework, defined in ISO/IEC 23090-5, that is already used for dynamic point cloud compression and volumetric video. In this paper, the authors described their vision of how dynamic mesh compression could be achieved, which corresponds to their technical response to the CfP. The presented objective and subjective results will show that the proposed solution outperforms the anchor in terms of objective metrics and subjective perceived visual quality for low bit rate use cases.

Rate-distortion-optimized predictive compression of dynamic 3D mesh sequences

In this paper, we propose a Multiple Description Coding (MDC) method for reliable transmission of compressed time consistent 3D dynamic meshes. It trades off reconstruction quality for error resilience to provide the best expected reconstruction of 3D mesh sequence at the decoder side. The method is based on partitioning the mesh vertices into two sets and encoding each set independently by a 3D dynamic mesh coder. The encoded independent bitstreams or socalled descriptions are transmitted independently. The 3D dynamic mesh coder is based on predictive coding with spatial and temporal layered decomposition. In addition, the proposed method allows for different redundancy allocations by duplicating a number of encoded spatial layers in both sets. The algorithm is evaluated with redundancy-rate-distortion curves and flexible trade-off between redundancy and side distortions can be achieved.

In this paper, we propose a Multiple Description Coding (MDC) method for reliable transmission of compressed time consistent 3D dynamic meshes. It trades off reconstruction quality for error resilience to provide the best expected reconstruction of 3D mesh sequence at the decoder side. The method is based on partitioning the mesh frames into two sets by temporal subsampling and encoding each set independently by a 3D dynamic mesh coder. The encoded independent bitstreams or so-called descriptions are transmitted independently. The 3D dynamic mesh coder is based on predictive coding with spatial and temporal layered decomposition. In addition, the proposed method allows for different redundancy allocations by including a number of encoded spatial layers of the frames in the other set. The algorithm is evaluated with redundancy-rate-distortion curves and it is shown that, when one of the descriptions is lost, acceptable quality can be achieved with around 50% redundancy.

3D triangle meshes are a common form for representing the geometry of static and dynamic 3D objects. They are employed already in many areas, e.g. e-commerce, video games, online museums, CGI or 3D animated films, etc. Static triangle meshes represent only a piecewise linear approximation of complex 3D objects. As a consequence the approximation error can be unacceptably high unless the number of triangles is sufficiently large. On the other hand a large number of triangles makes these meshes cumbersome to handle and expensive to store or to transmit. Consequently, there exists a demand for techniques for efficient compression of static and dynamic 3D meshes. In this article we start with basics on 3D meshes. Thereafter, we explain the key ideas behind different mesh compression approaches for static and dynamic 3D meshes, and highlight their similarities and differences. Finally, we introduce the upcoming MPEG standard for compression of dynamic 3D meshes, which is referred to as FAMC (Frame-based Animated Mesh Compression), and show comparative compression results.

A Survey on Coding of Static and Dynamic 3D Meshes

The Video-based Dynamic Mesh Coding (V-DMC) standard is a cutting-edge technology for compressing dynamic mesh data. V-DMC realizes parallel and partial decoding by introducing submesh frameworks, where dynamic meshes are segmented and processed independently. However, V-DMC may introduce submesh boundary errors due to misalignments in the existence or coordinates of vertices. Here, submesh boundary errors are defined as the misalignments of vertex coordinates and connectivity between adjacent submesh boundaries. These errors can create holes, thereby degrading both the objective and subjective quality of the decoded dynamic meshes. To minimize boundary errors and enhance coding performance, we propose a two-stage boundary error correction method integrated into V-DMC's preprocessing and encoding/decoding processes. Specifically, the first stage involves rearranging the preprocessing order to minimize boundary errors, and the second stage fills holes using boundary information. Experimental results demonstrate that the proposed method effectively minimizes boundary errors in V-DMC decoded meshes, significantly improving both objective and subjective quality compared to the V-DMC reference software. The D1 BD-Rate gain, indicative of coding efficiency, averages -16.3% for all intra and -25.9% for random access conditions.

Impact of vertex clustering on registration-based 3D dynamic mesh coding

Dynamic point clouds and meshes are used in a wide variety of applications such as gaming, visualization, medicine, and more recently AR/VR/MR. This paper presents two extensions of MPEG-I Video-based Point Cloud Compression (V-PCC) standard to support mesh coding. The extensions are based on Edgebreaker and TFAN mesh connectivity coding algorithms implemented in the Google Draco software and the MPEG SC3DMC software for mesh coding, respectively. Lossless results for the proposed frameworks on top of version 8.0 of the MPEG-I V-PCC test model (TMC2) are presented and compared with Draco for dense meshes.

We propose a complete system to enable progressive coding with quality scalability of the mesh geometry, in MPEG's state-of-the-art Video-based Dynamic Mesh Coding (V-DMC) framework. In particular, we propose an alternative method for encoding the subdivision wavelet coefficients in V-DMC, using a zerotree coding approach that works directly in the native 3D mesh space. This allows us to identify parent-child relationships amongst the wavelet coefficients across different subdivision levels, which can be used to achieve an efficient and versatile coding mechanism. We demonstrate that, given a starting base mesh, a target subdivision surface and a desired maximum number of zerotree passes, our system produces an elegant and visually attractive lossy-to-lossless mesh geometry reconstruction with no further user intervention. Moreover, lossless coefficient encoding with our approach requires nearly the same bitrate as the default displacement coding methods in V-DMC. Yet, our approach provides several quality resolution levels embedded in the same bitstream, while the current V-DMC solutions encode a single quality level only. To the best of our knowledge, this is the first time that a zerotree-based method has been proposed and demonstrated to work for the compression of dynamic time-varying meshes, and the first time that an embedded quality-scalable approach has been used in the V-DMC framework.

Perceptual quality assessment of 3D dynamic meshes: Subjective and objective studies

We present a new approach to dynamic mesh compression, which combines compression with simplification to achieve improved compression results, a natural support for incremental transmission and level of detail. The algorithm allows fast progressive transmission of dynamic 3D content. Our scheme exploits both temporal and spatial coherency of the input data, and is especially efficient for the case of highly detailed dynamic meshes. The algorithm can be seen as an ultimate extension of the clustering and local coordinate frame (LCF)‐based approaches, where each vertex is expressed within its own specific coordinate system. The presented results show that we have achieved better compression efficiency compared to the state of the art methods. Copyright © 2008 John Wiley & Sons, Ltd.

Application of 3D mesh model coding is first presented in this chapter. We then survey the typical existing algorithms in the area of compression of static and dynamic 3D meshes. In an introductory sub-section we introduce basic concepts of 3D mesh models, including data representations, model formats, data acquisitions and 3D display technologies. Furthermore, we introduce several typical 3D mesh formats and give an overview to coding principles of mesh compression algorithms in general, followed by describing the quantitative measures for 3D mesh compression. Then we describe some typical and state-of-the-art algorithms in 3D mesh compression. Compression and streaming of gigantic 3D models are specially introduced. At last, the MPEG4 3D mesh model coding standard is briefed. We conclude this chapter with a discussion providing an overall picture of developments in the mesh coding area and pointing out directions for future research.

In this paper we propose a novel method to create dynamic mesh sequences with fixed connectivity from multiple camera video streams. Fixed connectivity is useful for dynamic mesh coding as mesh connectivity has to be transmitted only once and not frame-wise. The proposed method runs automatically. It deploys mesh parametrizations and Voronoi diagrams from the computer graphics realm as well as 2D optical and 3D scene flow from computer vision. Assuming that a given dynamic mesh sequence does not necessarily have constant connectivity, motion estimation is performed by applying a 3D motion field to a static reconstruction in one frame. Afterwards, mesh connectivity is transferred patch-wise from one frame to the next one using remeshing techniques. The entire method is independent of static mesh resolution, hence supplying a basis for dynamic mesh coding and simplification.

With the evolution of mobile communication networks, three-dimensional graphics applications and services are fast developing fields in mobile graphics and wireless networks. While the data size of 3D meshes is increasing, the available bandwidth of wireless networks and the weak resources of mobile devices have become relatively limited. The transmission of graphics multimedia over wireless networks is a bottleneck problem. In this paper, a novel mobile 3D graphics compression approach for progressive transmission over wireless networks is presented. By improving the progressive coding framework using reverse Butterfly Scheme, a new progressive mesh coding algorithm based on the reverse Modified Loop scheme is proposed. The dense triangle mesh is decomposed into the progressive mesh which consists of a base mesh and a series of displacement values. Then the progressive mesh is compressed by the embedded zerotree mesh coding. The proposed approach allows the data of 3D mesh to be transmitted over wireless network using TCP and UDP protocols respectively. It speeds up 3D graphics browsing on mobile devices as well as optimizes the data size of 3D mesh for progressive transmission.