V3C-based Coding of Dynamic Meshes

Motion field coding is an essential technique for exploiting temporal correlations in the ongoing standard of Video-based Dynamic Mesh Coding (V-DMC), which is crucial for efficient dynamic mesh compression. In V-DMC, the motion field comprises a set of motion vectors (MVs) that represent the positional changes of vertices between corresponding decimated meshes (called base meshes in V-DMC) of current and reference frames. This paper presents an enhanced motion field coding approach that effectively reduces the number of MVs required. The proposed method leverages the observation that base meshes often contain duplicate vertices with identical MVs. By integrating these vertices, the method achieves a more compact representation, while distinct MVs for the few duplicate vertices are explicitly coded to preserve mesh quality. Experimental results using MPEG test sequences demonstrate that the proposed approach can decrease the MV data size by up to 20% without compromising the quality of the reconstructed meshes.

Chapter 14 - Coding of dynamic 3D meshes

This paper presents and analyzes the visual assessment process of the MPEG Call for Proposals (CfP) for Dynamic Mesh Coding. The subjective evaluation for this type of data requires an adapted evaluation. In the CfP, a common rendering process of video sequences is applied according to camera paths which are not revealed to the proponents. The formal visual evaluation is performed on the rendered video sequences following the well-established DCR/DSIS protocol. This approach asserts a transparent process, a fair treatment of all proposals, and reproducible MOS results. This is demonstrated by a narrow range of confidence intervals (maximum 0.59) of the resulting MOS. The MOS results are further analyzed with respect to objective metrics used in the context of the CfP as well as objective metrics computed from the rendered video sequences. The results underline the challenges in the relation between objective metrics and MOS. Metrics including texture information are shown to perform better than purely geometry-based approaches. The metrics computed on the rendered video sequences show a generally higher correlation with MOS, VMAF performing best in this category.

Recent developments in the compression of dynamic meshes or mesh sequences have shown that the statistical dependencies within a mesh sequence can be exploited well by predictive coding approaches. Coders introduced so far use experimentally determined or heuristic thresholds for tuning the algorithms. In video coding rate-distortion (RD) optimization is often used to avoid fixing of thresholds and to select a coding mode. We applied these ideas and present here an RD-optimized mesh coder. It includes different prediction modes as well as an RD cost computation that controls the mode selection across all possible spatial partitions of a mesh to find the clustering structure together with the associated prediction modes. The structure of the RD-optimized D3DMC coder is presented, followed by comparative results with mesh sequences at different resolutions.

The growth of computational power of contemporary hardware causes technologies working with 3D-data to expand. Examples of the use of this kind of data can be found in geography or gaming industry. 3D-data may not be only static, but also dynamic.One way of animated 3D-data representation is expressing them by ”dynamic triangle mesh”. This kind of data representation is usually voluminous and needs to be compressed for efficient storage and transmission. In this paper, we are dealing with the influence of vertex clustering on dynamic mesh compression. The mesh is divided into vertex clusters based on the vertex movement similarity and compressed per-partes to achieve higher compression performance. We use Coddyac as a basic compression algorithm and extend it by adding well known clustering algorithms to demonstrate the efficiency of this approach. We also addres the choice of optimal clustering strategy for the Coddyac algorithm.Keywords3D dynamic meshesData compressionComputer animationCoddyacClustering

Development of geometry data compression techniques in the past years has been limited by the lack of a metric with proven correlation with human perception of mesh distortion. Many algorithms have been proposed, but usually the aim has been to minimise mean squared error, or some of its derivatives.In the field of dynamic mesh compression, the situation has changed with the recent proposal of the STED metric, which has been shown to capture the human perception of mesh distortion much better than previous metrics. In this paper we show how existing algorithms can be steered to provide optimal results with respect to this metric, and we propose a novel dynamic mesh compression algorithm, based on trajectory space PCA and Laplacian coordinates, specifically designed to minimise the newly proposed STED error. Our experiments show that using the proposed algorithm, we were able to reduce the required data rate by up to 50% while preserving the introduced STED error.

Laplacian mesh compression, also known as high‐pass mesh coding, is a popular technique for efficiently storing both static and dynamic triangle meshes that gained further recognition with the advent of perceptual mesh distortion evaluation metrics. Currently, the usual rule of thumb that drives the decision for a mesh compression algorithm is whether or not accuracy in absolute scale is required: Laplacian mesh encoding is chosen when perceptual quality is the main objective, while other techniques provide better results in terms of mechanistic error measures such as mean squared error.In this work, we present a modification of the Laplacian mesh encoding algorithm that preserves its benefits while it substantially reduces the resulting absolute error. Our approach is based on analyzing the reconstruction stage and modifying the quantization of differential coordinates, so that the decoded result stays close to the input even in areas that are distant from anchor points. In our approach, we avoid solving an overdetermined system of linear equations and thus reduce data redundancy, improve conditioning and achieve faster processing. Our approach can be directly applied to both static and dynamic mesh compression and we provide quantitative results comparing our approach with the state of the art methods.

We present a new approach to dynamic mesh compression, which combines compression with simplification to achieve improved compression results, a natural support for incremental transmission and level of detail. The algorithm allows fast progressive transmission of dynamic 3D content. Our scheme exploits both temporal and spatial coherency of the input data, and is especially efficient for the case of highly detailed dynamic meshes. The algorithm can be seen as an ultimate extension of the clustering and local coordinate frame (LCF)‐based approaches, where each vertex is expressed within its own specific coordinate system. The presented results show that we have achieved better compression efficiency compared to the state of the art methods. Copyright © 2008 John Wiley & Sons, Ltd.

This article describes the main video coding technologies included in a joint proposal submitted by Qualcomm and Technicolor, in response to a Call for Proposals (CfP) issued by ITU-T SG16 WP3 Q.6 (VCEG) and ISO/IEC JTC1/SC29/WG11 (MPEG) in Oct. 2017. The proposal contains the majority of the tools that have been adopted into the Joint Exploration Model (JEM), developed in the exploratory phase that preceded the CfP. A flexible multi-tree type (MTT) block-partitioning scheme is proposed to extend the quadtree and binary tree (QTBT) based partitioning in JEM by including triple tree (TT) and asymmetric binary tree (ABT) partitions. In addition, several JEM tools in intra and inter prediction, transforms and arithmetic coding are modified, and new tools such as sign prediction and motion compensated padding are proposed. Objective standard dynamic range (SDR) gains of 43.1% and 15.5% in terms of average luma BD-rate improvement have been achieved for the CfP constraint set 1 (random-access configuration) relative to HEVC/H.265 (HM) and JEM anchors, respectively. For the CfP constraint set 2 (low-delay configuration), the average luma BD-rate improvements are 33.7% relative to the HM anchor and 12.7% relative to the JEM anchor. The proposed codec scored highly in both subjective evaluations and objective metrics and was among the best-performing CfP proposals.

The ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group issued a Call for Proposals (CfP) on video compression with capability beyond HEVC in October 2017. The CfP considered three categories of content – Standard Dynamic Range, High Dynamic Range and Wide Colour Gamut (HDR/WCG), and 360° Omni-directional video. As a result of the CfP process, the development of a new video coding standard, named Versatile Video Coding (VVC), was initiated. The goal of this paper is to provide an overview of the CfP responses for the HDR/WCG category. The paper includes a summary of work leading to the development of the CfP, a presentation of the CfP results for the HDR/WCG category, and a description of the specific HDR/WCG technologies submitted to the CfP.

In the context of the development of compression standards for visual media, typically, most decision making relies on the measurement with one or more objective quality metrics. In many cases, a small number of very simple metrics, such as the PSNR or the SSIM, are applied in decision making processes, e.g., in the context of adoption of coding tools in to a draft specification. THis applies to a variety of visual media under consideration, such as classical 2D video or various representations of immersive visual media like dynamic point clouds or meshes. Given the rise of learning-based coding tools and -apparently- competitive end-to-end learned coding schemes, as well as the increasing number of filtering blocks inside or outside of the coding loop of conventional coding schemes, the suitability of such metrics may be questioned. This is due to a potential lack of correlation with mean opinion scores acquired by subjective assessment, especially if specific artifacts, such as temporal consistency, are not well reflected by the metric. This problem can be even more significant for more advanced, potentially learning-based metrics, which may show unexpected behavior if being applied to compression artifacts which have not been known or seen by the time of training the corresponding metric.Advisory Group ISO/IEC SC 29/AG 5 MPEG Visual Quality Assessment is tasked with evaluating and recommending metrics and testing procedures for the use in standardization projects inside the body of MPEG Working Groups developing compression standards for visual media. This webinar presents recent insights in the performance of metrics and subjective assessment methods for a variety of visual media types. The evaluation includes laboratory tests as well as remote and on-site expert viewing sessions which are frequently conducted during MPEG standardization meetings. The results and the performance of such subjective tests are assessed and used to benchmark objective metrics commonly used or considered for application in the development process. Furthermore and outlook is provided to the dataset of compressed video for study of quality metrics (CVQM) which is currently being developed in AG 5 and which includes reconstructed video sequences from a set of conventional and learning-based coding schemes.

The increasing popularity of virtual, augmented, and mixed reality (VR/AR/MR) applications is driving the media industry to explore the creation and delivery of new immersive experiences. One of the trends is volumetric video, which allows users to explore content unconstrained by the traditional two-dimensional window of director’s view.The ISO/IEC joint technical committee 1 subcommittee 29, better known as the Moving Pictures Experts Group (MPEG), has recently finalized a group of standards, under the umbrella of Visual Volumetric Video-based Coding (V3C). These standards aim to efficiently code, store, and transport immersive content with 6 degrees of freedom. The V3C family of standards currently consists of three documents: 1) ISO/IEC 23090-5 defines the generic concepts of volumetric video-based coding and its application to dynamic point cloud data; 2) ISO/IEC 23090-12 specifies another application that enables compression of volumetric video content captured by multiple cameras; and 3) ISO/IEC 23090-10 describes how to store and deliver V3C compressed volumetric video content. Each standard leverages the capabilities of traditional 2D video coding and delivery solutions, allowing for re-use of existing infrastructures which facilitates fast deployment of volumetric video.This article provides an overview of the generic concepts of V3C, as defined in ISO/IEC 23090-5. Furthermore, it describes V3C carriage related functionalities specified in ISO/IEC 23090-10 and offers best practices for the community with respect to storage and delivery of volumetric video.

This article presents a new compression scheme for 3D dynamic meshes, referred to as Video and Subdivision based Mesh Coding (VSMC). The VSMC approach combines a displaced subdivision surface model with video-based coding in order to achieve efficient compression performance and real-time, low-power decoding and playback. In addition, VSMC supports a rich set of functionalities including scalability (spatial, temporal, and quality) and progressive transmission. The proposed scheme [1] was shown to outperform the anchor for the MPEG Call for Proposals on Dynamic Mesh coding [2] and was recently selected by the ISO MPEG 3D Graphics Coding group as the basis for the upcoming Video-based Dynamic Mesh Coding standard.

LiDAR point cloud (LPC) compression is an indispensable component for 3D vision tasks, especially for dynamic point clouds. However, the existing methods based on traditional spatial-temporal attention are immature, causing little improvement in inter-frame feature extraction. In this paper, we propose Diverse Attention-based Point Cloud Compression (DAPCC), an LPC compression entropy model combining aggregation embedding modules for temporal point matching and spatial-temporal attention blocks for dynamic Octree node encoding, which can effectively utilize the change information of dynamic point clouds. Specifically, we first introduce aggregation embedding to match the Octree sequences from two sweeps to establish temporal correlation. To effectively capture the feature details, we further design local and global combined attention for the spatial-temporal information of point clouds which can focus on the whole context. Finally, we organize a symmetric MLP module capable of strengthening vital features. We conduct experiments of static and dynamic compression on both indoor/outdoor point cloud benchmark datasets (<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i>, ScanNet, SemanticKITTI, and MPEG Common Test Conditions (CTC) Category 3 datasets) and downstream applications (<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i>, vehicle detection and semantic segmentation). Compared with the previous state-of-the-art methods, our method achieves up to 14.7% bpp and 45% decoding time savings and adapts to the downstream tasks with almost no impact on performance.

Mesh-based animation, usually represented as dynamic meshes with fixed connectivity, is becoming more and more prevalent in movies, games and other graphics applications nowadays, and there is a growing need to compactly store and rapidly transmit these meshes for practical use, especially for those with high-quality geometric details. In this paper, we explore a novel key-frame based dynamic mesh compression method, wherein we apply pose-similarity with spectral techniques to define piece-wise manifold harmonic bases to reduce spatial-temporal redundancy. We first partition the sequence into several clusters with similar poses, and then decompose the meshes in each cluster into primary poses and geometric details using the manifold harmonic bases derived from the extracted key-frame in that cluster. The primary poses can be characterized as linear combinations of manifold harmonic bases, and the geometric details can be recovered by deformation transfer technique. Thus, we only need a small number of key-frames and a few coefficients for compressing dynamic meshes, which saves a significant amount of storage comparing to traditional methods in which bases are stored explicitly. Furthermore, we apply a second-order linear prediction coding to the harmonic coefficients to further reduce the temporal redundancy. Our extensive experiments and evaluations on various datasets have manifested that our novel method could obtain a high compression ratio while preserving high-fidelity geometry details and guaranteeing limited human perceived distortion rate simultaneously.