Formal Visual Evaluation and Study of Objective Metrics for MPEG Dynamic Mesh Coding

Abstract

This paper presents and analyzes the visual assessment process of the MPEG Call for Proposals (CfP) for Dynamic Mesh Coding. The subjective evaluation for this type of data requires an adapted evaluation. In the CfP, a common rendering process of video sequences is applied according to camera paths which are not revealed to the proponents. The formal visual evaluation is performed on the rendered video sequences following the well-established DCR/DSIS protocol. This approach asserts a transparent process, a fair treatment of all proposals, and reproducible MOS results. This is demonstrated by a narrow range of confidence intervals (maximum 0.59) of the resulting MOS. The MOS results are further analyzed with respect to objective metrics used in the context of the CfP as well as objective metrics computed from the rendered video sequences. The results underline the challenges in the relation between objective metrics and MOS. Metrics including texture information are shown to perform better than purely geometry-based approaches. The metrics computed on the rendered video sequences show a generally higher correlation with MOS, VMAF performing best in this category.