Adaptive representation of dynamic 3D meshes for low-latency applications

Abstract

Recently, 3D visual representations of highly deformable 3D models, such as dynamic 3D meshes, are becoming popular due to their capability to represent realistically the motion of real-world objects/humans, paving the road for new and more advanced immersive virtual, augmented and mixed reality experiences. However, the real-time streaming of such models introduces increasing challenges related to low cost, low-latency and scalable coding of the acquired information. In view of this, this article proposes an efficient scalable coding mechanism, that decomposes a mesh sequence into spatial and temporal layers that remove a single vertex at each layer. The removed vertices are predicted by performing Laplacian interpolation of the motion vectors. The artifacts that are introduced in low-resolution representations are mitigated using a subspace based normal-vector denoising procedure, that is optimized to support low-latency streaming scenarios using incremental SVD. A novel initialization strategy offers robustness to outliers generated due to local deformations. An extensive evaluation study using several synthetic and scanned dynamic 3D meshes highlights the benefits of the proposed approach in terms of both execution time and reconstruction quality even in very low throughput scenarios of bit-per-vertex-per-frame (bpvf).