JAVRE: A Joint Asymmetric Video Rendering and Encoding Approach to Enable Optimized Cloud Mobile 3D Virtual Immersive User Experience

Abstract

With the growing adoption of mobile devices as the primary device for information and media consumption, and recent growth in availability of mobile devices with auto-stereoscopic 3D displays, the time has come to enable rich 3D virtual immersive experiences on mobile devices, like 3D display gaming, 3D virtual classroom, 3D virtual art gallery, etc., which are currently enabled on PC platforms. Instead of developing such compute-intensive applications on mobile devices, a scalable and portable approach is to take advantage of the elastic cloud resources. In this paper, we explore the possibility of developing a cloud based mobile 3D virtual immersive application architecture, where the 3D video rendering and encoding are performed on cloud servers, with the resulting 3D video streamed to mobile 3D devices through wireless networks. However, with the significantly higher bit rate requirement for 3D videos, ensuring user experience is a challenge considering the inherent dynamic variability of mobile networks. In this paper, in order to address this challenge, we propose a joint asymmetric graphics rendering and video encoding approach called JAVRE, where both the encoding quality and rendering richness of the left view and the right view can be asymmetric. Specifically, we first develop two mathematical models (user experience model and video bit rate model) through subjective and objective tests. Then we propose an optimization algorithm that can automatically choose the video encoding settings and graphics rendering settings for the left view and the right view to ensure the best user experience given the network conditions. Experiments conducted using real 4G-LTE network profiles and a commercial cloud service demonstrate significant improvement in user experience when JAVRE is used.