Optimizing Camera Setup for In-Home First-person Rendering Mixed Reality Gaming System

Abstract

With the advent of 3D humanoid reconstruction techniques, using a realistic 3D human avatar in a serious game has become popular. This realistic representation in the virtual environment could be achieved using a single low-cost RGB-D camera such as Kinect. However, properly setting up such a camera system for high-quality rendering can be challenging due to the relatively restricted in-home environment. In this paper, we address the challenge of finding optimized camera setup guidelines for an in-home first-person perspective mixed reality (MR) gaming system. We use an MR system with personalized humanoids to simulate the texture reconstruction for a user under a specific camera configuration. Then, a derivative-free optimization is leveraged as a black box approach to search for the optimized camera setup through iterative simulation. We also introduce a novel skeleton-based calibration to address the effects of physically varying the camera's position. For evaluation, two experiments are carried out to evaluate the correctness and effectiveness of the proposed calibration. Furthermore, we conduct a case study using simulation-based optimization for reconstructing lower limb amputees. This work can potentially help locate a proper camera setup for an MR system within the constraints of an in-home environment.