Mixed Reality TVs: Applying Motion Parallax for Enhanced Viewing and Control Experiences on Consumer TVs

Abstract

Smart TVs have been popularized in the last years, presenting continued innovation regarding different topics such as connectivity, platform support, and options for interaction. Examples of LG and Samsung present interaction options such as voice and gesture control, showing these emerging technologies that were previously tackled mainly on research investigations, as market alternative solutions for the already established remote control. In this paper, we address the interactivity with Smart TVs by using body gestures combined with the visualization modification through the Motion Parallax effect. We show that by tracking the user viewpoint and hand tips, plus applying the Parallax effect, it is possible to fully calibrate the virtual environment demonstrated by the TV with the real world. Once the calibration is performed, the TV becomes a Mixed Reality display, showing its content coupled with the real world, and allowing the user to perceive this coupling through the depth cue provided by the distortions of the Parallax effect. This calibration enables the use of a set of interaction techniques such as enabling the user to visually pinpoint and touch the virtual items of a TV menu. Moreover, we show that by modeling the interaction with the TV content as a 3D virtual world interaction, it is possible to use new metaphors based on physical interactions such as collisions, lights and shadows and magnetic attractions.