GestureFlow: QoE-Aware Streaming of Multi-TouchGestures in Interactive Multimedia Applications

Abstract

With the proliferation of multi-touch mobile devices, such as smartphones and tablets, users interact with devices in non-conventional gesture-intensive ways. As a new way to interact with mobile devices, gestures have been proven to be intuitive and natural with a minimal learning curve, and can be used in interactive multimedia applications. In order for multiple users to collaborate in an interactive manner, we propose that gestures can be streamed in multiple broadcast sessions, with each session corresponding to one of the users as the source of a gesture stream. During the interactive session, the Quality of Experience (QoE) of mobile users hinges upon delays from when gestures are entered by the source to when they are recognized by each of the receivers, which we refer to as gesture recognizing delays. In this paper, we present the design of GestureFlow, a gesture broadcast protocol designed specifically for concurrent gesture streams in multiple broadcast sessions, such that the gesture recognizing delay in each session is minimized. We motivate the effectiveness and practicality of using inter-session network coding, and address challenges introduced by the linear dependence of coded packets. We evaluate our protocol design using an extensive array of real-world experiments on mobile devices, involving a new gesture-intensive interactive multimedia application, called MusicScore, that we developed from scratch.