Real-Time Continuous Detection and Recognition of Subject-Specific Smart TV Gestures via Fusion of Depth and Inertial Sensing

Abstract

This paper presents a real-time detection and recognition approach to identify actions of interest involved in the smart TV application from continuous action streams via simultaneous utilization of a depth camera and a wearable inertial sensor. Continuous action streams mean when actions of interest are performed continuously and randomly among arbitrary actions of non-interest. The developed approach consists of a detection part and a recognition part. In the detection part, two support vector data descriptor classifiers corresponding to the two sensing modalities are used to separate actions of interest from actions of non-interest in continuous action streams. The actions detected as actions of interest by both of the sensing modalities are then passed to the recognition part. In this part, actions of interest are classified by fusing the decisions from two collaborative representation classifiers, one classifier using skeleton joint positions and the other classifier using inertial signals. The developed approach is applied to the hand gestures in the smart TV application. The experimental results obtained indicate the effectiveness of the developed approach to detect and recognize smart TV gestures in continuous action streams.