Radar-Based Air-Writing Gesture Recognition Using a Novel Multistream CNN Approach

Abstract

Hand gestures, being a convenient and natural way of communication, is getting huge attention for human–computer interface designs. Among these gestures, detecting mid-air writing is one of the most promising applications. Existing radar-based solutions often perform the mid-air writing recognition by tracking the hand trajectory using multiple monostatic or bistatic radars. This article presents a multistream convolutional neural network (MS-CNN)-based in-air digits recognition method using a frequency-modulated continuous-wave (FMCW) radar. With one FMCW radar comprising of two receiving channels, a novel three-stream CNN network with range-time, Doppler-time, and angle-time spectrograms as inputs is constructed and the features are fused together in the later stage before making a final recognition. Unlike the traditional CNN, MS-CNN with multiple independent input layers enables the creation of a multidimensional deep-learning model for FMCW radars. Twelve human volunteers were invited to writing the digits from zero to nine in the air in both home and lab environments. The three-stream CNN architecture-based air writing for digits has shown a promising accuracy of 95%. A comparison of the proposed MS-CNN system was made with 45 different variants of CNN and preliminary results shows that MS-CNN outperforms the other traditional CNN architectures for air-writing application. The gestures radar data have also been made available to the research community.