SonicGest: Ultrasonic Gesture Recognition System Combined with GAN on Smartphones

Abstract

In recent years, with the widespread popularity of mobile devices, gesture recognition as a way of human-computer interaction has received more and more attention. However, existing gesture recognition methods have their limitations, such as requiring additional hardware devices, invading user privacy, and causing difficulty in data collection. To address these issues, we propose SonicGest, a recognition system that utilizes acoustic signals to sense in-air gestures. The system only needs the built-in speaker and microphone of the smartphone, without any additional hardware and no privacy disclosure. SonicGest transforms the features of the acoustic Doppler effect caused by gesture movements into a spectrogram, uses spectrogram enhancement techniques to remove noise interference, and then builds a convolutional neural network (CNN) classification model to recognize different gestures. To solve the problem of data collection difficulties, we utilize the Wasserstein distance based on gradient penalty to optimize the loss function of the generative adversarial network (GAN) to generate high-quality spectrograms to expand the dataset. The experimental results show that SonicGest has a recognition accuracy of 98.9% for ten kinds of gestures.