WiFi-based Gesture Recognition Research Articles

HandGest: Hierarchical Sensing for Robust-in-the-Air Handwriting Recognition With Commodity WiFi Devices

Recent advances in wireless sensing techniques have made it possible to recognize hand gestures using channel state information (CSI) in commodity WiFi devices. Existing WiFi-based gesture recognition systems mainly use learning-based pattern recognition methods to recognize different gestures, however, these methods fail to work well when the locations of transceivers, the relative location and orientation of the hand with respect to transceivers, and/or the hand gesturing size change, leading to inconsistent signal patterns caused by those factors. Although some recent efforts have been made to address the so-called “domain-dependent” gesture recognition problem, they either require prior knowledge on initial locations of the hand and WiFi devices or need to train several classifiers for the specific domains. Different from the state-of-the-art methods, we construct two distinct features from a hand-oriented view (rather than from a transceiver’s view), namely, the dynamic phase vector (DPV) and motion rotation variable (MRV), which are quite consistent in characterizing a big set of handwriting gestures, despite significant change in locations of transceivers, the relative location and orientation of the hand with respect to transceivers, and the drawing sizes. We further incorporate a hierarchical sensing framework and develop HandGest—a real-time handwriting gesture recognition system using commodity WiFi devices, to precisely recognize a great number of “in-the-air” handwritings based on the aforementioned two domain-independent features and a pipeline of specific features. Extensive experiments have been done in practical settings with 20 volunteers, evaluation results demonstrate that HandGest outperforms state-of-the-art methods on a large number of handwritings with different transceivers’ location, different initial hand locations and orientations, as well as different drawing sizes. Given its superior performance, we believe that HandGest paves a new way to enhance the real-world practicality of WiFi-based gesture recognition.

WiAdv

WiFi-based gesture recognition systems have attracted enormous interest owing to the non-intrusive of WiFi signals and the wide adoption of WiFi for communication. Despite boosted performance via integrating advanced deep neural network (DNN) classifiers, there lacks sufficient investigation on their security vulnerabilities, which are rooted in the open nature of the wireless medium and the inherent defects (e.g., adversarial attacks) of classifiers. To fill this gap, we aim to study adversarial attacks to DNN-powered WiFi-based gesture recognition to encourage proper countermeasures. We design WiAdv to construct physically realizable adversarial examples to fool these systems. WiAdv features a signal synthesis scheme to craft adversarial signals with desired motion features based on the fundamental principle of WiFi-based gesture recognition, and a black-box attack scheme to handle the inconsistency between the perturbation space and the input space of the classifier caused by the in-between non-differentiable processing modules. We realize and evaluate our attack strategies against a representative state-of-the-art system, Widar3.0 in realistic settings. The experimental results show that the adversarial wireless signals generated by WiAdv achieve over 70% attack success rate on average, and remain robust and effective across different physical settings. Our attack case study and analysis reveal the vulnerability of WiFi-based gesture recognition systems, and we hope WiAdv could help promote the improvement of the relevant systems.

Towards Robust Gesture Recognition by Characterizing the Sensing Quality of WiFi Signals

WiFi-based gesture recognition emerges in recent years and attracts extensive attention from researchers. Recognizing gestures via WiFi signal is feasible because a human gesture introduces a time series of variations to the received raw signal. The major challenge for building a ubiquitous gesture recognition system is that the mapping between each gesture and the series of signal variations is not unique, exact the same gesture but performed at different locations or with different orientations towards the transceivers generates entirely different gesture signals (variations). To remove the location dependency, prior work proposes to use gesture-level location-independent features to characterize the gesture instead of directly matching the signal variation pattern. We observe that gesture-level features cannot fully remove the location dependency since the signal qualities inside each gesture are different and also depends on the location. Therefore, we divide the signal time series of each gesture into segments according to their qualities and propose customized signal processing techniques to handle them separately. To realize this goal, we characterize signal's sensing quality by building a mathematical model that links the gesture signal with the ambient noise, from which we further derive a unique metric i.e., error of dynamic phase index (EDP-index) to quantitatively describe the sensing quality of signal segments of each gesture. We then propose a quality-oriented signal processing framework that maximizes the contribution of the high-quality signal segments and minimizes the impact of low-quality signal segments to improve the performance of gesture recognition applications. We develop a prototype on COTS WiFi devices. The extensive experimental results demonstrate that our system can recognize gestures with an accuracy of more than 94% on average, and significant improvements compared with state-of-arts.

Just‐in‐Time Human Gesture Recognition Using WiFi Signals

In-air gesture recognition using wireless signals acts as a key enabler for various applications including smart homes, remote healthcare, shared autopilot, etc. Although researchers have conducted extensive research on WiFi-based gesture recognition, it remains an open question of providing accurate, robust, and in-time recognition system with the commodity WiFi infrastructure. We present FaSee, a just-in-time WiFi-based gesture recognition system by identifying the fine-grained Channel state information (CSI) features upon off-the-shelf WiFi devices. The core of FaSee is essentially a novel hybrid recognition algorithm, which combines the classical K-Means algorithm with Dynamic time warping (DTW) together, to transform the feature matching in traditional gesture recognition schemes into a hierarchical manner, thereby significantly improving the recognition efficiency. Experimental results show that FaSee recognizes 9 representative gestures with an average accuracy of 94.75% without tedious per-person training, while achieving 30% signal processing delay saving when compared with the state-of-the-arts gesture recognition schemes.

Context-Aware Wireless-Based Cross-Domain Gesture Recognition

Recently, significant efforts have been made to enable WiFi-based gesture recognition. However, models trained with data collected from specific domain suffer from significant performance degradation when applied in a new domain. In practice, various WiFi sensing techniques have provided us with a full knowledge of domain information including discrete variables, i.e., environment and subject, as well as continuous variables, i.e., location and orientation. Previous works haven't fully explored these domain information or need to integrate substantial links' information to use them. Intuitively, we can boost gesture recognition accuracy by accounting for all these domain information with different properties. We propose a new framework not being restricted to link number which combines an adversarial learning scheme with feature disentanglement modules. They together conduct two-stage alignment between each of the source domains and the target domain to eliminate all gesture irrespective information. We also present an attention scheme based on discriminative information of each source and target domain to promote positive transfer from source to target domain. Our model is evaluated on the Widar 3.0 data set and achieves an improvement of 3%-12.7% in cross-domain average accuracy, demonstrating the superiority.

Towards fully domain‐independent gesture recognition using COTS WiFi device

This letter proposes a fully domain-independent WiFi-based gesture recognition system based on the multi-label adversarial network. Unlike pioneer machine learning works, our system does not require retraining in the target domain, which benefits from our key idea of eliminating fully domain information such as orientations, locations, environment information. The system proposed by us includes three parts: Feature extractor, domain discriminator, and gesture recogniser. The feature extractor attempts to deceive the domain discriminator based on the adversarial network structure so that it is difficult to judge the input domain label, thereby obtaining domain-independent features and realising fully domain-independent gesture recognition. Extensive experiments are conducted on the Widar 3.0 dataset and our dataset to evaluate system performance. The results show that our system can achieve an in-domain recognition accuracy of 93.2% and a cross-domain recognition accuracy of 87.1%, which is superior to other state-of-the-art works.

A Novel Gesture Recognition System Based on CSI Extracted from a Smartphone with Nexmon Firmware.

In this paper, we propose a novel gesture recognition system based on a smartphone. Due to the limitation of Channel State Information (CSI) extraction equipment, existing WiFi-based gesture recognition is limited to the microcomputer terminal equipped with Intel 5300 or Atheros 9580 network cards. Therefore, accurate gesture recognition can only be performed in an area relatively fixed to the transceiver link. The new gesture recognition system proposed by us breaks this limitation. First, we use nexmon firmware to obtain 256 CSI subcarriers from the bottom layer of the smartphone in IEEE 802.11ac mode on 80 MHz bandwidth to realize the gesture recognition system’s mobility. Second, we adopt the cross-correlation method to integrate the extracted CSI features in the time and frequency domain to reduce the influence of changes in the smartphone location. Third, we use a new improved DTW algorithm to classify and recognize gestures. We implemented vast experiments to verify the system’s recognition accuracy at different distances in different directions and environments. The results show that the system can effectively improve the recognition accuracy.

WiGAN: A WiFi Based Gesture Recognition System with GANs.

In recent years, a series of research experiments have been conducted on WiFi-based gesture recognition. However, current recognition systems are still facing the challenge of small samples and environmental dependence. To deal with the problem of performance degradation caused by these factors, we propose a WiFi-based gesture recognition system, WiGAN, which uses Generative Adversarial Network (GAN) to extract and generate gesture features. With GAN, WiGAN expands the data capacity to reduce time cost and increase sample diversity. The proposed system extracts and fuses multiple convolutional layer feature maps as gesture features before gesture recognition. After fusing features, Support Vector Machine (SVM) is exploited for human activity classification because of its accuracy and convenience. The key insight of WiGAN is to generate samples and merge multi-grained feature maps in our designed GAN, which not only enhances the data but also allows the neural network to select different grained features for gesture recognition. According to the result of experiments conducted on two existing datasets, the average recognition accuracy of WiGAN reaches 98% and 95.6%, respectively, outperforming the existing system. Moreover, the recognition accuracy under different experimental environments and different users shows the robustness of WiGAN.

Deep transfer learning for gesture recognition with WiFi signals

Different gestures have different action speeds, directions, and trajectories that can cause distinctive effects on the propagation of WiFi signals. In this paper, we present a new approach that uses deep transfer learning techniques to recognize gestures based on the channel state information (CSI) extracted from WiFi signals. Firstly, the CSI streams of gestures are captured and the gesture segments are extracted based on the CSI amplitude changes, and then the WiFi-based gesture recognition problem is innovatively converted to an image classification problem by expressing CSI streams as an image matrix. After that, two deep transfer learning methods are applied to recognize gestures using high-level features extracted by deep convolutional neural network (CNN) and fine-tuned CNN models. We evaluated our method using a collected dataset with 12 gestures in two environments, and the experimental results demonstrated that the proposed method outperformed other state-of-the-art WiFi-based gesture recognition methods.

WiFi based Multi-User Gesture Recognition

WiFi based gesture recognition has received significant attention overthe past few years. However, the key limitation of prior WiFi based gesture recognition systems is that they cannot recognize the gestures of multiple users performing them simultaneously. In this article, we address this limitation and propose WiMU, a WiFi based Multi-User gesture recognition system. The key idea behind WiMU is that when it detects that some users have performed some gestures simultaneously, it first automatically determines the number of simultaneously performed gestures (Na) and then, using the training samples collected from a single user, generates virtual samples for various plausible combinations of Na gestures. The key property of these virtual samples is that the virtual samples for any given combination of gestures are identical to the real samples that would result from real users performing that combination of gestures. WiMU compares the detected sample against these virtual samples and recognizes the simultaneously performed gestures. We implemented and extensively evaluated WiMU using commodity WiFi devices. Our results show that WiMU recognizes 2, 3, 4, 5, 6, 7, and 8 simultaneously performed gestures with accuracies of 95.6, 94.9, 93.9, 92.7, 91.6, 91.0, and 90.1 percent, respectively.

Towards Location Independent Gesture Recognition with Commodity WiFi Devices

Recently, WiFi-based gesture recognition has attracted increasing attention. Due to the sensitivity of WiFi signals to environments, an activity recognition model trained at a specific place can hardly work well for other places. To tackle this challenge, we propose WiHand, a location independent gesture recognition system based on commodity WiFi devices. Leveraging the low rank and sparse decomposition, WiHand separates gesture signal from background information, thus making it resilient to location variation. Extensive evaluations showed that WiHand can achieve an average accuracy of 93% for various locations. In addition, WiHand works well under through the wall scenario.

WiGrus: A Wifi-Based Gesture Recognition System Using Software-Defined Radio

With the proliferation of WiFi devices and infrastructures, the ubiquitous WiFi signals are used to transmit user data. Besides it is also capable of sensing and identifying human gestures. In this paper, we propose a WiFi-based gesture recognition system, namely WiGrus, which solves the problems of user privacy and energy consumption compared with the approaches using wearable sensors and depth cameras. WiGrus leverages the fine-grained Channel State Information (CSI) extracted from WiFi signals to recognize a set of hand gestures. First of all, we utilize timestamps attached to the extracted CSI values to split continuously received WiFi packets into gesture instances. Second, a Principal Component Analysis (PCA)-based method and the first order difference are employed to reduce the noise and mitigate multipath effects caused by the environment changes. Then, massive features are extracted from the processed CSI values to present the intrinsic characteristics of each gesture. Finally, a 2-stage-RF algorithm is proposed to classify the gestures. Our experiments are implemented with a wireless router and a Software Defined Radio (SDR) device, more specifically Universal Software Radio Peripheral (USRP), which are used as WiFi signal transmitter and receiver respectively. The experimental results demonstrate that WiGrus can achieve an average accuracy of 96% in Line-of-sight (LOS) scenario and 92% in Non-Line-of-Sight (NLOS) scenario in the office environment and is robust to the environment changes.

Augmenting User Identification with WiFi Based Gesture Recognition

Over the last few years, researchers have proposed several WiFi based gesture recognition systems that can recognize predefined gestures performed by users at runtime. As most environments are inhabited by multiple users, the true potential of WiFi based gesture recognition can be unleashed only when each user can independently define the actions that the system should take when the user performs a certain predefined gesture. To enable this, a gesture recognition system should not only be able to recognize any given predefined gesture, but should also be able to identify the user that performed it. Unfortunately, none of the prior WiFi based gesture recognition systems can identify the user performing the gesture. In this paper, we propose WiID, a WiFi and gesture based user identification system that can identify the user as soon as he/she performs a predefined gesture at runtime. WiID integrates with the WiFi based gesture recognition systems as an add-on module whose sole objective is to identify the users that perform the predefined gestures. The design of WiID is based on our novel result which states that the timeseries of the frequencies that appear in WiFi channel's frequency response while performing a given gesture are different in the samples of that gesture performed by different users, and are similar in the samples of that gesture performed by the same user. We implemented and extensively evaluated WiID in a variety of environments using a comprehensive data set comprising over 25,000 gesture samples.

WiGeR: WiFi-Based Gesture Recognition System

Recently, researchers around the world have been striving to develop and modernize human–computer interaction systems by exploiting advances in modern communication systems. The priority in this field involves exploiting radio signals so human–computer interaction will require neither special devices nor vision-based technology. In this context, hand gesture recognition is one of the most important issues in human–computer interfaces. In this paper, we present a novel device-free WiFi-based gesture recognition system (WiGeR) by leveraging the fluctuations in the channel state information (CSI) of WiFi signals caused by hand motions. We extract CSI from any common WiFi router and then filter out the noise to obtain the CSI fluctuation trends generated by hand motions. We design a novel and agile segmentation and windowing algorithm based on wavelet analysis and short-time energy to reveal the specific pattern associated with each hand gesture and detect duration of the hand motion. Furthermore, we design a fast dynamic time warping algorithm to classify our system’s proposed hand gestures. We implement and test our system through experiments involving various scenarios. The results show that WiGeR can classify gestures with high accuracy, even in scenarios where the signal passes through multiple walls.