Device-free Human Activity Recognition Research Articles

Exploiting channel state information of WiFi signal for human activity detection: an experimental study

Ubiquitous computing aims to seamlessly integrate computing into our daily lives, and requires reliable information on human activities and state for various applications. In this paper, we propose a device-free human activity recognition system that leverages the rich information behind WiFi signals to detect human activities in indoor environments, including walking, sitting, and standing. The key idea of our system is to use the dynamic features of activities, which we carefully examine and analyze through the characteristics of channel state information. We evaluate the impact of location changes on WiFi signal distribution for different activities and design an activity detection system that employs signal processing techniques to extract discriminative features from wireless signals in the frequency and temporal domains. We implement our system on a single off-the-shelf WiFi device connecting to a commercial wireless access point and evaluate it in laboratory and conference room environments. Our experiments demonstrate the feasibility of using WiFi signals for device-free human activity recognition, which could provide a practical and non-intrusive solution for indoor monitoring and ubiquitous computing applications.

Design of a Low-Cost and Device-Free Human Activity Recognition Model for Smart LED Lighting Control

Design of a Low-Cost and Device-Free Human Activity Recognition Model for Smart LED Lighting Control

SecureSense: Defending Adversarial Attack for Secure Device-Free Human Activity Recognition

Deep neural networks have empowered accurate device-free human activity recognition, which has wide applications. Deep models can extract robust features from various sensors and generalize well even in challenging situations such as data-insufficient cases. However, these systems could be vulnerable to input perturbations, i.e. adversarial attacks. We empirically demonstrate that both black-box Gaussian attacks and modern adversarial white-box attacks can render their accuracies to plummet. In this paper, we firstly point out that such phenomenon can bring severe safety hazards to device-free sensing systems, and then propose a novel learning framework, SecureSense, to defend common attacks. SecureSense aims to achieve consistent predictions regardless of whether there exists an attack on its input or not, alleviating the negative effect of distribution perturbation caused by adversarial attacks. Extensive experiments demonstrate that our proposed method can significantly enhance the model robustness of existing deep models, overcoming possible attacks. The results validate that our method works well on wireless human activity recognition and person identification systems. To the best of our knowledge, this is the first work to investigate adversarial attacks and further develop a novel defense framework for wireless human activity recognition in mobile computing research.

CSI-Based Human Continuous Activity Recognition Using GMM–HMM

Recently, device-free human activity recognition has become a research hotspot, and great progress has been made in ubiquitous computing. Among the different kinds of implementations, activity recognition based on WiFi channel state information (CSI) has attracted enormous attention for its superiority compared with conventional approaches. In this article, a device-free human continuous activity recognition system based on WiFi CSI is proposed. First, the CSI phase difference expansion matrix is constructed as a more obvious activity recognition feature, and a method based on threshold combined with labeling is used to achieve continuous activity segmentation. Then, the Gaussian mixture model–hidden Markov model (GMM–HMM) is used to model the CSI feature data of each activity, which is originally used for human 3-D skeleton-based activity recognition. The approach is of great value not only for its high accuracy compared with other classification approaches, such as long short-term memory (LSTM) and convolutional neural network (CNN), but also for its tremendous advantage that a pretty short CSI time series could be used to identify human activities, thus saving computer memory, reducing system calculation time greatly, and improving the error tolerance rate of the segmentation. Experiments on measured activity datasets and methods comparison demonstrate the effectiveness and superiority of the proposed system. The factors affecting system performance, such as the length of the CSI time series, have been discussed in this article.

Device-Free Human Activity Recognition Based on Dual-Channel Transformer Using WiFi Signals

Human activity recognition plays a significant role in smart building applications, healthcare services, and security monitoring. In particular, WiFi-based indoor wireless sensing system becomes increasingly popular due to its noninvasiveness. This work presents the design and implementation of DARMS, a Device-free human Activity Recognition and Monitoring System that can be deployed with low-cost commodity WiFi devices. DARMS is a passive wireless sensing system, and it can accurately distinguish various daily activities without the user wearing any sensor. DARMS makes two key technical contributions. First, an effective signal processing methodology is designed to extract the CSI features both in the time domain and frequency domain. Second, a dual-channel neural network that combines temporal and frequency information is proposed to achieve fine-grained activity recognition. In our experiments, DARMS shows outstanding performance in different indoor environments, with an average accuracy of 96.9% for fall detection and 93.3% for human activity recognition.

RF-Based Human Activity Recognition Using Signal Adapted Convolutional Neural Network

Human activity recognition (HAR) plays a critical role in a wide range of real-world applications, and it is traditionally achieved via wearable sensing. Recently, to avoid the burden and discomfort caused by wearable devices, device-free approaches exploiting radio-frequency (RF) signals arise as a promising alternative for HAR. Most of the latest device-free approaches require training a large deep neural network model in either time or frequency domain, entailing extensive storage to contain the model and intensive computations to infer human activities. Consequently, even with some major advances on device-free HAR, current device-free approaches are still far from practical in real-world scenarios where the computation and storage resources possessed by, for example, edge devices, are limited. To overcome these weaknesses, we introduce HAR-SAnet which is a novel RF-based HAR framework. It adopts an original signal adapted convolutional neural network architecture: instead of feeding the handcraft features of RF signals into a classifier, HAR-SAnet fuses them adaptively from both time and frequency domains to design an end-to-end neural network model. We apply point-wise grouped convolution and depth-wise separable convolutions to confine the model scale and to speed up the inference execution time. The experiment results show that the recognition accuracy of HAR-SAnet substantially outperforms the state-of-the-art algorithms and systems.

Wi-Fi-Based Location-Independent Human Activity Recognition via Meta Learning

Wi-Fi-based device-free human activity recognition has recently become a vital underpinning for various emerging applications, ranging from the Internet of Things (IoT) to Human–Computer Interaction (HCI). Although this technology has been successfully demonstrated for location-dependent sensing, it relies on sufficient data samples for large-scale sensing, which is enormously labor-intensive and time-consuming. However, in real-world applications, location-independent sensing is crucial and indispensable. Therefore, how to alleviate adverse effects on recognition accuracy caused by location variations with the limited dataset is still an open question. To address this concern, we present a location-independent human activity recognition system based on Wi-Fi named WiLiMetaSensing. Specifically, we first leverage a Convolutional Neural Network and Long Short-Term Memory (CNN-LSTM) feature representation method to focus on location-independent characteristics. Then, in order to well transfer the model across different positions with limited data samples, a metric learning-based activity recognition method is proposed. Consequently, not only the generalization ability but also the transferable capability of the model would be significantly promoted. To fully validate the feasibility of the presented approach, extensive experiments have been conducted in an office with 24 testing locations. The evaluation results demonstrate that our method can achieve more than 90% in location-independent human activity recognition accuracy. More importantly, it can adapt well to the data samples with a small number of subcarriers and a low sampling rate.

Low-Cost and Device-Free Human Activity Recognition Based on Hierarchical Learning Model.

Human activity recognition (HAR) has been a vital human–computer interaction service in smart homes. It is still a challenging task due to the diversity and similarity of human actions. In this paper, a novel hierarchical deep learning-based methodology equipped with low-cost sensors is proposed for high-accuracy device-free human activity recognition. ESP8266, as the sensing hardware, was utilized to deploy the WiFi sensor network and collect multi-dimensional received signal strength indicator (RSSI) records. The proposed learning model presents a coarse-to-fine hierarchical classification framework with two-level perception modules. In the coarse-level stage, twelve statistical features of time–frequency domains were extracted from the RSSI measurements filtered by a butterworth low-pass filter, and a support vector machine (SVM) model was employed to quickly recognize the basic human activities by classifying the signal statistical features. In the fine-level stage, the gated recurrent unit (GRU), a representative type of recurrent neural network (RNN), was applied to address issues of the confused recognition of similar activities. The GRU model can realize automatic multi-level feature extraction from the RSSI measurements and accurately discriminate the similar activities. The experimental results show that the proposed approach achieved recognition accuracies of 96.45% and 94.59% for six types of activities in two different environments and performed better compared the traditional pattern-based methods. The proposed hierarchical learning method provides a low-cost sensor-based HAR framework to enhance the recognition accuracy and modeling efficiency.

DeepMV

Recently, significant efforts are made to explore device-free human activity recognition techniques that utilize the information collected by existing indoor wireless infrastructures without the need for the monitored subject to carry a dedicated device. Most of the existing work, however, focuses their attention on the analysis of the signal received by a single device. In practice, there are usually multiple devices "observing" the same subject. Each of these devices can be regarded as an information source and provides us an unique "view" of the observed subject. Intuitively, if we can combine the complementary information carried by the multiple views, we will be able to improve the activity recognition accuracy. Towards this end, we propose DeepMV, a unified multi-view deep learning framework, to learn informative representations of heterogeneous device-free data. DeepMV can combine different views' information weighted by the quality of their data and extract commonness shared across different environments to improve the recognition performance. To evaluate the proposed DeepMV model, we set up a testbed using commercialized WiFi and acoustic devices. Experiment results show that DeepMV can effectively recognize activities and outperform the state-of-the-art human activity recognition methods.

Features extraction and analysis for device-free human activity recognition based on channel statement information in b5G wireless communications

Features extraction and analysis for human activity recognition (HAR) have been studied for decades in the 5th generation (5G) and beyond the 5th generation (B5G) era. Nowadays, with the extensive use of unmanned aerial vehicles (UAVs) in the civil field, integrating wireless signal receivers on UAVs could be a better choice to receive hearable signals more conveniently. In recent years, the HAR system based on CSI based on WiFi radar has received widespread attention due to its low cost and privacy protection property. However, in the existing CSI-based HAR system, there are two disadvantages: (1) The detection threshold is manually set, which limits its adaptability and immediacy in different wireless environments. (2) A sole classifier is used to complete the recognition, resulting in poor robustness and relatively low recognition accuracy. In this paper, we propose a CSI-based device-free HAR (CDHAR) system with WiFi-sensing radar integrated on UAVs to recognize everyday human activities. Firstly, by using machine learning, CDHAR applies kernel density estimation (KDE) to obtain adaptive detection thresholds to complete the extraction of activity duration. Second, we proposed a random subspace classifier ensemble method for classification, which applies the frequency domain feature instead of the time domain feature, and we choose each kind of feature in the same amount. Finally, we prototype CDHAR on commercial WiFi devices and evaluate its performance in both indoor environment and outdoor environments. The experiment results tell that even if experimental scenario varies, the accuracy of activity durations extraction can reach 98% and 99.60% whether in outdoor or indoor environments. According to the extracted data, the recognition accuracy in outdoor and indoor environments can reach 91.2% and 90.2%, respectively. CDHAR ensures high recognition accuracy while improving the adaptability and instantaneity.

Device free human activity and fall recognition using WiFi channel state information (CSI)

Human activity recognition (HAR) is a broad research area. While there exist solutions based on sensors and vision-based technologies, these solutions suffer from considerable limitations. Thus in order to mitigate or avoid these limitations, device free solutions based on radio signals like (home) WiFi, in particular 802.11 are considered. Recently, channel state information (CSI), available in WiFi 802.11n networks have been proposed for fine-grained analysis. We are able to detect human activities like Walk, Sit, Stand, Run (in the sequel, any human activity used for classification is capitalised, i.e. is denoted by its corresponding label. For example, “standing“ is denoted as Stand, the activity “sitting“ is denoted by Sit and so on), etc. in a line-of-sight (LOS) scenario and a non-line-of-sight (N-LOS) scenario within an indoor environment. We propose two algorithms—one using a support vector machine (SVM) for classification and another one using a long short-term memory (LSTM) recurrent neural network. While the former uses sophisticated pre-processing and feature extraction techniques based on wavelet analysis, the latter processes the raw data directly (after denoising). We show that it is possible to characterize activities and/or human body presence with high accuracy and we compare both approaches with regard to accuracy and performance. Furthermore, we extend the experimental setup to detect human falls, too which is a relevant use-case in the context of ambient assisted living (AAL) and show that with the developed algorithms it is possible to detect falls with high accuracy. In addition, we also show that the algorithms can be used to count the number of people in a room based on the CSI-data, which is a first step towards detecting more complex social behavior and activities. Our paper is an extended version of the paper (Damodaran and Schäfer, Device free human activity recognition using wifi channel state information, in: 16th IEEE International Conference on Ubiquitous Intelligence and Computing (UIC 2019), 5th IEEE Smart World Congress, Leicester, vol 16, IEEE, 2019).

Multi-Variations Activity Based Gaits Recognition Using Commodity WiFi

With the prevalence of commercial WiFi devices and the development of Internet of Things (IoT), researchers have extended the usage of WiFi from communication to sensing. Recently, device-free human activity recognition has been applied to support WiFi based remote control and human-computer interaction. However, prior works usually recognize each individual activity by extracting the feature of corresponding WiFi signals, and hence distinguishing differences between human activities. Once the activity has multiple variations of different body parts (such as head, arm and leg), distinguishing such sub-activities is extremely difficult. In this paper, we propose a Multi-variations Activity Recognition (MAR) system to identify multiple-variations of body parts. Our work is based on an observation that the Channel State Information (CSI) is sensitive to the activities of body parts. Firstly, CANDECOMP/ PARAFAC (CP) decomposition and Dynamic Time Warping (DTW) are applied to recognize multi-variations activities. Secondly, we theoretically analyse the uniqueness of CP decomposition. Then, we design specific experiment to verify the reliability and stability of uniqueness. Finally, we apply MAR in gaits recognition of multiple volunteers to evaluate the accuracy performance. The experiment results demonstrate that MAR achieves average 95% accuracy in gaits recognition.

Deep activity recognition in smart buildings with commercial Wi-Fi devices

Activity recognition acts as a key enabler of smart building applications, such as behaviour analysis, health diagnosis and user authentication. However, existing methods either require burdensome equipment, or light and line-of-sight (LOS) working conditions. To address this challenge, we propose DeepAR, a device-free human activity recognition system with prevailing Wi-Fi signals, which circumvents the use of dedicated devices. DeepAR mainly exploits two key techniques to recognise human daily activities. Firstly, a novel principle component extraction method is presented to capture the motion-induced distortions and discard the irrelevant interferences. Secondly, deep feature maps are constructed with time and frequency domain characteristics, and a deep convolutional neural network (CNN) model is further applied to classify the activity labels. DeepAR is implemented with commercial Wi-Fi devices, and the performance is evaluated through extensive experiments. Experiment results show that DeepAR can achieve an average accuracy of 98.6% in a meeting room and 96.4% in a student office.

Temporal-Frequency Attention-Based Human Activity Recognition Using Commercial WiFi Devices

Human activity recognition has been growing for decades in a variety of technological disciplines. However, in the existing WiFi-based human activity recognition systems, there are the following problems: Firstly, in the processing of channel state information (CSI) data, mainly for the removal of noise in the superimposed signal, there is no effective removal of useless multipath signals; Secondly, the data segmentation algorithm based on the empirical threshold requires manual adjustment of the threshold in different environments, resulting in poor robustness and unstable segmentation; Thirdly, simple learning classification is applied without specific design for CSI data structure and sufficiently abstracting information features. In this paper, a device-free human activity recognition system with a temporal-frequency attention mechanism is proposed, which can be deployed on commercial WiFi devices to identify human's daily activities. Firstly, the multipath signal affected by the channel change is extracted by using the difference of the propagation delay of different multipath, thereby eliminating the delay and invalid multipath signals that have undergone multiple reflections and refractions. Secondly, a neural network model based on attention mechanism is proposed, which assigns different weights to different characteristics and sequences by imitating the human brain to dedicate more attention to important information. Then, the long short-term memory (LSTM) model is used to learn the correlation features of different dimensions to realize human activity recognition. Finally, the system performance is evaluated in different environments, and the experimental results show that our syetem holds a better performance in both line-of-sight (LOS) and non-line-of-sight (NLOS) than the existing human activity recognition systems.

Multiple Kernel Semi-Representation Learning With Its Application to Device-Free Human Activity Recognition

In the research of smart buildings, human activity recognition is an important cornerstone for numerous emerging applications. Although several sensing techniques have been proposed for human activity identification, they require either the user instrumentation or additional infrastructure, that are inconvenient, privacy-intrusive, and expensive. To address these issues, a WiFi-enabled device-free human activity recognition system, namely SmartSense, is proposed in this paper. By upgrading commercial WiFi routers with our designed firmware, fine-grained channel state information (CSI) from PHY layer can be directly extracted from the existing WiFi traffic. In this paper, we propose SmartSense, a device-free human activity recognition system that only leverages existing commercial off-the-shelf WiFi routers. By exploiting the prevalence of WiFi infrastructure in buildings, we developed a novel CSI-enabled Internet of Things platform to collect the CSI measurements from regular data frames. To identify different human activities, a novel machine learning tool, namely multiple kernel semi-representation learning (MKSRL) method is established. MKSRL allows the input of expert domain knowledge in a flexible way and conducts automatic and effective multikernel representation learning for the activity recognition task. Each stage of MKSRL is computationally efficient and theoretically guaranteed, and they can be integrated seamlessly within the reproducing kernel framework for the overall information extraction, representation, and fusion. We conducted experiments to comprehensively evaluate the performance of SmartSense in three common indoor environments. Experimental results validate that SmartSense can provide an activity recognition accuracy of 98%, which achieves significant performance gain over the existing methods.

Device-Free People Counting in IoT Environments: New Insights, Results, and Open Challenges

In the last years multiple Internet of Things (IoT) solutions have been developed to detect, track, count, and identify human activity from people that do not carry any device nor participate actively in the detection process. When WiFi radio receivers are employed as sensors for device-free human activity recognition, channel quality measurements are preprocessed in order to extract predictive features toward performing the desired activity recognition via machine learning (ML) models. Despite the variety of predictors in the literature, there is no universally outperforming set of features for all scenarios and applications. However, certain feature combinations could achieve a better average detection performance compared to the use of a thorough feature portfolio. Such predictors are often obtained by feature engineering and selection techniques applied before the learning process. This manuscript elaborates on the feature engineering and selection methodology for counting device-free people by solely resorting to the fluctuation and variation of WiFi signals exchanged by IoT devices. We comprehensively review the feature engineering and ML models employed in the literature from a critical perspective, identifying trends, research niches, and open challenges. Furthermore, we present and provide the community with a new open database with WiFi measurements in several indoor environments (i.e., rooms, corridors, and stairs) where up to five people can be detected. This dataset is used to exhaustively assess the performance of different ML models with and without feature selection, from which insightful conclusions are drawn regarding the predictive potential of different predictors across scenarios of diverse characteristics.

Radios as Sensors

Radio receivers, besides acting as wireless network nodes participating to the Internet of Things (IoT) communication task, may act as opportunistic sensors participating to the IoT sensing task. In particular, a radio receiver is intrinsically an electronic sensor which may be used for device-free human activity recognition. In this paper, we analyze recent results on how the identification of the human body presence and movement can be carried out analyzing the RF signals transmitted by sources of opportunity. The impact of channel bandwidth, transmission mode, carrier frequency, and signal descriptors on the recognition performance is discussed. Moreover, we present a novel crowd counting system and assess the performance considering two different types of signal descriptors. Results prove the effectiveness of the presented crowd counting system and allow to get more insights into the relation among the specific sensed environment, chosen signal descriptors, and classification accuracy.

A Robust and Device-Free System for the Recognition and Classification of Elderly Activities.

Human activity recognition, tracking and classification is an essential trend in assisted living systems that can help support elderly people with their daily activities. Traditional activity recognition approaches depend on vision-based or sensor-based techniques. Nowadays, a novel promising technique has obtained more attention, namely device-free human activity recognition that neither requires the target object to wear or carry a device nor install cameras in a perceived area. The device-free technique for activity recognition uses only the signals of common wireless local area network (WLAN) devices available everywhere. In this paper, we present a novel elderly activities recognition system by leveraging the fluctuation of the wireless signals caused by human motion. We present an efficient method to select the correct data from the Channel State Information (CSI) streams that were neglected in previous approaches. We apply a Principle Component Analysis method that exposes the useful information from raw CSI. Thereafter, Forest Decision (FD) is adopted to classify the proposed activities and has gained a high accuracy rate. Extensive experiments have been conducted in an indoor environment to test the feasibility of the proposed system with a total of five volunteer users. The evaluation shows that the proposed system is applicable and robust to electromagnetic noise.