A Hybrid CNN–LSTM Network for the Classification of Human Activities Based on Micro-Doppler Radar

Jianping Zhu,Wenbin Ye,Haiquan Chen

doi:10.1109/access.2020.2971064

Jianping Zhu, Wenbin Ye + Show 1 more

Open Access

https://doi.org/10.1109/access.2020.2971064

Copy DOI

Abstract

Many deep learning (DL) models have shown exceptional promise in radar-based human activity recognition (HAR) area. For radar-based HAR, the raw data is generally converted into a 2-D spectrogram by using short-time Fourier transform (STFT). All the existing DL methods treat the spectrogram as an optical image, and thus the corresponding architectures such as 2-D convolutional neural networks (2D-CNNs) are adopted in those methods. These 2-D methods that ignore temporal characteristics ordinarily lead to a complex network with a huge amount of parameters but limited recognition accuracy. In this paper, for the first time, the radar spectrogram is treated as a time sequence with multiple channels. Hence, we propose a DL model composed of 1-D convolutional neural networks (1D-CNNs) and long short-term memory (LSTM). The experiments results show that the proposed model can extract spatio-temporal characteristics of the radar data and thus achieves the best recognition accuracy and relatively low complexity compared to the existing 2D-CNN methods.

Highlights

Human activity recognition (HAR) provides excellent potential for various applications, including personal health systems (PHS), human-computer interaction (HCI), and antiterrorism monitoring [1]–[3]
PROPOSED METHOD Fig. 1 shows the overall network architecture for HAR tasks, which consists of an short-time Fourier transform (STFT) for data preprocessing, 1-D convolutional neural networks (1D-CNNs) for local feature learning, an long shortterm memory (LSTM) layer for global temporal information extraction, and a fully connected layer for classification
After processed by STFT and 1D-CNNs, the feature map can be seen as a 1-D time series with multiple channels, which retains unbroken temporal characteristics

Summary

Introduction

Human activity recognition (HAR) provides excellent potential for various applications, including personal health systems (PHS), human-computer interaction (HCI), and antiterrorism monitoring [1]–[3]. There are generally two types of HAR: video-based HAR and sensor-based HAR [4]. Video-based HAR takes advantages of the videos or images from optical cameras to resolve human motion, whereas sensor-based HAR relies on the data from smart sensors such as a gyroscope, accelerometer, and radars. Sensor-based HAR is becoming more popular and extensively used. Radar-based devices offer unique advantages, such as penetrating opaque objects, adapting to any lighting conditions, and working around the clock [5]. Radarbased HAR methods are attracting growing interests

Methods

Results

Conclusion