micro-Doppler Spectrogram Research Articles

Blind Universal Denoising for Radar Micro-Doppler Spectrograms Using Identical Dual Learning and Reciprocal Adversarial Training

Blind Universal Denoising for Radar Micro-Doppler Spectrograms Using Identical Dual Learning and Reciprocal Adversarial Training

A micro-Doppler spectrogram denoising algorithm for radar human activity recognition

Radar signal recognition based on micro-Doppler spectrogram has been widely used in human action recognition tasks. However, in practical application scenarios, radar signals inevitably have noise, which leads to different degrees of deformation of the spectrogram graph structure and affects the accuracy of subsequent recognition algorithms. In this paper, we present “ACFL”, a novel algorithm for micro-Doppler spectrogram denoising, which aims to reduce the impact of noise on human action recognition. ACFL employs amplitude–frequency two-dimensional clustering and fuzzy logic clustering selection mechanism to remove noise elements from the spectrogram. Moreover, to address the issue of noise leakage or target missing under time-varying noise and action conditions, ACFL adopts spectrogram segmentation based on short-term Rényi entropy. By dividing the spectrogram into intervals with different time–frequency distributions, the dynamic spectrogram denoise over time is achieved. Simulation and measured data experiments demonstrate that the proposed algorithm not only achieves a higher-quality denoised spectrogram but also significantly improves the accuracy of human action recognition under noisy conditions.

HUMAN ACTIVITY DETECTION USING MMWAVE SENSOR

Human Activity Recognition (HAR) is becoming increasingly useful for applications such as well-being monitoring and personalizing smart spaces. Traditional methods for HAR often require wearable devices or cameras. The former is not feasible for every environment and the latter has strong privacy concerns. The mmWave radar has been shown to be a promising alternative as it does not imply the same privacy concerns and does not require the users to have wearable devices. In this paper we have used a low- cost mmWave radar to generate micro-Doppler spectrograms to ultimately classify different activities. For this, multiple classifiers and methods of spectrogram filtering have been examined. Finally, a Time-Distributed Convolutional Neural Network in conjunction with a Bi-Directional Long Short-Term Memory has attained an average accuracy of 99.62% on a dataset of 5 activities, involving 2 participants.

Urban Bird-Drone Classification With Synthetic Micro-Doppler Spectrograms

Urban Bird-Drone Classification With Synthetic Micro-Doppler Spectrograms

Comparative Analysis of Audio Processing Techniques on Doppler Radar Signature of Human Walking Motion Using CNN Models.

Artificial intelligence (AI) radar technology offers several advantages over other technologies, including low cost, privacy assurance, high accuracy, and environmental resilience. One challenge faced by AI radar technology is the high cost of equipment and the lack of radar datasets for deep-learning model training. Moreover, conventional radar signal processing methods have the obstacles of poor resolution or complex computation. Therefore, this paper discusses an innovative approach in the integration of radar technology and machine learning for effective surveillance systems that can surpass the aforementioned limitations. This approach is detailed into three steps: signal acquisition, signal processing, and feature-based classification. A hardware prototype of the signal acquisition circuitry was designed for a Continuous Wave (CW) K-24 GHz frequency band radar sensor. The collected radar motion data was categorized into non-human motion, human walking, and human walking without arm swing. Three signal processing techniques, namely short-time Fourier transform (STFT), mel spectrogram, and mel frequency cepstral coefficients (MFCCs), were employed. The latter two are typically used for audio processing, but in this study, they were proposed to obtain micro-Doppler spectrograms for all motion data. The obtained micro-Doppler spectrograms were then fed to a simplified 2D convolutional neural networks (CNNs) architecture for feature extraction and classification. Additionally, artificial neural networks (ANNs) and 1D CNN models were implemented for comparative analysis on various aspects. The experimental results demonstrated that the 2D CNN model trained on the MFCC feature outperformed the other two methods. The accuracy rate of the object classification models trained on micro-Doppler features was 97.93%, indicating the effectiveness of the proposed approach.

Classification and Discrimination of Birds and Small Drones Using Radar Micro-Doppler Spectrogram Images

This paper investigates the use of micro-Doppler spectrogram signatures of flying targets, such as drones and birds, to aid in their remote classification. Using a custom-designed 10-GHz continuous wave (CW) radar system, measurements from different scenarios on a variety of targets were recorded to create datasets for image classification. Time/velocity spectrograms generated for micro-Doppler analysis of multiple drones and birds were used for target identification and movement classification using TensorFlow. Using support vector machines (SVMs), the results showed an accuracy of about 90% for drone size classification, about 96% for drone vs. bird classification, and about 85% for individual drone and bird distinction between five classes. Different characteristics of target detection were explored, including the landscape and behavior of the target.

Objective Evaluation of Clutter Suppression for Micro-Doppler Spectrograms of Hand Gesture/Sign Language Based on Pseudo-Reference Image

Gesture and sign language recognition technology enables machines to understand the meaning of human hand movements. In human-computer interaction, it is expected that gesture/sign language recognition technologies will overcome equipment size and application environment constraints; in information communication, gesture/sign language recognition technology will assist healthy people in more easily entering the world of deaf people and better understanding and meeting their inner emotional needs; and in patient monitoring, gesture/sign language recognition technologies will detect abnormal behavior in the elderly or patients and reduce possible safety issues. As a result, it has significant implications for both research and broad application. Because the radar sensor can work normally in a wide range of illumination and weather conditions, as well as penetrate the shelter to receive the moving object echo signal and preserve individual privacy, it is becoming increasingly popular in a variety of recognition tasks. A primary step in using radar sensors for gesture/sign language recognition is to suppress clutter to highlight useful motion information. To evaluate the clutter suppression effect, however, an objective metric is required. We present an objective assessment metric based on the pseudo-reference image and an automatic threshold selection method based on Otsu for clutter suppression, as well as subjective and objective experiments demonstrating their effectiveness and universality in gesture/sign language recognition. Notably, our proposed metric can be used for any recognition task that uses micro-Doppler spectrograms as the dataset.

Editorial: RSN Editorial 2023

It is a pleasure to report that the Premium Award for the best paper published in the journal in 2020/21 is made to Samiur Rahman and Duncan Robertson of the University of St Andrews, Scotland, for their paper Classification of drones and birds using convolutional neural networks applied to radar micro-Doppler spectrogram images, which was published in the May 2020 issue, (Vol. 14 No. 5, pp. 653–661). Drones are becoming more and more widespread in both civil and military operations. It is necessary to be able to detect and classify these targets reliably, and radar is one of the most widely used sensors for the purpose. An important part of this involves distinguishing between drones and birds, which is complicated by the fact that birds and drones have comparable radar cross sections and travel at comparable speeds. However, the micro-Doppler modulation due to the flapping of wings of a bird or the rotation of propeller or rotor blades of a drone are significantly different, so provides a means of solving the problem. The paper describes and demonstrates a classification method based on Convolutional Neural Networks (CNNs) to recognise the features of the micro-Doppler signatures. Experimental data was gathered using two FMCW radars, one operating at 24 GHz and one at 94 GHz, at a local falconry. Excellent results were obtained, with classification accuracies better than 94% and in some cases very close to 100%. The paper is judged to be outstanding because it represents a substantial amount of work—both theoretical and experimental, because of its excellent and significant results, and because of the clarity of its writing. We warmly congratulate the authors on this award. The names of the members of the Editorial Board are listed on the journal page of the Wiley Online Library https://ietresearch.onlinelibrary.wiley.com/journal/17518792; the members are distinguished researchers who represent the full scope of the journal, and they come from all over the world. They help with reaching decisions on submitted papers, encouraging authors to submit their work to the journal, and proposing topics for Special Issues and Special Sections. Several have now reached the end of their term of appointment, and we thank them for their service over many years. We have also appointed a number of new members, and we welcome them to their task and look forward to their inputs. Finally, we express our gratitude to the reviewers for their time, expertise and hard work in refereeing papers and maintaining the reputation of the journal, both for quality and for rapid publication. This is an essential part of the publication process, but necessarily the reviewers must remain anonymous. This editorial is one place where we can acknowledge their contribution and record our thanks to them. Data sharing not applicable to this article as no datasets were generated or analysed during the current study.

Hand Gesture Recognition Using Radial and Transversal Dual Micromotion Features

Most of the work in hand gesture recognition (HGR) focuses on developing diverse classification algorithms based on micro-Doppler (mD) spectrogram, that is 1-D motion along the radial direction. In this work, we exert effort on the radar system and preprocessing methods to extract 2-D motions for HGR. Specifically, we utilize an interferometric radar with two widely spaced receivers to obtain both radial and transversal micromotion features of hand gestures. In the preprocessing stage, as pre-interferometry is nonlinear multiplication in time domain, both the increased noise level and unwanted cross-terms may reduce its usefulness for HGR. To solve these problems, we propose a post-interferometric preprocessing method in frequency domain, which is capable of reducing noise level of the obtained spectrogram and suppressing the nuisance cross-terms. We measure four pairs of symmetric hand gestures from three persons and compare the HGR accuracy using different preprocessing methods. Experimental results show that the mD processing combined with post-interferometry give the highest HGR accuracy of over 99%.

MmGesture: Semi-supervised gesture recognition system using mmWave radar

Gesture recognition has found versatile applications in natural human–computer interaction (HCI). Compared with traditional camera-based or wearable sensors-based solutions, gesture recognition using the millimeter wave (mmWave) radar has attracted growing attention for its characteristics of contact-free, privacy-preserving and less environment-dependence. Recently, most of studies adopted one of the Range Doppler Image (RDI), Range Angle Image (RAI), Doppler Angle Image (DAI) or Micro-Doppler Spectrogram extracted from the raw radar signal as the input of a deep neural network to realize gesture recognition. However, the effectiveness of these four inputs in gesture recognition has attracted little attention so far. Moreover, the lack of large amounts of labeled data restricts the performance of traditional supervised learning network. In this paper, we first conducted extensive experiments to compare the effectiveness of these four inputs in the gesture recognition, respectively. Then we proposed a semi-supervised leaning framework by utilizing few labeled data in the source domain and large amounts of unlabeled data in the target domain. Specially, we combine the ∏-model and some specific data augmentation tricks on the mmWave signal to realize the domain-independent gesture recognition. Extensive experiments on a public mmWave gesture dataset demonstrate the superior effectiveness of the proposed system.

Patient behavior detection based on time-frequency fusion of FMCW radar

This paper proposes a novel time-frequency feature fusion method to recognise patients’ behaviours based on the Frequency Modulated Continuous Wave (FMCW) radar system, which can locate patients as well as recognise their current actions and thus is expected to solve the shortage of medical staff caused by the novel coronavirus pneumonia (COVID-19). To recognise the patient’s behaviour, the FMCW radar is utilised to acquire point clouds reflected by the human body, and the micro-Doppler spectrogram is generated by human motion. Then features are extracted and fused from the time-domain information of point clouds and the frequency-domain information of the micro-Doppler spectrogram respectively. According to the fused features, the patient’s behaviour is recognised by a Bayesian optimisation random forest algorithm, where the role of Bayesian optimisation is to select the best hyper-parameters for the random forest, i.e. the number of random forest decision trees, the depth of leaves, and the number of features. The experimental results show that an average accuracy of 99.3% can be achieved by using the time-frequency fusion with the Bayesian optimisation random forest model to recognise six actions.

Towards a Low-Cost Solution for Gait Analysis Using Millimeter Wave Sensor and Machine Learning.

Human Activity Recognition (HAR) that includes gait analysis may be useful for various rehabilitation and telemonitoring applications. Current gait analysis methods, such as wearables or cameras, have privacy and operational constraints, especially when used with older adults. Millimeter-Wave (MMW) radar is a promising solution for gait applications because of its low-cost, better privacy, and resilience to ambient light and climate conditions. This paper presents a novel human gait analysis method that combines the micro-Doppler spectrogram and skeletal pose estimation using MMW radar for HAR. In our approach, we used the Texas Instruments IWR6843ISK-ODS MMW radar to obtain the micro-Doppler spectrogram and point clouds for 19 human joints. We developed a multilayer Convolutional Neural Network (CNN) to recognize and classify five different gait patterns with an accuracy of 95.7 to 98.8% using MMW radar data. During training of the CNN algorithm, we used the extracted 3D coordinates of 25 joints using the Kinect V2 sensor and compared them with the point clouds data to improve the estimation. Finally, we performed a real-time simulation to observe the point cloud behavior for different activities and validated our system against the ground truth values. The proposed method demonstrates the ability to distinguish between different human activities to obtain clinically relevant gait information.

Cone-Shaped Space Target Inertia Characteristics Identification by Deep Learning With Compressed Dataset

An effective method for identifying inertia characteristics of cone-shaped space target based on deep learning is proposed. The inertia ratio is determined by the time-varying scattering fields from the cone-shaped targets. The multistatic method is introduced to reduce the evaluation time of time-varying scattering fields. The micro-Doppler spectrogram (MDS) dataset is constructed by the time–frequency analysis with numerical simulation method, point scattering model, and experimental tests. The compressed dataset is further achieved by truncated singular value decomposition (SVD). Finally, the micromotion parameter identification model is constructed to identify the inertia ratio for the cone-shaped space target. The interaction loss function and the feedforward denoising convolutional neural networks (DnCNNs) are employed to improve the identification accuracy. Parameters identification of the precession frequency, precession angle, spin frequency, and inertia ratio with both simulation and experiment datasets demonstrate the validity of the proposed method.

Dop-DenseNet: Densely Convolutional Neural Network-Based Gesture Recognition Using a Micro-Doppler Radar

Hand gesture recognition is an efficient and practical solution for the non-contact human–machine interaction in smart devices. To date, vision-based methods are widely used in this research area, but they are susceptible to light conditions. To address this issue, radar-based gesture recognition using micro-Doppler signatures can be applied as an alternative. Accordingly, the use of a novel densely convolutional neural network model, Dop-DenseNet, is proposed in this paper for improving hand gesture recognition in terms of classification accuracy and latency. The model was designed with cross or skip connections in a dense architecture so that the former features, which can be lost in the forward-propagation process, can be reused. We evaluated our model with different numbers of filter channels and experimented with it using the Dop-Net dataset, with different time lengths of input data. As a result, it was found that the model with 64 3 × 3 filters and 200 time bins of micro-Doppler spectrogram data could achieve the best performance trade-off, with 99.87% classification accuracy and 3.1 ms latency. In comparison, our model remarkably outperformed the selected state-of-the-art neural networks (GoogLeNet, Res- Net-50, NasNet-Mobile, and MobileNet-V2) using the same Dop-Net dataset.

Human Activity Recognition Based on WRGAN-GP-Synthesized Micro-Doppler Spectrograms

In recent years, deep convolutional neural networks (DCNNs) have demonstrated the prominent performance in the radar-based human activity recognition. However, collecting and labeling the radar data are usually expensive and time-consuming. It is very challenging to obtain sufficient training data in practice. The DCNNs often suffer from the overfitting problem due to the lack of data and cannot effectively exert their performance. A large number of synthetic micro-Doppler spectrograms similar to the real micro-Doppler spectrograms can be produced by generation adversarial networks (GANs) to increase the training dataset. As a fact of matter, the quality and diversity of synthetic spectrograms are particularly important for the training process of DCNNs. In this paper, we propose a more stable and effective Wasserstein refined generative adversarial network with gradient penalty (WRGAN-GP) to generate the spectrograms for expanding the training dataset. The DCNN is trained to recognize human activities through the new training data composed of real spectrograms and the augmented synthetic spectrograms. The experimental results show that compared with existing GANs-based spectrogram augmentation methods, the proposed WRGAN-GP method provides higher stability and accuracy in generating the augmented spectrograms. Accordingly, the classification accuracy of human activities can also be greatly increased.

Distributed radar fusion and recurrent networks for classification of continuous human activities

Continuous Human Activity Recognition (HAR) in arbitrary directions is investigated in this paper using a network of five spatially distributed pulsed Ultra-Wideband radars. While activities performed continuously and in unconstrained trajectories provide a more realistic and natural scenario for HAR, the network of radar sensors is proposed to address the issue of unfavourable or occluded perspectives when using only a single sensor. Different techniques to combine the relevant information from the multiple radars in the network are investigated, focussing on signal level fusion directly applied on Range-Time maps, and the selection of radar nodes based on location and velocity of the target derived from multilateration processing and tracking. Recurrent Neural Networks with and without bidirectionality are used to classify the activities based on the micro-Doppler (μD) spectrograms obtained for sensor fusion techniques. To assess classification performances, novel evaluation metrics accounting for the continuous nature of the sequence of activities and inherent imbalances in the dataset are proposed and compared with existing metrics. It is shown that the conventional accuracy metric may not capture all the important aspects for continuous HAR, and the proposed metrics can be considered for a more comprehensive evaluation.

Radar-Based Human Activity Recognition Using Hyperdimensional Computing

Aiming at realizing on-chip real-time human activity recognition based on the miniaturized and integrated radar with limited computational resources, hyperdimensional computing (HDC) is a promising method due to its unique advantages including high energy efficiency, excellent robustness, lightweight model, and superfast learning process. This article reports on the first use of HDC in the radar system to recognize human activities. First, we design an image mask method to minimize the impact of interference in original radar micro-Doppler spectrograms and apply a fast envelope extraction method to fully reflect features of different human activities. Subsequently, we select the most applicable representation methods for HDC to encode and recognize these features. To better accelerate the convergence speed, enable subtle adjustments of the HDC model, and improve the recognition accuracy, we propose a new adaptive retraining method, Mask-Retraining. Finally, the accuracy of recognizing six human activities using HDC with Mask-Retraining reaches 92.93%. Experimental results and comparisons between HDC and ten recognition algorithms demonstrate the validity of our methods. In most cases, HDC outperforms other algorithms being compared both in recognition accuracy and computational cost. Especially, HDC becomes the prime and irreplaceable candidate when only small datasets or finite data points are available.

SimHumalator: An Open-Source End-to-End Radar Simulator for Human Activity Recognition

Radio-frequency-based noncooperative monitoring of humans has numerous applications ranging from law enforcement to ubiquitous sensing applications such as ambient assisted living and biomedical applications for nonintrusively monitoring patients. Large training datasets, almost unlimited memory capacity, and ever-increasing processing speeds of computers could drive forward the data-driven deep-learning-focused research in the abovementioned applications. However, generating and labeling large volumes of high-quality, diverse radar datasets is an onerous task. Furthermore, unlike the fields of vision and image processing, the radar community has limited access to databases that contain large volumes of experimental data. Therefore, in this article, we present an open-source motion capture data-driven simulation tool, SimHumalator, that can generate large volumes of human micro-Doppler radar data in passive WiFi scenarios. The simulator integrates IEEE 802.11 WiFi Standards (IEEE 802.11 g, n, and ad) compliant transmissions with the human animation data to generate the micro-Doppler features that incorporate the diversity of human motion characteristics and the sensor parameters. The simulated signatures have been validated with experimental data gathered using an in-house-built hardware prototype. This article describes simulation methodology in detail and provides case studies on the feasibility of using simulated micro-Doppler spectrograms for data augmentation tasks.

FMNet: Latent Feature-Wise Mapping Network for Cleaning Up Noisy Micro-Doppler Spectrogram

Micro-Doppler signatures contain considerable information about target dynamics. However, the radar sensing systems are easily affected by noisy surroundings, resulting in uninterpretable motion patterns on the micro-Doppler spectrogram. Meanwhile, radar returns often suffer from multipath, clutter and interference. These issues lead to difficulty in, for example motion feature extraction, activity classification using micro Doppler signatures ($\mu$-DS), etc. In this paper, we propose a latent feature-wise mapping strategy, called Feature Mapping Network (FMNet), to transform measured spectrograms so that they more closely resemble the output from a simulation under the same conditions. Based on measured spectrogram and the matched simulated data, our framework contains three parts: an Encoder which is used to extract latent representations/features, a Decoder outputs reconstructed spectrogram according to the latent features, and a Discriminator minimizes the distance of latent features of measured and simulated data. We demonstrate the FMNet with six activities data and two experimental scenarios, and final results show strong enhanced patterns and can keep actual motion information to the greatest extent. On the other hand, we also propose a novel idea which trains a classifier with only simulated data and predicts new measured samples after cleaning them up with the FMNet. From final classification results, we can see significant improvements.

Semisupervised Human Activity Recognition With Radar Micro-Doppler Signatures

Human activity recognition (HAR) plays a vital role in many applications, such as surveillance, in-home monitoring, and health care. Portable radar sensor has been increasingly used in HAR systems in combination with deep learning (DL). However, it is both difficult and time-consuming to obtain a large-scale radar dataset with reliable labels. Insufficient labeled data often limit the generalization of DL models. As a result, the performance of DL models will drop when being applied to a new scenario. In this sense, only labeling a small portion of data in the large-scale radar dataset is more feasible. In this article, we propose a semisupervised transfer learning (TL) algorithm, “ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">joint domain and semantic transfer learning</i> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">JDS-TL</i> ),” for radar-based HAR, which is composed of two modules: unsupervised domain adaptation (DA) and supervised semantic transfer. By employing a sparsely labeled dataset to train the HAR model, the proposed method alleviates the need of labeling a significantly large number of radar signals. We adopt a public radar micro-Doppler spectrogram dataset including six human activities to evaluate <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">JDS-TL</i> . Experiments show that the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">JDS-TL</i> is able to recognize the six activities with an average accuracy of 87.6% when there are only 10% instances labeled in the training dataset. Ablation analysis also demonstrates the efficiency of the DA and the semantic transfer modules.