Radar Sensor Data Research Articles

Learning Omni-Dimensional Spatio-Temporal Dependencies for Millimeter-Wave Radar Perception

Reliable environmental perception capabilities are a prerequisite for achieving autonomous driving. Cameras and LiDAR are sensitive to illumination and weather conditions, while millimeter-wave radar avoids these issues. Existing models rely heavily on image-based approaches, which may not be able to fully characterize radar sensor data or efficiently further utilize them for perception tasks. This paper rethinks the approach to modeling radar signals and proposes a novel U-shaped multilayer perceptron network (U-MLPNet) that aims to enhance the learning of omni-dimensional spatio-temporal dependencies. Our method involves innovative signal processing techniques, including a 3D CNN for spatio-temporal feature extraction and an encoder–decoder framework with cross-shaped receptive fields specifically designed to capture the sparse and non-uniform characteristics of radar signals. We conducted extensive experiments using a diverse dataset of urban driving scenarios to characterize the sensor’s performance in multi-view semantic segmentation and object detection tasks. Experiments showed that U-MLPNet achieves competitive performance against state-of-the-art (SOTA) methods, improving the mAP by 3.0% and mDice by 2.7% in RD segmentation and AR and AP by 1.77% and 2.03%, respectively, in object detection. These improvements signify an advancement in radar-based perception for autonomous vehicles, potentially enhancing their reliability and safety across diverse driving conditions.

Introducing the double validation metric for radar sensor models

In automated vehicles, environment perception is performed by various sensor types, such as cameras, radars, lidars, and ultrasonics. Simulation models of these sensors, as required in virtual validation methods, are available in various degrees of detail. However, proving the validity of such models is a subject of research. New metrics and methods for credibility assessment of simulation are needed to standardize the validation process in the future. The so-called double validation metric (DVM) has shown advantages and allows an intuitive interpretability of the validation results. The DVM has so far only been applied to lidar sensor models. In this paper, an extension to the DVM is introduced, which is called the DVM Map. A static measurement scenario is conducted in reality and transferred into simulation. The novel method is demonstrated on the obtained real and simulated radar sensor data. In this simple scenario special focus is put on the position accuracy of GNSS reference sensors. Therefore, their impact on the result of sensor model validation is discussed. The paper shows that the method provides a more detailed and accurate validation in comparison to the state of the art of a radar simulation, revealing previously undetected simulation errors. Errors due to the environment model, signal propagation, and signal processing are separated and satellite imagery is used for intuitive visualization of the results. This method is a complementary tool to existing validation techniques to improve the interpretability and judging the trustworthiness of radar simulations.

Human Activity Recognition Based on Deep Learning and Micro-Doppler Radar Data.

Activity recognition is one of the significant technologies accompanying the development of the Internet of Things (IoT). It can help in recording daily life activities or reporting emergencies, thus improving the user's quality of life and safety, and even easing the workload of caregivers. This study proposes a human activity recognition (HAR) system based on activity data obtained via the micro-Doppler effect, combining a two-stream one-dimensional convolutional neural network (1D-CNN) with a bidirectional gated recurrent unit (BiGRU). Initially, radar sensor data are used to generate information related to time and frequency responses using short-time Fourier transform (STFT). Subsequently, the magnitudes and phase values are calculated and fed into the 1D-CNN and Bi-GRU models to extract spatial and temporal features for subsequent model training and activity recognition. Additionally, we propose a simple cross-channel operation (CCO) to facilitate the exchange of magnitude and phase features between parallel convolutional layers. An open dataset collected through radar, named Rad-HAR, is employed for model training and performance evaluation. Experimental results demonstrate that the proposed 1D-CNN+CCO-BiGRU model demonstrated superior performance, achieving an impressive accuracy rate of 98.2%. This outperformance of existing systems with the radar sensor underscores the proposed model's potential applicability in real-world scenarios, marking a significant advancement in the field of HAR within the IoT framework.

A Multi-Sensor Approach to Characterize Winter Water-Level Drawdown Patterns in Lakes

Artificial manipulation of lake water levels through practices like winter water-level drawdown (WD) is prevalent across many regions, but the spatiotemporal patterns are not well documented due to limited in situ monitoring. Multi-sensor satellite remote sensing provides an opportunity to map and analyze drawdown frequency and metrics (timing, magnitude, duration) at broad scales. This study developed a cloud computing framework to process time series of synthetic aperture radar (Sentinel 1-SAR) and optical sensor (Landsat 8, Sentinel 2) data to characterize WD in 166 lakes across Massachusetts, USA, during 2016–2021. Comparisons with in situ logger data showed that the Sentinel 1-derived surface water area captured relative water-level fluctuations indicative of WD. A machine learning approach classified lakes as WD versus non-WD based on seasonal water-level fluctuations derived from Sentinel 1-SAR data. The framework mapped WD lakes statewide, revealing prevalence throughout Massachusetts with interannual variability. Results showed WDs occurred in over 75% of lakes during the study period, with high interannual variability in the number of lakes conducting WD. Mean WD magnitude was highest in the wettest year (2018) but % lake area exposure did not show any association with precipitation and varied between 8% to 12% over the 5-year period. WD start date was later and duration was longer in wet years, indicating climate mediation of WD implementation driven by management decisions. The data and tools developed provide an objective information resource to evaluate ecological impacts and guide management of this prevalent but understudied phenomenon. Overall, the results and interactive web tool developed as part of this study provide new hydrologic intelligence to inform water management and policies related to WD practices.

Imaging radar and LiDAR image translation for 3-DOF extrinsic calibration

The integration of sensor data is crucial in the field of robotics to take full advantage of the various sensors employed. One critical aspect of this integration is determining the extrinsic calibration parameters, such as the relative transformation, between each sensor. The use of data fusion between complementary sensors, such as radar and LiDAR, can provide significant benefits, particularly in harsh environments where accurate depth data is required. However, noise included in radar sensor data can make the estimation of extrinsic calibration challenging. To address this issue, we present a novel framework for the extrinsic calibration of radar and LiDAR sensors, utilizing CycleGAN as a method of image-to-image translation. Our proposed method employs translating radar bird-eye-view images into LiDAR-style images to estimate the 3-DOF extrinsic parameters. The use of image registration techniques, as well as deskewing based on sensor odometry and B-spline interpolation, is employed to address the rolling shutter effect commonly present in spinning sensors. Our method demonstrates a notable improvement in extrinsic calibration compared to filter-based methods using the MulRan dataset.

Flood Detection in Polarimetric SAR Data Using Deformable Convolutional Vision Model

Floods represent a significant natural hazard with the potential to inflict substantial damage on human society. The swift and precise delineation of flood extents is of paramount importance for effectively supporting flood response and disaster relief efforts. In comparison to optical sensors, Synthetic Aperture Radar (SAR) sensor data acquisition exhibits superior capabilities, finding extensive application in flood detection research. Nonetheless, current methodologies exhibit limited accuracy in flood boundary detection, leading to elevated instances of both false positives and false negatives, particularly in the detection of smaller-scale features. In this study, we proposed an advanced flood detection method called FWSARNet, which leveraged a deformable convolutional visual model with Sentinel-1 SAR images as its primary data source. This model centered around deformable convolutions as its fundamental operation and took inspiration from the structural merits of the Vision Transformer. Through the introduction of a modest number of supplementary parameters, it significantly extended the effective receptive field, enabling the comprehensive capture of intricate local details and spatial fluctuations within flood boundaries. Moreover, our model employed a multi-level feature map fusion strategy that amalgamated feature information from diverse hierarchical levels. This enhancement substantially augmented the model’s capability to encompass various scales and boost its discriminative power. To validate the effectiveness of the proposed model, experiments were conducted using the ETCI2021 dataset. The results demonstrated that the Intersection over Union (IoU) and mean Intersection over Union (mIoU) metrics for flood detection achieved impressive values of 80.10% and 88.47%, respectively. These results surpassed the performance of state-of-the-art (SOTA) models. Notably, in comparison to the best results documented on the official ETCI2021 dataset competition website, our proposed model in this paper exhibited a remarkable 3.29% improvement in flood prediction IoU. The experimental outcomes underscore the capability of the FWSARNet method outlined in this paper for flood detection using Synthetic Aperture Radar (SAR) data. This method notably enhances the accuracy of flood detection, providing essential technical and data support for real-world flood monitoring, prevention, and response efforts.

국제해상충돌예방규칙을 고려한 Event Triggered NMPC 기반의 선박 충돌 회피 알고리즘

About 75% of vessel collision accidents are caused by human error, which causes enormous economic loss, environmental pollution, and human casualties, thus research on automatic collision avoidance of vessels is being actively conducted. In addition, vessels must comply with the COLREGs rules stipulated by IMO when performing collision avoidance with other vessels in motion. In this study, the collision risk was calculated by estimating the position and velocity of other vessels through the Probabilistic Data Association Filter (PDAF) algorithm based on RADAR sensor data. When a collision risk is detected, we propose an event-triggered Nonlinear Model Predict Control (NMPC) algorithm that geometrically creates waypoints that satisfy COLREGs and follows them. To verify the proposed algorithm, simulations through MATLAB are performed.

FedRadar: Federated Multi-Task Transfer Learning for Radar-Based Internet of Medical Things

Owing to increasing connected medical devices and rapid development of deep learning technologies, the Internet of Medical Things (IoMT) has attracted great attention for healthcare monitoring with intelligent sensors. Radar serves as a non-contact healthcare device, continuously measuring human’s vital signs and behavior to provide daily and comprehensive long-term record on health status. However, radar sensor data collected from various users and families usually involve sensitive personal information, while traditional deep learning technologies using IoMT radar data may present high privacy leakage risks. This paper proposes FedRadar, a novel federated multi-task transfer learning framework for radar-based heartbeat rate and activity monitoring in IoMT to solve these challenges. It deals with the decentralized structure, training personalized multi-task models collaboratively, combining the shared relevance knowledge of human’s physiological information while keeping personal radar data locally for privacy protection. First, the multi-task neural network is built on the spatial-temporal radar data to capture the potential relationship and shared representations between human’s vital sign and activity. Furthermore, the federated learning with knowledge transfer scheme is designed to achieve personalized local models by transferring coarse relevant features and keeping fine-grained individual information. Extensive experiments demonstrate the effectiveness and robustness of FedRadar with 2.8% and 2.5% superior than local training model respectively on the accuracy of heartbeat rate estimation and activity classification in realistic constructed radar datasets. In addition, FedRadar is extensible and suitable to continuously monitor multiple health indicators with privacy protection in IoMT. The FedRadar codes and constructed radar datasets are available on https://github.com/bupt-uwb/FedRadar.

Multiposture leg tracking for temporarily vision restricted environments based on fusion of laser and radar sensor data

AbstractLeg tracking is an established field in mobile robotics and machine vision in general. These algorithms, however, only distinguish the scene between leg and nonleg detections. In application fields like firefighting, where people tend to choose squatting or crouching over standing postures, those methods will inevitably fail. Further, tracking based on a single sensor system may reduce the overall reliability if brought to outdoor or complex environments with limited vision on the target objectives. Therefore, we extend our recent work to a multiposture detection system based on laser and radar sensors, that are fused to allow for maximal reliability and accuracy in scenarios as complex as indoor firefighting with vastly limited vision. The proposed tracking pipeline is trained and extensively validated on a new data set. We show that the radar tracker reaches state‐of‐the‐art performance, and that laser and fusion tracker outperform recent methods.

Tracking With Sequentially Fused Radar and Acoustic Sensor Data With Propagation Delay

Tracking With Sequentially Fused Radar and Acoustic Sensor Data With Propagation Delay

On-Road Object Collision Point Estimation by Radar Sensor Data Fusion

This paper proposes an object collision point estimation scheme by developing a new data fusion method in a multi-radar network environment. In order to reduce radar’s estimation error due to measurement uncertainty, we first design radar accuracy models determined by the position of each object. Then, an interacting multiple model (IMM) filter based on occupancy zones is designed for accurate object estimation. For a multi-radar network’s object estimation, we also design a radar data fusion method using the estimated object information through the IMM instead of the object estimation information given by the radars. A collision point identification problem, where multiple sensors calculate the different vehicle surface points of the same object, is solved by developing the data fusion method to estimate the object surface’s collision point closest to the ego vehicle center. The utility of the proposed scheme was validated through a scenario-based object estimation experiment. We confirmed that the proposed data fusion method produced substantially improved error distributions over conventional methods.

Ghost Target Suppression Using Deep Neural Network in Radar-Based Indoor Environment Mapping

In this paper, we propose a method to remove ghost targets that occur in radar-based indoor environment mapping. In this work, the ghost target refers to an object that does not physically exist in actual space, but is detected by multiple reflection paths of a radar signal. We first acquire target detection results in an indoor hallway environment using a millimeter-wave band frequency-modulated continuous wave radar sensor. In the detection results, the estimated distance, velocity, and angle values of real and ghost targets have different distributions, which can be used as criteria to classify both types of targets. Therefore, we design a deep neural network (DNN)-based classifier using the target features as input vectors. Through the proposed DNN-based classifier, the ghost targets can be identified and removed from the detection results. We verify the performance of the proposed ghost target suppression method on two different types of radar sensor data sets: 1) radar sensor data acquired from the hallway environment used to train the DNN and 2) radar sensor data acquired from new hallway environments not used for training the DNN. Our proposed method can remove up to 87.36% of ghost targets on average in three different indoor environments.

Monitoring of Coastal Aquaculture Sites in the Philippines through Automated Time Series Analysis of Sentinel-1 SAR Images

With the unprecedented expansion of aquaculture around the world, there is a critical need to monitor its progress. In Palawan, Philippines, coastal aquaculture is gaining momentum towards increasing fish production, a pressure that presents a challenge to the sustainability of these areas. In this paper, we explore the application of Earth observation methods to map coastal aquaculture development in Palawan and evaluate the extent of its change. The European Space Agency Sentinel-1 synthetic aperture radar and Sentinel-2 multispectral instrument sensor data were applied in fully automatic mode to build maps of Palawan coastal aquaculture. The maps were validated using Google Earth high-resolution optical images and in situ observations in Malampaya Sound, and demonstrated a successful detection rate of 72%, while the false alarm rate was less than 7.5%. Objects only 5 m across, four times smaller than the spatial resolution of the Sentinel-1 sensors, were successfully detected using the developed methodology, thus exceeding the capabilities of other published methods that are limited to detecting large groups of aquaculture structures. The maps revealed aquaculture structures in high quantities in Malampaya Sound, Taytay Bay, and other locations in the coastal waters of Palawan, Philippines. A significant change of aquaculture spatial distribution was identified by comparing aquaculture maps generated with an interval of three years. This new automated methodology was validated as robust for mapping aquaculture objects in Palawan, Philippines, and can be applied to aquaculture studies in other regions worldwide.

Digit Recognition in Air-Writing Using Single Millimeter-Wave Band Radar System

In this paper, we propose an air-writing method in a millimeter-wave band radar system. In particular, a method for removing undesired detection results due to a hand movement is proposed. In our experiments, we use a frequency-modulated continuous wave (FMCW) radar system using 62 GHz as the center frequency and 3 GHz as the bandwidth, which has a range resolution of several centimeters. After installing the FMCW radar on the table, radar sensor data is acquired by having subjects write single-digit numbers (i.e., numbers 0 to 9) in the air. However, in the case of writing numbers 4 and 5, even unnecessary hand movements can be detected by the radar sensor. To identify the numbers in which such undesired detection results occur, the Hough transform is applied to the detection result in the horizontal direction. Then, using different features for each number in the Hough transform domain, undesired detection results due to the hand movement that interfere with number recognition is removed. Finally, we evaluate the digit recognition performance with a convolutional neural network-based classifier. When undesired detection results are removed by the proposed method, the numbers can be recognized with an average accuracy of 97%.

High Level Integrated System of Space Detector Network Integration

In order to meet the requirements of high reliability and low power consumption during the operation of aircraft, this paper adopts the high-level synthesis method to map the high-level language into RTL level code that can run directly on FPGA, and proposes an intelligent integration system for aircraft based on high-level synthesis. The input of aircraft system is image radar sensor data, visible image data and infrared image output. The output of the aircraft system is target identification. Including convolutional IP core, pooled IP core and fully connected IP core. The whole system uses 7-layer deep learning neural network structure, which is convolution IP core, pooled IP core, convolution IP core, pooled IP core, fully connected IP core, fully connected IP core, fully connected IP core.

Mitigation of Rain Effect on Wave Height Measurement Using X-Band Radar Sensor

The presence of rain can negatively affect the performance of many sensors such as X-band radar. In this paper, an effective approach is proposed to mitigate the effect of rain on significant wave height ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${H}_{s}$ </tex-math></inline-formula> ) estimation from X-band radar sensor data along with a machine-learning (ML)-based method. First of all, the haze removal algorithm is applied to rain-contaminated radar images as pre-processing. Then, three different features are extracted from the processed radar images. Different combinations of these three features are utilized to estimate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${H}_{s}$ </tex-math></inline-formula> under the rain condition by using support vector regression (SVR)-based and temporal convolutional network (TCN)-based regression methods. It is found that the root-mean-square-errors (RMSEs) of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${H}_{s}$ </tex-math></inline-formula> estimation results using two typical methods (signal-to-noise ratio (SNR)-based and ensemble empirical mode decomposition (EEMD)-based linear fitting methods) are decreased by 0.14 m and 0.48 m after introducing the haze removal algorithm, respectively. Also, a relatively high accuracy can be achieved using the SVR-based regression method with the combination of SNR and gray level co-occurrence matrix (GLCM) features. Compared to the SNR-based and EEMD-based linear regression methods, the proposed SVR-based method further improves the estimation accuracy, with reductions of RMSE by 0.19 m and 0.82 m, respectively.

Data portability for activities of daily living and fall detection in different environments using radar micro-doppler

The health status of an older or vulnerable person can be determined by looking into the additive effects of aging as well as any associated diseases. This status can lead the person to a situation of ‘unstable incapacity’ for normal aging and is determined by the decrease in response to the environment and to specific pathologies with apparent decrease of independence in activities of daily living (ADL). In this paper, we use micro-Doppler images obtained using a frequency-modulated continuous wave radar (FMCW) operating at 5.8 GHz with 400 MHz bandwidth as the sensor to perform assessment of this health status. The core idea is to develop a generalized system where the data obtained for ADL can be portable across different environments and groups of subjects, and critical events such as falls in mature individuals can be detected. In this context, we have conducted comprehensive experimental campaigns at nine different locations including four laboratory environments and five elderly care homes. A total of 99 subjects participated in the experiments where 1453 micro-Doppler signatures were recorded for six activities. Different machine learning, deep learning algorithms and transfer learning technique were used to classify the ADL. The support vector machine (SVM), K-nearest neighbor (KNN) and convolutional neural network (CNN) provided adequate classification accuracies for particular scenarios; however, the autoencoder neural network outperformed the mentioned classifiers by providing classification accuracy of ~ 88%. The proposed system for fall detection in elderly people can be deployed in care centers and is application for any indoor settings with various age group of people. For future work, we would focus on monitoring multiple older adults, concurrently in indoor settings using continuous radar sensor data stream which is limitation of the present system.

Comparative study of satellite multispectral image data processing with MapReduce and classification algorithm

Now that Big Data has amassed a significant amount of data, it is available in both structured and unstructured formats. Unstructured data processing is difficult to generate by individuals (e.g. Twitter data) or even sensors (e.g. satellites, videos) with data sizes ranging from gigabytes to terabytes and peta-bytes. We can easily analyse and classify different trends in unstructured data sets if the right analytical approach is used, while keeping data quality and size in mind. Early warning forecasts, which are based on satellite imagery and radar sensor data, are a major problem in the real world. To obtain a better understanding of Big data a proper architecture for the analysis of various classifications of satellite imagery patterns using Hadoop technology should be proposed. Different classification methods for different satellite imagery pattern classification methods are segregated in the proposed architecture to reduce time and improve efficiency and will get scalability results to increase the performance of classification patterns to large data sets.

Compressive Sensing-Based SAR Image Reconstruction from Sparse Radar Sensor Data Acquisition in Automotive FMCW Radar System.

In this paper, we propose a method for reconstructing synthetic aperture radar (SAR) images by applying a compressive sensing (CS) technique to sparsely acquired radar sensor data. In general, SAR image reconstruction algorithms require radar sensor data acquired at regular spatial intervals. However, when the speed of the radar-equipped platform is not constant, it is difficult to consistently perform regular data acquisitions. Therefore, we used the CS-based signal recovery method to efficiently reconstruct SAR images even when regular data acquisition was not performed. In the proposed method, we used the l1-norm minimization to overcome the non-uniform data acquisition problem, which replaced the Fourier transform and inverse Fourier transform in the conventional SAR image reconstruction method. In addition, to reduce the phase distortion of the recovered signal, the proposed method was applied to each of the in-phase and quadrature components of the acquired radar sensor data. To evaluate the performance of the proposed method, we conducted experiments using an automotive frequency-modulated continuous wave radar sensor. Then, the quality of the SAR image reconstructed with data acquired at regular intervals was compared with the quality of images reconstructed with data acquired at non-uniform intervals. Using the proposed method, even if only 70% of the regularly acquired radar sensor data was used, a SAR image having a correlation of 0.83 could be reconstructed.

Modeling sub-boreal forest canopy bulk density in Minnesota, USA, using synthetic aperture radar and optical satellite sensor data

BackgroundNational estimates of canopy bulk density (CBD; kg m−3) for fire behavior modeling are generated and supported by the LANDFIRE program. However, locally derived estimates of CBD at finer scales are preferred over national estimates if they exist, as the absolute accuracy of the LANDFIRE CBD product is low and varies regionally. Active sensors (e.g., lidar or radar) are better suited for this task, as passive sensors are ill equipped to detect differences among key vertical fuel structures, such as coniferous surface fuels (≤2 m high) and canopy fuels above this threshold—a key categorical fuel distinction in fire behavior modeling. However, previous efforts to map CBD using lidar sensor data in the Superior National Forest (SNF) of Minnesota, USA, yielded substandard results. Therefore, we use a combination of dormant-season synthetic aperture radar (SAR) and optical satellite sensor data to (1) expand detectability of coniferous fuels among mixed forest canopies to improve the accuracy of CBD modeling and (2) better understand the influence of surface fuels in this regard. Response variables included FuelCalc output and indirect estimates of maximum burnable fuel based on canopy gap fraction (CGF) measured at ground level and 2 m above ground level.ResultsSAR variables were important predictors of CBD and total fuel density (TFD) in all independent model calibrations with ground data, in which we define TFD as the sum of CBD and primarily live coniferous surface fuel density (SFD) 0 to 2 m above ground. Exploratory estimates of TFD appeared biased to the presence of sapling-stage conifer fuel on measures of CGF at the ground level. Thus, modeling efforts to calibrate SFD with satellite sensor data failed. Both CGF-based and FuelCalc-based field estimates of CBD yielded close unity with satellite-calibrated estimates, although substantial differences in data distributions existed. Estimates of CBD from the widest CGF zenith angle range (0 to 38°) correlated best with FuelCalc-based CBD estimates, while both resulted in maximum biomass values that exceeded those considered typical for the SNF. Model results from the narrowest zenith angle range (0 to 7°) produced estimates of CBD that were more in line with values considered typical. LANDFIRE’s estimates of CBD were weakly, but significantly (P = 0.05), correlated to both narrow- and wide-angle CGF-based estimates of CBD, but not with FuelCalc-based estimates.ConclusionsThe combined use of field estimates of CBD, based on indirect measures of CGF according to Keane et al. (Canadian Journal of Forest Research 35:724–739, 2005), with SAR and optical satellite sensor data demonstrates the potential of this method for mapping CBD in the Upper Midwest, USA. Results suggested that the presence of live, coniferous surface fuels neither confounds remote detection nor precludes mapping of CBD in this region using SAR satellite sensor data, as C- and L-band idiosyncrasies likely limit the visibility of these smaller understory fuels from space. Nevertheless, research using direct measures of burnable SFD for calibrations with SAR satellite sensor data should be conducted to more definitively answer this remote detection question, as we suspect substantial bias among measures of CGF from ground level when estimating SFD as the difference between TFD and CBD.