Radar Data Fusion Research Articles

A Novel Multi-Sensor Fusion Based Object Detection and Recognition Algorithm for Intelligent Assisted Driving

The object detection and recognition algorithm based on the fusion of millimeter-wave radar and high-definition video data can improve the safety of intelligent-driving vehicles effectively. However, due to the different data modalities of millimeter-wave radar and video, how to fuse the two effectively is the key point. The difficulty lies in the data fusion methods such as insufficient adaptability of image distortion in data alignment and coordinate transformation and also the mismatching of information levels of the data to be fused. To solve the problem of data fusion of millimeter wave radar and video, this paper proposes a decision-level fusion method of millimeter-wave radar and high-definition video data based on angular alignment. Specifically, through the joint calibration and approximate interpolation, projected to polar coordinate system, the radar and the camera are angularly aligned in the horizontal direction. Then objects are detected by a deep neural network model from video data, and combined with those detected by radar to make the joint decision. Finally, object detection and recognition task based on the fusion of the two kinds of data is completed. Theoretical analysis and experimental results indicate that the accuracy of the algorithm based on the two data fusion is superior to that of the single detection and recognition algorithm on the basis of millimeter-wave radar or video data.

Real-time lidar and radar fusion for road-objects detection and tracking

In this paper, a real-time road-object detection and tracking (LR_ODT) method for autonomous driving is proposed. The method is based on the fusion of lidar and radar measurement data, where they a...

Effective Automatic Radar Bias Estimation Scheme Using Targets of Opportunity

In multi-radar air surveillance scenario bias in individual radar measurements is an important prerequisite for effective fusion of radar data. Conventional bias computation approaches depend on GPS fitted air sorties for computation of bias in radar measurements. The method proposed in this paper computes radar bias automatically using GPS data available from Automatic Dependent Surveillance Broadcast (ADS-B) measurements from targets of opportunity. The proposed method computes the bias in radar measurements without the need of costly dedicated calibration sorties or multiple radars and thereby reducing the cost and manual effort. This paper provides details about the mathematical frame works of recursive weighted least square based bias estimation algorithm and the associated data conversion formats in different coordinate systems. The proposed method preprocess the correlated radar and AD-B measurements and selects measurement pairs from linear segments for time alignment. This paper provides a detailed scheme of automatic bias estimation process with different parameters involved in controlling the accuracy and periodicity of the bias to be computed. The scheme developed in this paper also includes a bias validation module and a unique averaging method to ensure accuracy and smooth variations across the measurement batches considered for computation. The simulation results obtained in this paper suggest the utility of the proposed approach for practical bias computation applications in a cost effective way. The method developed in this paper is based on insight obtained from analysis of recorded data from field deployed radars.

Kalman-filter-based sensor fusion applied to road-objects detection and tracking for autonomous vehicles

In this article, a real-time road-Object Detection and Tracking (LR_ODT) method for autonomous driving is proposed. This method is based on the fusion of lidar and radar measurement data, where they are installed on the ego car, and a customized Unscented Kalman Filter is employed for their data fusion. The merits of both devices are combined using the proposed fusion approach to precisely provide both pose and velocity information for objects moving in roads around the ego car. Unlike other detection and tracking approaches, the balanced treatment of both pose estimation accuracy and its real-time performance is the main contribution in this work. The proposed technique is implemented using the high-performance language C++ and utilizes highly optimized math and optimization libraries for best real-time performance. Simulation studies have been carried out to evaluate the performance of the LR_ODT for tracking bicycles, cars, and pedestrians. Moreover, the performance of the Unscented Kalman Filter fusion is compared to that of the Extended Kalman Filter fusion showing its superiority. The Unscented Kalman Filter has outperformed the Extended Kalman Filter on all test cases and all the state variable levels (−24% average Root Mean Squared Error). The employed fusion technique shows how outstanding is the improvement in tracking performance compared to the use of a single device (−29% Root Mean Squared Error with lidar and −38% Root Mean Squared Error with radar).

Fusion of Radar and Optical Data for Land Cover Classification Using Machine Learning Approach

This study evaluates the advantages of combining traditional space borne optical data with longer wavelengths of radar for land cover mapping. Land cover classification was carried out using Optical, radar data and combination of both for the Bardiya district using Random Forest algorithm. The fusion of optical and radar shows better land cover discrimination with 96.98% overall accuracy in compared to using radar data and optical data separately with overall accuracy of 69.2% and 95.89% respectively. Additionally, the qualitative result demonstrates that the combined utilization of optical and radar imagery yields useful land cover information over those obtained using either type of image on its own.

High-latitude wetland mapping using multidate and multisensor Earth observation data: a case study in the Northwest Territories

The main objective in our study was to derive an accurate wetland inventory of the Dınàgà Wek’èhodì region, Northwest Territories, while also enhancing our previously established wetland mapping workflow. Our methods used multidate optical and radar satellite imagery and fused these data with ArcticDEM topographic variables. Additionally, few studies to date have assessed the ArcticDEM for wetland mapping; our research helps fill this critical gap in the literature. A machine-learning, object-based approach was employed to classify the fused data stacks and included both mean and standard deviation image-object feature extractions. In this study, 18 random forest models were tested, each including various sensor inputs and feature extractions. The highest accuracy was achieved using a fusion of optical, radar, and ArcticDEM data and included both the mean and standard deviation of image objects (88.17% overall accuracy and kappa 0.858). Vegetated wetlands had producer accuracies ranging from 74% to 86%, whereas open water was 92%. Feature importance rankings indicated that 16 of the top 20 variables were derived from optical data, three from radar, and one from the ArcticDEM. The results of our study will be used to assist governments and other interested parties in advancing conservation initiatives for this significant high-latitude region.

Combining and Fusion of Multispectral and RADAR Data for Better Identification of Geological Features

Combining and Fusion of Multispectral and RADAR Data for Better Identification of Geological Features

Review of Remote Sensing Methods to Map Coffee Production Systems

The coffee sector is working towards sector-wide commitments for sustainable production. Yet, knowledge of where coffee is cultivated and its environmental impact remains limited, in part due to the challenges of mapping coffee using satellite remote sensing. We recognize the urgency to capitalize on recent technological advances to improve remote sensing methods and generate more accurate, reliable, and scalable approaches to coffee mapping. In this study, we provide a systematic review of satellite-based approaches to mapping coffee extent, which produced 43 articles in the peer-reviewed and gray literature. We outline key considerations for employing effective approaches, focused on the need to balance data affordability and quality, classification complexity and accuracy, and generalizability and site-specificity. We discuss research opportunities for improved approaches by leveraging the recent expansion of diverse satellite sensors and constellations, optical/Synthetic Aperture Radar data fusion approaches, and advances in cloud computing and deep learning algorithms. We highlight the need for differentiating between production systems and the need for research in important coffee-growing geographies. By reviewing the range of techniques successfully used to map coffee extent, we provide technical recommendations and future directions to enable accurate and scalable coffee maps.

Parking Area Data Collection and Scenario Extraction for the Purpose of Automatic Parking ADAS Function

Automatic Parking is the most significant ADAS function that could achieving mass production in the field of unmanned driving. For the purpose of automatic parking function, we’re foundamentally facing the corresponding data collection and data analysis.In view of three classical parking scenarios,this paper is concentrating on three contents including the sensor hardware construction, the perception algorithm development,and scenario data analysis.For the the hardware construction,we focus on the sensor installation,time calibration,and space unification.Based on the front camera and RADAR data fusion,we put great emphasis on the multi-target matching and fusion algorithm. And for the purpose of the scenario extraction and big data analysis work, we are developing automatic big data processing Python code,realizing TB level original data labeled, and data completely processed in short time.The whole technology process and tool chain in this program,have extensively high level scientific research and commercial value

Vehicle Trajectory Prediction and Collision Warning via Fusion of Multisensors and Wireless Vehicular Communications.

Driver inattention is one of the leading causes of traffic crashes worldwide. Providing the driver with an early warning prior to a potential collision can significantly reduce the fatalities and level of injuries associated with vehicle collisions. In order to monitor the vehicle surroundings and predict collisions, on-board sensors such as radar, lidar, and cameras are often used. However, the driving environment perception based on these sensors can be adversely affected by a number of factors such as weather and solar irradiance. In addition, potential dangers cannot be detected if the target is located outside the limited field-of-view of the sensors, or if the line of sight to the target is occluded. In this paper, we propose an approach for designing a vehicle collision warning system based on fusion of multisensors and wireless vehicular communications. A high-level fusion of radar, lidar, camera, and wireless vehicular communication data was performed to predict the trajectories of remote targets and generate an appropriate warning to the driver prior to a possible collision. We implemented and evaluated the proposed vehicle collision system in virtual driving environments, which consisted of a vehicle–vehicle collision scenario and a vehicle–pedestrian collision scenario.

Использование слияния оптических и радарных данных при космическом мониторинге водных объектов

Использование слияния оптических и радарных данных при космическом мониторинге водных объектов

A Review of the Application of Remote Sensing Data for Abandoned Agricultural Land Identification with Focus on Central and Eastern Europe

This study aims to analyze and assess studies published from 1992 to 2019 and listed in the Web of Science (WOS) and Current Contents (CC) databases, and to identify agricultural abandonment by application of remote sensing (RS) optical and microwave data. We selected 73 studies by applying structured queries in a field tag form and Boolean operators in the WOS portal and by expert analysis. An expert assessment yielded the topical picture concerning the definitions and criteria for the identification of abandoned agricultural land (AAL). The analysis also showed the absence of similar field research, which serves not only for validation, but also for understanding the process of agricultural abandonment. The benefit of the fusion of optical and radar data, which supports the application of Sentinel-1 and Sentinel-2 data, is also evident. Knowledge attained from the literary sources indicated that there exists, in the world literature, a well-covered problem of abandonment identification or biomass estimation, as well as missing works dealing with the assessment of the natural accretion of biomass in AAL.

Radar Data Fusion in the Stradar System

Abstract The main task of the Polish Border Guard is protection of the country’s border which requires utilization of multimedia surveillance systems automatically gathering, processing and sharing various data. The paper presents such a system developed for the Maritime Division of the Polish Border Guard within the STRADAR project and the problem of fusion of radar data in this system. The system, apart from providing communication means, gathers data from AIS, GPS and radar receivers: ARPA and SCANTER 2001. In the paper the concept of the radar data gathering in STRADAR system is provided with detailed presentation of radar servers, Radar INT modules and a reduplication (fusion) module and the proposition of the algorithm for radar data fusion.

Supervised cross-fusion method: a new triplet approach to fuse thermal, radar, and optical satellite data for land use classification

This study presents a new fusion method namely supervised cross-fusion method to improve the capability of fused thermal, radar, and optical images for classification. The proposed cross-fusion method is a combination of pixel-based and supervised feature-based fusion of thermal, radar, and optical data. The pixel-based fusion was applied to fuse optical data of Sentinel-2 and Landsat 8. According to correlation coefficient (CR) and signal to noise ratio (SNR), among the used pixel-based fusion methods, wavelet obtained the best results for fusion. Considering spectral and spatial information preservation, CR of the wavelet method is 0.97 and 0.96, respectively. The supervised feature-based fusion method is a fusion of best output of pixel-based fusion level, land surface temperature (LST) data, and Sentinel-1 radar image using a supervised approach. The supervised approach is a supervised feature selection and learning of the inputs based on linear discriminant analysis and sparse regularization (LDASR) algorithm. In the present study, the non-negative matrix factorization (NMF) was utilized for feature extraction. A comparison of the obtained results with state of the art fusion method indicated a higher accuracy of our proposed method of classification. The rotation forest (RoF) classification results improvement was 25% and the support vector machine (SVM) results improvement was 31%. The results showed that the proposed method is well classified and separated four main classes of settlements, barren land, river, river bank, and even the bridges over the river. Also, a number of unclassified pixels by SVM are very low compared to other classification methods and can be neglected. The study results showed that LST calculated using thermal data has had positive effects on improving the classification results. By comparing the results of supervised cross-fusion without using LST data to the proposed method results, SVM and RoF classifiers showed 38% and 7% of classification improvement, respectively.

Web-based visualisation for look-ahead ground imaging in tunnel boring machines

Abstract Tunnel Boring Machines (TBMs) are large multi-million pound machines used to excavate underground tunnels. In order to make best use of the high-speed performance of a TBM and guarantee the safety of excavation, it is important to know the local geology, structures and ground properties ahead of the TBM cutter head, especially in complex geological conditions (e.g. karst caves). By working with experienced geophysical experts, tunnelling engineers/consultants and TBM manufacturers, we propose a novel web-based visualisation platform to help TBM operators efficiently manage, process and visualise the TBM parameters, the geology map created by geo-experts based on boreholes, and especially the imaging data captured by an on-board ground imaging system for “seeing through” the ground beyond the excavation surface. Informative visualisation interfaces were designed to facilitate interpretation of the imaging data and adding annotation by users; algorithms were developed for automatic detection of features and probable events by fusion of radar and seismic imaging data; and a back-end database was designed to store all such relevant information for supporting more advanced interpretation in the future. The web-based architecture not only allows the visualisation platform to be directly linked to on-board sensors (e.g. ground penetrating radars, seismic sensors), but also allows users away from the job site to access the captured data using a standard web browser, enabling a collaborative interpretation process. The data processing, management and visualisation platform presented in this paper is flexible with respect to different imaging sensors and modalities, so it is highly adaptable for any other ground imaging systems for tunnel geology inspection, underground utility surveys, etc.

High-Resolution Urban Land Mapping in China from Sentinel 1A/2 Imagery Based on Google Earth Engine

Accurate and timely urban land mapping is fundamental to supporting large area environmental and socio-economic research. Most of the available large-area urban land products are limited to a spatial resolution of 30 m. The fusion of optical and synthetic aperture radar (SAR) data for large-area high-resolution urban land mapping has not yet been widely explored. In this study, we propose a fast and effective urban land extraction method using ascending/descending orbits of Sentinel-1A SAR data and Sentinel-2 MSI (MultiSpectral Instrument, Level 1C) optical data acquired from 1 January 2015 to 30 June 2016. Potential urban land (PUL) was identified first through logical operations on yearly mean and standard deviation composites from a time series of ascending/descending orbits of SAR data. A Yearly Normalized Difference Vegetation Index (NDVI) maximum and modified Normalized Difference Water Index (MNDWI) mean composite were generated from Sentinel-2 imagery. The slope image derived from SRTM DEM data was used to mask mountain pixels and reduce the false positives in SAR data over these regions. We applied a region-specific threshold on PUL to extract the target urban land (TUL) and a global threshold on the MNDWI mean, and slope image to extract water bodies and high-slope regions. A majority filter with a three by three window was applied on previously extracted results and the main processing was carried out on the Google Earth Engine (GEE) platform. China was chosen as the testing region to validate the accuracy and robustness of our proposed method through 224,000 validation points randomly selected from high-resolution Google Earth imagery. Additionally, a total of 735 blocks with a size of 900 × 900 m were randomly selected and used to compare our product’s accuracy with the global human settlement layer (GHSL, 2014), GlobeLand30 (2010), and Liu (2015) products. Our method demonstrated the effectiveness of using a fusion of optical and SAR data for large area urban land extraction especially in areas where optical data fail to distinguish urban land from spectrally similar objects. Results show that the average overall, producer’s and user’s accuracies are 88.03%, 94.50% and 82.22%, respectively.

Cooperative Passive Coherent Location: A Promising 5G Service to Support Road Safety

5G promises many new vertical service areas beyond simple communication and data transfer. We propose CPCL (cooperative passive coherent location), a distributed MIMO radar service, which can be offered by mobile radio network operators as a service for public user groups. CPCL comes as an inherent part of the radio network and takes advantage of the most important key features proposed for 5G. It extends the well-known idea of passive radar (also known as passive coherent location, PCL) by introducing cooperative principles. These range from cooperative, synchronous radio signaling, and MAC up to radar data fusion on sensor and scenario levels. By using software-defined radio and network paradigms, as well as real-time mobile edge computing facilities intended for 5G, CPCL promises to become a ubiquitous radar service which may be adaptive, reconfigurable, and perhaps cognitive. As CPCL makes double use of radio resources (both in terms of frequency bands and hardware), it can be considered a green technology. Although we introduce the CPCL idea from the viewpoint of vehicle-to-vehicle/infrastructure (V2X) communication, it can definitely also be applied to many other applications in industry, transport, logistics, and for safety and security applications.

THE SEN1-2 DATASET FOR DEEP LEARNING IN SAR-OPTICAL DATA FUSION

Abstract. While deep learning techniques have an increasing impact on many technical fields, gathering sufficient amounts of training data is a challenging problem in remote sensing. In particular, this holds for applications involving data from multiple sensors with heterogeneous characteristics. One example for that is the fusion of synthetic aperture radar (SAR) data and optical imagery. With this paper, we publish the SEN1-2 dataset to foster deep learning research in SAR-optical data fusion. SEN1-2 comprises 282;384 pairs of corresponding image patches, collected from across the globe and throughout all meteorological seasons. Besides a detailed description of the dataset, we show exemplary results for several possible applications, such as SAR image colorization, SAR-optical image matching, and creation of artificial optical images from SAR input data. Since SEN1-2 is the first large open dataset of this kind, we believe it will support further developments in the field of deep learning for remote sensing as well as multi-sensor data fusion.

Radar cross‐section of pedestrians in the low‐THz band

Low-terahertz (THz) (above 150 GHz) radar sensing is one of the promising technologies to provide safe navigation for autonomous cars due to its expected high-resolution imaging capability. It is anticipated that for robust operation at high levels of autonomy the sensor suite should provide a fusion of video and radar data and its efficiency depends on radar ability to deliver a resolution high enough to be compatible with that of the optical image. Performance of low-THz radar, capable to deliver required resolution, is considered in this study, with the focus on reflectivities of pedestrians at frequencies within the low-THz region – 150 and 300 GHz. Backscatter returns are collected in a controlled environment at a number of frequency bands and at different aspect angles. Measurement methodology is presented and several indicators of reflectivities are calculated. Results are compared with values of radar cross-section reported for current automotive frequency standards 24 and 77 GHz. The effect of clothing on reflectivities is also considered in this study.

APPLICATION OF DEEP LEARNING OF MULTI-TEMPORAL SENTINEL-1 IMAGES FOR THE CLASSIFICATION OF COASTAL VEGETATION ZONE OF THE DANUBE DELTA

Abstract. Land cover is a fundamental variable for regional planning, as well as for the study and understanding of the environment. This work propose a multi-temporal approach relying on a fusion of radar multi-sensor data and information collected by the latest sensor (Sentinel-1) with a view to obtaining better results than traditional image processing techniques. The Danube Delta is the site for this work. The spatial approach relies on new spatial analysis technologies and methodologies: Deep Learning of multi-temporal Sentinel-1. We propose a deep learning network for image classification which exploits the multi-temporal characteristic of Sentinel-1 data. The model we employ is a Gated Recurrent Unit (GRU) Network, a recurrent neural network that explicitly takes into account the time dimension via a gated mechanism to perform the final prediction. The main quality of the GRU network is its ability to consider only the important part of the information coming from the temporal data discarding the irrelevant information via a forgetting mechanism. We propose to use such network structure to classify a series of images Sentinel-1 (20 Sentinel-1 images acquired between 9.10.2014 and 01.04.2016). The results are compared with results of the classification of Random Forest.