Comparison of 2D and 3D semantic segmentation in urban areas using fused hyperspectral and lidar data

Abstract. Limitations and deficiencies of different remote sensing sensors in extraction of different objects caused fusion of data from different sensors to become more widespread for improving classification results. Using a variety of data which are provided from different sensors, increase the spatial and the spectral accuracy. Lidar (Light Detection and Ranging) data fused together with hyperspectral images (HSI) provide rich data for classification of the surface objects. Lidar data representing high quality geometric information plays a key role for segmentation and classification of elevated features such as buildings and trees. On the other hand, hyperspectral data containing high spectral resolution would support high distinction between the objects having different spectral information such as soil, water, and grass. This paper presents a fusion methodology on Lidar and hyperspectral data for improving classification accuracy in urban areas. In first step, we applied feature extraction strategies on each data separately. In this step, texture features based on GLCM (Grey Level Co-occurrence Matrix) from Lidar data and PCA (Principal Component Analysis) and MNF (Minimum Noise Fraction) based dimension reduction methods for HSI are generated. In second step, a Maximum Likelihood (ML) based classification method is applied on each feature spaces. Finally, a fusion method is applied to fuse the results of classification. A co-registered hyperspectral and Lidar data from University of Houston was utilized to examine the result of the proposed method. This data contains nine classes: Building, Tree, Grass, Soil, Water, Road, Parking, Tennis Court and Running Track. Experimental investigation proves the improvement of classification accuracy to 88%.

The fusion of dissimilar data modalities in neural networks presents a significant challenge, particularly in the case of multimodal hyperspectral and lidar data. Hyperspectral data, typically represented as images with potentially hundreds of bands, provide a wealth of spectral information, while lidar data, commonly represented as point clouds with millions of unordered points in 3D space, offer structural information. The complementary nature of these data types presents a unique challenge due to their fundamentally different representations requiring distinct processing methods. In this work, we introduce an alternative hyperspectral data representation in the form of a hyperspectral point cloud (HSPC), which enables ingestion and exploitation with point cloud processing neural network methods. Additionally, we present a composite fusion-style, point convolution-based neural network architecture for the semantic segmentation of HSPC and lidar point cloud data. We investigate the effects of the proposed HSPC representation for both unimodal and multimodal networks ingesting a variety of hyperspectral and lidar data representations. Finally, we compare the performance of these networks against each other and previous approaches. This study paves the way for innovative approaches to multimodal remote sensing data fusion, unlocking new possibilities for enhanced data analysis and interpretation.

In this paper, the fusion of hyperspectral and Li-DAR data is taken into account in order to develop a new classification framework for the accurate analysis of urban areas. In this method, an attribute profile is considered in order to model the spatial information of LiDAR and hyper-spectral data. In parallel, in order to reduce the redundancy of the hyperspectral data and address the so-called curse of dimensionality, a supervised feature extraction technique is used. Then, the new features obtained by the attribute profile and the supervised feature extraction technique are concatenated into a stacked vector. The final classification map is achieved by using a Random Forest classifier. Results infer that the proposed method can provide very good results in terms of classification accuracy and CPU processing time in an automatic manner.

Estimation of forest leaf water content through inversion of a radiative transfer model from LiDAR and hyperspectral data

3D semantic segmentation is important for the performance improvement of a robot vision task. Detection and identification of a large variety of objects in a scene are challenging for autonomous robotic manipulation. With the semantic segmented depth information, a robotic arm can obtain the 3D regions of target objects in a complicated scene. In this paper, we present the development of a 3D semantic segmentation camera based on the Mask Regional Convolutional Neural Network (Mask R-CNN) for the understanding of 3D images at the color level. Each object class is assigned by color for representation. It efficiently separates different objects in an image while concurrently produces a high-quality segmentation for each object. This camera is developed with a structured light technique to obtain 3D information with high accuracy and density. The experimental results demonstrate that the 3D information and semantic segmentation can be combined to achieve reliable scene perception.

Large-scale coastal reclamation has caused significant changes in Spartina alterniflora (S. alterniflora) distribution in coastal regions of China. However, few studies have focused on estimation of the wetland vegetation biomass, especially of S. alterniflora, in coastal regions using LiDAR and hyperspectral data. In this study, the applicability of LiDAR and hypersectral data for estimating S. alterniflora biomass and mapping its distribution in coastal regions of China was explored to attempt problems of wetland vegetation biomass estimation caused by different vegetation types and different canopy height. Results showed that the highest correlation coefficient with S. alterniflora biomass was vegetation canopy height (0.817), followed by Normalized Difference Vegetation Index (NDVI) (0.635), Atmospherically Resistant Vegetation Index (ARVI) (0.631), Visible Atmospherically Resistant Index (VARI) (0.599), and Ratio Vegetation Index (RVI) (0.520). A multivariate linear estimation model of S. alterniflora biomass using a variable backward elimination method was developed with R squared coefficient of 0.902 and the residual predictive deviation (RPD) of 2.62. The model accuracy of S. alterniflora biomass was higher than that of wetland vegetation for mixed vegetation types because it improved the estimation accuracy caused by differences in spectral features and canopy heights of different kinds of wetland vegetation. The result indicated that estimated S. alterniflora biomass was in agreement with the field survey result. Owing to its basis in the fusion of LiDAR data and hyperspectral data, the proposed method provides an advantage for S. alterniflora mapping. The integration of high spatial resolution hyperspectral imagery and LiDAR data derived canopy height had significantly improved the accuracy of mapping S. alterniflora biomass.

Abstract. Hyperspectral imagery is a rich source of spectral information and plays very important role in discrimination of similar land-cover classes. In the past, several efforts have been investigated for improvement of hyperspectral imagery classification. Recently the interest in the joint use of LiDAR data and hyperspectral imagery has been remarkably increased. Because LiDAR can provide structural information of scene while hyperspectral imagery provide spectral and spatial information. The complementary information of LiDAR and hyperspectral data may greatly improve the classification performance especially in the complex urban area. In this paper feature level fusion of hyperspectral and LiDAR data is proposed where spectral and structural features are extract from both dataset, then hybrid feature space is generated by feature stacking. Support Vector Machine (SVM) classifier is applied on hybrid feature space to classify the urban area. In order to optimize the classification performance, two issues should be considered: SVM parameters values determination and feature subset selection. Bees Algorithm (BA) is powerful meta-heuristic optimization algorithm which is applied to determine the optimum SVM parameters and select the optimum feature subset simultaneously. The obtained results show the proposed method can improve the classification accuracy in addition to reducing significantly the dimension of feature space.

The complexity of urban areas makes it difficult for single-source remotely sensed data to meet all urban application requirements. Airborne light detection and ranging (lidar) can provide precise horizontal and vertical point cloud data, while hyperspectral images can provide hundreds of narrow spectral bands which are sensitive to subtle differences in surface materials. The main objectives of this study are to explore: (1) the performance of fused lidar and hyperspectral data for urban land-use classification, especially the contribution of lidar intensity and height information for land-use classification in shadow areas; and (2) the efficiency of combined pixel- and object-based classifiers for urban land-use classification. Support vector machine (SVM), maximum likelihood classification (MLC), and object-based classifiers were used to classify lidar, hyperspectral data and their derived features, such as the normalized digital surface model (nDSM), normalized difference vegetation index (NDVI), and texture measures, into 15 urban land-use classes. Spatial attributes and rules were used to minimize misclassification of the objects showing similar spectral properties, and accuracy assessments were carried out for the classification results. Compared with hyperspectral data alone, hyperspectral–lidar data fusion improved overall accuracy by 6.8% (from 81.7 to 88.5%) when the SVM classifier was used. Meanwhile, compared with SVM alone, the combined SVM and object-based method improved OA by 7.1% (from 87.6 to 94.7%). The results suggest that hyperspectral–lidar data fusion is effective for urban land-use classification, and the proposed combined pixel- and object-based classifiers are very efficient and flexible for the fusion of hyperspectral and lidar data.

3D point cloud semantic segmentation toward large-scale unstructured agricultural scene classification

Abstract. As urbanization continues to increase and extreme climatic events become more prevalent, urban planners and engineers are actively implementing adaptive measures to protect urban assets and communities. To support the urban planning adaptation process, mapping of impervious and pervious areas is essential to understanding the hydrodynamic environment within urban areas for flood risk planning. The application of advance geospatial data and analytical techniques using remote sensing and GIS can improve land surface characterization to better quantify surface run-off and infiltration. This study presents a method to combine airborne hyperspectral and LiDAR data for classifying pervious (e.g. vegetation, gravel, and soil) and impervious (e.g. asphalt and concrete) areas within road allowance areas for the City of Surrey, British Columbia, Canada. Hyperspectral data was acquired using the Compact Airborne Spectrographic Imager (CASI) at 1 m ground spatial resolution, consisting of 72 spectral bands, and LiDAR data acquired from Leica Airborne LiDAR system at a density of 20 points/m2. A spectral library was established using 10 cm orthophotography and GIS data to identify surface features. In addition to spectral functions such as mean and standard deviation, several spectral indices were developed to discriminate between asphalt, concrete, gravel, vegetation, and shadows respectively. A spectral analysis of selected endmembers was conducted and an initial classification technique was applied using Spectral Angle Mapper (SAM). The classification results (i.e. shadows) were improved by integrating LIDAR data with the hyperspectral data.

Synergistic applications based on integrated hyperspectral and lidar data are receiving a growing interest from the remote-sensing community. A prerequisite for the optimum sensor fusion of hyperspectral and lidar data is an accurate geometric coalignment. The simple unadjusted integration of lidar elevation and hyperspectral reflectance causes a substantial loss of information and does not exploit the full potential of both sensors. This paper presents a novel approach for the geometric coalignment of hyperspectral and lidar airborne data, based on their respective adopted return intensity information. The complete approach incorporates ray tracing and subpixel procedures in order to overcome grid inherent discretization. It aims at the correction of extrinsic and intrinsic (camera resectioning) parameters of the hyperspectral sensor. In additional to a tie-point-based coregistration, we introduce a ray-tracing-based back projection of the lidar intensities for area-based cost aggregation. The approach consists of three processing steps. First is a coarse automatic tie-point-based boresight alignment. The second step coregisters the hyperspectral data to the lidar intensities. Third is a parametric coalignment refinement with an area-based cost aggregation. This hybrid approach of combining tie-point features and area-based cost aggregation methods for the parametric coregistration of hyperspectral intensity values to their corresponding lidar intensities results in a root-mean-square error of 1/3 pixel. It indicates that a highly integrated and stringent combination of different coalignment methods leads to an improvement of the multisensor coregistration.

Comparison of 2D and 3D semantic segmentation in urban areas using fused hyperspectral and lidar data

Comparison of 2D and 3D semantic segmentation in urban areas using fused hyperspectral and lidar data

Comparison of 2D and 3D semantic segmentation in urban areas using fused hyperspectral and lidar data

The pre-harvest estimation of seed composition from standing crops is imperative for field management practices and plant phenotyping. This paper presents for the first time the potential of Unmanned Aerial Vehicles (UAV)-based high-resolution hyperspectral and LiDAR data acquired from in-season stand crops for estimating seed protein and oil compositions of soybean and corn using multisensory data fusion and automated machine learning. UAV-based hyperspectral and LiDAR data was collected during the growing season (reproductive stage five (R5)) of 2020 over a soybean test site near Columbia, Missouri and a cornfield at Urbana, Illinois, USA. Canopy spectral and texture features were extracted from hyperspectral imagery, and canopy structure features were derived from LiDAR point clouds. The extracted features were then used as input variables for automated machine-learning methods available with the H2O Automated Machine-Learning framework (H2O-AutoML). The results presented that: (1) UAV hyperspectral imagery can successfully predict both the protein and oil of soybean and corn with moderate accuracies; (2) canopy structure features derived from LiDAR point clouds yielded slightly poorer estimates of crop-seed composition compared to the hyperspectral data; (3) regardless of machine-learning methods, the combination of hyperspectral and LiDAR data outperformed the predictions using a single sensor alone, with an R2 of 0.79 and 0.67 for corn protein and oil and R2 of 0.64 and 0.56 for soybean protein and oil; and (4) the H2O-AutoML framework was found to be an efficient strategy for machine-learning-based data-driven model building. Among the specific regression methods evaluated in this study, the Gradient Boosting Machine (GBM) and Deep Neural Network (NN) exhibited superior performance to other methods. This study reveals opportunities and limitations for multisensory UAV data fusion and automated machine learning in estimating crop-seed composition.