Multi-sensor Classification Research Articles

Multi-Sensor Classification Framework of Urban Vegetation for Improving Ecological Services Management

Recent climatic changes have profoundly impacted the urban microclimate, exposing city dwellers to harsh living conditions. One effective approach to mitigating these events involves incorporating more green infrastructure into the cityscape. The ecological services provided by urban vegetation play a crucial role in enhancing the sustainability and livability of cities. However, monitoring urban vegetation and accurately estimating its status pose challenges due to the heterogeneous nature of the urban environment. In response to this, the current study proposes utilizing a remote sensing-based classification framework to enhance data availability, thereby improving practices related to urban vegetation management. The aim of the current research is to explore the spatial pattern of vegetation and enhance the classification of tree species within diverse and complex urban environments. This study combines various remote sensing observations to enhance classification capabilities. High-resolution colored rectified aerial photographs, LiDAR-derived products, and hyperspectral data are merged and analyzed using advanced classifier methods, specifically partial least squares-discriminant analysis (PLS-DA) and object-based image analysis (OBIA). The OBIA method demonstrates an impressive overall accuracy of 95.30%, while the PLS-DA model excels with a remarkable overall accuracy of 100%. The findings validate the efficacy of incorporating OBIA, aerial photographs, LiDAR, and hyperspectral data in improving tree species classification and mapping within the context of PLS-DA. This classification framework holds significant potential for enhancing management practices and tools, thereby optimizing the ecological services provided by urban vegetation and fostering the development of sustainable cities.

Sensor Fusion with Deep Learning for Autonomous Classification and Management of Aquatic Invasive Plant Species

Recent advances in deep learning, including the development of AlexNet, Residual Network (ResNet), and transfer learning, offer unprecedented classification accuracy in the field of machine vision. A developing application of deep learning is the automated identification and management of aquatic invasive plants. Classification of submersed aquatic vegetation (SAV) presents a unique challenge, namely, the lack of a single source of sensor data that can produce robust, interpretable images across a variable range of depth, turbidity, and lighting conditions. This paper focuses on the development of a multi-sensor (RGB and hydroacoustic) classification system for SAV that is robust to environmental conditions and combines the strengths of each sensing modality. The detection of invasive Hydrilla verticillata (hydrilla) is the primary goal. Over 5000 aerial RGB and hydroacoustic images were generated from two Florida lakes via an unmanned aerial vehicle and boat-mounted sonar unit, and tagged for neural network training and evaluation. Classes included “HYDR”, containing hydrilla; “NONE”, lacking SAV, and “OTHER”, containing SAV other than hydrilla. Using a transfer learning approach, deep neural networks with the ResNet architecture were individually trained on the RGB and hydroacoustic datasets. Multiple data fusion methodologies were evaluated to ensemble the outputs of these neural networks for optimal classification accuracy. A method incorporating logic and a Monte Carlo dropout approach yielded the best overall classification accuracy (84%), with recall and precision of 84.5% and 77.5%, respectively, for the hydrilla class. The training and ensembling approaches were repeated for a DenseNet model with identical training and testing datasets. The overall classification accuracy was similar between the ResNet and DenseNet models when averaged across all approaches (1.9% higher accuracy for the ResNet vs. the DenseNet).

Attention-Guided Multispectral and Panchromatic Image Classification

Multi-sensor image can provide supplementary information, usually leading to better performance in classification tasks. However, the general deep neural network-based multi-sensor classification method learns each sensor image separately, followed by a stacked concentrate for feature fusion. This way requires a large time cost for network training, and insufficient feature fusion may cause. Considering efficient multi-sensor feature extraction and fusion with a lightweight network, this paper proposes an attention-guided classification method (AGCNet), especially for multispectral (MS) and panchromatic (PAN) image classification. In the proposed method, a share-split network (SSNet) including a shared branch and multiple split branches performs feature extraction for each sensor image, where the shared branch learns basis features of MS and PAN images with fewer learn-able parameters, and the split branch extracts the privileged features of each sensor image via multiple task-specific attention units. Furthermore, a selective classification network (SCNet) with a selective kernel unit is used for adaptive feature fusion. The proposed AGCNet can be trained by an end-to-end fashion without manual intervention. The experimental results are reported on four MS and PAN datasets, and compared with state-of-the-art methods. The classification maps and accuracies show the superiority of the proposed AGCNet model.

An ensemble deep learning method as data fusion system for remote sensing multisensor classification

Because of the great achievements in designing remote sensing sensors, the extraction of useful information from multisource remote sensing data remains a challenging problem. Most of the recent research projects have applied single deep learning systems for data fusion and classification. The idea of using ensemble deep learning algorithms through a multisensor fusion system can improve the performance of data fusion tasks. In this research, however, a multi-sensor classification strategy, which is based on deep learning ensemble procedure and decision fusion framework, is investigated for the fusion of Light Detection and Ranging (LiDAR), Hyperspectral Images (HS), and very high-resolution Visible (Vis) images. This research proposes a basic classifier based on deep Convolutional Neural Network (CNN) in which the softmax layer is replaced by a Support Vector Machine (CNN-SVM). Then, a random feature selection is applied to generate two separate CNN-SVM ensemble systems, one for LiDAR and Vis and the other one for HS data. To overcome the similarity and overfitting between the deep features and the classifiers provided by two ensemble systems and to select the best subsets of the classifiers, two diversity measures select the most diverse combinations of the classifiers. Finally, a decision fusion method combines the obtained diverse classifiers from CNN ensembles. Results demonstrate that the proposed method achieves higher accuracy, and its performance outperforms some of the existing methods. The proposed ensemble CNN method improved single deep CNN, random forest, and Adaboost between 2% to 10% in terms of classification accuracy.

A CAUSAL HIERARCHICAL MARKOV FRAMEWORK FOR THE CLASSIFICATION OF MULTIRESOLUTION AND MULTISENSOR REMOTE SENSING IMAGES

Abstract. In this paper, a multiscale Markov framework is proposed in order to address the problem of the classification of multiresolution and multisensor remotely sensed data. The proposed framework makes use of a quadtree to model the interactions across different spatial resolutions and a Markov model with respect to a generic total order relation to deal with contextual information at each scale in order to favor applicability to very high resolution imagery. The methodological properties of the proposed hierarchical framework are investigated. Firstly, we prove the causality of the overall proposed model, a particularly advantageous property in terms of computational cost of the inference. Secondly, we prove the expression of the marginal posterior mode criterion for inference on the proposed framework. Within this framework, a specific algorithm is formulated by defining, within each layer of the quadtree, a Markov chain model with respect to a pixel scan that combines both a zig-zag trajectory and a Hilbert space-filling curve. Data collected by distinct sensors at the same spatial resolution are fused through gradient boosted regression trees. The developed algorithm was experimentally validated with two very high resolution datasets including multispectral, panchromatic and radar satellite images. The experimental results confirm the effectiveness of the proposed algorithm as compared to previous techniques based on alternate approaches to multiresolution fusion.

Automated detection of archaeological mounds using machine-learning classification of multisensor and multitemporal satellite data

This paper presents an innovative multisensor, multitemporal machine-learning approach using remote sensing big data for the detection of archaeological mounds in Cholistan (Pakistan). The Cholistan Desert presents one of the largest concentrations of Indus Civilization sites (from ca 3300 to 1500 BC). Cholistan has figured prominently in theories about changes in water availability, the rise and decline of the Indus Civilization, and the transformation of fertile monsoonal alluvial plains into an extremely arid margin. This paper implements a multisensor, multitemporal machine-learning approach for the remote detection of archaeological mounds. A classifier algorithm that employs a large-scale collection of synthetic-aperture radar and multispectral images has been implemented in Google Earth Engine, resulting in an accurate probability map for mound-like signatures across an area that covers ca 36,000 km2 The results show that the area presents many more archaeological mounds than previously recorded, extending south and east into the desert, which has major implications for understanding the archaeological significance of the region. The detection of small (<5 ha) to large mounds (>30 ha) suggests that there were continuous shifts in settlement location. These shifts are likely to reflect responses to a dynamic and changing hydrological network and the influence of the progressive northward advance of the desert in a long-term process that culminated in the abandonment of much of the settled area during the Late Harappan period.

Classification of Data Stream in Sensor Network With Small Samples

Compared with conventional narrowband radar, ultra-wideband (UWB) radar has strong anti-interference performance, low-frequency, and wide-frequency characteristics, a good penetrating ability, a high-resolution range, and a good target-recognition ability. It is effective for the detection of weak signals such as breathing or motion of the human body. Therefore, a UWB radar is widely used in the field of human body target detection for earthquake and snow disasters. At present, through-wall human target detection technology using a UWB radar has two challenging problems. First, UWB radar data has a dynamic acquisition process, whereas existing detection technologies mostly employ static learning models, resulting in an inability to process and learn data features online. There is a need to address target recognition based on sensor-network data stream. Second, UWB radar faces the problem of unbalanced data classification in the identification of through-wall targets, i.e., target recognition under small-sample conditions. To address these issues, this paper proposes an adaptive incremental recursive least-squares regression parameter estimation method based on an adaptive variable sliding window, which performs Gaussian function fitting on the data streams and adapts to the mean square error and self-adaptation variable sliding window threshold comparison to adaptively block dynamic data streams. The tensor space theory is used to extract and fuse the multisensor data, and the tensor depth learning algorithm is used to improve the recognition accuracy of the target detection under small-sample conditions. To evaluate the feasibility of the proposed algorithm, we use three UWB radars to build a multistate recognition system for human targets. The data stream adaptive block effect, single and multisensor classification effects were tested under small-sample conditions. The experimental results indicate that the proposed algorithm not only accurately segmented the dynamic data streams but also effectively realized multisensor information fusion and improved the real-time target monitoring under small-sample conditions.

Variational Gaussian process for multisensor classification problems

This paper proposes a new model for multi-sensory data classification. To tackle this problem, probabilistic modeling and variational Bayesian inference are used. A Gaussian Process (GP) classifier is built upon the introduced modeling. Its posterior distribution is approximated using variational Bayesian inference. Finally, labels of test samples are predicted employing this classifier. Very importantly, and in contrast to alternative approaches, the proposed method does not discard samples with missing features and utilizes all available information for training. Furthermore, to take into account that the quality of the information provided by each sensor may differ (some modalities/sensors may provide more reliable/distinctive information than others), we introduce two versions of the algorithm. In the first one, the parameters modeling each sensor performance are shared while in the second one, each sensor parameters are estimated independently. Synthetic and real datasets are utilized to examine the validity of the proposed models. The results obtained for binary classification problems justify their use and confirm their superiority over existing fusion architectures.

Classification of Multisensor and Multiresolution Remote Sensing Images Through Hierarchical Markov Random Fields

This letter proposes two methods for the supervised classification of multisensor optical and synthetic aperture radar images with possibly different spatial resolutions. Both the methods are formulated within a unique framework based on hierarchical Markov random fields. Distinct quad-trees associated with the individual information sources are defined to jointly address multisensor, multiresolution, and possibly multifrequency fusion, and are integrated with finite mixture models and the marginal posterior mode criterion. Experimental validation is conducted with Pleiades, COSMO-SkyMed, RADARSAT-2, and GeoEye-1 data.

A decision-based multi-sensor classification system using thermal hyperspectral and visible data in urban area

ABSTRACTMulti-sensor data fusion has become more and more popular for classification applications. The fusion of multisource remote-sensing data can provide more information about the same observed site results in a superior comprehension of the scene. In this field of study, a combination of very high-resolution data collected by a digital color camera and a new coarse resolution hyperspectral data in the long-wave infrared range for urban land-cover classification has been extensively enticed much consideration and turned into a research hot spot in image analysis and data fusion research community. In this paper, a decision-based multi-sensor classification system is proposed to completely use the advantages of both sensors to attain enhanced land-cover classification results. In this context, spectral, textural and spatial features are extracted for the proposed multilevel classification. Then, a land-cover separability preprocessing is employed to identify how the proposed method can fully utilize the sensor advantages. Next, a support vector machine is applied to classify road classes by using thermal hyperspectral image data; plants, roofs and bare soils are classified by the joint use of sensors via Dempster–Shafer classifier fusion. Finally, an object-based post-processing is employed to improve the classification results. Experiments carried out on the dataset of 2014 IEEE GRSS data fusion contest indicate the superiority of the proposed methodology for the potentialities and possibilities of the joint utilization of sensors and refine the classification outcomes when evaluated against single sensor data. Meanwhile, the obtained classification accuracy can be a competitor against the results issued by the 2014 IEEE GRSS data fusion contest.

Multisensor Classification System for Triggering FES in Order to Support Voluntary Swallowing.

In dysphagia the ability of elevating the larynx and hyoid is usually impaired. Electromyography (EMG) and Bioimpedance (BI) measurements at the neck can be used to trigger functional electrical stimulation (FES) of swallowing related muscles. Nahrstaedt et al.1 introduced an algorithm to trigger the stimulation in phase with the voluntary swallowing to improve the airway closure and elevation speed of the larynx and hyoid. However, due to non-swallow related movements like speaking, chewing or head turning, stimulations might be unintentionally triggered. So far a switch was used to enable the BI/EMG-triggering of FES when the subject was ready to swallow, which is inconvenient for practical use. In this contribution, a range image camera system is introduced to obtain data of head, mouth, and jaw movements. This data is used to apply a second classification step to reduce the number of false stimulations. In experiments with healthy subjects, the amount of potential false stimulations could be reduced by 47% while 83% of swallowing intentions would have been correctely supported by FES.

Collaborative Multi-Sensor Classification Via Sparsity-Based Representation

In this paper, we propose a general collaborative sparse representation framework for multi-sensor classification, which takes into account the correlations as well as complementary information between heterogeneous sensors simultaneously while considering joint sparsity within each sensor’s observations. We also robustify our models to deal with the presence of sparse noise and low-rank interference signals. Specifically, we demonstrate that incorporating the noise or interference signal as a low-rank component in our models is essential in a multi-sensor classification problem when multiple co-located sources/sensors simultaneously record the same physical event. We further extend our frameworks to kernelized models which rely on sparsely representing a test sample in terms of all the training samples in a feature space induced by a kernel function. A fast and efficient algorithm based on alternative direction method is proposed where its convergence to an optimal solution is guaranteed. Extensive experiments are conducted on several real multi-sensor data sets and results are compared with the conventional classifiers to verify the effectiveness of the proposed methods.

Regional Urban Extent Extraction Using Multi-Sensor Data and One-Class Classification

Stable night-time light data from the Defense Meteorological Satellite Program (DMSP) Operational Line-scan System (OLS) provide a unique proxy for anthropogenic development. This paper presents a regional urban extent extraction method using a one-class classifier and combinations of DMSP/OLS stable night-time light (NTL) data, MODIS normalized difference vegetation index (NDVI) data, and land surface temperature (LST) data. We first analyzed how well MODIS NDVI and LST data quantify the properties of urban areas. Considering that urban area is the only class of interest, we applied the one-class support vector machine (OCSVM) to classify different combinations of the three datasets. We evaluated the effectiveness of the proposed method and compared with the locally optimized threshold method in regional urban extent mapping in China. The experimental results demonstrate that DMSP/OLS NTL data, MODIS NDVI and LST data provide different but complementary information sources to quantify the urban extent at a regional scale. The results also indicate that the OCSVM classification of the combination of all three datasets generally outperformed the locally optimized threshold method. The proposed method effectively and efficiently extracted the urban extent at a regional scale, and is applicable to other study areas.

Land cover classification of Lago Grande de Curuai floodplain (Amazon, Brazil) using multi-sensor and image fusion techniques

Given the limitations of different types of remote sensing images, automated land-cover classifications of the Amazon várzea may yield poor accuracy indexes. One way to improve accuracy is through the combination of images from different sensors, by either image fusion or multi-sensor classifications. Therefore, the objective of this study was to determine which classification method is more efficient in improving land cover classification accuracies for the Amazon várzea and similar wetland environments - (a) synthetically fused optical and SAR images or (b) multi-sensor classification of paired SAR and optical images. Land cover classifications based on images from a single sensor (Landsat TM or Radarsat-2) are compared with multi-sensor and image fusion classifications. Object-based image analyses (OBIA) and the J.48 data-mining algorithm were used for automated classification, and classification accuracies were assessed using the kappa index of agreement and the recently proposed allocation and quantity disagreement measures. Overall, optical-based classifications had better accuracy than SAR-based classifications. Once both datasets were combined using the multi-sensor approach, there was a 2% decrease in allocation disagreement, as the method was able to overcome part of the limitations present in both images. Accuracy decreased when image fusion methods were used, however. We therefore concluded that the multi-sensor classification method is more appropriate for classifying land cover in the Amazon várzea.

Supervised Classification of Multisensor and Multiresolution Remote Sensing Images With a Hierarchical Copula-Based Approach

In this paper, we develop a novel classification approach for multiresolution, multisensor [optical and synthetic aperture radar (SAR)], and/or multiband images. This challenging image processing problem is of great importance for various remote sensing monitoring applications and has been scarcely addressed so far. To deal with this classification problem, we propose a two-step explicit statistical model. We first design a model for the multivariate joint class-conditional statistics of the coregistered input images at each resolution by resorting to multivariate copulas. Such copulas combine the class-conditional marginal probability density functions (pdfs) of each input channel that are estimated by finite mixtures of well-chosen parametric families. We consider different distribution families for the most common types of remote sensing imagery acquired by optical and SAR sensors. We then plug the estimated joint pdfs into a hierarchical Markovian model based on a quad-tree structure, where each tree-scale corresponds to the different input image resolutions and to corresponding multiscale decimated wavelet transforms, thus preventing a strong resampling of the initial images. To obtain the classification map, we resort to an exact estimator of the marginal posterior mode. We integrate a prior update in this model in order to improve the robustness of the developed classifier against noise and speckle. The resulting classification performance is illustrated on several remote sensing multiresolution data sets, including very high resolution and multisensor images acquired by COSMO-SkyMed and GeoEye-1.

Mapping natural and urban environments using airborne multi-sensor ADS40–MIVIS–LiDAR synergies

The recent and forthcoming availability of high spatial resolution imagery from satellite and airborne sensors offers the possibility to generate an increasing number of remote sensing products and opens new promising opportunities for multi-sensor classification. Data fusion strategies, applied to modern airborne Earth observation systems, including hyperspectral MIVIS, color-infrared ADS40, and LiDAR sensors, are explored in this paper for fine-scale mapping of heterogeneous urban/rural landscapes. An over 1000-element array of supervised classification results is generated by varying the underlying classification algorithm (Maximum Likelihood/Spectral Angle Mapper/Spectral Information Divergence), the remote sensing data stack (different multi-sensor data combination), and the set of hyperspectral channels used for classification (feature selection). The analysis focuses on the identification of the best performing data fusion configuration and investigates sensor-derived marginal improvements. Numerical experiments, performed on a 20-km stretch of the Marecchia River (Italy), allow for a quantification of the synergies of multi-sensor airborne data. The use of Maximum Likelihood and of the feature space including ADS40, LiDAR derived normalized digital surface, texture layers, and 24 MIVIS bands represents the scheme that maximizes the classification accuracy on the test set. The best classification provides high accuracy (92.57% overall accuracy) and demonstrates the potential of the proposed approach to define the optimized data fusion and to capture the high spatial variability of natural and human-dominated environments. Significant inter-class differences in the identification schemes are also found by indicating possible sub-optimal solutions for landscape-driven mapping, such as mixed forest, floodplain, urban, and agricultural zones.

A study of visual and tactile terrain classification and classifier fusion for planetary exploration rovers

SUMMARYKnowledge of the physical properties of terrain surrounding a planetary exploration rover can be used to allow a rover system to fully exploit its mobility capabilities. Terrain classification methods provide semantic descriptions of the physical nature of a given terrain region. These descriptions can be associated with nominal numerical physical parameters, and/or nominal traversability estimates, to improve mobility prediction accuracy. Here we study the performance of multisensor classification methods in the context of Mars surface exploration. The performance of two classification algorithms for color, texture, and range features are presented based on maximum likelihood estimation and support vector machines. In addition, a classification method based on vibration features derived from rover wheel–terrain interaction is briefly described. Two techniques for merging the results of these “low-level” classifiers are presented that rely on Bayesian fusion and meta-classifier fusion. The performance of these algorithms is studied using images from NASA's Mars Exploration Rover mission and through experiments on a four-wheeled test-bed rover operating in Mars-analog terrain. Also a novel approach to terrain sensing based on fused tactile and visual features is presented. It is shown that accurate terrain classification can be achieved via classifier fusion from visual and tactile features.

Constructing consonant belief functions from sample data using confidence sets of pignistic probabilities

A new method is proposed for building a predictive belief function from statistical data in the transferable belief model framework. The starting point of this method is the assumption that, if the probability distribution P X of a random variable X is known, then the belief function quantifying our belief regarding a future realization of X should have its pignistic probability distribution equal to P X . When P X is unknown but a random sample of X is available, it is possible to build a set P of probability distributions containing P X with some confidence level. Following the least commitment principle, we then look for a belief function less committed than all belief functions with pignistic probability distribution in P . Our method selects the most committed consonant belief function verifying this property. This general principle is applied to arbitrary discrete distributions as well as exponential and normal distributions. The efficiency of this approach is demonstrated using a simulated multi-sensor classification problem.

Performance evaluation of multi-sensor classification systems

A common problem in classification is to use one/more sensors to observe repeated measurements of a target's features/attributes, and in turn update the targets' posterior classification probabilities to aid in target identification. This paper addresses the following questions: 1. How do we quantify the classification performance of a sensor? 2. What happens to the posterior probabilities as the number of measurements increase? 3. Will the targets be classified correctly? While the Kalman filter allows for off-line estimation of kinematic performance (covariance matrix), a comparable approach for studying classification accuracy has not been done previously. We develop a new analytical approach for computing the long-run classification performance of a sensor and also present recursive formulas for efficient calculation of the same. We show that, under a minimal condition, a sensor will eventually classify all targets perfectly. We also develop a methodology for evaluating the classification performance of multi-sensor fusion systems involving sensors of varying quality. The contributions of this paper are 1. A simple metric to quantify a sensor's ability to discriminate between the targets being identified, and its use in comparing multiple sensors, 2. An approximate formula based on this metric to compute off-line estimates of the rate of convergence toward perfect classification, and the number of measurements required to achieve a desired level of classification accuracy, and 3. The use of this metric to evaluate classification performance of multi-sensor fusion systems.

Performance evaluation of multi-sensor classification systems

Performance evaluation of multi-sensor classification systems