Object-based Convolutional Neural Network Research Articles

A classification framework with multi-level fusion of object-based analysis and convolutional neural network: a case study for land use classification in mining areas

ABSTRACT To enhance the accuracy of land use classification in mining areas, the Object-based Convolutional Neural Network (OCNN) method has been widely used. However, existing researches tend to neglect the importance of decision-level fusion, focusing only on feature-level fusion. This study proposes a new classification framework with Multi-level Fusion of Object-based analysis and CNN (MFOCNN) to achieves high-accuracy land use classification in mining areas. First, Simple Linear Iterative Cluster (SLIC) is employed to generate image objects, which serve as the basic unit for classification. Second, an improved DenseNet is proposed to extract deep features from image patches, which represent image objects, and provide the classification result. Third, handcrafted features including spectral, textural and geometric of the image objects are extracted and fused with the deep features to obtain the classification result with random forest classifier. Finally, the Dempster-Shafer (DS) evidence theory is applied to fuse the two previously described classification results at the decision-level to obtain the final result. Experiments conducted in the mining areas of Erdos using Gaofen-6 images demonstrate that the proposed MFOCNN achieves the best visual performance and accuracy among all tested methods. The MFOCNN, with its feature-level fusion and decision-level fusion, significantly improves the accuracy of land use classification in mining areas. The results suggest that the proposed MFOCNN is a promising method for achieving high-accuracy land use classification in mining areas.

Building Extraction From High Spatial Resolution Remote Sensing Images of Complex Scenes by Combining Region-Line Feature Fusion and OCNN

Building extraction from remote sensing imagery has been a research hotspot for some time with the advancement of AI in remote sensing. However, the edges of buildings extracted using existing techniques are commonly broken and inaccurate for the complex scenes in suburban and rural areas. This study proposes a framework for extracting structures by combining region-line feature fusion with object-based convolutional neural networks to solve this problem. First, a building edge detection network known as the Multichannel Attention-based Dense Extreme Inception Network for Edge Detection (MA-DexiNed) is constructed, which is considered more accurate for building edge extraction in complicated image scenes. Second, the probability map of the building edges obtained by MA-DexiNed is refined. According to rule judgment, breakpoints are linked by an edge thinning connection algorithm to obtain single-pixel, contiguous building line features. Third, the geometric boundaries of buildings are obtained by combining region attributes derived by unsupervised image segmentation and line features obtained from deep learning supervised segmentation. Finally, the pretrained AlexNet is employed to identify the class characteristics of buildings. The suggested framework was used for two GF-2 images and one Google Earth image from various regions and with numerous types of complicated scenes. The experimental findings demonstrated that this approach could extract more precise and complete building edges for complex image scenes compared with several existing methods. This advancement results from constrained regional image segmentation using deep semantic edge features. This methodology can offer a benchmark for subsequent building extraction tasks from high resolution imagery.

A Joint Bayesian Optimization for the Classification of Fine Spatial Resolution Remotely Sensed Imagery Using Object-Based Convolutional Neural Networks

In recent years, deep learning-based image classification has become widespread, especially in remote sensing applications, due to its automatic and strong feature extraction capability. However, as deep learning methods operate on rectangular-shaped image patches, they cannot accurately extract objects’ boundaries, especially in complex urban settings. As a result, combining deep learning and object-based image analysis (OBIA) has become a new avenue in remote sensing studies. This paper presents a novel approach for combining convolutional neural networks (CNN) with OBIA based on joint optimization of segmentation parameters and deep feature extraction. A Bayesian technique was used to find the best parameters for the multiresolution segmentation (MRS) algorithm while the CNN model learns the image features at different layers, achieving joint optimization. The proposed classification model achieved the best accuracy, with 0.96 OA, 0.95 Kappa, and 0.96 mIoU in the training area and 0.97 OA, 0.96 Kappa, and 0.97 mIoU in the test area, outperforming several benchmark methods including Patch CNN, Center OCNN, Random OCNN, and Decision Fusion. The analysis of CNN variants within the proposed classification workflow showed that the HybridSN model achieved the best results compared to 2D and 3D CNNs. The 3D CNN layers and combining 3D and 2D CNN layers (HybridSN) yielded slightly better accuracies than the 2D CNN layers regarding geometric fidelity, object boundary extraction, and separation of adjacent objects. The Bayesian optimization could find comparable optimal MRS parameters for the training and test areas, with excellent quality measured by AFI (0.046, −0.037) and QR (0.945, 0.932). In the proposed model, higher accuracies could be obtained with larger patch sizes (e.g., 9 × 9 compared to 3 × 3). Moreover, the proposed model is computationally efficient, with the longest training being fewer than 25 s considering all the subprocesses and a single training epoch. As a result, the proposed model can be used for urban and environmental applications that rely on VHR satellite images and require information about land use.

Temporal Sequence Object-based CNN (TS-OCNN) for crop classification from fine resolution remote sensing image time-series

Accurate crop distribution mapping is required for crop yield prediction and field management. Due to rapid progress in remote sensing technology, fine spatial resolution (FSR) remotely sensed imagery now offers great opportunities for mapping crop types in great detail. However, within-class variance can hamper attempts to discriminate crop classes at fine resolutions. Multi-temporal FSR remotely sensed imagery provides a means of increasing crop classification from FSR imagery, although current methods do not exploit the available information fully. In this research, a novel Temporal Sequence Object-based Convolutional Neural Network (TS-OCNN) was proposed to classify agricultural crop type from FSR image time-series. An object-based CNN (OCNN) model was adopted in the TS-OCNN to classify images at the object level (i.e., segmented objects or crop parcels), thus, maintaining the precise boundary information of crop parcels. The combination of image time-series was first utilized as the input to the OCNN model to produce an ‘original’ or baseline classification. Then the single-date images were fed automatically into the deep learning model scene-by-scene in order of image acquisition date to increase successively the crop classification accuracy. By doing so, the joint information in the FSR multi-temporal observations and the unique individual information from the single-date images were exploited comprehensively for crop classification. The effectiveness of the proposed approach was investigated using multitemporal SAR and optical imagery, respectively, over two heterogeneous agricultural areas. The experimental results demonstrated that the newly proposed TS-OCNN approach consistently increased crop classification accuracy, and achieved the greatest accuracies (82.68% and 87.40%) in comparison with state-of-the-art benchmark methods, including the object-based CNN (OCNN) (81.63% and 85.88%), object-based image analysis (OBIA) (78.21% and 84.83%), and standard pixel-wise CNN (79.18% and 82.90%). The proposed approach is the first known attempt to explore simultaneously the joint information from image time-series with the unique information from single-date images for crop classification using a deep learning framework. The TS-OCNN, therefore, represents a new approach for agricultural landscape classification from multi-temporal FSR imagery. Besides, it is readily generalizable to other landscapes (e.g., forest landscapes), with a wide application prospect.

Improved object-based convolutional neural network (IOCNN) to classify very high-resolution remote sensing images

ABSTRACT The land cover classification of very high-resolution (VHR) remote sensing images is a challenging task. VHR images depict many complex objects with various shapes in complicated contexts. The deep learning-based method is a solution for such dif- ficult task and feature extraction. Nevertheless, this method cannot efficiently deal with images with complex scene structures. An improved object-based convolutional neural network (IOCNN) is designed to classify VHR images with zone division and convolutional position sampling techniques in this study. The method can achieve the best performance of each zone at its own optimized scales. Based on multi-scale convolutional deep features extracted from VHR images, the objects with irregular shapes can be classified using the approach. In this study, the zone-level scale adaption and multi-scale recognition of complex objects are achieved. The performance of IOCNN is compared with the state-of-the-art methods for feature extraction, including five object-based CNN approaches and two fully convolutional networks (FCNs). The results show that the classification performance of IOCNN is considerably stronger than that of state-of-the-art methods. The overall accuracies of the land cover classification in IOCNN are 91.65% and 93.49% on two tested images. The results demonstrate the practicability of IOCNN.

BTS: a binary tree sampling strategy for object identification based on deep learning

ABSTRACT Object-based convolutional neural networks (OCNNs) have achieved great performance in the field of land-cover and land-use classification. Studies have suggested that the generation of object convolutional positions (OCPs) largely determines the performance of OCNNs. Optimized distribution of OCPs facilitates the identification of segmented objects with irregular shapes. In this study, we propose a morphology-based binary tree sampling (BTS) method that provides a reasonable, effective, and robust strategy to generate evenly distributed OCPs. The proposed BTS algorithm consists of three major steps: 1) calculating the required number of OCPs for each object, 2) dividing a vector object into smaller sub-objects, and 3) generating OCPs based on the sub-objects. Taking the object identification in land-cover and land-use classification as a case study, we compare the proposed BTS algorithm with other competing methods. The results suggest that the BTS algorithm outperforms all other competing methods, as it yields more evenly distributed OCPs that contribute to better representation of objects, thus leading to higher object identification accuracy. Further experiments suggest that the efficiency of BTS can be improved when multi-thread technology is implemented.

Iterative Deep Learning (IDL) for agricultural landscape classification using fine spatial resolution remotely sensed imagery

The agricultural landscape can be interpreted at different semantic levels, such as fine low-level crop (LLC) classes (e.g., Wheat, Almond, and Alfalfa) and broad high-level crop (HLC) classes (e.g., Winter crops, Tree crops, and Forage). The LLC and HLC are hierarchically correlated with each other, but such intrinsically hierarchical relationships have been overlooked in previous crop classification studies in remote sensing. In this research, a novel Iterative Deep Learning (IDL) framework was proposed for the classification of complex agricultural landscapes using remotely sensed imagery. The IDL adopts an object-based convolutional neural network (OCNN) as the basic classifier for both the LLC and HLC classifications, which has the advantage of maintaining precise crop parcel boundaries. In IDL, the HLC classification implemented by the OCNN is conditional upon the LLC classification probabilities, whereas the HLC probabilities combined with the original imagery are, in turn, re-used as inputs to the OCNN to enhance the LLC classification. Such an iterative updating procedure forms a Markov process, where both the LLC and HLC classifications are refined and evolve collaboratively. The effectiveness of the IDL was tested on two heterogeneous agricultural fields using fine spatial resolution (FSR) SAR and optical imagery. The experimental results demonstrate that the iterative process of IDL helps to resolve contradictions within the class hierarchies. The new proposed IDL consistently increased the accuracies of both the LLC and HLC classifications with iteration, and achieved the highest accuracies for each at four iterations. The average overall accuracies were 88.4% for LLC and 91.2% for HLC, for both study sites, far greater than the accuracies of the state-of-the-art benchmarks, including the pixel-wise CNN (81.7% and 85.9%), object-based image analysis (OBIA) (84.0% and 85.8%), and OCNN (84.0% and 88.4%). To the best of our knowledge, the proposed model is the first to identify and use the relationship between the class levels in an ontological hierarchy in a remote sensing classification process. It is applied here to increase progressively the accuracy of classification at two levels for a complex agricultural landscape. As such IDL represents an entirely new paradigm for remote sensing image classification. Moreover, the promising results demonstrate the great potential of the proposed IDL with wide application prospect.

A Scale Sequence Object-based Convolutional Neural Network (SS-OCNN) for crop classification from fine spatial resolution remotely sensed imagery

ABSTRACT The highly dynamic nature of agro-ecosystems in space and time usually leads to high intra-class variance and low inter-class separability in the fine spatial resolution (FSR) remotely sensed imagery. This makes traditional classifiers essentially relying on spectral information for crop mapping from FSR imagery an extremely challenging task. To mine effectively the rich spectral and spatial information in FSR imagery, this paper proposed a Scale Sequence Object-based Convolutional Neural Network (SS-OCNN) that classifies images at the object level by taking segmented objects (crop parcels) as basic units of analysis, thus, ensuring that the boundaries between crop parcels are delineated precisely. These segmented objects were subsequently classified using a CNN model integrated with an automatically generated scale sequence of input patch sizes. This scale sequence can fuse effectively the features learned at different scales by transforming progressively the information extracted at small scales to larger scales. The effectiveness of the SS-OCNN was investigated using two heterogeneous agricultural areas with FSR SAR and optical imagery, respectively. Experimental results revealed that the SS-OCNN consistently achieved the most accurate classification results. The SS-OCNN, thus, provides a new paradigm for crop classification over heterogeneous areas using FSR imagery, and has a wide application prospect.

Explainable identification and mapping of trees using UAV RGB image and deep learning

The identification and mapping of trees via remotely sensed data for application in forest management is an active area of research. Previously proposed methods using airborne and hyperspectral sensors can identify tree species with high accuracy but are costly and are thus unsuitable for small-scale forest managers. In this work, we constructed a machine vision system for tree identification and mapping using Red–Green–Blue (RGB) image taken by an unmanned aerial vehicle (UAV) and a convolutional neural network (CNN). In this system, we first calculated the slope from the three-dimensional model obtained by the UAV, and segmented the UAV RGB photograph of the forest into several tree crown objects automatically using colour and three-dimensional information and the slope model, and lastly applied object-based CNN classification for each crown image. This system succeeded in classifying seven tree classes, including several tree species with more than 90% accuracy. The guided gradient-weighted class activation mapping (Guided Grad-CAM) showed that the CNN classified trees according to their shapes and leaf contrasts, which enhances the potential of the system for classifying individual trees with similar colours in a cost-effective manner—a useful feature for forest management.

Mapping Urban Tree Cover Changes Using Object-Based Convolution Neural Network (OB-CNN)

Urban trees provide social, economic, environmental and ecosystem services benefits that improve the liveability of cities and contribute to individual and community wellbeing. There is thus a need for effective mapping, monitoring and maintenance of urban trees. Remote sensing technologies can effectively map and monitor urban tree coverage and changes over time as an efficient and low-cost alternative to field-based measurements, which are time consuming and costly. Automatic extraction of urban land cover features with high accuracy is a challenging task, and it demands object based artificial intelligence workflows for efficiency and thematic accuracy. The aim of this research is to effectively map urban tree cover changes and model the relationship of such changes with socioeconomic variables. The object-based convolutional neural network (CNN) method is illustrated by mapping urban tree cover changes between 2005 and 2015/16 using satellite, Google Earth imageries and Light Detection and Ranging (LiDAR) datasets. The training sample for CNN model was generated by Object Based Image Analysis (OBIA) using thresholds in a Canopy Height Model (CHM) and the Normalised Difference Vegetation Index (NDVI). The tree heatmap produced from the CNN model was further refined using OBIA. Tree cover loss, gain and persistence was extracted, and multiple regression analysis was applied to model the relationship with socioeconomic variables. The overall accuracy and kappa coefficient of tree cover extraction was 96% and 0.77 for 2005 images and 98% and 0.93 for 2015/16 images, indicating that the object-based CNN technique can be effectively implemented for urban tree coverage mapping and monitoring. There was a decline in tree coverage in all suburbs. Mean parcel size and median household income were significantly related to tree cover loss (R2 = 58.5%). Tree cover gain and persistence had positive relationship with tertiary education, parcel size and ownership change (gain: R2 = 67.8% and persistence: R2 = 75.3%). The research findings demonstrated that remote sensing data with intelligent processing can contribute to the development of policy input for management of tree coverage in cities.

Mapping Tea Plantations from VHR Images Using OBIA and Convolutional Neural Networks

Tea is an important economic plant, which is widely cultivated in many countries, particularly in China. Accurately mapping tea plantations is crucial in the operations, management, and supervision of the growth and development of the tea industry. We propose an object-based convolutional neural network (CNN) to extract tea plantations from very high resolution remote sensing images. Image segmentation was performed to obtain image objects, while a fine-tuned CNN model was used to extract deep image features. We conducted feature selection based on the Gini index to reduce the dimensionality of deep features, and the selected features were then used for classifying tea objects via a random forest. The proposed method was first applied to Google Earth images and then transferred to GF-2 satellite images. We compared the proposed classification with existing methods: Object-based classification using random forest, Mask R-CNN, and object-based CNN without fine-tuning. The results show the proposed method achieved a higher classification accuracy than other methods and produced smaller over- and under-classification geometric errors than Mask R-CNN in terms of shape integrity and boundary consistency. The proposed approach, trained using Google Earth images, achieved comparable results when transferring to the classification of tea objects from GF-2 images. We conclude that the proposed method is effective for mapping tea plantations using very high-resolution remote sensing images even with limited training samples and has huge potential for mapping tea plantations in large areas.

Exploring multiscale object-based convolutional neural network (multi-OCNN) for remote sensing image classification at high spatial resolution

Convolutional Neural Network (CNN) has been increasingly used for land cover mapping of remotely sensed imagery. However, large-area classification using traditional CNN is computationally expensive and produces coarse maps using a sliding window approach. To address this problem, object-based CNN (OCNN) becomes an alternative solution to improve classification performance. However, previous studies were mainly focused on urban areas or small scenes, and implementation of OCNN method is still needed for large-area classification over heterogeneous landscape. Additionally, the massive labeling of segmented objects requires a practical approach for less computation, including object analysis and multiple CNNs. This study presents a new multiscale OCNN (multi-OCNN) framework for large-scale land cover classification at 1-m resolution over 145,740 km2. Our approach consists of three main steps: (i) image segmentation, (ii) object analysis with skeleton-based algorithm, and (iii) application of multiple CNNs for final classification. Also, we developed a large benchmark dataset, called IowaNet, with 1 million labeled images and 10 classes. In our approach, multiscale CNNs were trained to capture the best contextual information during the semantic labeling of objects. Meanwhile, skeletonization algorithm provided morphological representation (“medial axis”) of objects to support the selection of convolutional locations for CNN predictions. In general, proposed multi-OCNN presented better classification accuracy (overall accuracy ~87.2%) compared to traditional patch-based CNN (81.6%) and fixed-input OCNN (82%). In addition, the results showed that this framework is 8.1 and 111.5 times faster than traditional pixel-wise CNN16 or CNN256, respectively. Multiple CNNs and object analysis have proved to be essential for accurate and fast classification. While multi-OCNN produced a high-level of spatial details in the land cover product, misclassification was observed for some classes, such as road versus buildings or shadow versus lake. Despite these minor drawbacks, our results also demonstrated the benefits of IowaNet training dataset in the model performance; overfitting process reduces as the number of samples increases. The limitations of multi-OCNN are partially explained by segmentation quality and limited number of spectral bands in the aerial data. With the advance of deep learning methods, this study supports the claim of multi-OCNN benefits for operational large-scale land cover product at 1-m resolution.

Standardized object-based dual CNNs for very high-resolution remote sensing image classification and standardization combination effect analysis

ABSTRACT Advances in the object-based convolutional neural network (CNN) have demonstrated the superiority of CNNs for image classification. However, any object-based CNN, regardless of its model structure, only stacks the square images with different scales when extracting features. The impact of background information around the segmented object (the number of pixels around the segmented object) for the classification accuracy is neglected. In addition, blurred object boundaries and feature representation, as well as huge computational redundancy, restrict the application for very high-resolution remote sensing image (VHRI) classification. To solve these problems, a novel standardized object-based dual CNN (SOD-CNN) is proposed for VHRI classification. First, based on geographic object-based image analysis, the image is segmented into homogeneous regions. Second, these less-segmented objects are over-segmented into superpixels with high compactness to provide crisp and accurate boundary delineation at the pixel level. Third, four standardization methods are developed to limit the number of pixels around the segmented object. The standardized less-segmented object and over-segmented object are fed into two different CNNs to capture different perspectives of features at local and global scales. Finally, feature fusion based on the full connection is performed to integrate the class-specific classification results. The effectiveness of the proposed method was verified by using two VHRI, which achieved excellent classification accuracy, consistently outperforming the benchmark comparisons. The overall and per-class classification accuracy was investigated under different standardization combinations. We found that (1) the proposed standardization method not only reduced redundancy of information in the object-based CNN but also highlighted the features of segmented objects; (2) different segmented objects had different optimal standardization combinations; and (3) the classification accuracy was reasonably controlled by the foundation number of training samples.

A multi-level context-guided classification method with object-based convolutional neural network for land cover classification using very high resolution remote sensing images

Classification of very high resolution imagery (VHRI) is challenging due to the difficulty in mining complex spatial and spectral patterns from rich image details. Various object-based Convolutional Neural Networks (OCNN) for VHRI classification have been proposed to overcome the drawbacks of the redundant pixel-wise CNNs, owing to their low computational cost and fine contour-preserving. However, classification performance of OCNN is still limited by geometric distortions, insufficient feature representation, and lack of contextual guidance. In this paper, an innovative multi-level context-guided classification method with the OCNN (MLCG-OCNN) is proposed. A feature-fusing OCNN, including the object contour-preserving mask strategy with the supplement of object deformation coefficient, is developed for accurate object discrimination by learning simultaneously high-level features from independent spectral patterns, geometric characteristics, and object-level contextual information. Then pixel-level contextual guidance is used to further improve the per-object classification results. The MLCG-OCNN method is intentionally tested on two validated small image datasets with limited training samples, to assess the performance in applications of land cover classification where a trade-off between time-consumption of sample training and overall accuracy needs to be found, as it is very common in the practice. Compared with traditional benchmark methods including the patch-based per-pixel CNN (PBPP), the patch-based per-object CNN (PBPO), the pixel-wise CNN with object segmentation refinement (PO), semantic segmentation U-Net (U-NET), and DeepLabV3+(DLV3+), MLCG-OCNN method achieves remarkable classification performance (> 80 %). Compared with the state-of-the-art architecture DeepLabV3+, the MLCG-OCNN method demonstrates high computational efficiency for VHRI classification (4–5 times faster).

High-Resolution Imagery Classification Based on Different Levels of Information

Detailed land use and land cover (LULC) information is one of the important information for land use surveys and applications related to the earth sciences. Therefore, LULC classification using very-high resolution remotely sensed imagery has been a hot issue in the remote sensing community. However, it remains a challenge to successfully extract LULC information from very-high resolution remotely sensed imagery, due to the difficulties in describing the individual characteristics of various LULC categories using single level features. The traditional pixel-wise or spectral-spatial based methods pay more attention to low-level feature representations of target LULC categories. In addition, deep convolutional neural networks offer great potential to extract high-level features to describe objects and have been successfully applied to scene understanding or classification. However, existing studies has paid little attention to constructing multi-level feature representations to better understand each category. In this paper, a multi-level feature representation framework is first designed to extract more robust feature representations for the complex LULC classification task using very-high resolution remotely sensed imagery. To this end, spectral reflection and morphological and morphological attribute profiles are used to describe the pixel-level and neighborhood-level information. Furthermore, a novel object-based convolutional neural networks (CNN) is proposed to extract scene-level information. The object-based CNN method combines advantages of object-based method and CNN method and can perform multi-scale analysis at the scene level. Then, the random forest method is employed to carry out the final classification using the multi-level features. The proposed method was validated on three challenging remotely sensed imageries including a hyperspectral image and two multispectral images with very-high spatial resolution, and achieved excellent classification performances.

A hybrid OSVM-OCNN Method for Crop Classification from Fine Spatial Resolution Remotely Sensed Imagery

Accurate information on crop distribution is of great importance for a range of applications including crop yield estimation, greenhouse gas emission measurement and management policy formulation. Fine spatial resolution (FSR) remotely sensed imagery provides new opportunities for crop mapping at a detailed level. However, crop classification from FSR imagery is known to be challenging due to the great intra-class variability and low inter-class disparity in the data. In this research, a novel hybrid method (OSVM-OCNN) was proposed for crop classification from FSR imagery, which combines a shallow-structured object-based support vector machine (OSVM) with a deep-structured object-based convolutional neural network (OCNN). Unlike pixel-wise classification methods, the OSVM-OCNN method operates on objects as the basic units of analysis and, thus, classifies remotely sensed images at the object level. The proposed OSVM-OCNN harvests the complementary characteristics of the two sub-models, the OSVM with effective extraction of low-level within-object features and the OCNN with capture and utilization of high-level between-object information. By using a rule-based fusion strategy based primarily on the OCNN’s prediction probability, the two sub-models were fused in a concise and effective manner. We investigated the effectiveness of the proposed method over two test sites (i.e., S1 and S2) that have distinctive and heterogeneous patterns of different crops in the Sacramento Valley, California, using FSR Synthetic Aperture Radar (SAR) and FSR multispectral data, respectively. Experimental results illustrated that the new proposed OSVM-OCNN approach increased markedly the classification accuracy for most of crop types in S1 and all crop types in S2, and it consistently achieved the most accurate accuracy in comparison with its two object-based sub-models (OSVM and OCNN) as well as the pixel-wise SVM (PSVM) and CNN (PCNN) methods. Our findings, thus, suggest that the proposed method is as an effective and efficient approach to solve the challenging problem of crop classification using FSR imagery (including from different remotely sensed platforms). More importantly, the OSVM-OCNN method is readily generalisable to other landscape classes and, thus, should provide a general solution to solve the complex FSR image classification problem.

Exploring semantic elements for urban scene recognition: Deep integration of high-resolution imagery and OpenStreetMap (OSM)

Urban scenes refer to city blocks which are basic units of megacities, they play an important role in citizens’ welfare and city management. Remote sensing imagery with largescale coverage and accurate target descriptions, has been regarded as an ideal solution for monitoring the urban environment. However, due to the heterogeneity of remote sensing images, it is difficult to access their geographical content at the object level, let alone understanding urban scenes at the block level. Recently, deep learning-based strategies have been applied to interpret urban scenes with remarkable accuracies. However, the deep neural networks require a substantial number of training samples which are hard to satisfy, especially for high-resolution images. Meanwhile, the crowed-sourced Open Street Map (OSM) data provides rich annotation information about the urban targets but may encounter the problem of insufficient sampling (limited by the places where people can go). As a result, the combination of OSM and remote sensing images for efficient urban scene recognition is urgently needed. In this paper, we present a novel strategy to transfer existing OSM data to high-resolution images for semantic element determination and urban scene understanding. To be specific, the object-based convolutional neural network (OCNN) can be utilized for geographical object detection by feeding it rich semantic elements derived from OSM data. Then, geographical objects are further delineated into their functional labels by integrating points of interest (POIs), which contain rich semantic terms, such as commercial or educational labels. Lastly, the categories of urban scenes are easily acquired from the semantic objects inside. Experimental results indicate that the proposed method has an ability to classify complex urban scenes. The classification accuracies of the Beijing dataset are as high as 91% at the object-level and 88% at the scene level. Additionally, we are probably the first to investigate the object level semantic mapping by incorporating high-resolution images and OSM data of urban areas. Consequently, the method presented is effective in delineating urban scenes that could further boost urban environment monitoring and planning with high-resolution images.

Object-Based Convolutional Neural Networks for Cloud and Snow Detection in High-Resolution Multispectral Imagers

Cloud and snow detection is one of the most significant tasks for remote sensing image processing. However, it is a challenging task to distinguish between clouds and snow in high-resolution multispectral images due to their similar spectral distributions. The shortwave infrared band (SWIR, e.g., Sentinel-2A 1.55–1.75 µm band) is widely applied to the detection of snow and clouds. However, high-resolution multispectral images have a lack of SWIR, and such traditional methods are no longer practical. To solve this problem, a novel convolutional neural network (CNN) to classify cloud and snow on an object level is proposed in this paper. Specifically, a novel CNN structure capable of learning cloud and snow multiscale semantic features from high-resolution multispectral imagery is presented. In order to solve the shortcoming of “salt-and-pepper” in pixel level predictions, we extend a simple linear iterative clustering algorithm for segmenting high-resolution multispectral images and generating superpixels. Results demonstrated that the new proposed method can with better precision separate the cloud and snow in the high-resolution image, and results are more accurate and robust compared to the other methods.

Land Cover Mapping with Higher Order Graph-Based Co-Occurrence Model

Deep learning has become a standard processing procedure in land cover mapping for remote sensing images. Instead of relying on hand-crafted features, deep learning algorithms, such as Convolutional Neural Networks (CNN) can automatically generate effective feature representations, in order to recognize objects with complex image patterns. However, the rich spatial information still remains unexploited, since most of the deep learning algorithms only focus on small image patches that overlook the contextual information at larger scales. To utilize these contextual information and improve the classification performance for high-resolution imagery, we propose a graph-based model in order to capture the contextual information over semantic segments of the image. First, we explore semantic segments which build on the top of deep features and obtain the initial classification result. Then, we further improve the initial classification results with a higher-order co-occurrence model by extending the existing conditional random field (HCO-CRF) algorithm. Compared to the pixel- and object-based CNN methods, the proposed model achieved better performance in terms of classification accuracy.

An object-based convolutional neural network (OCNN) for urban land use classification

Urban land use information is essential for a variety of urban-related applications such as urban planning and regional administration. The extraction of urban land use from very fine spatial resolution (VFSR) remotely sensed imagery has, therefore, drawn much attention in the remote sensing community. Nevertheless, classifying urban land use from VFSR images remains a challenging task, due to the extreme difficulties in differentiating complex spatial patterns to derive high-level semantic labels. Deep convolutional neural networks (CNNs) offer great potential to extract high-level spatial features, thanks to its hierarchical nature with multiple levels of abstraction. However, blurred object boundaries and geometric distortion, as well as huge computational redundancy, severely restrict the potential application of CNN for the classification of urban land use. In this paper, a novel object-based convolutional neural network (OCNN) is proposed for urban land use classification using VFSR images. Rather than Pixel-wise convolutional processes, the OCNN relies on segmented objects as its functional units, and CNN networks are used to analyse and label objects such as to partition within-object and between-object variation. Two CNN networks with different model structures and window sizes are developed to predict linearly shaped objects (e.g. Highway, Canal) and general (other non-linearly shaped) objects. Then a rule-based decision fusion is performed to integrate the class-specific classification results. The effectiveness of the proposed OCNN method was tested on aerial photography of two large urban scenes in Southampton and Manchester in Great Britain. The OCNN combined with large and small window sizes achieved excellent classification accuracy and computational efficiency, consistently outperforming its sub-modules, as well as other benchmark comparators, including the Pixel-wise CNN, contextual-based MRF and object-based OBIA-SVM methods. The proposed method provides the first object-based CNN framework to effectively and efficiently address the complicated problem of urban land use classification from VFSR images.