Approach For Scene Classification Research Articles

Weighted ensemble approach for smoke-like scene classification in remote sensing images

Weighted ensemble approach for smoke-like scene classification in remote sensing images

A deep learning-based global and segmentation-based semantic feature fusion approach for indoor scene classification

This work proposes a novel approach that uses a semantic segmentation mask to obtain a 2D spatial layout of the segmentation-categories across the scene, designated by segmentation-based semantic features (SSFs). These features represent, per segmentation-category, the pixel count, as well as the 2D average position and respective standard deviation values. Moreover, a two-branch network, GS2F2App, that exploits CNN-based global features extracted from RGB images and the segmentation-based features extracted from the proposed SSFs, is also proposed. GS2F2App was evaluated in two indoor scene benchmark datasets: the SUN RGB-D and the NYU Depth V2, achieving state-of-the-art results on both datasets.

Deep learning techniques for remote sensing image scene classification: A comprehensive review, current challenges, and future directions

SummarySince last decade, deep learning has made exceptional progress in various fields of artificial intelligence including image and voice recognition, natural language processing. Inspired by these successes, researchers are now applying deep learning techniques to classification of scenes in remote sensing images. The purpose of remote sensing image scene classification is to classify remote sensing scenes according to their content. These images display a complex structure due to the variety of landforms as well as the distance between the image collection instrument and earth. In our review, we discussed 76 relevant papers published on this topic over the past 6 years. The review conducts a comparison analysis based on the overall accuracy parameter to provide insight into the effectiveness of different methods on different proportions of the dataset. The five classes of techniques we describe are convolutional neural networks, autoencoders, generative adversarial networks, vision transformers, and few‐shot learning. Future directions are discussed in this review in order to enhance the effectiveness of deep learning‐based scene classification approaches. This article concludes with an overview of the proposed method to enhance the accuracy in classifying remote sensing images.

DASFNet: Dense-Attention–Similarity-Fusion Network for scene classification of dual-modal remote-sensing images

• The cosine similarity and a gating mechanism are used by the SFM to fuse IRRG and DSM data and extract corresponding complementary features. • The MCFAM is used to extract multiscale context cues, solving the problem of object-scale variations in HRRSIs. • The DAMM forms a dense-attention branch that can adaptively capture both low- and high-level attentions. Although significant progress has been made in scene classification of high-resolution remote-sensing images (HRRSIs), dual-modal HRRSI scene classification is still an active and challenging issue. In this study, we introduce an end-to-end dense-attention–similarity-fusion network (DASFNet) for dual-modal HRRSIs. Specifically, we propose a dense-attention map module based on graph convolution, which adaptively captures long-range semantic cues and further directs shallow-attention cues to the deep level to guide the generation of high-level feature attention cues. At the encoder stage, DASFNet uses feature similarity to explore the correlation between dual-modal features; a similarity-fusion module extracts complementary information by fusing features from different modalities. A multiscale context-feature-aggregation module is used to strengthen the feature embedding of any two spatial scales; this solves the of scale change problem. A large number of experiments on two HRRSI benchmark datasets for scene classification indicate that the proposed DASFNet outperforms the outstanding scene classification approaches.

Multi-Scale Graph-Based Feature Fusion for Few-Shot Remote Sensing Image Scene Classification

Remote sensing image scene classification has drawn extensive attention for its wide application in various scenarios. Scene classification in many practical cases faces the challenge of few-shot conditions. The major difficulty of few-shot remote sensing image scene classification is how to extract effective features from insufficient labeled data. To solve these issues, a multi-scale graph-based feature fusion (MGFF) model is proposed for few-shot remote sensing image scene classification. In the MGFF model, a graph-based feature construction model is developed to transform traditional image features into graph-based features, which aims to effectively represent the spatial relations among images. Then, a graph-based feature fusion model is proposed to integrate graph-based features of multiple scales, which aims to enhance sample discrimination based on different scale information. Experimental results on two public remote sensing datasets prove that the MGFF model can achieve superior accuracy than other few-shot scene classification approaches.

Overview on remote sensing image scene classification using deep learning

Now a days, deep learning is becoming very famous dueto its power fulability of learning. Deep learning has been applied for remote sensing image analysis tasks including scene classification. Remote sensing has seen an enormous increase in the enhancement of digital images captured from satellites that cover almost all angle of the surface of the earth. This growth in data has forced the community of the remote sensing to apply deep learning algorithms to solve different remote sensing tasks. Convolutional neural networks have obtained huge success in the field of scene classification. The current best scene classification approaches are still far from achieving the human’s ability to classify scene types with a few labelled samples. This study provides a systematic survey of deep learning methods for remote sensing image scene classification by covering mover than 9papers.Inthispaper, we provide a brief discussion on Convolutional neural networks. Here we also discussed few-shot learning algorithm to overcome the gap between the current understanding level of machine and human-level performance.

Multi-View Urban Scene Classification with a Complementary-Information Learning Model

Traditional urban scene-classification approaches focus on images taken either by satellite or in aerial view. Although single-view images are able to achieve satisfactory results for scene classification in most situations, the complementary information provided by other image views is needed to further improve performance. Therefore, we present a complementary information-learning model (CILM) to perform multi-view scene classification of aerial and ground-level images. Specifically, the proposed CILM takes aerial and ground-level image pairs as input to learn view-specific features for later fusion to integrate the complementary information. To train CILM, a unified loss consisting of cross entropy and contrastive losses is exploited to force the network to be more robust. Once CILM is trained, the features of each view are extracted via the two proposed feature-extraction scenarios and then fused to train the support vector machine classifier for classification. The experimental results on two publicly available benchmark data sets demonstrate that CILM achieves remarkable performance, indicating that it is an effective model for learning complementary information and thus improving urban scene classification.

Local climate zone classification using a multi-scale, multi-level attention network

Local Climate Zones (LCZ) offer a climate-aware and standardized classification scheme composed of 17 urban and natural landscape classes. Recent deep learning-based LCZ classification studies have adopted a scene classification approach with computer vision-inspired models. In light of these advancements, this study introduces a multi-scale, multi-level attention network (MSMLA-Net) for deep learning-based LCZ classification. MSMLA-Net integrates a multi-scale (MS) module to generate multi-scale features from the input data and a novel multi-level attention (MLA) module as a branch unit from the model’s main ResNet backbone. MLA uses the convolutional block attention module (CBAM) at multiple stages to generate multi-level spatially and spectrally enhanced features for context aggregation. This study presents comprehensive model-based experiments on model depth, the individual and combined influence of MS and MLA modules, and the addition of attention mechanisms. Furthermore, data-based experiments are conducted to determine optimal Sentinel-2 spectral bands, while OpenStreetMap (OSM) building data, ALOS World 3D DSM height information, and a national land cover map are included as ancillary bands. LCZ classification tests are conducted on six major cities in South Korea. With regards to model-based performance, MSMLA-Net was built using a modified SE-ResNet50 backbone (MSMLA-50) and obtained the best classification results using 48 by 48-pixel input patches with a combination of all Sentinel-2 and ancillary bands, outperforming three state-of-the-art LCZ classification models. In particular, only MSMLA-50 reached over 70% built-up overall accuracy and maintained high accuracy even when tested on completely unseen areas in a “citywise” leave-one-out sampling strategy. For data-based results, the combination of Sentinel-2 Red Edge bands were helpful, mainly due to the greater number of available bands. Training only ancillary data generated up to 75.0% for built-up overall accuracy, albeit at the cost of low natural overall accuracy. Using all available bands and ancillary data produced the best results, but a combination of only OSM and Sentinel-2 bands also generated comparable accuracy. Ultimately, the proposed MSMLA-Net bridges advanced computer vision techniques such as attention mechanisms and multi-level context aggregation to improve LCZ classification. The MS and MLA modules can be applied on different backbones, sophisticated sampling schemes, and data inputs to flexibly achieve improved classification results.

Insights into few shot learning approaches for image scene classification.

Image understanding and scene classification are keystone tasks in computer vision. The development of technologies and profusion of existing datasets open a wide room for improvement in the image classification and recognition research area. Notwithstanding the optimal performance of exiting machine learning models in image understanding and scene classification, there are still obstacles to overcome. All models are data-dependent that can only classify samples close to the training set. Moreover, these models require large data for training and learning. The first problem is solved by few-shot learning, which achieves optimal performance in object detection and classification but with a lack of eligible attention in the scene classification task. Motivated by these findings, in this paper, we introduce two models for few-shot learning in scene classification. In order to trace the behavior of those models, we also introduce two datasets (MiniSun; MiniPlaces) for image scene classification. Experimental results show that the proposed models outperform the benchmark approaches in respect of classification accuracy.

An adaptive multilayer feature fusion strategy for remote sensing scene classification

ABSTRACT Remote sensing scene classification (RSSC) is one of the fundamental and challenging tasks for remote-sensing understanding and interpretation. How to construct discriminative features is crucial for scene classification. It is generally acknowledged that integrating convolutional features from different layers can significantly improve scene classification performance. However, some existing methods directly concatenate the high-layer and low-layer features without considering feature redundancy, semantic ambiguity and background noise, resulting in sub-optimal performance. In this paper, inspired by the attention mechanism, we propose a simple but effective strategy to fuse different convolutional features after feature selection operation, instead of directing concatenation. The basic idea of the feature selection operation is that we need to fuse those low-layer features, which are consistent with high-layer semantics. In particular, the proposed strategy is flexible and can be embedded into many neural architectures. Experimental results on two benchmark datasets demonstrate that the proposed method can obtain more valuable features and achieve competitive performance than other scene classification approaches.

BLAINDER-A Blender AI Add-On for Generation of Semantically Labeled Depth-Sensing Data.

Common Machine-Learning (ML) approaches for scene classification require a large amount of training data. However, for classification of depth sensor data, in contrast to image data, relatively few databases are publicly available and manual generation of semantically labeled 3D point clouds is an even more time-consuming task. To simplify the training data generation process for a wide range of domains, we have developed the BLAINDER add-on package for the open-source 3D modeling software Blender, which enables a largely automated generation of semantically annotated point-cloud data in virtual 3D environments. In this paper, we focus on classical depth-sensing techniques Light Detection and Ranging (LiDAR) and Sound Navigation and Ranging (Sonar). Within the BLAINDER add-on, different depth sensors can be loaded from presets, customized sensors can be implemented and different environmental conditions (e.g., influence of rain, dust) can be simulated. The semantically labeled data can be exported to various 2D and 3D formats and are thus optimized for different ML applications and visualizations. In addition, semantically labeled images can be exported using the rendering functionalities of Blender.

Compact Deep Color Features for Remote Sensing Scene Classification

Aerial scene classification is a challenging problem in understanding high-resolution remote sensing images. Most recent aerial scene classification approaches are based on Convolutional Neural Networks (CNNs). These CNN models are trained on a large amount of labeled data and the de facto practice is to use RGB patches as input to the networks. However, the importance of color within the deep learning framework is yet to be investigated for aerial scene classification. In this work, we investigate the fusion of several deep color models, trained using color representations, for aerial scene classification. We show that combining several deep color models significantly improves the recognition performance compared to using the RGB network alone. This improvement in classification performance is, however, achieved at the cost of a high-dimensional final image representation. We propose to use an information theoretic compression approach to counter this issue, leading to a compact deep color feature set without any significant loss in accuracy. Comprehensive experiments are performed on five remote sensing scene classification benchmarks: UC-Merced with 21 scene classes, WHU-RS19 with 19 scene types, RSSCN7 with 7 categories, AID with 30 aerial scene classes, and NWPU-RESISC45 with 45 categories. Our results clearly demonstrate that the fusion of deep color features always improves the overall classification performance compared to the standard RGB deep features. On the large-scale NWPU-RESISC45 dataset, our deep color features provide a significant absolute gain of 4.3% over the standard RGB deep features.

SceneNet: Remote sensing scene classification deep learning network using multi-objective neural evolution architecture search

The scene classification approaches using deep learning have been the subject of much attention for remote sensing imagery. However, most deep learning networks have been constructed with a fixed architecture for natural image processing, and they are difficult to apply directly to remote sensing images, due to the more complex geometric structural features. Thus, there is an urgent need for automatic search for the most suitable neural network architecture from the image data in scene classification, in which a powerful search mechanism is required, and the computational complexity and performance error of the searched network should be balanced for a practical choice. In this article, a framework for scene classification network architecture search based on multi-objective neural evolution (SceneNet) is proposed. In SceneNet, the network architecture coding and searching are achieved using an evolutionary algorithm, which can implement a more flexible hierarchical extraction of the remote sensing image scene information. Moreover, the computational complexity and the performance error of the searched network are balanced by employing the multi-objective optimization method, and the competitive neural architectures are obtained in a Pareto solution set. The effectiveness of SceneNet is demonstrated by experimental comparisons with several deep neural networks designed by human experts.

Remote-Sensing Image Scene Classification With Deep Neural Networks in JPEG 2000 Compressed Domain

To reduce the storage requirements, remote sensing (RS) images are usually stored in compressed format. Existing scene classification approaches using deep neural networks (DNNs) require to fully decompress the images, which is a computationally demanding task in operational applications. To address this issue, in this paper we propose a novel approach to achieve scene classification in JPEG 2000 compressed RS images. The proposed approach consists of two main steps: i) approximation of the finer resolution sub-bands of reversible biorthogonal wavelet filters used in JPEG 2000; and ii) characterization of the high-level semantic content of approximated wavelet sub-bands and scene classification based on the learnt descriptors. This is achieved by taking codestreams associated with the coarsest resolution wavelet sub-band as input to approximate finer resolution sub-bands using a number of transposed convolutional layers. Then, a series of convolutional layers models the high-level semantic content of the approximated wavelet sub-band. Thus, the proposed approach models the multiresolution paradigm given in the JPEG 2000 compression algorithm in an end-to-end trainable unified neural network. In the classification stage, the proposed approach takes only the coarsest resolution wavelet sub-bands as input, thereby reducing the time required to apply decoding. Experimental results performed on two benchmark aerial image archives demonstrate that the proposed approach significantly reduces the computational time with similar classification accuracies when compared to traditional RS scene classification approaches (which requires full image decompression).

Semantic Fisher Scores for Task Transfer: Using Objects to Classify Scenes.

The transfer of a neural network (CNN) trained to recognize objects to the task of scene classification is considered. A Bag-of-Semantics (BoS) representation is first induced, by feeding scene image patches to the object CNN, and representing the scene image by the ensuing bag of posterior class probability vectors (semantic posteriors). The encoding of the BoS with a Fisher vector (FV) is then studied. A link is established between the FV of any probabilistic model and the Q-function of the expectation-maximization (EM) algorithm used to estimate its parameters by maximum likelihood. This enables 1) immediate derivation of FVs for any model for which an EM algorithm exists, and 2) leveraging efficient implementations from the EM literature for the computation of FVs. It is then shown that standard FVs, such as those derived from Gaussian or even Dirichlet mixtures, are unsuccessful for the transfer of semantic posteriors, due to the highly non-linear nature of the probability simplex. The analysis of these FVs shows that significant benefits can ensue by 1) designing FVs in the natural parameter space of the multinomial distribution, and 2) adopting sophisticated probabilistic models of semantic feature covariance. The combination of these two insights leads to the encoding of the BoS in the natural parameter space of the multinomial, using a vector of Fisher scores derived from a mixture of factor analyzers (MFA). A network implementation of the MFA Fisher Score (MFA-FS), denoted as the MFAFSNet, is finally proposed to enable end-to-end training. Experiments with various object CNNs and datasets show that the approach has state-of-the-art transfer performance. Somewhat surprisingly, the scene classification results are superior to those of a CNN explicitly trained for scene classification, using a large scene dataset (Places). This suggests that holistic analysis is insufficient for scene classification. The modeling of local object semantics appears to be at least equally important. The two approaches are also shown to be strongly complementary, leading to very large scene classification gains when combined, and outperforming all previous scene classification approaches by a sizable margin.

A novel approach for scene classification from remote sensing images using deep learning methods

ABSTRACT Deep learning plays a major task in classification of unsupervised data, which utilises network enhanced learning. This technique proves to be powerful in remote sensing (RS) field for spatial data classification. In the existing environment, huge amounts of data from the earth observation satellites known as satellite images time series (SITS) are gathered, which can be utilised for observing the areas related to geography over through time. In this proposed model the time series model utilised is based on geography. There exists a challenge on how these types of information can be analysed in the field of remote sensing. Notable, techniques related to deep learning substantiated in dealing with remote sensing usually for classification of scene . In this paper, we propose an enhanced classification method involving Recurrent Neural Network (RNN) along with Random forest (RF) for land classification using satellite images, which are publicly available for various research purposes. We utilised spatial data gathered from the satellite images (i.e. time series). Our experimental classification is based on pixel and object-based classification. The attained analysis illustrates that the proposed model outperforms the other present day remote sensing classification techniques by producing 87% target accuracy of classification scene from satellite images.

A comprehensive system for image scene classification

In recent years, image scene classification based on low/high-level features has been considered as one of the most important and challenging problems faced in image processing research. The high-level features based on semantic concepts present a more accurate and closer model to the human perception of the image scene content. This paper presents a new multi-stage approach for image scene classification based on high-level semantic features extracted from image content. In the first stage, the object boundaries and their labels that represent the content are extracted. For this purpose, a combined method of a fully convolutional deep network and a combined network of a two-class SVM-fuzzy and SVR are used. Topic modeling is used to represent the latent relationships between the objects. Hence in the second stage, a new combination of methods consisting of the bag of visual words, and supervised document neural autoregressive distribution estimator is used to extract the latent topics (topic modeling) in the image. Finally, classification based on Bayesian method is performed according to the extracted features of the deep network, objects labels and the latent topics in the image. The proposed method has been evaluated on three datasets: Scene15, UIUC Sports, and MIT-67 Indoor. The experimental results show that the proposed approach achieves average performance improvement of 12%, 11% and 14% in the accuracy of object detection, and 0.5%, 0.6% and 1.8% in the mean average precision criteria of the image scene classification, compared to the previous state-of-the-art methods on these three datasets.

Contextual Classification of Sea-Ice Types Using Compact Polarimetric SAR Data

Automatic classification methods using satellite imagery are beneficial in the sea-ice-type mapping of the Arctic regions. In the near future, the RADARSAT Constellation Mission (RCM) will be launched, providing unique compact polarimetric (CP) synthetic aperture radar (SAR) data, expected to be an improvement over the current RADARSAT-2 dual-polarimetric SAR imagery. This motivates the implementation of a CP-dedicated automatic scene classification approach. First, an existing unsupervised segmentation algorithm called iterative region growing using semantics (IRGS) is used to segment ice-class homogeneous regions to reduce the impact of speckle noise. Second, a support vector machine (SVM) is used to classify the ice-type labels for each homogeneous region. Two complex quad-polarimetric RADARSAT-2 scenes are used to mathematically simulate the corresponding CP scenes for algorithm testing. Classification accuracy shows that using only the two CP intensity images leads to improved results compared with standard dual-polarimetric scenes. Using the CP data, the best classification results are obtained with the reconstructed QP data for the IRGS segmentation and all derived CP features for the SVM labeling. The results support the expected potential that CP scenes will provide improved sea-ice classification than the current operational dual-pol scenes.

Spectrotemporal Analysis Using Local Binary Pattern Variants for Acoustic Scene Classification

In this paper, we present an approach for acoustic scene classification, which aggregates spectral and temporal features. We do this by proposing the first use of the variable-Q transform (VQT) to generate the time–frequency representation for acoustic scene classification. The VQT provides finer control over the resolution compared to the constant-Q transform (CQT) or short time fourier transform and can be tuned to better capture acoustic scene information. We then adopt a variant of the local binary pattern (LBP), the adjacent evaluation completed LBP (AECLBP), which is better suited to extracting features from acoustic time–frequency images. Our results yield a 5.2% improvement on the DCASE 2016 dataset compared to the application of standard CQT with LBP. Fusing our proposed AECLBP with HOG features, we achieve a classification accuracy of 85.5%, which outperforms one of the top performing systems.

Acoustic Scene Classification Using Efficient Summary Statistics and Multiple Spectro-Temporal Descriptor Fusion

This paper presents a novel approach for acoustic scene classification based on efficient acoustic feature extraction using spectro-temporal descriptors fusion. Grounded on the finding in neuroscience—“auditory system summarizes the temporal details of sounds using time-averaged statistics to understand acoustic scenes”, we devise an efficient computational framework for sound scene classification by using multipe time-frequency descriptors fusion with discriminant information enhancement. To characterize rich information of sound, i.e., local structures on the time-frequency plane, we adopt 2-dimensional local descriptors. A more critical issue raised in how to logically ‘summarize’ those local details into a compact feature vector for scene classification. Although ‘time-averaged statistics’ is suggested by the psychological investigation, directly computing time average of local acoustic features is not a logical way, since arithmetic mean is vulnerable to extreme values which are anticipated to be generated by interference sounds which are irrelevant to the scene category. To tackle this problem, we develop time-frame weighting approach to enhance sound textures as well as to suppress scene-irrelevant events. Subsequently, robust acoustic feature for scene classification can be efficiently characterized. The proposed method had been validated by using Rouen dataset which consists of 19 acoustic scene categories with 3029 real samples. Extensive results demonstrated the effectiveness of the proposed scheme.