Natural Scene Categorization Research Articles

The Brain's Sensitivity to Real-world Statistical Regularity Does Not Require Full Attention.

Predictive coding accounts of perception state that the brain generates perceptual predictions in the service of processing incoming sensory data. These predictions are hypothesized to be afforded by the brain's ability to internalize useful patterns, that is, statistical regularities, from the environment. We have previously argued that the N300 ERP component serves as an index of the brain's use of representations of (real-world) statistical regularities. However, we do not yet know whether overt attention is necessary in order for this process to engage. We addressed this question by presenting stimuli of either high or low real-world statistical regularity in terms of their representativeness (good/bad exemplars of natural scene categories) to participants who either fully attended the stimuli or were distracted by another task (attended/distracted conditions). Replicating past work, N300 responses were larger to bad than to good scene exemplars, and furthermore, we demonstrate minimal impacts of distraction on N300 effects. Thus, it seems that overtly focused attention is not required to maintain the brain's sensitivity to real-world statistical regularity. Furthermore, in an exploratory analysis, we showed that providing additional, artificial regularities, formed by altering the proportions of good and bad exemplars within blocks, further enhanced the N300 effect in both attended and distracted conditions, shedding light on the relationship between statistical regularities learned in the real world and those learned within the context of an experiment.

Transformer based ensemble deep learning approach for remote sensing natural scene classification

ABSTRACT Very high resolution (VHR) remote sensing (RS) image classification is paramount for detailed Earth’s surface analysis. Feature extraction from VHR natural scenes is crucial, but it becomes a challenging task because of the overlapping edges present in images. Multiple open-source datasets exist in the literature to train robust models, and they have been benchmarked using deep learning models. However, different datasets contain different numbers of classes; a few of them could be absent in other datasets because they are independent of the other classes, and the classes are sometimes not mutually exclusive amongst the same dataset. Thus, it is very challenging to generalize a model trained on a single dataset to perform scene classification on unknown classes of multiple benchmark datasets and real-time images. Thus, this work introduces the Remote Sensing Natural Scenes 92 (RS_NS92) dataset, consisting of 36,785 images belonging to 92 classes, curated by selectively taking the union of all subclasses from five benchmark datasets. This class count is significantly higher than publicly available datasets and maintains a low-class imbalance and a comprehensive data distribution for robust model training. It also provides the remote sensing community with an extra platform to validate the performance on multiple benchmarks. Inspired by federated learning, an ensemble approach consisting of three feature extraction backbones: Vision Transformers, Swin Transformers, and ConvNeXt (termed the VSC_Ensemble model) is also introduced. This model can make extraordinary predictions across multiple datasets by finetuning weights using transfer learning. Experimental analysis with the proposed approach not only obtains a high test accuracy of 97.24% and an F1-score of 0.9587 for the 92 classes on a 90:10 split of the proposed benchmark dataset but also gets excellent results on unseen test images of other datasets, which are comparable to the state-of-the-art results.

Transformer-based convolutional neural network approach for remote sensing natural scene classification

Feature extraction in remote sensing is a challenging yet crucial operation for scene classification because of cloud cover and overlapping edges present in the data. Many architectures have been solely used as a backbone for feature extraction in complex computer vision tasks such as object detection and semantic segmentation. Though the remote sensing literature has compared deep learning models for scene classification, the comparison between different transformer-based architectures and convolution-based architectures has not been systematically addressed in the literature. Thus, this work comprehensively analyses different deep learning architectures on multiple scene classification datasets to understand the features and weigh the advantages of one or more functional connections in different convolutional neural networks. It has been done using five open-source benchmark datasets: UC-Merced land use, WHU-RS19, Optimal-31, RSI-CB256, and MLRSNet. Feature extraction for remote sensing natural scene classification is performed using pre-trained ImageNet22 k weights of convolution-based architectures such as VGG-16, ResNet50, EfficientNetB3 and ConvNeXt, and transformer architectures such as Vision transformers (ViT) and Swin transformers. Further, the networks are fine-tuned using the LinBnDrop block from the fastai framework before scene classification using the softmax layer. Our work obtains a new benchmark for all datasets on a 90:10 train-test split. An explanation to understand the use of each architecture based on the available data and its application is discussed in this work. The analysis of 42 experiments conducted in this work will help the research community analyze the scene classification datasets and give them better insights into fine-tuning applications.

Spatiotemporal cortical dynamics for visual scene processing as revealed by EEG decoding.

The human visual system rapidly recognizes the categories and global properties of complex natural scenes. The present study investigated the spatiotemporal dynamics of neural signals involved in visual scene processing using electroencephalography (EEG) decoding. We recorded visual evoked potentials from 11 human observers for 232 natural scenes, each of which belonged to one of 13 natural scene categories (e.g., a bedroom or open country) and had three global properties (naturalness, openness, and roughness). We trained a deep convolutional classification model of the natural scene categories and global properties using EEGNet. Having confirmed that the model successfully classified natural scene categories and the three global properties, we applied Grad-CAM to the EEGNet model to visualize the EEG channels and time points that contributed to the classification. The analysis showed that EEG signals in the occipital electrodes at short latencies (approximately 80 ~ ms) contributed to the classifications, whereas those in the frontal electrodes at relatively long latencies (200 ~ ms) contributed to the classification of naturalness and the individual scene category. These results suggest that different global properties are encoded in different cortical areas and with different timings, and that the combination of the EEGNet model and Grad-CAM can be a tool to investigate both temporal and spatial distribution of natural scene processing in the human brain.

Cross-Domain Transfer Learning for Natural Scene Classification of Remote-Sensing Imagery

Natural scene classification, which has potential applications in precision agriculture, environmental monitoring, and disaster management, poses significant challenges due to variations in the spatial resolution, spectral resolution, texture, and size of remotely sensed images of natural scenes on Earth. For such challenging problems, deep-learning-based algorithms have demonstrated amazing performances in recent years. Among these methodologies, transfer learning is a useful technique which employs the learned features already extracted from the pre-trained models from large-scale datasets for the problem at hand, resulting in quicker and more accurate models. In this study, we deployed cross-domain transfer learning for the land-cover classification of remotely sensed images of natural scenes. We conducted extensive experiments to measure the performance of the proposed method and explored the factors that affect the performance of the models. Our findings suggest that fine-tuning the ResNet-50 model outperforms various other models in terms of the classification accuracy. The experimental results showed that the deployed cross-domain transfer-learning system achieved outstanding (99.5% and 99.1%) accurate performances compared to previous benchmarks on the NaSC-TG2 dataset with the final tuning of the whole structure and only the last three layers, respectively.

Object-based Classification of Natural Scenes Using Machine Learning Methods

The replication of human intellectual processes by machines, particularly computer systems, is known as artificial intelligence (AI). AI is an intelligent tool that is utilized across sectors to improve decision making, increase productivity, and eliminate repetitive tasks. Machine learning (ML) is a key component of AI since it includes understanding and developing ways that can learn or improve performance on tasks. For the last decade, ML has been applied in computer vision (CV) applications. In computer vision, systems and computers extract meaningful data from digital videos, photos, and other visual sources and use that information to conduct actions or make suggestions. In this work, we have solved the image segmentation problem for the natural images to segment out water, land, and sky. Instead of applying image segmentation directly to the images, images are pre-processed, and statistical and textural features are then passed through a neural network for the pixel-wise semantic segmentation of the images. We chose the 5X5 window over the pixel-by-pixel technique since it requires less resources and time for training and testing.

Two-stage natural scene image classification with noise discovering and label-correlation mining

In natural scene classification, it is common that one natural scene image may belong to multiple categories concurrently; as a result, multi-label learning has become a research hotspot. Despite recent rapid developments in multi-label learning, the increasing amount of high-dimensional data poses great challenges—such as redundant features and high computational costs—to conventional multi-label learning models. Most contemporary strategies for dealing with this issue depend on forcing feature learning into multi-label models. Notably, however, these approaches seldom pay attention to the label correlation and propagation in the feature subspace. To address this issue, we introduce an alternative multi-label feature learning solution that incorporates both labeled and unlabeled information. Unlike existing multi-label learning models , which rely on clean and trustworthy training datasets, we argue that in semi-supervised learning scenarios, the unlabeled data can be easily corrupted by noise or outliers, which causes the model performance to degrade. We next extract the label correlation and propagate the label information to discover the noise or outliers. Subsequently, our model adaptively searches the optimal feature subspace to reduce the influence of redundant features for high-dimensional data. The effectiveness of the proposed model is demonstrated by experimental observations on artificial and real-world datasets.

Priming of natural scene categorization during continuous flash suppression

Continuous Flash Suppression (CFS) reduces conscious awareness of stimuli. Whether stimuli suppressed by CFS are processed at categorical or semantic levels is still debated. Here, we approached this question using a large set of indoor and outdoor scene photographs in a priming paradigm. Perceptually suppressed primes were followed by visible targets. Participants rapidly reported whether the targets showed an indoor or an outdoor scene. Responses were faster (and fast responses more accurate) when primes and targets came from a congruent superordinate category (e.g., both were outdoor scenes). During CFS, priming effects were relatively small (up to 10 ms) and modulated by prime visibility and stimulus onset asynchrony (SOA) of prime and target. Without CFS, the stimuli elicited consistent and more robust priming effects (about 24 ms). Our results imply that scene category is processed during CFS, although some residual prime visibility is likely necessary for significant priming effects to occur.

Study of Natural Scene Categories in Measurement of Perceived Image Quality

One challenge facing image quality assessment (IQA) is that current models designed or trained on the basis of exiting databases are intrinsically suboptimal and cannot deal with the real-world complexity and diversity of natural scenes. IQA models and databases are heavily skewed toward the visibility of distortions. It is critical to understand the wider determinants of perceived quality and use the new understanding to improve the predictive power of IQA models. Human behavioral categorization performance is powerful and essential for visual tasks. However, little is known about the impact of natural scene categories (SCs) on perceived image quality. We hypothesize that different classes of natural scenes influence image quality perception—how image quality is perceived is not only affected by the lower level image statistics and image structures shared between different categories but also by the semantic distinctions between these categories. In this article, we first design and conduct a fully controlled psychovisual experiment to verify our hypothesis. Then, we propose a computational framework that integrates the natural SC-specific component into image quality prediction. Research demonstrates the importance and plausibility of considering natural SCs in future IQA databases and models.

Sparse coded spatial pyramid matching and multi-kernel integrated SVM for non-linear scene classification

AbstractSupport vector machine (SVM) techniques and deep learning have been prevalent in object classification for many years. However, deep learning is computation-intensive and can require a long training time. SVM is significantly faster than Convolution Neural Network (CNN). However, the SVM has limited its applications in the mid-size dataset as it requires proper tuning. Recently the parameterization of multiple kernels has shown greater flexibility in the characterization of the dataset. Therefore, this paper proposes a sparse coded multi-scale approach to reduce training complexity and tuning of SVM using a non-linear fusion of kernels for large class natural scene classification. The optimum features are obtained by parameterizing the dictionary, Scale Invariant Feature Transform (SIFT) parameters, and fusion of multiple kernels. Experiments were conducted on a large dataset to examine the multi-kernel space capability to find distinct features for better classification. The proposed approach founds to be promising than the linear multi-kernel SVM approaches achieving 91.12 % maximum accuracy.

Do Humans and Deep Convolutional Neural Networks Use Visual Information Similarly for the Categorization of Natural Scenes?

The investigation of visual categorization has recently been aided by the introduction of deep convolutional neural networks (CNNs), which achieve unprecedented accuracy in picture classification after extensive training. Even if the architecture of CNNs is inspired by the organization of the visual brain, the similarity between CNN and human visual processing remains unclear. Here, we investigated this issue by engaging humans and CNNs in a two‐class visual categorization task. To this end, pictures containing animals or vehicles were modified to contain only low/high spatial frequency (HSF) information, or were scrambled in the phase of the spatial frequency spectrum. For all types of degradation, accuracy increased as degradation was reduced for both humans and CNNs; however, the thresholds for accurate categorization varied between humans and CNNs. More remarkable differences were observed for HSF information compared to the other two types of degradation, both in terms of overall accuracy and image‐level agreement between humans and CNNs. The difficulty with which the CNNs were shown to categorize high‐passed natural scenes was reduced by picture whitening, a procedure which is inspired by how visual systems process natural images. The results are discussed concerning the adaptation to regularities in the visual environment (scene statistics); if the visual characteristics of the environment are not learned by CNNs, their visual categorization may depend only on a subset of the visual information on which humans rely, for example, on low spatial frequency information.

The N300: An Index for Predictive Coding of Complex Visual Objects and Scenes.

Predictive coding models can simulate known perceptual or neuronal phenomena, but there have been fewer attempts to identify a reliable neural signature of predictive coding for complex stimuli. In a pair of studies, we test whether the N300 component of the event-related potential, occurring 250–350-ms poststimulus-onset, has the response properties expected for such a signature of perceptual hypothesis testing at the level of whole objects and scenes. We show that N300 amplitudes are smaller to representative (“good exemplars”) compared with less representative (“bad exemplars”) items from natural scene categories. Integrating these results with patterns observed for objects, we establish that, across a variety of visual stimuli, the N300 is responsive to statistical regularity, or the degree to which the input is “expected” (either explicitly or implicitly) based on prior knowledge, with statistically regular images evoking a reduced response. Moreover, we show that the measure exhibits context-dependency; that is, we find the N300 sensitivity to category representativeness when stimuli are congruent with, but not when they are incongruent with, a category pre-cue. Thus, we argue that the N300 is the best candidate to date for an index of perceptual hypotheses testing for complex visual objects and scenes.

Exploring human-nature interactions in national parks with social media photographs and computer vision.

Understanding the activities and preferences of visitors is crucial for managing protected areas and planning conservation strategies. Conservation culturomics promotes the use of user-generated online content in conservation science. Geotagged social media content is a unique source of in situ information on human presence and activities in nature. Photographs posted on social media platforms are a promising source of information, but analyzing large volumes of photographs manually remains laborious. We examined the application of state-of-the-art computer-vision methods to studying human-nature interactions. We used semantic clustering, scene classification, and object detection to automatically analyze photographs taken in Finnish national parks by domestic and international visitors. Our results showed that human-nature interactions can be extracted from user-generated photographs with computer vision. The different methods complemented each other by revealing broad visual themes related to level of the data set, landscape photogeneity, and human activities. Geotagged photographs revealed distinct regional profiles for national parks (e.g., preferences in landscapes and activities), which are potentially useful in park management. Photographic content differed between domestic and international visitors, which indicates differences in activities and preferences. Information extracted automatically from photographs can help identify preferences among diverse visitor groups, which can be used to create profiles of national parks for conservation marketing and to support conservation strategies that rely on public acceptance. The application of computer-vision methods to automatic content analysis of photographs should be explored further in conservation culturomics, particularly in combination with rich metadata available on social media platforms.

NaSC-TG2: Natural Scene Classification With Tiangong-2 Remotely Sensed Imagery

Scene classification is one of the most important applications of remote sensing. Researchers have proposed various datasets and innovative methods for remote sensing scene classification in recent years. However, most of the existing remote sensing scene datasets are collected uniquely from a single data source: Google Earth. In addition, scenes in different datasets are mainly human-made landscapes with high similarity. The lack of richness and diversity of data sources limits the research and applications of remote sensing classification. This article describes a large-scale dataset named “NaSC-TG2,” which is a novel benchmark dataset for remote sensing natural scene classification built from Tiangong-2 remotely sensed imagery. The goal of this dataset is to expand and enrich the annotation data for advancing remote sensing classification algorithms, especially for the natural scene classification. The dataset contains 20 000 images, which are equally divided into ten scene classes. The dataset has three primary advantages: 1) it is large scale, especially in terms of the number of each class, and the numbers of scenes are evenly distributed; 2) it has a large number of intraclass differences and high interclass similarity, because all images are carefully selected from different regions and seasons; and 3) it offers natural scenes with novel spatial scale and imaging performance compared with other datasets. All images are acquired from the new generation of wideband imaging spectrometer of Tiangong-2. In addition to RGB images, the corresponding multispectral scene images are also provided. This dataset is useful in supporting the development and evaluation of classification algorithms, as demonstrated in the present study.

A Multiple-Instance Densely-Connected ConvNet for Aerial Scene Classification.

In contrast with nature scenes, aerial scenes are often composed of many objects crowdedly distributed on the surface in bird's view, the description of which usually demands more discriminative features as well as local semantics. However, when applied to scene classification, most of the existing convolution neural networks (ConvNets) tend to depict global semantics of images, and the loss of low- and mid-level features can hardly be avoided, especially when the model goes deeper. To tackle these challenges, in this paper, we propose a multiple-instance densely-connected ConvNet (MIDC-Net) for aerial scene classification. It regards aerial scene classification as a multiple-instance learning problem so that local semantics can be further investigated. Our classification model consists of an instance-level classifier, a multiple instance pooling and followed by a bag-level classification layer. In the instance-level classifier, we propose a simplified dense connection structure to effectively preserve features from different levels. The extracted convolution features are further converted into instance feature vectors. Then, we propose a trainable attention-based multiple instance pooling. It highlights the local semantics relevant to the scene label and outputs the bag-level probability directly. Finally, with our bag-level classification layer, this multiple instance learning framework is under the direct supervision of bag labels. Experiments on three widely-utilized aerial scene benchmarks demonstrate that our proposed method outperforms many state-of-the-art methods by a large margin with much fewer parameters.

A classification algorithm based on weighted ML-kNN for multi-label data

The ML-kNN algorithm uses naive Bayesian classification to modify the traditional kNN algorithm to solve multi-label classification problems. However, the ML-kNN algorithm is prone to misjudgement or incomplete judgment of the unseen instance's label set in two special cases: when the number of labels in the training set is not balanced and when the training instances are unevenly distributed in space. Therefore, a weighted ML-kNN algorithm (i.e., wML-kNN) is proposed in this paper. The main idea is to assign different weights to each label according to the proportion of labels and mutual information of the spatial distribution of unseen instances to training instances. This method can reduce the probability of misjudgement of the unseen instance's label set. A comparative study was conducted on four multi-label datasets that included review classification and three other published benchmark multi-label datasets: yeast gene function analysis, natural scene classification, and musical sentiment classification. The results show that the performance of the wML-kNN algorithm is better than the other four multi-label learning algorithms, including ML-kNN.

Contribution of central vision and peripheral vision to categorization of natural and artificial scenes

Contribution of central vision and peripheral vision to categorization of natural and artificial scenes

Categorization Goals Modulate the Use of Natural Scene Statistics.

Understanding natural scenes involves the contribution of bottom-up analysis and top-down modulatory processes. However, the interaction of these processes during the categorization of natural scenes is not well understood. In the current study, we approached this issue using ERPs and behavioral and computational data. We presented pictures of natural scenes and asked participants to categorize them in response to different questions (Is it an animal/vehicle? Is it indoors/outdoors? Are there one/two foreground elements?). ERPs for target scenes requiring a "yes" response began to differ from those of nontarget scenes, beginning at 250 msec from picture onset, and this ERP difference was unmodulated by the categorization questions. Earlier ERPs showed category-specific differences (e.g., between animals and vehicles), which were associated with the processing of scene statistics. From 180 msec after scene onset, these category-specific ERP differences were modulated by the categorization question that was asked. Categorization goals do not modulate only later stages associated with target/nontarget decision but also earlier perceptual stages, which are involved in the processing of scene statistics.

Generalized Aggregation of Sparse Coded Multi-Spectra for Satellite Scene Classification

Satellite scene classification is challenging because of the high variability inherent in satellite data. Although rapid progress in remote sensing techniques has been witnessed in recent years, the resolution of the available satellite images remains limited compared with the general images acquired using a common camera. On the other hand, a satellite image usually has a greater number of spectral bands than a general image, thereby permitting the multi-spectral analysis of different land materials and promoting low-resolution satellite scene recognition. This study advocates multi-spectral analysis and explores the middle-level statistics of spectral information for satellite scene representation instead of using spatial analysis. This approach is widely utilized in general image and natural scene classification and achieved promising recognition performance for different applications. The proposed multi-spectral analysis firstly learns the multi-spectral prototypes (codebook) for representing any pixel-wise spectral data, and then, based on the learned codebook, a sparse coded spectral vector can be obtained with machine learning techniques. Furthermore, in order to combine the set of coded spectral vectors in a satellite scene image, we propose a hybrid aggregation (pooling) approach, instead of conventional averaging and max pooling, which includes the benefits of the two existing methods, but avoids extremely noisy coded values. Experiments on three satellite datasets validated that the performance of our proposed approach is very impressive compared with the state-of-the-art methods for satellite scene classification.

Semi-supervised multi-label feature selection via label correlation analysis with l1-norm graph embedding

In this paper, we propose a novel semi-supervised multi-label feature selection algorithm and apply it to three different applications: natural scene classification, web page annotation, and yeast gene functional classification. Compared with the previous works, there are two advantages of our algorithm: (1) Manifold learning which leverages the underlying geometric structure of the training data is imposed to utilize both labeled and unlabeled data. Besides, the underlying manifold structure is guaranteed to be clear by using the l1-norm regularization. (2) Shared subspace learning which has shown its efficiency in multi-label learning scenarios, is also considered in our feature learning algorithm. The proposed objective function involves l2,1-norm and l1-norm, making it non-smooth and difficult to solve. We also design an efficient iterative algorithm to optimize it. Experimental results demonstrate the effectiveness of our algorithm compared with sate-of-the-art algorithms on different tasks.