Natural Scene Images Research Articles

What can we experience and report on a rapidly presented image? Intersubjective measures of specificity of freely reported contents of consciousness.

Background: A majority of previous studies appear to support a view that human observers can only perceive coarse information from a natural scene image when it is presented rapidly (<100ms, masked). In these studies, participants were often forced to choose an answer from options that experimenters preselected. These options can underestimate what participants experience and can report on it. The current study aims to introduce a novel methodology to investigate how detailed information participants can report after briefly seeing a natural scene image. Methods: We used a novel free-report paradigm to examine what people can freely report following a rapidly presented natural scene image (67/133/267ms, masked). N = 670 online participants typed up to five words to report what they saw in the image together with confidence of the respective responses. We developed a novel index, Intersubjective Agreement (IA). IA quantifies how specifically the response words were used to describe the target image, with a high value meaning the word is not often reported for other images. Importantly, IA eliminates the need for experimenters to preselect response options. Results: The words with high IA values are often something detailed (e.g., a small object) in a particular image. With IA, unlike commonly believed, we demonstrated that participants reported highly specific and detailed aspects of the briefly (even at 67ms, masked) shown image. Further, IA is positively correlated with confidence, indicating metacognitive conscious access to the reported aspects of the image. Conclusion: These new findings challenge the dominant view that the content of rapid scene experience is limited to global and coarse gist. Our novel paradigm opens a door to investigate various contents of consciousness with a free-report paradigm.

Multi-Content Complementation Network for Salient Object Detection in Optical Remote Sensing Images

In the computer vision community, great progresses have been achieved in salient object detection from natural scene images (NSI-SOD); by contrast, salient object detection in optical remote sensing images (RSI-SOD) remains to be a challenging emerging topic. The unique characteristics of optical RSIs, such as scales, illuminations and imaging orientations, bring significant differences between NSI-SOD and RSI-SOD. In this paper, we propose a novel Multi-Content Complementation Network (MCCNet) to explore the complementarity of multiple content for RSI-SOD. Specifically, MCCNet is based on the general encoder-decoder architecture, and contains a novel key component named Multi-Content Complementation Module (MCCM), which bridges the encoder and the decoder. In MCCM, we consider multiple types of features that are critical to RSI-SOD, including foreground features, edge features, background features, and global image-level features, and exploit the content complementarity between them to highlight salient regions over various scales in RSI features through the attention mechanism. Besides, we comprehensively introduce pixel-level, map-level and metric-aware losses in the training phase. Extensive experiments on two popular datasets demonstrate that the proposed MCCNet outperforms 23 state-of-the-art methods, including both NSI-SOD and RSI-SOD methods. The code and results of our method are available at https://github.com/MathLee/MCCNet.

Bridging the Functional and Wiring Properties of V1 Neurons Through Sparse Coding.

The functional properties of neurons in the primary visual cortex (V1) are thought to be closely related to the structural properties of this network, but the specific relationships remain unclear. Previous theoretical studies have suggested that sparse coding, an energy-efficient coding method, might underlie the orientation selectivity of V1 neurons. We thus aimed to delineate how the neurons are wired to produce this feature. We constructed a model and endowed it with a simple Hebbian learning rule to encode images of natural scenes. The excitatory neurons fired sparsely in response to images and developed strong orientation selectivity. After learning, the connectivity between excitatory neuron pairs, inhibitory neuron pairs, and excitatory-inhibitory neuron pairs depended on firing pattern and receptive field similarity between the neurons. The receptive fields (RFs) of excitatory neurons and inhibitory neurons were well predicted by the RFs of presynaptic excitatory neurons and inhibitory neurons, respectively. The excitatory neurons formed a small-world network, in which certain local connection patterns were significantly overrepresented. Bidirectionally manipulating the firing rates of inhibitory neurons caused linear transformations of the firing rates of excitatory neurons, and vice versa. These wiring properties and modulatory effects were congruent with a wide variety of data measured in V1, suggesting that the sparse coding principle might underlie both the functional and wiring properties of V1 neurons.

REL-SAGAN: Relative Generation Adversarial Network Integrated With Attention Mechanism for Scene Data Augmentation of Remote Sensing

Deep learning shows a strong ability in target detection, scene classification, and change detection of remote sensing. However, the training process requires a large number of samples, and the production of most high-quality training samples requires a lot of time and human resources. With the resolution of remote sensing image gradually improved, and the scene information becoming more and more complicated, the sample augmentation methods at this stage have shown obvious defects, such as the loss of sample key information and the lack of diversity in augmented data sets. In order to solve these problems, this article proposes a new augmentation method for remote sensing scene data named Rel-SAGAN based on generative adversarial networks (GAN). First, combined with self-attention mechanism to improve the large-scale feature learning ability and reduce the calculation parameters of GAN. Second, using relativity adversarial loss function to improve the structural stability of GAN. Furthermore, increasing convolution kernel and deeper structural of GAN to improve the ability of global feature extraction and saving the training time. Finally, taking NWPU remote sensing image dataset as experimental data, the effectiveness of the method proposed in this article is verified through ResNet-18. The experimental results show that it can generate more diverse high-resolution remote sensing natural scene images, the overall classification accuracy of augmented training dataset of remote sensing used high-quality generated images selected based on inception score and Frechet inception distance evaluations is improved by more than 3% and the classification accuracy is generally higher than the traditional data augmentation methods.

Global Perception Network for Salient Object Detection in Remote Sensing Images

Despite recent works that have achieved remarkable progress on salient object detection for natural scene images, to detect various types and scales of objects, complex backgrounds in remote sensing images are still challenging. In this study, a novel global perception network (GPNet) is constructed for the salient object detection of remote sensing images. The proposed GPNet includes a global perception module (GPM), an axial attention block (AAB), and a feature distillation structure (FDS). The GPM is used to preserve the relationships of the entire dataset, the AAB is designed to capture the dependencies between the space and channel, the FDS is introduced to enable the helpful multilevel information flow into deep layers to enhance feature generation, and the global and the local attention information are mutually fused to enhance the network mode. Extensive experiments on three public datasets demonstrate that the proposed method outperforms other compared state-of-the-art methods both qualitatively and quantitatively (<uri>https://github.com/liuyu1002/GPnet</uri>).

Remote Sensing Object Detection Based on Gated Context-Aware Module

Recently, deep learning algorithms, especially feature pyramid network (FPN), have achieved significant progress in object detection of natural scene images. However, due to the complex scenes of remote sensing images and the diversity of remote sensing objects, FPN still faces the following drawback when applied to remote sensing object detection. Specifically, in the original FPN, the features of each proposal are extracted by RoIAlign. However, these features have limited effective receptive fields, making FPN lack of crucial contextual information to accurately classify and locate objects, as well as filter some background noises that possess similar appearance with objects. To alleviate the above problem, in this letter, we propose a gated context aware module (G-CAM), and replace the original RoIAlign in FPN with the proposed G-CAM to adaptively incorporate the useful local context surrounding each proposal and the global context of the whole image into FPN, enabling FPN to effectively detect objects in remote sensing images Extensive experiments have been conducted on the DIOR and RSOD datasets, which validates that the proposed method achieves superior performance to the considered state-of-the-art methods in terms of detection accuracy.

Multi-Oriented Rotation-Equivariant Network for Object Detection on Remote Sensing Images

Object detection has attracted a lot of attention in the field of image automatic interpretation. Detectors based on convolution neural networks (CNNs) applied in natural scene images encode detection results with horizontal bounding boxes (HBBs), which can not accurately calibrate the position and shape of the arbitrary-orientation objects on remote sensing images (RSIs). To solve these issues, we propose an object detection framework named multi-oriented rotation-equivariant network (MORE-Net) in this letter. The MORE-Net consists of a multi-oriented rotating filter (MORF) and a multi-oriented ground objects detector. The MORF generates rotation-equivariant features by rotating canonical filter according to the predefined discrete directions. Equipped with MORF, the multi-oriented ground objects detector extracts rotation-invariant semantic representations for each decoded rotated region of interest (RRoI). We also propose area smooth (AS) L1 loss to impose tighter shape and position constraints to the RROIs. Extensive experiments and comprehensive evaluations on the large-scale DOTA dataset demonstrate the effectiveness of the proposed framework, which achieves a mean average precision (mAP) value of 81.27 on the DOTA-v1.0 dataset and a mAP value of 78.03 on the DOTA-v1.5 dataset.

Apple image fusion algorithm based on binocular acquisition system

To solve the problem that single natural scene image acquisition information in an orchard cannot meet the requirements of accurate fruit recognition and target positioning, a new apple image fusion algorithm based on the binocular acquisition system named New Non-sub-sampled Contourlet Transform algorithm is proposed to obtain a high-quality fusion image. The binocular acquisition system is constructed with the Time of Flight industrial camera and the colour camera. In order to achieve a better fusion effect, a parameter optimisation algorithm based on an Artificial Bee Colony algorithm with discard strategy for brightness saliency function is proposed to optimise the low-frequency component parameters of the new fusion coefficient rule. The experiments were taken on series of apple images under three sunlight conditions in the orchard. The experimental results show that the six evaluation indicators obtained by the new algorithm achieve the expected fusion image effect under three different types of sunlight conditions.

CNN and Fuzzy Rules Based Text Detection and Recognition from Natural Scenes

In today’s real world, an important research part in image processing is scene text detection and recognition. Scene text can be in different languages, fonts, sizes, colours, orientations and structures. Moreover, the aspect ratios and layouts of a scene text may differ significantly. All these variations appear assignificant challenges for the detection and recognition algorithms that are considered for the text in natural scenes. In this paper, a new intelligent text detection and recognition method for detectingthe text from natural scenes and forrecognizing the text by applying the newly proposed Conditional Random Field-based fuzzy rules incorporated Convolutional Neural Network (CR-CNN) has been proposed. Moreover, we have recommended a new text detection method for detecting the exact text from the input natural scene images. For enhancing the presentation of the edge detection process, image pre-processing activities such as edge detection and color modeling have beenapplied in this work. In addition, we have generated new fuzzy rules for making effective decisions on the processes of text detection and recognition. The experiments have been directedusing the standard benchmark datasets such as the ICDAR 2003, the ICDAR 2011, the ICDAR 2005 and the SVT and have achieved better detection accuracy intext detection and recognition. By using these three datasets, five different experiments have been conducted for evaluating the proposed model. And also, we have compared the proposed system with the other classifiers such as the SVM, the MLP and the CNN. In these comparisons, the proposed model has achieved better classification accuracywhen compared with the other existing works.

A New Method for Arabic Text Detection in Natural Scene Images

Text in scene images can provide useful and vital information for content-based image analysis. Therefore, text detection and script identification in images are an important task. In this paper, we propose a new method for text detection in natural scene images, particularly for Arabic text, based on a bottom-up approach where four principal steps can be highlighted. The detection of extremely stable and homogeneous regions of interest (ROIs) is based on the Color Stability and Homogeneity Regions (CSHR) proposed technique. These regions are then labeled as textual or non-textual ROI. This identification is based on a structural approach. The textual ROIs are grouped to constitute zones according to spatial relations between them. Finally, the textual or non-textual nature of the constituted zones is refined. This last identification is based on handcrafted features and on features built from a Convolutional Neural Network (CNN) after learning. The proposed method was evaluated on the databases used for text detection in natural scene images: the competitions organized in 2017 edition of the International Conference on Document Analysis and Recognition (ICDAR2017), the Urdu-text database and our Natural Scene Image Database for Arabic Text detection (NSIDAT) database. The obtained experimental results seem to be interesting.

Pareidolia in Schizophrenia and Bipolar Disorder.

While there are many studies on pareidolia in healthy individuals and patients with schizophrenia, to our knowledge, there are no prior studies on pareidolia in patients with bipolar disorder. Accordingly, in this study, we, for the first time, measured pareidolia in patients with bipolar disorder (N = 50), and compared that to patients with schizophrenia (N = 50) and healthy controls (N = 50). We have used (a) the scene test, which consists of 10 blurred images of natural scenes that was previously found to produce illusory face responses and (b) the noise test which had 32 black and white images consisting of visual noise and 8 images depicting human faces; participants indicated whether a face was present on these images and to point to the location where they saw the face. Illusory responses were defined as answers when observers falsely identified objects that were not on the images in the scene task (maximum illusory score: 10), and the number of noise images in which they reported the presence of a face (maximum illusory score: 32). Further, we also calculated the total pareidolia score for each task (the sum number of images with illusory responses in the scene and noise tests). The responses were scored by two independent raters with an excellent congruence (kappa > 0.9). Our results show that schizophrenia patients scored higher on pareidolia measures than both healthy controls and patients with bipolar disorder. Our findings are agreement with prior findings on more impaired cognitive processes in schizophrenia than in bipolar patients.

A Change of Scenery: Does Exposure to Images of Nature Affect Delay Discounting and Food Desirability?

Previous research suggests that exposure to nature may reduce delay discounting (the tendency to discount larger future gains in favor of smaller immediate rewards) and thereby facilitate healthier dietary intake. This pre-registered study examined the impact of online exposure to images of natural scenes on delay discounting and food preferences. It was predicted that exposure to images of natural scenes (vs. images of urban scenes) would be associated with: (i) lower delay discounting; (ii) higher desirability for fruits and vegetables (and lower desirability for more energy-dense foods); and (iii) delay discounting would mediate the effect of nature-image exposure on food desirability. Adult participants (N = 109) were recruited to an online between-subjects experiment in which they viewed a timed sequence of six images either showing natural landscape scenes or urban scenes. They then completed measures of mood, delay discounting (using a five-trial hypothetical monetary discounting task) and rated their momentary desire to eat four fruits and vegetables (F&V), and four energy-dense foods. There was no statistically significant effect of experimental condition (natural vs. urban image exposure) on delay discounting or food desirability. Bayes factors supported the null hypothesis for discounting (BF01 = 4.89), and energy-dense food desirability (BF01 = 7.21), but provided no strong evidence for either hypothesis for F&V desirability (BF01 = 0.78). These findings indicate that brief online exposure to images of nature does not affect momentary impulsivity or energy-dense food preference, whereas for preference for less-energy dense foods, the evidence was inconclusive.

SDFCNv2: An Improved FCN Framework for Remote Sensing Images Semantic Segmentation

Semantic segmentation is a fundamental task in remote sensing image analysis (RSIA). Fully convolutional networks (FCNs) have achieved state-of-the-art performance in the task of semantic segmentation of natural scene images. However, due to distinctive differences between natural scene images and remotely-sensed (RS) images, FCN-based semantic segmentation methods from the field of computer vision cannot achieve promising performances on RS images without modifications. In previous work, we proposed an RS image semantic segmentation framework SDFCNv1, combined with a majority voting postprocessing method. Nevertheless, it still has some drawbacks, such as small receptive field and large number of parameters. In this paper, we propose an improved semantic segmentation framework SDFCNv2 based on SDFCNv1, to conduct optimal semantic segmentation on RS images. We first construct a novel FCN model with hybrid basic convolutional (HBC) blocks and spatial-channel-fusion squeeze-and-excitation (SCFSE) modules, which occupies a larger receptive field and fewer network model parameters. We also put forward a data augmentation method based on spectral-specific stochastic-gamma-transform-based (SSSGT-based) during the model training process to improve generalizability of our model. Besides, we design a mask-weighted voting decision fusion postprocessing algorithm for image segmentation on overlarge RS images. We conducted several comparative experiments on two public datasets and a real surveying and mapping dataset. Extensive experimental results demonstrate that compared with the SDFCNv1 framework, our SDFCNv2 framework can increase the mIoU metric by up to 5.22% while only using about half of parameters.

An innovative network based on double receptive field and Recursive Bi-directional Long Short-Term Memory

Sequence recognition of natural scene images has always been an important research topic in the field of computer vision. CRNN has been proven to be a popular end-to-end character sequence recognition network. However, the problem of wide characters is not considered under the setting of CRNN. The CRNN is less effective in recognizing long dense small characters. Aiming at the shortcomings of CRNN, we proposed an improved CRNN network, named CRNN-RES, based on BiLSTM and multiple receptive fields. Specifically, on the one hand, the CRNN-RES uses a dual pooling core to enhance the CNN network’s ability to extract features. On the other hand, by improving the last RNN layer, the BiLSTM is changed to a shared parameter BiLSTM network using recursive residuals, which reduces the number of network parameters and improves the accuracy. In addition, we designed a structure that can flexibly configure the length of the input data sequence in the RNN layer, called the CRFC layer. Comparing the CRNN-RES network proposed in this paper with the original CRNN network, the extensive experiments show that when recognizing English characters and numbers, the parameters of CRNN-RES is 8197549, which decreased 133,752 parameters compare with CRNN. In the public dataset ICDAR 2003 (IC03), ICDAR 2013 (IC13), IIIT 5k-word (IIIT5k), and Street View Text (SVT), the CRNN-RES obtain the accuracy of 96.90%, 89.85%, 83.63%, and 82.96%, which higher than CRNN by 1.40%, 3.15%, 5.43%, and 2.16% respectively.

A multiscale feature fusion method for cursive text detection in natural scene images

ABSTRACT Text detection in natural images is a challenging problem due to variations in text size, aspect ratio, alignment and background complexity. This paper proposes a multiscale feature fusion convolutional neural network method to detect cursive and multi-language text in natural images. The proposed method combines VGG-16 features at multi-scales and multi-layers and creates a new convolutional feature map of shallow and deep layers. On top of convolutional feature map, a vertical text proposal generation method is used that generates fixed-size text proposals. A recurrent layer is implemented which takes the convolutional feature maps of window as sequential input and updates the recurrent state internally in the hidden layers. The output of recurrent layer is mapped to the two fully connected layers to predict the text/non-text region proposals and bounding boxes regression. The model is evaluated on a custom-developed Urdu scene text dataset and the ICDAR-MLT17 Arabic text image dataset.

Deep-CNNTL: Text Localization from Natural Scene Images Using Deep Convolution Neural Network with Transfer Learning

Deep-CNNTL: Text Localization from Natural Scene Images Using Deep Convolution Neural Network with Transfer Learning

Estimation of ISO12233 Edge Spatial Frequency Response from Natural Scene Derived Step-Edge Data

Abstract The Natural Scene derived Spatial Frequency Response (NS-SFR) is a novel camera system performance measure that derives SFRs directly from images of natural scenes and processes them using ISO12233 edge-based SFR (e-SFR) algorithm. NS-SFR is a function of both camera system performance and scene content. It is measured directly from captured scenes, thus eliminating the use of test charts and strict laboratory conditions. The effective system e-SFR can be subsequently estimated from NS-SFRs using statistical analysis and a diverse dataset of scenes. This paper first presents the NS-SFR measuring framework, which locates, isolates, and verifies suitable step-edges from captures of natural scenes. It then details a process for identifying the most likely NS-SFRs for deriving the camera system e-SFR. The resulting estimates are comparable to standard e-SFRs derived from test chart inputs, making the proposed method a viable alternative to the ISO technique, with potential for real-time camera system performance measurements.

Analysis of Natural Scene Derived Spatial Frequency Responses for Estimating Camera ISO12233 Slanted-edge Performance

Abstract The Natural Scene derived Spatial Frequency Response (NS-SFR) framework automatically extracts suitable step-edges from natural pictorial scenes and processes these edges via the edge-based ISO12233 (e-SFR) algorithm. Previously, a novel methodology was presented to estimate the standard e-SFR from NS-SFR data. This paper implements this method using diverse natural scene image datasets from three characterized camera systems. Quantitative analysis was carried out on the system e-SFR estimates to validate accuracy of the method. Both linear and non-linear camera systems were evaluated. To investigate how scene content and dataset size affect system e-SFR estimates, analysis was conducted on entire datasets, as well as subsets of various sizes and scene group types. Results demonstrate that system e-SFR estimates strongly correlate with results from test chart inputs, with accuracy comparable to that of the ISO12233. Further work toward improving and fine-tuning the proposed methodology for practical implementation is discussed.

DeepRPN-BIQA: Deep architectures with region proposal network for natural-scene and screen-content blind image quality assessment

With the emerging use of technology and screen-oriented applications in our daily life, screen content images have gained the same importance as natural scene images. This results in many natural-scene and screen-content blind image quality assessment (BIQA) models to evaluate the perceptual quality without any prior information regarding the reference image. Recently, patch-based techniques for image quality assessment (IQA) have shown promising results. As per our knowledge, no IQA technique in literature is available that can be equally effective for both natural-scene and screen-content images. In this work, we have proposed a deep architecture with a region proposal network (RPN) for blind natural-scene and screen-content image quality assessment, named DeepRPN-BIQA. The proposed architecture computes visual saliency using RPN to extract important regions having a high contribution towards the image quality. Important regions are extracted by utilizing the texture and edges of images by sliding the network over the extracted feature map from deep architectures i.e., VGGNet and ResNet. The regions proposed (RP) that overlap more than 60% are merged into one proposal and are called the region of interest (ROI). The overlap between RPs is computed using anchors having 3 different scales and aspect ratios. A local quality score is computed over each ROI and the total quality score is computed by taking the average of all the local quality scores. Experimental results show that the DeepRPN-BIQA shows a high correlation between mean observer score and predicted quality score and performs better than other models for screen content images, synthetically distorted images, images taken in real-life conditions using mobile phone cameras, and large scale image quality assessment database.

Deep multi-task learning for image/video distortions identification

Identifying distortions in images and videos is important and useful in various visual applications, such as image quality enhancement and assessment techniques. Instead of applying them blindly, these techniques can be applied or adjusted depending on the type of distortion identified. In this paper, we propose a deep multi-task learning (MTL) model for identifying the types of distortion in both images and videos, considering both single and multiple distortions. The proposed MTL model is composed of one convolutional neural network (CNN) shared between all tasks and N parallel classifiers, where each classifier is dedicated to identify a type of distortion. The proposed architecture also allows to adjust the number of tasks according to the number of distortion types considered, making the solution scalable. The proposed method has been evaluated on natural scene images and laparoscopic videos databases, each presenting a rich set of distortions. The experimental results demonstrate that our model achieves the best performance among the state-of-the-art methods for both single and multiple distortions (Code is available at: https://github.com/zoubidaameur/Deep-Multi-Task-Learning-for-Image-Video-Distortions-Identification).