Natural Scenes Research Articles

Predicting image memorability from evoked feelings.

While viewing a visual stimulus, we often cannot tell whether it is inherently memorable or forgettable. However, the memorability of a stimulus can be quantified and partially predicted by a collection of conceptual and perceptual factors. Higher-level properties that represent the "meaningfulness" of a visual stimulus to viewers best predict whether it will be remembered or forgotten across a population. Here, we hypothesize that the feelings evoked by an image, operationalized as the valence and arousal dimensions of affect, significantly contribute to the memorability of scene images. We ran two complementary experiments to investigate the influence of affect on scene memorability, in the process creating a new image set (VAMOS) of hundreds of natural scene images for which we obtained valence, arousal, and memorability scores. From our first experiment, we found memorability to be highly reliable for scene images that span a wide range of evoked arousal and valence. From our second experiment, we found that both valence and arousal are significant but weak predictors of image memorability. Scene images were most memorable if they were slightly negatively valenced and highly arousing. Images that were extremely positive or unarousing were most forgettable. Valence and arousal together accounted for less than 8% of the variance in image memorability. These findings suggest that evoked affect contributes to the overall memorability of a scene image but, like other singular predictors, does not fully explain it. Instead, memorability is best explained by an assemblage of visual features that combine, in perhaps unintuitive ways, to predict what is likely to stick in our memory.

Cortical encoding of spatial structure and semantic content in 3D natural scenes.

Our visual system enables us to effortlessly navigate and recognize real-world visual environments. Functional magnetic resonance imaging (fMRI) studies suggest a network of scene-responsive cortical visual areas, but much less is known about the temporal order in which different scene properties are analysed by the human visual system. In this study, we selected a set of 36 full-colour natural scenes that varied in spatial structure and semantic content that our male and female human participants viewed both in 2D and 3D while we recorded magnetoencephalography (MEG) data. MEG enables tracking of cortical activity in humans at millisecond timescale. We compared the representational geometry in the MEG responses with predictions based on the scene stimuli using the representational similarity analysis framework. The representational structure first reflected the spatial structure in the scenes in time-window 90-125 ms, followed by the semantic content in time-window 140-175 ms after stimulus onset. The 3D stereoscopic viewing of the scenes affected the responses relatively late, from around 140 ms from stimulus onset. Taken together, our results indicate that the human visual system rapidly encodes a scene's spatial structure and suggest that this information is based on monocular instead of binocular depth cues.Significance statement Our visual system enables us to recognize and navigate our visual surroundings seemingly effortlessly, but what exactly happens in our brains remains poorly understood. With the help of time-resolved brain imaging (magnetoencephalography, MEG), we found that the brain first encodes the spatial structure of a scene (e.g., cluttered or navigable) before its semantic content (e.g., a car park or farm). Brain imaging studies typically use 2D pictures as stimuli. Here we asked whether binocular disparity, a depth cue which arises from our two eyes seeing the scene from slightly different angles, aids the coding of the spatial structure. Our results suggest that this 3D depth cue plays little role in the rapid, initial sensing of our spatial surroundings.

The Visual Integration of Semantic and Spatial Information of Objects in Naturalistic Scenes (VISIONS) database: attentional, conceptual, and perceptual norms.

The complex interplay between low- and high-level mechanisms governing our visual system can only be fully understood within ecologically valid naturalistic contexts. For this reason, in recent years, substantial efforts have been devoted to equipping the scientific community with datasets of realistic images normed on semantic or spatial features. Here, we introduce VISIONS, an extensive database of 1136 naturalistic scenes normed on a wide range of perceptual and conceptual norms by 185 English speakers across three levels of granularity: isolated object, whole scene, and object-in-scene. Each naturalistic scene contains a critical object systematically manipulated and normed regarding its semantic consistency (e.g., a toothbrush vs. a flashlight in a bathroom) and spatial position (i.e., left, right). Normative data are also available for low- (i.e., clarity, visual complexity) and high-level (i.e., name agreement, confidence, familiarity, prototypicality, manipulability) features of the critical object and its embedding scene context. Eye-tracking data during a free-viewing task further confirms the experimental validity of our manipulations while theoretically demonstrating that object semantics is acquired in extra-foveal vision and used to guide early overt attention. To our knowledge, VISIONS is the first database exhaustively covering norms about integrating objects in scenes and providing several perceptual and conceptual norms of the two as independently taken. We expect VISIONS to become an invaluable image dataset to examine and answer timely questions above and beyond vision science, where a diversity of perceptual, attentive, mnemonic, or linguistic processes could be explored as they develop, age, or become neuropathological.

Character/Word Modelling: A Two-Step Framework for Text Recognition in Natural Scene Images

Text recognition from images is a complex task in computer vision. Traditional text recognition methods typically rely on Optical Character Recognition (OCR); however, their limitations in image processing can lead to unreliable results. However, recent advancements in deep-learning models have provided an effective alternative for recognizing and classifying text in images. This study proposes a deep-learning-based text recognition system for natural scene images that incorporates character/word modeling, a two-step procedure involving the recognition of characters and words. In the first step, Convolutional Neural Networks (CNN) are used to differentiate individual characters from image frames. In the second step, the Viterbi search algorithm employs lexicon-based word recognition to determine the optimal sequence of recognized characters, thereby enabling accurate word identification in natural scene images. The system is tested using the ICDAR 2003 and ICDAR 2013 datasets from the Kaggle repository, and achieved accuracies of 79.8% and 81.5%, respectively.

Fine-grained restoration of Mongolian patterns based on a multi-stage deep learning network

Mongolian patterns are easily damaged by various factors in the process of inheritance and preservation, and the traditional manual restoration methods are time-consuming, laborious, and costly. With the development of deep learning technology and the rapid growth of the image restoration field, the existing image restoration methods are mostly aimed at natural scene images. They do not apply to Mongolian patterns with complex line texture structures and high saturation-rich colors. In order to solve this problem, this paper proposes a Mongolian pattern restoration model with a multi-stage network. In the first stage, a pyramid context encoder network is introduced to learn the contextual features of the image for global restoration; in the second stage, a local restoration network is constructed by combining the RIC convolutional layer and the MPD down-sampling module; and in the third stage, the global refinement restoration is carried out by using the U-Net network that incorporates the attention mechanism. The experimental results show that this paper’s method achieves remarkable results in the Mongolian pattern repair task, using four evaluation metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), LPIPS, and L1 Loss to compare with the existing method. The results show that this paper’s method performs well in the Mongolian pattern repair task. Also, the performance of the model on public datasets verifies its wide applicability. The method in this paper provides an efficient solution for the digital restoration of Mongolian motifs and has important application prospects.

Digital Characteristics of Spatial Layout in Urban Park Scene Space: Spatial Classification, Quantitative Indicators, and Design Applications Based on Completed Park Cases

Urban parks are important components of the urban green space system, providing residents with a variety of leisure options. The design’s focal point is the spatial layout of the different use scenarios within these parks. Previous studies have largely concentrated on the scene and macro layout of urban public spaces but have not thoroughly investigated the digital characteristics and corresponding parametric methods for the layout of different internal use scenes in urban green spaces. This research selected 18 urban parks from various global regions as case samples and categorized eight typical park scene space types based on common activity scene requirements in park design using AutoCAD to identify vector boundaries in each sample. To examine the digital characteristics of these scene space types, a quantitative index system was established, including spatial density and scale indicators, spatial connection relationship indicators, and spatial unit morphology indicators. The analysis of these indicators across the samples shows that the number of scene spaces is positively correlated with the total park area, while natural experience scenes constitute the largest proportion of urban parks. Different scene types exhibit distinct spatial layouts; for example, circulation spaces demonstrate high connectivity due to their role in directing visitor flow. Some spaces exhibit a more fixed scale and size, while others vary considerably. Finally, this research develops a parametric design framework using the Grasshopper platform. By taking a park in Nanjing as a case study, this paper illustrates how to utilize digital layout features to generate scene space layouts, offering insights into intelligent generative design. This approach provides a structured method to enhance urban park design through the application of digital and parametric tools, contributing to the broader field of urban park design.

Enhancement of satellite images based on CLAHE and augmented elk herd optimizer

Satellite images often have very narrow brightness value ranges, so it is necessary to enhance the contrast and brightness, maintain the quality of visual information, and preserve pertinent details in the images before conducting additional analysis. This is because improving the brightness and contrast of images is crucial to image processing and analysis as it makes it easier for people to identify and comprehend the images. The Incomplete Beta Function (IBF) is a popular transformation function for Image Contrast Enhancement (ICE). Nevertheless, IBF has modest efficiency in parameter selection, a small set of adjustable parameters for stretching regions with high or low gray levels, and image enhancement is almost ineffective with stretching at either end. Meta-heuristic algorithms have been utilized efficiently and effectively over the past few decades to solve complicated image processing problems. This paper presents an Augmented version of the Elk Herd Optimizer (AEHO) combined with other traditional ICE techniques to improve edge details, entropy, local contrast, and local brightness of low-contrast natural and satellite images. The AEHO method employs a multi-stage strategic procedure, where its mathematical model undergoes several enhancements before being applied to ICE to allow for further exploration and exploitation of its features. This method uses a pre-established fitness criterion for the purpose of optimizing a set of parameters to rework a well-known transformation function and an effective assessment technique as an objective standard for this purpose. In the proposed image enhancement model, contrast limited adaptive histogram equalization was first applied as a prior step to ameliorate the color intensity. Then, the optimal IBF’s parameters for ICE were adaptively determined using AEHO. After that, bilateral gamma correction was used to improve the visual quality of images without sacrificing edge details or natural color quality. The proposed AEHO-based image enhancement model is tested on natural scenes, certain standard images, and publicly available satellite images. In addition to other five techniques built on based on pre-existing meta-heuristics, the performance of the proposed method was compared against other well-known state-of-the-art image enhancement algorithms. The objective evaluation of the enhancement algorithms was achieved utilizing a variety of full-reference, no-reference, and pertinent performance evaluation norms. The experimental findings illustrated that the proposed image enhancement method can successfully outperform several other algorithms that employed the same image enhancement model as AEHO in addition to other conventional image enhancement methods included for comparison. The results on ten natural and satellite color images showed that the presented method performs better than all other comparative methods in the corresponding evaluation criteria in terms of average peak signal-to-noise ratio, average universal quality index, average structural contrast-quality index, and average values of discrete entropy results, which are more than 32.30, 94.0%, 0.98.9%, and 7.4, respectively. In a nutshell, AEHO can be an efficient method that can be used to tackle several image processing problems.

Stimulus-invariant aspects of the retinal code drive discriminability of natural scenes

Everything that the brain sees must first be encoded by the retina, which maintains a reliable representation of the visual world in many different, complex natural scenes while also adapting to stimulus changes. This study quantifies whether and how the brain selectively encodes stimulus features about scene identity in complex naturalistic environments. While a wealth of previous work has dug into the static and dynamic features of the population code in retinal ganglion cells (RGCs), less is known about how populations form both flexible and reliable encoding in natural moving scenes. We record from the larval salamander retina responding to five different natural movies, over many repeats, and use these data to characterize the population code in terms of single-cell fluctuations in rate and pairwise couplings between cells. Decomposing the population code into independent and cell-cell interactions reveals how broad scene structure is encoded in the retinal output. while the single-cell activity adapts to different stimuli, the population structure captured in the sparse, strong couplings is consistent across natural movies as well as synthetic stimuli. We show that these interactions contribute to encoding scene identity. We also demonstrate that this structure likely arises in part from shared bipolar cell input as well as from gap junctions between RGCs and amacrine cells.

Visual Field Asymmetries in Responses to ON and OFF Pathway Biasing Stimuli.

Recent reports suggest the ON and OFF pathways are differentially susceptible to selective vision loss in glaucoma. Thus, perimetric assessment of ON- and OFF-pathway function may serve as a useful diagnostic. However, this necessitates a developed understanding of normal ON/OFF pathway function around the visual field and as a function of input intensity. Here, using electroencephalography, we measured ON- and OFF-pathway biased contrast response functions in the upper and lower visual fields. Using the steady-state visually evoked potential paradigm, we flickered achromatic luminance probes according to a saw-tooth waveform, the fast phase of which biased responses towards the ON or OFF pathways. Neural responses from the upper and lower visual fields were simultaneously measured using frequency tagging - probes in the upper visual field modulated at 3.75Hz, while those in the lower visual field modulated at 3Hz. We find that responses to OFF/decrements are larger than ON/increments, especially in the lower visual field. In the lower visual field, both ON and OFF responses were well described by a sigmoidal non-linearity. In the upper visual field, the ON pathway function was very similar to that of the lower, but the OFF pathway function showed reduced saturation and more cross-subject variability. Overall, this demonstrates that the relationship between the ON and OFF pathways depends on the visual field location and contrast level, potentially reflective of natural scene statistics.

Optimal color sets to represent the colors of natural scenes by k-medoids clustering

Optimal color sets to represent the colors of natural scenes by k-medoids clustering

Text Font Correction and Alignment Method for Scene Text Recognition.

Text recognition is a rapidly evolving task with broad practical applications across multiple industries. However, due to the arbitrary-shape text arrangement, irregular text font, and unintended occlusion of font, this remains a challenging task. To handle images with arbitrary-shape text arrangement and irregular text font, we designed the Discriminative Standard Text Font (DSTF) and the Feature Alignment and Complementary Fusion (FACF). To address the unintended occlusion of font, we propose a Dual Attention Serial Module (DASM), which is integrated between residual modules to enhance the focus on text texture. These components improve text recognition by correcting irregular text and aligning it with the original feature extraction, thus complementing the overall recognition process. Additionally, to enhance the study of text recognition in natural scenes, we developed the VBC Chinese dataset under varying lighting conditions, including strong light, weak light, darkness, and other natural environments. Experimental results show that our method achieves competitive performance on the VBC dataset with an accuracy of 90.8% and an overall average accuracy of 93.8%.

Color crowding considered as adaptive spatial integration.

Crowding is the inability to recognize an object in clutter, classically considered a fundamental low-level bottleneck to object recognition. Recently, however, it has been suggested that crowding, like predictive phenomena such as serial dependence, may result from optimizing strategies that exploit redundancies in natural scenes. This notion leads to several testable predictions, such as crowding being greater for nonsalient targets and, counterintuitively, that flanker interference should be associated with higher precision in judgements, leading to a lower overall error rate. Here we measured color discrimination for targets flanked by stimuli of variable color. The results verified both predictions, showing that although crowding can affect object recognition, it may be better understood not as a processing bottleneck, but rather as a consequence of mechanisms evolved to efficiently exploit the spatial redundancies of the natural world. Analyses of reaction times of judgments shows that the integration occurs at sensory rather than decisional levels.

Exposure to nature can promote pro‐environmental behavior: Mediating role of temporal discounting

AbstractOne possible explanation for the lack of pro‐environmental behavior among the public is that the benefits of combating climate change are long‐term and distant. Temporal discounting is a pervasive tendency to prefer small immediate gains to large delayed benefits, which may lead to a lack of motivation to engage in pro‐environmental behavior. However, exposure to nature can reduce this tendency and by extension may promote pro‐environmental behavior. Two behavioral experiments were conducted to investigate whether exposure to natural (vs. urban) scenes would induce lower discounting and increase the tendency toward pro‐environmental behavior. We demonstrated that exposure to natural (vs. urban) scenes was associated with a lower level of discounting and a greater tendency toward pro‐environmental behavior, including energy‐saving use of air conditioning (Experiment 1), willingness to participate in beach cleaning (Experiment 2), and choosing meals with less environmental impact (Experiment 2). Mediation analysis indicated that the discounting tendency mediated the relationship between exposure to natural scenes and pro‐environmental behavior. This study provides the first experimental evidence that temporal discounting accounts for the association between exposure to natural (vs. urban) scenes and the tendency toward pro‐environmental behavior. Our findings support the development of a novel strategy for promoting pro‐environmental behavior.

Domain-specific representation of social inference by neurons in the human amygdala and hippocampus.

Inferring the intentions and emotions of others from behavior is crucial for social cognition. While neuroimaging studies have identified brain regions involved in social inference, it remains unknown whether performing social inference is an abstract computation that generalizes across different stimulus categories or is specific to certain stimulus domain. We recorded single-neuron activity from the medial temporal lobe (MTL) and the medial frontal cortex (MFC) in neurosurgical patients performing different types of inferences from images of faces, hands, and natural scenes. Our findings indicate distinct neuron populations in both regions encoding inference type for social (faces, hands) and nonsocial (scenes) stimuli, while stimulus category was itself represented in a task-general manner. Uniquely in the MTL, social inference type was represented by separate subsets of neurons for faces and hands, suggesting a domain-specific representation. These results reveal evidence for specialized social inference processes in the MTL, in which inference representations were entangled with stimulus type as expected from a domain-specific process.

The influence of functional components of natural scenes on episodic memory

Prior expectation for the structure of natural scenes is perhaps the most influential contributor to episodic memory for objects in scenes. While the influence of functional components of natural scenes on scene perception and visual search has been well studied, far less is known about the independent contributions of these components to episodic memory. In this investigation, we systematically removed three functional components of natural scenes: global-background, local spatial, and local associative information, to evaluate their impact on episodic memory. Results revealed that [partially] removing the global-background negatively impacted recall accuracy following short encoding times but had relatively little impact on memory after longer times. In contrast, systematically removing local spatial and associative relationships of scene objects negatively impacted recall accuracy following short and longer encoding times. These findings suggest that scene background, object spatial arrangements, and object relationships facilitate not only scene perception and object recognition, but also episodic memory. Interestingly, the impact of these components depends on how much encoding time is available to store information in episodic memory. This work has important implications for understanding how the inherent structure and function of the natural world interacts with memory and cognition in naturalistic contexts.

Plant leaf image segmentation in natural scenes: a multi-layer graph queries propagation approach

Plant leaf image segmentation in natural scenes: a multi-layer graph queries propagation approach

Correlation reconstruction mechanism based on dual wavelength imaging and neural network

In a natural light-field imaging scene, lighting is usually performed using a mixed light field of multiple wavelengths. To better meet the requirements of human vision, this study proposes a correlation reconstruction method based on dual-wavelength imaging and a neural network (compression sensing correlation fusion-reconstruction by dual-wavelength imaging and auto-encoder neural network, CSCFR-DWI-AENN). Two different wavelengths of light are mixed through an optical multiplexer unit (OMU) to form a dual-wavelength illumination light field, and the light intensity value reflected by the object is received by the detector. The object information was reconstructed using the compressed sensing ghost imaging algorithm, and the distribution of the illumination field was taken as prior information. Two single-wavelength illumination information and the detection value are used for compressed sensing reconstruction, and the high-quality reconstructed image of the target to be measured is obtained by combining the noise reduction advantage of the autoencoder neural network. Then the target information is fused by the New Sum of Modified Laplacian (NSML) algorithm. Two single-wavelength reconstruction, dual-wavelength direct reconstruction and the proposed reconstruction were compared respectively, and non-reference image quality evaluation functions such as EN, MI, EAV and SF were used to process and analyze the reconstruction effect. It is proved that this algorithm can reconstruct the object image comprehensively, reproduce the complete detail information, and achieve a more accurate, more comprehensive and more reliable description of the same object. It provides a theoretical basis for multi-wavelength imaging technology and has a good application prospect.

Wi-TCG: a WiFi gesture recognition method based on transfer learning and conditional generative adversarial networks

Abstract With the rapid progress of WiFi technology, WiFi-based wireless sensing technology has opened up new ways for contactless human-computer interaction. However, hand gesture recognition technology faces low-quality data sets, insufficient model robustness, poor universality when the application scenario changes, high training costs, and weak generalization ability. To this end, this study innovatively proposes the Wi-TCG method, which combines transfer learning and conditional generative adversarial network (CGAN) to optimize WiFi gesture recognition. This method uses commercial Wi-Fi devices to collect channel state information (CSI) of gesture actions. It innovatively extracts Doppler shift image data as the input of CGAN to generate virtual data with similar characteristics to expand the training sample set. The network is fine-tuned using transfer learning techniques to recognize multiple gesture action categories in different scenarios accurately. In tests of two new natural scenes and six new gesture categories, the Wi-TCG method achieved a high recognition accuracy of 93.1%, providing strong support for applying WiFi-based wireless sensing technology in contactless human-computer interaction.

Picking point identification and localization method based on swin-transformer for high-quality tea

In the nature scene, because of the high degree of similarity between the background and the tea buds, as well as the different growth postures of the tea buds, finding and precisely identifying the picking point is challenging. To solve these issues, this paper proposes a precise way to find the best picking point for tea buds by combining traditional algorithms with Swin-Transformer-based target detection and semantic segmentation algorithms, namely SORC-SFT. Firstly, an improved target detection algorithm, Swin-Oriented R-CNN (SORC), is used to realize the recognition of four types of high-quality tea. The mean Average Precision (mAP) of the four categories was 82.3% after replacing the feature fusion network FPN with PAFPN and adding the Coordinate Attention (CA) mechanism. Secondly, the corresponding segmentation mask of the four recognized categories is obtained by adding Semask, Feature Alignment Module (FAM), and Feature Selection Module (FSM) to the improved semantic segmentation algorithm Semask-Fa-Transformer (SFT). The mean Intersection over Union (mIoU) of the semantic segmentation algorithm for each category is 89.83%, 91.97%, 88.85%, and 89.68%, respectively. Finally, the morphology of different categories of tea buds is analyzed, and the traditional algorithm is used to realize the accurate localization of the identified tea buds. For the four tested categories, the proportion of correct samples in locating picking points is 96.18%, 91.28%, 93.85%, and 90.58%, respectively. The experimental results show that, out of all the algorithms, the proposed picking point identification and localization approach has the best performance and will make a strong contribution to the accurate identification of tea leaves during the intelligent picking process.

Collaborative Camouflaged Object Detection: A Large-Scale Dataset and Benchmark.

In this article, we provide a comprehensive study of a new task called collaborative camouflaged object detection (CoCOD), which aims to simultaneously detect camouflaged objects with the same properties from a group of relevant images. To this end, we meticulously construct the first large-scale dataset, termed CoCOD8K, which consists of 8528 high-quality and elaborately selected images with object mask annotations, covering five superclasses and 70 subclasses. The dataset spans a wide range of natural and artificial camouflage scenes with diverse object appearances and backgrounds, making it a very challenging dataset for CoCOD. Besides, we propose the first baseline model for CoCOD, named bilateral-branch network (BBNet), which explores and aggregates co-camouflaged cues within a single image and between images within a group, respectively, for accurate camouflaged object detection (COD) in given images. This is implemented by an interimage collaborative feature exploration (CFE) module, an intraimage object feature search (OFS) module, and a local-global refinement (LGR) module. We benchmark 18 state-of-the-art (SOTA) models, including 12 COD algorithms and six CoSOD algorithms, on the proposed CoCOD8K dataset under five widely used evaluation metrics. Extensive experiments demonstrate the effectiveness of the proposed method and the significantly superior performance compared to other competitors. We hope that our proposed dataset and model will boost growth in the COD community. The dataset, model, and results will be available at: https://github.com/zc199823/BBNet-CoCOD.