Natural Scenes Research Articles

Implementation of Text to Image using Diffusion Model

Text-to-image generation is a transformative field in artificial intelligence, aiming to bridge the semantic gap between textual descriptions and visual representations. This presents a comprehensive approach to tackle this challenging task. Leveraging the advancements in deep learning, natural language processing (NLP), and computer vision, this proposes a cutting-edge model for generating high-fidelity images from textual prompts. Trained on a vast and varied dataset of written descriptions and related images, this model combines an image decoder and a text encoder within a hierarchical framework. To enhance realism, this incorporates attention mechanisms and fine-grained semantic parsing. The model's performance is rigorously evaluated through both quantitative metrics and qualitative human assessments. Results demonstrate its ability to produce visually compelling and contextually accurate images across various domains, from natural scenes to specific object synthesis. This further explores applications in creative content generation, design automation, and virtual environments, showcasing the potential impact of our approach. Additionally, this releases a user-friendly API, empowering developers and designers to integrate our model into their projects, and fostering innovation and creativity. Key Words: image generation model, Deep learning, Natural language processing.

Changing: A Short Study on John Cheever’s “Swimmer”

American writer John Cheever’s “Swimmer” adopts a surreal or symbolic style that breaks away from the previous ideas and approaches of realism. The story’s hero Neddy Merrill embarks on an adventurous journey to swim home: an eight-mile journey that connects suburban swimming pools in his mind’s eye to form the magnificent Lucinda River, named for his wife. This paper examines the ever-changing natural scenes to explore the role of such scenes in portraying the characters and revealing the theme of the story. It also discusses the changing fates of the hero in two aspects-people’s changing attitudes towards Neddy and his own changing physical condition. The study considers the hero’s tragedy as a true portrayal of the loss and spiritual plight of the American middle class, which strengthens the sense of crisis and disillusionment in modern society, and expresses the author’s deep doubts about the lifestyle and thinking mode of the middle class, and his infinite worries about the future of the American middle class.

Vehicle trajectory prediction based on attention optimized with real-scene sampling

Advancements in autonomous vehicles and deep learning have notably improved vehicle trajectory prediction accuracy. However, extracting interaction features in complex driving scenarios, such as vehicle-to-vehicle interactions and lane constraints, presents challenges. Deep learning-based methods struggle to achieve optimal predictive performance under limited computational resources. This study introduces a global attention mechanism to enhance feature extraction from driving scene encodings, focusing the decoder on interactive behaviours and boosting long-term prediction performance. An adaptive scheduled sampling model is employed, using actual driving scenarios probabilistically for training, addressing slow learning of actual driving behaviours and lack of initial feature correction. This method increases attention to actual interactions, reducing reliance on natural scenes and improving model generalizability. On the NGSIM dataset, sampling attention encoder-decoder (SAED) achieves a 1–5 s average displacement error (ADE) of 1.34 m, with 4 s and 5 s final displacement errors (FDEs) of 1.64 and 2.06 m, respectively. Compared to methods based on long short-term memory (LSTM), SAED reduces the model's storage space by 24.68% under the same network layer count. That demonstrates its effectiveness in extracting interactive behaviours in complex scenarios and enhances the accuracy of long-term predictions.

Potsdam data set of eye movement on natural scenes (DAEMONS).

Potsdam data set of eye movement on natural scenes (DAEMONS).

The Brain's Sensitivity to Real-world Statistical Regularity Does Not Require Full Attention.

Predictive coding accounts of perception state that the brain generates perceptual predictions in the service of processing incoming sensory data. These predictions are hypothesized to be afforded by the brain's ability to internalize useful patterns, that is, statistical regularities, from the environment. We have previously argued that the N300 ERP component serves as an index of the brain's use of representations of (real-world) statistical regularities. However, we do not yet know whether overt attention is necessary in order for this process to engage. We addressed this question by presenting stimuli of either high or low real-world statistical regularity in terms of their representativeness (good/bad exemplars of natural scene categories) to participants who either fully attended the stimuli or were distracted by another task (attended/distracted conditions). Replicating past work, N300 responses were larger to bad than to good scene exemplars, and furthermore, we demonstrate minimal impacts of distraction on N300 effects. Thus, it seems that overtly focused attention is not required to maintain the brain's sensitivity to real-world statistical regularity. Furthermore, in an exploratory analysis, we showed that providing additional, artificial regularities, formed by altering the proportions of good and bad exemplars within blocks, further enhanced the N300 effect in both attended and distracted conditions, shedding light on the relationship between statistical regularities learned in the real world and those learned within the context of an experiment.

Streak tube imaging lidar with kilohertz laser pulses and few-photons detection capability

Lidar using active light illumination is capable of capturing depth and reflectivity information of target scenes. Among various technologies, streak tube imaging lidar (STIL) has garnered significant attention due to its high resolution and excellent precision. The echo signals of a STIL system using single laser pulse are often overwhelmed by noise in complex environments, making it difficult to discern the range of the target. By combining high-frequency laser pulses with the repetitive sweep circuit, the STIL system enables efficient detection of few-photons signal in weak-light environments. Additionally, we have developed a robust algorithm for estimating the depth and reflectivity images of targets. The results demonstrate that this lidar system achieves a depth resolution better than 0.5 mm and a ranging accuracy of 95 um. Furthermore, the imaging of natural scenes also validates the exceptional 3D imaging capability of this system.

Through their eyes: Multi-subject brain decoding with simple alignment techniques

Abstract To-date, brain decoding literature has focused on single-subject studies, that is, reconstructing stimuli presented to a subject under fMRI acquisition from the fMRI activity of the same subject. The objective of this study is to introduce a generalization technique that enables the decoding of a subject’s brain based on fMRI activity of another subject, that is, cross-subject brain decoding. To this end, we also explore cross-subject data alignment techniques. Data alignment is the attempt to register different subjects in a common anatomical or functional space for further and more general analysis. We utilized the Natural Scenes Dataset, a comprehensive 7T fMRI experiment focused on vision of natural images. The dataset contains fMRI data from multiple subjects exposed to 9,841 images, where 982 images have been viewed by all subjects. Our method involved training a decoding model on one subject’s data, aligning new data from other subjects to this space, and testing the decoding on the second subject based on information aligned to the first subject. We also compared different techniques for fMRI data alignment, specifically ridge regression, hyper alignment, and anatomical alignment. We found that cross-subject brain decoding is possible, even with a small subset of the dataset, specifically, using the common data, which are around 10% of the total data, namely 982 images, with performances in decoding comparable to the ones achieved by single-subject decoding. Cross-subject decoding is still feasible using half or a quarter of this number of images with slightly lower performances. Ridge regression emerged as the best method for functional alignment in fine-grained information decoding, outperforming all other techniques. By aligning multiple subjects, we achieved high-quality brain decoding and a potential reduction in scan time by 90%. This substantial decrease in scan time could open up unprecedented opportunities for more efficient experiment execution and further advancements in the field, which commonly requires prohibitive (20 hours) scan time per subject.

Visualization of the occurrence and spread of wildfires in three-dimensional natural scenes

Visualization of the occurrence and spread of wildfires in three-dimensional natural scenes

Transformer based ensemble deep learning approach for remote sensing natural scene classification

ABSTRACT Very high resolution (VHR) remote sensing (RS) image classification is paramount for detailed Earth’s surface analysis. Feature extraction from VHR natural scenes is crucial, but it becomes a challenging task because of the overlapping edges present in images. Multiple open-source datasets exist in the literature to train robust models, and they have been benchmarked using deep learning models. However, different datasets contain different numbers of classes; a few of them could be absent in other datasets because they are independent of the other classes, and the classes are sometimes not mutually exclusive amongst the same dataset. Thus, it is very challenging to generalize a model trained on a single dataset to perform scene classification on unknown classes of multiple benchmark datasets and real-time images. Thus, this work introduces the Remote Sensing Natural Scenes 92 (RS_NS92) dataset, consisting of 36,785 images belonging to 92 classes, curated by selectively taking the union of all subclasses from five benchmark datasets. This class count is significantly higher than publicly available datasets and maintains a low-class imbalance and a comprehensive data distribution for robust model training. It also provides the remote sensing community with an extra platform to validate the performance on multiple benchmarks. Inspired by federated learning, an ensemble approach consisting of three feature extraction backbones: Vision Transformers, Swin Transformers, and ConvNeXt (termed the VSC_Ensemble model) is also introduced. This model can make extraordinary predictions across multiple datasets by finetuning weights using transfer learning. Experimental analysis with the proposed approach not only obtains a high test accuracy of 97.24% and an F1-score of 0.9587 for the 92 classes on a 90:10 split of the proposed benchmark dataset but also gets excellent results on unseen test images of other datasets, which are comparable to the state-of-the-art results.

Visual social attention in SYNGAP1-related intellectual disability.

SYNGAP1-ID is a neurodevelopmental disorder caused by a mutation of the SYNGAP1 gene. Characterized by moderate to severe developmental delay, it is associated with several physical and behavioral issues as well as additional diagnoses, including autism. However, it is not known whether social cognitive differences seen in SYNGAP1-ID are similar to those previously identified in idiopathic or other forms of autism. This study therefore investigated visual social attention in SYNGAP1-ID. Eye movements were recorded across three passive viewing tasks (face scanning, pop-out, and social preference) of differing social complexity in 24 individuals with SYNGAP1-ID and 12 typically developing controls. We found that SYNGAP1-ID participants looked at faces less than the controls, and when they did look at faces, they had less time looking at and fewer fixations to the eyes. For the pop-out task, where social and nonsocial objects (Phone, car, face, bird, and face-noise) were presented in an array, those with SYNGAP1-ID spent significantly less time looking at the phone stimulus as well as fewer fixations to the face compared with the typically developing controls. When looking at two naturalistic scenes side by side, one social in nature (e.g., with children present) and the other not, there were no differences between the SYNGAP1-ID group and typically developing controls on any of the examined eye tracking measures. This study provides novel findings on the social attention of those with SYNGAP1-ID and helps to provide further evidence for using eye tracking as an objective measure of the social phenotype in this population in future clinical trials.

Visuospatial processing in early brain-based visual impairment is associated with differential recruitment of dorsal and ventral visual streams.

Visuospatial processing impairments are prevalent in individuals with cerebral visual impairment (CVI) and are typically ascribed to "dorsal stream dysfunction" (DSD). However, the contribution of other cortical regions, including early visual cortex (EVC), frontal cortex, or the ventral visual stream, to such impairments remains unknown. Thus, here, we examined fMRI activity in these regions, while individuals with CVI (and neurotypicals) performed a visual search task within a dynamic naturalistic scene. First, behavioral performance was measured with eye tracking. Participants were instructed to search and follow a walking human target. CVI participants took significantly longer to find the target, and their eye gaze patterns were less accurate and less precise. Second, we used the same task in the MRI scanner. Along the dorsal stream, activation was reduced in CVI participants, consistent with the proposed DSD in CVI. Intriguingly, however, visual areas along the ventral stream showed the complete opposite pattern, with greater activation in CVI participants. In contrast, we found no differences in either EVC or frontal cortex between groups. These results suggest that the impaired visuospatial processing abilities in CVI are associated with differential recruitment of the dorsal and ventral visual streams, likely resulting from impaired selective attention.

Nature's beauty versus urban bustle: Chinese folk music influences food choices by inducing mental imagery of different scenes

Previous research has demonstrated that music can impact people's food choices by triggering emotional states. We reported two virtual reality (VR) experiments designed to examine how Chinese folk music influences people's food choices by inducing mental imagery of different scenes. In both experiments, young healthy Chinese participants were asked to select three dishes from an assortment of two meat and two vegetable dishes while listening to Chinese folk music that could elicit mental imagery of nature or urban scenes. The results of Experiment 1 revealed that they chose vegetable-forward meals more frequently while listening to Chinese folk music eliciting mental imagery of nature versus urban scenes. In Experiment 2, the participants were randomly divided into three groups, in which the prevalence of their mental imagery was enhanced, moderately suppressed, or strongly suppressed by performing different tasks while listening to the music pieces. We replicated the results of Experiment 1 when the participants' mental imagery was enhanced, whereas no such effect was observed when the participants' mental imagery was moderately or strongly suppressed. Collectively, these findings suggest that music may influence the food choices people make in virtual food choice tasks by inducing mental imagery, which provides insights into utilizing environmental cues to promote healthier food choices.

Reinforced Res-Unet transformer for underwater image enhancement

Since light propagates through water with variable degrees of energy attenuation, the captured images suffer mainly from color deviations, lower contrast, and blurred details and textures. Compared with the traditional algorithm, the data-driven method has great advantages. However, the problems related to the reasonable network architecture, coding method and scarcity of databases still need to be explored to meet the requirements for high-quality reconstructed images in various tasks. In this paper, an underwater image enhancement network based on feature fusion is proposed, called RUTUIE. It combines the advantages of Resnet and the U-shape structure, mainly consists of an adaptation of multi-stage up-and-down sampling. Specifically, the U-shape structure serves as the backbone of Resnet, with two feature transformers on each side and a single level of up-and-down sampling. This structure aims to prevent insignificant features from being ignored during the process of the multi-stage up-and-down sampling. Moreover, the improved transformer encoder based on the feature-level attention mechanism combines the advantages of CNN, which allows the network to have both local and global perception. It can also consider the correlation between features in different subspaces during feature fusion. Then, we propose and demonstrate that embedding an adaptive feature selection module at appropriate locations can retain more learned feature representations. Moreover, we present how the previously proposed color transfer method can be utilized to synthesize underwater images and for network training. Extensive experiments demonstrate that our work effectively corrects color casts, reconstructs the rich texture information in natural scenes, and improves the contrast.

Advanced deep learning algorithm for instant discriminating of tea leave stress symptoms by smartphone-based detection

The primary challenges in tea production under multiple stress exposures have negatively affected its global market sustainability, so introducing an infield fast technique for monitoring tea leaves' stresses has tremendous urgent needs. Therefore, this study aimed to propose an efficient method for the detection of stress symptoms based on a portable smartphone with deep learning models. Firstly, a database containing over 10,000 images of tea garden canopies in complex natural scenes was developed, which included healthy (no stress) and three types of stress (tea anthracnose (TA), tea blister blight (TB) and sunburn (SB)). Then, YOLOv5m and YOLOv8m algorithms were adapted to discriminate the four types of stress symptoms; where the YOLOv8m algorithm achieved better performance in the identification of healthy leaves (98%), TA (92.0%), TB (68.4%) and SB (75.5%). Furthermore, the YOLOv8m algorithm was used to construct a model for differentiation of disease severity of TA, and a satisfactory result was obtained with the accuracy of mild, moderate, and severe TA infections were 94%, 96%, and 91%, respectively. Besides, we found that CNN kernels of YOLOv8m could efficiently extract the texture characteristics of the images at layer 2, and these characteristics can clearly distinguish different types of stress symptoms. This makes great contributions to the YOLOv8m model to achieve high-precision differentiation of four types of stress symptoms. In conclusion, our study provided an effective system to achieve low-cost, high-precision, fast, and infield diagnosis of tea stress symptoms in complex natural scenes based on smartphone and deep learning algorithms.

LCAMix: Local-and-contour aware grid mixing based data augmentation for medical image segmentation

Medical image segmentation often faces challenges related to overfitting, primarily due to the limited and complex training samples. This challenge often prompts the use of self-supervised learning and data augmentation. However, self-supervised learning requires well-defined hand-crafted tasks and multiple training stages. On the other hand, basic image augmentation techniques like cropping, rotation, and flipping, effective for natural scene images, have limited efficacy for medical images due to their isotropic nature.While regional dropout regularization data augmentation methods have proven effective in image recognition tasks, their application in image segmentation is not as extensively studied. Additionally, existing augmentation methods often operate on square regions, leading to the loss of crucial contour information. This is particularly problematic for medical image segmentation tasks dealing with regions of interest characterized by intricate shapes. In this work, we introduce LCAMix, a novel data augmentation approach designed for medical image segmentation. LCAMix operates by blending two images and their segmentation masks based on their superpixels, incorporating a local-and-contour-aware strategy. The training process on augmented images adopts two auxiliary pretext tasks: firstly, classifying local superpixels in augmented images using an adaptive focal margin, leveraging segmentation ground truth masks as prior knowledge; secondly, reconstructing the two source images using mixed superpixels as mutual masks, emphasizing spatial sensitivity. Our method stands out as a simple, one-stage, model-agnostic, and plug-and-play data augmentation solution applicable to various segmentation tasks. Notably, it requires no external data or additional models. Extensive experiments validate its superior performance across diverse medical segmentation datasets and tasks. The source codes are available at https://github.com/DanielaPlusPlus/DataAug4Medical.

Take a prior from other tasks for severe blur removal

Recovering clear structures from severely blurry inputs is a huge challenge due to the detail loss and ambiguous semantics. Although segmentation maps can help deblur facial images, their effectiveness is limited in complex natural scenes because they ignore the detailed structures necessary for deblurring. Furthermore, direct segmentation of blurry images may introduce error propagation. To alleviate the semantic confusion and avoid error propagation, we propose utilizing high-level vision tasks, such as classification, to learn a comprehensive prior for severe blur removal. We propose a feature learning strategy based on knowledge distillation, which aims to learn the priors with global contexts and sharp local structures. To integrate the priors effectively, we propose a semantic prior embedding layer with multi-level aggregation and semantic attention. We validate our method on natural image deblurring benchmarks by introducing the priors to various models, including UNet and mainstream deblurring baselines, to demonstrate its effectiveness and generalization ability. The results show that our approach outperforms existing methods on severe blur removal with our plug-and-play semantic priors.

YOLOv5s-CEDB: A robust and efficiency Camellia oleifera fruit detection algorithm in complex natural scenes

To solve the problem of poor recognition accuracy caused by various colors, uneven distribution, and occlusion by branches and leaves of Camellia oleifera fruits under natural growth conditions, this study proposes an improved deep learning network, YOLOv5s-CEDB, for Camellia oleifera fruit detection based on YOLOv5s. Coordinate Attention Mechanism (CoordAtt), Deformable Convolution (DConv), and Explicit Visual Center (EVC) are introduced to enhance the network’s local and global feature extraction performance. To improve the detection performance for small and dense targets, the feature fusion module of the network was replaced with the designed light-bidirectional feature pyramid (light-BiFPN) structure. GhostConv was used to reduce the parameters and inference speed of the structure. A dataset with different light conditions, colors, and density levels of Camellia oleifera fruits was established, and performance evaluation experiments were conducted. Experimental results showed that the mean Average Precision (mAP) and F1-score of the designed YOLOv5s-CEDB network reached 91.4 % and 89.6 %, respectively, which were 2.6 % and 3.6 % higher than those of the original YOLOv5s model, respectively, and the influencing frame rate arrived at 37.6 FPS. Under different colors, distribution densities, occlusion scenarios, and light intensities, the detection accuracy of the YOLOv5s-CEDB network model was significantly better than those of the YOLOv5s, YOLOv8s and Faster-RCNN networks. It was verified that the proposed YOLOv5s-CEDB network could significantly improve the accuracy and stability of Camellia oleifera fruit detection, satisfying the requirements of yield estimation and intelligent harvesting.

Feature-selective responses in macaque visual cortex follow eye movements during natural vision.

In natural vision, primates actively move their eyes several times per second via saccades. It remains unclear whether, during this active looking, visual neurons exhibit classical retinotopic properties, anticipate gaze shifts or mirror the stable quality of perception, especially in complex natural scenes. Here, we let 13 monkeys freely view thousands of natural images across 4.6 million fixations, recorded 883 h of neuronal responses in six areas spanning primary visual to anterior inferior temporal cortex and analyzed spatial, temporal and featural selectivity in these responses. Face neurons tracked their receptive field contents, indicated by category-selective responses. Self-consistency analysis showed that general feature-selective responses also followed eye movements and remained gaze-dependent over seconds of viewing the same image. Computational models of feature-selective responses located retinotopic receptive fields during free viewing. We found limited evidence for feature-selective predictive remapping and no viewing-history integration. Thus, ventral visual neurons represent the world in a predominantly eye-centered reference frame during natural vision.

A Real-Time License Plate Detection and Recognition Model in Unconstrained Scenarios.

Accurate and fast recognition of vehicle license plates from natural scene images is a crucial and challenging task. Existing methods can recognize license plates in simple scenarios, but their performance degrades significantly in complex environments. A novel license plate detection and recognition model YOLOv5-PDLPR is proposed, which employs YOLOv5 target detection algorithm in the license plate detection part and uses the PDLPR algorithm proposed in this paper in the license plate recognition part. The PDLPR algorithm is mainly designed as follows: (1) A Multi-Head Attention mechanism is used to accurately recognize individual characters. (2) A global feature extractor network is designed to improve the completeness of the network for feature extraction. (3) The latest parallel decoder architecture is adopted to improve the inference efficiency. The experimental results show that the proposed algorithm has better accuracy and speed than the comparison algorithms, can achieve real-time recognition, and has high efficiency and robustness in complex scenes.

The Influence of Competing Social and Symbolic Cues on Observers' Gaze Behaviour.

The effects of social (eye gaze, pointing gestures) and symbolic (arrows) cues on observers' attention are often studied by presenting such cues in isolation and at fixation. Here, we extend this work by embedding cues in natural scenes. Participants were presented with a single cue (Experiment 1) or a combination of cues (Experiment 2) embedded in natural scenes and were asked to 'simply look at the images' while their eye movements were recorded to assess the effects of the cues on (overt) attention. Single-gaze and pointing cues were fixated for longer than arrows but at the cost of shorter dwell times on the cued object. When presented together, gaze and pointing cues were fixated faster and for longer than simultaneously presented arrows. Attention to the cued object depended on the combination of cues and whether both cues were directed towards or away from the target object. Together, the findings confirm earlier observations that people attract attention more strongly than arrows but that arrows more strongly direct attention.