Natural Scenes Research Articles

Spotting the Unseen: Reciprocal Consensus Network Guided by Visual Archetypes

Humans often require only a few visual archetypes to spot novel objects. Based on this observation, we present a strategy rooted in ``spotting the unseen" by establishing dense correspondences between potential query image regions and a visual archetype, and we propose the Consensus Network (CoNet). Our method leverages relational patterns intra and inter images via Auto-Correlation Representation (ACR) and Mutual-Correlation Representation (MCR). Within each image, the ACR module is capable of encoding both local self-similarity and global context simultaneously. Between the query and support images, the MCR module computes the cross-correlation across two image representations and introduces a reciprocal consistency constraint, which can incorporate to exclude outliers and enhance model robustness. To overcome the challenges of low-resource training data, particularly in one-shot learning scenarios, we incorporate an adaptive margin strategy to better handle diverse instances. The experimental results indicate the effectiveness of the proposed method across diverse domains such as object detection in natural scenes, and text spotting in both historical manuscripts and natural scenes, which demonstrates its sparkling generalization ability. Our code is available at: https://github.com/infinite-hwb/conet.

ViTEraser: Harnessing the Power of Vision Transformers for Scene Text Removal with SegMIM Pretraining

Scene text removal (STR) aims at replacing text strokes in natural scenes with visually coherent backgrounds. Recent STR approaches rely on iterative refinements or explicit text masks, resulting in high complexity and sensitivity to the accuracy of text localization. Moreover, most existing STR methods adopt convolutional architectures while the potential of vision Transformers (ViTs) remains largely unexplored. In this paper, we propose a simple-yet-effective ViT-based text eraser, dubbed ViTEraser. Following a concise encoder-decoder framework, ViTEraser can easily incorporate various ViTs to enhance long-range modeling. Specifically, the encoder hierarchically maps the input image into the hidden space through ViT blocks and patch embedding layers, while the decoder gradually upsamples the hidden features to the text-erased image with ViT blocks and patch splitting layers. As ViTEraser implicitly integrates text localization and inpainting, we propose a novel end-to-end pretraining method, termed SegMIM, which focuses the encoder and decoder on the text box segmentation and masked image modeling tasks, respectively. Experimental results demonstrate that ViTEraser with SegMIM achieves state-of-the-art performance on STR by a substantial margin and exhibits strong generalization ability when extended to other tasks, e.g., tampered scene text detection. Furthermore, we comprehensively explore the architecture, pretraining, and scalability of the ViT-based encoder-decoder for STR, which provides deep insights into the application of ViT to the STR field. Code is available at https://github.com/shannanyinxiang/ViTEraser.

DICAM: Deep Inception and Channel-wise Attention Modules for underwater image enhancement

In underwater environments, imaging devices suffer from water turbidity, attenuation of lights, scattering, and particles, leading to low quality, poor contrast, and biased color images. This has led to great challenges for underwater condition monitoring and inspection using conventional vision techniques. In recent years, underwater image enhancement has attracted increasing attention due to its critical role in improving the performance of current computer vision tasks in underwater object detection and segmentation. As existing methods, built mainly from natural scenes, have performance limitations in improving the color richness and distributions we propose a novel deep learning-based approach namely Deep Inception and Channel-wise Attention Modules (DICAM) to enhance the quality, contrast, and color cast of the hazy underwater images. The proposed DICAM model enhances the quality of underwater images, considering both the proportional degradations and non-uniform color cast. Extensive experiments on two publicly available underwater image enhancement datasets have verified the superiority of our proposed model compared with several state-of-the-art conventional and deep learning-based methods in terms of full-reference and reference-free image quality assessment metrics. The source code of our DICAM model is available at https://github.com/hfarhaditolie/DICAM.

Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation

Adversarial Robustness Distillation (ARD) is a promising task to solve the issue of limited adversarial robustness of small capacity models while optimizing the expensive computational costs of Adversarial Training (AT). Despite the good robust performance, the existing ARD methods are still impractical to deploy in natural high-security scenes due to these methods rely entirely on original or publicly available data with a similar distribution. In fact, these data are almost always private, specific, and distinctive for scenes that require high robustness. To tackle these issues, we propose a challenging but significant task called Data-Free Adversarial Robustness Distillation (DFARD), which aims to train small, easily deployable, robust models without relying on data. We demonstrate that the challenge lies in the lower upper bound of knowledge transfer information, making it crucial to mining and transferring knowledge more efficiently. Inspired by human education, we design a plug-and-play Interactive Temperature Adjustment (ITA) strategy to improve the efficiency of knowledge transfer and propose an Adaptive Generator Balance (AGB) module to retain more data information. Our method uses adaptive hyperparameters to avoid a large number of parameter tuning, which significantly outperforms the combination of existing techniques. Meanwhile, our method achieves stable and reliable performance on multiple benchmarks.

End-to-End Real-Time Vanishing Point Detection with Transformer

In this paper, we propose a novel transformer-based end-to-end real-time vanishing point detection method, which is named Vanishing Point TRansformer (VPTR). The proposed method can directly regress the locations of vanishing points from given images. To achieve this goal, we pose vanishing point detection as a point object detection task on the Gaussian hemisphere with region division. Considering low-level features always provide more geometric information which can contribute to accurate vanishing point prediction, we propose a clear architecture where vanishing point queries in the decoder can directly gather multi-level features from CNN backbone with deformable attention in VPTR. Our method does not rely on line detection or Manhattan world assumption, which makes it more flexible to use. VPTR runs at an inferring speed of 140 FPS on one NVIDIA 3090 card. Experimental results on synthetic and real-world datasets demonstrate that our method can be used in both natural and structural scenes, and is superior to other state-of-the-art methods on the balance of accuracy and efficiency.

Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions

Non-native speakers with limited vocabulary often struggle to name specific objects despite being able to visualize them, e.g., people outside Australia searching for ‘numbats.’ Further, users may want to search for such elusive objects with difficult-to-sketch interactions, e.g., “numbat digging in the ground.” In such common but complex situations, users desire a search interface that accepts composite multimodal queries comprising hand-drawn sketches of “difficult-to-name but easy-to-draw” objects and text describing “difficult-to-sketch but easy-to-verbalize” object's attributes or interaction with the scene. This novel problem statement distinctly differs from the previously well-researched TBIR (text-based image retrieval) and SBIR (sketch-based image retrieval) problems. To study this under-explored task, we curate a dataset, CSTBIR (Composite Sketch+Text Based Image Retrieval), consisting of ~2M queries and 108K natural scene images. Further, as a solution to this problem, we propose a pretrained multimodal transformer-based baseline, STNet (Sketch+Text Network), that uses a hand-drawn sketch to localize relevant objects in the natural scene image, and encodes the text and image to perform image retrieval. In addition to contrastive learning, we propose multiple training objectives that improve the performance of our model. Extensive experiments show that our proposed method outperforms several state-of-the-art retrieval methods for text-only, sketch-only, and composite query modalities. We make the dataset and code available at: https://vl2g.github.io/projects/cstbir.

An Improved NLCS Algorithm Based on Series Reversion and Elliptical Model Using Geosynchronous Spaceborne–Airborne UHF UWB Bistatic SAR for Oceanic Scene Imaging

Geosynchronous spaceborne–airborne (GEO-SA) ultra-high-frequency ultra-wideband bistatic synthetic aperture radar (UHF UWB BiSAR) provides high-precision images for marine and polar environments, which are pivotal in glacier monitoring and sea ice thickness measurement for polar ocean mapping and navigation. Contrasting with traditional high-frequency BiSAR, it faces unique challenges, such as the considerable spatial variability, significant range–azimuth coupling, and vast volumes of echo data, which impede high-resolution image reconstruction. This paper presents an improved bistatic nonlinear chirp scaling (NLCS) algorithm for imaging oceanic scenes with GEO-SA UHF UWB BiSAR. This methodology extends the two-dimensional (2-D) spectrum up to the sixth order via the method of series reversion (MSR) to meet accuracy demands and then employs an elliptical model to elucidate the alterations in the azimuth frequency modulation (FM) rate mismatch. Initially, the imaging geometry and signal model are introduced, and then a separation of bistatic slant ranges based on the configuration is proposed. In addition, during range processing, after eliminating linear range cell migration (RCM), the derivation process for the sixth-order 2-D spectrum is detailed and an improved filter is applied to correct the high-order RCM. Finally, during azimuth processing, the causes of the FM rate mismatch are analyzed, a cubic perturbation function derived from the elliptical model is used for FM rate equalization, and a unified sixth-order filter is applied to complete the azimuth compression. Experimental results with point targets and natural oceanic scenes validate the outstanding efficacy of the proposed NLCS algorithm, particularly in imaging quality enhancements for GEO-SA UHF UWB BiSAR.

Predicted effectiveness of EnChroma multi-notch filters for enhancing color perception in anomalous trichromats

EnChroma filters are aids designed to improve color vision for anomalous trichromats. Their use is controversial because the results of lab-based assessments of their effectiveness have so far largely failed to agree with positive anecdotal reports. However, the effectiveness of EnChroma filters will vary depending on the conditions of viewing, including whether the stimuli are broadband reflective surfaces or colors presented on RGB displays, whether illumination spectra are broadband or narrowband, the transmission spectra of particular filters, and the cone spectral sensitivity functions of the observer. We created a model of anomalous trichromatic color vision to predict the effects of EnChroma filters on the color signals impaired in anomalous trichromacy. Using the model we varied illumination, filter type and observer cone sensitivity functions, and tested the effect of presenting colors as broadband reflective surfaces or on RGB displays. We also used hyperspectral images to assess the impact of the filters on anomalous trichromats’ color vision for natural scenes. Model results predicted that the filters should be broadly effective at enhancing anomalous trichromats’ equivalent to L/(L + M) chromatic contrasts under a range of viewing conditions, but are substantially more effective for deuteranomals than for protanomals. The filters are predicted to be more effective for broadband reflective surfaces presented under broadband illuminants than for surfaces presented under narrowband illuminants or for colors presented on RGB displays. Since the potential impacts of contrast adaptation and perceptual learning are not considered in the model, it needs to be empirically validated. Results of empirical tests of the effects of EnChroma filters on deuteranomalous color vision in comparison with model predictions are presented in an accompanying paper (Somers et al., in prep.).

SAM-IE: SAM-based image enhancement for facilitating medical image diagnosis with segmentation foundation model

The Segment Anything Model (SAM) is a large-scale model developed for general segmentation tasks in computer vision. Trained on a substantial dataset, SAM can accurately segment various objects in natural scene images. However, due to significant semantic differences between medical and natural images, directly applying SAM to medical image segmentation does not yield optimal results. Therefore, effectively utilizing such a comprehensive foundation model for medical image analysis is an emerging research topic. Despite SAM’s current suboptimal performance in medical image segmentation, it shows preliminary recognition and localization of tissues and lesions that radiologists focus on in medical images. This implies that SAM’s generated masks, features, and stability scores hold potential value for medical image diagnosis. Therefore, based on the model output of SAM, this study introduces a SAM-based Image Enhancement (SAM-IE) method for disease diagnosis. Targeting popular medical image classification models (e.g., ResNet50 and Swin Transformer), SAM-IE is proposed to enhance image inputs by combining the binary mask and contour mask generated by SAM with the original image to create attention maps, thereby improving diagnostic performance. To validate the effectiveness of SAM-IE for diagnosis, experiments were conducted on four medical image datasets for eight classification tasks. The results demonstrate the effectiveness of our proposed SAM-IE model, showcasing SAM’s potential value in medical image classification. This study provides a feasible approach for integrating SAM into disease diagnosis.

Cracks-suppression perceptual geometry coding for dynamic point clouds

Dynamic point clouds can effectively describe 3D objects and natural scenes, providing users with an immersive visual experience, but their huge amount of data requires efficient compression tools. To this end, the video-based point cloud compression (V-PCC) standard was developed, however, it may lead to cracks in the compressed point cloud at low bitrates. To solve the problem, this paper proposes a cracks-suppression perceptual geometry coding method for dynamic point cloud. Firstly, considering that the edge regions of 2D geometry patches in V-PCC are prone to cracks, a prior knowledge based crack region detection and preprocessing scheme is designed to reduce cracks. Secondly, considering the unique perceptual geometry characteristics of point cloud, a projected geometry curvature based structural similarity (PGC-SSIM) is proposed to evaluate geometry quality of point clouds, which contains the geometric perception information of point cloud, and is more consistent with the human visual perception. Finally, based on the PGC-SSIM, an adaptive quantization parameter adjustment strategy is designed for rate-distortion optimization in geometry coding of dynamic point clouds. The experimental results show that the proposed method can effectively reduce cracks in the compressed point cloud without reducing the coding performance compared to the V-PCC anchor. Moreover, the presented PGC-SSIM can be used to improve the visual quality of the compressed point cloud.

Lensless opto-electronic neural network with quantum dot nonlinear activation

With the swift advancement of neural networks and their expanding applications in many fields, optical neural networks have gradually become a feasible alternative to electrical neural networks due to their parallelism, high speed, low latency, and power consumption. Nonetheless, optical nonlinearity is hard to realize in free-space optics, which restricts the potential of the architecture. To harness the benefits of optical parallelism while ensuring compatibility with natural light scenes, it becomes essential to implement two-dimensional spatial nonlinearity within an incoherent light environment. Here, we demonstrate a lensless opto-electrical neural network that incorporates optical nonlinearity, capable of performing convolution calculations and achieving nonlinear activation via a quantum dot film, all without an external power supply. Through simulation and experiments, the proposed nonlinear system can enhance the accuracy of image classification tasks, yielding a maximum improvement of 5.88% over linear models. The scheme shows a facile implementation of passive incoherent two-dimensional nonlinearities, paving the way for the applications of multilayer incoherent optical neural networks in the future.

Enhanced Nuclei Segmentation and Classification via Category Descriptors in the SAM Model.

Segmenting and classifying nuclei in H&E histopathology images is often limited by the long-tailed distribution of nuclei types. However, the strong generalization ability of image segmentation foundation models like the Segment Anything Model (SAM) can help improve the detection quality of rare types of nuclei. In this work, we introduce category descriptors to perform nuclei segmentation and classification by prompting the SAM model. We close the domain gap between histopathology and natural scene images by aligning features in low-level space while preserving the high-level representations of SAM. We performed extensive experiments on the Lizard dataset, validating the ability of our model to perform automatic nuclei segmentation and classification, especially for rare nuclei types, where achieved a significant detection improvement in the F1 score of up to 12%. Our model also maintains compatibility with manual point prompts for interactive refinement during inference without requiring any additional training.

Understimulation resembles overstimulation: Effects on school children's attentional performance, affect, and environmental preference

Research on restorative environments has long examined the benefits of nature exposure for people whose resources have been depleted due to a situation that is too demanding for the individual. We argue that people's resources can also be taxed in understimulating situations, in which there is a lack of sensory, cognitive, social and/or behavioral stimulation, leading to a need for recovery. Exploring this phenomenon forms the main objective of this study. Given the potential benefits that contact with nature in educational centres has for children, we chose young children (N = 145, Mage = 9.34) at school as participants. We conducted a 3 (stimulation: under, over, and control) by 2 (environments: natural, urban) by 3 (time: before stimulation treatment, just after, and after exposure to environment) mixed-mode experiment. The environments were simulated through a series of slides presented on a screen in the classrooms where also the initial part of the experiment took place. Our results show that both under- and overstimulating situations are taxing for children's capacity for attention, and that understimulation decreases children's positive affect. After overstimulation, exposure to natural scenes restored attentional capacities, while exposure to urban scenes did not. After understimulation, exposure to any of the environments (nature; urban) restored children's attentional capabilities and lifted their mood. Future research could focus especially on low stimulation/low meaning situations to better understand their negative effects on attention and mood.

A new framework for improving semantic segmentation in aerial imagery

High spatial resolution (HSR) remote sensing imagery presents a rich tapestry of foreground-background intricacies, rendering semantic segmentation in aerial contexts a formidable and vital undertaking. At its core, this challenge revolves around two pivotal questions: 1) Mitigating Background Interference and Enhancing Foreground Clarity. 2) Accurate Segmentation in Dense Small Object Cluster. Conventional semantic segmentation methods primarily cater to the segmentation of large-scale objects in natural scenes, yet they often falter when confronted with aerial imagery’s characteristic traits such as vast background areas, diminutive foreground objects, and densely clustered targets. In response, we propose a novel semantic segmentation framework tailored to overcome these obstacles. To address the first challenge, we leverage PointFlow modules in tandem with the Foreground-Scene (F-S) module. PointFlow modules act as a barrier against extraneous background information, while the F-S module fosters a symbiotic relationship between the scene and foreground, enhancing clarity. For the second challenge, we adopt a dual-branch structure termed disentangled learning, comprising Foreground Precedence Estimation and Small Object Edge Alignment (SOEA). Our foreground saliency guided loss optimally directs the training process by prioritizing foreground examples and challenging background instances. Extensive experimentation on the iSAID and Vaihingen datasets validates the efficacy of our approach. Not only does our method surpass prevailing generic semantic segmentation techniques, but it also outperforms state-of-the-art remote sensing segmentation methods.

Composite descriptor based on contour and appearance for plant species identification

In this paper, we propose an effective composite descriptor combining contour and appearance features for plant species identification. The composite descriptor uses two different fusion schemes: the first is a method based on the fusion of low-level local contour and appearance features; that is, we use local triangle features and local speeded-up robust features (SURF) to describe leaf images, respectively. Then, we utilize the Hausdorff distance to measure the leaf discrepancy to complete the plant identification task. The second is a method of global feature fusion based on high-level shape and appearance features. Specifically, the Fisher vector encodes the local triangle, and SURF features into high-level shape and appearance features. Then, the Euclidean distance is used to compute the dissimilarity of leaves to realize plant leaf recognition. The composite descriptor based on these two fusion schemes can well describe the characteristics of leaf images and improve the ability to identify plant leaves. This composite descriptor is extensively tested on several standard plant databases, including simple, compound, and natural scene plant leaf datasets. Experimental results demonstrate that the composite descriptor achieves high recognition accuracy, outperforming existing state-of-the-art benchmarks. In addition, we further applied our method directly to shape recognition and achieved good recognition results, demonstrating the wide range of applications of our method.

Can natural scenes cue attention to multiple locations? Evidence from eye-movements in contextual cueing

Humans find visual targets more quickly when the target appears at the same location in a repeated configuration of other stimuli. However, when the target alternates between two locations in the repeated configuration, the benefit for visual search is smaller. This reduction of benefits has been explained as the result of an averaging of a benefit for one location and a cost for the other location. In two experiments, we investigated this two-target-locations effect in real-world scenes using high-resolution eye-tracking. Experiment 1 adapted a study in which subjects searched for a small “T” or “L” superimposed on real-world photographs. Half of the trials showed repeated scenes with one possible target location each; half showed novel scenes. We replicated the pronounced contextual cueing effect in real-world scenes. In Experiment 2, two conditions were added. In one of them, targets appeared in repeated scenes alternating between two possible locations per scene. In the other condition, targets appeared in repeated scenes but at new locations, constrained to one side of the screen. Subjects were faster to search for and identify a target in repeated scenes than in novel scenes, including when the scene was paired with two alternating target locations and (after extensive training) even when the scene only predicted the hemifield. Separate analyses on the two possible target locations resulted in rejection of the suggestion of costs for the additional target location, since the contextual cueing effect was present in the second half of the experiment for both the favored and the less favored target location. The eye-tracking data demonstrated that contextual cueing influences searching fixations, characteristic of attentional guidance, rather than responding fixations, characteristic of facilitation of response processes. Further, these data revealed that adding another possible target location leads to less guidance, rather than impeding response processes. Thus, this study delivers evidence for a flexible and attentional guidance mechanism that is able to prioritize more than one location in natural contexts.

Individual differences in human gaze behavior generalize from faces to objects

Individuals differ in where they fixate on a face, with some looking closer to the eyes while others prefer the mouth region. These individual biases are highly robust, generalize from the lab to the outside world, and have been associated with social cognition and associated disorders. However, it is unclear, whether these biases are specific to faces or influenced by domain-general mechanisms of vision. Here, we juxtaposed these hypotheses by testing whether individual face fixation biases generalize to inanimate objects. We analyzed >1.8 million fixations toward faces and objects in complex natural scenes from 405 participants tested in multiple labs. Consistent interindividual differences in fixation positions were highly inter-correlated across faces and objects in all samples. Observers who fixated closer to the eye region also fixated higher on inanimate objects and vice versa. Furthermore, the inter-individual spread of fixation positions scaled with target size in precisely the same, non-linear manner for faces and objects. These findings contradict a purely domain-specific account of individual face gaze. Instead, they suggest significant domain-general contributions to the individual way we look at faces, a finding with potential relevance for basic vision, face perception, social cognition, and associated clinical conditions.

Unsupervised remote sensing image thin cloud removal method based on contrastive learning

AbstractCloud removal algorithm is a crucial step of remote sensing image preprocessing. The current mainstream remote sensing image cloud removal algorithms are implemented based on deep learning, and most of them are supervised. A large number of data pairs are required for training to achieve cloud removal. However, real with/without cloud image pairs datasets are difficult to obtain in the real world, and the models obtained by training on synthetic datasets often need to generalize better to natural scenes. And the existing unsupervised thin cloud removal methods based on Cycle‐GAN framework with considerable model complexity and unstable training are not an excellent solution to the problem of lack of paired datasets. Based on this, in this paper, the authors propose an unsupervised remote sensing image thin cloud removal method based on contrastive learning—GAN‐UD. It is a network consisting of a frequency‐spatial attention generator and a discriminator. In addition, the authors introduce local contrastive loss and global content loss to constrain the content of the generated images to ensure that the generated cloud‐free images are consistent with the input cloud images in terms of image content. Experimental results show that the proposed method in this paper can still effectively remove thin clouds from remote sensing images without paired training datasets, outperforms current unsupervised cloud removal methods, and achieves comparable performance to supervised methods.

An Optically Augmented Visual Aid for Individuals with Age-Related Macular Degeneration

Normal vision is a precious gift to mankind. Any vision defect or degradation is actually an intimidating problem for individuals and societies. Therefore, researchers are continually working to find effective solutions for vision disorders. In some retinal diseases such as Age-related Macular Degeneration (AMD), visual aids are required to improve vision ability and/or stop the progress of the disease. Recently, augmented vision techniques have been used to provide aid to people suffering from retinal impairment. However, in such techniques, the images of real scenes are electronically deformed to compensate for vision impairment. Therefore, the natural scene is displayed as an electronic image on glasses. Intuitively, it is annoying to the patient to see electronic rather than natural scenes. Moreover, these visual aids are bulky and produce electric fields that might be harmful with continuous use. In this work, a novel optical solution to provide a visual aid to patients with central vision loss has been proposed. The proposed optical solution deforms the wavefront of the scene to entirely fall on the healthy parts of the retina. This, in turn, conveys all scene information to the brain to be perceived by the patient. As it provides optical processing, the proposed solution overcomes all drawbacks of the electronic solutions. To prove the validity of the proposed solution, three lenses were designed, fabricated, and tested to visualize simple shapes, reading, and obtaining aid during walking and driving. Obtaining the expected results from these tests, they were tried by three volunteers to clinically prove the validity and feasibility of the proposed optical aid. The feedback from the three patients was promising since all of them could recognize some of the details they used to miss with at least one of the lenses.

A Real-Time Deep Learning-Based Facial Mask Detection System for Preventing the Transmission of Respiratory Viruses

In order to prevent the spread of respiratory viruses like COVID-19 and influenza, an effective protection method is to wear facial masks in densely populated areas. This has led to a growing need for smart services that automatically detect facial masks and replace manual reminding. To address this challenging task and to contribute towards health safety, this paper introduces MedNetV2 system, which is an efficient deep learning-based facial mask detector with a low computational cost. In comparison with existing systems, the main specificities of the proposed system are: (1) the adoption of an effective deep learning-based framework to deal with both the large scale diversity and position variations of masked faces involved in natural scenes, (2) the interaction between face localization and facial mask detection modules to achieve the overall system goal, (3) the lightweight design and the real-time response well suited for real-world scenarios. Extensive experiments on public dataset and real-world video streaming are carried out to validate quantitatively and visually the effectiveness of the proposed system. Promising results, in terms of detection accuracy as well as time response, are achieved when compared it with other state-of-the-art systems.