Natural Scene Images Research Articles

Façade Design Pattern Optimization Workflow Through Visual Spatial Frequency Analysis and Structural Safety Assessment

As the demand for highly efficient yet aesthetically pleasing, complex building envelope structures is rising worldwide, computational analysis and generative design tools are becoming ever so relevant. Previous methods for achieving a natural distribution of structural or shading elements in non-uniform façades are mostly based either on computer-generated pseudo-randomness or a literal biomorphic approach where a naturally occurring pattern is directly projected on the façade surface. As an alternative, this research introduces a novel technique for optimisation that utilises a two-dimensional Power Spectrum Analysis, suitable for numerically assessing the alignment of designed geometry with natural patterns. By integrating this optimisation method into the design process, the façade pattern generation can be automated and optimal design can be selected by evaluating multiple design solutions. Instead of using repetitive geometrical patterns or generated pseudo-randomness, patterns objectively similar to those occurring in nature can be created without directly copying natural structures. The distribution of the structural and shading elements controls the way natural light permeates the building and, considering the data gathered from images of natural scenes, this method can be used to design structures not only with optimal structural and energy performance but also with visual and psychological occupant comfort in mind.

A Comprehensive Review on Text Detection and Recognition in Scene Images

Detecting and recognizing text in natural scene images and videos is vital for several real-world applications, such as in the analysis of Crime scene CCTV footage, sports videos, and autonomous driving, to name a few. Therefore, one can expect several challenges, namely, arbitrarily oriented and shaped Text detection and identification in movies and natural environments. Many methods have been developed in the past to address these challenges, including advanced deep-learning models and transformers. Due to several methods available in the literature, it is not so easy to understand the open challenges, applications, directions, scope, limitations, and weaknesses of the methods. Therefore, there is a need to write a survey/review to highlight and discuss the strengths and weaknesses of the developed methods. This survey/review presents different categories of work and discusses their importance, limitations, new challenges, applications, and, finally, directions such that readers can choose appropriate methods and directions to carry out research work in the field of text detection/recognition in the natural scene and videos.

Soft set-based MSER end-to-end system for occluded scene text detection, recognition and prediction

The presence of unpredictable occlusions on natural scene text is a significant challenge, exacerbating the difficulties already posed on text detection and recognition by the variability of such images. Addressing the need for a robust, consistently performing approach that can effectively address the above challenges, this paper presents a new Soft Set-based end-to-end system for text detection, recognition and prediction in occluded natural scene images. This is the first approach to integrate text detection, recognition and prediction, unlike existing systems developed for end-to-end text spotting (text detection and recognition) only. For candidate text components detection, the proposed combination of Soft Sets with Maximally Stable Extremal Regions (SS-MSER) improves text detection and spotting in natural scene images, irrespectively of the presence of arbitrarily orientated and shaped text, complex backgrounds and occlusion. Furthermore, a Graph Recurrent Neural Network is proposed for grouping candidate text components into text lines and for fitting accurate bounding boxes to each word. Finally, a Convolutional Recurrent Neural Network (CRNN) is proposed for the recognition of text and for predicting missing characters due to occlusion. Experimental results on a new occluded scene text dataset (OSTD) and on the most relevant benchmark natural scene text datasets demonstrate that the proposed system outperforms the state-of-the-art in text detection, recognition and prediction. The code and dataset are available at https://github.com/alloydas/Softset-MSER-Based-Occluded-Scene-Text-Spotting/blob/master/Soft_set_MSER.ipynb

An End-to-End Scene Text Recognition for Bilingual Text

Text localization and recognition from natural scene images has gained a lot of attention recently due to its crucial role in various applications, such as autonomous driving and intelligent navigation. However, two significant gaps exist in this area: (1) prior research has primarily focused on recognizing English text, whereas Arabic text has been underrepresented, and (2) most prior research has adopted separate approaches for scene text localization and recognition, as opposed to one integrated framework. To address these gaps, we propose a novel bilingual end-to-end approach that localizes and recognizes both Arabic and English text within a single natural scene image. Specifically, our approach utilizes pre-trained CNN models (ResNet and EfficientNetV2) with kernel representation for localization text and RNN models (LSTM and BiLSTM) with an attention mechanism for text recognition. In addition, the AraElectra Arabic language model was incorporated to enhance Arabic text recognition. Experimental results on the EvArest, ICDAR2017, and ICDAR2019 datasets demonstrated that our model not only achieves superior performance in recognizing horizontally oriented text but also in recognizing multi-oriented and curved Arabic and English text in natural scene images.

GINet:Graph interactive network with semantic-guided spatial refinement for salient object detection in optical remote sensing images

There are many challenging scenarios in the task of salient object detection in optical remote sensing images (RSIs), such as various scales and irregular shapes of salient objects, cluttered backgrounds, etc. Therefore, it is difficult to directly apply saliency models targeting natural scene images to optical RSIs. Besides, existing models often do not give sufficient exploration for the potential relationship of different salient objects or different parts of the salient object. In this paper, we propose a graph interaction network (i.e. GINet) with semantic-guided spatial refinement to conduct salient object detection in optical RSIs. The key advantages of GINet lie in two points. Firstly, the graph interactive reasoning (GIR) module conducts information exchange of different-level features via the graph interaction operation, and enhances features along spatial and channel dimensions via the graph reasoning operation. Secondly, we designed the global content-aware refinement (GCR) module, which incorporates the foreground and background feature-based local information and the semantic feature-based global information simultaneously. Experiments results on two public optical RSIs datasets clearly show the effectiveness and superiority of the proposed GINet when compared with the state-of-the-art models.

The minimal computational substrate of fluid intelligence

The quantification of cognitive powers rests on identifying a behavioural task that depends on them. Such dependence cannot be assured, for the powers a task invokes cannot be experimentally controlled or constrained a priori, resulting in unknown vulnerability to failure of specificity and generalisability. Evaluating a compact version of Raven's Advanced Progressive Matrices (RAPM), a widely used clinical test of fluid intelligence, we show that LaMa, a self-supervised artificial neural network trained solely on the completion of partially masked images of natural environmental scenes, achieves representative human-level test scores a prima vista, without any task-specific inductive bias or training. Compared with cohorts of healthy and focally lesioned participants, LaMa exhibits human-like variation with item difficulty, and produces errors characteristic of right frontal lobe damage under degradation of its ability to integrate global spatial patterns. LaMa's narrow training and limited capacity suggest matrix-style tests may be open to computationally simple solutions that need not necessarily invoke the substrates of reasoning.

Unsupervised learning as a computational principle works in visual learning of natural scenes, but not of artificial stimuli.

The question of whether we learn exposed visual features remains a subject of controversy. A prevalent computational model suggests that visual features frequently exposed to observers in natural environments are likely to be learned. However, this unsupervised learning model appears to be contradicted by the significant body of experimental results with human participants that indicates visual perceptual learning (VPL) of visible task-irrelevant features does not occur with frequent exposure. Here, we demonstrate a resolution to this controversy with a new finding: Exposure to a dominant global orientation as task-irrelevant leads to VPL of the orientation, particularly when the orientation is derived from natural scene images, whereas VPL did not occur with artificial images even with matched distributions of local orientations and spatial frequencies to natural scene images. Further investigation revealed that this disparity arises from the presence of higher-order statistics derived from natural scene images-global structures such as correlations between different local orientation and spatial frequency channels. Moreover, behavioral and neuroimaging results indicate that the dominant orientation from these higher-order statistics undergoes less attentional suppression than that from artificial images, which may facilitate VPL. Our results contribute to resolving the controversy by affirming the validity of unsupervised learning models for natural scenes but not for artificial stimuli. They challenge the assumption that VPL occurring in everyday life can be predicted by laws governing VPL for conventionally used artificial stimuli.

Recognition And Classification of Indian Scripts in Natural Scene Images

In the field of computer vision and document analysis, the identification and categorization of Indian scripts in natural scene images pose a difficult yet crucial challenge. The variety of characters and intricate writing styles in Indian scripts require reliable solutions for precise identification under different environmental conditions. This study presents a novel CNN model designed for identifying scripts in Indian multilingual document images captured by cameras. Experimental evaluations of the model's performance were conducted with two regional languages (Odia and Telugu) and one national language (Hindi). The average accuracy in script recognition for the three language combinations is 95.66%, with Odia achieving 99.00%, Hindi 90.33%, and Telugu 98.12%. The model achieved the highest accuracy in recognition. The model achieved the highest accuracy in recognition Keywords: Text Recognition, Image Augmentation, CNN, LSTM, VGG, ResNet, DenseNet, Datasets, Natural Images

Individual differences in face salience and rapid face saccades.

Humans saccade to faces in their periphery faster than to other types of objects. Previous research has highlighted the potential importance of the upper face region in this phenomenon, but it remains unclear whether this is driven by the eye region. Similarly, it remains unclear whether such rapid saccades are exclusive to faces or generalize to other semantically salient stimuli. Furthermore, it is unknown whether individuals differ in their face-specific saccadic reaction times and, if so, whether such differences could be linked to differences in face fixations during free viewing. To explore these open questions, we invited 77 participants to perform a saccadic choice task in which we contrasted faces as well as other salient objects, particularly isolated face features and text, with cars. Additionally, participants freely viewed 700 images of complex natural scenes in a separate session, which allowed us to determine the individual proportion of first fixations falling on faces. For the saccadic choice task, we found advantages for all categories of interest over cars. However, this effect was most pronounced for images of full faces. Full faces also elicited faster saccades compared with eyes, showing that isolated eye regions are not sufficient to elicit face-like responses. Additionally, we found consistent individual differences in saccadic reaction times toward faces that weakly correlated with face salience during free viewing. Our results suggest a link between semantic salience and rapid detection, but underscore the unique status of faces. Further research is needed to resolve the mechanisms underlying rapid face saccades.

Universal scale laws for colors and patterns in imagery

Distribution of colors and patterns in images is observed through cascades that adjust spatial resolution and dynamics. Cascades of colors reveal the emergent universal property that Fully Colored Images (FCIs) of natural scenes adhere to the debated continuous linear log-scale law (slope −2.00±0.01) (L1). Cascades of discrete 2×2 patterns are derived from pixel square reductions onto the seven unlabeled rotation-free textures (0000, 0001, 0011, 0012, 0101, 0102, 0123). They exhibit an unparalleled universal entropy maximum of 1.74±0.013 at some dynamics regardless of spatial scale (L2). Patterns also adhere to the Integral Fluctuation Theorem (1.00±0.01) (L3), pivotal in studies of chaotic systems. Images with fewer colors exhibit quadratic shift and bias from L1 and L3 but adhere to L2. Randomized Hilbert fractal FCIs better match the laws than basic-to-AI-based simulations. Those results are of interest in Neural Networks, out-of-equilibrium physics, and spectral imagery.

Deep Learning Techniques for Detecting and Segmenting Text in Natural Scene Images: Review

Text detection and segmentation in natural scene images is an active research problem in computer vision and document analysis. Unlike scanned documents, scene text exhibits significant diversity in appearance, orientation, scale, font, and lighting conditions. In this review, survey the current state-of-the-art in techniques and methodologies aimed at detecting and segmenting text regions from images of natural scenes are presented. Both traditional approaches using hand-crafted features as well as modern data-driven deep learning methods will be discussed. The review will analyze common datasets, evaluation protocols and metrics used for benchmarking. Limitations of existing methods and open challenges in handling multilingual text, curved text, and efficiency will be highlighted. Promising future directions towards robust and generalizable scene text extraction systems will be identified. In summary, the review will provide a comprehensive overview of the advances, remaining challenges and future opportunities in developing automated systems for detecting and segmenting text in unconstrained natural images.

FPS-U2Net: Combining U2Net and multi-level aggregation architecture for fire point segmentation in remote sensing images

Traditional methods for fire point segmentation (FPS) in satellite remote sensing images (RSIs) overly rely on threshold judgment, which are greatly affected by factors such as regional time and show poor generalization. Besides, due to the difference between natural scene images (NSIs) and RSIs, directly apply NSIs-based deep learning methods to forest fire RSIs without any modification fails to achieve satisfactory results. To address these issues, first, we construct a Landsat8 RSI-FPS dataset covering different years, seasons and regions. Then, for the first time, we apply salient object detection (SOD) to FPS in forest fire monitoring and propose a novel network FPS-U2Net to improve the performance of FPS. FPS-U2Net is based on U2Netp (a lightweight U2Net), to make better use of the multi-level features from adjacent encoders, we propose multi-level aggregation module (MAM), which is placed between the encoder and decoder at the same stage to aggregate the adjacent multi-scale features and capture richer contextual information. To make up for the weakness of BCE loss, we employ the hybrid loss, BCE + IoU, for the training of the network, which can guide the network learn the salient information from pixel and map levels. Extensive experiments on three datasets demonstrate that our FPS-U2Net significantly outperforms the state-of-the-art semantic segmentation and SOD methods. FPS-U2Net can accurately segment fire regions and predict clear local details.

LCAMix: Local-and-contour aware grid mixing based data augmentation for medical image segmentation

Medical image segmentation often faces challenges related to overfitting, primarily due to the limited and complex training samples. This challenge often prompts the use of self-supervised learning and data augmentation. However, self-supervised learning requires well-defined hand-crafted tasks and multiple training stages. On the other hand, basic image augmentation techniques like cropping, rotation, and flipping, effective for natural scene images, have limited efficacy for medical images due to their isotropic nature.While regional dropout regularization data augmentation methods have proven effective in image recognition tasks, their application in image segmentation is not as extensively studied. Additionally, existing augmentation methods often operate on square regions, leading to the loss of crucial contour information. This is particularly problematic for medical image segmentation tasks dealing with regions of interest characterized by intricate shapes. In this work, we introduce LCAMix, a novel data augmentation approach designed for medical image segmentation. LCAMix operates by blending two images and their segmentation masks based on their superpixels, incorporating a local-and-contour-aware strategy. The training process on augmented images adopts two auxiliary pretext tasks: firstly, classifying local superpixels in augmented images using an adaptive focal margin, leveraging segmentation ground truth masks as prior knowledge; secondly, reconstructing the two source images using mixed superpixels as mutual masks, emphasizing spatial sensitivity. Our method stands out as a simple, one-stage, model-agnostic, and plug-and-play data augmentation solution applicable to various segmentation tasks. Notably, it requires no external data or additional models. Extensive experiments validate its superior performance across diverse medical segmentation datasets and tasks. The source codes are available at https://github.com/DanielaPlusPlus/DataAug4Medical.

MetaSem: metamorphic testing based on semantic information of autonomous driving scenes

AbstractThe development of artificial intelligence and information communication technology has significantly propelled advancements in autonomous driving. The advent of autonomous driving has a profound impact on societal development and transportation methods. However, as intelligent systems, autonomous driving systems (ADSs) often make wrong judgements in specific scenarios, resulting in accidents. There is an urgent need for comprehensive testing and validation of ADSs. Metamorphic testing (MT) techniques have demonstrated effectiveness in testing ADSs. Nevertheless, existing testing methods primarily encompass relatively simple metamorphic relations (MRs) that only verify ADSs from a single perspective. To ensure the safety of ADSs, it is essential to consider the various elements of driving scenarios during the testing process. Therefore, this paper proposes MetaSem, a novel metamorphic testing method based on semantic information of autonomous driving scenes. Based on semantic information of the autonomous driving scenes and traffic regulations, we design 11 MRs targeting different scenario elements. Three transformation modules are developed to execute addition, deletion and replacement operations on various scene elements within the images. Finally, corresponding evaluation metrics are defined based on MRs. MetaSem automatically discovers inconsistent behaviours according to the evaluation metrics. Our empirical study on three advanced and popular autonomous driving models demonstrates that MetaSem not only efficiently generates visually natural and realistic scene images but also detects 11,787 inconsistent behaviours on three driving models.

A Real-Time License Plate Detection and Recognition Model in Unconstrained Scenarios.

Accurate and fast recognition of vehicle license plates from natural scene images is a crucial and challenging task. Existing methods can recognize license plates in simple scenarios, but their performance degrades significantly in complex environments. A novel license plate detection and recognition model YOLOv5-PDLPR is proposed, which employs YOLOv5 target detection algorithm in the license plate detection part and uses the PDLPR algorithm proposed in this paper in the license plate recognition part. The PDLPR algorithm is mainly designed as follows: (1) A Multi-Head Attention mechanism is used to accurately recognize individual characters. (2) A global feature extractor network is designed to improve the completeness of the network for feature extraction. (3) The latest parallel decoder architecture is adopted to improve the inference efficiency. The experimental results show that the proposed algorithm has better accuracy and speed than the comparison algorithms, can achieve real-time recognition, and has high efficiency and robustness in complex scenes.

Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions

Non-native speakers with limited vocabulary often struggle to name specific objects despite being able to visualize them, e.g., people outside Australia searching for ‘numbats.’ Further, users may want to search for such elusive objects with difficult-to-sketch interactions, e.g., “numbat digging in the ground.” In such common but complex situations, users desire a search interface that accepts composite multimodal queries comprising hand-drawn sketches of “difficult-to-name but easy-to-draw” objects and text describing “difficult-to-sketch but easy-to-verbalize” object's attributes or interaction with the scene. This novel problem statement distinctly differs from the previously well-researched TBIR (text-based image retrieval) and SBIR (sketch-based image retrieval) problems. To study this under-explored task, we curate a dataset, CSTBIR (Composite Sketch+Text Based Image Retrieval), consisting of ~2M queries and 108K natural scene images. Further, as a solution to this problem, we propose a pretrained multimodal transformer-based baseline, STNet (Sketch+Text Network), that uses a hand-drawn sketch to localize relevant objects in the natural scene image, and encodes the text and image to perform image retrieval. In addition to contrastive learning, we propose multiple training objectives that improve the performance of our model. Extensive experiments show that our proposed method outperforms several state-of-the-art retrieval methods for text-only, sketch-only, and composite query modalities. We make the dataset and code available at: https://vl2g.github.io/projects/cstbir.

SAM-IE: SAM-based image enhancement for facilitating medical image diagnosis with segmentation foundation model

The Segment Anything Model (SAM) is a large-scale model developed for general segmentation tasks in computer vision. Trained on a substantial dataset, SAM can accurately segment various objects in natural scene images. However, due to significant semantic differences between medical and natural images, directly applying SAM to medical image segmentation does not yield optimal results. Therefore, effectively utilizing such a comprehensive foundation model for medical image analysis is an emerging research topic. Despite SAM’s current suboptimal performance in medical image segmentation, it shows preliminary recognition and localization of tissues and lesions that radiologists focus on in medical images. This implies that SAM’s generated masks, features, and stability scores hold potential value for medical image diagnosis. Therefore, based on the model output of SAM, this study introduces a SAM-based Image Enhancement (SAM-IE) method for disease diagnosis. Targeting popular medical image classification models (e.g., ResNet50 and Swin Transformer), SAM-IE is proposed to enhance image inputs by combining the binary mask and contour mask generated by SAM with the original image to create attention maps, thereby improving diagnostic performance. To validate the effectiveness of SAM-IE for diagnosis, experiments were conducted on four medical image datasets for eight classification tasks. The results demonstrate the effectiveness of our proposed SAM-IE model, showcasing SAM’s potential value in medical image classification. This study provides a feasible approach for integrating SAM into disease diagnosis.

Enhanced Nuclei Segmentation and Classification via Category Descriptors in the SAM Model.

Segmenting and classifying nuclei in H&E histopathology images is often limited by the long-tailed distribution of nuclei types. However, the strong generalization ability of image segmentation foundation models like the Segment Anything Model (SAM) can help improve the detection quality of rare types of nuclei. In this work, we introduce category descriptors to perform nuclei segmentation and classification by prompting the SAM model. We close the domain gap between histopathology and natural scene images by aligning features in low-level space while preserving the high-level representations of SAM. We performed extensive experiments on the Lizard dataset, validating the ability of our model to perform automatic nuclei segmentation and classification, especially for rare nuclei types, where achieved a significant detection improvement in the F1 score of up to 12%. Our model also maintains compatibility with manual point prompts for interactive refinement during inference without requiring any additional training.

Non-uniformity correction algorithm for DoFP adapted to integration time variations.

Division of the focal plane (DoFP) polarization detector is a pivotal technology in real-time polarization detection. This technology integrates a micropolarization array (MPA) onto the conventional focal plane, introducing a more intricate non-uniformity than traditional focal plane detectors. Current non-uniformity correction algorithms for DoFP are difficult to adapt to changes in integration time and perform poorly in low-polarization scenarios. Analyzing the characteristics of DoFP, formulating a pixel response model, and introducing an adaptive non-uniformity correction algorithm tailored for varying integration time. The DoFP analysis vectors are decomposed into average polarization response and unit analysis vectors for correction separately to improve the performance of the correction algorithm in different polarization scenarios. The performance of modern correction algorithms was tested and evaluated using standard uniform images, and the proposed method outperformed existing algorithms in terms of polarization measurement accuracy under the root mean square error (RMSE) metric. Moreover, in natural scene images, our proposed algorithm shows favorable visual effects and distinguishes itself from its superior stability amid changes in the integration time.

Global and Multiscale Aggregate Network for Saliency Object Detection in Optical Remote Sensing Images

Salient Object Detection (SOD) is gradually applied in natural scene images. However, due to the apparent differences between optical remote sensing images and natural scene images, directly applying the SOD of natural scene images to optical remote sensing images has limited performance in global context information. Therefore, salient object detection in optical remote sensing images (ORSI-SOD) is challenging. Optical remote sensing images usually have large-scale variations. However, the vast majority of networks are based on Convolutional Neural Network (CNN) backbone networks such as VGG and ResNet, which can only extract local features. To address this problem, we designed a new model that employs a transformer-based backbone network capable of extracting global information and remote dependencies. A new framework is proposed for this question, named Global and Multiscale Aggregate Network for Saliency Object Detection in Optical Remote Sensing Images (GMANet). In this framework, the Pyramid Vision Transformer (PVT) is an encoder to catch remote dependencies. A Multiscale Attention Module (MAM) is introduced for extracting multiscale information. Meanwhile, a Global Guiled Brach (GGB) is used to learn the global context information and obtain the complete structure. Four MAMs are densely connected to this GGB. The Aggregate Refinement Module (ARM) is used to enrich the details of edge and low-level features. The ARM fuses global context information and encoder multilevel features to complement the details while the structure is complete. Extensive experiments on two public datasets show that our proposed framework GMANet outperforms 28 state-of-the-art methods on six evaluation metrics, especially E-measure and F-measure. It is because we apply a coarse-to-fine strategy to merge global context information and multiscale information.