Natural Images Research Articles

Specular highlight removal using Quaternion transformer

Specular highlight removal is a very important issue, because specular highlight reflections in images with illumination changes can give very negative effects on various computer vision and image processing tasks. Numerous state-of-the-art networks for the specular removal use convolutional neural networks (CNN), which cannot learn global context effectively. They capture spatial information while overlooking 3D structural correlation information of an RGB image. To address this problem, we introduce a specular highlight removal network based on Quaternion transformer (QformerSHR), which employs a transformer architecture based on Quaternion representation. In particular, a depth-wise separable Quaternion convolutional layer (DSQConv) is proposed to enhance computational performance of QformerSHR, while efficiently preserving the structural correlation of an RGB image by utilizing the Quaternion representation. In addition, a Quaternion transformer block (QTB) based on DSQConv learns global context. As a result, QformerSHR consisting of DSQConv and QTB performs the specular removal from natural and text image datasets effectively. Experimental results demonstrate that it is significantly more effective than state-of-the-art networks for the specular removal, in terms of both quantitative performance and subjective quality.

Feature-specific divisive normalization improves natural image encoding for depth perception.

Vision science and visual neuroscience seek to understand how stimulus and sensor properties limit the precision with which behaviorally-relevant latent variables are encoded and decoded. In the primate visual system, binocular disparity-the canonical cue for stereo-depth perception-is initially encoded by a set of binocular receptive fields with a range of spatial frequency preferences. Here, with a stereo-image database having ground-truth disparity information at each pixel, we examine how response normalization and receptive field properties determine the fidelity with which binocular disparity is encoded in natural scenes. We quantify encoding fidelity by computing the Fisher information carried by the normalized receptive field responses. Several findings emerge from an analysis of the response statistics. First, broadband (or feature-unspecific) normalization yields Laplace-distributed receptive field responses, and narrowband (or feature-specific) normalization yields Gaussian-distributed receptive field responses. Second, the Fisher information in narrowband-normalized responses is larger than in broadband-normalized responses by a scale factor that grows with population size. Third, the most useful spatial frequency decreases with stimulus size and the range of spatial frequencies that is useful for encoding a given disparity decreases with disparity magnitude, consistent with neurophysiological findings. Fourth, the predicted patterns of psychophysical performance, and absolute detection threshold, match human performance with natural and artificial stimuli. The current computational efforts establish a new functional role for response normalization, and bring us closer to understanding the principles that should govern the design of neural systems that support perception in natural scenes.

When guided diffusion model meets zero-shot image super-resolution

Existing deep learning-based single-image super-resolution (SR) methods typically rely on vast quantities of paired data. As an essential solution, zero-shot SR methods require only a single image to handle image-specific degradation. However, these methods still struggle to recover fine-grained details due to the lack of supervised information. In this work, we propose a novel guided Diffusion model for Zero-shot image SR (ZeroDiff) to explicitly direct image quality enhancement. Specifically, we elaborate two key guidance strategies: (1) high-frequency guidance and (2) content-consistent guidance. The former concentrates on boosting fine-grained textures by embedding high-frequency information into the cross-attention mechanism of the noise estimator. The latter avoids the sampling deviating from the original image in terms of structure and low-frequency content. Specifically, the noisy images at each diffusion step are injected into the corresponding sampling step, encouraging the sampled image to be consistent with that of the corresponding diffusion step. Moreover, we design a progressive zoom-in paradigm by gradually enlarging the image size and enriching the image details to boost the sampling efficiency of diffusion models, while enabling high-quality image reconstruction. Extensive experiments reveal that our method achieves comparable results with other state-of-the-art methods in quantitative and qualitative evaluations on both face and natural images from synthetic and real-world datasets.

Embedded U-shaped network with cross-hierarchical feature adaptation fusion for remote sensing image haze removal

ABSTRACT Recently, U-Net architecture has been extensively explored for remote sensing (RS) image haze removal, demonstrating remarkable performance. However, most existing RS image haze removal methods based on the U-Net fail to fully utilize feature information extracted in the encoding layers during decoding, resulting in unsatisfactory haze removal results. Moreover, most haze removal algorithms neglect the interaction of cross-level feature information, which is particularly important for scene recovery and context information constraints. To address these issues, this paper presents an Embedded U-shaped Network with Cross-Hierarchical Feature Adaptation Fusion Network (EUCHA). Specifically, an Embedded U-shaped Network Framework (EUNF) is proposed to fuse the features extracted from different scale encoding layers as supplementary information into the corresponding decoding layers by embedding multiple parallel U-shaped subnetworks, which alleviates feature information dilution during the decoding process and enhances the network receptive field. Furthermore, we propose a Cross-Hierarchical Feature Adaptive Fusion (CHAF) module, which effectively and adaptively fuses multiple adjacent hierarchical features extracted by channel attention and pixel attention through learnable factors to enhance the expressive ability of features. Experiments were conducted on both real and synthetic datasets and compared with the nine most advanced algorithms. The results showed that EUCHA algorithm performs better in removing artefacts in both RS images and natural images. The source code is available at https://github.com/GongHongY/EUCHA.

A novel GAN-based three-axis mutually supervised super-resolution reconstruction method for rectal cancer MR image

Background and objectiveThis study aims to enhance the resolution in the axial direction of rectal cancer magnetic resonance (MR) imaging scans to improve the accuracy of visual interpretation and quantitative analysis. MR imaging is a critical technique for the diagnosis and treatment planning of rectal cancer. However, obtaining high-resolution MR images is both time-consuming and costly. As a result, many hospitals store only a limited number of slices, often leading to low-resolution MR images, particularly in the axial plane. Given the importance of image resolution in accurate assessment, these low-resolution images frequently lack the necessary detail, posing substantial challenges for both human experts and computer-aided diagnostic systems. Image super-resolution (SR), a technique developed to enhance image resolution, was originally applied to natural images. Its success has since led to its application in various other tasks, especially in the reconstruction of low-resolution MR images. However, most existing SR methods fail to account for all anatomical planes during reconstruction, leading to unsatisfactory results when applied to rectal cancer MR images. MethodsIn this paper, we propose a GAN-based three-axis mutually supervised super-resolution reconstruction method tailored for low-resolution rectal cancer MR images. Our approach involves performing one-dimensional (1D) intra-slice SR reconstruction along the axial direction for both the sagittal and coronal planes, coupled with inter-slice SR reconstruction based on slice synthesis in the axial direction. To further enhance the accuracy of super-resolution reconstruction, we introduce a consistency supervision mechanism across the reconstruction results of different axes, promoting mutual learning between each axis. A key innovation of our method is the introduction of Depth-GAN for synthesize intermediate slices in the axial plane, incorporating depth information and leveraging Generative Adversarial Networks (GANs) for this purpose. Additionally, we enhance the accuracy of intermediate slice synthesis by employing a combination of supervised and unsupervised interactive learning techniques throughout the process. ResultsWe conducted extensive ablation studies and comparative analyses with existing methods to validate the effectiveness of our approach. On the test set from Shanxi Cancer Hospital, our method achieved a Peak Signal-to-Noise Ratio (PSNR) of 34.62 and a Structural Similarity Index (SSIM) of 96.34 %. These promising results demonstrate the superiority of our method.

Human EEG and artificial neural networks reveal disentangled representations of object real-world size in natural images

Human EEG and artificial neural networks reveal disentangled representations of object real-world size in natural images

Neural Responses to Natural Versus AI-generated Affective Images

Neural Responses to Natural Versus AI-generated Affective Images

Quantifying the role of perceived curvature in the processing of natural object images

Quantifying the role of perceived curvature in the processing of natural object images

Perceiving natural images may consume less cognitive resources: evidence from image memorability, edge magnitudes, and spectral content

Perceiving natural images may consume less cognitive resources: evidence from image memorability, edge magnitudes, and spectral content

The high-dimensional structure of natural image representations varies systematically across visual cortex

The high-dimensional structure of natural image representations varies systematically across visual cortex

Relative depth discrimination in natural images of paired human body joints

Relative depth discrimination in natural images of paired human body joints

Free-viewing of static natural images and movies in marmoset monkeys.

Free-viewing of static natural images and movies in marmoset monkeys.

The effect of image size and defocus on children’s reflex vergence eye movements to natural images

The effect of image size and defocus on children’s reflex vergence eye movements to natural images

Leukocyte classification using relative-relationship-guided contrastive learning

Hematologic diseases and blood disorders can be studied through microscopic examination of blood smear images or chemical assays. Many researchers are focused on utilizing deep learning (DL) to identify, quantify, and classify various types of blood cells, which is crucial for addressing theoretical and practical challenges in disease diagnosis and treatment planning. However, there is a significant deficiency in annotations for leukocyte classification, limiting the effectiveness of current DL methods. Contrastive learning (CL) facilitates the extraction of prior knowledge from unlabeled data, reducing the dependency on manual annotations. While CL demonstrates remarkable achievements in classifying natural images, the direct application to leukocyte classification presents certain defects. We observe that standard data augmentation techniques in CL exhibit varying sensitivity to different categories of white blood cell images. Employing a uniform augmentation strategy may lead the network into an optimization trap, acquiring inadequate representations from erroneous sample relationships. To address this issue, we propose a Relative-Relationship-Guided Contrastive Learning Representation (ReCLR) framework for the leukocyte classification. ReCLR mines positive and negative samples based on relative distance knowledge. Specifically, we provide adversarial guidance for the positive sample, restricting the distance between the positive sample and the original sample less than but as close as possible to the distance between the original sample and the farthest negative sample. Then, we utilize the entropy constraint to regulate the relative distance relationship between the negative and original samples. Finally, the guided positive and negative samples are employed for contrastive learning in leukocyte classification. The extensive experiments, including linear evaluation, domain transfer, and fine-tuning, demonstrate the effectiveness of the proposed method. Our ReCLR achieves accuracies of 92.07%, 65.36%, and 92.49% on three real-world leukocyte datasets, respectively, outperforming several state-of-the-art methods. The source code is released at https://github.com/AlchemyEmperor/ReCLR.

The Adaption of Recent New Concepts in Neural Radiance Fields and Their Role for High-Fidelity Volume Reconstruction in Medical Images.

Volume reconstruction techniques are gaining increasing interest in medical domains due to their potential to learn complex 3D structural information from sparse 2D images. Recently, neural radiance fields (NeRF), which implicitly model continuous radiance fields based on multi-layer perceptrons to enable volume reconstruction of objects at arbitrary resolution, have gained traction in natural image volume reconstruction. However, the direct application of NeRF to medical volume reconstruction presents unique challenges due to differences in imaging principles, internal structure requirements, and boundary delineation. In this paper, we evaluate different NeRF techniques developed for natural images, including sampling strategies, feature encoding, and the use of complimentary features, by applying them to medical images. We evaluate three state-of-the-art NeRF techniques on four datasets of medical images of different complexity. Our goal is to identify the strengths, limitations, and future directions for integrating NeRF into the medical domain.

Fixational eye movements enhance the precision of visual information transmitted by the primate retina

Fixational eye movements alter the number and timing of spikes transmitted from the retina to the brain, but whether these changes enhance or degrade the retinal signal is unclear. To quantify this, we developed a Bayesian method for reconstructing natural images from the recorded spikes of hundreds of retinal ganglion cells (RGCs) in the macaque retina (male), combining a likelihood model for RGC light responses with the natural image prior implicitly embedded in an artificial neural network optimized for denoising. The method matched or surpassed the performance of previous reconstruction algorithms, and provides an interpretable framework for characterizing the retinal signal. Reconstructions were improved with artificial stimulus jitter that emulated fixational eye movements, even when the eye movement trajectory was assumed to be unknown and had to be inferred from retinal spikes. Reconstructions were degraded by small artificial perturbations of spike times, revealing more precise temporal encoding than suggested by previous studies. Finally, reconstructions were substantially degraded when derived from a model that ignored cell-to-cell interactions, indicating the importance of stimulus-evoked correlations. Thus, fixational eye movements enhance the precision of the retinal representation.

Insights into the Development Trends of Industrial Large Language Models

In recent years, Large Language Models (LLMs) with massive parameters and complex structures have achieved significant breakthroughs in fields such as natural language processing and image generation, driving their widespread application in industrial sectors. Despite the enormous potential of industrial AI models in areas like design and development, monitoring and management, quality control, and maintenance, their actual construction and deployment still face a lot of challenges, including inherent model deficiencies and difficulties in aligning with industrial requirements. Future technological development trends include the generation of customized industrial datasets, the collaborative optimization of large and small models, the enhancement of adaptive capabilities, and the application of Retrieval-Augmented Generation (RAG) technology. These trends are expected to improve the effectiveness and scalability of AI models, better meeting the needs of the industrial domain. This paper systematically discusses the challenges, technological development trends, and practical applications and deployment of industrial AI models, providing valuable insights for future directions.

Quaternion-Aware Low-Rank Prior for Blind Color Image Deblurring

Blind image deblurring is a critical and challenging task in the field of imaging science due to its severe ill-posedness. Appropriate prior information and regularizations are normally introduced to alleviate this problem. Inspired by the fact that the matrix representing a natural image is intrinsically low-rank or approximately low-rank, we employ the low-rank matrix approximation (LRMA) approach for tackling blind image deblurring problems with unknown kernels. When applied to color image restoration tasks, making use of the quaternion representation in the hypercomplex domain enables us to better illustrate the inner relationships among color channels and thus more accurately characterize color image structure. Following this idea, we develop a novel model for color image blind deblurring by implementing the quaternion representation to the LRMA method. This proposed model facilitates better results for blur kernel estimation through preserving the sharper color intermediate latent image, which is first implemented for addressing the blind color image deblurring problem. Extensive numerical experiments demonstrate that our proposed quaternion-aware low-rank prior model greatly improves the performance when compared with the conventional low-rank based scheme and outperforms some of the state-of-the-art methods in terms of some criteria and visual quality.

An End-to-End Scene Text Recognition for Bilingual Text

Text localization and recognition from natural scene images has gained a lot of attention recently due to its crucial role in various applications, such as autonomous driving and intelligent navigation. However, two significant gaps exist in this area: (1) prior research has primarily focused on recognizing English text, whereas Arabic text has been underrepresented, and (2) most prior research has adopted separate approaches for scene text localization and recognition, as opposed to one integrated framework. To address these gaps, we propose a novel bilingual end-to-end approach that localizes and recognizes both Arabic and English text within a single natural scene image. Specifically, our approach utilizes pre-trained CNN models (ResNet and EfficientNetV2) with kernel representation for localization text and RNN models (LSTM and BiLSTM) with an attention mechanism for text recognition. In addition, the AraElectra Arabic language model was incorporated to enhance Arabic text recognition. Experimental results on the EvArest, ICDAR2017, and ICDAR2019 datasets demonstrated that our model not only achieves superior performance in recognizing horizontally oriented text but also in recognizing multi-oriented and curved Arabic and English text in natural scene images.

Impact of Display Sub-Pixel Arrays on Perceived Gloss and Transparency.

In recent years, improvements in display image quality have made it easier to perceive rich object information, such as gloss and transparency, from images, known as shitsukan. Do the different display specifications in the world affect their appearance? Clarifying the effects of differences in pixel structure on shitsukan perception is necessary to realize shitsukan management for displays with different hardware structures, which has not been fully clarified before. In this study, we experimentally investigated the effects of display pixel arrays on the perception of glossiness and transparency. In a visual evaluation experiment, we investigated the effects of three types of sub-pixel arrays (RGB, RGBW, and PenTile) on the perception of glossiness and transparency using natural images. The results confirmed that sub-pixel arrays affect the appearance of glossiness and transparency. A general relationship of RGB > PenTile > RGBW for glossiness and RGB > RGBW > PenTile for transparency was found; however, detailed analysis, such as cluster analysis, confirmed that the relative superiority of these sub-pixel arrays may vary depending on the observer and image content.