Human Visual Characteristics Research Articles

Content-Based Image Retrieval and Feature Extraction: Analysing the Literature

A significant amount of multimedia data consists of digital images, and multimedia content analysis is used in many real-world computer vision applications. Multimedia information, especially photos, has become much more complicated in the last several years. Every day, millions of photos are posted to various websites, such as Instagram, Facebook, and Twitter. Finding a suitable image in an archive is a difficult research subject for the field of computer vision. Most search engines use standard text-based techniques that depend on metadata and captions in order to fetch photos. Over the past 20 years, a great deal of research has been conducted on content-based image retrieval (CBIR), picture categorization, and analysis. In image classification models and CBIR, high-level picture representations are represented as feature vectors made up of numerical values. Empirical evidence indicates a considerable disparity between picture feature representation and human visual understanding. Reducing the semantic gap between human visual understanding and picture feature representation is the aim of this study. This study aims to do a thorough analysis of the latest advancements in the domains of Content-Based picture Retrieval and picture representation. We performed a comprehensive analysis of many models for image retrieval and picture representation, encompassing the most recent advancements in semantic deep-learning methods and feature extraction. This paper provides an in-depth analysis of the key ideas and important studies related to image representation and content-based picture retrieval. In an effort to stimulate more research in this field, it also offers a preview of potential future study topics.

Towards 360 VR Sickness Mitigation: From Virtual Reality Eye-tracking to Visual Communication.

Most 360 virtual reality (VR) contents have been developed without considering that users could be affected by VR sickness. Accordingly, users' viewing safety has been steadily highlighted as a critical problem in the VR market. In this study, we investigate a novel VR sickness mitigation framework based on human visual characteristics for the rendered VR content. First, we build a large-scale 360 VR content database termed VRSP360 (VR Sickness and Presence 360) dedicated to the analysis of VR sickness and thoroughly conduct eye-tracking experiments to measure human perception. In the experiment, we observe that the users' gaze distribution is highly center-biased when they experience excessive VR sickness. From this observation, we design a foveated filtering framework that limits high-frequency textures in the peripheral view to mitigate VR sickness. Particularly, given the human visual system's (HVS) non-uniform resolution with respect to the fovea, we also adopt the foveation-based filtering method using the trade-off between sickness mitigation and presence conservation, which reduces any loss in perceptual quality despite the filtering. We further demonstrate that our framework can effectively compress visual information by applying foveated compression. In addition, we develop two metrics (visual texture index and perceptual information index) to measure the effective preservation of user-perceived information despite the filtration of peripheral vision textures by our proposed mitigation method. Through rigorous subjective evaluation on both original content and its VR-sickness-mitigated version, we demonstrate that the proposed framework successfully mitigates VR sickness with a reduction rate of ∼19% on the proposed dataset.

A modified atmospheric scattering model and degradation image clarification algorithm for haze environments

Addressing the limitation of existing image dehazing algorithms based on atmospheric scattering models, which often overlook the underlying physical causes of color distortion in hazy images, resulting in suboptimal algorithm robustness and image clarity when faced with varying levels of haze pollution, this study introduces a refined atmospheric scattering model. It also presents a haze image clarity algorithm based on this modified atmospheric scattering model by analyzing the fundamental factors contributing to image degradation and distortion caused by haze. Initially, the hazy image undergoes conversion to a new HSI color space. The hazy image is aligned more closely with human visual perception characteristics by implementing an adaptive saturation enhancement technique and adjusting contribution coefficients for the three RGB channels to the luminance component. Subsequently, the hazy image is partitioned into dense haze and non-dense haze regions based on haze distribution characteristics within the image. A global light intensity computation method is devised for the hazy image utilizing a quad-tree segmentation algorithm. Following this, calculations are performed to determine the atmospheric light value and transmittance for dense and non-dense haze regions separately. The global atmospheric light and transmittance values are then derived based on the probability density function of the dense haze region using Alpha Fusion. Finally, corrections are applied to the global atmospheric light values using an atmospheric light correction vector, while denoising the global transmittance is achieved through a fast non-local mean filtering algorithm. Experimental findings demonstrate that the proposed haze image clarity algorithm, grounded in the modified atmospheric scattering model, exhibits adaptability across various degrees and types of haze images. Comparative analysis against other prominent image dehazing algorithms reveals its efficacy in addressing severe color distortion in dense haze scenarios, achieving a more natural clarity effect, and showcasing superior performance across all numerical evaluation metrics.

A Low-Latency Noise-Aware Tone Mapping Operator for Hardware Implementation with a Locally Weighted Guided Filter

A tone mapping operator (TMO) is a module in the image signal processing pipeline that is used to convert high dynamic range images to low dynamic range images for display. Currently, state-of-the-art TMOs typically take complex algorithms and are implemented on graphics processing units, making it difficult to run with low latency on edge devices, and TMOs implemented in hardware circuits often lack additional noise suppression because of latency and hardware resource constraints. To address these issues, we proposed a low-latency noise-aware TMO for hardware implementation. Firstly, a locally weighted guided filter is proposed to decompose the luminance image into a base layer and a detail layer, with the weight function symmetric concerning the central pixel value of a window. Secondly, the mean and standard deviation of the basic layer and the detail layer are used to estimate the noise visibility according to the human visual characteristics. Finally, the gain for the detail layer is calculated to achieve adaptive noise suppression. In this process, luminance is first processed by the log2 function before being filtered and then symmetrically converted back to the linear domain by the exp2 function after compression. Meanwhile, the algorithms within the proposed TMO were optimized for hardware implementation to minimize latency and cache, achieving a low latency of 60.32 μs under video specification of 1080 P at 60 frames per second and objective metric smoothness in dark flat regions could be improved by more than 10% compared to similar methods.

Human gaze prediction for 3D light field display based on multi-attention fusion network

Existing methods for simulating human visual attention primarily focus on 2D displays and limited research has been conducted on predicting visual attention in three-dimensional (3D) light field content. 3D light field displays provide a heightened sense of stereoscopic realism to viewers. To ensure that the content of the 3D light field display appears more consistent with human visual characteristics, we proposed a novel method for predicting human eye fixation in 3D light field display images. Firstly, we collected real eye movement data and utilized it to create an eye movement dataset based on 3D light field display images. This solves the problem of missing datasets in the field of human gaze based on three-dimensional light field images. Then, we proposed a convolutional neural network model with multiple inputs and outputs, integrating attention modules. This model was trained and used to predict eye fixation within the constructed eye movement dataset. A correlation exists between predicted human gaze of multiple distinct views of same light field image. Finally, we predicted the human gaze area of light field multi-view images based on our model. Experimental results demonstrate that our model accurately predicts human gaze regions across different views of a 3D light field image. The human gaze predicted by the model on each view is basically consistent and relatively accurate. By leveraging proposed method, we can effectively anticipate where viewers will focus their attention on the 3D light field display, which is beneficial for targeted improvement of 3D light field display content.

A Spiking Artificial Vision Architecture Based on Fully Emulating the Human Vision.

Intelligent vision necessitates the deployment of detectors that are always-on and low-power, mirroring the continuous and uninterrupted responsiveness characteristic of human vision. Nonetheless, contemporary artificial vision systems attain this goal by the continuous processing of massive image frames and executing intricate algorithms, thereby expending substantial computational power and energy. In contrast, biological data processing, based on event-triggered spiking, has higher efficiency and lower energy consumption. Here, this work proposes an artificial vision architecture consisting of spiking photodetectors and artificial synapses, closely mirroring the intricacies of the human visual system. Distinct from previously reported techniques, the photodetector is self-powered and event-triggered, outputting light-modulated spiking signals directly, thereby fulfilling the imperative for always-on with low-power consumption. With the spiking signals processing through the integrated synapse units, recognition of graphics, gestures, and human action has been implemented, illustrating the potent image processing capabilities inherent within this architecture. The results prove the 90% accuracy rate in human action recognition within a mere five epochs utilizing a rudimentary artificial neural network. This novel architecture, grounded in spiking photodetectors, offers a viable alternative to the extant models of always-on low-power artificial vision system.

Optimization of regional cultural modern liquor packaging design based on computer vision algorithm

<p>In alcohol packaging, graphics, colors and text are social symbols. This article first uses the relevant principles of visual grammar to conduct a multimodal discourse analysis on the packaging pattern of Fen Liquor blue and white vases from multiple angles. Then, this paper uses human visual characteristics to improve product packaging styling effects. At the same time, a Gaussian filter is used to denoise the noisy wine packaging image, and then the image is used as an input image for grayscale transformation to obtain the guidance image. Research shows that in the multimodal discourse of Fen Liquor blue and white bottles, various forms such as images, text and colors can constitute and convey product information, mapping the history and culture of the product. This stimulates consumers’ visual senses and arouses consumers’ spiritual resonance to achieve the goal of promoting consumption.</p>

Research on military targets camouflage effect rating based on background anomaly detection and deep transfer features learning

Manual evaluation of military target camouflage often fails to fully capture human visual characteristics. This paper introduces a method for intelligent camouflage feature extraction and effect evaluation using deep neural networks. Firstly, military target and background datasets are constructed, with annotation methods optimized through perceptual experiments. A GANomaly model for background anomaly detection and a transfer learning feature model are developed. These models evaluate data similarity to background and target features to categorize camouflage into three levels. PCA dimensionality reduction and RFE screening identify final features. Two logistic regression models are then combined to create a grading model. In experiments, the GANomaly model was trained and optimized, achieving 86.8% accuracy in distinguishing level 1 from non-level 1 camouflage using 28-dimensional features. Among three transfer models, the MobileNetV2 model identified 33-dimensional features with 85.1% accuracy in distinguishing level 3 from non-level 3 camouflage. Combined, the models reached an 84.3% grading accuracy on the test set, with Micro and Macro average AUC values of 0.939 and 0.945, respectively. This performance surpasses that of BP network models and linear models based on manual features, as well as metrics like SSIM, UIQI, MSE, and PSNR. The results indicate that the models effectively align with human annotations and can reliably assess military target camouflage levels.

F-DARTS: Foveated Differentiable Architecture Search Based Multimodal Medical Image Fusion.

Multimodal medical image fusion (MMIF) is highly significant in such fields as disease diagnosis and treatment. The traditional MMIF methods are difficult to provide satisfactory fusion accuracy and robustness due to the influence of such possible human-crafted components as image transform and fusion strategies. Existing deep learning based fusion methods are generally difficult to ensure image fusion effect due to the adoption of a human-designed network structure and a relatively simple loss function and the ignorance of human visual characteristics during weight learning. To address these issues, we have presented the foveated differentiable architecture search (F-DARTS) based unsupervised MMIF method. In this method, the foveation operator is introduced into the weight learning process to fully explore human visual characteristics for the effective image fusion. Meanwhile, a distinctive unsupervised loss function is designed for network training by integrating mutual information, sum of the correlations of differences, structural similarity and edge preservation value. Based on the presented foveation operator and loss function, an end-to-end encoder-decoder network architecture will be searched using the F-DARTS to produce the fused image. Experimental results on three multimodal medical image datasets demonstrate that the F-DARTS performs better than several traditional and deep learning based fusion methods by providing visually superior fused results and better objective evaluation metrics.

A Remote Sensing Image Quality Interpretation Scale Characterization Method Based on the TTP Criterion

Accurate grading of remote sensing image interpretation is crucial for improving image classification and screening efficiency. Through extensive research, the General Image Quality Equation (GIQE) based on the National Imagery Interpretability Rating Scale (NIIRS) has been developed. However, poor reliability and low accuracy issues remain due to the failure to consider human visual characteristics. This paper introduces the Target Task Performance (TTP) criterion as a key parameter to reflect the cascading degradation factors of human visual perception characteristics and system imaging links, which improves the reliability of the model. A New Optimized Remote Sensing Image Quality Equation (NORSIQE), which effectively predicted the interpretability of image information, is constructed. Using 200 sets of test data, the quantitative relationship between key parameters (GSD, TTP, and SNR) of the NORSIQE and the subjective NIIRS level is obtained by a least squares regression fit, and the determination coefficient of the model is as high as 0.916. The model is evaluated for accuracy using 120 sets of validation data, showing an 87% improvement compared to the GIQE4. This method provides theoretical support for the development of new methods for remote sensing image quality evaluation and the design of payloads for remote sensing imaging systems.

Sharing Secret Colour Images with Embedded Visual Cryptography Using the Stamping Algorithm and OTP Procedure

Finding a way to ensure the safety of media is becoming increasingly common in the modern world as digital media usage increases. Visual cryptography (VC) offers an efficient method for sending images securely. Images that have been protected using visual encryption can be decoded using features of human vision. Emails are not a highly safe method of exchanging private data because someone else can quickly weaken the content. In the visual cryptography technique, we presented for colour pictures, the divided shares are enclosed in additional pictures using stamping. Using a random number generator, the shares are created. Visual cryptography schemes (VCS) are a method of encoding pictures that conceals the secret information which is present in images. A secret image is encrypted using a straightforward visual cryptography technique by splitting it into n shares, and the stamping operation is carried out by overlapping k shares. It can be beneficial for hiding a secret image. There is a chance that employing cryptography for information exchange could cause security problems because the process of decryption of simple visual cryptographic algorithms can be completed by the human eye. To address this issue, we are using the OTP procedure. In the past, static ID and passwords were employed, making them susceptible to replay and eavesdropping attacks. One Time Password technology, which generates a unique password each time, is utilized to solve this issue. The suggested approach strengthens the security of the created transparencies by applying an envelope to each share and employing a stamping technique to address security vulnerabilities that the previous methods had, such as pixel expansion and noise.

A dynamic holographic modelling method of digital twin scenes for bridge construction

ABSTRACT Holographic projection technology can provide a more intuitive and efficient visualization effect for a digital twin bridge construction scene. However, pre-rendering methods in the existing research work are usually used to implement holographic visualization, which is static display. The above-mentioned methods for static display have many shortcomings, such as poor adaptability, low rendering efficiency and lack of real-time. A dynamic holographic modelling approach is proposed for the augmented visualization of digital twin scenes for bridge construction. Firstly, a dynamic segmentation algorithm with adaptive screen size was designed to high-efficiently generate holographic scenes. Secondly, a motion blur control method was designed to improve the rendering efficiency of holographic scenes according to human visual characteristics. Finally, a prototype system was developed, and the corresponding experimental analysis was completed. The experimental results show that the method proposed in this article can support adaptive screen size image segmentation and real-time generation of holographic scenes for bridge construction. The amount of scene data can be reduced to more than 30%, which significantly improves rendering efficiency and reduces glare.

Perceptually-guided Dual-mode Virtual Reality System For Motion-adaptive Display.

An ideal Virtual reality (VR) device should simultaneously provide retina-level resolution, wide field-of-view (FOV), and high refresh rate display, thereby bringing users into a deeply immersive virtual world. However, directly providing such high-quality display poses great challenges for display panel fabrication, real-time rendering, and data transfer. To address this issue, we introduce a dual-mode virtual reality system based on the spatio-temporal perception characteristics of human vision. The proposed VR system has a novel optical architecture. It can switch display modes according to the user's perceptual requirements for different display scenes to adaptively adjust the display spatial and temporal resolution based on a given display budget, thus providing users with the optimal visual perception quality. In this work, a complete design pipeline for the dual-mode VR optical system is proposed, and a bench-top prototype is built with only off-the-shelf hardware and components to verify its capability. Compared to the conventional VR system, our proposed scheme is more efficient and flexible in utilizing the display budget, and this work is expected to facilitate the development of the VR device based on the human visual system.

Retinex‐inspired contrast stretch and detail boosting for lowlight image enhancement

AbstractLowlight images with low brightness and contrast, blurry details usually bring us an uncomfortable visual experience. To promote the quality of these deviation images, this paper presents a new and efficient approach, named MFMR, for enhancing lowlight images in the hue‐saturation‐value (HSV) colour space. Concretely, the multi‐angle filter is first applied to estimate the artifact‐free illumination and reflection component of the V‐channel. Afterward, the adaptive bi‐interval histogram with human visual characteristics and morphological operations is employed to process the former, adaptive gamma correction to process the latter for generating various feature maps. In the end, these feature maps are united via adaptive multi‐scale fusion strategy to reconstruct high‐quality images, which are characterized by high contrast and brightness, vivid colour, and clearer details. Extensive experiments show that this method is a well‐proven low‐light image enhancement approach, which outperforms the state‐of‐the‐art comparison methods. Furthermore, the proposed method also can yield satisfying images in the heavy foggy, yellow sand, underwater, and other severe conditions.

Blind omnidirectional image quality assessment with representative features and viewport oriented statistical features

With the development of information technologies, various types of streaming images are generated, such as videos, graphics, Virtual Reality (VR)/omnidirectional images (OIs), etc. Among them, the OIs usually have a broader view and a higher resolution, which provides human an immersive visual experience in a head-mounted display. However, the current image quality assessment works cannot achieve good performance without considering representative human visual features and visual viewing characteristics of OIs, which limited OIs’ further development. Motivated by the above problem, this work proposes a blind omnidirectional image quality assessment (BOIQA) model based on representative features and viewport oriented statistical features. Specifically, we apply the local binary pattern operator to encoder the cross-channel color information, and apply the weighted LBP to extract the structural features. Then the local natural scene statistics (NSS) features are extracted by using the viewport sampling to boost the performance. Finally, we apply support vector regression to predict the OIs’ quality score, and experimental results on CVIQD2018 and OIQA2018 Databases prove that the proposed model achieves better performance than state-of-the-art OIQA models.

Research on distortion quality evaluation of computer network shared image based on visual sensitivity

Shared image distortion will affect the user's experience, and then damage people's life and entertainment experience. In view of this, this research starts with the evaluation and classification of network shared image distortion quality, improves the shared image distortion quality evaluation algorithm combined with the sensitive characteristics of human vision, and verifies its performance superiority through comparative experiments. The results show that the performance of some improved reference quality evaluation algorithms reaches the highest values, which are 0.7923, 0.3224, 0.7931 and 0.8213, respectively. The improved non-reference quality evaluation algorithm achieves the highest values of positive indicators in the comparison of performance values, which are 0.487 and 0.287, respectively, while the lowest value of negative indicators is 0.902. It can be seen that the improved shared image quality evaluation algorithm conforms to the sensitive characteristics of human eyes, has high computational efficiency and has broad application prospects.

A deep learning model based on fusion images of chest radiography and X-ray sponge images supports human visual characteristics of retained surgical items detection.

Although a novel deep learning software was proposed using post-processed images obtained by the fusion between X-ray images of normal post-operative radiography and surgical sponge, the association of the retained surgical item detectability with human visual evaluation has not been sufficiently examined. In this study, we investigated the association of retained surgical item detectability between deep learning and human subjective evaluation. A deep learning model was constructed from 2987 training images and 1298 validation images, which were obtained from post-processing of the image fusion between X-ray images of normal post-operative radiography and surgical sponge. Then, another 800 images were used, i.e., 400 with and 400 without surgical sponge. The detection characteristics of retained sponges between the model and a general observer with 10-year clinical experience were analyzed using the receiver operator characteristics. The following values from the deep learning model and observer were, respectively, derived: Cutoff values of probability were 0.37 and 0.45; areas under the curves were 0.87 and 0.76; sensitivity values were 85% and 61%; and specificity values were 73% and 92%. For the detection of surgical sponges, we concluded that the deep learning model has higher sensitivity, while the human observer has higher specificity. These characteristics indicate that the deep learning system that is complementary to humans could support the clinical workflow in operation rooms for prevention of retained surgical items.

Haptic Texture Rendering of 2D Image Based on Adaptive Fractional Differential Method

The fractional differential algorithm has a good effect on extracting image textures, but it is usually necessary to select an appropriate fractional differential order for textures of different scales, so we propose a novel approach for haptic texture rendering of two-dimensional (2D) images by using an adaptive fractional differential method. According to the fractional differential operator defined by the Grünvald–Letnikov derivative (G–L) and combined with the characteristics of human vision, we propose an adaptive fractional differential method based on the composite sub-band gradient vector of the sub-image obtained by wavelet decomposition of the image texture. We apply these extraction results to the haptic display system to reconstruct the three-dimensional (3D) texture force filed to render the texture surface of two-dimensional (2D) images. Based on this approach, we carry out the quantitative analysis of the haptic texture rendering of 2D images by using the multi-scale structural similarity (MS-SSIM) and image information entropy. Experimental results show that this method can extract the texture features well and achieve the best texture force file for 2D images.

A No-Reference Quality Assessment Method for Screen Content Images Based on Human Visual Perception Characteristics

The widespread application of screen content images (SCIs) has met the needs of remote display and online working. It is a topic that is challenging and worthwhile discussing in research on quality assessment for SCIs. However, existing methods focus on extracting artificial features to predict image quality, which are subjective and incomplete, or lack good interpretability. To overcome these problems, we propose an effective quality assessment method for SCIs based on human visual perceptual characteristics. The proposed method simulates the multi-channel working mechanism of the human visual system (HVS) through pyramid decomposition and the information extraction process of brains with the help of dictionary learning and sparse coding. The input SCIs are first decomposed at multiple scales, and then dictionary learning and sparse coding are applied to the images at each scale. Furthermore, the sparse representation results are analyzed from multiple perspectives. First, a pooling scheme about generalized Gaussian distribution and log-normal distribution is designed to describe the sparse coefficients with and without zero values, respectively. Then the sparse coefficients are used to characterize the energy characteristics. Additionally, the probability of each atom is calculated to describe the statistical property of SCIs. Since the above process only deals with brightness, color-related features are also added to make the model more general and robust. Experimental results on three public SCI databases show that the proposed method can achieve better performance than existing methods.

Perceptual ultra-high definition video coding based on adaptive just noticeable distortion model

Ultra-high definition (UHD) video is gradually applied into many fields. The huge amount of data in UHD video limits the realization of these applications. Perception video coding can exploit the visual redundancy, and improve the compression performance. So far, the perpetual video coding based on just noticeable distortion (JND) model cannot fully exploit the perceptual redundancy of ultra-high definition (UHD) video/image in 10-bit depth. Thus, an adaptive JND model based on perceptual noise Bayesian network (PNBN) is firstly established for UHD video/image. Then, combined perceptual noise channel encoding/decoding with human visual characteristics, hidden side information of image is dynamically estimated by PNBN. Thirdly, the adaptive JND model is considered both luminance masking effect and texture complexity, which can adaptively exploit redundancy in different texture regions of image. Finally, an effective perceptual video coding (PVC) scheme is proposed for versatile video coding. A novel distortion compensation factor is appropriately designed for rate-distortion optimization. Experimental results show that the proposed PVC scheme achieves significant bitrate reduction with better subjective video quality than the state-of-the-art PVC schemes.