Characteristics Of Human Visual System Research Articles

A Spatial-Temporal Video Quality Assessment Method via Comprehensive HVS Simulation.

The quality of videos is the primary concern of video service providers. Built upon deep neural networks, video quality assessment (VQA) has rapidly progressed. Although existing works have introduced the knowledge of the human visual system (HVS) into VQA, there are still some limitations that hinder the full exploitation of HVS, including incomplete modeling with few HVS characteristics and insufficient connection among these characteristics. In this article, we present a novel spatial-temporal VQA method termed HVS-5M, wherein we design five modules to simulate five characteristics of HVS and create a bioinspired connection among these modules in a cooperative manner. Specifically, on the side of the spatial domain, the visual saliency module first extracts a saliency map. Then, the content-dependency and the edge masking modules extract the content and edge features, respectively, which are both weighted by the saliency map to highlight those regions that human beings may be interested in. On the other side of the temporal domain, the motion perception module extracts the dynamic temporal features. Besides, the temporal hysteresis module simulates the memory mechanism of human beings and comprehensively evaluates the video quality according to the fusion features from the spatial and temporal domains. Extensive experiments show that our HVS-5M outperforms the state-of-the-art VQA methods. Ablation studies are further conducted to verify the effectiveness of each module toward the proposed method. The source code is available at https://github.com/GZHU-DVL/HVS-5M.

A Method on Classification and Recognition of Noisy Plant Images Based on Visual Domain Perception

At present, some achievements have been made in the research of plant leaf classification such as the introduction of artificial intelligence algorithm. But there are still some problems. First, the existing achievements do not consider the subjective perception mechanism and role of human visual system in leaf classification data labels. Second, the implementation of the deep learning algorithm completely depends on the computing power level of the high-cost machine hardware and the large-scale image database. Finally, these research results rarely consider the noise pollution of leaf image samples. In order to solve the above problems, the paper fully considered the subjective perception principle and characteristics of human vision system (HVS), and proposed a lightweight classification method of noisy plant leaves (LCM-NPLs) based on visual domain perception. First, the most suitable HVS front-end perception characteristics were applied to the physical visual processing of leaves. Then the plant leaves were denoised through the information processing mechanism of HVS back-end. The visual effect of regular and orderly plant leaves is obtained. Finally, the classification is realized by principal component analysis (PCA) and third-order nearest neighbor algorithm. The results of ablation contrast experiments show that the classification accuracy of the method in this paper is 82.50% for plant leaves in the presence of serious noise interference with PSNR of 10.2421, more than 90% for plant leaves with general noise pollution transmission with PSNR of more than 15.3759, and 98.33% for plant leaves of light pollution with PSNR of 20.5659. The proposed method has achieved very good results. The proposed method can not only accurately classify plant leaves in different growth periods, but also maintain a high classification accuracy rate in the presence of serious noise interference.

Hybrid and Invisible Digital Image Watermarking Technique Using IWT-DCT and Hopfield Neural Network

According to the characteristic of HVS (Human Visual System) and the association memory ability of the neural network, an adaptive image watermarking algorithm based on a neural network is proposed invisible image watermarking is a secret embedding scheme for hiding secret images into a cover image file and the purpose of invisible watermarking is copyrights protection. Wavelet transformation-based image watermarking techniques provide better robustness for statistical attacks in comparison to Discrete Cosine Transform domain-based image watermarking. The joined method of IWT (Integer Wavelet Transform) and DCT (Discrete Cosine Transform) gives benefits to the two procedures. The IWT has an impediment of portion misfortune in embedding which increments the mean square estimate as SIM and results in diminishing PSNR. The capacity of drawing in is improved by pretreatment and re-treatment of image scrambling and Hopfield neural network. The proposed algorithm presents a hybrid integer wavelet transform and discrete cosine transform-based watermarking technique to obtain increased imperceptibility and robustness compared to the IWT-DCT-based watermarking technique. The proposed watermarking technique reduces the fractional loss compared to DWT-based watermarking.

Image quality assessment based on the perceived structural similarity index of an image.

Image quality assessment (IQA) has a very important role and wide applications in image acquisition, storage, transmission and processing. In designing IQA models, human visual system (HVS) characteristics introduced play an important role in improving their performances. In this paper, combining image distortion characteristics with HVS characteristics, based on the structure similarity index (SSIM) model, a novel IQA model based on the perceived structure similarity index (PSIM) of image is proposed. In the method, first, a perception model for HVS perceiving real images is proposed, combining the contrast sensitivity, frequency sensitivity, luminance nonlinearity and masking characteristics of HVS; then, in order to simulate HVS perceiving real image, the real images are processed with the proposed perception model, to eliminate their visual redundancy, thus, the perceived images are obtained; finally, based on the idea and modeling method of SSIM, combining with the features of perceived image, a novel IQA model, namely PSIM, is proposed. Further, in order to illustrate the performance of PSIM, 5335 distorted images with 41 distortion types in four image databases (TID2013, CSIQ, LIVE and CID) are used to simulate from three aspects: overall IQA of each database, IQA for each distortion type of images, and IQA for special distortion types of images. Further, according to the comprehensive benefit of precision, generalization performance and complexity, their IQA results are compared with those of 12 existing IQA models. The experimental results show that the accuracy (PLCC) of PSIM is 9.91% higher than that of SSIM in four databases, on average; and its performance is better than that of 12 existing IQA models. Synthesizing experimental results and theoretical analysis, it is showed that the proposed PSIM model is an effective and excellent IQA model.

Joint Graph Attention and Asymmetric Convolutional Neural Network for Deep Image Compression

Recent deep image compression methods have achieved prominent progress by using nonlinear modeling and powerful representation capabilities of neural networks. However, most existing learning-based image compression approaches employ customized convolutional neural network (CNN) to utilize visual features by treating all pixels equally, neglecting the effect of local key features. Meanwhile, the convolutional filters in CNN usually express the local spatial relationship within the receptive field and seldom consider the long-range dependencies from distant locations. This results in the long-range dependencies of latent representations not being fully compressed. To address these issues, an end-to-end image compression method is proposed by integrating graph attention and asymmetric convolutional neural network (ACNN). Specifically, ACNN is used to strengthen the effect of local key features and reduce the cost of model training. Graph attention is introduced into image compression to address the bottleneck problem of CNN in modeling long-range dependencies. Meanwhile, regarding the limitation that existing attention mechanisms for image compression hardly share information, we propose a self-attention approach which allows information flow to achieve reasonable bit allocation. The proposed self-attention approach is in compliance with the perceptual characteristics of human visual system, as information can interact with each other via attention modules. Moreover, the proposed self-attention approach takes into account channel-level relationship and positional information to promote the compression effect of rich-texture regions. Experimental results demonstrate that the proposed method achieves state-of-the-art rate-distortion performances after being optimized by MS-SSIM compared to recent deep compression models on the benchmark datasets of Kodak and Tecnick. The project page with the source code can be found in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://mic.tongji.edu.cn</uri> .

A perceptual framework for infrared–visible image fusion based on multiscale structure decomposition and biological vision

Infrared–visible image fusion is of great value in many applications due to their highly complementary information. However, it is hard to obtain high-quality fused image for current fusion algorithms. In this paper, we reveal an underlying deficiency in current fusion framework limiting the quality of fusion, i.e., the visual features used in the fusion can be easily affected by external physical conditions (e.g., the characteristics of different sensors and environmental illumination), indicating that those features from different sources have not been ensured to be fused on a consistent basis during the fusion. Inspired by biological vision, we derive a framework that transforms the image intensities into the visual response space of human visual system (HVS), within which all features are fused in the same perceptual state, eliminating the external physical factors that may influence the fusion process. The proposed framework incorporates some key characteristics of HVS that facilitate the simulation of human visual response in complex scenes, and is built on a new variant of multiscale decomposition, which can accurately localize image structures of different scales in visual-response simulation and feature fusion. A bidirectional saliency aggregation is proposed to fuse the perceived contrast features within the visual response space of HVS, along with an adaptive suppression of noise and intensity-saturation in this space prior to the fusion. The final fused image is obtained by transforming the fusion results in human visual response space back to the physical domain. Experiments demonstrate the significant improvement of fusion quality brought about by the proposed method.

68‐3: Importance of Individual Adaptation in Visually Fidelitous Dynamic‐Range Compression from HDR to SDR Images

Reproducing fidelitous videos/images that include the individual impressions induced when the real scene is viewed, is important. Our previous psychological studies and the latest physiological research result conducted by Kubota et al. newly suggest that the human visual system characteristics change dynamically on circadian rhythm. This study also suggests that they significantly depend on individuality, i.e., personal behavior.

Image Quality Assessment Based on Three Features Fusion in Three Fusion Steps

The objective image quality assessment (IQA) method was developed to replace subjective observer image quality evaluations in various applications. A reliable full reference color IQA method that allows reference and distorted images to be compared in a symmetric way is designed via three fusion steps described in this article. The three fusion steps include luminance channels fusion, similarity maps fusion, and features fusion. A fusion weight coefficient is designed to fuse the luminance channels of input images as an enhancement operator for features. The extracted SR (spectral residual), gradient, and chrominance features, by means of symmetric calculations for the reference and distorted images, are conducted via similarity fusion processing. Then, based on the human visual system (HVS) characteristics of achromatic and chromatic information receiving, a features fusion map represents the weighted sum of three similarity fusion maps. Finally, a deviation pooling strategy is utilized to export the quality score after features fusion. The novel method is called the features fusion similarity index (FFS). Various experiments are carried out based on statistical evaluation criteria to optimize the parameters of FFS, after which the proposed method of FFS is compared with other state-of-the-art IQA methods using large-scale benchmark single distortion databases. The results show that FFS performs with higher consistency with respect to subjective scores in terms of prediction accuracy, e.g., the PLCC can achieve at least 0.9116 accuracy and at most 0.9774 accuracy for four databases. In addition, the average running time of FFS is 0.0657 s—a value representing a higher computational efficiency.

Machine learning based video segmentation of moving scene by motion index using IO detector and shot segmentation

Video segmentation has a major role in several applications like autonomous vehicles, medical image analysis, video surveillance, and augmented reality. Portrait segmentation, a subset of semantic image segmentation, is the key step of preprocessing step in numerous applications and a few among them are entertainment applications, security systems, video conferences, and so on. For the given video, every object exhibiting independent motion in at least a single frame is segmented. This is formulated as a learning problem and designed as an Ensemble Soft Voting Algorithm based on Supervised Machine Learning (ESVA-SML). The motion stream requires an ensemble learning method trained on synthetic videos for segmenting independent objects at motion in the optical flow field. The spatial–temporal features are extracted from each sub-bands which efficiently resembles the human visual system characteristics concerning the feature extract using Gray Level Size Zone Matrix (GLSZM). This method linear combines the previous vectors within a stipulated time interval thereby describing each spatial–temporal feature vector. Based on the prediction errors, shot boundaries of every shot are efficiently identified from which at least a single keyframe is extracted for further analysis. The proposed method is extensively evaluated using SegTrack-v2, DAVIS, and Fusion databases by comparing eight standard methods in various parameters. By analyzing all these three databases, SegTrack-v2 is best in terms of accuracy, precision, and recall. While both DAVIS and Fusion databases produce less MAE. Finally, Fusion produces less RMSE and RAE with minimal response time.

A no-Reference Stereoscopic Image Quality Assessment Network Based on Binocular Interaction and Fusion Mechanisms.

In contemporary society full of stereoscopic images, how to assess visual quality of 3D images has attracted an increasing attention in field of Stereoscopic Image Quality Assessment (SIQA). Compared with 2D-IQA, SIQA is more challenging because some complicated features of Human Visual System (HVS), such as binocular interaction and binocular fusion, must be considered. In this paper, considering both binocular interaction and fusion mechanisms of the HVS, a hierarchical no-reference stereoscopic image quality assessment network (StereoIF-Net) is proposed to simulate the whole quality perception of 3D visual signals in human cortex, including two key modules: BIM and BFM. In particular, Binocular Interaction Modules (BIMs) are constructed to simulate binocular interaction in V2-V5 visual cortex regions, in which a novel cross convolution is designed to explore the interaction details in each region. In the BIMs, different output channel numbers are designed to imitate various receptive fields in V2-V5. Furthermore, a Binocular Fusion Module (BFM) with automatic learned weights is proposed to model binocular fusion of the HVS in higher cortex layers. The verification experiments are conducted on the LIVE 3D, IVC and Waterloo-IVC SIQA databases and three indices including PLCC, SROCC and RMSE are employed to evaluate the assessment consistency between StereoIF-Net and the HVS. The proposed StereoIF-Net achieves almost the best results compared with advanced SIQA methods. Specifically, the metric values on LIVE 3D, IVC and WIVC-I are the best, and are the second-best on the WIVC-II.

Image quality assessment based on the image contents visual perception

Image quality assessment (IQA) is widely used in image transmission, processing, and storage, which has important application value. To obtain an IQA model with the excellent performance, combining with the image content features and human visual system characteristics, based on the contrast definition in visual psychology, an IQA method and its mathematical model are proposed. In this method, first based on the contrast definition, combining with the visual characteristics, the contrast between the distorted image and the reference image is used to describe the difference between them; further, based on the difference, a definition method for image quality is proposed. Second, based on the gray and gradient co-occurrence matrix, a concept, which is called as the image gray-gradient expectation (GGE), is proposed, and its calculation method is also illustrated. And based on the GGE values and local contrast of image, a method for describing the image content and its visual perception is proposed. Finally, based on the image content features and the proposed definition method of image quality, an IQA method and its mathematical model are proposed. Further, they were tested using the 142 reference images and 7220 distorted images in the LIVE, CSIQ, TID2008, TID2013, CIDIQ, and IVC databases. And the results were compared with those of seven existing typical IQA models in terms of accuracy, complexity, and generalization performance. These experimental results show that, in the six databases, the IQA accuracy Pearson linear correlation coefficient of the proposed model can all be more than 0.7892 and reach 0.9638 highest, whose comprehensive efficiency is better than ones of the seven existing IQA models. These analysis and comparison show that the proposed model is an excellent IQA model on performance.

Perceptual VVC quantization refinement with ensemble learning

Compressing videos while maintaining an acceptable level of Quality of Experience (QoE) is indispensable. To this aim, a feasible method is to further increase the Quantization Parameter (QP) of video stream to eliminate visual redundancy, simultaneously utilizing perceptual characteristics of Human Visual System (HVS) to impose a threshold constraint on the maximum QP. In this paper, we employ Just Noticeable Distortion (JND) to characterize the aforementioned threshold constraint, thereby avoiding perceptual loss during QP refinement process. We propose an effective JND-based algorithm for QP optimization, in which a video saliency detection is introduced to extract regions of interest, a refinement model based on a lightweight network is designed to predict QP value and an ensemble learning method to improve generalization performance. Theoretical analysis and experimental results demonstrate that the proposed algorithm has been successfully applied to Versatile Video Coding (VVC) to achieve significant bitrate reduction without sacrificing perceived quality.

An Effective Multi-Task Two-Stage Network with the Cross-Scale Training Strategy for Multi-Scale Image Super Resolution

Convolutional neural networks and the per-pixel loss function have shown their potential to be the best combination for super-resolving severely degraded images. However, there are still challenges, such as the massive number of parameters requiring prohibitive memory and vast computing and storage resources as well as time-consuming training and testing. What is more, the per-pixel loss measured by L2 and the Peak Signal-to-Noise Ratio do not correlate well with human perception of image quality, since L2 simply does not capture the intricate characteristics of human visual systems. To address these issues, we propose an effective two-stage hourglass network with multi-task co-optimization, which enables the entire network to focus on training and testing time and inherent image patterns such as local luminance, contrast, structure and data distribution. Moreover, to avoid overwhelming memory overheads, our model is capable of performing real-time single image multi-scale super-resolution, so it is memory-friendly, meaning that memory space is utilized efficiently. In addition, in order to best use the underlying structure and perception of image quality and the intermediate estimates during the inference process, we introduce a cross-scale training strategy with 2×, 3× and 4× image super-resolution. This effective multi-task two-stage network with the cross-scale strategy for multi-scale image super-resolution is named EMTCM. Quantitative and qualitative experiment results show that the proposed EMTCM network outperforms state-of-the-art methods in recovering high-quality images.

Foveated light culling

In this paper, we propose a novel Foveated Light Culling (FLC) method to efficiently approximate global illumination for foveated rendering in virtual reality applications. The key idea is to cull the virtual point lights (VPLs) in accordance with their contribution to the visual perception, which makes the loss of details caused is imperceptible to the human eyes. Based on this idea and the features of human visual system (HVS), we calculate the contribution of each VPL according to samples generated from a perceptual importance map. By estimating the local visibility between VPLs and samples, our method can greatly improve the performance during the process of evaluating contributions. Our method is able to produce high-quality single-bounce diffuse-to-diffuse indirect illumination in the foveal region at real-time frame rate while supporting dynamic scenes and light sources. Furthermore, the experiments show that our approach makes a good balance between foveal quality and render efficiency.

Visual Perception Based Algorithm for Fast Depth Intra Coding of 3D-HEVC

3D-HEVC (The 3D Extension of High Efficiency Video Coding) is the newest 3D video coding standard, which enriches multimedia applications with the video format of multi-view plus depth. For the depth map coding in 3D-HEVC, the advanced coding tools enhance the coding efficiency of the depth map and the quality of the synthesized view. However, the time consumption and complexity of 3D-HEVC also increase significantly. This paper utilizes the characteristics of human visual system to propose a fast algorithm based on visual perception for the acceleration of the depth intra coding of 3D-HEVC. The depth map is segmented into different regions by Otsu's auto-thresholding. The dominate edge direction is categorized for each prediction unit. We detect the perceptual edge based on just noticeable depth difference model to extract the area that may affect the visual perception. According to depth map segmentation and edge distribution, we reduce the corresponding intra angular modes and determine whether to perform depth modelling mode. We also incorporate the boundary continuity and rate-distortion cost thresholding to propose the fast coding unit decision. The experimental results show that the proposed algorithm eliminates 53.09% of the depth coding time with only 0.15% BD-BR on average. The coding performance of the proposed algorithm outperforms the previous works significantly.

Advanced image steganography based on exploiting modification direction and neutrosophic set

A new variant of image steganography based on exploiting modification directions (EMD) named advanced EMD (AEMD) is introduced. In this method the secret digits in mn -ary notional systems are embedded into a group of n pixels of the cover image. To increase data hiding capacity, edge masking characteristics of human visual system is exploited to embed more bits at image edge pixels than non-edge pixels. To do this, a neutrosophic set edge detector, named as MNSED is also introduced to classify cover image into 2 × 2 non-overlapping edge and non-edge blocks without any overhead information. Hence the secret digits in two different bases are embedded into the edge and non-edge blocks. Experimental results show that the proposed method provides higher embedding capacity and better quality compared to the existing schemes, while its resistance to steganographic attack is still higher.

No-reference light field image quality assessment based on depth, structural and angular information

Light field imaging can capture more abundant scene information, including angular and spatial information, compared with traditional imaging technologies. However, light field image (LFI) processing will inevitably introduce distortion which is different from traditional imaging process. Therefore, LFI quality assessment has become one of important research issues on LFI processing. In this paper, we propose a depth, structural and angular information based no-reference LFI quality assessment method. Firstly, the horizontal and vertical mean difference images (MDIs) are defined to integrate the difference images of LFI's sub-aperture images (SAIs) and reflect the horizontal and vertical depth and structural information of LFI. Considering the multi-channel characteristics of human visual system, an effective feature extraction scheme with Curvelet decomposition is designed for MDIs and SAIs of the distorted LFI. As one of the representations of LFI, epipolar plane images (EPIs) contain LFI's angular and depth information, which reflect the angular consistency of LFI more intuitively. Therefore, an algorithm, namely local maximum similarity index statistics, is designed to extract the directional features from horizontal and vertical EPIs. Finally, the quality of distorted LFI is predicted by pooling these extracted features. Experimental results on three LFI datasets show that the proposed method can achieve better performance, compared with the representative traditional image quality assessment methods as well as the state-of-the-art LFI quality assessment methods.

Robust Blind Watermarking Algorithm Based on Contourlet Transform with Singular Value Decomposition

In this Letter, we propose a blind and robust multiple watermarking scheme using Contourlet transform and singular value decomposition (SVD). The host image is first decomposed by Contourlet transform. Singular values of Contourlet coefficient blocks are adopted to embed watermark information, and a fast calculation method is proposed to avoid the heavy computation of SVD. The watermark is embedded in both low and high frequency Contourlet coefficients to increase the robustness against various attacks. Moreover, the proposed scheme intrinsically exploits the characteristics of human visual system and thus can ensure the invisibility of the watermark. Simulation results show that the proposed scheme outperforms other related methods in terms of both robustness and execution time.

A full‐reference stereoscopic image quality assessment index based on stable aggregation of monocular and binocular visual features

In stereoscopic image quality assessment, human visual system has been universally taken into account to detect perceptual characteristics. A novel full-reference stereoscopic image assessment metric by considering both monocular and binocular visual features of human visual system is proposed. In particular, a new region segmentation algorithm is firstly proposed to divide 3D images into occluded and non-occluded regions. The just noticeable difference model is employed on the occluded regions to formulate the monocular vision, while the binocular just noticeable difference model is applied to the non-occluded regions to reveal the binocular vision of the human visual system. In the proposed region segmentation, disparity information and Euclidean distance between stereo pairs are both adopted to solve the unstable segmentation problem of traditional methods. A new pooling strategy based on global edge features is then presented to aggregate the just noticeable difference and binocular just noticeable difference evaluation maps. In addition, some local image features as supplementary of just noticeable difference to describe visual characteristics of the human visual system are also extracted. Finally, an overall quality score is calculated based on the above-mentioned features to measure the visual quality of distorted stereo pairs. Experimental results show that the proposed metric achieves high consistency with the human visual system, and outperforms state-of-the-art algorithms on stereoscopic image quality assessment.

An Overview of the Attention Mechanisms in Computer Vision

Deep convolutional neural network (CNN) plays an important role in the field of computer vision and image processing. In order to further improve the performance of CNN, scholars have conducted a series of new explorations, such as the improvement of activation functions, the construction of new loss functions, the regularization of parameters and the development of new network structures. However, every breakthrough of CNN comes from the innovation of network structure, whose design can be inspired by exploring the cognitive process of human brain. As one of the important features of human visual system, visual attention mechanism is essential in image generation, scene classification, target detection and tracking when applied in the field of computer vision. Focusing on the models of attention mechanisms commonly used in computer vision, their categorizations, principles, and outlook are summarized in this overview.