Properties Of Human Visual System Research Articles

PW-360IQA: Perceptually-Weighted Multichannel CNN for Blind 360-Degree Image Quality Assessment.

Image quality assessment of 360-degree images is still in its early stages, especially when it comes to solutions that rely on machine learning. There are many challenges to be addressed related to training strategies and model architecture. In this paper, we propose a perceptually weighted multichannel convolutional neural network (CNN) using a weight-sharing strategy for 360-degree IQA (PW-360IQA). Our approach involves extracting visually important viewports based on several visual scan-path predictions, which are then fed to a multichannel CNN using DenseNet-121 as the backbone. In addition, we account for users' exploration behavior and human visual system (HVS) properties by using information regarding visual trajectory and distortion probability maps. The inter-observer variability is integrated by leveraging different visual scan-paths to enrich the training data. PW-360IQA is designed to learn the local quality of each viewport and its contribution to the overall quality. We validate our model on two publicly available datasets, CVIQ and OIQA, and demonstrate that it performs robustly. Furthermore, the adopted strategy considerably decreases the complexity when compared to the state-of-the-art, enabling the model to attain comparable, if not better, results while requiring less computational complexity.

Two calibration models for compensation of the individual elements properties of self-emitting displays

In this paper, we examine the applicability limits of different methods of compensation of the individual properties of self-emitting displays with significant non-uniformity of chromaticity and maximum brightness. The aim of the compensation is to minimize the perceived image non-uniformity. Compensation of the displayed image non-uniformity is based on minimizing the perceived distance between the target (ideally displayed) and the simulated image displayed by the calibrated screen. The S-CIELAB model of the human visual system properties is used to estimate the perceived distance between two images. In this work, we compare the efficiency of the channel-wise and linear (with channel mixing) compensation models depending on the models of variation in the characteristics of display elements (subpixels). It was found that even for a display with uniform chromatic subpixels characteristics, the linear model with channel mixing is superior in terms of compensation accuracy.

Quality Assessment of DIBR-Synthesized Views Based on Sparsity of Difference of Closings and Difference of Gaussians.

Images synthesized using depth-image-based-rendering (DIBR) techniques may suffer from complex structural distortions. The goal of the primary visual cortex and other parts of brain is to reduce redundancies of input visual signal in order to discover the intrinsic image structure, and thus create sparse image representation. Human visual system (HVS) treats images on several scales and several levels of resolution when perceiving the visual scene. With an attempt to emulate the properties of HVS, we have designed the no-reference model for the quality assessment of DIBR-synthesized views. To extract a higher-order structure of high curvature which corresponds to distortion of shapes to which the HVS is highly sensitive, we define a morphological oriented Difference of Closings (DoC) operator and use it at multiple scales and resolutions. DoC operator nonlinearly removes redundancies and extracts fine grained details, texture of an image local structure and contrast to which HVS is highly sensitive. We introduce a new feature based on sparsity of DoC band. To extract perceptually important low-order structural information (edges), we use the non-oriented Difference of Gaussians (DoG) operator at different scales and resolutions. Measure of sparsity is calculated for DoG bands to get scalar features. To model the relationship between the extracted features and subjective scores, the general regression neural network (GRNN) is used. Quality predictions by the proposed DoC-DoG-GRNN model show higher compatibility with perceptual quality scores in comparison to the tested state-of-the-art metrics when evaluated on four benchmark datasets with synthesized views, IRCCyN/IVC image/video dataset, MCL-3D stereoscopic image dataset and IST image dataset.

Zonal MPEG-2

This paper presents a simple but effective approach to improve the performance of MPEG-2. This study is motivated by (i) the energy compaction property in DCT and (ii) the property of human visual system. Both (i) and (ii) indicate that low-frequency coefficients are more important than others in the DCT transformed blocks. Moreover, the quantized transformed coefficients in high-frequency region are zero or approximate zero in general. Those coefficients contribute nothing or little to PSNR and may waste some bit rate in the MPEG-2. Therefore, the zonal coding scheme is applied to the MPEG-2 where some high-frequency coefficients are dis- carded. The proposed approach is called zonal MPEG-2 which is totally compatible with the tra- ditional MPEG-2. The zonal MPEG-2 is verified by examples and shown to be able to signifi- cantly reduce the bit rate with acceptable perceptual quality degradation when compared with the traditional MPEG-2.

HEVC compatible perceptual multiple description video coding for reliable video transmission over packet networks

Multiple description coding (MDC) has been an effective scheme for reliable transmission of videos over error prone networks but requires higher data rates. Recently, perceptual video coding schemes are able to encode videos at a lower data rate, but with the same perceived decoding quality. Such schemes exploit human visual system (HVS) properties to encode videos. Although HVS characteristics have been used in MDC, but only for residual information and are not fully standard compatible. To this end, we propose a high efficiency video coding (HEVC) standard compatible perceptual multiple description video coding (PMDVC) framework. Our proposed framework temporally sub-samples the input video into even and odd frame sub-sequences, which are individually encoded (using HEVC) in a perceptual manner. We encode each description using a context based visual saliency model to obtain visual saliency mask, which is optimally thresholded into salient and non-salient regions. Coding tree unit (CTU) level perceptual relevance mask of each frame is generated by dividing binary saliency mask into five different groups. The quantization parameter at CTU level of each perceptual relevant group is adjusted in such a manner that data rate is minimized while maintaining the perceived quality. Our proposed HEVC based PMDVC scheme is evaluated under lossless and lossy channel conditions. Our results show better performance in data rate reduction, perceptual peak signal-to-noise-ratio, multi- scale structural similarity index, and difference mean opinion score, when compared with HEVC based temporally sub-sampled multiple description video coding scheme.

Deep Template-Based Watermarking

Traditional watermarking algorithms have been extensively studied. As an important type of watermarking schemes, template-based approaches maintain a very high embedding rate. In such scheme, the message is often represented by some dedicatedly designed templates, and then the message embedding process is carried out by additive operation with the templates and the host image. To resist potential distortions, these templates often need to contain some special statistical features so that they can be successfully recovered at the extracting side. But in existing methods, most of these features are handcrafted and too simple, thus making them not robust enough to resist serious distortions unless very strong and obvious templates are used. Inspired by the powerful feature learning capacity of deep neural network, we propose the first deep template-based watermarking algorithm in this paper. Specifically, at the embedding side, we first design two new templates for message embedding and locating, which is achieved by leveraging the special properties of human visual system, i.e., insensitivity to specific chrominance components, the proximity principle and the oblique effect. At the extracting side, we propose a novel two-stage deep neural network, which consists of an auxiliary enhancing sub-network and a classification sub-network. Thanks to the power of deep neural networks, our method achieves both digital editing resilience and camera shooting resilience based on typical application scenarios. Through extensive experiments, we demonstrate that the proposed method can achieve much better robustness than existing methods while guaranteeing the original visual quality.

Blind video quality assessment via spatiotemporal statistical analysis of adaptive cube size 3D‐DCT coefficients

There is an urgent need for a robust video quality assessment (VQA) model that can efficiently evaluate the quality of a video content varying in terms of the distortion and content type in the absence of the reference video. Considering this need, a novel no reference (NR) model relying on the spatiotemporal statistics of the distorted video in a three-dimensional (3D)-discrete cosine transform (DCT) domain is proposed in this study. While developing the model, as the first contribution, the video contents are adaptively segmented into the cubes of different sizes and spatiotemporal contents in line with the human visual system (HVS) properties. Then, the 3D-DCT is applied to these cubes. Following that, as the second contribution, different efficient features (i.e. spectral behaviour, energy variation, distances between spatiotemporal frequency bands, and DC variation) associated with the contents of these cubes are extracted. After that, these features are associated with the subjective experimental results obtained from the EPFL-PoliMi video database using the linear regression analysis for building the model. The evaluation results present that the proposed model, unlike many top-performing NR-VQA models (e.g. V-BLIINDS, VIIDEO, and SSEQ), achieves high and stable performance across the videos with different contents and distortions.

Development of a machine learning model for extracting image prominent colors

AbstractAlthough an image can consist of myriad different colors, due to the limitations of the human visual system and the mechanism of selective visual attention, only a limited number of them are prominent, that is, they stand out or are noticeable at first sight. In this article, a framework for building a model for extracting image prominent colors based on machine learning is presented. The model is learned on human‐extracted themes of prominent colors and uses numerous features, which were defined based on the properties of human visual system. For the purpose of this study, we constructed a database of images with their associated human‐extracted themes of prominent colors, which are open to the public and available to other researchers. The analysis of observers' data shows a high interobserver agreement on prominent color categories as well as high diversity of prominent colors. According to our results, the most influential factors on the perception of prominent colors are associated with color coverage (which should be adjusted with a saliency map), color properties—lightness and chroma, and diversity of colors. To the best of our knowledge, this is the first study on image prominent colors and first attempt to extract them with a model trained on the real data. The presented model has a high practical importance, since it can be used for extracting image colors in different scenarios, for example, for automatic color design, image categorization, as a descriptor in content‐based image retrieval, and image content analysis frameworks.

Digital Medical Diagnostic Displays

Digital medical imaging technology is developing rapidly in the last few decades and widely used as medical diagnostic tool. Medical image visual content analysis and interpretation is the most often method applied in detection and tracking of the pathogen behaviour of the imaged tissue. The human visual perception of the displayed image is limited by human visual system properties, display device characteristics and illumination environment influences during observation. In this review article we are analyzing how medical image specificities, human eye visual properties and display technology ultimate performances could be used to define medical image monitor technical requirements according to named application. We are focused on the analysis of the physical processes involved and technical aspects leading to optimization of the medical display requirements definition. This will help engineering and medical specialists to understand better medical display properties and provide more objective assessment of the display diagnostic suitability.

Asymmetric Foveated Just-Noticeable-Difference Model for Images With Visual Field Inhomogeneities

Just noticeable difference (JND) models reveal visibility limitation of human visual system (HVS) and exploit perceptual redundancy in the visual signal. Conventional just noticeable difference (JND) model supposes the visual acuity is homogeneous with or without considerations on the isoeccentric locations. Recent research on vision science indicates that the effect of eccentricity on visual sensitivity is not homogeneous across the visual field (VF). Relevant studies have found visual acuity at the isoeccentric locations is better along the horizontal meridian than along the vertical meridian, i.e., horizontal-vertical anisotropy (HVA). Along the vertical meridian, better performance has been reported for the inferior meridian compared to the superior meridian, i.e., vertical-meridian asymmetry (VMA). These properties of HVS should be taken into account to yield a better JND estimation, for better exploiting the perceptual redundancy. In this paper, we use two hemi-ellipses to represent the iso-acuity contour, which can better express the HVA and VMA effects. Moreover, we design psychophysical experiments under different test conditions, where the results show the hemi-ellipse model performs better when compared to the circular model. Finally, we build an asymmetric foveated JND model using two hemi-ellipses for approximation. When comparing with traditional JND models, experimental results demonstrate the effectiveness of the proposed model.

Visual IoT Security: Data Hiding in AMBTC Images Using Block-Wise Embedding Strategy.

This study investigates combining the property of human vision system and a 2-phase data hiding strategy to improve the visual quality of data-embedded compressed images. The visual Internet of Things (IoT) is indispensable in smart cities, where different sources of visual data are collected for more efficient management. With the transmission through the public network, security issue becomes critical. Moreover, for the sake of increasing transmission efficiency, image compression is widely used. In order to respond to both needs, we present a novel data hiding scheme for image compression with Absolute Moment Block Truncation Coding (AMBTC). Embedding secure data in digital images has broad security uses, e.g., image authentication, prevention of forgery attacks, and intellectual property protection. The proposed method embeds data into an AMBTC block by two phases. In the intra-block embedding phase, a hidden function is proposed, where the five AMBTC parameters are extracted and manipulated to embed the secret data. In the inter-block embedding phase, the relevance of high mean and low mean values between adjacent blocks are exploited to embed additional secret data in a reversible way. Between these two embedding phases, a halftoning scheme called direct binary search is integrated to efficiently improve the image quality without changing the fixed parameters. The modulo operator is used for data extraction. The advantages of this study contain two aspects. First, data hiding is an essential area of research for increasing the IoT security. Second, hiding in compressed images instead of original images can improve the network transmission efficiency. The experimental results demonstrate the effectiveness and superiority of the proposed method.

Weighting Quantization Matrices for HEVC/H.265-Coded RGB Videos

In the HEVC/H.265 video coding standard, weighting quantization matrices (WQMs) are supported to take advantage of the characteristics of the human visual system (HVS). However, the default WQMs utilized in HEVC are developed for YCbCr videos instead of RGB videos. In this paper, a set of new WQMs is proposed for video coding in RGB color space. First, we utilize the spatial contrast sensitivity function (CSF) to model the bandpass property of HVS. To derive the parameters of the spatial CSF, a series of subjective experiments is conducted to obtain the just-noticeable distortion (JND) thresholds of several selected DCT subbands. In addition, the sensitivities of different DCT subbands in one color channel, as well as among R, G, and B channels, are considered to design the WQMs of intra-coded $8 \times 8$ blocks. Moreover, to reduce the data size of WQMs, the WQMs for other block sizes are derived from intra $8 \times 8$ WQMs. The proposed WQMs are then applied into HEVC to directly code RGB videos. The experimental results demonstrate that when the PSNRs of G, B, and R channels are combined with a ratio of 4:1:1, the proposed WQMs can achieve an average BD-rate saving of 12.64% and 20.51%, respectively, in all-intra (AI) and low-delay (LD) profiles compared to HEVC without WQMs. The proposed scheme also enjoys a better video quality metric (VQM) performance.

Video analytical coding: When video coding meets video analysis

Leveraging on the properties of human visual system, most of the well-designed video coding standards utilize rate–distortion optimization techniques by maximizing a fidelity cost function (e.g., peak signal noise ratio, PSNR) under an available bit rate budget constrain. However, a huge amount of video data is consumed by computers rather than by human beings in several application scenarios. In view of this, this paper proposes a new coding framework called video analytical coding (VAC) for video analysis. We use the term “analytical distortion” to denote the difference of video analysis performance when video quality degrades and analytical distortion is estimated by compression distortion. Meanwhile, we develop a new rate–analytical-distortion optimization (RADO) method to trade off the bit rate and the analytical distortion. Specifically, we consider moving object detection as the analysis task and develop a rate analytical distortion (RAD) model and a quantization parameter adaptation strategy for video coding, where the analytical distortion is related to the object detection performance represented as F1-measure. Experimental results show that the performance of the video analysis task can be significantly improved (up to 40% reduction of analytical distortion).

Bivariate analysis of 3D structure for stereoscopic image quality assessment

Human visual system (HVS) identifies each unit of stereo-pair into binocular or monocular perception depending on distortions and depth perception. However, these HVS properties still have a large room for exploration in stereoscopic image quality assessment (SIQA) research field. In this paper, a bivariate natural scene statistics (NSS) model is proposed to capture image quality by extracting features from binocular and monocular perception regions, respectively. In the implementation details, the stereo-pair is first segmented into various regions based on spatial information of its disparity. Then the regions are classified into categories of binocular fusion, binocular rivalry and binocular suppression. Bivariate statistics of spatially adjacent Gabor response of image are extracted from each category of regions, based on which features are calculated for image quality representation. In particular, the extraction strategy depends on the type of image patch. Experimental results show that the proposed model is promising at handling the task of SIQA on LIVE 3D Image Quality Database.

Inverse Tone Mapping Operator Using Sequential Deep Neural Networks Based on the Human Visual System

Conventional digital displays show images with much less dynamic range than that of human visual perception. High-dynamic range (HDR) displays are being developed for viewing images with higher dynamic range than that of conventional low-dynamic range (LDR) images. However, to view existing LDR images on HDR displays, an inverse tone mapping operator (ITMO), which is a process to extend the dynamic range of LDR images, is required. In this paper, we propose an adaptive ITMO by effectively learning the differences between LDR and HDR images using a sequential learning process: dynamic range learning followed by color difference learning. Our proposed method enables visualization of colors similar to real-world colors better than conventional ITMOs by learning color differences based on the human visual system properties. For the objective comparison, seven different evaluation metrics optimized for HDR image evaluation were used. Our method resulted in 10%–25% improved HDR-visual difference predictor-2.2 values over those of other ITMOs. Other metrics also demonstrated the superior performance of our method. For the subjective comparison, eight human observers evaluated the estimated HDR images in terms of color appearance and overall preferences. Our method received an average 4.7 out of 5 score, whereas other ITMOs received below 3.5 scores in the evaluation of color appearance. The objective and subjective evaluations’ results showed that our proposed method outperformed the conventional ITMOs in estimating the dynamic range and color appearance of ground-truth HDR images.

Quaternion representation based visual saliency for stereoscopic image quality assessment

In this paper, a novel visual saliency detection method for stereoscopic images is proposed for the stereoscopic image quality assessment (SIQA) by considering the disparity map and difference image between the stereo image pairs. Firstly, a new quaternion representation (QR) of each stereo image (left/right view image) is constructed, which comprises the image content, the inter-view disparity, and the difference map. The quaternion Fourier transform (QFT) is performed on the constructed QR to generate the visual saliency maps for left and right views of stereoscopic image pairs, respectively. The generated visual saliency maps are further incorporated into the quality metrics for SIQA. Experimental results demonstrate that the visual saliency maps generated by the proposed method can help significantly boost the performance of SIQA, comparing with other visual saliency models proposed for stereoscopic images. It further confirms that the proposed visual saliency model can accurate depict the acuity property of human visual system (HVS) in judging the perceptual quality of stereoscopic images.

Perceptual Metrics Quality

A 3D mesh can be subjected to different types of operations, such as compression, watermarking etc. Such processes lead to geometric distortions compared to the original version. In this context, quantifying the resultant modifications to the original mesh and evaluating the perceptual quality of degraded meshes become a critical issue. The perceptual 3D meshes quality is central in several applications to preserve the visual appearance of these treatments. The used metrics results have to be well correlated to the visual perception of humans. Although there are objective metrics, they do not allow the prediction of the perceptual quality, and do not include the human visual system properties. In the current work, a comparative study between the perceptual quality assessment metrics for 3D meshes was conducted. The experimental study on subjective database published by LIRIS / EPFL was used to test and to validate the results of six metrics. The results established that the Mesh Structural Distortion Measure metric achieved superior results compared to the other metrics.

Robust watermark technique using masking and Hermite transform.

The following paper evaluates a watermark algorithm designed for digital images by using a perceptive mask and a normalization process, thus preventing human eye detection, as well as ensuring its robustness against common processing and geometric attacks. The Hermite transform is employed because it allows a perfect reconstruction of the image, while incorporating human visual system properties; moreover, it is based on the Gaussian functions derivates. The applied watermark represents information of the digital image proprietor. The extraction process is blind, because it does not require the original image. The following techniques were utilized in the evaluation of the algorithm: peak signal-to-noise ratio, the structural similarity index average, the normalized crossed correlation, and bit error rate. Several watermark extraction tests were performed, with against geometric and common processing attacks. It allowed us to identify how many bits in the watermark can be modified for its adequate extraction.

The detection algorithm of irregular dynamic objects

Using remote sensing data for marine monitoring, marine rescue, marine pollution monitoring which has a broad monitoring area and fast response time, so in the last 10 years, satellite borne or airborne optical and SAR sensor data have been increasingly used in the field of ocean monitoring. However, based on the dynamic video to carry out the study of the detection of irregular debris blocks from the wrecked ship, aircraft is less. To a great extent, it affects the efficiency of maritime rescue. This paper, based on the dynamic objects detection, Color Space Quantization and property of Human Vision System, takes the irregular wreckage patches as research. And a detection algorithm for dynamic objects is developed with the foundation of Lab channels. To verify that, the calculation of the Euclidean distance between the characteristic parameters of irregular wreckage patches and regular floating ones, and the average Euclidean distance is 0.8245 between the irregular1 and the regular while that between irregular2 and the regular is 4.3645. At the same time, the Hausdorff distance was calculated and verified, and the average distance between the irregular block 1 and the regular block was 2.5975. The average distance between the block 2 and the rule block was 13.8962. Experimental results show that the method is consistent with the results of the second methods, which proves that the first method is scientific and operational. Therefore, it can be found that some parameters including divergence, elongation and eccentricity fluctuate dramatically from those in irregular ones to the regular one, which could be extracted to recognize or distinguish the patches. That is why this paper is of importance in marine rescue and detection of objects on the sea surface.

Radiometric Compensation of Images Projected on Non-White Surfaces by Exploiting Chromatic Adaptation and Perceptual Anchoring.

Flat surfaces in our living environment to be used as replacements of a projection screen are not necessarily white. We propose a perceptual radiometric compensation method to counteract the effect of color projection surfaces on image appearance. It reduces color clipping while preserving the hue and brightness of images based on the anchoring property of human visual system. In addition, it considers the effect of chromatic adaptation on perceptual image quality and fixes the color distortion caused by non-white projection surfaces by properly shifting the color of the image pixels toward the complementary color of the projection surface. User ratings show that our method outperforms the existing methods in 974 out of 1020 subjective tests.