Psychovisual Experiments Research Articles

A Dataset and Model for the Visual Quality Assessment of Inversely Tone-Mapped HDR Videos.

To enhance the viewer experience of standard dynamic range (SDR) video content on high dynamic range (HDR) displays, inverse tone mapping (ITM) is employed. Objective visual quality assessment (VQA) models are needed for effective evaluation of ITM algorithms. However, there is a lack of specialized VQA models for assessing the visual quality of inversely tone-mapped HDR videos (ITM-HDR-Videos). This paper addresses both an algorithmic and a dataset gap by introducing a novel SDR referenced HDR (SD-R-HD) VQA model tailored for ITM-HDR-Videos, along with the first public dataset specifically constructed for this purpose. The innovations of the SD-R-HD VQA model include 1) utilizing available SDR video as a reference signal, 2) extracting features that characterize standard ITM operations such as global mapping and local compensation, and 3) directly modeling interframe inconsistencies introduced by ITM operations. The newly created ITM-HDR-VQA dataset comprises 200 ITM-HDR-Videos annotated with mean opinion scores, gathered over 320 man-hours of psychovisual experiments. Experimental results demonstrate that the SD-R-HD VQA model significantly outperforms existing state-of-the-art VQA models.

Visual vs internal attention mechanisms in deep neural networks for image classification and object detection

The so-called “attention mechanisms” in Deep Neural Networks (DNNs) denote an automatic adaptation of DNNs to capture representative features given a specific classification task and related data. Such attention mechanisms perform both globally by reinforcing feature channels and locally by stressing features in each feature map. Channel and feature importance are learnt in the global end-to-end DNNs training process. In this paper, we present a study and propose a method with a different approach, adding supplementary visual data next to training images. We use human visual attention maps obtained independently with psycho-visual experiments, both in task-driven or in free viewing conditions, or powerful models for prediction of visual attention maps. We add visual attention maps as new data alongside images, thus introducing human visual attention into the DNNs training and compare it with both global and local automatic attention mechanisms. Experimental results show that known attention mechanisms in DNNs work pretty much as human visual attention, but still the proposed approach allows a faster convergence and better performance in image classification tasks.

Visual Characteristics of Afterimage under Dark Surround Conditions

Three psycho-visual experiments were carried out to investigate visual afterimage characteristics of high luminance LEDs under dark surround conditions. The results show that the luminance of illumination, exposure time and background luminance are the primary factors influencing the afterimage’s duration, the color difference between stimulus and background, and visibility. Besides, visual afterimage characteristics are strongly correlated with the luminance contrast between the stimulus and the background, but not for the color difference with a white background. A third-order polynomial regression model was proposed to accurately estimate the afterimage duration, the color difference between stimulus and background, and visibility. The model performance showed high R-squared, low root-mean-square error (RMSE) and mean absolute error (MAE) values between the predicted and visual characteristics.

Color and Quality Enhancement of Videoconferencing Whiteboards

Abstract Whiteboards are commonly used as a medium of instant illustration of ideas during several activities including presentations, lectures, meetings, and related others through videoconferencing systems. However, the acquisition of whiteboard contents is inhibited by issues inherent to the camera technologies, the whiteboard glossy surfaces along with other environmental issues such as room lighting or camera positioning. The contents of whiteboards are mostly invisible due to the low luminance contrast and other related color degradation problems. This article presents an account of a work aimed at extracting the whiteboard image and consequently enhancing its perceptual quality and legibility. Two different methods based on color balancing and color warping are introduced to improve the global and local luminance contrast as well as color saturation of the contents. The methods are implemented based on different general models of the videoconferencing environment for avoiding color shifts and unnaturalness of results. Our evaluations, through psycho-visual experiments, reveal the significance of the proposed method’s improvements over the state of the art methods in terms of visual quality and visibility.

Perceptual Lightness Modeling for High-Dynamic-Range Imaging

The human visual system (HVS) non-linearly processes light from the real world, allowing us to perceive detail over a wide range of illumination. Although models that describe this non-linearity are constructed based on psycho-visual experiments, they generally apply to a limited range of illumination and therefore may not fully explain the behavior of the HVS under more extreme illumination conditions. We propose a modified experimental protocol for measuring visual responses to emissive stimuli that do not require participant training, nor requiring the exclusion of non-expert participants. Furthermore, the protocol can be applied to stimuli covering an extended luminance range. Based on the outcome of our experiment, we propose a new model describing lightness response over an extended luminance range. The model can be integrated with existing color appearance models or perceptual color spaces. To demonstrate the effectiveness of our model in high dynamic range applications, we evaluate its suitability for dynamic range expansion relative to existing solutions.

Quality evaluation metric for greyscale error diffusion halftone images based on texture and visual characteristics

ABSTRACTHalftone images produced by error diffusion algorithms suffer from texture structures that decrease their visual qualities, such as worm-like and stripe-like defects. An effective quality evaluation metric is needed to evaluate various error diffusion halftoning algorithms and their corresponding halftone images. A texture distortion evaluation metric (TDEM) is proposed to measure the texture distortions within the referenced halftoned greyscale images. First, the weights of different tonal regions were assigned based on the two types of visual objects available and the locations of the texture structures. Then, based on image structure correlations and human visual characteristics, a block operation was performed and TDEM values were obtained by summing the local block-weighted variances based on image information. The proposed metric was validated using objective comparison and psycho-visual experiments. Results indicate that the metric can measure texture distortions effectively, and that the results are consistent with subjective visual perception.

Towards a Perceptually-Motivated Color Space for High Dynamic Range Imaging

To reproduce the appearance of real world scenes, a number of color appearance models have been proposed thanks to adapted psycho-visual experiments. Most of them were designed and intended for a limited dynamic range, or address only dynamic range compression applications. However, given the increasing availability of displays with higher luminance and contrast ranges, a detailed analysis of appearance attributes is also necessary for dynamic range expansion scenarios. In this study, we propose a psycho-visual experimental setup, designed by adapting and combining the adjustment and partition scaling methodologies, which we employ for measuring perceptual colorfulness of color patches with different levels of lightness, chroma and hue. The proposed setup reduces the complexity and increases the efficiency relative to previous experimental setups and allows both expert and non-expert participants to be included. From the collected data, a modified color space is obtained and a new saturation model for dynamic range compression and expansion is derived for high dynamic range imaging.

Motion parallax in stereo 3D

Binocular disparity is the main depth cue that makes stereoscopic images appear 3D. However, in many scenarios, the range of depth that can be reproduced by this cue is greatly limited and typically fixed due to constraints imposed by displays. For example, due to the low angular resolution of current automultiscopic screens, they can only reproduce a shallow depth range. In this work, we study the motion parallax cue, which is a relatively strong depth cue, and can be freely reproduced even on a 2D screen without any limits. We exploit the fact that in many practical scenarios, motion parallax provides sufficiently strong depth information that the presence of binocular depth cues can be reduced through aggressive disparity compression. To assess the strength of the effect we conduct psycho-visual experiments that measure the influence of motion parallax on depth perception and relate it to the depth resulting from binocular disparity. Based on the measurements, we propose a joint disparity-parallax computational model that predicts apparent depth resulting from both cues. We demonstrate how this model can be applied in the context of stereo and multiscopic image processing, and propose new disparity manipulation techniques, which first quantify depth obtained from motion parallax, and then adjust binocular disparity information accordingly. This allows us to manipulate the disparity signal according to the strength of motion parallax to improve the overall depth reproduction. This technique is validated in additional experiments.

Assessment of pedestrian discomfort glare from urban LED lighting

Due to different visual tasks and gaze patterns, the discomfort glare experienced by pedestrians may differ from that experienced by drivers. This paper investigates the discomfort glare experienced by pedestrians under various urban LED luminaires through psychovisual experiments conducted on a test track. The ability of state-of-the-art models to predict the level of discomfort glare, measured on the de Boer rating scale, for this application is also investigated. With one exception, the models all overestimate the mean subjective discomfort glare compared to the experimental data. Models proposed by Lin et al. (2015) and Bullough et al. (2008, 2011) perform well. However, the implementation of these models is not straightforward because choices are needed to estimate some of the variables such as the background luminance and the glare source area.

Adaptive Image-Space Sampling for Gaze-Contingent Real-time Rendering

With ever-increasing display resolution for wide field-of-view displays---such as head-mounted displays or 8k projectors---shading has become the major computational cost in rasterization. To reduce computational effort, we propose an algorithm that only shades visible features of the image while cost-effectively interpolating the remaining features without affecting perceived quality. In contrast to previous approaches we do not only simulate acuity falloff but also introduce a sampling scheme that incorporates multiple aspects of the human visual system: acuity, eye motion, contrast (stemming from geometry, material or lighting properties), and brightness adaptation. Our sampling scheme is incorporated into a deferred shading pipeline to shade the image's perceptually relevant fragments while a pull-push algorithm interpolates the radiance for the rest of the image. Our approach does not impose any restrictions on the performed shading. We conduct a number of psycho-visual experiments to validate scene- and task-independence of our approach. The number of fragments that need to be shaded is reduced by 50% to 80%. Our algorithm scales favorably with increasing resolution and field-of-view, rendering it well-suited for head-mounted displays and wide-field-of-view projection.

User Preferences Modeling and Learning for Pleasing Photo Collage Generation

In this article, we consider how to automatically create pleasing photo collages created by placing a set of images on a limited canvas area. The task is formulated as an optimization problem. Differently from existing state-of-the-art approaches, we here exploit subjective experiments to model and learn pleasantness from user preferences. To this end, we design an experimental framework for the identification of the criteria that need to be taken into account to generate a pleasing photo collage. Five different thematic photo datasets are used to create collages using state-of-the-art criteria. A first subjective experiment where several subjects evaluated the collages, emphasizes that different criteria are involved in the subjective definition of pleasantness. We then identify new global and local criteria and design algorithms to quantify them. The relative importance of these criteria are automatically learned by exploiting the user preferences, and new collages are generated. To validate our framework, we performed several psycho-visual experiments involving different users. The results shows that the proposed framework allows to learn a novel computational model which effectively encodes an inter-user definition of pleasantness. The learned definition of pleasantness generalizes well to new photo datasets of different themes and sizes not used in the learning. Moreover, compared with two state-of-the-art approaches, the collages created using our framework are preferred by the majority of the users.

No reference image quality classification for JPEG-distorted images

In this paper, we address the Image Quality Assessment (IQA) of JPEG-distorted images. We approach the IQA field by focusing on a classification problem that maps different objective metrics into different categorical quality classes. To this end, we adopt a machine learning classification approach, where No Reference (NR) metrics are considered as features, while the assigned classes come from psycho-visual experiments. Eleven NR metrics have been considered: seven specific for blockiness and four general purpose. We evaluate the performance of single metrics and investigate if a pool of metrics can reach better performances than each of the single ones. Five as well as three quality classes are considered, and the corresponding classifiers are tested on two well known databases available in the literature (LIVE and MICT), and on a new database (IVL) presented in this paper.

Psychovisual evaluations of many luminous environments on the internet

In the process of lighting design, usually many lighting alternatives are assessed against criteria such as energy consumption, lamp type, price and also user preferences collected through psychovisual experiments. However, these experiments induce subject fatigue, which limits the number of lighting alternatives that can be evaluated. To overcome this limitation, we propose a new protocol based on (1) the use of the internet and (2) an incomplete experimental design in which each subject judges only a subset of the whole set of visual stimuli. This protocol is validated by comparing three experiments conducted in the laboratory and online. No statistical difference was highlighted between the results of the experiments. Extrapolations suggest that this protocol could be used to assess up to 3000 visual stimuli.

A no-reference metric for demosaicing artifacts that fits psycho-visual experiments

The present work concerns the analysis of how demosaicing artifacts affect image quality and proposes a novel no-reference metric for their quantification. This metric that fits the psycho-visual data obtained by an experiment analyzes the perceived distortions produced by demosaicing algorithms. The demosaicing operation consists of a combination of color interpolation (CI) and anti-aliasing (AA) algorithms and converts a raw image acquired with a single sensor array, overlaid with a color filter array, into a full-color image. The most prominent artifact generated by demosaicing algorithms is called zipper. The zipper artifact is characterized by segments (zips) with an On–Off pattern. We perform psycho-visual experiments on a dataset of images that covers nine different degrees of distortions, obtained using three CI algorithms combined with two AA algorithms. We then propose our no-reference metric based on measures of blurriness, chromatic and achromatic distortions to fit the psycho-visual data. With this metric demosaicing algorithms could be evaluated and compared.

A perceptual model for disparity

Binocular disparity is an important cue for the human visual system to recognize spatial layout, both in reality and simulated virtual worlds. This paper introduces a perceptual model of disparity for computer graphics that is used to define a metric to compare a stereo image to an alternative stereo image and to estimate the magnitude of the perceived disparity change. Our model can be used to assess the effect of disparity to control the level of undesirable distortions or enhancements (introduced on purpose). A number of psycho-visual experiments are conducted to quantify the mutual effect of disparity magnitude and frequency to derive the model. Besides difference prediction, other applications include compression, and re-targeting. We also present novel applications in form of hybrid stereo images and backward-compatible stereo. The latter minimizes disparity in order to convey a stereo impression if special equipment is used but produces images that appear almost ordinary to the naked eye. The validity of our model and difference metric is again confirmed in a study.

A WAVELET-BASED PERCEPTUAL IMAGE CODER INCORPORATING A NEW MODEL FOR COMPRESSION OF COLOR IMAGES

A wavelet-based perceptual image coder for the compression of color images is proposed here in which the coding structure is coupled with Human Visual System models to produce high quality images. The major contribution is the development of a new model for the compression of the color components based on psychovisual experiments, which quantifies the optimum amount of compression that can be applied to the color components for a given rate. The model is developed for YCbCr color space and the perceptually uniform CIE Lab color space. A complete coding structure for the compression of color images is developed by incorporating the new perceptual model. The performance of the proposed coder is compared with a wavelet-based coder that uses the quantization stage of the JPEG2000 standard. The perceptual quality of the compressed images is tested using the wavelet-based subjective and objective perceptual quality matrices such as Mean Opinion Score, Visual Information Fidelity and Visual Signal to Noise Ratio. Though the model is developed for a perceptually lossless high quality image compression, results obtained reveal that the proposed structure gives very good perceptual quality compared to the existing schemes for lower bit rates. These advantages make the proposed coder a candidate for replacing the encoder stage of the current image compression standards.

ESTIMATION OF APPROPRIATE PARAMETER VALUES FOR DOCUMENT BINARIZATION TECHNIQUES

Most of the document binarization techniques have many parameters that must initially be specified. Usually, document binarization evaluation is subjective and employs human observers for the estimation of the proper set of parameters, for each one of the binarization techniques. The selection of the proper values for these parameters is crucial for the final binarization result. However, there is no set of parameters that guarantees the best binarization result for all document images. Thus, the proper parameters must be adapted to each one of the document images. This paper proposes a new technique which permits the estimation of the proper parameters values for each one of the document binarization techniques. The proposed approach is based on a statistical performance analysis of a set of binarization results obtained by the application of a binarization technique, using different parameters values. Using the statistical performance analysis, the best document binarization result of a set of document binarization techniques can also be estimated. From several experimental results, we verify that the proposed evaluation technique is successfully applied to different types of document images. In addition, psycho-visual experiments show that the selection of the best binarization technique, obtained by the proposed approach, agrees in most of the cases, with the human perception ability.

Psychovisually tuned attribute operators for pre-processing digital video

In video compression, image pre-processing is used to improve the overall compression performance by removing noise from the image sequence. The compressibility can further be improved by removing other visually redundant information that the human visual system is not sensitive to, provided that the visual quality of the image sequence is preserved. A new approach to pre-processing is presented, based on attribute morphology in which extrema regions are removed provided that they meet some perceptual criterion, given by the attribute limit. Both the standard area and a new attribute, based on the power within the removed component, are investigated. Psychovisual experiments are used to determine the psychovisually lossless attribute limits and the limits expressed in a generalised form. The performance gain achieved by this approach is determined by comparing the codec outputs for original and the pre-processed images. Results show the compressibility of the pre-processed images, assessed by a number of compression methods, is significantly improved, thus demonstrating the advantages of the attribute morphology methods, with the power attribute providing the greatest improvement.

Performance comparison of masking models based on a new psychovisual test method with natural scenery stimuli

Various image processing applications exploit a model of the human visual system (HVS). One element of HVS-models describes the masking-effect, which is typically parameterized by psycho-visual experiments that employ superimposed sinusoidal stimuli. Those stimuli are oversimplified with respect to real images and can capture only very elementary masking-effects. To overcome these limitations a new psycho-visual test method is proposed. It is based on natural scenery stimuli and operates in the wavelet domain. The collected psycho-visual data is finally used to evaluate the performance of various masking models under conditions as found in real image processing applications like compression.

Design and Evaluation of an Entirely Psychovisual-Based Coding Scheme

In this paper a new psychovisual-based coding scheme is proposed. The analysis and the quantization stages, the two main functions which determine the performances of a coding scheme, are based on the human visual system properties. In the first stage, a filter bank decomposes images into subimages of perceptual significance when a contrast transformation is applied. Analytic cortex filters have been used because they provide an accurate modelization of visual receptive fields. The choice of subbands lies on psychovisual experiments led in the laboratory. It was found that visual information is processed through 17 channels. In the second stage the use of the local band-limited contrast yields very interesting properties concerning the quantization. A scalar and vector quantization have been considered. In this latter case the vector's construction methodology preserves the main properties of the human visual system about perception of quantization impairments and takes into account the masking effect due to interaction between subbands with the same radial frequency but with different orientations. The vector's components are the local band limited contrasts Cij (m, n) defined as the ratio between the luminance Lij at the point (m, n), which belongs to the radial subband i and angular sector j and the average luminance at this location. Hence the vector's dimension depends on the orientation selectivity of the chosen decomposition. The low pass subband, which is nondirectional is scalar quantized. A methodology for automatic subsampling matrix design was also developed. The performance have been evaluated on a set of images in terms of peak SNR, true bit rates, and visual quality. For the latter, no impairments are visible at a distance of four times the height of the used high quality TV monitor. The SNRs are about 6 to 8 dB under the ones of classical subband image coding schemes when producing the same visual quality. Another particularity of this approach, due to the use of the local band limited contrast, lies in the structure of the reconstruction image error which is found to be highly correlated to the structure of the original image.