High Dynamic Range Content Research Articles

HDR-ChipQA: No-reference quality assessment on High Dynamic Range videos

We present a no-reference video quality model and algorithm that delivers standout performance for High Dynamic Range (HDR) videos, which we call HDR-ChipQA. HDR videos represent wider ranges of luminances, details, and colors than Standard Dynamic Range (SDR) videos. The growing adoption of HDR in massively scaled video networks has driven the need for video quality assessment (VQA) algorithms that better account for distortions on HDR content. In particular, standard VQA models may fail to capture conspicuous distortions at the extreme ends of the dynamic range, because the features that drive them may be dominated by distortions that pervade the mid-ranges of the signal. We introduce a new approach whereby a local expansive nonlinearity emphasizes distortions occurring at the higher and lower ends of the local luma range, allowing for the definition of additional quality-aware features that are computed along a separate path. These features are not HDR-specific, and also improve VQA on SDR video contents, albeit to a reduced degree. We show that this preprocessing step significantly boosts the power of distortion-sensitive natural video statistics (NVS) features when used to predict the quality of HDR content. In similar manner, we separately compute novel wide-gamut color features using the same nonlinear processing steps. We have found that our model significantly outperforms SDR VQA algorithms on the only publicly available, comprehensive HDR database, while also attaining state-of-the-art performance on SDR content.

HDRC: a subjective quality assessment database for compressed high dynamic range image

Evaluating the quality of high dynamic range (HDR) images has emerged as a challenging and contemporary topic with the proliferation of HDR content. In this work, a new HDR compression (HDRC) database is proposed, aiming to provide a benchmark for the development of full-reference HDR image quality assessment (IQA) algorithms when facing the latest HDR compression distortions. In particular, the proposed HDRC database is the first HDR-IQA database to incorporate Versatile Video Coding (VVC) compression distortions, closely associated with practical application scenarios. Furthermore, the proposed HDRC database is currently the largest HDR-IQA database, including 80 reference images and 400 distorted images. Extensive experiments are conducted by studying the performance compared to existing HDR-IQA databases when evaluating three HDR-specific IQA models and nine IQA models prevalent for low dynamic range (LDR) content, revealing the challenges the proposed HDRC database brings. The results indicate that the existing IQA models demonstrate noticeable decreases in accuracy when assessing new compression distortions, underscoring the need for the development of novel HDR-IQA models. Consequently, the suggested HDRC database can serve as a potential database for HDR-IQA research, fostering a comprehensive exploration of the associated fields.

A Study of Subjective and Objective Quality Assessment of HDR Videos.

As compared to standard dynamic range (SDR) videos, high dynamic range (HDR) content is able to represent and display much wider and more accurate ranges of brightness and color, leading to more engaging and enjoyable visual experiences. HDR also implies increases in data volume, further challenging existing limits on bandwidth consumption and on the quality of delivered content. Perceptual quality models are used to monitor and control the compression of streamed SDR content. A similar strategy should be useful for HDR content, yet there has been limited work on building HDR video quality assessment (VQA) algorithms. One reason for this is a scarcity of high-quality HDR VQA databases representative of contemporary HDR standards. Towards filling this gap, we created the first publicly available HDR VQA database dedicated to HDR10 videos, called the Laboratory for Image and Video Engineering (LIVE) HDR Database. It comprises 310 videos from 31 distinct source sequences processed by ten different compression and resolution combinations, simulating bitrate ladders used by the streaming industry. We used this data to conduct a subjective quality study, gathering more than 20,000 human quality judgments under two different illumination conditions. To demonstrate the usefulness of this new psychometric data resource, we also designed a new framework for creating HDR quality sensitive features, using a nonlinear transform to emphasize distortions occurring in spatial portions of videos that are enhanced by HDR, e.g., having darker blacks and brighter whites. We apply this new method, which we call HDRMAX, to modify the widely-deployed Video Multimethod Assessment Fusion (VMAF) model. We show that VMAF+HDRMAX provides significantly elevated performance on both HDR and SDR videos, exceeding prior state-of-the-art model performance. The database is now accessible at: https://live.ece.utexas.edu/research/LIVEHDR/LIVEHDR_index.html. The model will be made available at a later date at: https://live.ece.utexas.edu//research/Quality/index_algorithms.htm.

Luminance and Detail Enhancement for HDR Images Based on Surround-Aware Perceptual Quantization Under Ambient Illumination

The lighting conditions of mobile devices could severely affect the viewing experience of the displayed image, such as loss of details and contrast, especially for high dynamic range (HDR) contents. In this paper, a content independent luminance and detail enhancement method is proposed for HDR images under ambient illumination. The perceptual luminance at the given lighting conditions is first quantified based on the human visual adaption. Combining the quantization curve and the display model, the mapping operator for the luminance channel could then be efficiently constructed. Both psychophysical evaluation and objective measurement indicate that the proposed method significantly improves the perceptual quality of HDR images under ambient illumination.

Multi-Scale Attention-Guided Non-Local Network for HDR Image Reconstruction.

High-dynamic-range (HDR) image reconstruction methods are designed to fuse multiple Low-dynamic-range (LDR) images captured with different exposure values into a single HDR image. Recent CNN-based methods mostly perform local attention- or alignment-based fusion of multiple LDR images to create HDR contents. Depending on a single attention mechanism or alignment causes failure in compensating ghosting artifacts, which can arise in the synthesized HDR images due to the motion of objects or camera movement across different LDR image inputs. In this study, we propose a multi-scale attention-guided non-local network called MSANLnet for efficient HDR image reconstruction. To mitigate the ghosting artifacts, the proposed MSANLnet performs implicit alignment of LDR image features with multi-scale spatial attention modules and then reconstructs pixel intensity values using long-range dependencies through non-local means-based fusion. These modules adaptively select useful information that is not damaged by an object’s movement or unfavorable lighting conditions for image pixel fusion. Quantitative evaluations against several current state-of-the-art methods show that the proposed approach achieves higher performance than the existing methods. Moreover, comparative visual results show the effectiveness of the proposed method in restoring saturated information from original input images and mitigating ghosting artifacts caused by large movement of objects. Ablation studies show the effectiveness of the proposed method, architectural choices, and modules for efficient HDR reconstruction.

Deep Controllable Backlight Dimming for HDR Displays

High dynamic range (HDR) displays with dual-panels are one type of displays that can provide HDR content. These are composed of a white backlight panel and a colour LCD panel. Local dimming algorithms are used to control the backlight panel in order to reproduce content with high dynamic range and contrast at a high fidelity. However, existing local dimming algorithms usually process low dynamic range (LDR) images, which are not suitable for processing HDR images. In addition, these methods use hand-crafted features to estimate the backlight values, which may not be suitable for many kind of images. In this work, a novel deep learning based local dimming method is proposed for rendering HDR images on dual-panel HDR displays. The method uses a Convolutional Neural Network (CNN) to directly predict backlight values, using as input the HDR image that is to be displayed. The model is designed and trained via a controllable power parameter that allows a user to trade off between power and quality. The proposed method is evaluated against seven other methods on a test set of 105 HDR images, using a variety of quantitative quality metrics. Results demonstrate improved display quality and better power consumption when using the proposed method compared to the best alternatives.

Image Segmentation Methods for Subpicture Partitioning in the VVC Video Encoder

The emergence of the new generation video coding standard, Versatile Video Coding (VVC), has brought along novel features rendering the new standard more efficient and flexible than its predecessors. Aside from efficient compression of 8 k or higher camera-captured content, VVC also supports a wide range of applications, including computer-generated content, high dynamic range (HDR) content, multilayer and multi-view coding, video region extraction, as well as 360° video. One of the newly introduced coding tools in VVC, offering extraction and independent coding of rectangular sub-areas within a frame, is called Subpicture. In this work, we turn our attention to frame partitioning using Subpictures in VVC, and more particularly, a content-aware partitioning is considered. To achieve that, we make use of image segmentation algorithms and properly modify them to operate on a per Coding Tree Unit (CTU) basis in order to render them compliant with the standard’s restrictions. Additionally, since subpicture boundaries need to comply with slice boundaries, we propose two methods for properly partitioning a frame using tiles/slices aiming to avoid over-partitioning of a frame. The proposed algorithms are evaluated regarding both compression efficiency and image segmentation effectiveness. Our evaluation results indicate that the proposed partitioning schemes have a negligible impact on compression efficiency and video quality

DenSE SwinHDR: SDRTV to HDRTV Conversion Using Densely Connected Swin Transformer With Squeeze and Excitation Module

Modern displays have the capability of rendering video contents encoded with high dynamic range (HDR) standards. HDR contents deliver more realistic visual experiences by wider color gamut and luminance compared to standard dynamic range (SDR) contents. However, most of the available contents are encoded with SDR standards. To provide HDR contents, the technology converts existing SDR contents to HDR contents, what we call SDRTV-to-HDRTV, is highly demanded by providers such as IPTV or broadcasting services. In this paper, we divide SDRTV-to-HDRTV conversion problem into global and local mapping problems. Transformers recently achieve significant performance and are known for conducting global mapping effectively. Convolution Neural Networks (CNN) are specialized for extracting and converting local features. In this regard, we introduce a combined model with a transformer for global mapping and CNN for local mapping, which solves the SDRTD-to-HDRTV problem in a complementary manner. We intensively explore the best combination strategy for transformers and CNNs. Through comprehensive objective/subjective experiments, we verified that the proposed method achieves the highest performance compared to the existing models in both fidelity and visual quality perspectives. To the best of our knowledge, we are the first to utilize Vision Transformer for the SDRTV-to-HDRTV conversion problem. To boost the performance, we combined Vision Transformer with architectural strategies which are previously applied on convolutional neural networks such as residual connection, dense connection, and squeeze-and-excitation module. We introduce a new Vision Transformer architecture denoted as DenSE-SwinHDR. Our method outperforms in terms of objective scores and visual quality compared to the state-of-the-art methods. Specifically, DenSE-SwinHDR achieved 0.79 dB PSNR, 0.93 dB PU-PSNR gain over HDRTVNet. Also, our proposed method achieve best performance on subjective quality assessment.

Unified implementation of global high dynamic range image tone-mapping algorithms.

High dynamic range (HDR) images and video require tone-mapping for display on low dynamic range (LDR) screens. Many tone-mapping operators have been proposed to convert HDR content to LDR, but almost each has a different implementation structure and requires a different execution time. We propose a unified structure that can represent any global tone-mapping algorithm with an array of just 256 coefficients. These coefficients extracted offline for every HDR image or video frame can be used to convert them to LDR in real time using linear interpolation. The produced LDR images are identical to the images produced by the original implementation of the algorithm. This unified implementation can replicate any global tone-mapping function and requires very low and fixed execution time, which is independent of algorithm and type of content and depends only on image size. Experimental studies are presented to show the accuracy and time efficiency of the proposed implementation.

Developments in International Video Coding Standardization After AVC, With an Overview of Versatile Video Coding (VVC)

In the last 17 years, since the finalization of the first version of the now-dominant H.264/Moving Picture Experts Group-4 (MPEG-4) Advanced Video Coding (AVC) standard in 2003, two major new generations of video coding standards have been developed. These include the standards known as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC). HEVC was finalized in 2013, repeating the ten-year cycle time set by its predecessor and providing about 50% bit-rate reduction over AVC. The cycle was shortened by three years for the VVC project, which was finalized in July 2020, yet again achieving about a 50% bit-rate reduction over its predecessor (HEVC). This article summarizes these developments in video coding standardization after AVC. It especially focuses on providing an overview of the first version of VVC, including comparisons against HEVC. Besides further advances in hybrid video compression, as in previous development cycles, the broad versatility of the application domain that is highlighted in the title of VVC is explained. Included in VVC is the support for a wide range of applications beyond the typical standard- and high-definition camera-captured content codings, including features to support computer-generated/screen content, high dynamic range content, multilayer and multiview coding, and support for immersive media such as 360° video.

HDR

<fig orientation="portrait" position="float" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <graphic orientation="portrait" position="float" xlink:href="devli-3094680.tif"/> </fig> Ultrahigh-definition (UHD) and high-dynamic range (HDR) seem to be ubiquitous. We can now generate UHD HDR content from the phones in our pockets just like the professional media industry creates using programming. Therefore, the technology must be stable, and there is no more work to do, right?

Deep HDR Hallucination for Inverse Tone Mapping.

Inverse Tone Mapping (ITM) methods attempt to reconstruct High Dynamic Range (HDR) information from Low Dynamic Range (LDR) image content. The dynamic range of well-exposed areas must be expanded and any missing information due to over/under-exposure must be recovered (hallucinated). The majority of methods focus on the former and are relatively successful, while most attempts on the latter are not of sufficient quality, even ones based on Convolutional Neural Networks (CNNs). A major factor for the reduced inpainting quality in some works is the choice of loss function. Work based on Generative Adversarial Networks (GANs) shows promising results for image synthesis and LDR inpainting, suggesting that GAN losses can improve inverse tone mapping results. This work presents a GAN-based method that hallucinates missing information from badly exposed areas in LDR images and compares its efficacy with alternative variations. The proposed method is quantitatively competitive with state-of-the-art inverse tone mapping methods, providing good dynamic range expansion for well-exposed areas and plausible hallucinations for saturated and under-exposed areas. A density-based normalisation method, targeted for HDR content, is also proposed, as well as an HDR data augmentation method targeted for HDR hallucination.

HDR-BVQM: High dynamic range blind video quality model

Considerable progress has been made toward developing standard dynamic range (SDR) blind video quality assessment (BVQA) models that do not require any baseline reference for quality prediction. However, there is no such method for the high dynamic range (HDR) content. Unlike SDR video, HDR video represents a high-fidelity representation of the real-world scene by preserving the wide luminance range and color gamut. Therefore, SDR BVQA models are not suitable for HDR BVQA. Towards ameliorating this, a first-of-its-kind BVQA model for HDR content is presented in this work. The proposed HDR blind video quality model (HDR-BVQM) is inspired by the spatio-temporal natural scene statistics model, previously employed in SDR blind quality assessment metrics. To build our proposed model, we first develop a comprehensive subjective HDR video quality dataset, including 228 distorted videos generated through three different (H.264, HEVC, packet drop) distortion processes from 19 pristine HDR videos. The developed dataset is then used to extract HDR relevant features, which vary in different distortion types, to train and test the proposed HDR-BVQM. The features are based on the pointwise, pairwise log-derivative, and motion coherence based statistics. Finally, detailed validation and performance comparison is performed with full-reference HDR and no-reference SDR quality assessment methods. The results reveal that the quality prediction by HDR-BVQM correlates with the human judgment of quality.

Query-by-example HDR image retrieval based on CNN

Due to the expension of High Dynamic Range (HDR) imaging applications into various aspects of daily life, an efficient retrieval system, tailored to this type of data, has become a pressing challenge. In this paper, the reliability of Convolutional Neural Networks (CNN) descriptor and its investigation for HDR image retrieval are studied. The main idea consists in exploring the use of CNN to determine HDR image descriptor. Specifically, a Perceptually Uniform (PU) encoding is initially applied to the HDR content to map the luminance values in a perceptually uniform scale. Afterward, the CNN features, using Fully Connected (FC) layer activation, are extracted and classified by applying the Support Vector Machines (SVM) algorithm. Experimental evaluation demonstrates that the CNN descriptor, using the VGG19 network, achieves satisfactory results for describing HDR images on public available datasets such as PascalVoc2007, Cifar-10 and Wang. The experimental results also show that the features, after a PU processing, are more descriptive than those directly extracted from HDR contents. Finally, we show the superior performance of the proposed method against a recent state-of-the-art technique.

A Comprehensive Analysis of Crowdsourcing for Subjective Evaluation of Tone Mapping Operators

Tone mapping operators (TMO) are pivotal in rendering High Dynamic Range (HDR) content on limited dynamic range media. Analysing the quality of tone mapped images depends on several objective factors and a combination of several subjective factors like aesthetics, fidelity etc. Objective Image quality assessment (IQA) metrics are often used to evaluate TMO quality but they do not always reflect the ground truth. A robust alternative to objective IQA metrics is subjective quality assessment. Although, subjective experiments provide accurate results, they can be time-consuming and expensive to conduct. Over the last decade, crowdsourcing experiments have become more popular for collecting large amount of data within a shorter period of time for a lesser cost. Although they provide more data requiring less resources, lack of controlled environment for the experiment results in noisy data. In this work1, we propose a comprehensive analysis of crowdsourcing experiments with two different groups of participants. Our contributions include a comparative study and a collection of methods to detect unreliable participants in crowdsourcing experiments in a TMO quality evaluation scenario. These methods can be utilized by the scientific community to increase the reliability of the gathered data.

Deep Tone-Mapping Operator Using Image Quality Assessment Inspired Semi-Supervised Learning

Tone-mapping operator (TMO) is intended to convert high dynamic range (HDR) content into a lower dynamic range so that it can be displayed on a standard dynamic range (SDR) device. The tone-mapped result of HDR content is usually stored as SDR image. For different HDR scenes, traditional TMOs are able to obtain a satisfying SDR image only under manually fine-tuned parameters. In this paper, we address this problem by proposing a learning-based TMO using deep convolutional neural network (CNN). We explore different CNN structure and adopt multi-scale and multi-branch fully convolutional design. When training deep CNN, we introduce image quality assessments (IQA), specifically, tone-mapped image quality assessment, and implement it as semi-supervised loss terms. We discuss and prove the effectiveness of semi-supervised loss terms, CNN structure, data pre-processing, etc. by several experiments. Finally, we demonstrate that our approach can produce appealing results under diversified HDR scenes.

Human Visual System Model-Based Optimized Tone Mapping of High Dynamic Range Images

High dynamic range (HDR) image and video technology can provide significant picture quality improvement over the standard dynamic range (SDR). However, when HDR content is represented on an SDR display, dynamic range compression may result in image quality deterioration. To address this problem, we propose an optimized human visual system (HVS) response model-based tone-mapping algorithm to preserve the perceptual responses between the HDR image and its tone-mapped image. First, we measure the HVS response differences using a 2D histogram when an HDR image is displayed on an HDR device and when its tone-mapped image is displayed on an SDR device. Then, we formulate an optimization problem to minimize the differences. By efficiently solving the optimization problem, we obtain an optimal tone-mapping curve. Experimental results on actual displays demonstrate that the proposed algorithm provides superior image quality compared with conventional algorithms in terms of both subjective and objective evaluations.

Highlights Analysis System (HAnS) for Low Dynamic Range to High Dynamic Range Conversion of Cinematic Low Dynamic Range Content

We propose a novel and efficient algorithm for detection of specular reflections and light sources (highlights) in cinematic content. The detection of highlights is important for reconstructing them properly in the conversion of the low dynamic range (LDR) to high dynamic range (HDR) content. Highlights are often difficult to be distinguished from bright diffuse surfaces, due to their brightness being reduced in the conventional LDR content production. Moreover, the cinematic LDR content is subject to the artistic use of effects that change the apparent brightness of certain image regions (e.g. limiting depth of field, grading, complex multi-lighting setup, etc.). To ensure the robustness of highlights detection to these effects, the proposed algorithm goes beyond considering only absolute brightness and considers five different features. These features are: the size of the highlight relative to the size of the surrounding image structures, the relative contrast in the surrounding of the highlight, its absolute brightness expressed through the luminance (luma feature), through the saturation in the color space (maxRGB feature) and through the saturation in white (minRGB feature). We evaluate the algorithm on two different image data-sets. The first one is a publicly available LDR image data-set without cinematic content, which allows comparison to the broader State of the art. Additionally, for the evaluation on cinematic content, we create an image data-set consisted of manually annotated cinematic frames and real-world images. For the purpose of demonstrating the proposed highlights detection algorithm in a complete LDR-to-HDR conversion pipeline, we additionally propose a simple inverse-tone-mapping algorithm. The experimental analysis shows that the proposed approach outperforms conventional highlights detection algorithms on both image data-sets, achieves high quality reconstruction of the HDR content and is suited for use in LDR-to-HDR conversion.

TMO-Net: A Parameter-Free Tone Mapping Operator Using Generative Adversarial Network, and Performance Benchmarking on Large Scale HDR Dataset

Currently published tone mapping operators (TMO) are often evaluated on a very limited test set of high dynamic range (HDR) images. Thus, the resulting performance index is highly subject to extensive hyperparameter tuning, and many TMOs exhibit sub-optimal performance when tested on a broader spectrum of HDR images. This indicates that there are deficiencies in the generalizable applicability of these techniques. Finally, it is a challenge developing parameter-free tone mapping operators using data-hungry advanced deep learning methods due to the paucity of large scale HDR datasets. In this paper, these issues are addressed through the following contributions: a) a large scale HDR image benchmark dataset (LVZ-HDR dataset) with multiple variations in sceneries and lighting conditions is created to enable performance evaluation of TMOs across a diverse conditions and scenes that will also contribute to facilitate the development of more robust TMOs using state-of-the-art deep learning methods; b) a deep learning-based tone mapping operator (TMO-Net) is presented, which offers an efficient and parameter-free method capable of generalizing effectively across a wider spectrum of HDR content; c) finally, a comparative analysis, and performance benchmarking of 19 state-of-the-art TMOs on the new LVZ-HDR dataset are presented. Standard metrics including the Tone Mapping Quality Index (TMQI), Feature Similarity Index for Tone Mapped images (FSITM), and Natural Image Quality Evaluator (NIQE) are used to qualitatively evaluate the performance index of the benchmarked TMOs. Experimental results demonstrate that the proposed TMO-Net qualitatively and quantitatively outperforms current state-of-the-art TMOs.

An experimental study on the perceived quality of natively graded versus inverse tone mapped high dynamic range video content on television

High Dynamic Range (HDR) television promises to display higher brightness and deeper black levels and thus more vivid and realistic images. However, home video distribution and video broadcasting were historically designed for what we now call standard dynamic range screens (SDR). In order to display SDR content on an HDR screen, it is explicitly or implicitly converted, in a process called inverse tone mapping (iTMO). This paper’s goal is to assess the perceived quality of converted SDR content in comparison to natively graded HDR content. In doing so, this paper aims to enable content creators/distributors to make informed choices between creating/broadcasting HDR content or relying on conversion. To this end, a psychophysical experiment was performed to tests how viewers evaluate the difference between natively graded HDR and a set of SDR to HDR conversion options in a television setup. Results indicate that viewers prefer natively graded HDR content, followed by inverse tone mapping algorithms starting from videos with a compressed dynamic range. When comparing conversion options, users clearly prefer conversion from ‘compressed dynamic range’ SDR over ‘clipped dynamic range’ SDR. Users disliked videos that were naively stretched from standard SDR. In addition, a significant effect of type of sequence was found, with a preference for light scenes with low contrast.