High Dynamic Range Video Research Articles

Neural Sensors: Learning Pixel Exposures for HDR Imaging and Video Compressive Sensing With Programmable Sensors.

Camera sensors rely on global or rolling shutter functions to expose an image. This fixed function approach severely limits the sensors' ability to capture high-dynamic-range (HDR) scenes and resolve high-speed dynamics. Spatially varying pixel exposures have been introduced as a powerful computational photography approach to optically encode irradiance on a sensor and computationally recover additional information of a scene, but existing approaches rely on heuristic coding schemes and bulky spatial light modulators to optically implement these exposure functions. Here, we introduce neural sensors as a methodology to optimize per-pixel shutter functions jointly with a differentiable image processing method, such as a neural network, in an end-to-end fashion. Moreover, we demonstrate how to leverage emerging programmable and re-configurable sensor-processors to implement the optimized exposure functions directly on the sensor. Our system takes specific limitations of the sensor into account to optimize physically feasible optical codes and we evaluate its performance for snapshot HDR and high-speed compressive imaging both in simulation and experimentally with real scenes.

JSI-GAN: GAN-Based Joint Super-Resolution and Inverse Tone-Mapping with Pixel-Wise Task-Specific Filters for UHD HDR Video

Joint learning of super-resolution (SR) and inverse tone-mapping (ITM) has been explored recently, to convert legacy low resolution (LR) standard dynamic range (SDR) videos to high resolution (HR) high dynamic range (HDR) videos for the growing need of UHD HDR TV/broadcasting applications. However, previous CNN-based methods directly reconstruct the HR HDR frames from LR SDR frames, and are only trained with a simple L2 loss. In this paper, we take a divide-and-conquer approach in designing a novel GAN-based joint SR-ITM network, called JSI-GAN, which is composed of three task-specific subnets: an image reconstruction subnet, a detail restoration (DR) subnet and a local contrast enhancement (LCE) subnet. We delicately design these subnets so that they are appropriately trained for the intended purpose, learning a pair of pixel-wise 1D separable filters via the DR subnet for detail restoration and a pixel-wise 2D local filter by the LCE subnet for contrast enhancement. Moreover, to train the JSI-GAN effectively, we propose a novel detail GAN loss alongside the conventional GAN loss, which helps enhancing both local details and contrasts to reconstruct high quality HR HDR results. When all subnets are jointly trained well, the predicted HR HDR results of higher quality are obtained with at least 0.41 dB gain in PSNR over those generated by the previous methods. The official Tensorflow code is available at https://github.com/JihyongOh/JSI-GAN.

Efficient Debanding Filtering for Inverse Tone Mapped High Dynamic Range Videos

When a low or standard dynamic range video is inverse tone mapped to high dynamic range (HDR), there can be banding artifacts in the output HDR video. We design a highly integrated architecture that can detect and alleviate banding artifacts while preserving true edges and details in an extremely efficient way. This is achieved by a 7-tap edge-aware selective sparse filter. Coding artifacts, such as blocky artifacts, can also be reduced by this filter. The filter includes some parameters that depend on the strengths of the banding artifacts. A parameter selection mechanism is presented which considers smoothness of the banding regions and fidelity of the filtering output. The filter yields significant PSNR gain at the regions of artifacts. Subjective tests demonstrate the great quality improvement achieved by the proposed filter, compared to the quality before filtering. The visual quality provided by the filter is better than or similar to that of algorithms which are far more complex.

SR-ITM-GAN: Learning 4K UHD HDR With a Generative Adversarial Network

Currently, high dynamic range (HDR) videos with high resolution (HR) have become popular due to the display and the rendered technological advancements. However, making ultra-high definition (UHD) with HDR videos is expensive. The legacy low-resolution (LR) standard dynamic range (SDR) format is still largely used in practice. It is necessary to search for a solution to transform LR SDR videos into UHD HDR format. In this paper, we consider joint super resolution and learning inverse tone mapping an issue of high-frequency reconstruction and local contrast enhancement, and we propose an architecture based on a generative adversarial network to apply joint SR-ITM learning. Specifically, we include the residual ResNeXt block (RRXB) as a basic module to better capture high-frequency textures and adopt YUV interpolation to achieve local contrast enhancement. By adopting a generative adversarial network as a pivotal training mechanism, our designs show advantages in both integration and performance. Our code is now available on GitHub: SR-ITM-GAN.

Learning-Based JND-Directed HDR Video Preprocessing for Perceptually Lossless Compression With HEVC

The final consumer of videos is mostly human. Therefore, if videos can be compressed by fully utilizing the perception characteristics of human visual systems (HVS), the bitrates of the compressed videos can be significantly reduced with subjective visual quality degradation as little as possible. Based on this, we newly propose a learning-based Just Noticeable Distortion (JND)-directed preprocessing scheme for perceptual video compression, especially for 10-bit High Dynamic Range (HDR) videos, which is called the HDR-JNDNet. Our HDR-JNDNet effectively suppresses the perceptual redundancy of 10-bit HDR video signals so that the compression efficiency can be significantly enhanced for the HEVC main10 profile encoder. To our best knowledge, our work is the first approach to training a CNN-based model to directly generate the JND-directed suppressed frames of 10-bit HDR video with the negligible perceptual quality difference between the decoded frames for the original HDR video input with and without the preprocessing by our HDR-JNDNet. Via intensive experiments, when the HDR-JNDNet is applied as preprocessing for the HDR video input before compression, it allows to remarkably save the required bitrates up to the maximum (average) 40.66% (18.37%) for 4K-UHD/HDR test videos, with little subjective video quality degradation without increasing the computational complexity.

Efficient Color Artifact Removal Algorithm Based on High-Efficiency Video Coding (HEVC) for High-Dynamic Range Video Sequences

High efficiency video coding (HEVC) has been developed rapidly to support new generation display devices and their ultra high definition (UHD) with high dynamic range (HDR) and wide color gamut (WCG). To support HDR/WCG sequences on the HEVC standard, pre-/post-processing technique has been designed. After an HDR video is compressed, a reconstructed frame exhibits chromatic distortions that resemble color smearing. To remove this color artifact, we herein propose a block-level quantization parameter (QP) offset-control-based efficient compression algorithm for the HDR sequence. First, we extract the candidate coding units (CUs) with the annoying area to the human eye based on the just noticeable distortion (JND) model. Subsequently, the chromatic distorted blocks are verified by the activity function as the chromatic artifact is observed at the nearby strong edge. For the verified artifact blocks, we reassign the $QP\text{s}$ for the Cb and Cr chroma components. Our experimental results show that the proposed method yields an average gain in BD-rate of 3.3% for U, and 3.4% for V with a negligible bitrate increase of 0.3% on average.

High Dynamic Range Video Synthesis Using Superpixel-Based Illuminance-Invariant Motion Estimation

We propose a robust high dynamic range (HDR) video synthesis algorithm using the superpixel-based illuminance-invariant motion estimation technique. The proposed algorithm first selects an input frame in an alternating exposed input video as the reference. Then, the correspondences between two adjacent frames are estimated by employing a feature descriptor, which is robust against illuminance variation, and a superpixel segmentation technique. Next, the input frames are warped to the reference frame using the estimated motion maps. Finally, the final HDR frame is synthesized by constructing a weight map, which can handle complex motions and poor exposures by considering the underlying structures in the input frames. Experimental results on real test sequences show that the proposed algorithm can provide high-quality HDR videos compared with those obtained by state-of-the-art algorithms in terms of both subjective and objective evaluations.

A Novel Chroma Processing Scheme for Improved Color Accuracy of HDR Video Content

Our eyes do not perceive all colors the same way, being sensitive to changes in some colors more than others. In this work, by taking advantage of this characteristic of the human eye, we propose a chroma processing scheme that reduces the color errors generated by the quantization and subsampling processes in the HDR video delivery pipeline. To this end, our method uses the de-correlated color space of ICtCp and re-distributes chroma code-words in favor of those colors that our eyes are most sensitive to. Performance evaluations have shown that our proposed chroma processing can improve color accuracy of the tested HDR video content by an average of 18.6% and 26.2% in terms of color accuracy compared to that of the 10-bit YCbCr for the 4:4:4 and 4:2:0 formats, respectively. Additionally, subjective evaluations showed that the proposed chroma processing scheme does not negatively affect the overall compression performance while it preserves colors through the quantization and subsampling processes much more accurately than existing methods.

Global Rate-Distortion Optimization-Based Rate Control for HEVC HDR Coding

High dynamic range (HDR) video compression technology, which is capable of delivering a wider range of luminance and a larger colour gamut than standard dynamic range (SDR) technology, has been widely used in recent years in many fields, including industrial image processing, digital entertainment, and machine vision. Rate control (RC) is of paramount importance to HDR compression and transmission; accordingly, an RC scheme for HDR in High Efficiency Video Coding (HEVC) is proposed in this paper. First, considering the HDR characteristics, we propose an HDR-Visual Difference Predictor (VDP)-2-based rate-distortion (R-D) model to improve the coding performance. Second, we directly utilize $\lambda $ rather than the bit rate in the optimization process to obtain the optimal solution. Finally, we propose a new model parameter estimation method to further reduce the RC errors. According to our experimental results, significant bit rate reductions in terms of HDR-VDP-2, the Video Quality Metric (VQM) and the mean peak-signal-to-noise ratio (mPSNR) can be achieved on average compared with the state-of-the-art algorithm used in HM16.19.

A NOVEL METHOD FOR HDR VIDEO ENCODING, COMPRESSION AND QUALITY EVALUATION

The high dynamic range imaging/video has become more prominent in the consumer electronic realm as the it allows the human to view a picture e or a video at a minimum eye adaption. The high dynamic ranging is capable enough to capture a full range light from the scene unlike the traditional methods that captures only 8-f-stops. Though the high dynamic range video are produced using more standard methodologies, it still needs continuous developments in the distribution phase. so the paper discourses the shortcomings in the present method of high dynamic range video coding and puts forth the novel method of two layered backward compatible video encoding methods accompanied by the Chroma subsampling and the luminance masking to improve the color accuracy and remove the nonvisible portion respectively of the video content. The performance measurement of the novel method HDR video encoding compression (N-HVEC) in term of mean opinion source, bit rate savings and he HDR-VQM metric is compared with the HDR10 and HDR-HLG

High Dynamic Range Video Coding Technology in Responses to the Joint Call for Proposals on Video Compression With Capability Beyond HEVC

The ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group issued a Call for Proposals (CfP) on video compression with capability beyond HEVC in October 2017. The CfP considered three categories of content – Standard Dynamic Range, High Dynamic Range and Wide Colour Gamut (HDR/WCG), and 360° Omni-directional video. As a result of the CfP process, the development of a new video coding standard, named Versatile Video Coding (VVC), was initiated. The goal of this paper is to provide an overview of the CfP responses for the HDR/WCG category. The paper includes a summary of work leading to the development of the CfP, a presentation of the CfP results for the HDR/WCG category, and a description of the specific HDR/WCG technologies submitted to the CfP.

35.3: Invited Paper: Effects of Luminance, Contrast and Saturation of HDR QLED Display on Visual System Based on Eye Movement

This paper investigated the effects of luminance, contrast and saturation of HDR video presented on QLED display on visual system. Results showed that mainly, contrast and saturation influenced visual neural systems controlling eye movements, as well as subjective evaluation of display quality and viewing preference, while luminance generally had no significant influence.

The effect of labeling on the perceived quality of HDR video transmission

At the time of this paper, high dynamic range (HDR) visualization has already emerged in both the industry and the commercial sector, with HDR displays already present on the consumer market and the technology marching towards the goal of becoming the de facto format of multimedia. HDR is thus often looked at and praised as the next logical step in the evolution of audiovisual entertainment. However, there is no such thing as a single, universal HDR standard, and the competing market projects a future with even more diversity in format specifications, display capabilities and content characteristics. As the competitors attempt to surpass each other and obtain a bigger share of the global market, they inevitably bombard the potential customers and users with brief but effective labels that reflect excellence and superior quality. In this paper, the cognitive effect of such labels is investigated. As home video entertainment is possibly the most numerous instance of the future usage of this visualization technology, video streaming is particularly addressed. Since real-time video transmission services are bound to suffer playback interruptions upon insufficient data rates and uncompensated drops in the available bandwidth, stalling events experienced in conventional multimedia streaming shall apply to HDR video as well. The paper presents four separate experiments, studying how the cognitive bias caused by the labeling effect influences the perception of HDR quality aspects and stalling events, and how the cognitive load varies for stalling detection thresholds between conventional and HDR visualization.

Uniform Color Space-Based High Dynamic Range Video Compression

Recently, there has been a significant progress in the research and development of the high dynamic range (HDR) video technology and the state-of-the-art video pipelines are able to offer a higher bit depth support to capture, store, encode, and display HDR video content. In this paper, we introduce a novel HDR video compression algorithm, which uses a perceptually uniform color opponent space, a novel perceptual transfer function to encode the dynamic range of the scene, and a novel error minimization scheme for accurate chroma reproduction. The proposed algorithm was objectively and subjectively evaluated against four state-of-the-art algorithms. The objective evaluation was conducted across a set of 39 HDR video sequences, using the latest x265 10-bit video codec along with several perceptual and structural quality assessment metrics at 11 different quality levels. Furthermore, a rating-based subjective evaluation ( $n=40$ ) was conducted with six sequences at two different output bitrates. Results suggest that the proposed algorithm exhibits the lowest coding error amongst the five algorithms evaluated. Additionally, the rate–distortion characteristics suggest that the proposed algorithm outperforms the existing state-of-the-art at bitrates ≥ 0.4 bits/pixel.

22‐2: HDR Dynamic Tone Mapping with Enhanced Rendering Control

High dynamic range (HDR) video standards have been evolving from static tone mapping (e.g. HDR10) to dynamic tone mapping. This enables tone mapping customized for each scene or frame. This paper introduces an enhanced tone mapping curve formulation which helps improve rendering precision for shadow details and addresses the display's low‐end luminance characteristics.

P‐33: Effects of Luminance, Contrast and Saturation of HDR QLED Display on Visual System Based on Eye Movement

The effects of screen luminance, contrast and saturation of HDR video presented on QLED display on visual system were investigated. Results showed that contrast and saturation, rather than luminance, influenced visual neural systems controlling eye movements, and affected subjective display quality and viewing preference.

A high-dynamic range CMOS camera based on dual-gain channels

To overcome the ghosting phenomenon of multi-exposure technology, a new high-dynamic range (HDR) image processing method is proposed in this paper, which combines the features from dual-gain channel images. Further, a complete CMOS camera based on the HDR method is implemented, which produces a real-time HDR live video streams. This camera can capture the details of bright and dark areas in the scene completely with an extended dynamic range up to 95 dB. An ALTERA FPGA is the core processing unit of the entire camera, and it completes all the functional modules of the camera efficiently, including dual-channel video capture, image caching, HDR synthesis and tone mapping. Finally, the real-time HDR video flow has a display resolution of 1920 × 1080 and a frame rate of 60 fps.

Deep HDR Video from Sequences with Alternating Exposures

AbstractA practical way to generate a high dynamic range (HDR) video using off‐the‐shelf cameras is to capture a sequence with alternating exposures and reconstruct the missing content at each frame. Unfortunately, existing approaches are typically slow and are not able to handle challenging cases. In this paper, we propose a learning‐based approach to address this difficult problem. To do this, we use two sequential convolutional neural networks (CNN) to model the entire HDR video reconstruction process. In the first step, we align the neighboring frames to the current frame by estimating the flows between them using a network, which is specifically designed for this application. We then combine the aligned and current images using another CNN to produce the final HDR frame. We perform an end‐to‐end training by minimizing the error between the reconstructed and ground truth HDR images on a set of training scenes. We produce our training data synthetically from existing HDR video datasets and simulate the imperfections of standard digital cameras using a simple approach. Experimental results demonstrate that our approach produces high‐quality HDR videos and is an order of magnitude faster than the state‐of‐the‐art techniques for sequences with two and three alternating exposures.

Video Luminance Levels for Human Skin Tones in Hybrid Log-Gamma Video

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">This paper presents current work undertaken by the British Broadcasting Corp. (BBC) to look at the representation of human skin tones in International Telecommunication Union-Radiocommunication (ITU-R) Recommendation BT.2100 high dynamic range video systems. Specifically, the author presents luma values for a range of skin tones, categorized by the Fitzpatrick scale, which can be used as a lineup guide by cameramen, racks/shader operators, and colorists. The effect of makeup is also covered.</i>

An Adaptive Quantizer for High Dynamic Range Content: Application to Video Coding

In this paper, we propose an adaptive perceptual quantization method to convert the representation of high dynamic range (HDR) content from the floating point data type to integer, which is compatible with the current image/video coding and display systems. The proposed method considers the luminance distribution of the HDR content, as well as the detectable contrast threshold of the human visual system, in order to preserve more contrast information than the perceptual quantizer (PQ) in integer representation. Aiming to demonstrate the effectiveness of this quantizer for HDR video compression, we implemented it in a mapping function on the top of the HDR video coding system based on high efficiency video coding standard. Moreover, a comparison function is also introduced to decrease the additional bit-rate of side information, generated by the mapping function. Objective quality measurements and subjective tests have been conducted in order to evaluate the quality of the reconstructed HDR videos. Subjective test results have shown that the proposed method can improve, in a significant manner, the perceived quality of some reconstructed HDR videos. In the objective assessment, the proposed method achieves improvements over PQ in terms of the average bit-rate gain for metrics used in the measurement.