High Dynamic Range Video Research Articles

Unified implementation of global high dynamic range image tone-mapping algorithms.

High dynamic range (HDR) images and video require tone-mapping for display on low dynamic range (LDR) screens. Many tone-mapping operators have been proposed to convert HDR content to LDR, but almost each has a different implementation structure and requires a different execution time. We propose a unified structure that can represent any global tone-mapping algorithm with an array of just 256 coefficients. These coefficients extracted offline for every HDR image or video frame can be used to convert them to LDR in real time using linear interpolation. The produced LDR images are identical to the images produced by the original implementation of the algorithm. This unified implementation can replicate any global tone-mapping function and requires very low and fixed execution time, which is independent of algorithm and type of content and depends only on image size. Experimental studies are presented to show the accuracy and time efficiency of the proposed implementation.

Formulating Event-Based Image Reconstruction as a Linear Inverse Problem With Deep Regularization Using Optical Flow.

Event cameras are novel bio-inspired sensors that measure per-pixel brightness differences asynchronously. Recovering brightness from events is appealing since the reconstructed images inherit the high dynamic range (HDR) and high-speed properties of events; hence they can be used in many robotic vision applications and to generate slow-motion HDR videos. However, state-of-the-art methods tackle this problem by training an event-to-image Recurrent Neural Network (RNN), which lacks explainability and is difficult to tune. In this work we show, for the first time, how tackling the combined problem of motion and brightness estimation leads us to formulate event-based image reconstruction as a linear inverse problem that can be solved without training an image reconstruction RNN. Instead, classical and learning-based regularizers are used to solve the problem and remove artifacts from the reconstructed images. The experiments show that the proposed approach generates images with visual quality on par with state-of-the-art methods despite only using data from a short time interval. State-of-the-art results are achieved using an image denoising Convolutional Neural Network (CNN) as the regularization function. The proposed regularized formulation and solvers have a unifying character because they can be applied also to reconstruct brightness from the second derivative. Additionally, the formulation is attractive because it can be naturally combined with super-resolution, motion-segmentation and color demosaicing. Code is available at https://github.com/tub-rip/event_based_image_rec_inverse_problem.

Real Time and Content Adaptive Tone Mapping Operator for Hdr Video Content

Real Time and Content Adaptive Tone Mapping Operator for Hdr Video Content

Image Quality Evaluation in Professional HDR/WCG Production Questions the Need for HDR Metrics.

In the quality evaluation of high dynamic range and wide color gamut (HDR/WCG) images, a number of works have concluded that native HDR metrics, such as HDR visual difference predictor (HDR-VDP), HDR video quality metric (HDR-VQM), or convolutional neural network (CNN)-based visibility metrics for HDR content, provide the best results. These metrics consider only the luminance component, but several color difference metrics have been specifically developed for, and validated with, HDR/WCG images. In this paper, we perform subjective evaluation experiments in a professional HDR/WCG production setting, under a real use case scenario. The results are quite relevant in that they show, firstly, that the performance of HDR metrics is worse than that of a classic, simple standard dynamic range (SDR) metric applied directly to the HDR content; and secondly, that the chrominance metrics specifically developed for HDR/WCG imaging have poor correlation with observer scores and are also outperformed by an SDR metric. Based on these findings, we show how a very simple framework for creating color HDR metrics, that uses only luminance SDR metrics, transfer functions, and classic color spaces, is able to consistently outperform, by a considerable margin, state-of-the-art HDR metrics on a varied set of HDR content, for both perceptual quantization (PQ) and Hybrid Log-Gamma (HLG) encoding, luminance and chroma distortions, and on different color spaces of common use.

LOCAL FEATURE EXTRACTION IN IMAGES

The methods of the local feature point extraction are analyzed. The analysis shows that the most effective detectors are based on the brightness gradient determination. They usually use the Harris angle detector, which is complex in calcu­la­tions. The algorithm complexity minimization contradicts both the detector effective­ness and to the high dynamic range of the analyzed image. As a result, the high-speed methods could not recognize the feature points in the heavy luminance conditions. The modification of the high dynamic range (HDR) image compression algorithm based on the Retinex method is proposed. It contains an adaptive filter, which preserves the image edges. The filter is based on a set of feature detectors perfor­ming the Harris-Laplace transform which is much simpler than the Harris angle detector. A prototype of the HDR video camera is designed which provides sharp images. Its structure simplifies the design of the artificial intelligence engine, which is implemented in FPGA of medium or large size.

Robust HDR video watermarking method based on saliency extraction and T-SVD

Robust HDR video watermarking method based on saliency extraction and T-SVD

HDR-BVQM: High dynamic range blind video quality model

Considerable progress has been made toward developing standard dynamic range (SDR) blind video quality assessment (BVQA) models that do not require any baseline reference for quality prediction. However, there is no such method for the high dynamic range (HDR) content. Unlike SDR video, HDR video represents a high-fidelity representation of the real-world scene by preserving the wide luminance range and color gamut. Therefore, SDR BVQA models are not suitable for HDR BVQA. Towards ameliorating this, a first-of-its-kind BVQA model for HDR content is presented in this work. The proposed HDR blind video quality model (HDR-BVQM) is inspired by the spatio-temporal natural scene statistics model, previously employed in SDR blind quality assessment metrics. To build our proposed model, we first develop a comprehensive subjective HDR video quality dataset, including 228 distorted videos generated through three different (H.264, HEVC, packet drop) distortion processes from 19 pristine HDR videos. The developed dataset is then used to extract HDR relevant features, which vary in different distortion types, to train and test the proposed HDR-BVQM. The features are based on the pointwise, pairwise log-derivative, and motion coherence based statistics. Finally, detailed validation and performance comparison is performed with full-reference HDR and no-reference SDR quality assessment methods. The results reveal that the quality prediction by HDR-BVQM correlates with the human judgment of quality.

High Speed and High Dynamic Range Video with an Event Camera.

Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our quantitative experiments show that our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality ( ), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos ( frames per second) of high-speed phenomena (e.g., a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. As an additional contribution, we demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data. We release the reconstruction code, a pre-trained model and the datasets to enable further research.

51‐3: Color Shifting in High Dynamic Range OLED Displays

Displaying High Dynamic Range (HDR) video on an organic light emitting diode (OLED) display aged for 1200 hours produced noticeable color shifting in the HDR mode playback. The experiment and measurements on commercial high end OLED panels from two different companies prove the damaging effect and introduces a proposed measurement system to quantify the color shift in aged OLED panels.

Image Quality Comparison between LCD and OLED Display

The image qualities of LCD and OLED monitors set to their own default settings were compared using forced-choice experiment method. Both displays’ peak white luminance was around 1,000 cd/m2 and color gamut setting was fixed as DCI-P3 for LCD while OLED’s gamut was set as DCI-P3 or BT.2020. The twelve image quality evaluation keywords were collected through the focus group Interview and eleven HDR video clips were selected as test stimuli. During the experiment, the test video was shown on two displays placed side-by-side and thirteen naïve participants were asked to select the display having the better image quality. The experimental results showed that OLED has higher image quality than LCD does because of higher colorfulness in general. Black luminance level affected the image quality for the dark images but the images having large bright area, colorfulness affected the overall image quality. This result shows that not display technology but the color characteristics affects the image quality.

A New Method for the Simultaneous Creation of Live HDR and SDR Video

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Deploying separate high dynamic range (HDR) and standard dynamic range (SDR) production facilities to cover the same event simultaneously can be economically prohibitive. In this case study, we present a new method for creating both ultrahigh-definition (UHD) HDR and HD SDR streams from a single UHD HDR production workflow, developed through a series of real-world UHD HDR production trials in 2018 and 2019, concluding with the 2019 Football Association Challenge Cup (FA Cup), a major U.K. sporting event. We describe the findings from those trials and the lessons learned and include descriptions of what worked well and what worked less well. The workflow solution presented allows for the integration of both HDR camera sources and SDR camera sources (e.g., super slow-motion and specialist cameras), prerecorded or archive SDR content and SDR graphics into a single HDR production environment. It can exploit the backward-compatible nature of the hybrid log-gamma (HLG) HDR solution and recent advances in color volume management. We show how a production’s two outputs, HDR and SDR, can be made using current SDR shading/racking monitoring, with only a single HDR vision guarantee monitor</i> .

Auto Complementary Exposure Control for High Dynamic Range Video Capturing

Recent trends in consumer electronics have led to a proliferation of high dynamic range (HDR) video acquisition and transmission studies. High-end video capturing systems increase the bit depth of each pixel to adapt to the high-contrast ambient light. However, the extended bit depth is pathetically meager compared with the far wider dynamic range of the incident light. What’s more, incompatibility issues for the subsequent compression and transmission system may be arisen along with the increased bit depth. In this paper, we propose an auto complementary exposure (ACE) algorithm for optimal exposure parameter estimation under Siamese Trigger (ST) mode to maximize the entropy of the output HDR image. With the proposed algorithm, neither the sensor nor other related hardware needs to be modified. Only the auto-exposure algorithm in the firmware needs to be updated. In the proposed ACE algorithm, every two consecutive images are treated as Siamese twins, and the exposure parameters for the twin images are calculated simultaneously. The target of the ACE algorithm is not to maximize the entropy of each image but to maximize that of the fused image. Compared to existing auto exposure or bracketed fusing algorithms, the proposed algorithm has three significant advantages. Firstly, no extra bit-depth is needed to embrace the details of the scene. Thus the image transmission system is not affected. Secondly, only two raw images are necessary for one output fused HDR image. The computation load is relatively low. Thirdly, the proposed method can fit video recording, which has a broader application prospect than HDR image capturing. Experiment results on a self-built database and publicly accessible database demonstrate that the algorithm works well under various types of scenarios. Furthermore, the experiment results on the real-time system show that the captured video image reveals many details under high dynamic scenes.

Learning Hdr Video Reconstruction for Dual-Exposure Sensors with Temporally-Alternating Exposures

Learning Hdr Video Reconstruction for Dual-Exposure Sensors with Temporally-Alternating Exposures

An experimental study on the perceived quality of natively graded versus inverse tone mapped high dynamic range video content on television

High Dynamic Range (HDR) television promises to display higher brightness and deeper black levels and thus more vivid and realistic images. However, home video distribution and video broadcasting were historically designed for what we now call standard dynamic range screens (SDR). In order to display SDR content on an HDR screen, it is explicitly or implicitly converted, in a process called inverse tone mapping (iTMO). This paper’s goal is to assess the perceived quality of converted SDR content in comparison to natively graded HDR content. In doing so, this paper aims to enable content creators/distributors to make informed choices between creating/broadcasting HDR content or relying on conversion. To this end, a psychophysical experiment was performed to tests how viewers evaluate the difference between natively graded HDR and a set of SDR to HDR conversion options in a television setup. Results indicate that viewers prefer natively graded HDR content, followed by inverse tone mapping algorithms starting from videos with a compressed dynamic range. When comparing conversion options, users clearly prefer conversion from ‘compressed dynamic range’ SDR over ‘clipped dynamic range’ SDR. Users disliked videos that were naively stretched from standard SDR. In addition, a significant effect of type of sequence was found, with a preference for light scenes with low contrast.

ITU-R Study Group 6 Progress Report 2019–2020

The International Telecommunication Union (ITU) has and continues to develop globally harmonized standards that have progressively enhanced the viewing and listening quality of all media. The ITU championed the global switch to digital broadcasting and the introduction of high-definition television (HDTV). It also pioneered ultrahigh-definition television (UHDTV), high-dynamic range television (HDR-TV), and the developing standards for advanced sound systems, which together bring increased realism and a dramatic increase in the quality of images and sound no matter how they are distributed.

A Perception-Based Inverse Tone Mapping Operator for High Dynamic Range Video Applications

The drastic visual improvements introduced by High Dynamic Range (HDR) technologies open new markets for a wide range of industries. Among them, a significant opportunity is offered to owners of legacy Standard Dynamic Range (SDR) content that can be converted to the new standard to take advantage of the enhanced capabilities of the HDR displays. Similarly, since SDR broadcasting infrastructure will continue to be around for the time being, such conversion process is becoming an obvious necessity. To this end, different approaches have tried to efficiently convert SDR images and videos to HDR format, a procedure well known as inverse Tone Mapping. In this paper, we propose a novel high visual quality video inverse Tone Mapping Operator (iTMO) that addresses the inadequacies of the state-of-the-art methods, resulting in high visual quality HDR videos that match the capabilities of the HDR technology. Our approach is based on human visual perception and employs a segmentation method according to the Human Visual System (HVS) sensitivity to brightness changes in different regions of the frame and constructs the mapping curve using the brightness distribution information of these regions. Our iTMO uses a hybrid approach to achieve an optimal balance between the overall contrast and brightness of the output HDR frame by maximizing a weighted sum of contrast and brightness difference between input SDR and generated HDR frame. The proposed iTMO works equally well for all levels of brightness, eliminating any visual artifacts by dynamically maintaining changes at non-perceivable levels. Subjective and objective evaluations validated the superior visual performance of our proposed iTMO over state-of-the-art methods.

De-ghosted HDR video acquisition for embedded systems

This paper proposes a novel ghost-free High Dynamic Range (HDR) multi-exposure video acquisition suitable for real-time implementation in embedded systems. While the method is limited to stationary cameras, it achieves, with low requirements on resources, results comparable to state-of-the-art de-ghosting methods that are often very computationally expensive and almost impossible to implement in smart cameras and embedded systems. The paper describes the method itself and includes an evaluation of the performance on selected embedded platforms and a comparison of the results to the state of the art using HDR datasets.

UHD-2/8K Camera Recorder Using Organic CMOS Image Sensor

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">We developed an ultrahigh-definition (UHD)-2/8K camera recorder that uses a complementary metal–oxide semiconductor (CMOS) image sensor with an organic thin-film, instead of photodiodes for the opto-electrical conversion element. The advent of UHD-2/8K Super Hi-Vision broadcasts and an increase in the production of UHD-2/8K programs across a variety of genres necessitate the development of a sensor with a good signal-to-noise ratio (SNR) and high-dynamic range (HDR) to capture the full range of HDR video. To ensure successful use of this sensor, we studied the level diagrams (e.g., balancing of sensitivity, SNR, and dynamic range) that would best fit the sensor’s characteristics and use its strengths, and then proceeded accordingly. For the current UHD-2/8K location shooting, a workflow is needed to record raw data and post-process them into a practical video format. To implement a more efficient workflow, we chose to use Grass Valley HQX compression encoding while adding features that make it possible to store video data in a compressed form and simultaneously generate high-definition television (HDTV) proxy data. The workflow that we envisioned uses proxy data for offline editing and relinks them using online editing equipment.</i>

Guest Editorial Introduction to the Special Section on the Joint Call for Proposals on Video Compression With Capability Beyond HEVC

Standardization for digital video compression has shown significant evolution over the last three decades. Starting in 1988 with ITU-T H.261 as the first such standard that was practical for consumer use, ISO/IEC MPEG-1 and H.262/MPEG-2 video (the latter jointly standardized by ITU-T and ISO/IEC) were developed very soon thereafter, creating the first wave of broad usage of digital technology in consumer video, such as broadcast and disc player applications. Later, the H.264/MPEG-4 Advanced Video Coding (AVC) standard was again developed jointly by ITU-T and ISO/IEC experts, with its High Profile becoming dominant from 2004 in HD broadcast and storage, as well as network-based streaming services and private capture of video. With ever-increasing demands for higher quality and the advent of flat-panel displays, the H.265/MPEG-H High Efficiency Video Coding (HEVC) standard became the next generation of video compression standard; with its first version defined in 2013, HEVC has been especially instrumental for the recent deployment of Ultra High Definition (UHD, a.k.a. 4K video). As time has moved forward, video content has continued to become an increasing presence in our lives, with an ever-growing diversification of usage models and continuing demands for higher quality. For example, flat panels evolved towards support of high dynamic range (HDR) video with a wider color gamut, and new modalities for consuming video have appeared, such as head-mounted displays (HMDs).

Fast Opto-electrical Transformation of HDR Videos based on a LUT for Piecewise Linear Approximation

This paper proposes fast opto-electrical transformation of high dynamic range (HDR) signals based on piecewise linear approximation. Non-linear transformation for the opto-electrical transfer function (OETF) is accelerated via two-level look-up table (LUT) linear approximation. First, the range of an input optical signal is divided into 10 intervals of a log10 scale. Then, each log10 scale interval is again divided into multiple sub-intervals. For each sub-interval, the parameters of the linear function for approximation of non-linear transformation are stored in a LUT. Without computation of the parameters for a range, the output of the electrical signal level is easily computed with the linear function. The proposed algorithm was evaluated along with the OETF mapping algorithm (the so-called LUT-linear mapping) in the HDRTools package developed by the Moving Picture Experts Group (MPEG). The experimental results show that the proposed algorithm achieves time savings on average of 42.73% and has a negligible loss in peak signal-to-noise ratio compared to the LUT-linear mapping method.