High Dynamic Range Video Research Articles

An experimental study on the impact of cinema theater reflections on users’ perceived intra-frame dynamic range

Due to rapid technological progress in high dynamic range (HDR) video capture and display, perceived contrast by users of HDR technology is becoming increasingly important in the visual signal processing chain. More specifically, when considering a higher contrast and brightness range in cinema, it is essential to understand the limits of visual perception under specific viewing conditions and brightness levels. A field experiment was conducted to test the effect of light scattering and reflections from the cinema theater on users’ perceivable dynamic range. We investigated how the perceivable dynamic range evolves with various background levels and different strengths and shapes of local highlights. Results showed a significant effect of image composition and the average picture level on the perceivable near black dynamic range. In addition to the limitations of the human visual system, one needs to consider the limitations of a theatrical projection setup where bright elements in the image via reflections on the walls, ceiling, floor, seats, and audience contaminate the black elements of the image.

Pupillometry of HDR Video Viewing

Pupillometry of HDR Video Viewing

Visual perception based rate distortion optimization method for high dynamic range video coding

In view of the drastic increase of storage resources and transmission bandwidth requirement for high dynamic range (HDR) video compared to the traditional low dynamic range (LDR) video, we propose a dynamic rate distortion optimization algorithm based on visual perception for HDR Video encoding to improve the performance of high efficiency video coding (HEVC) Main 10 for coding HDR video. With the information of visual selective attention, we design a non-uniform distortion weight distribution strategy to different regions of interest and improve the conventional method of distortion calculation. At the same time, in order to further eliminate the perceptive redundancy in HDR video coding, the texture characteristics of video content are used to adjust Lagrange multipliers adaptively, which is applied to the encoder to dynamically adjust the quantization parameters to realize reasonably the trade-off between coded bits and distortion perception. The experimental results show that the proposed algorithm can save an average of 7.46% and 6.53% bitrate with the same HDR-visible difference predictor-2.2(HDR-VDP-2.2) and PSNR_DE compared with HEVC Main 10, saving the maximum of 18.52 % and 11.49% respectively. The proposed algorithm can effectively reduce the consumption of the overall bitrates and still maintain the visual quality of the reconstructed HDR video.

A bio-inspired synergistic virtual retina model for tone mapping

Real-world radiance values span several orders of magnitudes which have to be processed by artificial systems in order to capture visual scenes with a high visual sensitivity. Interestingly, it has been found that similar processing happens in biological systems, starting at the retina level. So our motivation in this paper is to develop a new video tone mapping operator (TMO) based on a synergistic model of the retina. We start from the so-called Virtual Retina model, which has been developed in computational neuroscience. We show how to enrich this model with new features to use it as a TMO, such as color management, luminance adaptation at photoreceptor level and readout from a heterogeneous population activity. Our method works for video but can also be applied to static images (by repeating images in time). It has been carefully evaluated on standard benchmarks in the static case, giving comparable results to the state-of-the-art using default parameters, while offering user control for finer tuning. Results on HDR videos are also promising, specifically w.r.t. temporal luminance coherency. As a whole, this paper shows a promising way to address computational photography challenges by exploiting the current research in neuroscience about retina processing.

Video surveillance systems-current status and future trends

Within this survey an attempt is made to document the present status of video surveillance systems. The main components of a surveillance system are presented and studied thoroughly. Algorithms for image enhancement, object detection, object tracking, object recognition and item re-identification are presented. The most common modalities utilized by surveillance systems are discussed, putting emphasis on video, in terms of available resolutions and new imaging approaches, like High Dynamic Range video. The most important features and analytics are presented, along with the most common approaches for image / video quality enhancement. Distributed computational infrastructures are discussed (Cloud, Fog and Edge Computing), describing the advantages and disadvantages of each approach. The most important deep learning algorithms are presented, along with the smart analytics that they utilize. Augmented reality and the role it can play to a surveillance system is reported, just before discussing the challenges and the future trends of surveillance.

A multi-modeling electro-optical transfer function for display and transmission of high dynamic range content

This paper proposes a multi-modeling Electro-Optical Transfer Function (EOTF) enabling efficient use of the code word to preserve the contrast details in the quantized High Dynamic Range (HDR) content. Different from the classic EOTF that employs one single model, the proposed EOTF considers four models to adjust the reproducible contrast according to the luminance level of the HDR content. Furthermore, the proposed EOTF and its inverse process functions can be integrated on HDR cameras and displays with a low computational cost. In order to evaluate the performance of the proposed EOTF, it has been applied into a HDR video transmission chain based on the High Efficiency Video Coding (HEVC) codec. The standardized Perceptual Quantizer (PQ) EOTF and Hybrid Log-Gamma (HLG) Opto-Electronic Transfer Function (OETF) are also introduced and compared with the proposed EOTF in terms of the reproducible contrast. The objective evaluation results show that the proposed EOTF brings a better quality of the decoded HDR video than that generated by the standard function. A subjective test was also conducted in a commercial HDR television and the results show that the proposed EOTF can present more visible details than PQ, particularly for the HDR content with dark scenes. Therefore, the proposed multi-modeling EOTF is useful for the transmission and display of the HDR content in the consumer electronics.

2017 Update on High Dynamic Range Television

2017 Update on High Dynamic Range Television

Model-Based Multiscale Gigapixel Image Formation Pipeline on GPU

We present an efficient and flexible GPU implementation of a highly-parallelizable and scalable image formation pipeline for gigapixel images based on the MapReduce framework. The presented implementation was developed to operate with the AWARE multiscale gigapixel cameras, but it is also able to efficiently form gigapixel images from any source. The AWARE cameras are compact camera arrays that simultaneously collect images that span a wide field-of-view to generate high-resolution and high dynamic range panoramic images and video. The proposed GPU implementation exploits the mutiscale nature of the AWARE image acquisition, not only enabling the fast composition of gigapixel-scale panoramas, but also the rapid formation of images of arbitrary portions of the field-of-view at current display-scale resolutions at video rates.

Improved High Dynamic Range Video Coding with a Nonlinearity Based on Natural Image Statistics

Improved High Dynamic Range Video Coding with a Nonlinearity Based on Natural Image Statistics

Context-aware HDR video distribution for mobile devices

Context-aware HDR video distribution for mobile devices

A comparative review of tone-mapping algorithms for high dynamic range video

Tone-mapping constitutes a key component within the field of high dynamic range HDR imaging. Its importance is manifested in the vast amount of tone-mapping methods that can be found in the literature, which are the result of an active development in the area for more than two decades. Although these can accommodate most requirements for display of HDR images, new challenges arose with the advent of HDR video, calling for additional considerations in the design of tone-mapping operators TMOs. Today, a range of TMOs exist that do support video material. We are now reaching a point where most camera captured HDR videos can be prepared in high quality without visible artifacts, for the constraints of a standard display device. In this report, we set out to summarize and categorize the research in tone-mapping as of today, distilling the most important trends and characteristics of the tone reproduction pipeline. While this gives a wide overview over the area, we then specifically focus on tone-mapping of HDR video and the problems this medium entails. First, we formulate the major challenges a video TMO needs to address. Then, we provide a description and categorization of each of the existing video TMOs. Finally, by constructing a set of quantitative measures, we evaluate the performance of a number of the operators, in order to give a hint on which can be expected to render the least amount of artifacts. This serves as a comprehensive reference, categorization and comparative assessment of the state-of-the-art in tone-mapping for HDR video.

A Biologically-Inspired Approach to Making HDR Video Quality Assessment Easier

Quality is easy to recognize by the eye. Quantifying it can be more problematic. For high dynamic range (HDR), it is even harder because convenient methods, such as peak signal-to-noise ratio and structural similarity metric, do not work well. Some newer HDR-aware metrics are now available; however, they can be slow and computationally cumbersome, therefore discouraging use in realtime monitoring. This paper introduces an alternative, simpler technique that could be used in post-production and realtime monitoring during distribution. Specifically, this paper presents a biologically inspired method of creating a “spatial detail” representation that can be used instead of the original HDR image as a basis for quantifying video distortions and artifacts. Simple correlation analysis of the luma spatial detail signal yields an intuitive and sensitive metric of distortion in HDR video. As an added benefit, the spatial detail signal provides a guide that enables distortions in HDR highlights to be evaluated separately from the remainder of the video.

High Speed High Dynamic Range Video

High-speed digital video is a common measurement tool used in the analysis of explosive or similarly energetic events. The dynamic range of a digital camera is limited to bit depth of the analog to digital converter and deep well capacity of the sensor. The dynamic range of an explosive event can exceed the camera’s dynamic range by many orders of magnitude, often in excess of 120 dB causing loss of data from sensor saturation. A method has been developed to utilize current off the shelf synchronized high-speed cameras to film events consisting of extremely high dynamic range events. This paper will describe the process for the design and implementation of an optimized system composed of any number of cameras, and demonstrates the capabilities of a constructed system to capture a video spanning 169 dB range (28 stops) at 79,000 pps using three and four high speed cameras.

Conversion of Conventional Video Display for High Dynamic Range Televisions

This paper presents the results of subjective experiments undertaken by the BBC R&D to create a reversible solution for the display of conventional video content on a Hybrid Log-Gamma high dynamic range (HDR) television. This algorithm is simple to implement in both hardware and software and is designed to allow both the use of archive footage and conventional cameras in an HDR television service and the generation of a non-HDR service in such a way that the transmission-chain of the non-HDR service can reverse the algorithm. This ensures no quality reduction for the non-HDR viewer.

A Maximum a Posteriori Estimation Framework for Robust High Dynamic Range Video Synthesis.

High dynamic range (HDR) image synthesis from multiple low dynamic range exposures continues to be actively researched. The extension to HDR video synthesis is a topic of significant current interest due to potential cost benefits. For HDR video, a stiff practical challenge presents itself in the form of accurate correspondence estimation of objects between video frames. In particular, loss of data resulting from poor exposures and varying intensity makes conventional optical flow methods highly inaccurate. We avoid exact correspondence estimation by proposing a statistical approach via maximum a posterior estimation, and under appropriate statistical assumptions and choice of priors and models, we reduce it to an optimization problem of solving for the foreground and background of the target frame. We obtain the background through rank minimization and estimate the foreground via a novel multiscale adaptive kernel regression technique, which implicitly captures local structure and temporal motion by solving an unconstrained optimization problem. Extensive experimental results on both real and synthetic data sets demonstrate that our algorithm is more capable of delivering high-quality HDR videos than current state-of-the-art methods, under both subjective and objective assessments. Furthermore, a thorough complexity analysis reveals that our algorithm achieves better complexity-performance tradeoff than conventional methods.

HDR video past, present and future: A perspective

High dynamic range (HDR) video has emerged from research labs around the world and entered the realm of consumer electronics. The dynamic range that a human can see in a scene with minimal eye adaption (approximately 1,000,000:1) is vastly greater than traditional imaging technology which can only capture about 8 f-stops (256:1). HDR technology, on the other hand, has the potential to capture the full range of light in a scene; even more than a human eye can see. This paper examines the field of HDR video from capture to display: past, present and future. In particular the paper looks beyond the current marketing hype around HDR to show how HDR video in the future can and, indeed, should bring about a step change in imaging, analogous to the change from black and white to colour.

Video frame synthesizing method for HDR video capturing system with four image sensors

High Dynamic Range (HDR) imaging has recently been applied to video systems, including the next-generation ultrahigh definition television (UHDTV) format. This format requires a camera with a dynamic range of over 15 f-stops and an S/N ratio that is the same as that of HDTV systems. Current UHDTV cameras cannot satisfy these conditions, as their small pixel size decreases the full-well capacity of UHDTV camera image sensors in comparison with that of HDTV sensors. We propose a four-chip capturing method combining threechip and single-chip systems. A prism divides incident light into two rays with intensities in the ratio m:1. Most of the incident light is directed to the three-chip capturing block; the remainder is directed to a single-chip capturing block, avoiding saturation in high-exposure videos. High quality HDR video can then be obtained by synthesizing the high-quality image obtained from the three-chip system with the low saturation image from the singlechip. Herein, we detail this image synthesis method, discuss the smooth matching method between spectrum characteristics of the two systems, and consider the modulation transfer function (MTF) response differences between the three- and single-chip capturing systems by means of analyzing using human visual models.

Optimization of CMOS Image Sensor Utilizing Variable Temporal Multi-Sampling Partial Transfer Technique to Achieve Full-frame High Dynamic Range with Superior Low Light and Stop Motion Capability

Differential Binary Pixel Technology is a threshold-based timing, readout and image reconstruction method that utilizes sub-frame partial charge transfer technique in a standard four-transistor (4T) pixel CMOS Image sensor (CIS) to achieve HDR video with Stop Motion. This technology improves low light signal-tonoise ratio (SNR) by up to 21dB. The method is verified in silicon using a TSMC 65nm 1.1μm pixel technology 1 megapixel (MP) test chip array and is compared with a traditional 4× oversampling technique using full charge transfer. The test chip is also compared with the iPhone 6s rear view camera to show superior HDR video capability.

A full-resolution 8K single-chip portable camera system

We have developed an 8K "full-resolution," 60-fps, portable camera system using a 133-megapixel complementary metaloxide-semiconductor (CMOS) single-chip image sensor. The camera head weighs less than 7 kg, whereas the conventional 8K full-resolution three-chip camera weighs over 50 kg. The new camera can use both commercial 35-mm full-frame lenses and super 35-mm lenses by using lens adapters. It employs a compact 100-Gbps optical transceiver that can transmit the 8K fullresolution video signal to the camera control unit (CCU). The size of the CCU is 3U, which is comparable to CCUs used for broadcasting high definition television (HDTV) cameras. The camera supports the wide-color-gamut and high dynamic range (HDR) video formats, which were standardized in ITU-R BT. 2020 and BT. 2100, respectively. Moreover, a streaking noise correction circuit is implemented in the CCU. The 8K signal output interface from the CCU is compliant with the ITU-R BT. 2077 (U-SDI) standard. By performing an image shooting experiment, we confirmed that the limiting resolution of this camera was more than 4000 TV lines and the signal-to-noise (S/N) ratio was 57 dB at a sensitivity of F4.0/2000 lux.

Video cascade accumulation of the total solar eclipse on Svalbard 2015

Abstract. This work presents a novel image accumulation filter technique that reveals small-scale features and details from intense luminosity or high dynamic range (HDR) video recordings. It was discovered and developed from the analyses of the Norwegian Broadcasting Corporation (NRK) film of the total solar eclipse that occurred Friday 20 March 2015 in Longyearbyen (78° N, 15° E) on Svalbard, Norway. The result of the filter is fused with a HDR image of the corona and the Solar Dynamic Observatory (SDO) image of the solar disk.