Inverse Tone Mapping Research Articles

A brightness-adaptive kernel prediction network for inverse tone mapping

The inverse tone mapping (iTM) technique that produces a high dynamic range (HDR) image from one single standard dynamic range (SDR) image has received much attention in industry and academia recently. However, existing methods to recover HDR images mainly focus on overexposed regions but ignore underexposed regions. The underexposed regions in an image are susceptible to noise and artifacts, which will reduce the quality of the image and the user’s visual experience. Therefore, in this paper, we propose a brightness-adaptive iTM model based on deep learning to focus on the content restoration of both the overexposed and underexposed regions in an SDR image simultaneously. In this model, instead of directly predicting the HDR output, we adopt an encoder-decoder network to predict spatially adaptive kernels, which further convolute the input SDR image to produce the HDR result. With the spatially adaptive kernels, the input regions with different exposures can be adaptively mapped by making full use of neighborhood information. Importantly, brightness-adaptive skip connections in the encoder-decoder network, as well as a region loss, are designed to force the proposed model to attach importance to overexposed and underexposed regions. Besides, a global branch is employed in our encoder to exploit both the global and local brightness. Extensive qualitative and quantitative experiments demonstrate that the proposed approach outperforms recent methods on multiple metrics.

Deep HDR Hallucination for Inverse Tone Mapping.

Inverse Tone Mapping (ITM) methods attempt to reconstruct High Dynamic Range (HDR) information from Low Dynamic Range (LDR) image content. The dynamic range of well-exposed areas must be expanded and any missing information due to over/under-exposure must be recovered (hallucinated). The majority of methods focus on the former and are relatively successful, while most attempts on the latter are not of sufficient quality, even ones based on Convolutional Neural Networks (CNNs). A major factor for the reduced inpainting quality in some works is the choice of loss function. Work based on Generative Adversarial Networks (GANs) shows promising results for image synthesis and LDR inpainting, suggesting that GAN losses can improve inverse tone mapping results. This work presents a GAN-based method that hallucinates missing information from badly exposed areas in LDR images and compares its efficacy with alternative variations. The proposed method is quantitatively competitive with state-of-the-art inverse tone mapping methods, providing good dynamic range expansion for well-exposed areas and plausible hallucinations for saturated and under-exposed areas. A density-based normalisation method, targeted for HDR content, is also proposed, as well as an HDR data augmentation method targeted for HDR hallucination.

HDR-Like Image from Pseudo-Exposure Image Fusion: A Genetic Algorithm Approach

Image distortion during multiple-exposure capture is a common scenario, and most of the existing legacy image content is in the low dynamic range (LDR). For both of these problems, pseudo multiple exposure image generation from a single image is a possible solution. Hence, in this article, we propose a novel method for high dynamic range (HDR)-like image generation by fusing pseudo-multi-exposure images that are produced from only one single image. The proposed pseudo-exposure image generation method is based on intensity mapping function (IMF). First, our system learns the relation between different exposure images with the help of genetic algorithm and generates long and short exposure images. In the second step, we fuse images with different exposures to generate the HDR-like image. Our method outperforms several inverse tone mapping operator (ITMO)-based methods. The simplicity and good performance of the proposed method makes it suitable for mobile phones and other consumer electronics. We also develop a demo software to validate the proposed method.

A review on high dynamic range (HDR) image quality assessment

Abstract This paper presents a literature review on the method of measuring high dynamic range (HDR) image quality. HDR technology can help maximize user satisfaction level when using HDR images-based visual services. The advance of HDR technology indirectly presents a more difficult challenge to the image quality assessment method due to the high sensitivity of the human visual system (HVS) to various kinds of distortions that may arise in HDR images. This is related to the process of HDR image generation, which in general can be classified into two broad categories: the formation using the multiple exposure fusion (MEF) method and the inverse tone mapping operator (ITMO) method. In this paper, we will outline how HDR image quality measurement method works and describe some examples of these measurement methods which are related to the way the HDR images are fabricated. From these methods, it can be seen that most of them are still focused on full-reference and no-reference quality models. We argue that there is still room for the development of reduced-reference HDR image quality assessment.

A Fully Automatic Content Adaptive Inverse Tone Mapping Operator With Improved Color Accuracy

High Dynamic Range (HDR) technology offers a higher visual quality compared to its Standard Dynamic Range (SDR) counterpart, as it tries to imitate the way our eyes perceive brightness and color information. Converting SDR content to HDR format - using inverse Tone Mapping Operators (iTMOs) - to take advantage of the superior visual quality offered by HDR displays, is an attractive proposition to SDR content owners and real-time broadcasters. In this paper, we propose a novel content adaptive iTMO that works in the perceptual domain to model the sensitivity of the human eye to brightness changes in different areas of a scene. To preserve the overall visual impression, our proposed iTMO utilizes an entropy-based brightness segmentation, which also makes our method content adaptive. In addition, we propose a novel perception-based color adjustment method that can maintain the color accuracy between input SDR and generated HDR frames. By performing the color adjustment in the perceptual domain, our iTMO prevents hue shifts and generates HDR colors that closely follow their SDR counterparts. Our subjective evaluations indicate that our proposed method outperforms other state-of-the-art methods by an average of 81% in terms of visual quality, and 76% in terms of how closely the HDR colors match their SDR counterparts. In addition to subjective evaluations, we also performed objective evaluations using the HDR-VDP 2.2 and PU-SSIM metrics and concluded that, on average, our proposed iTMO outperforms the state-of-the-art methods in terms of these two metrics.

Supervised Learning in Automatic Selection of Preferred Inverse Tone-Mapping Operator for HDR Display

Supervised Learning in Automatic Selection of Preferred Inverse Tone-Mapping Operator for HDR Display

An experimental study on the perceived quality of natively graded versus inverse tone mapped high dynamic range video content on television

High Dynamic Range (HDR) television promises to display higher brightness and deeper black levels and thus more vivid and realistic images. However, home video distribution and video broadcasting were historically designed for what we now call standard dynamic range screens (SDR). In order to display SDR content on an HDR screen, it is explicitly or implicitly converted, in a process called inverse tone mapping (iTMO). This paper’s goal is to assess the perceived quality of converted SDR content in comparison to natively graded HDR content. In doing so, this paper aims to enable content creators/distributors to make informed choices between creating/broadcasting HDR content or relying on conversion. To this end, a psychophysical experiment was performed to tests how viewers evaluate the difference between natively graded HDR and a set of SDR to HDR conversion options in a television setup. Results indicate that viewers prefer natively graded HDR content, followed by inverse tone mapping algorithms starting from videos with a compressed dynamic range. When comparing conversion options, users clearly prefer conversion from ‘compressed dynamic range’ SDR over ‘clipped dynamic range’ SDR. Users disliked videos that were naively stretched from standard SDR. In addition, a significant effect of type of sequence was found, with a preference for light scenes with low contrast.

Method for reconstructing a high dynamic range image based on a single-shot filtered low dynamic range image.

Traditional cameras are limited by sensors and cannot directly capture single-shot high dynamic range (HDR) images. We propose an improved single-shot HDR image reconstruction method that uses a single-exposure filtered low dynamic range (FLDR) image. First, by adding an optical filter in front of the camera lens, a FLDR image with different RGB channel exposure states and luminance ranges can be captured in a single-shot, unlike the traditional LDR image. Second, a deep inverse tone mapping network (DITMnet) with multibranch features extraction and multioutput images synthesis is designed to reconstruct an HDR image from a single FLDR image. Experimentally, under different exposure states and color spaces, our method outperforms similar algorithms.

Vision models fine-tuned by cinema professionals for High Dynamic Range imaging in movies

Many challenges that deal with processing of HDR material remain very much open for the film industry, whose extremely demanding quality standards are not met by existing automatic methods. Therefore, when dealing with HDR content, substantial work by very skilled technicians has to be carried out at every step of the movie production chain. Based on recent findings and models from vision science, we propose in this work effective tone mapping and inverse tone mapping algorithms for production, post-production and exhibition. These methods are automatic and real-time, and they have been both fine-tuned and validated by cinema professionals, with psychophysical tests demonstrating that the proposed algorithms outperform both the academic and industrial state-of-the-art. We believe these methods bring the field closer to having fully automated solutions for important challenges for the cinema industry that are currently solved manually or sub-optimally. Another contribution of our research is to highlight the limitations of existing image quality metrics when applied to the tone mapping problem, as none of them, including two state-of-the-art deep learning metrics for image perception, are able to predict the preferences of the observers.

A Perception-Based Inverse Tone Mapping Operator for High Dynamic Range Video Applications

The drastic visual improvements introduced by High Dynamic Range (HDR) technologies open new markets for a wide range of industries. Among them, a significant opportunity is offered to owners of legacy Standard Dynamic Range (SDR) content that can be converted to the new standard to take advantage of the enhanced capabilities of the HDR displays. Similarly, since SDR broadcasting infrastructure will continue to be around for the time being, such conversion process is becoming an obvious necessity. To this end, different approaches have tried to efficiently convert SDR images and videos to HDR format, a procedure well known as inverse Tone Mapping. In this paper, we propose a novel high visual quality video inverse Tone Mapping Operator (iTMO) that addresses the inadequacies of the state-of-the-art methods, resulting in high visual quality HDR videos that match the capabilities of the HDR technology. Our approach is based on human visual perception and employs a segmentation method according to the Human Visual System (HVS) sensitivity to brightness changes in different regions of the frame and constructs the mapping curve using the brightness distribution information of these regions. Our iTMO uses a hybrid approach to achieve an optimal balance between the overall contrast and brightness of the output HDR frame by maximizing a weighted sum of contrast and brightness difference between input SDR and generated HDR frame. The proposed iTMO works equally well for all levels of brightness, eliminating any visual artifacts by dynamically maintaining changes at non-perceivable levels. Subjective and objective evaluations validated the superior visual performance of our proposed iTMO over state-of-the-art methods.

Learning to Generate Multi-Exposure Stacks With Cycle Consistency for High Dynamic Range Imaging

Inverse tone mapping aims at recovering the lost scene radiances from a single exposure image. With the successful use of deep learning in numerous applications, many inverse tone mapping methods use convolution neural networks in a supervised manner. As these approaches are trained with many pre-fixed high dynamic range (HDR) images, they fail to flexibly expand the dynamic ranges of images. To overcome this limitation, we consider a multiple exposure image synthesis approach for HDR imaging. In particular, we propose a pair of neural networks that represent mappings between images that have exposure levels one unit apart (stop-up/down network). Therefore, it is possible to construct two positive-feedback systems to generate images with greater or lesser exposure. Compared to previous works using the conditional generative adversarial learning framework, the stop-up/down network employs HDR friendly network structures and several techniques to stabilize the training processes. Experiments on HDR datasets demonstrate the advantages of the proposed method compared to conventional methods. Consequently, we apply our approach to restore the full dynamic range of scenes agilely with only two networks and generate photorealistic images in complex lighting situations.

JSI-GAN: GAN-Based Joint Super-Resolution and Inverse Tone-Mapping with Pixel-Wise Task-Specific Filters for UHD HDR Video

Joint learning of super-resolution (SR) and inverse tone-mapping (ITM) has been explored recently, to convert legacy low resolution (LR) standard dynamic range (SDR) videos to high resolution (HR) high dynamic range (HDR) videos for the growing need of UHD HDR TV/broadcasting applications. However, previous CNN-based methods directly reconstruct the HR HDR frames from LR SDR frames, and are only trained with a simple L2 loss. In this paper, we take a divide-and-conquer approach in designing a novel GAN-based joint SR-ITM network, called JSI-GAN, which is composed of three task-specific subnets: an image reconstruction subnet, a detail restoration (DR) subnet and a local contrast enhancement (LCE) subnet. We delicately design these subnets so that they are appropriately trained for the intended purpose, learning a pair of pixel-wise 1D separable filters via the DR subnet for detail restoration and a pixel-wise 2D local filter by the LCE subnet for contrast enhancement. Moreover, to train the JSI-GAN effectively, we propose a novel detail GAN loss alongside the conventional GAN loss, which helps enhancing both local details and contrasts to reconstruct high quality HR HDR results. When all subnets are jointly trained well, the predicted HR HDR results of higher quality are obtained with at least 0.41 dB gain in PSNR over those generated by the previous methods. The official Tensorflow code is available at https://github.com/JihyongOh/JSI-GAN.

Dynamic Range Expansion Using Cumulative Histogram Learning for High Dynamic Range Image Generation

In modern digital photographs, most images have low dynamic range (LDR) formats, which means that the range of light intensities from the darkest to the brightest is much lower than the range that can be perceived by the human eye. Therefore, to visualize images as naturally as possible on devices that display them in high dynamic range (HDR) format, the LDR images need to be converted into HDR images. The aim of this study was to develop an adaptive inverse tone mapping operator (iTMO) that can convert a single LDR image into a realistic HDR image based on artificial neural networks. In contrast to conventional iTMO algorithms, our technique was developed by learning the complicated relationship between various LDR–HDR pair images, which enabled nearly ground-truth HDR images to be generated from various types of LDR images. The novel learning technique is called cumulative histogram-based learning and color difference learning. The superior performance of our technique over conventional methods was assessed through objective evaluations of various types of LDR and HDR images.

Repeated Look-up Tables.

Efficient hardware implementations routinely approximate mathematical functions with look-up tables, while keeping the error of the approximation under control. For a certain class of commonly occurring 1D functions, namely monotonically increasing or decreasing functions, we found that it is possible to approximate such functions by repeated application of a very low resolution 1D look-up table. There are many advantages to cascading multiple identical LUTs, including the promise of a very simple hardware design and the use of standard linear interpolation. Further, the complexity associated with unequal bin sizes can be avoided. We show that for realistic applications, including gamma correction, high dynamic range encoding and decoding curves, as well as tone mapping and inverse tone mapping applications, multiple cascaded look-up tables can reduce the approximation error by more than 50% compared to a single look-up table with the same total memory footprint.

An optimization framework for inverse tone mapping using a single low dynamic range image

Conventional inverse tone-mapping (ITM) methods tend to produce contrast distortions such as contrast loss and contrast reversal in reconstructed high dynamic range (HDR) images. This paper proposes a novel ITM optimization framework based on the assumption that the input low dynamic range (LDR) image is similar to the LDR image obtained by tone mapping a true HDR image. In the proposed framework, an HDR image is initially reconstructed by applying a conventional tone-mapping function in a reverse manner, and then the reconstructed HDR image is iteratively modified toward the optimum HDR image by minimizing the difference between the input LDR image and a tone-mapped LDR image obtained from the reconstructed HDR image. The experimental results demonstrate that the proposed framework effectively reconstructs a high-quality HDR image and outperforms other conventional methods in terms of objective quality.

ITM-Net: Deep Inverse Tone Mapping Using Novel Loss Function Considering Tone Mapping Operator

In this paper, we propose a novel inverse tone mapping network, called “iTM-Net.” For training iTM-Net, we also propose a novel loss function that considers the non-linear relation between low dynamic range (LDR) and high dynamic range (HDR) images. For inverse tone mapping with convolutional neural networks (CNNs), we first point out that training CNNs with a standard loss function causes a problem due to the non-linear relation between the LDR and HDR images. To overcome the problem, the novel loss function non-linearly tone-maps target HDR images into LDR ones on the basis of a tone mapping operator, and the distance between the tone-mapped images and predicted ones are then calculated. The proposed loss function enables us not only to normalize the HDR images but also to reduce the non-linear relation between LDR and HDR ones. The experimental results show that the HDR images predicted by the proposed iTM-Net have higher-quality than the HDR ones predicted by conventional inverse tone mapping methods, including the state of the art, in terms of both HDR-VDP-2.2 and PU encoding + MS-SSIM. In addition, compared with loss functions that do not consider the non-linear relation, the proposed loss function is shown to improve the performance of CNNs.

Deep Inverse Tone Mapping for Compressed Images

Converting a single low dynamic range (LDR) image into a high dynamic range (HDR) image, which is the so-called inverse tone mapping (ITM), is a challenging ill-posed problem since a lot of information is lost during compression and storage. Traditional ITM techniques mainly focus on high-quality LDR images without compression artifacts. However, in practice, LDR images are usually stored as a lossy compression format for the convenience of transmission, which will cause artifacts, i.e., blocking and ringing. Hence, these ITM methods suffer from severe performance drop for some real-world applications. In this paper, we propose a novel decomposition-based network to reconstruct HDR images from compressed LDR images; in other words, the proposed network can simultaneously remove blocking/ringing artifacts and recover high-quality HDR information. Considering compression artifacts mainly embedded in high frequency part, we thus decompose the input image into the high-frequency component (also known as detail layer) and low-frequency component (also known as a base layer). The detail layer mainly contains the information of texture, noise, and artifacts, while the base layer contains the information of structure and large object. Based on this, we design two subnetworks, i.e., detail layer recovery subnetwork and base layer recovery subnetwork, to restore the two parts separately. The detail layer recovery subnetwork is responsible for the artifacts removal with texture preserving, while the base layer recovery subnetwork is designed for tone expansion and overexposed/underexposed region restoration. Finally, in order to further reconstruct the serious overexposed regions, we adapt a merge subnetwork to fuse the result from the previous two subnetworks. The experimental results on compressed images demonstrate that the proposed method significantly outperforms other state-of-the-art methods.

Illumination consistency based on single low dynamic range images

In this paper, a light consistency solution for generating high dynamic range (HDR) images based on a single low dynamic range image(LDR) is proposed, and the virtual object is rendered by illumination. The solution can reduce the time of image acquisition and processing, and solve the problems caused by the limitations of image acquisition equipment. The solution is divided into three stages: image preprocessing, high dynamic range image generation and virtual object relighting. Firstly, in the stage of image pretreatment, the wavelet noise reduction method based on a Gaussian mixture model is used to remove image noise and avoid image detail distortion. The inverse camera response function is utilized to linearize the image, the pixel brightness range is expanded based on the inverse tone mapping function, and the threshold segmentation method is combined with flooding Gaussian smoothing to calculate the highlight spread diagram to compensate for scene highlights lost during camera shooting. Then, the extended dynamic range image is interpolated linearly by using the specular expansion image to get the high dynamic range image. Based on the analysis and experimental simulation, compared with other methods, it is found that using a single low-dynamic-range image can greatly reduce the time of image acquisition and processing and reduce the limitations of image acquisition equipment, while maintaining good light fusion. Based on the simulation results, the efficiency of light consistency processing is improved.

Perceptual brightness-based inverse tone mapping for high dynamic range imaging

This paper proposes a novel high dynamic range imaging method based on inverse tone mapping for generating a high dynamic range image from a single low dynamic range image. With the increasing expectations for image quality, displays providing high dynamic range images have been released. However, although low dynamic range images are currently commonly used, they do not use the high representation capability of high dynamic range displays. Moreover, high dynamic range contents are scarce. Therefore, high dynamic range imaging techniques are gaining attention. The proposed method involves adaptive specular region detection to determine the bright and dark areas for high dynamic range imaging, output mapping range decision to consider perceptual brightness, final mapping-level decision to prevent degradation of perceptual image quality, and inverse pixel mapping for luminance mapping. The simulation results of the proposed method yielded an HDR-VDP-2 quality score of 59.51, which was higher than those of the conventional methods. Furthermore, the execution time of the proposed algorithm was 0.66 s, which had the lowest complexity compared to the conventional methods. For subjective performance evaluation, we analyzed the perceived image quality on the display with the DRIM, which confirmed the superiority of the proposed method over the conventional methods.

Fine-grained detection of inverse tone mapping in HDR images

Abstract High dynamic range (HDR) imaging enables to capture the full range of physical luminance of a real-world scene, and is expected to progressively replace traditional low dynamic range (LDR) pictures and videos. Despite the increasing HDR popularity, very little attention has been devoted to new forensic problems that are characteristic to this content. In this paper, we address for the first time such kind of problem, by identifying the source of an HDR picture. Specifically, we consider the two currently most common techniques to generate an HDR image: by fusing multiple LDR images with different exposure time, or by inverse tone mapping an LDR picture. We show that, in order to apply conventional forensic tools to HDR images, they need to be properly preprocessed, and we propose and evaluate a few simple HDR forensic preprocessing strategies for this purpose. In addition, we propose a new forensic feature based on Fisher scores, calculated under Gaussian mixture models. We show that the proposed feature outperforms the popular SPAM features in classifying the HDR image source on image blocks as small as 3 × 3, which makes our method suitable to detect composite forgeries combining HDR patches originating from different acquisition processes.