Standard Dynamic Range Images Research Articles

High-Dynamic-Range Tone Mapping in Intelligent Automotive Systems

Intelligent driver assistance systems are becoming increasingly popular in modern passenger vehicles. A crucial component of intelligent vehicles is the ability to detect vulnerable road users (VRUs) for an early and safe response. However, standard imaging sensors perform poorly in conditions of strong illumination contrast, such as approaching a tunnel or at night, due to their dynamic range limitations. In this paper, we focus on the use of high-dynamic-range (HDR) imaging sensors in vehicle perception systems and the subsequent need for tone mapping of the acquired data into a standard 8-bit representation. To our knowledge, no previous studies have evaluated the impact of tone mapping on object detection performance. We investigate the potential for optimizing HDR tone mapping to achieve a natural image appearance while facilitating object detection of state-of-the-art detectors designed for standard dynamic range (SDR) images. Our proposed approach relies on a lightweight convolutional neural network (CNN) that tone maps HDR video frames into a standard 8-bit representation. We introduce a novel training approach called detection-informed tone mapping (DI-TM) and evaluate its performance with respect to its effectiveness and robustness in various scene conditions, as well as its performance relative to an existing state-of-the-art tone mapping method. The results show that the proposed DI-TM method achieves the best results in terms of detection performance metrics in challenging dynamic range conditions, while both methods perform well in typical, non-challenging conditions. In challenging conditions, our method improves the detection F2 score by 13%. Compared to SDR images, the increase in F2 score is 49%.

HDR-LFNet: Inverse tone mapping using fusion network

To capture the real-world luminance values, High Dynamic Range (HDR) image processing has been developed. HDR images have a richer content than the widely-used Standard Dynamic Range (SDR) images, and are used in a number of situations, e.g. in film industry. As HDR displays are more and more commercially available, we need to be able to process HDR images as well as SDR ones (for example, devising denoising algorithms, inpainting or anti-aliasing). The most powerful methods to process images are now deep neural networks. However, the training of such networks calls for a lot of images, and HDR images datasets are relatively small.One way to generate HDR images is inverse tone mapping operators (iTMOs). They are algorithms that expand the dynamic range of SDR images. In this paper, we propose HDR-LFNet, a novel iTMO, and its HDR training dataset. Our method merges several existing handcrafted iTMOs, combined in a supervised neural network to produce an HDR output. Our lightweight network requires less training images than state-of-the-art methods, and has faster training phase. Besides, the quality of the generated images is similar to the state-of-the-art. We present the architecture as well as the subjective and experimental evaluations of our method.

Deep HDR estimation with generative detail reconstruction

AbstractWe study the problem of High Dynamic Range (HDR) image reconstruction from a Standard Dynamic Range (SDR) input with potential clipping artifacts. Instead of building a direct model that maps from SDR to HDR images as in previous work, we decompose an input SDR image into a base (low frequency) and detail layer (high frequency), and treat reconstructing these two layers as two separate problems. We propose a novel architecture that comprises individual components specially designed to handle both tasks. Specifically, our base layer reconstruction component recovers low frequency content and remaps the color gamut of the input SDR, whereas our detail layer reconstruction component, which builds upon prior work on image inpainting, hallucinates missing texture information. The output HDR prediction is produced by a final refinement stage. We present qualitative and quantitative comparisons with existing techniques where our method achieves state‐of‐the‐art performance.

Deep Tone-Mapping Operator Using Image Quality Assessment Inspired Semi-Supervised Learning

Tone-mapping operator (TMO) is intended to convert high dynamic range (HDR) content into a lower dynamic range so that it can be displayed on a standard dynamic range (SDR) device. The tone-mapped result of HDR content is usually stored as SDR image. For different HDR scenes, traditional TMOs are able to obtain a satisfying SDR image only under manually fine-tuned parameters. In this paper, we address this problem by proposing a learning-based TMO using deep convolutional neural network (CNN). We explore different CNN structure and adopt multi-scale and multi-branch fully convolutional design. When training deep CNN, we introduce image quality assessments (IQA), specifically, tone-mapped image quality assessment, and implement it as semi-supervised loss terms. We discuss and prove the effectiveness of semi-supervised loss terms, CNN structure, data pre-processing, etc. by several experiments. Finally, we demonstrate that our approach can produce appealing results under diversified HDR scenes.

Object Detection Under Challenging Lighting Conditions Using High Dynamic Range Imagery

Most Convolution Neural Network (CNN) based object detectors, to date, have been optimized for accuracy and/or detection performance on datasets typically comprised of well exposed 8-bits/pixel/channel Standard Dynamic Range (SDR) images. A major existing challenge in this area is to accurately detect objects under extreme/difficult lighting conditions as SDR image trained detectors fail to accurately detect objects under such challenging lighting conditions. In this paper, we address this issue for the first time by introducing High Dynamic Range (HDR) imaging to object detection. HDR imagery can capture and process ≈13 orders of magnitude of scene dynamic range similar to the human eye. HDR trained models are therefore able to extract more salient features from extreme lighting conditions leading to more accurate detections. However, introducing HDR also presents multiple new challenges such as the complete absence of resources and previous literature on such an approach. Here, we introduce a methodology to generate a large scale annotated HDR dataset from any existing SDR dataset and validate the quality of the generated dataset via a robust evaluation technique. We also discuss the challenges of training and validating HDR trained models using existing detectors. Finally, we provide a methodology to create an out of distribution (OOD) HDR dataset to test and compare the performance of HDR and SDR trained detectors under difficult lighting condition. Results suggest that using the proposed methodology, HDR trained models are able to achieve 10 – 12% more accuracy compared to SDR trained models on real-world OOD dataset consisting of high-contrast images under extreme lighting conditions.

A Perception-Based Inverse Tone Mapping Operator for High Dynamic Range Video Applications

The drastic visual improvements introduced by High Dynamic Range (HDR) technologies open new markets for a wide range of industries. Among them, a significant opportunity is offered to owners of legacy Standard Dynamic Range (SDR) content that can be converted to the new standard to take advantage of the enhanced capabilities of the HDR displays. Similarly, since SDR broadcasting infrastructure will continue to be around for the time being, such conversion process is becoming an obvious necessity. To this end, different approaches have tried to efficiently convert SDR images and videos to HDR format, a procedure well known as inverse Tone Mapping. In this paper, we propose a novel high visual quality video inverse Tone Mapping Operator (iTMO) that addresses the inadequacies of the state-of-the-art methods, resulting in high visual quality HDR videos that match the capabilities of the HDR technology. Our approach is based on human visual perception and employs a segmentation method according to the Human Visual System (HVS) sensitivity to brightness changes in different regions of the frame and constructs the mapping curve using the brightness distribution information of these regions. Our iTMO uses a hybrid approach to achieve an optimal balance between the overall contrast and brightness of the output HDR frame by maximizing a weighted sum of contrast and brightness difference between input SDR and generated HDR frame. The proposed iTMO works equally well for all levels of brightness, eliminating any visual artifacts by dynamically maintaining changes at non-perceivable levels. Subjective and objective evaluations validated the superior visual performance of our proposed iTMO over state-of-the-art methods.

Study of naturalness in tone-mapped images

Nowadays, images can be obtained in various ways such as capturing photos in single-exposure mode, applying Multiple Exposure Fusion algorithms to generate an image from multiple shoots of the same scene, mapping High Dynamic Range (HDR) images to Standard Dynamic Range (SDR) images, converting raw formats to displayable formats, or applying post-processing techniques to enhance image quality, aesthetic quality,…When looking at some photos, one might have a feeling of unnaturalness. This paper deals with the problem of developing a model firstly to estimate if an image looks natural or not to humans and the second purpose is to try to understand how the unnaturalness feeling is induced by a photo: Are there specific unnaturalness clues or is unnaturalness a general feeling when looking at a photo? The study focuses on SDR images, especially on tone-mapped images. The first contribution of the paper is the setting of an experiment gathering human naturalness opinions on 1900 SDR images mainly obtained from tone mapping operators. Based on the collected data, the second contribution is to study the efficiency of different feature types including handcrafted features and learned features for image naturalness analysis. A binary classification model is then developed based on the determined features to classify if an image looks natural or unnatural.

Referenceless Quality Evaluation of Tone-Mapped HDR and Multiexposure Fused Images

Nowadays, the standard dynamic range (SDR) image acquired at a fixed exposure exposes weakness in portraying fine-grained details of real scenes. The high dynamic range (HDR) image and other types of SDR images generated by multiexposure fusion techniques provide us new choices for scene representation. To display on SDR screens, an HDR image must be tone-mapped to an SDR one. Since different tone-mapping/fusion algorithms produce images with varying visual quality levels, it naturally desires a quality evaluation model for comparison. This article proposes an effective model in the absence of the reference image. By analyzing the characteristics of tone-mapped HDR and multiexposure fused images, we first extract multiple quality-sensitive features from the following aspects: 1) colorfulness; 2) exposure; and 3) naturalness. Then, the model is built by bridging all extracted features and associated subjective ratings via support vector regression. Extensive experiments on publicly available databases prove the superiority of our model over the state-of-the-art referenceless quality evaluation ones.

Learning-Based Low-Complexity Reverse Tone Mapping With Linear Mapping

Although high dynamic range (HDR) display has become popular recently, the legacy content such as standard dynamic range (SDR) video is still in service and needs to be properly converted on HDR display devices. Therefore, it is desirable for HDR TV sets to have the capability of automatically converting input SDR video into HDR video, which is called reverse tone mapping (RTM). In this paper, we propose a novel learning-based low-complexity RTM scheme that not only expands the suppressed dynamic ranges (DR) of the SDR videos (or images), but also effectively restores lost detail in the SDR videos. Most existing conventional RTM schemes have focused on how to expand the DR of global contrast, resulting in limitations in recovering lost detail of SDR videos. On the other hand, the recent convolutional neural network-based approaches show promising results, but they are too complex to be applied on the users’ devices in practice. In this paper, our learning-based RTM scheme is computationally simple but effective in recovering lost detail. To learn the SDR-to-HDR relation, training “SDR-HDR” images are first separated into their base layer components and detail layer components by applying a guided filter. The detail layer components of the “SDR-HDR” pairs are used to train the SDR-to-HDR mapping. The mapping matrices are computed based on kernel ridge regression. In the meantime, the global contrast of the base layers is expanded by a nonlinear function that suppresses darker regions and amplifies brighter regions to fit the full DR of a target HDR display. To verify the effectiveness of our learning-based RTM scheme, we performed subjective quality assessment for images and videos. The experimental results show that our RTM scheme outperforms the existing RTM scheme with the successful restoration of lost detail in SDR images.

A New Dataset for Source Identification of High Dynamic Range Images.

Digital source identification is one of the most important problems in the field of multimedia forensics. While Standard Dynamic Range (SDR) images are commonly analyzed, High Dynamic Range (HDR) images are a less common research subject, which leaves space for further analysis. In this paper, we present a novel database of HDR and SDR images captured in different conditions, including various capturing motions, scenes and devices. As a possible application of this dataset, the performance of the well-known reference pattern noise-based source identification algorithm was tested on both kinds of images. Results have shown difficulties in source identification conducted on HDR images, due to their complexity and wider dynamic range. It is concluded that capturing conditions and devices themselves can have an impact on source identification, thus leaving space for more research in this field.

Large-Scale Crowdsourced Study for Tone-Mapped HDR Pictures.

Measuring digital picture quality, as perceived by human observers, is increasingly important in many applications in which humans are the ultimate consumers of visual information. Standard dynamic range (SDR) images provide 8 b/color/pixel. High dynamic range (HDR) images, usually created from multiple exposures of the same scene, can provide 16 or 32 b/color/pixel, but need to be tonemapped to SDR for display on standard monitors. Multiexposure fusion (MEF) techniques bypass HDR creation by fusing an exposure stack directly to SDR images to achieve aesthetically pleasing luminance and color distributions. Many HDR and MEF databases have a relatively small number of images and human opinion scores, obtained under stringently controlled conditions, thereby limiting realistic viewing. Moreover, many of these databases are intended to compare tone-mapping algorithms, rather than being specialized for developing and comparing image quality assessment models. To overcome these challenges, we conducted a massively crowdsourced online subjective study. The primary contributions described in this paper are: 1) the new ESPL-LIVE HDR Image Database that we created containing diverse images obtained by tone-mapping operators and MEF algorithms, with and without post-processing; 2) a large-scale subjective study that we conducted using a crowdsourced platform to gather more than 300 000 opinion scores on 1811 images from over 5000 unique observers; and 3) a detailed study of the correlation performance of the state-of-the-art no-reference image quality assessment algorithms against human opinion scores of these images. The database is available at http://signal.ece.utexas.edu/%7Edebarati/HDRDatabase.zip.

P‐216L: Late‐News Poster: Reproduction of Perceptual Reality in Standard‐Dynamic‐Range (SDR) Environments Using High‐Dynamic‐Range (HDR) Images Compressed by Global Tone Mapping

We conduct subjective experiments in real HDR environments. The results suggest that (1) most observers could perceive correct luminance correlations by global adaptation rather than local adaptation, but (2) Weber‐ratios were increased. Our solution is to estimate global tone mapping curves in accordance with the above results. The resulting SDR images offer perceptual reality.

No-Reference Quality Assessment of Tone-Mapped HDR Pictures.

Being able to automatically predict digital picture quality, as perceived by human observers, has become important in many applications where humans are the ultimate consumers of displayed visual information. Standard dynamic range (SDR) images provide 8 b/color/pixel. High dynamic range (HDR) images, which are usually created from multiple exposures of the same scene, can provide 16 or 32 b/color/pixel, but must be tonemapped to SDR for display on standard monitors. Multi-exposure fusion techniques bypass HDR creation, by fusing the exposure stack directly to SDR format while aiming for aesthetically pleasing luminance and color distributions. Here, we describe a new no-reference image quality assessment (NR IQA) model for HDR pictures that is based on standard measurements of the bandpass and on newly conceived differential natural scene statistics (NSS) of HDR pictures. We derive an algorithm from the model which we call the HDR IMAGE GRADient-based Evaluator. NSS models have previously been used to devise NR IQA models that effectively predict the subjective quality of SDR images, but they perform significantly worse on tonemapped HDR content. Toward ameliorating this we make here the following contributions: 1) we design HDR picture NR IQA models and algorithms using both standard space-domain NSS features as well as novel HDR-specific gradient-based features that significantly elevate prediction performance; 2) we validate the proposed models on a large-scale crowdsourced HDR image database; and 3) we demonstrate that the proposed models also perform well on legacy natural SDR images. The software is available at: http://live.ece.utexas.edu/research/Quality/higradeRelease.zip.

Evaluating the Color Fidelity of ITMOs and HDR Color Appearance Models

With the increasing availability of high-dynamic-range (HDR) displays comes the need to remaster existing content in a way that takes advantage of the extended range of luminance and contrast that such displays offer. At the same time, it is crucial that the creative intent of the director is preserved through such changes as much as possible. In this article, we compare several approaches for dynamic range extension to assess their ability to correctly reproduce the color appearance of standard dynamic range (SDR) images on HDR displays. A number of state-of-the-art inverse tone mapping operators (ITMOs) combined with a standard chromatic adaptation transform (CAT) as well as some HDR color appearance models have been evaluated through a psychophysical study, making use of an HDR display as well as HDR ground-truth data. We found that global ITMOs lead to the most reliable performance when combined with a standard CAT, while more complex methods were found to be more scene dependent, and often less preferred than the unprocessed SDR image. HDR color appearance models, albeit being the most complete solutions for accurate color reproduction, were found to not be well suited to the problem of dynamic range expansion, suggesting that further research may be necessary to provide accurate color management in the context of inverse tone mapping.

Probabilistic Exposure Fusion

The luminance of a natural scene is often of high dynamic range (HDR). In this paper, we propose a new scheme to handle HDR scenes by integrating locally adaptive scene detail capture and suppressing gradient reversals introduced by the local adaptation. The proposed scheme is novel for capturing an HDR scene by using a standard dynamic range (SDR) device and synthesizing an image suitable for SDR displays. In particular, we use an SDR capture device to record scene details (i.e., the visible contrasts and the scene gradients) in a series of SDR images with different exposure levels. Each SDR image responds to a fraction of the HDR and partially records scene details. With the captured SDR image series, we first calculate the image luminance levels, which maximize the visible contrasts, and then the scene gradients embedded in these images. Next, we synthesize an SDR image by using a probabilistic model that preserves the calculated image luminance levels and suppresses reversals in the image luminance gradients. The synthesized SDR image contains much more scene details than any of the captured SDR image. Moreover, the proposed scheme also functions as the tone mapping of an HDR image to the SDR image, and it is superior to both global and local tone mapping operators. This is because global operators fail to preserve visual details when the contrast ratio of a scene is large, whereas local operators often produce halos in the synthesized SDR image. The proposed scheme does not require any human interaction or parameter tuning for different scenes. Subjective evaluations have shown that it is preferred over a number of existing approaches.

FPGA implementation of a real-time biologically inspired image enhancement algorithm

This paper presents an FPGA implementation of a novel image enhancement algorithm, which compensates for the under-/over-exposed image regions, caused by the limited dynamic range of contemporary standard dynamic range image sensors. The algorithm, which is motivated by the attributes of the shunting center-surround cells of the human visual system, is implemented in Altera Stratix II GX: EP2SGX130GF1508C5 FPGA device. The proposed implementation, which is synthesized in an FPGA technology, employs reconfigurable pipeline, structured memory management, and data reuse in spatial operations, to render in real-time the huge amount of input data that the video signal comprises. It also avoids the use of computationally intensive operations, achieving the required specifications in terms of flexibility, timing, performance and visual quality. The proposed implementation allows real-time processing of color images with sizes up to 2.5 Mpixels, at frame rate of 25 fps. As a result, the architectural solution described in this work offers a low-cost implementation for automatic exposure correction in real-time video systems.

Fast centre–surround contrast modification

A new algorithm for fast contrast modification of standard dynamic range (SDR) images (8 bits/ channel) is presented. Its thrust is to enhance the contrast in the under-/over-exposed regions of SDR images, caused by the low dynamic range of the capturing device. It is motivated by the attributes of the shunting centre - surround cells of the human visual system. The main advantage of the proposed algorithm is its O(N ) complexity which results in very fast execution, even when executed on a conventional personal computer (0.2 s/frame for a 640 � 480 pixel resolution on a 3 GHz Pentium 4). Thus, it moderately increases the computational burden if it is used as a pre-processing stage for other image processing algorithms. The proposed method is compared with other established algorithms, which can enhance the contrast in the under-/over-exposed regions of SDR images: the multi-scale Retinex with colour rendition, the McCann Retinex (McCann99), the rational mapping function and the automatic colour equalisation. The results obtained by this comparison indicate that the proposed algorithm exhibits at least comparable results in contrast modification tasks to the other algorithms, in significantly reduced execution times.