High Dynamic Range Image Compression Research Articles

Weighted side-window based gradient guided image filtering

Image filtering under guidance image, known as guided filtering (GF), has been successfully applied to a variety of applications. Existing GF methods utilize either conventional full window-based framework (FWF) or simple uniformly weighted aggregation strategy (UWA); thereby they suffer from edge-blurring. In this paper, based upon gradient guided filtering (GGF), a weighted side-window based gradient guided filtering (WSGGF) is proposed to address the aforementioned problem. First, both regression and adaptive regularization terms in GGF are improved upon eight side windows by introducing side window-based framework (SWF). L1 norm is adopted to choose the results calculated in side windows. Second, UWA strategy in GGF is replaced by a refined variance-based weighted average (VWA) aggregation. In VWA, the value of each weight is chosen inversely proportional to the corresponding estimator. We show that with these improvements our method can well retain the edge sharpness and is robust to visual artifacts. To cut down the time consumption, a fast version of WSGGF (FWSGGF) is further proposed by incorporating a simple but effective down-sampling strategy, which is about four times faster while maintaining the superior performance. By comparing with the state-of-the-art (SOTA) methods on edge-aware smoothing, detail enhancement, high dynamic range image (HDR) compression, image luminance adjustment, depth map upsampling and single image haze removal, the effectiveness and flexibility of our proposed methods are verified. The source code is available at: https://github.com/weimin581/WSGGF

Intrinsic Image Transfer for Illumination Manipulation.

This paper presents a novel intrinsic image transfer (IIT) algorithm for image illumination manipulation, which creates a local image translation between two illumination surfaces. This model is built on an optimization-based framework composed of illumination, reflectance and content photo-realistic losses, respectively. Each loss is firstly defined on the corresponding sub-layers factorized by an intrinsic image decomposition and then reduced under the well-known spatial-varying illumination illumination-invariant reflectance prior knowledge. We illustrate that all losses, with the aid of an "exemplar" image, can be directly defined on images without the necessity of taking an intrinsic image decomposition, thereby giving a closed-form solution to image illumination manipulation. We also demonstrate its versatility and benefits to several illumination-related tasks: illumination compensation, image enhancement and tone mapping, and high dynamic range (HDR) image compression, and show their high-quality results on natural image datasets.

High dynamic range image compression based on the multi-peak S-shaped tone curve.

Tone mapping methods aim to compress the high dynamic range (HDR) images so that they can be displayed on common devices. The tone curve plays a key role in many tone mapping methods, which can directly adjust the range of the HDR image. The S-shaped tone curves can produce impressive performances due to their flexibility. However, the conventional S-shaped tone curve in tone mapping methods is single and had the problem of excessive compressing of the dense grayscale areas, resulting in the loss of details in this area, and insufficient compressing of the sparse grayscale areas, resulting in low contrast of tone mapped image. This paper proposes a multi-peak S-shaped (MPS) tone curve to address these problems. Specifically, the grayscale interval of the HDR image is divided according to the significant peak and valley distribution of the grayscale histogram, and each interval is tone mapped by an S-shaped tone curve. We further propose an adaptive S-shaped tone curve based on the luminance adaptation mechanism of the human visual system, which can effectively reduce the compression in the dense grayscale areas and increase the compression in the sparse grayscale areas, preserving details while improving the contrast of tone mapped images. Experiments show that our MPS tone curve replaces the single S-shaped tone curve in relevant methods for better performance and outperforms the state-of-the-art tone mapping methods.

OoDHDR-Codec: Out-of-Distribution Generalization for HDR Image Compression

Recently, deep learning has been proven to be a promising approach in standard dynamic range (SDR) image compression. However, due to the wide luminance distribution of high dynamic range (HDR) images and the lack of large standard datasets, developing a deep model for HDR image compression is much more challenging. To tackle this issue, we view HDR data as distributional shifts of SDR data and the HDR image compression can be modeled as an out-of-distribution generalization (OoD) problem. Herein, we propose a novel out-of-distribution (OoD) HDR image compression framework (OoDHDR-codec). It learns the general representation across HDR and SDR environments, and allows the model to be trained effectively using a large set of SDR datases supplemented with much fewer HDR samples. Specifically, OoDHDR-codec consists of two branches to process the data from two environments. The SDR branch is a standard blackbox network. For the HDR branch, we develop a hybrid system that models luminance masking and tone mapping with white-box modules and performs content compression with black-box neural networks. To improve the generalization from SDR training data on HDR data, we introduce an invariance regularization term to learn the common representation for both SDR and HDR compression. Extensive experimental results show that the OoDHDR codec achieves strong competitive in-distribution performance and state-of-the-art OoD performance. To the best of our knowledge, our proposed approach is the first work to model HDR compression as OoD generalization problems and our OoD generalization algorithmic framework can be applied to any deep compression model in addition to the network architectural choice demonstrated in the paper. Code available at https://github.com/caolinfeng/OoDHDR-codec.

LOCAL FEATURE EXTRACTION IN IMAGES

The methods of the local feature point extraction are analyzed. The analysis shows that the most effective detectors are based on the brightness gradient determination. They usually use the Harris angle detector, which is complex in calcu­la­tions. The algorithm complexity minimization contradicts both the detector effective­ness and to the high dynamic range of the analyzed image. As a result, the high-speed methods could not recognize the feature points in the heavy luminance conditions. The modification of the high dynamic range (HDR) image compression algorithm based on the Retinex method is proposed. It contains an adaptive filter, which preserves the image edges. The filter is based on a set of feature detectors perfor­ming the Harris-Laplace transform which is much simpler than the Harris angle detector. A prototype of the HDR video camera is designed which provides sharp images. Its structure simplifies the design of the artificial intelligence engine, which is implemented in FPGA of medium or large size.

High dynamic range image compression based on visual saliency

Abstract With wider luminance range than conventional low dynamic range (LDR) images, high dynamic range (HDR) images are more consistent with human visual system (HVS). Recently, JPEG committee releases a new HDR image compression standard JPEG XT. It decomposes an input HDR image into base layer and extension layer. The base layer code stream provides JPEG (ISO/IEC 10918) backward compatibility, while the extension layer code stream helps to reconstruct the original HDR image. However, this method does not make full use of HVS, causing waste of bits on imperceptible regions to human eyes. In this paper, a visual saliency-based HDR image compression scheme is proposed. The saliency map of tone mapped HDR image is first extracted, then it is used to guide the encoding of extension layer. The compression quality is adaptive to the saliency of the coding region of the image. Extensive experimental results show that our method outperforms JPEG XT profile A, B, C and other state-of-the-art methods. Moreover, our proposed method offers the JPEG compatibility at the same time.

An extensive performance evaluation of full-reference HDR image quality metrics

High dynamic range (HDR) image and video technology has recently attracted a great deal of attention in the multimedia community, as a mean to produce truly realistic video and further improve the quality of experience (QoE) of emerging multimedia services. In this context, measuring the quality of compressed HDR content plays a fundamental role. However, full-reference (FR) HDR visual quality assessment poses new challenges with respect to the conventional low dynamic range case. Quality metrics have to be redesigned or adapted to HDR, and understanding their reliability to predict users’ judgments is even more critical due to the still limited availability of HDR displays to perform subjective evaluations. The goal of this paper is to provide a complete and thorough survey of the performance of the most popular HDR FR image quality metrics. To this end, we gather several existing HDR image databases with subjective quality annotations, in addition to a new one created by ourselves. After aligning the scores in these databases, we obtain an extensive set of 690 compressed HDR images, along with their subjective quality. Next, we analyze in depth many FR metrics, including those used in MPEG standardization, using both classical correlation analyses and classification accuracy. We believe that our results could serve as the most complete and comprehensive benchmark of image quality metrics in the field of HDR image compression.

High dynamic range image compression with improved logarithmic transformation

High dynamic range image compression with improved logarithmic transformation

Evaluation of JPEG XT for high dynamic range cameras

Modern cameras can be used to capture high dynamic range (HDR) images by taking multiple low dynamic range (LDR) pictures at different exposure settings. However, DSLR cameras often store the multiple exposure pictures, which increases storage requirements proportionally to the number of exposures, while smartphones and tablets only store a tone-mapped representation of the HDR image instead of the original HDR data. In this paper, we investigate the bit rate reduction that could be achieved by storing HDR images compressed with JPEG XT, a JPEG backward-compatible HDR image compression standard, over multiple exposures JPEG. We also investigate the quality increase that could be achieved by fusing images with linear values and native sensor bit depth over pictures with sRGB gamma encoded values and 8 bits per sample. To investigate the impact of the different parameters on the performance, we perform a set of experiments with 14 contents, 4 resolutions (ranging from 2.5 megapixels (MP) to 16MP), 4 exposures configurations (2, 3, 5, and 7 exposures), 5 tone mapping operators (TMOs) used to generate the JPEG XT base layer, and 5 quality levels representative of values commonly used in smartphones and DSLR cameras. Results show that the performance is dependent on the TMO and number of exposures, with maximum average bit rate savings ranging from 10% for two exposures up to 70% for seven exposures. Additionally, quality improvements of 1.7dB to 5.1dB can be achieved when using the native image sensor data format.

High dynamic range image compression by optimizing tone mapped image quality index.

Tone mapping operators (TMOs) aim to compress high dynamic range (HDR) images to low dynamic range (LDR) ones so as to visualize HDR images on standard displays. Most existing TMOs were demonstrated on specific examples without being thoroughly evaluated using well-designed and subject-validated image quality assessment models. A recently proposed tone mapped image quality index (TMQI) made one of the first attempts on objective quality assessment of tone mapped images. Here, we propose a substantially different approach to design TMO. Instead of using any predefined systematic computational structure for tone mapping (such as analytic image transformations and/or explicit contrast/edge enhancement), we directly navigate in the space of all images, searching for the image that optimizes an improved TMQI. In particular, we first improve the two building blocks in TMQI—structural fidelity and statistical naturalness components—leading to a TMQI-II metric. We then propose an iterative algorithm that alternatively improves the structural fidelity and statistical naturalness of the resulting image. Numerical and subjective experiments demonstrate that the proposed algorithm consistently produces better quality tone mapped images even when the initial images of the iteration are created by the most competitive TMOs. Meanwhile, these results also validate the superiority of TMQI-II over TMQI.

Tone mapping-based high-dynamic-range image compression: study of optimization criterion and perceptual quality

We study the issue of quality assessment in tone mapping- based high-dynamic-range (HDR) image compression. In this, there are two stages at which a decision should be made regarding perceptual vis- ual quality: (a) for finding the optimal parameters of the dynamic range reduction function so that the visual quality is maximized, and (b) visual quality judgment of the decompressed image. We first investigate two objective optimization criteria, namely mean squared error and structural similarity index measure, toward optimization of a tone mapping model- based HDR image compression method. We then conduct a comprehen- sive subjective study to evaluate the visual quality of the compressed HDR images. Therefore, we consider both objective and subjective aspects for HDR image compression. To our knowledge, no systematic and compre- hensive studies exist in the current literature which shed light on the issue of quality assessment in HDR compression. So this study brings in new knowledge and perspective for the relatively less investigated topic of HDR compression from the view point of perceptual quality. We further evaluate the prediction performances of four objective methods on the 140 compressed HDR images that have been subjectively rated.

Methods for improving the tone mapping for backward compatible high dynamic range image and video coding

Backward compatibility for high dynamic range image and video compression forms one of the essential requirements in the transition phase from low dynamic range (LDR) displays to high dynamic range (HDR) displays. In a recent work [1], the problems of tone mapping and HDR video coding are originally fused together in the same mathematical framework, and an optimized solution for tone mapping is achieved in terms of the mean square error (MSE) of the logarithm of luminance values. In this paper, we improve this pioneer study in three aspects by considering its three shortcomings. First, the proposed method [1] works over the logarithms of luminance values which are not uniform with respect to Human Visual System (HVS) sensitivity. We propose to use the perceptually uniform luminance values as an alternative for the optimization of tone mapping curve. Second, the proposed method [1] does not take the quality of the resulting tone mapped images into account during the formulation in contrary to the main goal of tone mapping research. We include the LDR image quality as a constraint to the optimization problem and develop a generic methodology to compromise the trade-off between HDR and LDR image qualities for coding. Third, the proposed method [1] simply applies a low-pass filter to the generated tone curves for video frames to avoid flickering during the adaptation of the method to the video. We instead include an HVS based flickering constraint to the optimization and derive a methodology to compromise the trade-off between the rate-distortion performance and flickering distortion. The superiority of the proposed methodologies is verified with experiments on HDR images and video sequences.

Context-dependent JPEG backward-compatible high-dynamic range image compression

High-dynamic range (HDR) imaging is expected, together with ultrahigh definition and high-frame rate video, to become a technology that may change photo, TV, and film industries. Many cameras and displays capable of capturing and rendering both HDR images and video are already available in the market. The popularity and full-public adoption of HDR content is, however, hindered by the lack of standards in evalu- ation of quality, file formats, and compression, as well as large legacy base of low-dynamic range (LDR) displays that are unable to render HDR. To facilitate the wide spread of HDR usage, the backward compatibility of HDR with commonly used legacy technologies for storage, rendering, and compression of video and images are necessary. Although many tone-mapping algorithms are developed for generating viewable LDR con- tent from HDR, there is no consensus of which algorithm to use and under which conditions. We, via a series of subjective evaluations, demonstrate the dependency of the perceptual quality of the tone-mapped LDR images on the context: environmental factors, display parameters, and image con- tent itself. Based on the results of subjective tests, it proposes to extend JPEG file format, the most popular image format, in a backward compat- ible manner to deal with HDR images also. An architecture to achieve such backward compatibility with JPEG is proposed. A simple implementation of lossy compression demonstrates the efficiency of the proposed archi- tecture compared with the state-of-the-art HDR image compression. © 2013

Adaptive quantization using piecewise companding and scaling for Gaussian mixture

Quantization is fundamental to analog-to-digital converter (ADC) and signal compression. In this paper, we propose an adaptive quantizer with piecewise companding and scaling for signals of Gaussian mixture model (GMM). Our adaptive quantizer operates under three modes, each of which corresponds to different types of GMM. Moreover, we propose a reconfigurable architecture to implement our adaptive quantizer in an ADC. We also use it to quantize images and design the tone mapping algorithm for high dynamic range (HDR) image compression. Our experimental results show that (1) the proposed quantizer is able to achieve performance close to the optimal quantizer (i.e., Lloyd–Max quantizer for GMM) in the sense of mean squared error (MSE), at much lower computational cost than it; (2) the proposed quantizer is able to achieve much better MSE performance than a uniform quantizer, at a cost similar to the uniform quantizer. The proposed adaptive quantizer holds great potential in the appilcations of the existing ADC and HDR image compression.

Optimizing a Tone Curve for Backward-Compatible High Dynamic Range Image and Video Compression

For backward compatible high dynamic range (HDR) video compression, the HDR sequence is reconstructed by inverse tone-mapping a compressed low dynamic range (LDR) version of the original HDR content. In this paper, we show that the appropriate choice of a tone-mapping operator (TMO) can significantly improve the reconstructed HDR quality. We develop a statistical model that approximates the distortion resulting from the combined processes of tone-mapping and compression. Using this model, we formulate a numerical optimization problem to find the tone-curve that minimizes the expected mean square error (MSE) in the reconstructed HDR sequence. We also develop a simplified model that reduces the computational complexity of the optimization problem to a closed-form solution. Performance evaluations show that the proposed methods provide superior performance in terms of HDR MSE and SSIM compared to existing tone-mapping schemes. It is also shown that the LDR image quality resulting from the proposed methods matches that produced by perceptually-based TMOs.

HDR image compression and evaluation based on local adaptation using a retinal model

Abstract— The retinal adaptation process helps the human visual system to see high‐dynamic‐range (HDR) scenes in the real world. A simple static local adaptation method for HDR image compression based on a retinal model is presented. The proposed simple model aims at recreating the same sensations of the human visual system between the real‐world scene and the range compressed image on the display device when viewed after the human visual system reaches the steady local adaptation state, respectively. In computing scene local adaptation, the use of a non‐linear edge‐preserving bilateral filter not only presents a better tonal rendition in compressing local contrast and preserving details but also avoids banding artifacts across high‐gradient edges. Our new model relates the display adaptation with the scene adaptation based on the retinal model. In order to verify the effectiveness, a subjective evaluation is made by comparing the real scene and the display image using the paired comparison technique.

High Dynamic Range Image Compression by Fast Integrated Surround Retinex Model

A novel compressing method of high dynamic range image based on fast integrated surround Retinex model is proposed in this paper. The proposed method has two novelties. First, multiscale surround images are integrated to a single surround field, which is applied to center/surround single-scale Retinex (SSR) model. The method reduces the "banding artifact" seen in normal SSR and simplifies the complicated computational steps in conventional multiscale Retinex. Second, the Gaussian pyramid method is introduced to cut the computation time for generating a large-scale surround by tracing a "reduction" and "expansion" sequences using down and up sampling followed by linear interpolation. The computational expense is dramatically saved less than 1/100 for getting a surround by Gaussian convolution with large kernel size. The proposed model worked well in compressing the dynamic range and improving the visibility in heavy shadow areas of natural color images while preserving pleasing contrast.

HDR Image Compression by Local Adaptation for Scene and Display Using Retinal Model

Adaptation process of retina helps human visual system to see a high dynamic range scene in real world. This paper presents a simple static local adaptation method for high dynamic range image compression based on a retinal model. The proposed simple model aims at recreating the same sensations between the real scene and the range compressed image on display device when viewed after reaching steady state local adaptation respectively. Our new model takes the display adaptation into account in relation to the scene adaptation based on the retinal model. In computing local adaptation, the use of nonlinear edge preserving bilateral filter presents a better tonal rendition in preserving the local contrast and details while avoiding banding artifacts normally seen in local methods. Finally, we demonstrate the effectiveness of the proposed model by estimating the color difference between the recreated image and the target visual image obtained by trial and error method.