Local Tone Mapping Research Articles

LTM-NeRF: Embedding 3D Local Tone Mapping in HDR Neural Radiance Field.

Recent advances in Neural Radiance Fields (NeRF) have provided a new geometric primitive for novel view synthesis. High Dynamic Range NeRF (HDR NeRF) can render novel views with a higher dynamic range. However, effectively displaying the scene contents of HDR NeRF on diverse devices with limited dynamic range poses a significant challenge. To address this, we present LTM-NeRF, a method designed to recover HDR NeRF and support 3D local tone mapping. LTM-NeRF allows for the synthesis of HDR views, tone-mapped views, and LDR views under different exposure settings, using only the multi-view multi-exposure LDR inputs for supervision. Specifically, we propose a differentiable Camera Response Function (CRF) module for HDR NeRF reconstruction, globally mapping the scene's HDR radiance to LDR pixels. Moreover, we introduce a Neural Exposure Field (NeEF) to represent the spatially varying exposure time of an HDR NeRF to achieve 3D local tone mapping, for compatibility with various displays. Comprehensive experiments demonstrate that our method can not only synthesize HDR views and exposure-varying LDR views accurately but also render locally tone-mapped views naturally. For video results, please view our supplemental materials or visit the project page at: https://xhuangcv.github.io/LTM-NeRF/.

An Improved Inception Network to classify black tea appearance quality

To conveniently and precisely evaluate orthodox black tea appearance quality, we here present a novel method based on computer vision integrated image processing and deep learning. Tea images were collected using the custom-built image acquisition device and processed with Adaptive Local Tone Mapping (ALTM), and Unsharp Masking to optimize illumination and sharpness. These images were then used to train six convolutional neural networks (CNNs) to find suitable network structure by transfer learning. It was found that the CNNs constructed with the MBconv modules had better performance in this task. Consequently, a CNN classification model (Improved Inception Network) based on the MBconv modules was constructed and trained, which yielded a test accuracy of 95%, performed better than the other CNNs; the test accuracy of original Inception V3 was 89%. The proposed method obtained 97.22% accuracy for independent set in validation, which demonstrated the viability of applying image processing and deep learning approaches to solve practical problems in the field of tea assessment.

Impact of ISP Tuning on Object Detection.

In advanced driver assistance systems (ADAS) or autonomous vehicle research, acquiring semantic information about the surrounding environment generally relies heavily on camera-based object detection. Image signal processors (ISPs) in cameras are generally tuned for human perception. In most cases, ISP parameters are selected subjectively and the resulting image differs depending on the individual who tuned it. While the installation of cameras on cars started as a means of providing a view of the vehicle's environment to the driver, cameras are increasingly becoming part of safety-critical object detection systems for ADAS. Deep learning-based object detection has become prominent, but the effect of varying the ISP parameters has an unknown performance impact. In this study, we analyze the performance of 14 popular object detection models in the context of changes in the ISP parameters. We consider eight ISP blocks: demosaicing, gamma, denoising, edge enhancement, local tone mapping, saturation, contrast, and hue angle. We investigate two raw datasets, PASCALRAW and a custom raw dataset collected from an advanced driver assistance system (ADAS) perspective. We found that varying from a default ISP degrades the object detection performance and that the models differ in sensitivity to varying ISP parameters. Finally, we propose a novel methodology that increases object detection model robustness via ISP variation data augmentation.

New visual noise measurement on a versatile laboratory setup in HDR conditions for smartphone camera testing

Cameras, especially cameraphones, are using a large panel of technologies, such as multi-frame stacking and local tone mapping to capture and render scenes with high dynamic range. ISO defined charts for OECF estimation and visual noise measurement are not really designed for these specific use cases, especially when no manual control of the camera is available. Moreover, these charts are limited to one measurement. We developed a versatile laboratory setup to evaluate image quality attributes, like autofocus, exposure and details preservation. It is tested in various lighting conditions, with several dynamic ranges up to 7EV difference within the scene, under different illuminants. Latest visual noise measurements proposed by IEEE P1858 or ISO-15739 are not giving fully satisfactory results on our laboratory scene, due to differences in the chart, framing and lighting conditions used. We performed subjective visual experiments to build a quality ruler of noisy grey patches, and use it as a dataset to develop and validate an improved version of a visual noise measurement. In the experiments we also studied the impact of different environment conditions of the grey patches to assess their relevance to our algorithm. Our new visual noise measurement uses a luminance sensitivity function multiplied by the square root of the weighted sum of the variances of the Lab coordinates of the patches. A non-linear JND scaling is applied afterwards to get a visual noise measurement in units of JND of noisiness.

Learning Tone Curves for Local Image Enhancement

Image enhancement methods can be formulated as global transformations, local transformations, pixel-wise processing, or a mixture of these operations. Global transformations are limited in enhancing local image regions. Existing local and pixel-wise methods mitigate this issue, but give rise to the additional challenge of limited interpretability. Bridging the gap between global and local methods, we propose a local tone mapping network (LTMNet) that learns a grid of tone curves to locally enhance an image. Tone curves are commonly used by photo-editing software and offer an intuitive representation to photographers, facilitating subsequent customization of the image. Tone curves are also widely used in image signal processors (ISPs), making our method easy to deploy on cameras. Because existing datasets contain image enhancement and photofinishing beyond global and local tone mapping, we also propose a new dataset representative of local tone mapping—the LTM dataset. We evaluate our method on this new dataset as well as MIT-Adobe and HDR+ datasets. We show that the proposed LTMNet outperforms existing methods in local tone mapping while achieving competitive performance modeling additional photofinishing. Furthermore, we show that our method can be assistive in user-interactive photo-editing tools. Our code, model, and data will be released publicly.

Layer Decomposition based Local Tone Mapping Operator

Dynamic range of an image directly defines the number of levels it can accommodate in an image. In general, a dynamic range of 256 is in use. Defining dynamic range of an image need to be done by considering the display unit on which the image will be displayed on. If the dynamic range extends over 256, the dynamic range is said to be high dynamic range. This range may extend up to 10,000. This kind of images can’t be displayed on display units which can’t differentiate that many pictorial information. This kind of pictorial information need to be converted so that the regular display units can adapt the format. This must be done without losing much information. This process is very crucial and is done by several tone mapping operators. Tone mapping operators are of two types, global and local. Global tone mapping operators apply same mapping function throughout the image while the local tone mapping operators use different mapping functions for local regions of the image. Though the quality of global TMOs is better, there are many halo effects in the converted image. In this paper, a global TMO based on decomposition is proposed intended to reduce these effects. Image is decomposed in to two layers, base, and detail. A hybrid decomposition and optimization are proposed to improve the quality of converted image.

Warm-cool color-based high-speed decolorization: an empirical approach for tone mapping applications

Grayscale images are fundamental to many image processing applications like data compression, feature extraction, printing and tone mapping. However, some image information is lost when converting from color to grayscale. In this paper, we propose a light-weight and high-speed image decolorization method based on human perception of color temperatures. Chromatic aberration results from differential refraction of light depending on its wavelength. It causes some rays corresponding to cooler colors (like blue, green) to converge before the warmer colors (like red, orange). This phenomena creates a perception of warm colors "advancing" toward the eye, while the cool colors to be "receding" away. In this proposed color to gray conversion model, we implement a weighted blending function to combine red (perceived warm) and blue (perceived cool) channel. Our main contribution is threefold: First, we implement a high-speed color processing method using exact pixel by pixel processing, and we report a $5.7\times$ speed up when compared to other new algorithms. Second, our optimal color conversion method produces luminance in images that are comparable to other state of the art methods which we quantified using the objective metrics (E-score and C2G-SSIM) and a subjective user study. Third, we demonstrate that an effective luminance distribution can be achieved using our algorithm by using global and local tone mapping applications.

Visible and Near-Infrared Image Acquisition and Fusion for Night Surveillance

Image fusion combines images with different information to create a single, information-rich image. The process may either involve synthesizing images using multiple exposures of the same scene, such as exposure fusion, or synthesizing images of different wavelength bands, such as visible and near-infrared (NIR) image fusion. NIR images are frequently used in surveillance systems because they are beyond the narrow perceptual range of human vision. In this paper, we propose an infrared image fusion method that combines high and low intensities for use in surveillance systems under low-light conditions. The proposed method utilizes a depth-weighted radiance map based on intensities and details to enhance local contrast and reduce noise and color distortion. The proposed method involves luminance blending, local tone mapping, and color scaling and correction. Each of these stages is processed in the LAB color space to preserve the color attributes of a visible image. The results confirm that the proposed method outperforms conventional methods.

基于自适应阈值和局部色调映射的低照度图像处理

基于自适应阈值和局部色调映射的低照度图像处理

Research on defect detection method for high-reflective-metal surface based on high dynamic range imaging

The high dynamic range imaging (HDRI) technology can realistically reproduce the scene and acquire more image information for analysis and processing. In this paper, a defect detection method based on HDRI is proposed to solve the visual inspection problem of industrial parts with high-reflective surface. Firstly, a fast calibration method is put forward to increase the calibration speed of the camera response function (CRF). Then, a novel algorithm named adaptive tone mapping based on color correction model in gradient domain is designed to adaptively adjust image chromaticity information. Finally, adaptive threshold segmentation and Haar-like feature extraction algorithm are used to detect surface defects in synthesized high dynamic range (HDR) images. The experimental results show that the HDR images processed by our algorithm can meet the detection requirements for high-reflective-metal surface. The number of sampling points required by our fast calibration algorithm of CRF is 1/10 times than that of the traditional algorithm, and the running time in the program is about 1/4 times than that of the traditional algorithm. What’s more, our improved local tone mapping algorithm is obviously improved in information entropy, contrast and average gradient.

An Adaptive Global and Local Tone Mapping Algorithm Implemented on FPGA

We present a fast global and locally adaptive tone mapping algorithm and its field-programmable gate array (FPGA) implementation. The specially designed tone mapping function, which is based on local histogram equalization, controls global, and local characteristics individually. In contrast to other tonemap operators, our algorithm manages light/dark halos separately and by using local tonemap function alone, it can effectively suppress noise. We validated the effectiveness of our algorithms using subjective and objective assessment. Using an average of the bins, we achieve fast smoothed local histogram estimation with fewer bins while maintaining high accuracy. Our new implementation method requires minimal data access and reduced memory as it operates with a downscaled frame size of 240 × 135 pixels. Relative local area size is 248 × 248 @Full-HD resolution (1920 × 1080). For low-latency pixel output, the system performs the tone mapping using pixel information from the previous frame. When we implemented the system on FPGA (TB-7K-325TIMG and Xilinx Kintex-7), we achieved lightweight hardware as the total usage rate is about 25% of the available FPGA resource. Using an online 1080p video we demonstrate, a real-time video processing using our hardware tone mapping system.

Temporal adaptation control for local tone mapping operator

Abstract High dynamic range (HDR) imaging has gained great popularity over the past twenty years. Tone mapping operator (TMO) is the key component that enables reproduction of HDR images on the standard low dynamic range (LDR) display devices. When it comes to the HDR video, design of the TMO becomes especially challenging since temporal control of TMO parameters is needed in order to avoid possible artifacts. Since temporal and spatial contrast cannot be met simultaneously, existing solutions are usually designed to optimize one of these two requirements. We present novel local tone mapping operator that preserves details and simultaneously provides good local and global contrast of processed images. Tunable temporal control enables trade-off between spatial and temporal contrast of a tone mapped video. Flexible control presented in this paper ensures that both requirements can be met with a single operator just by using different tuning of the control block.When compared to the state-of-the-art TMOs, proposed solution exhibits better results regarding overall image quality.

Color HDR video processing architecture for smart camera

This paper presents a novel FPGA architecture of high dynamic range (HDR) video processing pipeline, based on the capturing of a sequence of differently exposed images. An acquisition process enabling multi-exposure HDR as well as fast implementation of local tone mapping operator involving bilateral filtering is proposed. The HDR acquisition process is enhanced by the application of novel deghosting method, which is dedicated for hardware implementation and proposed in this paper. The hardware processing pipeline is designed with regards to efficiency and performance and the calculations are performed in fixed point arithmetic. The pipeline is suitable for programmable hardware (FPGA—Field Programmable Gate Arrays) implementation and it achieves real-time performance on full HD HDR video which overcomes state-of-the-art solutions that use local tone mapping and deghosting algorithm.

Local tone mapping algorithm and hardware implementation

A novel tone mapping algorithm and hardware implementation for displaying wide dynamic range (WDR) images are proposed. The algorithm processes WDR images in a pixel-by-pixel fashion in the logarithmic domain, and it uses the block-interpolated minimum and maximum pixel values. The hardware implementation can achieve real-time processing of WDR images and is resource-efficient. Experimental results show that the proposed algorithm and hardware implementation can produce images with good brightness and high contrast.

Enhanced local tone mapping for detail preserving reproduction of high dynamic range images

Enhanced Local Tone Mapping (ELTM) is a flexible tone mapping operator designed to provide a good global and local contrast simultaneously over various test scenes. Also, it has intuitive and decoupled tuning interface, providing the user with full control over final image appearance. ELTM is based on detail/base layer decomposition compressing the base plane in both linear and logarithmic domain. This provides robustness to ELTM, while modified tone compression function provides good local contrast. Results were validated using set of images with various content, brightness and resolution. In this testing ELTM performed as the best tone mapping operator, among 7 state-of-the-art global and local tone mapping operators. Even better overall results are achieved by using proposed brightness control, to handle extreme scenes. Robustness and flexibility to achieve desired appearance makes ELTM suitable for applications where user experience is the primary concern as is the case with consumer electronics products.

High Dynamic Range Imaging Using Multiple Exposures

It is challenging to capture a high-dynamic range (HDR) scene using a low-dynamic range (LDR) camera. This paper presents an approach for improving the dynamic range of cameras by using multiple exposure images of same scene taken under different exposure times. First, the camera response function (CRF) is recovered by solving a high-order polynomial in which only the ratios of the exposures are used. Then, the HDR radiance image is reconstructed by weighted summation of the each radiance maps. After that, a novel local tone mapping (TM) operator is proposed for the display of the HDR radiance image. By solving the high-order polynomial, the CRF can be recovered quickly and easily. Taken the local image feature and characteristic of histogram statics into consideration, the proposed TM operator could preserve the local details efficiently. Experimental result demonstrates the effectiveness of our method. By comparison, the method outperforms other methods in terms of imaging quality.

HDR tone mapping algorithm based on difference compression with adaptive reference values

Tone mapping remains a challenging problem since tone mapping operators need to produce high perceptual quality under all conditions. In this paper, we propose a new local tone mapping method based on difference compression with adaptive reference values, which can effectively reproduce the details of bright and shadow regions. We also use a global tone mapping method and blend the output images produced by the global and local methods based on objective quality metrics. To quantitatively measure output images, we developed a new objective quality metric for the tone mapped images. The proposed detailness metric measures detail loss in the bright and shadow regions, and shows good correlations with subjective quality. We combined this metric with the recently proposed tone mapped image quality index (TMQI) that may not sufficiently reflect the amount of local detail loss. The experiments show that the proposed algorithm provides better perceptual quality than existing methods.

Event-Based Tone Mapping for Asynchronous Time-Based Image Sensor.

The asynchronous time-based neuromorphic image sensor ATIS is an array of autonomously operating pixels able to encode luminance information with an exceptionally high dynamic range (>143 dB). This paper introduces an event-based methodology to display data from this type of event-based imagers, taking into account the large dynamic range and high temporal accuracy that go beyond available mainstream display technologies. We introduce an event-based tone mapping methodology for asynchronously acquired time encoded gray-level data. A global and a local tone mapping operator are proposed. Both are designed to operate on a stream of incoming events rather than on time frame windows. Experimental results on real outdoor scenes are presented to evaluate the performance of the tone mapping operators in terms of quality, temporal stability, adaptation capability, and computational time.

Enhanced high dynamic‐range image rendering using a surround map based on edge‐adaptive layer blurring

Many tone‐mapping methods have developed efficient tone compression and local contrast enhancement techniques for high dynamic‐range imaging. Local tone mapping enhances the image quality to reveal the details; however, it causes artefacts such as halos. Halo artefacts appear around the edges of an image and lead to deterioration in the overall image quality. Detail‐base separation and multi‐scale methods have been developed to reduce halo artefacts. Detail‐base separation methods divide the image into a detail layer and base layer using an edge‐preserving filter and process each layer independently. Multi‐scale methods alleviate artefacts such as halos, noise, and awkward global toning using a weighted sum of single‐scale images. These methods have several inherent problems including the complexity of the edge‐preserving filter and multi‐processing redundancy. Therefore, in this study, a novel surround map is presented to resolve halo artefacts and enhance local details in single‐scale processing.

A Tone Mapping Algorithm Based on Multi-scale Decomposition

A new Tone Mapping( TM) algorithm based on multi-scale decomposition was proposed to solve a High Dynamic Range( HDR) image displayed on an ordinary display device. The algorithm decomposed a HDR image into multiple scales using a Local Edge-Preserving( LEP) filter to smooth the details of the image effectively, while still retaining the salient edges. Then a dynamic range compression function with parameters was proposed according to the characteristics of the decomposed layers and the request of compression. By changing the parameters, the coarse scale layer was compressed and the fine scale layer was boosted, which resulted in compressing the dynamic range of the image and boosting the details. Finally, by restructuring the image and restoring the color, the image after mapping had a good visual quality. The experimental results demonstrate that the proposed method is better than the algorithm proposed by Gu et al.( GU B, LI W J, ZHU M Y, et al. Local edge-preserving multiscale decomposition for high dynamic range image tone mapping [J]. IEEE Transactions on Image Processing, 2013, 22( 1) : 70- 79) and Yeganeh et al.( YEGANEH H, WANG Z. Objective quality assessment of tonemapped images [J]. IEEE Transactions on Image Processing, 2013, 22( 2) : 657- 667) in naturalness, structural fidelity and quality assessment; moreover, it avoids the halo artifacts which is a common problem existing in the local tone mapping algorithms. The algorithm can be used for the tone mapping of the HDR image.