High Dynamic Range Scenes Research Articles

Multi-exposure microscopic image fusion-based detail enhancement algorithm

Traditional microscope imaging techniques are unable to retrieve the complete dynamic range of a diatom species with complex silica-based cell walls and multi-scale patterns. In order to extract details from the diatom, multi-exposure images are captured at variable exposure settings using microscopy techniques. A recent innovation shows that image fusion overcomes the limitations of standard digital cameras to capture details from high dynamic range scene or specimen photographed using microscopy imaging techniques. In this paper, we present a cell-region sensitive exposure fusion (CS-EF) approach to produce well-exposed fused images that can be presented directly on conventional display devices. The ambition is to preserve details in poorly and brightly illuminated regions of 3-D transparent diatom shells. The aforesaid objective is achieved by taking into account local information measures, which select well-exposed regions across input exposures. In addition, a modified histogram equalization is introduced to improve uniformity of input multi-exposure image prior to fusion. Quantitative and qualitative assessment of proposed fusion results reveal better performance than several state-of-the-art algorithms that substantiate the method’s validity.

Bionic orientation method based on polarization imaging in HDR scenes.

An increasing number of bio-inspired navigation approaches have been designed based on polarization cameras. However, digital cameras can sense a much narrower field of vision than the vision of insects or human beings. In this study, we propose an adaptive skylight polarized orientation method for high dynamic range (HDR) scenes. Initially, we built a model of the image acquisition pipeline that can recover HDR irradiance maps from polarization images. Subsequently, the orientation method was designed based on a combination of the irradiance maps and the least squares methods. Some preprocessing steps were utilized to eliminate occlusion interference. In addition, an autoexposure adjustment method was proposed using information entropy and heuristic segmentation. Finally, the experimental results show that the proposed method can improve the accuracy of bionic orientation and adaption to skylight with occlusions and interference in natural conditions.

A review of IEEE P2020 noise metrics

The IEEE P2020 standard addresses fundamental image quality attributes that are specifically relevant to cameras in automotive imaging systems. The Noise standard in IEEE P2020 is mostly based on existing standards on noise in digital cameras. However, it adjusts test conditions and procedures to make them more suitable for cameras for automotive applications, such as use of fisheye lenses, 16-32 bit data format in operation in high dynamic range (HDR) mode, HDR scenes, extended temperature range, and near-infrared imaging. The work presents methodology, procedures and experimental results that demonstrate extraction of camera characteristics from videos of HDR and other test charts that are recorded in raw format, including dark and photo signals, temporal noise, fixed-pattern noise, signal-to-noise ratio curves, photon transfer curve, transaction factor and effective full well capacity. The work also presents methodology and experimental results for characterization of camera noise in the dark array and signal falloff.

Objective image quality evaluation of HDR videos captured by smartphones

High Dynamic Range (HDR) videos attract industry and consumer markets thanks to their ability to reproduce wider color gamuts, higher luminance ranges and contrast. While the cinema and broadcast industries traditionally go through a manual mastering step on calibrated color grading hardware, consumer cameras capable of HDR video capture without user intervention are now available. The aim of this article is to review the challenges found in evaluating cameras capturing and encoding videos in an HDR format, and improve existing measurement protocols to objectively quantify the video quality produced by those systems. These protocols study adaptation to static and dynamic HDR scenes with illuminant changes as well as the general consistency and readability of the scene’s dynamic range. An experimental study has been made to compare the performances of HDR video capture to Standard Dynamic Range (SDR) video capture, where significant differences are observed, often with scene-specific content adaptation similar to the human visual system.

Quantitative Measurement of Perceptual Attributes and Artifacts for Tone-Mapped HDR Display

Measuring electronic display quality, as perceived by human observers, has attracted high attention in current consumer displays. With limited dynamic range of consumer-level standard dynamic range (SDR) displays, high dynamic range (HDR) scenes are often rendered by different tone mapping operators (TMOs). Due to the lack of quantitative measurement and analysis of tone mapping in the existing image quality measurement (IQM) methods, it is of considerable significance to establish new IQM protocols that can differentiate the display quality of SDR electronic devices. We first propose an IQM model to exhibit the essential perceptual attributes and artifacts that are peculiar to tone-mapping. Furthermore, we characterize the overall image quality (OiQ) resulting from linear regression and various machine learning techniques. Finally, the execution of without HDR reference ablation experiments demonstrate the relative contribution of these attribute measurement to the OiQ. The use of IQM protocols helps with well-founded quality measurement between TMOs during tone-mapping processing. Our effort is not only useful to get into the tone-mapping field or when implementing a TMO, but it also sets the stage for quantitative measurement of TMOs. By monitoring these attributes and artifacts after different tone-mapping process, user-driven or optimal display are made possible.

Tone mapping high dynamic range images based on region-adaptive self-supervised deep learning

This paper presents a region-adaptive self-supervised deep learning (RASSDL) technique for high dynamic range (HDR) image tone mapping. The RASSDL tone mapping operator (TMO) is a convolutional neural network (CNN) trained on local image regions that can seamlessly tone map images of arbitrary sizes. The training of RASSDL TMO is through the design of a self-supervising target that automatically adapts to the local image regions based on their information contents. The self-supervising target is designed to ensure the tone-mapped output achieves a balance between preserving the relative contrast of the original scene and the visibilities of the fine details to achieve faithful reproduction of the HDR scene. Distinguishing from many existing TMOs that require manual tuning of parameters, RASSDL is parameter-free and completely automatic. Experimental results demonstrate that RASSDL TMO can achieve state-of-the-art performance in terms of preserving overall contrasts, revealing fine details, and being free from visual artifacts.

High dynamic range 3D measurements based on space-time speckle correlation and color camera.

Structured light (SL) based three-dimensional (3D) measurement struggles to estimate high dynamic range (HDR) scenes, where both high and low reflectivity parts exist simultaneously. This paper proposes a method through the joint design and optimization of hardware and algorithms, in which only four frames are required to realize the 3D reconstruction of HDR scenes. The height information of each sub-area in the scene under test can be encoded effectively by temporally projecting two sets of complementary speckle patterns onto target surface. To decode the corresponding patterns captured by the cameras, we design a stereo matching strategy consisting of space-time binary feature (ST-BIF) descriptor preliminary screening and zero-mean normalized cross-correlation (ST-ZNCC) final retrieval. The ST-BIF descriptor based on neighborhood comparison is designed to describe the space-time relative intensity change of projected speckles. Besides the HDR adaptability, the ST-BIF descriptor can effectively improve the matching speed. In addition, the measurable dynamic range can be further improved by fusing all channel disparities as evaluated results, benefitting from the different response of R, G and B channels in color camera to monochromatic light. Experiments are conducted to demonstrate the feasibility of the proposed method. The results indicate that our method achieves the root mean square error 0.2516mm (vs. 1.0668 by commonly used ZNCC) and an average coverage rate up to 94.87% (vs. 93.35% by commonly used ZNCC). Furthermore, the experimental results show that the proposed method can achieve 3D reconstruction of HDR scenes including specular reflection region.

Perception of White for Stimuli with Luminance Beyond the Diffuse White

The appearance of color stimuli with luminance levels beyond the diffuse white is gaining importance due to the popularity of high dynamic range (HDR) displays. Past work on color appearance of stimuli, color appearance models, and uniform color spaces mainly focused on the stimuli with luminance levels below the diffuse white, which were produced using surface color samples or conventional standard dynamic range (SDR) displays. In this study, we focused on the perception of white appearance for stimuli with luminance beyond the diffuse white. Human observers adjusted the color appearance of a stimulus to the whitest under different adapting conditions, including a dark condition and 12 illuminated conditions. It was found that the chromaticities for producing the white appearance under the dark condition were generally similar to those under the 6500 K conditions, regardless of the adapting luminance levels. In comparison to a recent study focusing on the stimuli with luminance below the diffuse white, the perception of white under the conditions with the adapting CCT levels of 2700, 3500, and 5000 K was significantly affected by the lightness level of the stimulus, which cannot be accurately characterized by CAM02-UCS. The results can be used for reproducing white appearance for highlights in HDR scenes. Further investigations on uniform color spaces for characterizing stimuli with luminance beyond the diffuse white are urgently needed for processing and displaying HDR images.

Efficient Pipelined Circuits for Histogram-based Tone Mapping of Nonlinear CMOS Image Sensors

Abstract Tone mapping is extensively researched to address the issue of displaying high dynamic range (DR) scenes on low DR displays. Even though several tone-mapping operators (TMOs) exist, not all are designed for hard real time. The operator has to be capable of scaling up the spatial resolution without compromising the frame rate. The implementation of a TMO should also be simple enough to embed in low-cost platforms for imaging systems. A computationally efficient, and well accepted, class of TMOs are global ones based on histograms. This work presents a method to implement TMOs that use histograms. This method is suitable for low-cost field-programmable gate arrays (FPGAs), using simple components such as adders, multipliers, and random access memories, and is particularly suited for a nonlinear CMOS image sensor (CIS) operating continuously in hard real time. The authors develop a fixed-point design, validated in bit-true fashion using Xilinx and Altera tools, from a background algorithm implemented using Matlab. Our generic design uses pipelined circuits and operates with low latency. The use of a hardware description language to model our circuits guarantees portability and modularity. Moreover, the complete TMO is generated from design parameters and a design template. The architecture is robust and scales well from kilopixel to megapixel formats. The circuits achieve 30 frames per second, at high definition resolutions, while occupying only a small fraction of the available logic elements in a low-cost FPGA device.

High frame rate optical flow estimation from event sensors via intensity estimation

Optical flow estimation forms the core of several computer vision tasks and its estimation requires accurate spatial and temporal gradient information. However, if there are fast-moving objects in the scene or if the camera moves rapidly, then the acquired images will suffer from motion blur, which will lead to poor optical flow estimation. Such challenging cases can be handled by event sensors which are a novel generation of sensors that acquire pixel-level brightness changes as binary events at a very high temporal resolution. Brightness constancy constraint, which is the basis of several optical flow algorithms cannot be directly used on event sensors making it challenging to estimate optical flow. We overcome this challenge by imposing brightness constancy constraint on intensity images predicted from event sensor data. For this task, we design a recurrent neural network that jointly predicts a sparse optical flow and intensity images from the event data. While intensity estimation is supervised using ground truth frames, optical flow estimation is self-supervised using the predicted intensity frames. However, in our case the temporal resolution of the ground truth intensity frames is far lower than the temporal resolution of the predicted intensity frames, making it challenging to supervise. As we use recurrent neural network, such a challenge can be overcome by sharing the weights for each of the predicted intensity frames. Quantitatively our predicted optical flow is better than previously proposed algorithms for optical flow estimation from event sensors. We also show our algorithm’s robustness against challenging cases of fast motion and high dynamic range scenes.

Fast exposure fusion of detail enhancement for brightest and darkest regions

Multi-exposure fusion is the common approach to generate high dynamic range (HDR) images that combines multi-exposure images captured for the same scene, but the traditional multi-exposure fusion algorithms lose details in the brightest and darkest regions of the scene. Therefore, many detail enhancement-based exposure fusion algorithms have been proposed to extract these details. However, these algorithms have low efficiency because of the complexity of detail enhancement mechanism, and most of them excessively enhance all the pixels besides of the necessary brightest and darkest pixels. We propose a local detail enhancement mechanism to enhance only the details of brightest and darkest regions by using fast local Laplacian filtering (FLLF). A large number of experiments show that the proposed algorithm has much more high efficiency than the current detail enhancement-based exposure fusion algorithms, and the brightest and darkest details in the high dynamic range scene are preserved well.

Event-Based Stereo Visual Odometry

Event-based cameras are bioinspired vision sensors whose pixels work independently from each other and respond asynchronously to brightness changes, with microsecond resolution. Their advantages make it possible to tackle challenging scenarios in robotics, such as high-speed and high dynamic range scenes. We present a solution to the problem of visual odometry from the data acquired by a stereo event-based camera rig. Our system follows a parallel tracking-and-mapping approach, where novel solutions to each subproblem (three-dimensional (3-D) reconstruction and camera pose estimation) are developed with two objectives in mind: being principled and efficient, for real-time operation with commodity hardware. To this end, we seek to maximize the spatio-temporal consistency of stereo event-based data while using a simple and efficient representation. Specifically, the mapping module builds a semidense 3-D map of the scene by fusing depth estimates from multiple viewpoints (obtained by spatio-temporal consistency) in a probabilistic fashion. The tracking module recovers the pose of the stereo rig by solving a registration problem that naturally arises due to the chosen map and event data representation. Experiments on publicly available datasets and on our own recordings demonstrate the versatility of the proposed method in natural scenes with general 6-DoF motion. The system successfully leverages the advantages of event-based cameras to perform visual odometry in challenging illumination conditions, such as low-light and high dynamic range, while running in real-time on a standard CPU. We release the software and dataset under an open source license to foster research in the emerging topic of event-based simultaneous localization and mapping.

Deep Tone-Mapping Operator Using Image Quality Assessment Inspired Semi-Supervised Learning

Tone-mapping operator (TMO) is intended to convert high dynamic range (HDR) content into a lower dynamic range so that it can be displayed on a standard dynamic range (SDR) device. The tone-mapped result of HDR content is usually stored as SDR image. For different HDR scenes, traditional TMOs are able to obtain a satisfying SDR image only under manually fine-tuned parameters. In this paper, we address this problem by proposing a learning-based TMO using deep convolutional neural network (CNN). We explore different CNN structure and adopt multi-scale and multi-branch fully convolutional design. When training deep CNN, we introduce image quality assessments (IQA), specifically, tone-mapped image quality assessment, and implement it as semi-supervised loss terms. We discuss and prove the effectiveness of semi-supervised loss terms, CNN structure, data pre-processing, etc. by several experiments. Finally, we demonstrate that our approach can produce appealing results under diversified HDR scenes.

Fast Exposure Fusion Based on Camera Response Function

This paper proposed a fast multi exposure images fusion algorithm based on the camera response function for high dynamic range scene imaging, which greatly improve the efficiency and ensure the good quality of the fused image. The algorithm calibrated the camera response function in advance, and then calculated the mapping curve between the scene illuminance and pixel value according to the exposure time. When fuse multi-exposure images, the mapping curves of the images are connected to obtain a new mapping curve which can curve the whole luminance range. Then the fused image can be obtained through the new mapping curve. The experimental results demonstrate the good effectiveness and high efficiency of proposed algorithm.

Real-time high dynamic range 3D measurement using fringe projection

Fringe projection profilometry (FPP) is a widely used technique for real-time three-dimensional (3D) shape measurement. However, it tends to compromise when measuring objects that have a large variation range of surface reflectivity. In this paper, we present a FPP method that can increase the dynamic range for real-time 3D measurements. First, binary fringe patterns are projected to generate grayscale sinusoidal patterns with the defocusing technique. Each pattern is then captured twice with different exposure values in one projection period. With image fusion, surfaces under appropriate exposure are retained. To improve the real-time performance of high dynamic range (HDR) 3D shape measurements, we build a binocular fringe projection profilometry system that saves the number of patterns by geometry constraint. Further, to ensure the accuracy and robustness of HDR 3D measurements, we propose a mixed phase unwrapping method that can reduce phase unwrapping errors for dense fringe patterns. Experiment results show that the proposed method can realize accurate and real-time 3D measurement for HDR scenes at 28 frames per second.

High-speed high dynamic range 3D shape measurement based on deep learning

For many three-dimensional (3D) measurement techniques based on fringe projection profilometry (FPP), measuring the objects with a large variation range of surface reflectivity is always a very tricky problem due to the limited dynamic range of camera. Many high dynamic range (HDR) 3D measurement methods are developed for static scenes, which are fragile for dynamic objects. In this paper, we address the problem of phase information loss in HDR scenes, in order to enable 3D reconstruction from saturated or dark images by deep learning. By using a specifically designed convolutional neural network (CNN), we can accurately extract phase information in both the low signal-to-noise ratio (SNR) and saturation situations after proper training. Experimental results demonstrate the success of our network in 3D reconstruction for both static and dynamic HDR objects. Our method can improve the dynamic range of three-step phase-shifting by a factor of 4.8 without any additional projected images or hardware adjustment during measurement. And the final 3D measurement speed of our method is about 13.89 Hz (off-line).

Object tracking on event cameras with offline–online learning

Compared with conventional image sensors, event cameras have been attracting attention thanks to their potential in environments under fast motion and high dynamic range (HDR). To tackle the lost-track issue due to fast illumination changes under HDR scene such as tunnels, an object tracking framework has been presented based on event count images from an event camera. The framework contains an offline-trained detector and an online-trained tracker which complement each other: The detector benefits from pre-labelled data during training, but may have false or missing detections; the tracker provides persistent results for each initialised object but may suffer from drifting issues or even failures. Besides, process and measurement equations have been modelled, and a Kalman fusion scheme has been proposed to incorporate measurements from the detector and the tracker. Self-initialisation and track maintenance in the fusion scheme ensure autonomous real-time tracking without user intervene. With self-collected event data in urban driving scenarios, experiments have been conducted to show the performance of the proposed framework and the fusion scheme.

Research on Auto-Exposure Algorithm Based on Image Big Data and Information Entropy

An Auto-Exposure (AE) algorithm based on image big data and information entropy is proposed. On the basis of the traditional algorithm for automatic exposure adjustment based on image brightness, image big data analysis is introduced for the first time. Through the combination of ambient luminance evaluation and image information entropy, the dimension of information acquisition of the automatic exposure system is improved, thus improving the image effect and scene adaptability of the camera. Especially in high dynamic range scenes, compared with the traditional algorithm, the effect is significantly improved.

CAOS Smart Camera-based Robust Low Contrast Image Recovery over 90 dB Scene Linear Dynamic Range

Experimentally demonstrated for the first time is Coded Access Optical Sensor (CAOS) camera empowered robust and true white light High Dynamic Range (HDR) scene low contrast target image recovery over the full linear dynamic range. The 90 dB linear HDR scene uses a 16 element custom designed test target with low contrast 6 dB step scaled irradiances. Such camera performance is highly sought after in catastrophic failure avoidance mission critical HDR scenarios with embedded low contrast targets.

Notice of Violation of IEEE Publication Principles: Tone Mapping Beyond the Classical Receptive Field

Notice of Violation of IEEE Publication Principles<br><br> "Tone Mapping Beyond the Classical Receptive Field,"<br> by Shao-Bing Gao, Min-Jie Tan, Zhen He, and Yong-Jie Li<br> in the IEEE Transactions on Image Processing, vol. 29, February 2020, pp. 4174‐4187<br><br> After careful and considered review of the content of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE's Publication Principles. This paper contains significant portions of content from the paper cited below. The original content was copied without attribution (including appropriate references to the original author(s) and/or paper title) and without permission.<br><br> Due to the nature of this violation, reasonable effort should be made to remove all past references to this paper, and future references should be made to the following paper<br><br> "Colour Constancy Beyond the Classical Receptive Field,"<br> by Arash Akbarinia, and C. Alejandro Parraga,<br> in the IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 9,September 2018, pp. 2081‐2094<br> <br/> Some neurons in the primary visual cortex (V1) of human visual system (HVS) conduct dynamic center-surround computation, which is thought to contribute to compress the high dynamic range (HDR) scene and preserve the details. We simulate this dynamic receptive field (RF) property of V1 neurons to solve the so-called tone mapping (TM) task in this paper. The novelties of our method are as follows. (1) Cortical processing mechanisms of HVS are modeled to build a local TM operation based on two Gaussian functions whose kernels and weights adapt according to the center-surround contrast, thus reducing halo artifacts and effectively enhancing the local details of bright and dark parts of image. (2) Our method uses an adaptive filter that follows the contrast levels of the image, which is computationally very efficient. (3) The local fusion between the center and surround responses returned by a cortical processing flow and the global signals returned by a sub-cortical processing flow according to the local contrast forms a dynamic mechanism that selectively enhances the details. Extensive experiments show that the proposed method can efficiently render the HDR scenes with good contrast, clear details, and high structural fidelity. In addition, the proposed method can also obtain promising performance when applied to enhance the low-light images. Furthermore, by modeling these biological solutions, our technique is simple and robust considering that our results were obtained using the same parameters for all the datasets (e.g., HDR images or low-light images), that is, mimicking how HVS operates.