Single-shot High Dynamic Range Research Articles

MetaHDR: single shot high-dynamic range imaging and sensing using a multifunctional metasurface

We present MetaHDR, which is a single-shot high-dynamic range (HDR) imaging and sensing system using a multifunctional metasurface. The metasurface is capable of splitting an incident beam into multiple focusing beams with different amounts of power, simultaneously forming multiple low dynamic range (LDR) images with distinct irradiance on a photosensor. Then, the LDR images are jointly processed using a gradient-based HDR fusion algorithm, which is shown to be effective in attenuating the residual light artifacts incurred by the metasurface and the lens flare. MetaHDR achieves single-shot HDR photography and videography that increases the dynamic range by at least 50 dB compared to the original dynamic range of the photosensor. It can also perform single-shot HDR sensing, including reflectance calibration and surface curvature estimation of reflective materials. MetaHDR’s demonstrated functionalities could be broadly applied in surveillance and security, microscopic imaging, advanced manufacturing, etc.

Microlens array camera with variable apertures for single-shot high dynamic range (HDR) imaging.

We report a microlens array camera with variable apertures (MACVA) for high dynamic range (HDR) imaging by using microlens arrays with various sizes of apertures. The MACVA comprises variable apertures, microlens arrays, gap spacers, and a CMOS image sensor. The microlenses with variable apertures capture low dynamic range (LDR) images with different f-stops under single-shot exposure. The reconstructed HDR images clearly exhibit expanded dynamic ranges surpassing LDR images as well as high resolution without motion artifacts, comparable to the maximum MTF50 value observed among the LDR images. This compact camera provides, what we believe to be, a new perspective for various machine vision or mobile devices applications.

Single-shot 3D measurement of highly reflective objects with deep learning.

Three-dimensional (3D) measurement methods based on fringe projection profilometry (FPP) have been widely applied in industrial manufacturing. Most FPP methods adopt phase-shifting techniques and require multiple fringe images, thus having limited application in dynamic scenes. Moreover, industrial parts often have highly reflective areas leading to overexposure. In this work, a single-shot high dynamic range 3D measurement method combining FPP with deep learning is proposed. The proposed deep learning model includes two convolutional neural networks: exposure selection network (ExSNet) and fringe analysis network (FrANet). The ExSNet utilizes self-attention mechanism for enhancement of highly reflective areas leading to overexposure problem to achieve high dynamic range in single-shot 3D measurement. The FrANet consists of three modules to predict wrapped phase maps and absolute phase maps. A training strategy directly opting for best measurement accuracy is proposed. Experiments on a FPP system showed that the proposed method predicted accurate optimal exposure time under single-shot condition. A pair of moving standard spheres with overexposure was measured for quantitative evaluation. The proposed method reconstructed standard spheres over a large range of exposure level, where prediction errors for diameter were 73 µm (left) and 64 µm (right) and prediction error for center distance was 49 µm. Ablation study and comparison with other high dynamic range methods were also conducted.

Single-Shot High Dynamic Range Imaging via Multiscale Convolutional Neural Network

We propose a single-shot high dynamic range (HDR) imaging algorithm with row-wise varying exposures in a single raw image based on a deep convolutional neural network (CNN). We first convert a raw Bayer input image into a radiance map by calibrating rows with different exposures, and then we design a new CNN model to restore missing information at the under- and over-exposed pixels and reconstruct color information from the raw radiance map. The proposed CNN model consists of three branch networks to obtain multiscale feature maps for an image. To effectively estimate the high-quality HDR images, we develop a robust loss function that considers the human visual system (HVS) model, color perception model, and multiscale contrast. Experimental results on both synthetic and captured real images demonstrate that the proposed algorithm can achieve synthesis results of significantly higher quality than conventional algorithms in terms of structure, color, and visual artifacts.

Method for reconstructing a high dynamic range image based on a single-shot filtered low dynamic range image.

Traditional cameras are limited by sensors and cannot directly capture single-shot high dynamic range (HDR) images. We propose an improved single-shot HDR image reconstruction method that uses a single-exposure filtered low dynamic range (FLDR) image. First, by adding an optical filter in front of the camera lens, a FLDR image with different RGB channel exposure states and luminance ranges can be captured in a single-shot, unlike the traditional LDR image. Second, a deep inverse tone mapping network (DITMnet) with multibranch features extraction and multioutput images synthesis is designed to reconstruct an HDR image from a single FLDR image. Experimentally, under different exposure states and color spaces, our method outperforms similar algorithms.

An Image Fusion Scheme for Single-Shot High Dynamic Range Imaging with Spatially Varying Exposures

This paper proposes a novel multi-exposure image fusion (MEF) scheme for single-shot high dynamic range imaging with spatially varying exposures (SVE). Single-shot imaging with SVE enables us not only to produce images without color saturation regions from a single-shot image, but also to avoid ghost artifacts in the producing ones. However, the number of exposures is generally limited to two, and moreover it is difficult to decide the optimum exposure values before the photographing. In the proposed scheme, a scene segmentation method is applied to input multi-exposure images, and then the luminance of the input images is adjusted according to both of the number of scenes and the relationship between exposure values and pixel values. The proposed method with the luminance adjustment allows us to improve the above two issues. In this paper, we focus on dual-ISO imaging as one of single-shot imaging. In an experiment, the proposed scheme is demonstrated to be effective for single-shot high dynamic range imaging with SVE, compared with conventional MEF schemes with exposure compensation.

Pixel Binning for High Dynamic Range Color Image Sensor Using Square Sampling Lattice.

We propose a new pixel binning scheme for color image sensors. We minimized distortion caused by binning by requiring that the superpixels lie on a square sampling lattice. The proposed binning schemes achieve the equivalent of 4.42 times signal strength improvement with the image resolution loss of 5 times, higher in noise performance and in resolution than the existing binning schemes. As a result, the proposed binning has considerably less artifacts and better noise performance compared with the existing binning schemes. In addition, we provide an extension to the proposed binning scheme for performing single-shot high dynamic range image acquisition.

Adaptive dualISO HDR reconstruction

With the development of modern image sensors enabling flexible image acquisition, single shot high dynamic range (HDR) imaging is becoming increasingly popular. In this work, we capture single shot HDR images using an imaging sensor with spatially varying gain/ISO. This allows all incoming photons to be used in the imaging. Previous methods on single shot HDR capture use spatially varying neutral density (ND) filters which lead to wasting incoming light. The main technical contribution in this work is an extension of previous HDR reconstruction approaches for single shot HDR imaging based on local polynomial approximations (Kronander et al., Unified HDR reconstruction from raw CFA data, 2013; Hajisharif et al., HDR reconstruction for alternating gain (ISO) sensor readout, 2014). Using a sensor noise model, these works deploy a statistically informed filtering operation to reconstruct HDR pixel values. However, instead of using a fixed filter size, we introduce two novel algorithms for adaptive filter kernel selection. Unlike a previous work, using adaptive filter kernels (Signal Process Image Commun 29(2):203–215, 2014), our algorithms are based on analyzing the model fit and the expected statistical deviation of the estimate based on the sensor noise model. Using an iterative procedure, we can then adapt the filter kernel according to the image structure and the statistical image noise. Experimental results show that the proposed filter de-noises the noisy image carefully while well preserving the important image features such as edges and corners, outperforming previous methods. To demonstrate the robustness of our approach, we have exploited input images from raw sensor data using a commercial off-the-shelf camera. To further analyze our algorithm, we have also implemented a camera simulator to evaluate different gain patterns and noise properties of the sensor.