Radiance Map Research Articles

R 3LIVE++: A Robust, Real-Time, Radiance Reconstruction Package With a Tightly-Coupled LiDAR-Inertial-Visual State Estimator.

This work proposed a LiDAR-inertial-visual fusion framework termed R 3LIVE++ to achieve robust and accurate state estimation while simultaneously reconstructing the radiance map on the fly. R 3LIVE++ consists of a LiDAR-inertial odometry (LIO) and a visual-inertial odometry (VIO), both running in real-time. The LIO subsystem utilizes the measurements from a LiDAR for reconstructing the geometric structure, while the VIO subsystem simultaneously recovers the radiance information of the geometric structure from the input images. R 3LIVE++ is developed based on R 3LIVE and further improves the accuracy in localization and mapping by accounting for the camera photometric calibration and the online estimation of camera exposure time. We conduct more extensive experiments on public and self-collected datasets to compare our proposed system against other state-of-the-art SLAM systems. Quantitative and qualitative results show that R 3LIVE++ has significant improvements over others in both accuracy and robustness. Moreover, to demonstrate the extendability of R 3LIVE++, we developed several applications based on our reconstructed maps, such as high dynamic range (HDR) imaging, virtual environment exploration, and 3D video gaming.

Fast and Accurate Illumination Estimation Using LDR Panoramic Images for Realistic Rendering.

A high dynamic range (HDR) image is commonly used to reveal stereo illumination, which is crucial for generating high-quality realistic rendering effects. Compared to the high-cost HDR imaging technique, low dynamic range (LDR) imaging provides a low-cost alternative and is preferable for interactive graphics applications. However, the limited LDR pixel bit depth significantly bothers accurate illumination estimation using LDR images. The conflict between the realism and promptness of illumination estimation for realistic rendering is yet to be resolved. In this paper, an efficient method that accurately infers illuminations of real-world scenes using LDR panoramic images is proposed. It estimates multiple lighting parameters, including locations, types and intensities of light sources. In our approach, a new algorithm that extracts illuminant characteristics during the exposure attenuation process is developed to locate light sources and outline their boundaries. To better predict realistic illuminations, a new deep learning model is designed to efficiently parse complex LDR panoramas and classify detected light sources. Finally, realistic illumination intensities are calculated by recovering the inverse camera response function and extending the dynamic range of pixel values based on previously estimated parameters of light sources. The reconstructed radiance map can be used to compute high-quality image-based lighting of virtual models. Experimental results demonstrate that the proposed method is capable of efficiently and accurately computing comprehensive illuminations using LDR images. Our method can be used to produce better realistic rendering results than existing approaches.

A High Dynamic Range Imaging Method for Short Exposure Multiview Images

The restoration and enhancement of multiview low dynamic range (MVLDR) images captured in low lighting conditions is a great challenge. The disparity maps are hardly reliable in practical, real-world scenarios and suffers from holes and artifacts due to large baseline and angle deviation among multiple cameras in low lighting conditions. Furthermore, multiple images with some additional information (e.g., ISO/exposure time, etc.) are required for the radiance map and poses the additional challenges of deghosting to encounter motion artifacts. In this paper, we proposed a method to reconstruct multiview high dynamic range (MVHDR) images from MVLDR images without relying on disparity maps. We detect and accurately match the feature points among the involved input views and gather the brightness information from the neighboring viewpoints to optimize an image restoration function based on input exposure gain to finally generate MVHDR images. Our method is very reliable and suitable for a wide baseline among sparse cameras. The proposed method requires only one image per viewpoint without any additional information and outperforms others.

HDR Map Reconstruction From a Single LDR Sky Panoramic Image for Outdoor Illumination Estimation

Outdoor low dynamic range (LDR) panoramic images that contain the sun and sky are generally over-saturated because the sun is 10,000 times brighter than the regions that surround it. Because the luminance information in the region that contains the sun in these images is lost, it is difficult to identify the sun’s position and generate high dynamic range (HDR) environment maps that can be used to realistically relight virtual objects. Previous methods to reconstruct HDR maps did not consider that the sun covers a small area in an image but contains extremely high luminance values. These methods are therefore insufficient for estimating scene illumination. We propose a multi-faceted approach to reconstructing HDR maps from a single LDR sky panoramic image that considers the sun and sky regions separately. We encode an input image and transfer a multi-dimensional latent representation to two decoders, which reconstruct the luminance information in the sky and sun regions separately. To plausibly model sun illumination, we introduce two networks (Sunpose-net and Sunrad-net) that estimate the position and radiance of the sun. The generated sun radiance map is then merged with the output of the decoder that is responsible for sun regions. We demonstrated that the proposed method more plausibly reconstructs HDR maps than previous methods using the HDR-VDP-2.2 which measures the visual quality of reconstructed HDR maps against ground truth. The accuracy of the overall sun and sky illumination distribution in HDR maps reconstructed using the proposed method was evaluated using histogram distance measures.

Real‐time Deep Radiance Reconstruction from Imperfect Caches

AbstractReal‐time global illumination is a highly desirable yet challenging task in computer graphics. Existing works well solving this problem are mostly based on some kind of precomputed data (caches), while the final results depend significantly on the quality of the caches. In this paper, we propose a learning‐based pipeline that can reproduce a wide range of complex light transport phenomena, including high‐frequency glossy interreflection, at any viewpoint in real time (> 90 frames per‐second), using information from imperfect caches stored at the barycentre of every triangle in a 3D scene. These caches are generated at a precomputation stage by a physically‐based offline renderer at a low sampling rate (e.g., 32 samples per‐pixel) and a low image resolution (e.g., 64×16). At runtime, a deep radiance reconstruction method based on a dedicated neural network is then involved to reconstruct a high‐quality radiance map of full global illumination at any viewpoint from these imperfect caches, without introducing noise and aliasing artifacts. To further improve the reconstruction accuracy, a new feature fusion strategy is designed in the network to better exploit useful contents from cheap G‐buffers generated at runtime. The proposed framework ensures high‐quality rendering of images for moderate‐sized scenes with full global illumination effects, at the cost of reasonable precomputation time. We demonstrate the effectiveness and efficiency of the proposed pipeline by comparing it with alternative strategies, including real‐time path tracing and precomputed radiance transfer.

Formula omitted] cooling in the jovian aurora: Juno remote sensing observations and modeling

Cooling of Jupiter's auroral thermosphere by H3+ radiation to space is one of the processes controlling the energy balance in the auroral upper atmosphere. The UltraViolet Spectrograph (UVS) and the Jupiter InfraRed Auroral Mapper (JIRAM) on board Juno have observed the Jovian polar aurora from its polar orbit since August 2016. The UVS instrument measures the H2 Lyman and Werner bands whose brightness is a proxy of the precipitated auroral electron flux. The 3.3–3.6 μm spectral window of the JIRAM L-band imager maps the H3+ thermal radiance with unprecedented spatial resolution. Comparison of concurrent observations indicates that the morphological features are similar in the two spectral regions but differences are also observed in the spatial intensity contrast. We compare the total (direct and indirect) particle heating rate and the cooling by H3+ radiation derived from four pairs of simultaneous UVS and JIRAM images. The total auroral cooling power H3+ is in the range 2–4 terawatts in both hemispheres. In all cases, the H3+ cooling in the aurora is found to range between 0.45 and 0.67 time less than the particle collisional heating. In a second step, we use the ultraviolet H2 brightness and FUV color ratio to derive the characteristics of the electron precipitation and model the H3+ radiance for each UVS map pixel. The comparison of the H3+ modeled radiance map with the JIRAM observations shows general good agreement with some local differences. The four spatially integrated H3+ cooling power from the model are in very good agreement with the JIRAM values. These results are important constraints for global magnetosphere-ionosphere-atmosphere models.

Camera response function assessment in multispectral HDR imaging

Recently, spatially varying Bidirectional Reflectance Distribution Functions (svBRDF) is widely used as a model to characterize the appearance of materials with varying visual properties over the surface. One of the challenges in image-based svBRDF capture systems rises for surfaces with high specularity and sparkles, which require a dynamic range higher than the dynamic range of cameras. High Dynamic Range Imaging (HDRI) for svBRDF systems with multispectral camera has not been addressed properly in the literature. In HDRI, Camera Response Function (CRF) plays a crucial role in the precision of results specially when measuring metrological data such as spectral svBRDF. In this work, we investigate the effect of CRF assessment on the precision of measurement. Therefore, we have conducted two experiments to measure absolute CRF using reflective chart method as well as estimate relative CRF by Debevec and Malik’s method for a filter wheel multispectral camera to be used in a svBRDF setup. Results are evaluated on two levels: radiance map construction and reflectance calculation, by comparing to telespectroradiometer measurements as ground truth data. Results showed that although the HDRI with measured absolute CRF outputs radiance measurements with the same physical units and in the same scale as ground truth data, HDRI with estimated relative CRF outperformed in terms of the precision of reflectance measurement.

Enhanced Road Lane Detection Facing Sun Glare

There are several studies on road lane detection but very few address adverse conditions for acquisition such as sun glare. Loss of details in underexposed images captured facing a low sun leads to misleading road lane detection. High Dynamic Range Imaging methods are used to acquire most details in such scenes. Unfortunately, these techniques are heavy on computations and therefore unsuitable for real time road lane detection. In this paper, we propose a machine learning solution that avoids High Dynamic Range Imaging computations that are the radiance map estimation, tone-mapping algorithms and quality measures calculation. We train a neural network on a High Dynamic Range Imaging dataset. The resulting model produces suitable images for road lane detection facing sun glare, in real time. Subjective and objective comparisons with the most popular High Dynamic Range Imaging method, Mertens Algorithm, are conducted to prove the effectiveness of the proposed Neural Network. The delivered images demonstrated an improvement in road lane detection.

Computing light pollution indicators for environmental assessment

AbstractLight pollution modeling and monitoring has traditionally used zenith sky brightness as its main indicator. Several other indicators (e.g., average sky radiance, horizontal irradiance, and average sky radiance at given interval of zenith distances) may be more useful, both for general and specific purposes of ecology studies, night sky, and environmental monitoring. These indicators can be calculated after the whole sky radiance is known with sufficient angular detail. This means, for each site, to integrate the contribution in each direction of the sky of each light source in the radius of hundreds of km. This approach is extremely high time‐consuming if the mapping is desired for a large territory. Here, we present a way to obtain maps of large territories for a large subset of useful indicators, bypassing the need to calculate first the radiance map of the whole sky in each site to obtain from it the desired indicator in that site. For each indicator, a point spread function (PSF) is calculated from the whole sky radiance maps generated by a single source at sufficiently dense number of distances from the observing site. If the PSF is transversally shift‐invariant, that is, if it depends only on the relative position of source and observer, then we can further speed up the map calculation via the use of fast Fourier‐transform. We present here examples of maps for different indicators. Precise results can be calculated for any single site, taking into account the site and light sources' altitudes, by means of specific inhomogeneous (spatially variant) and anisotropic (nonrotationally symmetric) PSFs.Key points Light pollution is not anymore viewed as an astronomycentric issue. On the contrary, it is a multidisciplinary problem, involving, between others, physicists, ecologists, physiologists, engineers, politicians. Some of the disciplines that will directly take advantage from the methods explained here are: astronomy, ecology and ethology (in condition of altered night light), urban and landscape planning, environmental protection.

Visible and Near-Infrared Image Acquisition and Fusion for Night Surveillance

Image fusion combines images with different information to create a single, information-rich image. The process may either involve synthesizing images using multiple exposures of the same scene, such as exposure fusion, or synthesizing images of different wavelength bands, such as visible and near-infrared (NIR) image fusion. NIR images are frequently used in surveillance systems because they are beyond the narrow perceptual range of human vision. In this paper, we propose an infrared image fusion method that combines high and low intensities for use in surveillance systems under low-light conditions. The proposed method utilizes a depth-weighted radiance map based on intensities and details to enhance local contrast and reduce noise and color distortion. The proposed method involves luminance blending, local tone mapping, and color scaling and correction. Each of these stages is processed in the LAB color space to preserve the color attributes of a visible image. The results confirm that the proposed method outperforms conventional methods.

Observation of Postsunset OI 135.6 nm Radiance Enhancement Over South America by the GOLD Mission

AbstractThe Global‐scale Observation of Limb and Disk (GOLD) mission, for the first time, provides synoptic two‐dimensional (2D) maps of OI 135.6 nm observations. These maps describe the unambiguous and dynamic evolution of nighttime ionospheric F2‐peak electron densities (NmF2) as the 135.6 nm airglow emission radiance correlates well with NmF2 at night. On November 19, 2018, the 135.6 nm radiance measured by GOLD, NmF2 measured by a digisonde, and GPS total electron content (TEC) measurements at Cachoeira Paulista (CP) all showed a postsunset enhancement, with an increase near 22:30 universal time. The 135.6 nm radiance map showed that this enhancement was due to the southward movement of the southern equatorial ionization anomaly (EIA) crest. Therefore, the GOLD observation showed the linkage between postsunset enhancement of NmF2 and EIA movement. Furthermore, unlike the southward movement of the southern crest, the corresponding EIA northern crest, however, did not show northward motion. This is the first time that the EIA hemispheric asymmetry, which included both different densities and movement of two crests in a short time period (<2‐h), was captured. The cause of this asymmetric movement of the two crests is not clear and requires further investigation.

Robust 3D Reconstruction Using HDR-Based SLAM

3D reconstruction is an important topic in the field of the emerging applications such as smart robotics, virtual reality (VR), augmented reality (AR), and autonomous driving. RGB-D simultaneous localization and mapping (SLAM) technique is widely used in the reconstruction process. However, low light and low textured environment often results in insufficient point features and fails the reconstruction. To address this problem, we propose a robust RGB-D SLAM system using high dynamic range (HDR) image information called HDR-based SLAM. The deep learning based HDR generation method is adopted to map a single low dynamic range (LDR) image into a radiance map which is normalized to exclude the influence of exposure time. We retrained the ORB descriptor patch to fit the normalized radiance maps in the feature matching step. The proposed method can improve the quantitative camera trajectory accuracy and qualitative result of geometry reconstruction. Experimental results show that the proposed method has better performance compared to that of the standard range imaging SLAM under challenging low light environment, which helps expand the applicability of 3D reconstruction system.

Single-Shot High Dynamic Range Imaging via Multiscale Convolutional Neural Network

We propose a single-shot high dynamic range (HDR) imaging algorithm with row-wise varying exposures in a single raw image based on a deep convolutional neural network (CNN). We first convert a raw Bayer input image into a radiance map by calibrating rows with different exposures, and then we design a new CNN model to restore missing information at the under- and over-exposed pixels and reconstruct color information from the raw radiance map. The proposed CNN model consists of three branch networks to obtain multiscale feature maps for an image. To effectively estimate the high-quality HDR images, we develop a robust loss function that considers the human visual system (HVS) model, color perception model, and multiscale contrast. Experimental results on both synthetic and captured real images demonstrate that the proposed algorithm can achieve synthesis results of significantly higher quality than conventional algorithms in terms of structure, color, and visual artifacts.

Visible and Near-Infrared Image Synthesis Using PCA Fusion of Multiscale Layers

This study proposes a method of blending visible and near-infrared (NIR) images to enhance their edge details and local contrast based on the Laplacian pyramid and principal component analysis (PCA). In the proposed method, both the Laplacian pyramid and PCA are implemented to generate a radiance map. Using the PCA algorithm, the soft-mixing method and the mask-skipping filter were applied when the images were fused. The color compensation method uses the ratio between the radiance map fused by the Laplacian pyramid and the PCA algorithm and the luminance channel of the visible image to preserve the chrominance of the visible image. The results show that the proposed method improves edge details and local contrast effectively.

Optical effects on HDR calibration via a multiple exposure noise-based workflow

High dynamic range (HDR) technology allows more of the lighting in a specific scene to be captured at a set point in time, and thus is capable of delivering an overall view of the scene that more closely correlates with our visual experience in the real world, compared to standard, or low dynamic range (LDR) technology. Although HDR capabilities of single exposure capture systems are improving, the traditional method for creating HDR images still includes combing a number of different exposures, captured with an LDR system, into a single HDR image. Several use cases requiring absolute calibration of the resulting HDR luminance map have been undertaken, but none of these have provided a detailed analysis of the optical effects of glare on the results. We develop a calibrated HDR radiance map, including methodical linearization of captured image data, while characterizing the limitations due to the effects of optical glare. A purposely designed controlled test scene is used to challenge the calibrated reconstruction efforts, including low luminance levels, spatial inclusion of lens vignette over the full imaged area, and optical glare. Results demonstrate that even with careful processing and recombination of the LDR data, radiometric accuracy is limited as a result of glare. The proposed approach performs better than calibration methods in commercially available HDR recombination software.

Mobile Campaign of Sky Brightness Measurement in Indonesia

The bright sky caused by light pollution is a problem for astronomical research. One of the first steps in dealing with the effects of light pollution is to calculate the level of light pollution. Measurement of sky brightness using Sky Quality Meter (SQM) can be done massively through mobile campaign/observations. LAPAN conducted such observations at several places namely Sumatera Barat, Sumedang, Jawa Tengah, Jawa Timur and Kupang by using SQM attached on vehicles. Those campaigns covers 80 to 980 km journey and the total distance traveled was about 2870 km. The obtained data show that the sky brightness above regions surveyed vary from 10 magnitude per arcsec2 (mpsas), especially around Surabaya, Jawa Timur to 22 mpsas. The results then compared radiance map of the Earth surface from Visible Infrared Imaging Radiometer Suite (VIIRS) aboard Suomi National Polar Orbiting Partnership (NPP). Moderate correlations between in-situ sky brightness data and the radiance map are achieved. Further analyses are required prior to utilization of those relations in the development of Indonesia’s sky brightness map which is presumably more accurate than the existing global map of sky brightness.

Limits of Color Constancy: Comparison of the signatures of chromatic adaptation and spatial comparisons

Color Constancy has two hypothetical mechanisms: Chromatic Adaptation and Spatial Comparisons. These mechanisms have different fundamental properties. Adaptation models work with small individual scene segments. They combine radiance measurements of individual segments with the modeler’s selected parameters that scale the receptor’s cone quanta catches. Alternatively, spatial models use the radiance map of the entire field of view to calculate appearances of all image segments simultaneously. They achieve independence from spectral shifts in illumination by making spatial comparisons within each L, M, S color channel. These spatial comparisons respond to color crosstalk caused by the overlap of spectral sensitivities of cone visual pigments. L, M, and S cones respond to every visible wavelength. Crosstalk causes the spatial comparisons of cone responses to vary with changes in spectral illumination. Color Constancy works best in spatially uniform, and variable spectral illumination. Measurements of Color Constancy show systematic departures from perfect constancy. These limits of Color Constancy are predicted by spatial comparisons with cone Crosstalk. These limits do not correlate with Chromatic Adaptation models. This paper describes cone Crosstalk, and reviews a series of measurements of the limits of Color Constancy in variable spectral, spatial and real-life illuminations.

Detail Enhanced Multi-Exposure Image Fusion Based On Edge Preserving Filters

Recent computational photography techniques play a significant role to overcome the limitation of standard digital cameras for handling wide dynamic range of real-world scenes contain brightly and poorly illuminated areas. In many of such techniques [1,2,3], it is often desirable to fuse details from images captured at different exposure settings, while avoiding visual artifacts. One such technique is High Dynamic Range (HDR) imaging that provides a solution to recover radiance maps from photographs taken with conventional imaging equipment. The process of HDR image composition needs the knowledge of exposure times and Camera Response Function (CRF), which is required to linearize the image data before combining Low Dynamic Range (LDR) exposures into HDR image. One of the long-standing challenges in HDR imaging technology is the limited Dynamic Range (DR) of conventional display devices and printing technology. Due to which these devices are unable to reproduce full DR. Although DR can be reduced by using a tone-mapping, but this comes at an unavoidable trade-off with increased computational cost. Therefore, it is desirable to maximize information content of the synthesized scene from a set of multi-exposure images without computing HDR radiance map and tone-mapping.This research attempts to develop a novel detail enhanced multi-exposure image fusion approach based on texture features, which exploits the edge preserving and intra-region smoothing property of nonlinear diffusion filters based on Partial Differential Equations (PDE). With the captured multi-exposure image series, we first decompose images into Base Layers (BLs) and Detail Layers (DLs) to extract sharp details and fine details, respectively. The magnitude of the gradient of the image intensity is utilized to encourage smoothness at homogeneous regions in preference to inhomogeneous regions. In the next step texture features of the BL to generate a decision mask (i.e., local range) have been considered that guide the fusion of BLs in multi-resolution fashion. Finally, well-exposed fused image is obtained that combines fused BL and the DL at each scale across all the input exposures. The combination of edge-preserving filters with Laplacian pyramid is shown to lead to texture detail enhancement in the fused image.Furthermore, Non-linear adaptive filter is employed for BL and DL decomposition that has better response near strong edges. The texture details are then added to the fused BL to reconstruct a detail enhanced LDR version of the image. This leads to an increased robustness of the texture details while at the same time avoiding gradient reversal artifacts near strong edges that may appear in fused image after DL enhancement.Finally, we propose a novel technique for exposure fusion in which Weighted Least Squares (WLS) optimization framework is utilized for weight map refinement of BLs and DLs, which lead to a new simple weighted average fusion framework. Computationally simple texture features (i.e. DL) and color saturation measure are preferred for quickly generating weight maps to control the contribution from an input set of multi-exposure images. Instead of employing intermediate HDR reconstruction and tone-mapping steps, well-exposed fused image is generated for displaying on conventional display devices. Simulation results are compared with a number of existing single resolution and multi-resolution techniques to show the benefits of the proposed scheme for the variety of cases. Moreover, the approaches proposed in this thesis are effective for blending flash and no-flash image pair, and multi-focus images, that is, input images photographed with and without flash, and images focused on different targets, respectively. A further advantage of the present technique is that it is well suited for detail enhancement in the fused image.

HDR Image Processing from Embedded Cell Phone Camera

In photo-realistic image synthesis the incoming light from the environment is particularly important. In this work we focus on capturing the incoming light in the form of the radiance map using a common mobile device. This involves the reconstruction of the spherical panorama sky-dome in high dynamic range (HDR) quality and save it to a usable data format. Our setup requires a common smartphone devices with an additional fish-eye lens attached to the camera. In this paper we discuss the calibration of our setup and the implementation of selected tone mapping operator (TMO) allowing satisfactory display of HDR images on the mobile device screen. Built-in cameras on mobile devices do not generally capture the HDR image. In this work, we describe an algorithm for capturing an HDR image on the Android platform. Using an optimization method we are able to acquire the camera response curve needed in the reconstruction of the HDR image from multiple snapshots. Subsequent HDR panoramas represent radiance sphere-maps of the incoming light from an environment. These radiance sphere-maps are useful in realistic image synthesis to illuminate the objects in the 3D scenes.

A high dynamic range structured light means for the 3D measurement of specular surface

This paper presents a novel structured light approach for the 3D reconstruction of specular surface. The binary shifting strip is adopted as structured light pattern instead of conventional sinusoidal pattern. Based on the framework of conventional High Dynamic Range (HDR) imaging technique, an efficient means is first introduced to estimate the camera response function. And then, dynamic range of the generated radiance map is compressed in the gradient domain by introducing an attenuation function. Subject to the change of lighting conditions caused by projecting different structured light patterns, the structure light image with middle exposure level is selected as the reference image and used for the slight adjustment of the primary fused image. Finally, the regenerated structured light images with well exposing condition are used for 3D reconstruction of the specular surface. To evaluate performance of the method, some stainless stamping parts with strong reflectivity are used for the experiments. And the results showed that, different specular targets with various shapes can be precisely reconstructed by the proposed method.