Scene Illumination Research Articles

Multiple Graph Affinity Interactive Network and a Variable Illumination Dataset for RGBT Image Salient Object Detection

Salient object detection (SOD) of images refers to simulating the attention mechanism of human vision to capture the most attractive objects in an image. Current SOD mainly relies on RGB images captured by optical cameras. However, existing optical cameras are not comparable to the human visual system, especially in poorly illumination scenes. Our human visual system is able to resolve scenes well in low light conditions, while optical cameras can barely image without enough illumination. To make machine vision closer to the imaging of the human eye, we propose to use thermal infrared (T) images to compensate RGB images and build a variable illumination RGBT dataset named VI-RGBT1500 for SOD. This dataset is collected under three different illumination conditions including sufficient illumination, uneven illumination and insufficient illumination to fully demonstrate the superiority of the RGBT image combination. Furthermore, we propose a multiple graph affinity interactive (MGAI) network to validate the proposed dataset. Our network structure is simple using only the MGAI to fuse the features of different modalities. Meanwhile, the MGAI model highlights valuable information during the interaction, which facilitates feature representation under variable illumination. The proposed VI-RGBT1500 dataset and three publicly available RGBT SOD datasets are used for the comparison experiments, and the results with the state-of-the-art methods prove that our VI-RGBT1500 dataset is valuable and the performance of the MGAI network is competitive. The VI-RGBT1500 dataset and the MGAI network are available at: https://github.com/huanglm-me/VI-RGBT1500.

CVCC Model: Learning-Based Computer Vision Color Constancy with RiR-DSN Architecture.

To achieve computer vision color constancy (CVCC), it is vital but challenging to estimate scene illumination from a digital image, which distorts the true color of an object. Estimating illumination as accurately as possible is fundamental to improving the quality of the image processing pipeline. CVCC has a long history of research and has significantly advanced, but it has yet to overcome some limitations such as algorithm failure or accuracy decreasing under unusual circumstances. To cope with some of the bottlenecks, this article presents a novel CVCC approach that introduces a residual-in-residual dense selective kernel network (RiR-DSN). As its name implies, it has a residual network in a residual network (RiR) and the RiR houses a dense selective kernel network (DSN). A DSN is composed of selective kernel convolutional blocks (SKCBs). The SKCBs, or neurons herein, are interconnected in a feed-forward fashion. Every neuron receives input from all its preceding neurons and feeds the feature maps into all its subsequent neurons, which is how information flows in the proposed architecture. In addition, the architecture has incorporated a dynamic selection mechanism into each neuron to ensure that the neuron can modulate filter kernel sizes depending on varying intensities of stimuli. In a nutshell, the proposed RiR-DSN architecture features neurons called SKCBs and a residual block in a residual block, which brings several benefits such as alleviation of the vanishing gradients, enhancement of feature propagation, promotion of the reuse of features, modulation of receptive filter sizes depending on varying intensities of stimuli, and a dramatic drop in the number of parameters. Experimental results highlight that the RiR-DSN architecture performs well above its state-of-the-art counterparts, as well as proving to be camera- and illuminant-invariant.

The impact of variable illumination on vegetation indices and evaluation of illumination correction methods on chlorophyll content estimation using UAV imagery

BackgroundThe advancements in unmanned aerial vehicle (UAV) technology have recently emerged as an effective, cost-efficient, and versatile solution for monitoring crop growth with high spatial and temporal precision. This monitoring is usually achieved through the computation of vegetation indices (VIs) from agricultural lands. The VIs are based on the incoming radiance to the camera, which is affected when there is a change in the scene illumination. Such a change will cause a change in the VIs and subsequent measures, e.g., the VI-based chlorophyll-content estimation. In an ideal situation, the results from VIs should be free from the impact of scene illumination and should reflect the true state of the crop’s condition. In this paper, we evaluate the performance of various VIs computed on images taken under sunny, overcast and partially cloudy days. To improve the invariance to the scene illumination, we furthermore evaluated the use of the empirical line method (ELM), which calibrates the drone images using reference panels, and the multi-scale Retinex algorithm, which performs an online calibration based on color constancy. For the assessment, we used the VIs to predict leaf chlorophyll content, which we then compared to field measurements.ResultsThe results show that the ELM worked well when the imaging conditions during the flight were stable but its performance degraded under variable illumination on a partially cloudy day. For leaf chlorophyll content estimation, The r^2 of the multivariant linear model built by VIs were 0.6 and 0.56 for sunny and overcast illumination conditions, respectively. The performance of the ELM-corrected model maintained stability and increased repeatability compared to non-corrected data. The Retinex algorithm effectively dealt with the variable illumination, outperforming the other methods in the estimation of chlorophyll content. The r^2 of the multivariable linear model based on illumination-corrected consistent VIs was 0.61 under the variable illumination condition.ConclusionsOur work indicated the significance of illumination correction in improving the performance of VIs and VI-based estimation of chlorophyll content, particularly in the presence of fluctuating illumination conditions.

A novel framework for crowd counting using video and audio

Crowd counting is significant in many areas. The existing methods have poor accuracy for perspective scenes and low illumination scenes. Besides, the existing audio-assisted methods only use local audio, which fails to provide the spatial feature information of sound in all directions in space. To alleviate the above problems, a novel framework named Video and Audio-assisted Crowd Counting Network (VACCNet) is proposed. The framework consists of two submodules: Video Crowd Counting (VCC) module and Audio-assisted Crowd Counting (ACC) module. The visual features from the VCC module and the fused audio features from the ACC module are further combined to get the final density map. To prove the effects of VACCNet, a new self-collected dataset named multiPle dIrection Assistance couNting netwOrk (PIANO) is built. The experimental results based on existing benchmarks and PIANO show that the proposed method has a 14.23% improvement averagely to the conventional methods.

Adaptive Enhanced Detection Network for Low Illumination Object Detection

Any small environmental changes in the driving environment of a traffic vehicle can become a risk factor directly leading to major safety incidents. Therefore, it is necessary to assist drivers in automatically detecting risk factors during the driving process using algorithms. However, besides making it more difficult for drivers to judge environmental changes, the performance of automatic detection networks in low illumination scenarios can also be greatly affected and cannot be used directly. In this paper, we propose a risk factor detection model based on deep learning in low illumination scenarios and test the optimization of low illumination image enhancement problems. The overall structure of this model includes dual discriminators, encoder–decoders, etc. The model consists of two main stages. In the first stage, the input low illumination scene image is adaptively converted into a standard illumination image through a lighting conversion module. In the second stage, the converted standard illumination image is automatically assessed for risk factors. The experiments show that the detection network can overcome the impact of low lighting and has high detection accuracy.

Multiple-wavelength range-gated active imaging applied to the evaluation of simultaneous movement of millimeter-size objects moving in a given volume.

The combined multiple-wavelength range-gated active imaging (WRAI) principle is able to determine the position of a moving object in a four-dimensional space and to deduce its trajectory and its speed independently of the video frequency. However, when the scene size is reduced and the objects have a millimeter size, the temporal values intervening on the depth of the visualized zone in the scene cannot be reduced further because of technological limitations. To improve the depth resolution, the illumination type of the juxtaposed style of this principle has been modified. Therefore, it was important to evaluate this new context in the case of millimeter-size objects moving simultaneously in a reduced volume. Based on the rainbow volume velocimetry method, the combined WRAI principle was studied in accelerometry and velocimetry with four-dimensional images of millimeter-size objects. This basic principle combining two wavelength categories determines the depth of moving objects in the scene with the warm color category and the precise moment of moving objects position with the cold color category. The difference in this new, to the best of our knowledge, method is at the level of the scene illumination, which is obtained transversally by a pulsed light source having a wide spectral band limited in warm colors in order to get a better depth resolution. For cold colors, the illumination with pulsed beams of distinct wavelengths remains unchanged. Thus, it is possible independently of the video frequency to know from a single recorded image the trajectory, the speed, and the acceleration of millimeter-size objects moving simultaneously in 3D space and also the chronology of their passages. The experimental tests validated this modified multiple-wavelength range-gated active imaging method and confirmed the possibility of avoiding confusion when the object trajectories intersect.

Shadow invariant classification for high-resolution images

Shadows are one of the most commonly observed optical phenomena in real-world environments. Often they are encountered in a target-scene area depending on the angle of view, scene structure, scene illumination, and scene geometry. In practice, the performance of several image-driven analysis techniques, such as classification tasks, becomes unstable in the presence of shadows. Specifically, shadow regions are profoundly difficult for image classification applications resulting in false predictions because they are inherently complex and are quite unlike anything seen during training. This paper introduces a novel algorithm named shadow invariant classifier system (SICS) to mitigate the challenges posed by shadows in various classification tasks, as such studies are currently lacking. Also, we propose a general framework for automatically selecting an optimal threshold for the proposed similarity matching methods (SMMs). The proposed algorithms use spectral information alone and require no pre-knowledge of shadows and their environmental settings. They are resilient to the complexities associated with the presence of shadows. To validate the performance, we conducted several classification experiments using six different remote sensing images from multi-sensor and multiplatform modalities having a varying scale of shadows and lighting effects. A thorough evaluation using the high spatial, spectral resolution multispectral and hyperspectral imagery demonstrates the effective shadow insensitive classification performance obtained by the proposed algorithms

Fast image visibility enhancement based on active polarization and color constancy for operation in turbid water.

Vehicles operating in a water medium sometimes encounter harsh conditions with high turbidity and low scene illumination, making it challenging to obtain reliable target information through optical devices. Although many post-processing solutions were proposed, they are not applicable to continuous vehicle operations. Inspired by the advanced polarimetric hardware technology, a joint fast algorithm was developed in this study to address the above problems. Backscatter attenuation and direct signal attenuation were solved separately by utilizing the revised underwater polarimetric image formation model. A fast local adaptive Wiener filtering method was used to improve the backscatter estimation by reducing the additive noise. Further, the image was recovered using the fast local space average color method. By using a low-pass filter guided by the color constancy theory, the problems of nonuniform illumination caused by artificial light and direct signal attenuation were both addressed. The results of testing on images from laboratory experiments showed improved visibility and realistic chromatic rendition.

Local Adaptive Illumination-Driven Input-Level Fusion for Infrared and Visible Object Detection

Remote sensing object detection based on the combination of infrared and visible images can effectively adapt to the around-the-clock and changeable illumination conditions. However, most of the existing infrared and visible object detection networks need two backbone networks to extract the features of two modalities, respectively. Compared with the single modality detection network, this greatly increases the amount of calculation, which limits its real-time processing on the vehicle and unmanned aerial vehicle (UAV) platforms. Therefore, this paper proposes a local adaptive illumination-driven input-level fusion module (LAIIFusion). The previous methods for illumination perception only focus on the global illumination, ignoring the local differences. In this regard, we design a new illumination perception submodule, and newly define the value of illumination. With more accurate area selection and label design, the module can more effectively perceive the scene illumination condition. In addition, aiming at the problem of incomplete alignment between infrared and visible images, a submodule is designed for the rapid estimation of slight shifts. The experimental results show that the single modality detection algorithm based on LAIIFusion can ensure a large improvement in accuracy with a small loss of speed. On the DroneVehicle dataset, our module combined with YOLOv5L could achieve the best performance.

Performance and polarization response of slit homogenizers for the GeoCarb mission

Abstract. The observing strategy of the Geostationary Carbon Observatory (GeoCarb), which is a “step and stare” approach, can lead to distortions in the instrument spectral response function (ISRF) when there are gradients in brightness across instrument field of view. These distortions induce errors in the retrieved trace gases. In order to minimize these errors, the GeoCarb instrument design was modified to include a “slit homogenizer” whose purpose is to scramble the pattern of the incoming light and effectively remove the ISRF distortions caused by the variations in illumination across the slit. As a risk reduction, GeoCarb procured six different homogenizers and had them tested for performance in a benchtop optical system. The major finding is that the homogenizer performance depends strongly on the polarization of the incoming light, with the sensitivity growing as a function of wavelength. The width of the ISRF is substantially smaller when the light is vertically polarized (orthogonal to the slit length) compared to horizontally polarized (parallel to the slit length), and the throughput is accordingly reduced. These effects are due to the effects of the gold coating and high incidence angles present in the GeoCarb homogenizer design, which was verified using a polarization-dependent model generalized from previous homogenizer modeling work. The results strongly recommend controlling the polarization of the light entering a similar implementation using a polarizer, depolarizer, or polarization scrambler for other instruments attempting to mitigate scene illumination non-uniformity effects, as well as a robust characterization of the polarization sensitivity of all key subsystems.

HDR Map Reconstruction From a Single LDR Sky Panoramic Image for Outdoor Illumination Estimation

Outdoor low dynamic range (LDR) panoramic images that contain the sun and sky are generally over-saturated because the sun is 10,000 times brighter than the regions that surround it. Because the luminance information in the region that contains the sun in these images is lost, it is difficult to identify the sun’s position and generate high dynamic range (HDR) environment maps that can be used to realistically relight virtual objects. Previous methods to reconstruct HDR maps did not consider that the sun covers a small area in an image but contains extremely high luminance values. These methods are therefore insufficient for estimating scene illumination. We propose a multi-faceted approach to reconstructing HDR maps from a single LDR sky panoramic image that considers the sun and sky regions separately. We encode an input image and transfer a multi-dimensional latent representation to two decoders, which reconstruct the luminance information in the sky and sun regions separately. To plausibly model sun illumination, we introduce two networks (Sunpose-net and Sunrad-net) that estimate the position and radiance of the sun. The generated sun radiance map is then merged with the output of the decoder that is responsible for sun regions. We demonstrated that the proposed method more plausibly reconstructs HDR maps than previous methods using the HDR-VDP-2.2 which measures the visual quality of reconstructed HDR maps against ground truth. The accuracy of the overall sun and sky illumination distribution in HDR maps reconstructed using the proposed method was evaluated using histogram distance measures.

A non-uniform illumination image enhancement method based on fusion of events and frames

The dynamic range of standard frame-based cameras is much smaller than that of natural scenes. Images captured by standard cameras in non-uniform illumination scenes may be over-exposure, under-exposure, or a combination of both, limiting the visibility of the images. Enhancing images with uneven illumination is a tough assignment. Although most of the existing non-uniform illumination image enhancement methods can preserve details well in dark areas, they cannot regain details in bright regions and may produce noise and halos. Event cameras have a higher dynamic range and are less affected by scene lighting than standard cameras. In this paper, we utilize the complementary advantages of event cameras and standard cameras to improve the visibility of unevenly illumination images, enhance the brightness of dark areas, and recover details in bright areas. First, events and frames are converted into intensity frames by corresponding preprocessing methods, which are subjected to a gamma correction operation to obtain artificial multi-exposure image sequences. Then, the weight map of each artificially exposed image is designed using several evaluation terms, namely contrast, well-exposure, and average brightness. After that, the weight terms are combined with the Laplacian pyramid algorithm to get a fused image. Finally, the final image is created by color restoration and contrast enhancement. Quantitative and qualitative experimental results demonstrate that our method enhances the details and visibility of non-uniform illumination images more effectively than several contrast algorithms.

Multi-scale fusion framework via retinex and transmittance optimization for underwater image enhancement

Low contrast, poor color saturation, and turbidity are common phenomena of underwater sensing scene images obtained in highly turbid oceans. To address these problems, we propose an underwater image enhancement method by combining Retinex and transmittance optimized multi-scale fusion framework. Firstly, the grayscale of R, G, and B channels are quantized to enhance the image contrast. Secondly, we utilize the Retinex color constancy to eliminate the negative effects of scene illumination and color distortion. Next, a dual transmittance underwater imaging model is built to estimate the background light, backscattering, and direct component transmittance, resulting in defogged images through an inverse solution. Finally, the three input images and corresponding weight maps are fused in a multi-scale framework to achieve high-quality, sharpened results. According to the experimental results and image quality evaluation index, the method combined multiple advantageous algorithms and improved the visual effect of images efficiently.

A novel factor graph-based optimization technique for stereo correspondence estimation

Dense disparities among multiple views are essential for estimating the 3D architecture of a scene based on the geometrical relationship between the scene and the views or cameras. Scenes with larger extents of homogeneous textures, differing scene illumination among the multiple views and with occluding objects affect the accuracy of the estimated disparities. Markov random fields based methods for disparity estimation address these limitations using spatial dependencies among the observations and among the disparity estimates. These methods, however, are limited by spatially fixed and smaller neighborhood systems or cliques. Recent learning-based methods generate rich set of stereo features for generating cost volume and estimating disparity. In this work, we present a new factor graph-based probabilistic graphical model for disparity estimation that allows a larger and a spatially variable neighborhood structure determined based on the local scene characteristics. Our algorithm improves the accuracy of disparity estimates in stereo image pairs with varying texture and illumination characteristics by enforcing spatial dependencies among scene characteristics as well as among disparity estimates. We evaluated our method using the Middlebury benchmark stereo datasets and the Middlebury evaluation dataset version 3.0 and compared its performance with recent state-of-the-art disparity estimation algorithms. Our factor graph-based algorithm provided disparity estimates with higher accuracy when compared to the recent non-learning- and learning-based disparity estimation algorithms. The factor graph formulation can be used for obtaining maximum a posteriori estimates from models or optimization problems with complex dependency structure among hidden variables. The strategies of using a priori distributions with shorter support and spatial dependencies were useful for reducing the computational cost and improving message convergence in the model. The factor-graph algorithm is also useful for other dense estimation problems such as optical flow estimation.

Adaptive ORB feature detection with a variable extraction radius in RoI for complex illumination scenes

Feature detection is a crucial technique for a vision navigation system to estimate robot pose according to natural landmarks. It is difficult for the existing feature detection techniques to balance the feature quality, processing time and robustness for a vision-based robot in complex workspaces. An adaptive Oriented fast and Rotated Brief (ORB) feature detection method with a variable extraction radius in Region of Interest (RoI) is proposed to deal with these problems. Firstly, the original camera image is processed by means of the Laplace transform of Gaussian (LTOG) pyramid and the grayscale centroid method, in order to obtain the rotation and scale invariance for ORB features. Then, a RoI segmenting technique is developed to locate the image areas that contain potential ORB features due to obvious grayscale variation. Thirdly, the ORB features are extracted in RoIs by using a set of variable-radius templates, adaptive to different illumination conditions. Finally, a number of feature detection and robot localization experiments are conducted on a vision-based robot prototype in different scenes under complex illumination. The experimental results verify that the RoI segmenting technique can correctly preserve the grayscale-varying regions to search ORB features with scattered distribution but excluding the irrelevant areas to suppress feature noises, while the variable-radius template extraction method can detect more feature inliers in complex workspaces. Therefore, our adaptive ORB method can outperform other commonly-used algorithms in accuracy, efficiency and robustness.

Real Time Skin Color Detection Based on Adaptive HSV Thresholding

The detection of human skin color has been studied extensively during the past two decades. It is an essential task for various computer vision applications such as biometric authentication, face/hands tracking and gesture analysis. New machine learning methods are effective for skin color detection. However, they are not suitable for real time applications since they are computationally heavy. A lightweight approach for skin color detection consists of using segmentation rules extracted by an investigation on skin color distribution. The kin appearance varies with diversity of image types, acquisition parameters and scene illumination. There are no general segmentation rules that provide effective skin segmentation for different scene conditions. In this paper we present a real-time skin color detector which adapts itself according to tracked human parts. First, initial thresholds are calculated using two popular skin datasets. Those thresholds can also be calculated quickly using small training sets. The proposed skin color detector showed comparable skin segmentation to DeepLabV3++ application and an improvement in term of F1 measure when compared to methods that relies on static rules.

LIST: low illumination scene text detector with automatic feature enhancement

LIST: low illumination scene text detector with automatic feature enhancement

Quadratic Clustering-Based Simplex Volume Maximization for Hyperspectral Endmember Extraction

The existence of intra-class spectral variability caused by differential scene components and illumination conditions limits the improvement of endmember extraction accuracy, as most endmember extraction algorithms directly find pixels in the hyperspectral image as endmembers. This paper develops a quadratic clustering-based simplex volume maximization (CSVM) approach to effectively alleviate spectral variability and extract endmembers. CSVM first adopts spatial clustering based on simple linear iterative clustering to obtain a set of homogeneous partitions and uses spectral purity analysis to choose pure pixels. The average of the chosen pixels in each partition is taken as a representative endmember, which reduces the effect of local-scope spectral variability. Then an improved spectral clustering based on k-means is implemented to merge homologous representative endmembers to further reduce the effect of large-scope spectral variability, and final endmember collection is determined by the simplex with maximum volume. Experimental results show that CSVM reduces the average spectral angle distance on Samson, Jasper Ridge and Cuprite datasets to below 0.02, 0.06 and 0.09, respectively, provides the root mean square errors of abundance maps on Samson and Jasper Ridge datasets below 0.25 and 0.10, and exhibits good noise robustness. By contrast, CSVM provides better results than other state-of-the-art algorithms.

Illuminant segmentation for multi-illuminant scenes using latent illumination encoding

Color constancy is an important part of the Human Visual System that allows us to recognize colors of object invariant to the light that is illuminating them. Computational color constancy is the process of estimating the illumination of a scene using some computational method. However, this problem is inherently ill-posed. The RGB value of each pixel is dependent on spectral reflectance of the object and the spectral power distribution of the illumination. Hence, methods that try to solve the computational color constancy problem have to introduce assumptions about the illumination. One common assumption is that there is only one global source of illumination, i.e., that the illumination is constant across the whole scene. Under this assumption, modern color constancy methods achieve excellent results, usually predicting the illumination color with accuracy better than the human eye. This assumption is broken in many real-world multi-illuminant scenes, e.g., outdoor images where parts of the scene are illuminated by either sunlight or skylight. This leads to significant drop in accuracy of single-illuminant estimation methods. Therefore, in this work, we propose a novel method for segmenting images based on illumination in multi-illuminant scenes. This method detects regions where there is only one illuminant, thus detecting areas where the single-illuminant assumption holds. We show that our method produces excellent results and outperforms all baseline models by a large margin.

Space of basis functions to retrieve road surface reflection tables from luminance maps

Road surface reflection tables ( r-tables) relate scene illuminance to luminance seen by a car driver. They are important for many road lighting tasks accounting for road optical properties, such as new illumination design, new pavement texture or lighting design software, to reduce energy consumption without losses on safety and visibility. This paper aims first at finding a space of basis functions to describe r-tables. From a database of 34 r-tables covering a large variety of pavements, a principal component analysis allows to construct a 33-dimensional space, basis for r-table representations. From that statistical model, a method is exposed to retrieve r-table from a luminance map. The estimated r-table is then used to calculate a reconstructed luminance map. Road lighting quality criteria are also derived and they demonstrate the relevancy of the estimated r-table. Finally the model is tested with noisy input data and it remains stable and reliable, making it applicable with experimental luminance maps.