Low-light Scenes Research Articles

Modality and Camera Factors Bi-Disentanglement for NIR-VIS Object Re-Identification

Aiming to match object identities across different modalities of images, the challenging task namely cross-modality object re-identification (NIR-VIS object Re-ID), has attracted increasing attention due to its wide application in low-light scenes. However, dramatic modality-dependent and camera-related discrepancies between Near-InfraRed-spectrum (NIR) and VISible-spectrum (VIS) images lead to a considerable intra-class gap in feature space. To address the problem, we propose a novel Modality and Camera factors Bi-Disentanglement (MCBD) model to learn modality-independent and camera-unrelated features for NIR-VIS object Re-ID. Our model consists of three key modules, including Confused Modality Generation (CMG), Modality-independent Information Distillation (MID), and Cameras Factor Disentanglement (CFD). Firstly, aiming at aligning image style between NIR and VIS data, the CMG utilizes a designed channel-interactive generator to generate confused modality images which preserves the structure information of original images. Besides, CMG is trained with confused adversarial learning which bridges the modality gap at the image level. Nevertheless, training the model with the confused modality images discards identity-related information such as color and contrast, which is not conducive to the extraction of distinctive features. To solve this problem, the MID is presented to distill out modality-independent information by feeding the original images to the model and reconstructing corresponding NIR and VIS modalities features to the confused modality images. Finally, due to the complexity of camera-related information in images, identity representation inevitably contains camera-related elements such as background and perspective information, which may interfere the matching process. To address this issue, the CFD is introduced to disentangle camera-related factors by the designed three-streams network and two factor decoupling losses, i.e., Camera-Camera Factor loss (CCF) and Identity-Camera Factor (ICF) loss. Comprehensive experiments are carried out on two cross-modality pedestrian Re-ID datasets and a cross-modality vehicle Re-ID dataset to demonstrate that the MCBD is effective in cross-modality object Re-ID task.

Enhancing Conventional Geometry-Based Visual Odometry Pipeline Through Integration of Deep Descriptors

Geometry-based Visual Odometry (VO) techniques are renowned in the fields of computer vision and robotics. They use methods from multi-view geometry to estimate camera motion from visual data obtained from one or more cameras. Tracking the camera motion precisely between different views is dependent on the correct estimation of correspondences between salient points of the views. In practice, geometry-based methods are found to be quite effective but do not perform well in challenging cases due to tracking failures caused by abrupt motion, occlusions, textureless and low-light scenes, etc. On the contrary, end-to-end learning from visual data using deep neural networks is an emerging area of research and deals with challenging cases successfully. Despite being computationally expensive, these methods do not outperform their counterparts in conditions favorable to geometry-based methods. Considering these facts in this work, our goal is to integrate deep descriptors to improve the correspondence between image points for tracking in a traditional geometry-based VO pipeline. We propose a simple stereo VO pipeline inspired by popular techniques found in the literature. Two conventional and four deep descriptors have been used in our experiments conducted on various image sequences of the challenging KITTI benchmark dataset. We have determined empirically that deep descriptors can effectively minimize drift in the VO estimates and produce better camera trajectories. The experimental results on the KITTI dataset demonstrate that our VO method performs at par with the state-of-the-art works reported in the literature.

Interactive lighting editing system for single indoor low-light scene images with corresponding depth maps

We propose a novel interactive lighting editing system for lighting a single indoor RGB image based on spherical harmonic lighting. It allows users to intuitively edit illumination and relight the complicated low-light indoor scene. Our method not only achieves plausible global relighting but also enhances the local details of the complicated scene according to the spatially-varying spherical harmonic lighting, which only requires a single RGB image along with a corresponding depth map. To this end, we first present a joint optimization algorithm, which is based on the geometric optimization of the depth map and intrinsic image decomposition avoiding texture-copy, for refining the depth map and obtaining the shading map. Then we propose a lighting estimation method based on spherical harmonic lighting, which not only achieves the global illumination estimation of the scene, but also further enhances local details of the complicated scene. Finally, we use a simple and intuitive interactive method to edit the environment lighting map to adjust lighting and relight the scene. Through extensive experimental results, we demonstrate that our proposed approach is simple and intuitive for relighting the low-light indoor scene, and achieve state-of-the-art results.

Fusion of thermal and RGB images for automated deep learning based crack detection in civil infrastructure

Research has been continually growing toward the development of image-based structural health monitoring tools that can leverage deep learning models to automate damage detection in civil infrastructure. However, these tools are typically based on RGB images, which work well under ideal lighting conditions, but often have degrading performance in poor and low-light scenes. On the other hand, thermal images, while lacking in crispness of details, do not show the same degradation of performance in changing lighting conditions. The potential to enhance automated damage detection by fusing RGB and thermal images together within a deep learning network has yet to be explored. In this paper, RGB and thermal images are fused in a ResNET-based semantic segmentation model for vision-based inspections. A convolutional neural network is then employed to automatically identify damage defects in concrete. The model uses a thermal and RGB encoder to combine the features detected from both spectrums to improve its performance of the model, and a single decoder to predict the classes. The results suggest that this RGB-thermal fusion network outperforms the RGB-only network in the detection of cracks using the Intersection Over Union (IOU) performance metric. The RGB-thermal fusion model not only detected damage at a higher performance rate, but it also performed much better in differentiating the types of damage.

A predictive intelligence approach for low-light enhancement

Images captured in low-light scenes often suffer from poor contrast, severe loss of details and a lot of noise. These issues mentioned above can degrade the performance of some computer vision tasks such as object detection and recognition. The Retinex theory, which decomposes low-light images into reflection and illumination components, is an effective tool for predicting high-light images. Most Retinex-based methods cannot deal with noise reduction and over-smoothing simultaneously. In this paper, we propose a predictive intelligence approach of a multi-task framework to enhance the low-light image. The framework is composed of four coarse recovery sub-networks and three cross refinement sub-networks. Firstly, we use the LIME method to pre-process the low-light image and input the results to the coarse recovery sub-networks as priori knowledges along with the original low-light image. The residual connection and mixed-domain attention module are embedded in each coarse recovery sub-network so that the feature map processed by the attention mechanism is merged with the feature map from the main branch to improve the ability of predicting local bright color patches and effectively control the number of noises. The refinement sub-networks take the output of the coarse sub-networks as input and feed them into U-Net architectures in cross manner. The refinement sub-networks establish a multi-task framework with a cross-task distillation module that can significantly improve the predictive ability of the whole network. Extensive experiments demonstrate that our method achieves 4.1 in MOS, 19.82 in PSNR, 0.8470 in SSIM, 3.0271 in NIQE, and improves by more than 5% in both subjective metrics and objective metrics compared to state-of-the-art alternatives.

Optimized Bezier Curve Based Intensity Mapping Scheme for Low Light Image Enhancement

Low light enhancement has been very challenging for images acquired under extremely-low light circumstances. In this article, an optimized Bezier curve based intensity mapping scheme for low light image enhancement is proposed. The proposed algorithm is employed to adjust the intensity mapping curve in both dark and bright regions. To enhance the low visibility details and suppress the noise, the control points of the Bezier curve transformation are automatically obtained by the optimal weighting process. The optimal constraints of the weighted probability distribution function are calculated by whale optimization algorithm to solve the problem of under and over enhancement. Tests on different low light images are conducted to expose the efficiency and superiority of the proposed method over other existing methods in terms of improving the visual quality and bringing out the hidden details. The proposed method can preserve some minor details of the brighter portions and significantly reduce the intensive noise hidden in the low light scenes without any color distortion.

Enhancing underwater image via adaptive color and contrast enhancement, and denoising

Images captured under water are often characterized by low contrast, color distortion, and noise, hindering some visual tasks carried out on it. Despite remarkable breakthrough has been made in recent years, effective and robust enhancement of degraded image remains a challenging problem. To improve the quality of underwater images, we propose a novel scheme by constructing an adaptive color and contrast enhancement, and denoising (ACCE-D) framework. In the proposed framework, Difference of Gaussian (DoG) filter and bilateral filter are respectively employed to decompose the high-frequency and low-frequency components. Benefited from this separation, we utilize soft-thresholding operation to suppress the noise in the high-frequency component. Specially, the low-frequency component is enhanced by using an adaptive color and contrast enhancement (ACCE) strategy. Moreover, we derive a numerical solution for ACCE, and adopt a pyramid-based strategy to accelerate the solving procedure. Both qualitative and quantitative experiments demonstrate that our strategy is effective in color correction, contrast enhancement, and detail revealing. In the quantitative evaluations, by performing on the 890 real-world underwater images from UIEBD, the proposed method obtains 0.65 UCIQE, 1.59 UIQM, 0.81 FDUM, 1.34 PCQI, 0.62 CBPD, and 7.75 entropy scores, achieving average increase of 5% comparing with several state-of-the-art methods. Furthermore, we have verified the utility of our proposed ACCE-D for enhancing other types of degraded scenes, including foggy scene, sandstorm scene and low-light scene.

Acoustic detection of unmanned aerial vehicles using biologically inspired vision processing.

Robust detection of acoustically quiet, slow-moving, small unmanned aerial vehicles is challenging. A biologically inspired vision approach applied to the acoustic detection of unmanned aerial vehicles is proposed and demonstrated. The early vision system of insects significantly enhances signal-to-noise ratios in complex, cluttered, and low-light (noisy) scenes. Traditional time-frequency analysis allows acoustic signals to be visualized as images using spectrograms and correlograms. The signals of interest in these representations of acoustic signals, such as linearly related harmonics or broadband correlation peaks, essentially offer equivalence to meaningful image patterns immersed in noise. By applying a model of the photoreceptor stage of the hoverfly vision system, it is shown that the acoustic patterns can be enhanced and noise greatly suppressed. Compared with traditional narrowband and broadband techniques, the bio-inspired processing can extend the maximum detectable distance of the small and medium-sized unmanned aerial vehicles by between 30% and 50%, while simultaneously increasing the accuracy of flight parameter and trajectory estimations.

Learning With Nested Scene Modeling and Cooperative Architecture Search for Low-Light Vision.

Images captured from low-light scenes often suffer from severe degradations, including low visibility, color casts, intensive noises, etc. These factors not only degrade image qualities, but also affect the performance of downstream Low-Light Vision (LLV) applications. A variety of deep networks have been proposed to enhance the visual quality of low-light images. However, they mostly rely on significant architecture engineering and often suffer from the high computational burden. More importantly, it still lacks an efficient paradigm to uniformly handle various tasks in the LLV scenarios. To partially address the above issues, we establish Retinex-inspired Unrolling with Architecture Search (RUAS), a general learning framework, that can address low-light enhancement task, and has the flexibility to handle other challenging downstream vision tasks. Specifically, we first establish a nested optimization formulation, together with an unrolling strategy, to explore underlying principles of a series of LLV tasks. Furthermore, we design a differentiable strategy to cooperatively search specific scene and task architectures for RUAS. Last but not least, we demonstrate how to apply RUAS for both low- and high-level LLV applications (e.g., enhancement, detection and segmentation). Extensive experiments verify the flexibility, effectiveness, and efficiency of RUAS.

A Generalized Enhancement Framework for Hazy Images With Complex Illumination

Images captured under low-light conditions are generally characterized by poor illumination, low contrast, and nonignorable large amount of noise. In order to improve the visibility in weak illumination scenes, multiple artificial light sources are used, which leads to severe uneven illumination of the scene. The main challenges of dehazing images with complex illumination are to suppress the boosting of unsightly noise when enhancing contrast and avoid overenhancement in bright glow regions. To circumvent problems above, this letter proposes a generalized enhancement framework, which works well not only in uniform light conditions but also in strongly nonuniform illumination low-light scenes. To achieve this, we first decompose the input hazy image into a structure layer containing low-frequency illumination variance and a texture layer containing large amount of high-frequency details. Sequentially, benefit from two derived masks that are intrinsically similar to weight maps, the proposed framework can perform regional adaptive brightness adjustment on the structure layer according to the distribution of light in the input image. Meanwhile, regions of effective details in the texture layer are assigned higher weights, while regions that belong to noise are suppressed. Finally, adding the enhanced texture layer back to the brightened structure layer, visually appealing results are generated. Experimental results on various scenarios demonstrate the superiority of the proposed framework over state-of-the-art methods in terms of both qualitative and quantitative.

Adaptive color correction and detail restoration for underwater image enhancement.

Underwater images have different color casts due to different attenuation conditions, such as bluish, greenish, and yellowish. In addition, due to floating particles and special illumination, underwater images have problems such as the lack of detail and unnecessary noise. To handle the above problems, this paper proposes a new, to the best of our knowledge, three-step adaptive enhancement method. For the first step, adaptive color correction, the three channels are adjusted based on the intermediate color channel, which is calculated by considering the positional relationship of the histogram distribution. For the second step, denoise and restore details, we first transform the space to hue, saturation, value (HSV), a detailed restoration method based on the edge-preserving decomposition that restores the lost detail while removing the influence of some noise. For the third step, we improve the global contrast. Still in the HSV space, a simple linear stretch strategy is applied to the saturation channel. Experiments on the standard underwater image enhancement benchmark data set have proved that our method yields more natural colors and more valuable detailed information than several state-of-the-art methods. In addition, our method also improves the visibility of underwater images captured by low-light scenes and different hardware cameras.

Enhanced MSRCR optical frequency segmented filter algorithm for a low-light vehicle environment

To solve the problem of difficult face detection in a low illumination vehicle environment, a novel multi-scale retinex color restoration (MSRCR) approach exploiting the RGB three-channel decomposition and guided filtering (MSRCR-3CGF) is proposed. The MSRCR algorithm is employed to remove the artifacts and interference of low-light in the image based on the face detector using a multi-task cascaded convolutional neural network (MTCNN). The enhanced face image is decomposed into RGB, and GF is applied to each channel. The proposed method is tested on three widely used datasets: Dark Face, large-scale CelebFaces attributes (CelebA) and WIDER FACE, and an actual low-light scene in vehicles. The experimental results show that the proposed method suppresses the high-frequency noise of MSRCR, whilst improving the image enhancement and accuracy in the face detection in a low-light vehicle environment.

Generative adversarial network for low‐light image enhancement

Low-light image enhancement is rapidly gaining research attention due to the increasing demands of extreme visual tasks in various applications. Although numerous methods exist to enhance image qualities in low light, it is still undetermined how to trade-off between the human observation and computer vision processing. In this work, an effective generative adversarial network structure is proposed comprising both the densely residual block (DRB) and the enhancing block (EB) for low-light image enhancement. Specifically, the proposed end-to-end image enhancement method, consisting of a generator and a discriminator, is trained using the hyper loss function. The DRB adopts the residual and dense skip connections to connect and enhance the features extracted from different depths in the network while the EB receives unique multi-scale features to ensure feature diversity. Additionally, increasing the feature sizes allows the discriminator to further distinguish between fake and real images from the patch levels. The merits of the loss function are also studied to recover both contextual and local details. Extensive experimental results show that our method is capable of dealing with extremely low-light scenes and the realistic feature generator outperforms several state-of-the-art methods in a number of qualitative and quantitative evaluation tests.

TSFNet: Two-Stage Fusion Network for RGB-T Salient Object Detection

Salient object detection (SOD) based on convolutional neural networks has achieved remarkable success. However, further improving the detection performance on challenging scenes (e.g., low-light scenes) requires additional investigation. Thermal infrared imaging captures thermal radiation from the surface of objects. Thus, it is insensitive to lighting conditions and can provide uniform imaging of objects. Accordingly, we propose a two-stage fusion network (TSFNet) integrating RGB and thermal information for RGB-T SOD. For the first fusion stage, we propose a feature-wise fusion module that captures and aggregates united information and intersecting information in each local region of the RGB and thermal images, and then independent decoding is applied to the RGB and thermal features. For the second fusion stage, we propose a bilateral auxiliary fusion module that extracts auxiliary spatial features from the foreground and background of the thermal and RGB modalities. Finally, we use multiple supervision to further improve the SOD performance. Comprehensive experiments demonstrate that TSFNet outperforms 11 state-of-the-art models under various indicators on three RGB-T SOD datasets.

Gaze-Dependent Simulation of Light Perception in Virtual Reality.

The perception of light is inherently different inside a virtual reality (VR) or augmented reality (AR) simulation when compared to the real world. Conventional head-worn displays (HWDs) are not able to display the same high dynamic range of brightness and color as the human eye can perceive in the real world. To mimic the perception of real-world scenes in virtual scenes, it is crucial to reproduce the effects of incident light on the human visual system. In order to advance virtual simulations towards perceptual realism, we present an eye-tracked VR/AR simulation comprising effects for gaze-dependent temporal eye adaption, perceptual glare, visual acuity reduction, and scotopic color vision. Our simulation is based on medical expert knowledge and medical studies of the healthy human eye. We conducted the first user study comparing the perception of light in a real-world low-light scene to a VR simulation. Our results show that the proposed combination of simulated visual effects is well received by users and also indicate that an individual adaptation is necessary, because perception of light is highly subjective.

Strawberry Plant Alive Status Detection and Relative Pixel Based Plant Localization

Abstract Autonomous plant alive status monitoring and corresponding localization of individual plant are two important tasks in precision agriculture. In this study, two new methods that are crucial to such robotic operations are proposed. First, a low cost and light scene invariant approach is proposed to differentiate green and yellow leaves using distinct color-ratio index ranges. Second, based on the relative pixel information of neighboring plants, an extended Kalman filter is used to determine plant positions. Such a differential style localization method is shown to be capable of achieving a similar centimeter level accuracy as light detection and ranging (LIDAR) or real-time kinematic-global positioning system (RTK-GPS) based approaches, but with a much lower upfront and maintenance cost. These two new methods are successfully validated in a nearby commercial field.

Multi-Frame RGB/NIR Imaging for Low-Light Color Image Super-Resolution

We propose a method for super-resolution (SR) of low-resolution (LR) color images taken in low-light scenes. Our method is based on the multi-frame SR technique, which reconstructs a high-resolution color image by fusing multiple LR images that were taken at different camera positions so as to alter the alignment between scene details and pixels. Previous multi-frame SR methods have implicitly assumed that the LR images could be captured with less noise and blur. However, images taken in low-light scene will contain high level of noise and motion blur, making it difficult to achieve high quality SR. To overcome these problems, we utilize a single sensor that captures red, green, blue (RGB) and near-infrared (NIR) information. We capture multiple raw images of a low-light scene using our RGB/NIR single sensor with an NIR flash unit. Because a NIR flash can provide sufficient illumination for low-light scenes, the structural scene information that, contributes to an effective SR process, can be captured with high quality. With the help of the NIR information, we jointly perform deblurring, denoising and SR of LR RGB/NIR raw images. Our experiments demonstrate the effectiveness of the proposed method using synthetic and real RGB/NIR raw images.

Sensitivity Improvement of Extremely Low Light Scenes with RGB-NIR Multispectral Filter Array Sensor.

Recently, several red-green-blue near-infrared (RGB-NIR) multispectral filter arrays (MFAs), which include near infrared (NIR) pixels, have been proposed. For extremely low light scenes, the RGB-NIR MFA sensor has been extended to receive NIR light, by adding NIR pixels to supplement for the insufficient visible band light energy. However, the resolution reconstruction of the RGB-NIR MFA, using demosaicing and color restoration methods, is based on the correlation between the NIR pixels and the pixels of other colors; this does not improve the RGB channel sensitivity with respect to the NIR channel sensitivity. In this paper, we propose a color restored image post-processing method to improve the sensitivity and resolution of an RGB-NIR MFA. Although several linear regression based color channel reconstruction methods have taken advantage of the high sensitivity NIR channel, it is difficult to accurately estimate the linear coefficients because of the high level of noise in the color channels under extremely low light conditions. The proposed method solves this problem in three steps: guided filtering, based on the linear similarity between the NIR and color channels, edge preserving smoothing to improve the accuracy of linear coefficient estimation, and residual compensation for lost spatial resolution information. The results show that the proposed method is effective, while maintaining the NIR pixel resolution characteristics, and improving the sensitivity in terms of the signal-to-noise ratio by approximately 13 dB.

Learning in the dark: 3D integral imaging object recognition in very low illumination conditions using convolutional neural networks

We propose a framework for three-dimensional (3D) object recognition and classification in very low illumination environments using convolutional neural networks (CNNs). 3D images are reconstructed using 3D integral imaging (InIm) with conventional visible spectrum image sensors. After imaging the low light scene using 3D InIm, the 3D reconstructed image has a higher signal-to-noise ratio than a single 2D image, which is a result of 3D InIm being optimal in the maximum likelihood sense for read-noise dominant images. Once 3D reconstruction has been performed, the 3D image is denoised and regions of interest are extracted to detect 3D objects in a scene. The extracted regions are then inputted into a CNN, which was trained under low illumination conditions using 3D InIm reconstructed images, to perform object recognition. To the best of our knowledge, this is the first report of utilizing 3D InIm and convolutional neural networks for 3D training and 3D object classification under very low illumination conditions.

Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High-Speed Scenarios

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. These cameras do not suffer from motion blur and have a very high dynamic range, which enables them to provide reliable visual information during high speed motions or in scenes characterized by high dynamic range. However, event cameras output only little information when the amount of motion is limited, such as in the case of almost still motion. Conversely, standard cameras provide instant and rich information about the environment most of the time (in low-speed and good lighting scenarios), but they fail severely in case of fast motions, or difficult lighting such as high dynamic range or low light scenes. In this paper, we present the first state estimation pipeline that leverages the complementary advantages of these two sensors by fusing in a tightly-coupled manner events, standard frames, and inertial measurements. We show on the publicly available Event Camera Dataset that our hybrid pipeline leads to an accuracy improvement of 130% over event-only pipelines, and 85% over standard-frames-only visual-inertial systems, while still being computationally tractable. Furthermore, we use our pipeline to demonstrate - to the best of our knowledge - the first autonomous quadrotor flight using an event camera for state estimation, unlocking flight scenarios that were not reachable with traditional visual-inertial odometry, such as low-light environments and high-dynamic range scenes.