Structure-guided lensless reconstruction via physics-aware decomposition in low-light conditions

Lensless imaging offers a lightweight, compact alternative to traditional lens-based systems, ideal for exploration in space-constrained environments. However, the absence of a focusing lens and limited lighting in such environments often results in low-light conditions, where the measurements suffer from complex noise interference due to insufficient capture of photons. This study presents a robust reconstruction method for high-quality imaging in low-light scenarios, employing two complementary perspectives: model-driven and data-driven. First, we apply a physics-model-driven perspective to reconstruct the range space of the pseudo-inverse of the measurement model-as a first guidance to extract information in the noisy measurements. Then, we integrate a generative-model-based perspective to suppress residual noises-as the second guidance to suppress noises in the initial noisy results. Specifically, a learnable Wiener filter-based module generates an initial, noisy reconstruction. Then, for fast and, more importantly, stable generation of the clear image from the noisy version, we implement a modified conditional generative diffusion module. This module converts the raw image into the latent wavelet domain for efficiency and uses modified bidirectional training processes for stabilization. Simulations and real-world experiments demonstrate substantial improvements in overall visual quality, advancing lensless imaging in challenging low-light environments.

Night photography often struggles with challenges like low light and blurring, stemming from dark environments and prolonged exposures. Current methods either disregard priors and directly fitting end-to-end networks, leading to inconsistent illumination, or rely on unreliable handcrafted priors to constrain the network, thereby bringing the greater error to the final result. We believe in the strength of data-driven high-quality priors and strive to offer a reliable and consistent prior, circumventing the restrictions of manual priors. In this paper, we propose Clearer Night Image Restoration with Vector-Quantized Codebook (VQCNIR) to achieve remarkable and consistent restoration outcomes on real-world and synthetic benchmarks. To ensure the faithful restoration of details and illumination, we propose the incorporation of two essential modules: the Adaptive Illumination Enhancement Module (AIEM) and the Deformable Bi-directional Cross-Attention (DBCA) module. The AIEM leverages the inter-channel correlation of features to dynamically maintain illumination consistency between degraded features and high-quality codebook features. Meanwhile, the DBCA module effectively integrates texture and structural information through bi-directional cross-attention and deformable convolution, resulting in enhanced fine-grained detail and structural fidelity across parallel decoders. Extensive experiments validate the remarkable benefits of VQCNIR in enhancing image quality under low-light conditions, showcasing its state-of-the-art performance on both synthetic and real-world datasets. The code is available at https://github.com/AlexZou14/VQCNIR.

Real-world, imaging systems acquire measurements that are degraded by noise, optical aberrations, and other imperfections that make image processing for human viewing and higher-level perception tasks challenging. Conventional cameras address this problem by compartmentalizing imaging from high-level task processing. As such, conventional imaging involves processing the RAW sensor measurements in a sequential pipeline of steps, such as demosaicking, denoising, deblurring, tone-mapping, and compression. This pipeline is optimized to obtain a visually pleasing image. High-level processing, however, involves steps such as feature extraction, classification, tracking, and fusion. While this silo-ed design approach allows for efficient development, it also dictates compartmentalized performance metrics without knowledge of the higher-level task of the camera system. For example, today’s demosaicking and denoising algorithms are designed using perceptual image quality metrics but not with domain-specific tasks such as object detection in mind. We propose an end-to-end differentiable architecture that jointly performs demosaicking, denoising, deblurring, tone-mapping, and classification (see Figure 1). The architecture does not require any intermediate losses based on perceived image quality and learns processing pipelines whose outputs differ from those of existing ISPs optimized for perceptual quality, preserving fine detail at the cost of increased noise and artifacts. We show that state-of-the-art ISPs discard information that is essential in corner cases, such as extremely low-light conditions, where conventional imaging and perception stacks fail. We demonstrate on captured and simulated data that our model substantially improves perception in low light and other challenging conditions, which is imperative for real-world applications such as autonomous driving, robotics, and surveillance. Finally, we found that the proposed model also achieves state-of-the-art accuracy when optimized for image reconstruction in low-light conditions, validating the architecture itself as a potentially useful drop-in network for reconstruction and analysis tasks beyond the applications demonstrated in this work. Our proposed models, datasets, and calibration data are available at https://github.com/princeton-computational-imaging/DirtyPixels .

In order to achieve high-precision positioning of unmanned vehicles in low-light environments, based on the system framework of the VINS-Fusion algorithm, a fusion positioning algorithm LL- VI G for unmanned vehicles under low-light conditions is proposed. Aiming at the problems of low contrast, noise, and difficulty in feature extraction under low-light conditions, A multi-layer fusion image enhancement algorithm is proposed to improve the number of corner points extracted under low light conditions. For the problems of cumulative error in VI-SLAM and GNSS signals being easily interfered, a graph optimization method is used to integrate the GNSS global image. The fusion of positioning information and VI-SLAM positioning results reduces the cumulative error of VI-SLAM to a certain extent, and at the same time provides high-precision positioning in the absence of GNSS signals, improving the positioning accuracy and robustness of unmanned vehicles. The multi-layer fusion image enhancement algorithm proposed in this paper is experimentally verified based on the New Tsukuba Stereo dataset. The results show that the image enhanced by this algorithm can effectively increase the number of corner extractions. The LL-VIG algorithm proposed in this paper is experimentally verified based on the KITTI public data set and real vehicle scenarios. The results show that the positioning accuracy of LL- VI G is significantly higher than that of the comparison algorithm VINS-Fusion.

In low light condition, color (RGB) images captured by visible sensors suffer from severe noise causing loss of colors and textures. However, near infrared (NIR) images captured by NIR sensors are robust to noise even in low light condition without color. Since RGB and NIR images are complementary in low light condition, the multispectral fusion of RGB and NIR images provides a viable solution to low light imaging. In this paper, we propose multispectral fusion of RGB and NIR images using weighted least squares (WLS) and convolution neural networks (CNNs). We combine traditional WLS filtering for layer decomposition and denoising with latest deep learning for image enhancement and texture transfer into the multispectral fusion to take both advantages. We build two networks based on CNN: image enhancement network (IEN) for image enhancement and texture transfer network (TTN) for NIR texture transfer. First, we perform RGB image denoising based on WLS filtering and generate the base layer. We use both RGB and NIR images for WLS filtering as weights to filter out noise in low light RGB images. Second, we conduct IEN to enhance contrast of the base layer. Third, we perform TTN to deliver NIR details completely and naturally to the fusion result. The combination of WLS, TTN and IEN leads to noise reduction, contrast enhancement, and detail preservation in fusion. Experimental results show that the proposed method achieves good performance in both noise reduction and detail transfer as well as outperforms state-of-the-art methods in terms of visual quality and quantitative measurements.

In low light conditions, color (RGB) images taken by cameras contain a lot of noise and loss of detail and color. However, multispectral images can provide spectral information, which can be fused by neural network models . In this paper, an improved U-Net model is proposed for multi-spectral image fusion to achieve color imaging in low illumination environment. The U-Net model is a symmetric convolutional neural network that helps extract and combine features at various image scales. Our improved U-Net model integrates residual blocks and attention mechanisms, utilizing multilevel feature extraction and contextual information fusion to significantly enhance imaging quality. To meet the requirements of low-light conditions, the model design incorporates a multi-scale feature fusion strategy, bolstering robustness against weak light and noise. We conducted multiple experiments at different light levels to validate the effectiveness of the model. The quality of the fused color images was evaluated with objective assessment metrics such as peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and chromatic aberration (ΔE). The experimental results demonstrated the effectiveness of the proposed method, which can generate color images with high color reproduction and rich detail under low light conditions. Compared to traditional methods, our approach shows substantial improvements in image clarity, noise suppression, and color authenticity, indicating significant practical value. In summary, this study combines deep learning with multispectral image fusion to propose an effective method for low-light color imaging. It offers new insights and technical solutions for addressing low-light imaging challenges in practical applications.

In low light condition, color (RGB) images captured by increasing the camera ISO contain much noise and detail loss. However, near infrared (NIR) images are robust to noise and have clear textures without color. In this paper, we propose scale-aware multispectral fusion of RGB and NIR images based on alternating guidance. Low light RGB images provide large-scale image structure and color information, while NIR images have fine details lost in RGB images. Since they are complementary, we adopt alternating guidance for the fusion of them using weighted least squares (WLS). First, we perform the first guidance to denoise the RGB image and obtain base layer. Then, we conduct the second guidance for scale-aware detail transfer of the NIR image and yield detail layer. Finally, we combine the base and detail layers to generate a fusion image. We maximize the multispectral advantage of RGB and NIR images for fusion based on alternating guidance. Experimental results show that the proposed method achieves good performance in noise reduction, detail transfer and color reproduction, and is superior to the state-of-the-art ones in terms of quantitative measurement and computational efficiency.

Low-light wheat image enhancement using an explicit inter-channel sparse transformer

Multiple images with different exposures are used to produce a high dynamic range (HDR) image. Sometimes high-sensitivity setting is needed for capturing images in low light condition as in an indoor room. However, current digital cameras do not produce a high-quality HDR image when noise occurs in low light condition or high-sensitivity setting. In this paper, we propose a noise reduction method in generating HDR images using a set of low dynamic range (LDR) images with different exposures, where ghost artifacts are effectively removed by image registration and local motion information. In high-sensitivity setting, motion information is used in generating a HDR image. We analyze the characteristics of the proposed method and compare the performance of the proposed and existing HDR image generation methods, in which Reinhard et al.’s global tone mapping method is used for displaying the final HDR images. Experiments with several sets of test LDR images with different exposures show that the proposed method gives better performance than existing methods in terms of visual quality and computation time.

LVIF-Net: Learning synchronous visible and infrared image fusion and enhancement under low-light conditions

In the realms of the Internet of Things (IoT) and artificial intelligence (AI) security, ensuring the integrity and quality of visual data becomes paramount, especially under low-light conditions, where low-light image enhancement emerges as a crucial technology. However, the current methods for enhancing images under low-light conditions still face some challenging issues, including the inability to effectively handle uneven illumination distribution, suboptimal denoising performance, and insufficient correlation among a branch network. Addressing these issues, the Multi-Scale Branch Network is proposed. It utilizes multi-scale feature extraction to handle uneven illumination distribution, introduces denoising functions to mitigate noise issues arising from image enhancement, and establishes correlations between network branches to enhance information exchange. Additionally, our approach incorporates a vision transformer to enhance feature extraction and context understanding. The process begins with capturing raw RGB data, which are then optimized through sophisticated image signal processor (ISP) techniques, resulting in a refined visual output. This method significantly improves image brightness and reduces noise, achieving remarkable improvements in low-light image enhancement compared to similar methods. Using the LOL-V2-real dataset, we achieved improvements of 0.255 in PSNR and 0.23 in SSIM, with decreases of 0.003 in MAE and 0.009 in LPIPS, compared to the state-of-the-art methods. Rigorous experimentation confirmed the reliability of this approach in enhancing image quality under low-light conditions.

Photosynthetic light-use efficiency (LUE) has gained wide interest as an input to modeling forest gross primary productivity (GPP). The photochemical reflectance index (PRI) has been identified as a principle means to inform LUE-based models, using airborne and satellite-based observations of canopy reflectance. More recently, low-cost electronics have become available with the potential to provide for dense in situ time-series measurements of PRI. A recent design makes use of interference filters to record light transmission within narrow wavebands. Uncertainty remains as to the dynamic range of these sensors and performance under low light conditions, the placement of the reference band, and methodology for reflectance calibration. This paper presents a low-cost sensor design and is tested in a laboratory set-up, as well in the field. The results demonstrate an excellent performance against a calibration standard (R2 = 0.9999) and at low light conditions. Radiance measurements over vegetation demonstrate a reversible reduction in green reflectance that was, however, seen in both the reference and signal wavebands. Time-series field measurements of PRI in a Douglas-fir canopy showed a weak correlation with eddy-covariance-derived LUE and a significant decline in PRI over the season. Effects of light quality, bidirectional scattering effects, and possible sensor artifacts on PRI are discussed.

Since images captured under low light conditions have low dynamic range and are seriously degraded by noise, it is a challengeable task to achieve both contrast enhancement and noise reduction from low light images. In this paper, we propose a readability enhancement method of low light images based on dual-tree complex wavelet transform (DTCWT). We perform contrast enhancement and noise reduction for low light images based on wavelet coefficients. First, we conduct illumination compensation to contain fine details and fully utilize dynamic range. Then, we decompose the image into high-pass and low-pass sub-bands by DTCWT, and perform contrast limited adaptive histogram equalization (CLAHE) and a nonlinear transform in low-pass and high-pass sub-bands, respectively, to achieve both contrast enhancement and noise reduction. Finally, we perform color correction to deal with the color distortion problem caused by contrast enhancement. Experimental results demonstrate that the proposed method outperforms state-of-the-art ones in contrast enhancement, noise reduction, and color reproduction in terms of both subjective and objective evaluations.

The missing of graph attributes poses a significant challenge in graph representation learning. Some existing graph attribute completion methods adopt the shared-space hypothesis or employ end-to-end frameworks to perform single-attribute imputation. However, these models can only generate one single attribute with a few specific patterns that either adhere to prior knowledge or are optimal for downstream tasks, making it difficult to capture the full range of variations in the target attribute distribution. This limitation negatively impacts the model's generalizability and efficiency. Therefore, to address this issue, we proposed a new method based on a graph denoising diffusion model, called Multi-attribute Imputation Graph Denoising Diffusion Model (MIGDiff), which can generate multiple high-quality attributes. Specifically, it employs a Dual-source Auto-encoder on existing attributes and graph topology to extract reliable knowledge, which serves as a condition for training the diffusion module. Within diffusion, noise is added to the structural embeddings of nodes without attributes in the forward process. In the reverse process, a Structure-aware Denoising Network is devised to integrate feature and structural information via an attention mechanism and to perform neighbor-guided refinement based on graph connectivity, thereby enhancing denoising and accurately recovering missing attributes while effectively maintaining structural consistency and distributional fidelity. During generation, multiple initial values are sampled to produce diverse attribute imputations, avoiding focusing on a few easy-to-learn patterns. Extensive experiments conducted on four public datasets highlight the state-of-the-art performance of MIGDiff in both attribute imputation and node classification tasks.

Egocentric recognition is exciting computer vision research by acquiring images and video from the first-person overview. However, an image becomes noisy and dark under low illumination conditions, making subsequent hand detection tasks difficult. Thus, image enhancement is necessary to make buried detail more visible. This article addresses the challenge of egocentric hand grasp recognition in low light conditions by utilizing the flex sensor and image enhancement algorithm based on adaptive gamma correction with weighting distribution. Initially, a flex sensor is installed to the thumb for object manipulation. The thumb placement that holds in a different position on the object of each grasp affects the voltage changing of the flex sensor circuit. The average voltages are used to configure the weighting parameter to improve images in the image enhancement stage. Moreover, the contrast and gamma function are used to adjust varies the low light condition. These grasp images are then separated to be training and testing with pre-trained deep neural networks as the feature extractor in YOLOv2 detection network for the grasp recognition system. The proposed of using a flex sensor significantly improves the grasp recognition rate in low light conditions.