Image Enhancement Tasks Research Articles

Mitigating gradient conflicts via expert squads in multi-task learning

The foundation of multi-task learning lies in the collaboration and interaction among tasks. However, in numerous real-world scenarios, certain tasks usually necessitate distinct, specialized knowledge. The mixing of these different task-specific knowledge often results in gradient conflicts during the optimization process, posing a significant challenge in the design of effective multi-task learning systems. This study proposes a straightforward yet effective multi-task learning framework that employs groups of expert networks to decouple the learning of task-specific knowledge and mitigate such gradient conflicts. Specifically, this approach partitions the feature channels into task-specific and shared components. The task-specific subsets are processed by dedicated experts to distill specialized knowledge. The shared features are captured by a point-wise aggregation layer from the whole outputs of all experts, demonstrating superior performance in capturing inter-task interactions. By considering both task-specific knowledge and shared features, the proposed approach exhibits superior performance in multi-task learning. Extensive experiments conducted on the PASCAL-Context and NYUD-v2 datasets have demonstrated the superiority of the proposed approach compared to other state-of-the-art methods. Furthermore, a benchmark dataset for multi-task learning in underwater scenarios has been developed, encompassing object detection and underwater image enhancement tasks. Comprehensive experiments on this dataset consistently validate the effectiveness of the proposed multi-task learning strategy. The source code is available at https://github.com/chenjie04/Multi-Task-Learning-PyTorch.

SFDiff: Diffusion model with sufficient spatial‐Fourier frequency information interaction for low‐light image enhancement

AbstractDiffusion models are increasingly applied in low‐light image enhancement tasks due to their exceptional capability to model data distributions, but most current methods focus only on the original pixel space and neglect the potential of Fourier frequency information. In this article, SFDiff is proposed, a novel low‐light image enhancement method that integrates Fourier frequency information into the diffusion process. Specifically, Fourier transforms are applied at both the image and feature levels to separately enhance the amplitude and phase components, which restores global illumination degradation and positional information. Then a Spatial‐Frequency Fusion (SFF) block is used to fully integrate and interact with the information across spatial and frequency domains. Since illumination degradation is primarily manifested in the amplitude component, a loss function based on maximum likelihood learning is employed to constrain the amplitude component at each step of the sampling process, ensuring that the reverse process maintains an optimal trajectory. Owing to the streamlined network design and the fact that the Fourier transform requires no extra parameters, SFDiff achieves a reduction in parameters of over compared to several state‐of‐the‐art (SOTA) diffusion models and delivers high‐quality enhancement results on multiple real‐world datasets. The code is available at https://github.com/MrWan001/SFDiff.

DCPNet: Deformable Control Point Network for image enhancement

In this paper, we present a novel image enhancement network consisting of global and local color enhancement. The proposed network model is constructed using global transformation functions, which are formed by a set of piece-wise quadratic curves and a local color enhancement network based on the encoder–decoder network. To adaptively and dynamically control the ranges of each piece-wise curve, we introduce deformable control points (DCPs), which determine the overall structure of the global transformation functions. The parameters for piece-wise quadratic curve fitting and DCPs are estimated using the proposed DCP network (DCPNet). DCPNet processes a down-sampled image to derive the DCP parameters: The DCP offsets and the curve parameters. Then, we obtain a set of DCPs from the DCP offsets and connect each adjacent DCP pair by using the curve parameter to construct a global transformation function for each color channel. The original input images are then transformed based on the resulting transformation functions to obtain globally enhanced images. Finally, the intermediate image is fed into the local enhancement network, which has a U-Net architecture, to produce the spatially enhanced images. Extensive experimental results demonstrate the superiority of the proposed method over state-of-the-art methods in various image enhancement tasks, such as image retouching, tone-mapping, and underwater image enhancement.

SeBIR: Semantic-guided burst image restoration

Burst image restoration methods offer the possibility of recovering faithful scene details from multiple low-quality snapshots captured by hand-held devices in adverse scenarios, thereby attracting increasing attention in recent years. However, individual frames in a burst typically suffer from inter-frame misalignments, leading to ghosting artifacts. Besides, existing methods indiscriminately handle all burst frames, struggling to seamlessly remove the corrupted information due to the neglect of multi-frame spatio-temporal varying degradation. To alleviate these limitations, we propose a general semantic-guided model named SeBIR for burst image restoration incorporating the semantic prior knowledge of Segment Anything Model (SAM) to enable adaptive recovery. Specifically, instead of relying solely on a single aligning scheme, we develop a joint implicit and explicit strategy that sufficiently leverages semantic knowledge as guidance to achieve inter-frame alignment. To further adaptively modulate and aggregate aligned features with spatio-temporal disparity, we elaborate a semantic-guided fusion module using the intermediate semantic features of SAM as an explicit guide to weaken the inherent degradation and strengthen the valuable complementary information across frames. Additionally, a semantic-guided local loss is designed to boost local consistency and image quality. Extensive experiments on synthetic and real-world datasets demonstrate the superiority of our method in both quantitative and qualitative evaluations for burst super-resolution, burst denoising, and burst low-light image enhancement tasks.

BGFlow: Brightness-guided normalizing flow for low-light image enhancement

Low-light image enhancement poses significant challenges due to its ill-posed nature. Recently, deep learning-based methods have attempted to establish a unified mapping relationship between normal-light images and their low-light versions but frequently struggle to capture the intricate variations in brightness conditions. As a result, these methods often suffer from overexposure, underexposure, amplified noise, and distorted colors. To tackle these issues, we propose a brightness-guided normalizing flow framework, dubbed BGFlow, for low-light image enhancement. Specifically, we recognize that low-frequency sub-bands in the wavelet domain carry significant brightness information. To effectively capture the intricate variations in brightness within an image, we design a transformer-based multi-scale wavelet-domain encoder to extract brightness information from the multi-scale features of the low-frequency sub-bands. The extracted brightness feature maps, at different scales, are then injected into the brightness-guided affine coupling layer to guide the training of the conditional normalizing flow module. Extensive experimental evaluations demonstrate the superiority of BGFlow over existing deep learning-based approaches in both qualitative and quantitative assessments. Moreover, we also showcase the exceptional performance of BGFlow on the underwater image enhancement task.

LL-Diff: Low-Light Image Enhancement Utilizing Langevin Sampling Diffusion

In this paper, we propose a new algorithm called LL-Diff, which is innovative compared to traditional augmentation methods in that it introduces the sampling method of Langevin dynamics. This sampling approach simulates the motion of particles in complex environments and can better handle noise and details in low-light conditions. We also incorporate a causal attention mechanism to achieve causality and address the issue of confounding effects. This attention mechanism enables us to better capture local information while avoiding over-enhancement. We have conducted experiments on the LOL-V1 and LOL-V2 datasets, and the results show that LL-Diff significantly improves computational speed and several evaluation metrics, demonstrating the superiority and effectiveness of our method for low-light image enhancement tasks. The code will be released on GitHub when the paper has been accepted.

SwiftWater: A lightweight SwiftFormer-based underwater image enhancement network with auxiliary prompt module

Due to variations in light attenuation underwater and the complexities of light propagation in aquatic environments, underwater images commonly suffer from degradation, such as color distortion, blur, and low contrast. These challenges pose significant obstacles to researchers studying marine resources. In recent years, numerous solutions have been proposed for underwater image enhancement (UIE) tasks, with deep learning-based methods, including those employing convolution and transformer structures, demonstrating promising performance. However, deep learning approaches often encounter issues related to large parameter sizes and substantial computational demands. Addressing these concerns, this paper introduces SwiftWater, a lightweight transformer-based network specifically designed for UIE tasks, featuring a U-Net style architecture. Leveraging the efficient SwiftFormer encoder as the network backbone, we introduce a Hybrid Squeeze and Excitation Block (HSEB) to dynamically adjust representations within skip connections. Additionally, we introduce an Auxiliary Prompt Module (APM) with a Lightweight Prompt Block (LPB) that utilizes the HSV color space of underwater images to enhance training. LPB within APM comprises two modules: the Prompt Generation Module (PGM) for prompt generation and the Lightweight Prompt Interaction Module (LPIM) for prompt-feature interaction, generating rich auxiliary image representations. Compared to existing mainstream UIE methods employing transformer structures, our proposed approach offers reduced parameter counts while achieving state-of-the-art (SOTA) performance across both quantitative and qualitative metrics.

Underwater image enhancement based on global features and prior distribution guided

Underwater images often suffer from substantial image blur and color distortion due to the variability of water conditions and the physical location of optical equipment, which significantly impacts the underwater intelligent system's environmental perception. Standard methods exhibit limited generalization capabilities, leading to considerable performance fluctuations when handling images with uncontrolled degradation. In this research, we leverage global features and the prior distribution of ground truth images to guide our enhancement model, introducing a novel conditional Variational Auto-Encoder-based model, named UWG-VAE, to address these challenges. UWG-VAE enhances model controllability by incorporating prior distribution information and classes of degraded styles into the decoder of the enhancement model. We assess the performance of UWG-VAE in underwater image enhancement tasks across four challenging real underwater image datasets, comparing it to state-of-the-art models. UWG-VAE demonstrates a substantial enhancement in visual quality, with notable improvements in UIQM, UCIQE, and URanker evaluation metrics when compared to existing state-of-the-art models.

Object detection on low-resolution images with two-stage enhancement

Although deep learning-based object detection methods have achieved superior performance on conventional benchmark datasets, it is still difficult to detect objects from low-resolution (LR) images under diverse degradation conditions. To this end, a two-stage enhancement method for the LR image object detection (TELOD) framework is proposed. In the first stage, an extremely lightweight task disentanglement enhancement network (TDEN) is developed as a super-resolution (SR) sub-network before the detector. In the TDEN, the SR images can be obtained by applying the recurrent connection manner between an image restoration branch (IRB) and a resolution enhancement branch (REB) to enhance the input LR images. Specifically, the TDEN reduces the difficulty of image reconstruction by dividing the total image enhancement task into two sub-tasks, which are accomplished by the IRB and REB, respectively. Furthermore, a shared feature extractor is applied across two sub-tasks to explore common and accurate feature representations. In the second stage, an auxiliary feature enhancement head (AFEH) driven by high-resolution (HR) image priors is designed to improve the task-specific features produced by the detection Neck without any extra inference costs. In particular, the feature interaction module is built into the AFEH to integrate the features from the enhancement and detection phases to learn comprehensive information for detection. Extensive experiments show that the proposed TELOD significantly outperforms other methods. Specifically, the TELOD achieves mAP improvements of 1.8% and 3.3% over the second best method AERIS on degraded VOC and COCO datasets, respectively.

DAF‐Retinex: Preserve the image detailed features and restore the reflected image

AbstractCurrently, deep learning methods for low‐light image enhancement tasks mainly focus on the illumination of images, while neglecting the problems of image noise and feature loss. To address this issue, this paper proposes a novel low‐light image enhancement network called DAF‐Retinex, based on the Retinex‐Net. To address the issue of image noise, different from traditional image denoise methods, this paper utilizes a fully convolutional neural network to denoise the reflection component, additionally, a denoising loss function is introduced to suppress noise. For preserving image details and extracting features, this paper creatively introduces self‐calibrated convolutions into low‐light image enhancement tasks, furthermore, a feature augmented attention block consisting of feature‐guided attention (FGA) is designed for feature learning to effectively enhance image illumination and extract image detail features. Experimental results demonstrate that the proposed algorithm in this paper effectively removes image noise and extracts detailed features, resulting in visually improved outcomes. On public datasets, the average improvement in objective evaluation metrics of image quality such as PSNR, SSIM, and NIQE are 1.13%, 4.12%, and 1.28%, respectively.

Enhanced pyramidal residual networks for single image super-resolution

Several super-resolution (SR) techniques are introduced in the literature, including traditional and machine learning-based algorithms. Especially, deep learning-based SR approaches emerge with demands for better quality images providing deeper subpixel enhancement. Dealing with the image enhancement task in the satellite images domain, a new SR method for single image SR, namely Enhanced Deep Pyramidal Residual Networks, is introduced in this study. The proposed method overcomes the potential instability problem of Enhanced Deep Residual Networks for Single Image Super-Resolution (EDSR) approach by gradually increasing the feature maps depending upon Pyramidal Residual Networks architecture. The EDSR itself is a good algorithm in the SR domain. However, it has a strict structure for increasing the block size. To overcome this problem with the aim of increasing the algorithm’s performance, the pyramidal residual networks gradually increasing hypothesis is utilized in the proposed approach, which is the main contribution and novelty of this study. Besides, by using the pyramidal residual networks gradually increasing hypothesis in the proposed approach, the parameter size of the models is also reduced, which affects the computational time. Two different models are proposed by considering addition and multiplication manners, and the proposed models are evaluated using well-known remote sensing datasets NWPU-RESISC45 and UC Merced. The results obtained by the proposed model are compared with the results of traditional image enhancement algorithms together with the EDSR itself, EDSR with deeper structure, Super-Resolution Generative Adversarial Networks approach, and Residual Local Feature Networks approach in terms of peak signal to noise ratio (PSNR) and structural similarity index measure (SSIM) metrics and showed that the proposed models present better quality images. Moreover, considering the computational time and complexity, it is shown that some proposed models achieve approximately 27% less output parameter having similar PSNR and SSIM values and computational time for EDSR itself and 65% less output parameter having better PSNR and SSIM values and 16% lower computational time for EDSR with deeper structure.

Night Time Image Enhancement

Night time image enhancement plays a crucial role in various applications such as surveillance, autonomous driving, and photography. However, capturing high-quality images in low-light conditions remains challenging due to limited visibility and increased noise levels. In this project, we propose a novel approach for enhancing nighttime images using MIRNet, a state-of-the-art deep learning architecture specifically designed for low-light image enhancement tasks. We collect a dataset of low-light images paired with their corresponding well-exposed counterparts and train the MIRNet model to learn the mapping between the two modalities. The architecture of MIRNet incorporates convolutional layers with residual connections to effectively capture low-light image features and generate visually pleasing enhancements. We evaluate the performance of our approach on a diverse range of nighttime scenes and compare the results against existing methods. Our experiments demonstrate that MIRNet produces superior results in enhancing nighttime images, significantly improving visibility, reducing noise, and preserving image details. The proposed approach holds promise for real-world applications where high-quality nighttime imagery is essential for decision-making and visual analysis. Keywords: Night time image enhancement, MIRNet, Deep learning, Low-light imaging, Image Processing,Convolutional neural networks (CNNs),Residual connections, Supervised learning, Dataset preparation,Imagequalityimprovement,Noisereduction,Visibilityenhancement,Surveillance,Autonomousdriving,Photography.

Multi-Domain Multi-Scale Diffusion Model for Low-Light Image Enhancement

Diffusion models have achieved remarkable progress in low-light image enhancement. However, there remain two practical limitations: (1) existing methods mainly focus on the spatial domain for the diffusion process, while neglecting the essential features in the frequency domain; (2) conventional patch-based sampling strategy inevitably leads to severe checkerboard artifacts due to the uneven overlapping. To address these limitations in one go, we propose a Multi-Domain Multi-Scale (MDMS) diffusion model for low-light image enhancement. In particular, we introduce a spatial-frequency fusion module to seamlessly integrates spatial and frequency information. By leveraging the Multi-Domain Learning (MDL) paradigm, our proposed model is endowed with the capability to adaptively facilitate noise distribution learning, thereby enhancing the quality of the generated images. Meanwhile, we propose a Multi-Scale Sampling (MSS) strategy that follows a divide-ensemble manner by merging the restored patches under different resolutions. Such a multi-scale learning paradigm explicitly derives patch information from different granularities, thus leading to smoother boundaries. Furthermore, we empirically adopt the Bright Channel Prior (BCP) which indicates natural statistical regularity as an additional restoration guidance. Experimental results on LOL and LOLv2 datasets demonstrate that our method achieves state-of-the-art performance for the low-light image enhancement task. Codes are available at https://github.com/Oliiveralien/MDMS.

SBIT-Fuse: Infrared and visible image fusion based on Symmetrical Bilateral interaction and Transformer

Infrared and visible image fusion, as a typical image enhancement task, aims to generate an image containing complementary information. Existing deep learning-based methods usually use a dual-stream model to extract features, which suffers from insufficient feature interaction in the feature extraction process. In order to effectively utilize information, this work proposes a Symmetrical Bilateral Interaction and Transformer fusion method (SBIT-Fuse) that is simple and efficient for constructing a dual-stream interaction network. Specifically, inspired by the problem of dead ReLU, we propose a Symmetrical Bilateral Interaction (SBI) module which consists of a certain number of cascaded cross domain activation interaction (CDAI) layers, where the deactivation information of the ReLU rectifier is transferred from one flow to another instead of being discarded. The proposed module exploits the drawbacks of ReLU to efficiently build the interaction of two streams and combine spatial and channel attention mechanisms, achieving the enhancement of feature extraction. Besides, our network is end-to-end and a transformer module is employed for extracting long-range semantic information. Moreover, combining a novel designed loss function, the proposed method can generate higher-quality fusion images. The comparative experiments with existing advanced methods and ablation studies on publicly datasets demonstrate the effectiveness of the proposed approach. Our code can be found at https://github.com/ljx111790/SBIT-Fuse.

Cycle-Retinex: Unpaired Low-Light Image Enhancement via Retinex-Inline CycleGAN

Low-light image enhancement aims to recover normal-light images from the images captured under dim environments. Most existing methods could just improve the light appearance globally whereas failing to handle other degradation such as dense noise, color offset and extremely low-light. Moreover, unsupervised methods proposed in recent years lack reliable physical model as the basis, thus universality is greatly limited. To address these problems, we propose a novel low-light image enhancement method via Retinex-inline cycle-consistent generative adversarial network named Cycle-Retinex, whose training is totally dependent on unpaired datasets. Specifically, we organically combine Retinex theory with CycleGAN, by which we decouple low-light image enhancement task into two sub-tasks, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> illumination map enhancement and reflectance map restoration. Retinex theory helps CycleGAN simplify low-light image enhancement problem and CycleGAN provides synthetic paired images to guide the training of Retinex decomposition network. We further introduce a self-augmented method to address the color distortion and noise problem, thus making the network learn to enhance low-light images adaptively. Extensive experiments show that the proposed method can achieve promising results. The source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/mummmml/Cycle-Retinex</uri> .

ENHANCEMENT OF MEDICAL MRI IMAGES BASED ON FRACTAL OPERATORS

This article presents the research of texture enhancement algorithms on medical images. Medical MRI brain scans contain large areas with low level grey colors that carry useful information for doctors. Texture improvement allow to highlight large grey areas on images for future detailed recognition. Based on the study of existing texture enhancement methods, it was determined that fractal operators are effective for processing medical images. The mathematical framework of fractal operators is presented based on the approximation equation of the Grünwald-Letnikov fractional derivatives. The creation of fractal differential masks and the algorithm of masks usage for image enhancement are described based on this equation. The approximation error of the Grunwald-Letnikov derivative is calculated in comparison with the analytical value of the Grunwald-Letnikov derivative. The algorithm based on the fractal derivative shows improvements in image parameters such as contrast, correlation, energy, and homogeneity compared to the original image parameters. A comparison of the results of the algorithm based on the fractal differential with other algorithms for improving the texture of images is also given. It is concluded that the fractal differential algorithm is well-suited for MRI image enhancement tasks, unlike other algorithms, both in visual comparisons and numerical metrics, and thus can be applied to solve real-world problems.

DedustGAN: Unpaired learning for image dedusting based on Retinex with GANs

Image dedusting has gained increasing attention as a preprocessing step for computer vision tasks. Current traditional image dedusting methods rely on a variety of constraints or priors, which are easy to be limited in real complex dusty scenes. Recently, some paired learning CNN methods have been proposed. However, they usually imitate the idea of image dehazing and fail to consider the inherent natural characteristics of dusty images. To bridge this gap, we propose an unpaired learning algorithm for image dedusting based on Retinex with GANs (DedustGAN). Specifically, we first design the DedustGAN with two branches, one branch uses Retinex to obtain one preliminary dedusted image in a physical way, and the other branch uses GANs to obtain another preliminary dedusted image in a learning way. Then, we design a no-reference image fusion strategy to fuse the two preliminary dedusted images for outputting the final dedusted image. The proposed DedustGAN breaks the restriction of existing paired learning methods that require paired images for network training and provide an effective physical dedusting model. Qualitative and quantitative experimental comparisons demonstrate the powerful dedusting ability of the Retinex model and the superiority of DedustGAN over the existing methods. In addition, to validate the strong generalization of the proposed DedustGAN, we further performed experiments on underwater image enhancement and low-light image enhancement tasks. We found that the effectiveness of DedustGAN even outperforms the existing methods in their domains. The code of the proposed DedustGAN is available at https://github.com/MXL696/DedustGAN.

UIESC: An Underwater Image Enhancement Framework via Self-Attention and Contrastive Learning

Low contrast, color distortion and blurred details are common problems that perplex vision-guided underwater robots. To this end, we propose an underwater image enhancement framework via self-attention and contrastive learning (UIESC) to solve these problems. In this work, local features and global dependencies are constructed through space and channel dual attention, and criss-cross attention is used to solve the high computational complexity of self-attention. Moreover, contrastive learning is introduced into network training as a loss function, and contrastive regularization ensures that the enhanced images are closer to clear positive samples and away from source negative samples. Finally, smoothed-histogram equalization is adopted for further optimization to accommodate complex and variable underwater scenes. Extensive experiments have shown that our framework outperforms state-of-the-art methods in underwater image enhancement tasks.

LAE-GAN: a novel cloud-based Low-light Attention Enhancement Generative Adversarial Network for unpaired text images

With the widespread adoption of mobile multimedia devices, the deployment of compute-intensive inference tasks on edge and resource-constrained devices, particularly in the context of low-light text detection, remains a formidable challenge. Existing deep learning approaches have shown limited effectiveness in restoring images for extremely dark scenes. To address these limitations, this paper presents a novel cloud-based Low-light Attention Enhancement Generative Adversarial Network for unpaired text images (LAE-GAN) for the non-paired text image enhancement task in extremely low-light conditions. In the first stage, compressed low-light images are transmitted from edge devices to a cloud server for image enhancement. The LAE-GAN, an end-to-end network comprising a Zero-DCE and AGM-net generator, is designed with a global and local discriminator structure. The initial illumination restoration of extremely low-light images is accomplished using the Zero-DCE network. To enhance text details, we propose an Enhanced Text Attention Mechanism (ETAM) that transforms text information into a comprehensive text attention mechanism across the entire network. The Sobel operator is employed to extract text edge information, while attention is focused on text region details through constraints imposed on the attention map and edge map. Additionally, an AGM-Net module is integrated to reduce noise and fine-tune illumination. In the second stage, the cloud server makes decisions based on user requirements and processes requests in parallel, scaling with the quantity of requests. In the third stage, the enhanced results are transmitted back to edge devices for text detection. Experimental results on widely used LOL and SID low-light datasets demonstrate significant improvements in both quantitative and qualitative analysis, surpassing state-of-the-art enhancement methods in terms of image restoration and text detection.

Robust underwater image enhancement with cascaded multi-level sub-networks and triple attention mechanism

With the growing exploration of marine resources, underwater image enhancement has gained significant attention. Recent advances in convolutional neural networks (CNN) have greatly impacted underwater image enhancement techniques. However, conventional CNN-based methods typically employ a single network structure, which may compromise robustness in challenging conditions. Additionally, commonly used UNet networks generally force fusion from low to high resolution for each layer, leading to inaccurate contextual information encoding. To address these issues, we propose a novel network called Cascaded Network with Multi-level Sub-networks (CNMS), which encompasses the following key components: (a) a cascade mechanism based on local modules and global networks for extracting feature representations with richer semantics and enhanced spatial precision, (b) information exchange between different resolution streams, and (c) a triple attention module for extracting attention-based features. CNMS selectively cascades multiple sub-networks through triple attention modules to extract distinct features from underwater images, bolstering the network’s robustness and improving generalization capabilities. Within the sub-network, we introduce a Multi-level Sub-network (MSN) that spans multiple resolution streams, combining contextual information from various scales while preserving the original underwater images’ high-resolution spatial details. Comprehensive experiments on multiple underwater datasets demonstrate that CNMS outperforms state-of-the-art methods in image enhancement tasks.