Image Enhancement Tasks Research Articles

Multi-Domain Rapid Enhancement Networks for Underwater Images

Images captured during marine engineering operations suffer from color distortion and low contrast. Underwater image enhancement helps to alleviate these problems. Many deep learning models can infer multi-source data, where images with different perspectives exist from multiple sources. To this end, we propose a multichannel deep convolutional neural network (MDCNN) linked to a VGG that can target multi-source (multi-domain) underwater image enhancement. The designed MDCNN feeds data from different domains into separate channels and implements parameters by linking VGGs, which improves the domain adaptation of the model. In addition, to optimize performance, multi-domain image perception loss functions, multilabel soft edge loss for specific image enhancement tasks, pixel-level loss, and external monitoring loss for edge sharpness preprocessing are proposed. These loss functions are set to effectively enhance the structural and textural similarity of underwater images. A series of qualitative and quantitative experiments demonstrate that our model is superior to the state-of-the-art Shallow UWnet in terms of UIQM, and the performance evaluation conducted on different datasets increased by 0.11 on average.

Dual Cross-Attention Multi-Stage Embedding Network for Low-Light Image Enhancement

The low-light image enhancement task aims to improve the visibility of information in the dark to obtain more data and utilize it, while also improving the visual quality of the image. In this paper, we propose a dual cross-attention multi-stage embedding network (DCMENet) for fast and accurate enhancement of low-light images into high-quality images with high visibility. The problem that enhanced images tend to have more noise in them, which affects the image quality, is improved by introducing an attention mechanism in the encoder–decoder structure. In addition, the encoder–decoder can focus most of its attention on the dark areas of the image and better attend to the detailed features in the image that are obscured by the dark areas. In particular, the poor performance of the Transformer when the dataset size is small is solved by fusing the CNN-Attention and Transformer in the encoder. Considering the purpose of the low-light image enhancement task, we raise the importance of recovering image detail information to the same level as reconstructing the lighting. For features such as texture details in images, cascade extraction using spatial attention and pixel attention can reduce the model complexity while the performance is also improved. Finally, the global features obtained by the encoder–decoder are fused into the shallow feature extraction structure to reconstruct the illumination while guiding the network for the focused extraction of information in the dark. The proposed DCMENet achieves the best results in both objective quality assessment and subjective evaluation, while for the computer vision tasks working in low-light environments as well, the enhanced images using the DCMENet proposed in this paper show the best performance.

Beyond Staircasing Effect: Robust Image Smoothing via ℓ0 Gradient Minimization and Novel Gradient Constraints

In this paper, we propose robust image-smoothing methods based on ℓ0 gradient minimization with novel gradient constraints to effectively suppress pseudo-edges. Simultaneously minimizing the ℓ0 gradient, i.e., the number of nonzero gradients in an image, and the ℓ2 data fidelity results in a smooth image. However, this optimization often leads to undesirable artifacts, such as pseudo-edges, known as the “staircasing effect”, and halos, which become more visible in image enhancement tasks, like detail enhancement and tone mapping. To address these issues, we introduce two types of gradient constraints: box and ball. These constraints are applied using a reference image (e.g., the input image is used as a reference for image smoothing) to suppress pseudo-edges in homogeneous regions and the blurring effect around strong edges. We also present an ℓ0 gradient minimization problem based on the box-/ball-type gradient constraints using an alternating direction method of multipliers (ADMM). Experimental results on important applications of ℓ0 gradient minimization demonstrate the advantages of our proposed methods compared to existing ℓ0 gradient-based approaches.

DLL-GAN: Degradation-level-based learnable adversarial loss for image enhancement

Image enhancement has recently gained considerable attention owing to its benefits in cleaning input images before final processing and, thus, helping decision systems. The outstanding success of generative adversarial networks has been exploited in image enhancement to handle various image generation challenges, and complex image enhancers incorporating new architectures and/or the use of metrics as loss functions have been proposed. However, degradation-aware loss functions that would help the generator deal with degradation have not been investigated. In this study, we propose a new adversarial learning paradigm, referred to as a degradation-level-based learnable generative adversarial network (DLL-GAN). In addition to the discriminator and L1 losses, a new loss is included in the DLL-GAN; it is calculated by using a convolutional-neural-network-based regressor that estimates the amount of degradation that is still present in the generator output and must be optimized during the learning process. The DLL-GAN is validated on three widely adopted and challenging image enhancement tasks: super-resolution, denoising, and JPEG artifact removal. An extensive experimental analysis reveals that the DLL-GAN improves the baseline performance through significant gains, which are confirmed in visual results. The DLL-GAN outperforms the state-of-the-art methods in three applications, especially at high and challenging degradation levels.

Multi-image transformer for multi-focus image fusion

Multi-Focus Image Fusion (MFIF) is an image enhancement task that fuses images in which different regions are in focus to achieve an all-in-focus image. In recent years, Generative Adversarial Networks (GANs)-based approaches have significantly improved the MFIF on Convolutional Neural Network (CNN) architectures. However, despite vision transformers (ViTs) achieving more successful results than CNNs in many high and low-level vision problems due to their ability to provide global connectivity, they have not been employed for MFIF until this study. We develop a Multi-image Transformer (MiT) for MFIF by being inspired by a Spatial-Temporal Transformer Network (STTN) so that global connection can be modeled along multiple input images. We call the proposed transformers-based MFIF model MiT-MFIF as it uses the developed MiT as a core component. We have made various modifications to the baseline transformer to be able to utilize vision transformers in MFIF tasks. Comprehensive experiments on standard MFIF datasets demonstrate the effectiveness of the proposed MiT-MFIF. Moreover, proposed method does not require any post-processing step like in competitor GAN-based methods while outperforming the state-of-the-art MFIF methods.

A Semi-Supervised Network Framework for low-light image enhancement

Existing supervised learning-based low-light image enhancement algorithms treat all degradations as a whole, resulting in limited enhancement performance. It is difficult for fully unsupervised learning to recover more hidden details, making the augmented results unsatisfactory for visual needs. To overwhelm the limitations of supervised and unsupervised learning, we propose a Semi-Supervised Network Framework (SSNF) to enhance low-light images. Specifically, we decouple the low-light image enhancement task into two stages. In the first stage of the SSNF, we employ methods based on information entropy and Retinex to improve the visibility of images. It is worth mentioning that this stage is a lightweight self-supervised network, which only needs to input low-light images and undergo minute-level training to achieve brightness improvement. In the second stage of the SSNF, we utilize U-Net and residual networks to remove problems such as noise and degradation existing in the first-stage enhancement results, thereby improving the visual properties of the enhanced images. It overwhelms the challenge of dealing with low-light images directly. We conduct extensive experiments on datasets such as LOL, synthetic, DICM, etc. The experimental results show that SSNF exhibits better visual effects and outperforms other advanced methods in performance metrics.

Triangle-Mesh-Rasterization-Projection (TMRP): An Algorithm to Project a Point Cloud onto a Consistent, Dense and Accurate 2D Raster Image.

The projection of a point cloud onto a 2D camera image is relevant in the case of various image analysis and enhancement tasks, e.g., (i) in multimodal image processing for data fusion, (ii) in robotic applications and in scene analysis, and (iii) for deep neural networks to generate real datasets with ground truth. The challenges of the current single-shot projection methods, such as simple state-of-the-art projection, conventional, polygon, and deep learning-based upsampling methods or closed source SDK functions of low-cost depth cameras, have been identified. We developed a new way to project point clouds onto a dense, accurate 2D raster image, called Triangle-Mesh-Rasterization-Projection (TMRP). The only gaps that the 2D image still contains with our method are valid gaps that result from the physical limits of the capturing cameras. Dense accuracy is achieved by simultaneously using the 2D neighborhood information (rx,ry) of the 3D coordinates in addition to the points P(X,Y,V). In this way, a fast triangulation interpolation can be performed. The interpolation weights are determined using sub-triangles. Compared to single-shot methods, our algorithm is able to solve the following challenges. This means that: (1) no false gaps or false neighborhoods are generated, (2) the density is XYZ independent, and (3) ambiguities are eliminated. Our TMRP method is also open source, freely available on GitHub, and can be applied to almost any sensor or modality. We also demonstrate the usefulness of our method with four use cases by using the KITTI-2012 dataset or sensors with different modalities. Our goal is to improve recognition tasks and processing optimization in the perception of transparent objects for robotic manufacturing processes.

A transformer-based network for perceptual contrastive underwater image enhancement

Vision-based underwater image enhancement methods have received much attention for application in the fields of marine engineering and marine science. The absorption and scattering of light in real underwater scenes leads to severe information degradation in the acquired underwater images, thus limiting further development of underwater tasks. To solve these problems, a novel transformer-based perceptual contrastive network for underwater image enhancement methods (TPC-UIE) is proposed to achieve visually friendly and high-quality images, where contrastive learning is applied to the underwater image enhancement (UIE) task for the first time. Specifically, to address the limitations of the pure convolution-based network, we embed the transformer into the UIE network to improve its ability to capture global dependencies. Then, the limits of the transformer are then taken into account as convolution is reintroduced to better capture local attention. At the same time, the dual-attention module strengthens the network’s focus on the spatial and color channels that are more severely attenuated. Finally, a perceptual contrastive regularization method is proposed, where a multi-loss function made up of reconstruction loss, perceptual loss, and contrastive loss jointly optimizes the model to simultaneously ensure texture detail, contrast, and color consistency. Experimental results on several existing datasets show that the TPC-UIE obtains excellent performance in both subjective and objective evaluations compared to other methods. In addition, the visual quality of the underwater images is significantly improved by the enhancement of the method and effectively facilitates further development of the underwater task.

Unsupervised Low-Light Image Enhancement via Virtual Diffraction Information in Frequency Domain

With the advent of deep learning, significant progress has been made in low-light image enhancement methods. However, deep learning requires enormous paired training data, which is challenging to capture in real-world scenarios. To address this limitation, this paper presents a novel unsupervised low-light image enhancement method, which first introduces the frequency-domain features of images in low-light image enhancement tasks. Our work is inspired by imagining a digital image as a spatially varying metaphoric “field of light”, then subjecting the influence of physical processes such as diffraction and coherent detection back onto the original image space via a frequency-domain to spatial-domain transformation (inverse Fourier transform). However, the mathematical model created by this physical process still requires complex manual tuning of the parameters for different scene conditions to achieve the best adjustment. Therefore, we proposed a dual-branch convolution network to estimate pixel-wise and high-order spatial interactions for dynamic range adjustment of the frequency feature of the given low-light image. Guided by the frequency feature from the “field of light” and parameter estimation networks, our method enables dynamic enhancement of low-light images. Extensive experiments have shown that our method performs well compared to state-of-the-art unsupervised methods, and its performance approximates the level of the state-of-the-art supervised methods qualitatively and quantitatively. At the same time, the light network structure design allows the proposed method to have extremely fast inference speed (near 150 FPS on an NVIDIA 3090 Ti GPU for an image of size 600×400×3). Furthermore, the potential benefits of our method to object detection in the dark are discussed.

Ultra-High-Definition Low-Light Image Enhancement: A Benchmark and Transformer-Based Method

As the quality of optical sensors improves, there is a need for processing large-scale images. In particular, the ability of devices to capture ultra-high definition (UHD) images and video places new demands on the image processing pipeline. In this paper, we consider the task of low-light image enhancement (LLIE) and introduce a large-scale database consisting of images at 4K and 8K resolution. We conduct systematic benchmarking studies and provide a comparison of current LLIE algorithms. As a second contribution, we introduce LLFormer, a transformer-based low-light enhancement method. The core components of LLFormer are the axis-based multi-head self-attention and cross-layer attention fusion block, which significantly reduces the linear complexity. Extensive experiments on the new dataset and existing public datasets show that LLFormer outperforms state-of-the-art methods. We also show that employing existing LLIE methods trained on our benchmark as a pre-processing step significantly improves the performance of downstream tasks, e.g., face detection in low-light conditions. The source code and pre-trained models are available at https://github.com/TaoWangzj/LLFormer.

Zero-referenced low-light image enhancement with adaptive filter network

The current strive toward efficient intelligent visual systems suffers from challenges in the task of low-light image enhancement. To improve image perception, the low-light scenes in different illumination conditions must be properly focused. However, typical CNN-based methods use the same set of parameters for all images, which limits the capability for handling complex scenes. Meanwhile, the existing deep models integrate the low-level and high-level features by simply adding or concatenating operations, lacking unique designs for the low-light image enhancement task. To address the above challenges, we propose a zero-referenced adaptive filter network (ZAFN) for low-light image enhancement. Specifically, the adaptive filters are generated by the integration of high-level contents from multiple partial scenes. The iterative enlightening process is then conducted using the low-level features that are dynamically modulated with the adaptive filters. To alleviate the requirement of paired training data and enable zero-referenced learning, we propose a color enhancement loss, a global consistency loss, and a self-regularized denoising loss for high-quality results. Our ZAFN model, which has a light model size and low computational cost, outperforms other state-of-the-art zero-referenced methods on four popular datasets.

LEPF-Net: Light Enhancement Pixel Fusion Network for Underwater Image Enhancement

Underwater images often suffer from degradation due to scattering and absorption. With the development of artificial intelligence, fully supervised learning-based models have been widely adopted to solve this problem. However, the enhancement performance is susceptible to the quality of the reference images, which is more pronounced in underwater image enhancement tasks because the ground truths are not available. In this paper, we propose a light-enhanced pixel fusion network (LEPF-Net) to solve this problem. Specifically, we first introduce a novel light enhancement block (LEB) based on the residual block (RB) and the light enhancement curve (LE-Curve) to restore the cast color of the images. The RB is adopted to learn and obtain the feature maps from an original input image, and the LE-Curve is used to renovate the color cast of the learned images. To realize the superb detail of the repaired images, which is superior to the reference images, we develop a pixel fusion subnetwork (PF-SubNet) that adopts a pixel attention mechanism (PAM) to eliminate noise from the underwater image. The PAM adapts weight allocation to different levels of a feature map, which leads to an enhancement in the visibility of severely degraded areas. The experimental results show that the proposed LEPF-Net outperforms most of the existing underwater image enhancement methods. Furthermore, among the five classic no-reference image quality assessment (NRIQA) indicators, the enhanced images obtained by LEPF-Net are of higher quality than the ground truths from the UIEB dataset.

Multiple transformation function estimation for image enhancement

Most deep learning-based image enhancement algorithms have been developed based on the image-to-image translation approach, in which enhancement processes are difficult to interpret. In this paper, we propose a novel interpretable image enhancement algorithm that estimates multiple transformation functions to describe complex color mapping. First, we develop a histogram-based multiple transformation function estimation network (HMTF-Net) to estimate multiple transformation functions by exploiting both the spatial and statistical information of the input images. Second, we estimate pixel-wise weight maps, which indicate the contribution of each transformation function at each pixel, based on the local structures of the input image and the transformed images obtained by each transformation function. Finally, we obtain the enhanced image as the weighted sum of the transformed images using the estimated weight maps. Extensive experiments confirm the effectiveness of the proposed approach and demonstrate that the proposed algorithm outperforms state-of-the-art image enhancement algorithms for different image enhancement tasks.

Underwater image enhancement combining dual color space and contrast learning

With the continuous increase of human society’s demand for marine resources, underwater vision plays an increasingly important role in marine information acquisition. However, due to the absorption and scattering of light, underwater images have characteristics such as blurring, low contrast, and color distortion, which are not conducive to the acquisition of underwater image information. The rapid development of deep learning and powerful feature learning capabilities have been widely used in underwater image enhancement tasks, but current research still has problems such as loss of local details and oversaturation. Aiming at the above problems, this paper proposes an underwater image enhancement method combining dual color space and contrast learning, including RGB color space and Lab color space. The RGB color space is used for color correction and artifact removal, and the Lab color space is used to enhance saturation and detail texture. The contrastive learning of positive and negative samples makes the generated image closer to the clear underwater image and deviates from the real underwater image. Experimental results on real underwater datasets show that the proposed method achieves the best results on PSNR, SSIM and UIQM compared to other underwater image enhancement methods.

Area Contrast Distribution Loss for Underwater Image Enhancement

In this paper, we aim to design a lightweight underwater image enhancement algorithm that can effectively solve the problem of color distortion and low contrast in underwater images. Recently, enhancement methods typically optimize a perceptual loss function, using high-level features extracted from pre-trained networks to train a feed-forward network for image enhancement tasks. This loss function measures the perceptual and semantic differences between images, but it is applied globally across the entire image and does not consider semantic information within the image, which limits the effectiveness of the perceptual loss. Therefore, we propose an area contrast distribution loss (ACDL), which trains a flow model to achieve real-time optimization of the difference between output and reference in training. Additionally, we propose a novel lightweight neural network. Because underwater image acquisition is difficult, our experiments have shown that our model training can use only half the amount of data and half the image size compared to Shallow-UWnet. The RepNet network reduces the parameter size by at least 48% compared to previous algorithms, and the inference time is 5 times faster than before. After incorporating ACDL, SSIM increased by 2.70% and PSNR increased by 9.72%.

SurroundNet: Towards effective low-light image enhancement

Although Convolution Neural Networks (CNNs) have made substantial progress in the low-light image enhancement task, one critical problem of CNNs is the paradox of model complexity and performance. This paper presents a novel SurroundNet that only involves less than 150K parameters (about 80–98 percent size reduction compared to SOTAs) and achieves very competitive performance. The proposed network comprises several Adaptive Retinex Blocks (ARBlock), which can be viewed as a novel extension of Single Scale Retinex in feature space. The core of our ARBlock is an efficient illumination estimation function called Adaptive Surround Function (ASF). It can be regarded as a general form of surround functions and be implemented by convolution layers. In addition, we also introduce a Low-Exposure Denoiser (LED) to smooth the low-light image before the enhancement. We evaluate the proposed method on two real-world low-light datasets. Experimental results demonstrate the superiority of our submitted SurroundNet in both performance and network parameters against State-of-the-Art low-light image enhancement methods. The code is available at https://github.com/ouc-ocean-group/SurroundNet.

All-in-one aerial image enhancement network for forest scenes.

Drone monitoring plays an irreplaceable and significant role in forest firefighting due to its characteristics of wide-range observation and real-time messaging. However, aerial images are often susceptible to different degradation problems before performing high-level visual tasks including but not limited to smoke detection, fire classification, and regional localization. Recently, the majority of image enhancement methods are centered around particular types of degradation, necessitating the memory unit to accommodate different models for distinct scenarios in practical applications. Furthermore, such a paradigm requires wasted computational and storage resources to determine the type of degradation, making it difficult to meet the real-time and lightweight requirements of real-world scenarios. In this paper, we propose an All-in-one Image Enhancement Network (AIENet) that can restore various degraded images in one network. Specifically, we design a new multi-scale receptive field image enhancement block, which can better reconstruct high-resolution details of target regions of different sizes. In particular, this plug-and-play module enables it to be embedded in any learning-based model. And it has better flexibility and generalization in practical applications. This paper takes three challenging image enhancement tasks encountered in drone monitoring as examples, whereby we conduct task-specific and all-in-one image enhancement experiments on a synthetic forest dataset. The results show that the proposed AIENet outperforms the state-of-the-art image enhancement algorithms quantitatively and qualitatively. Furthermore, extra experiments on high-level vision detection also show the promising performance of our method compared with some recent baselines.

PUGAN: Physical Model-Guided Underwater Image Enhancement Using GAN with Dual-Discriminators.

Due to the light absorption and scattering induced by the water medium, underwater images usually suffer from some degradation problems, such as low contrast, color distortion, and blurring details, which aggravate the difficulty of downstream underwater understanding tasks. Therefore, how to obtain clear and visually pleasant images has become a common concern of people, and the task of underwater image enhancement (UIE) has also emerged as the times require. Among existing UIE methods, Generative Adversarial Networks (GANs) based methods perform well in visual aesthetics, while the physical model-based methods have better scene adaptability. Inheriting the advantages of the above two types of models, we propose a physical model-guided GAN model for UIE in this paper, referred to as PUGAN. The entire network is under the GAN architecture. On the one hand, we design a Parameters Estimation subnetwork (Par-subnet) to learn the parameters for physical model inversion, and use the generated color enhancement image as auxiliary information for the Two-Stream Interaction Enhancement sub-network (TSIE-subnet). Meanwhile, we design a Degradation Quantization (DQ) module in TSIE-subnet to quantize scene degradation, thereby achieving reinforcing enhancement of key regions. On the other hand, we design the Dual-Discriminators for the style-content adversarial constraint, promoting the authenticity and visual aesthetics of the results. Extensive experiments on three benchmark datasets demonstrate that our PUGAN outperforms state-of-the-art methods in both qualitative and quantitative metrics. The code and results can be found from the link of https://rmcong.github.io/proj PUGAN.html.

Trinity-Net: Gradient-Guided Swin Transformer-Based Remote Sensing Image Dehazing and Beyond

Haze superimposes a veil over remote sensing images, which severely limits the extraction of valuable military information. To this end, we present a novel trinity model to restore realistic surface information by integrating the merits of both prior-based and deep learning-based strategies. Concretely, the critical insight of our Trinity-Net is to investigate how to incorporate prior information into CNNs and Swin Transformer for reasonable estimation of haze parameters. Then, haze-free images are obtained by reconstructing the remote sensing image formation model. Although Swin Transformer has shown tremendous potential in the dehazing task, which typically results in ambiguous details. We devise a gradient guidance module that naturally inherits structure priors of gradient maps, guiding the deep model to generate visually pleasing details. In light of the generality of image formation parameters, we successfully promote Trinity-Net to natural image dehazing and underwater image enhancement tasks. Notably, the acquisition of large-scale remote sensing hazy images and natural hazy images in military scenes is not feasible in practice. To bridge this gap, we construct a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Remote Sensing Image Dehazing Benchmark</i> (RSID) and a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Natural Image Dehazing Benchmark</i> (NID), including 1000 real-world hazy images with corresponding ground truth images, respectively. To our knowledge, this is the first exploration to develop dehazing benchmarks in the military field, alleviating the dilemma of data scarcity. Extensive experiments on three vision tasks illustrate the superiority of our Trinity-Net against multiple state-of-the-art methods. The datasets and code are available at https://github.com/chi-kaichen/Trinity-Net.

Discriminator feature-based progressive GAN inversion

Generative Adversarial Networks (GANs) receive extensive attention due to its capability of high-fidelity image synthesis. The synthesized content can be edited by imposing transformations on the latent codes. To edit real images, the GAN inversion methods mainly focus on learning an inverse mapping from the data space to the latent space of a well-trained GAN through a separate encoder. In this paper, we propose a Discriminator Feature-based Progressive Inversion (DFPI) model for GAN-based image reconstruction and enhancement. We find that the expressiveness of the generator feature is richer than that of the latent code for a given input image. To achieve high-quality reconstruction, we estimate the generator features in a light-weight optimization process, conditioned on off-the-shelf discriminator features. Specifically, a recomposition matrix is learnt to map high-level discriminator features to the high-level generator features due to their reverse structure. To progressively improve the reconstruction quality, we adopt a convolutional correction module to alleviate the estimation error at a lower generator layer, conditioned on the features of a lower discriminator layer. By performing the progressive correction process, the input images can be accurately reconstructed. Note that the resulting recomposition matrix and correction module can be generalized to unseen images. The experiments demonstrate the superior performance of DFPI over competing methods on GAN inversion as well as a variety of image enhancement tasks.