Image Restoration Research Articles

An efficient hybrid conjugate gradient method with an adaptive strategy and applications in image restoration problems

In this study, we introduce a novel hybrid conjugate gradient method with an adaptive strategy called asHCG method. The asHCG method exhibits the following characteristics. (i) Its search direction guarantees sufficient descent property without dependence on any line search. (ii) It possesses strong convergence for the uniformly convex function using a weak Wolfe line search, and under the same line search, it achieves global convergence for the general function. (iii) Employing the Armijo line search, it provides an approximate guarantee for worst-case complexity for the uniformly convex function. The numerical results demonstrate promising and encouraging performances in both unconstrained optimization problems and image restoration problems.

Restoration of Semantic-Based Super-Resolution Aerial Images

Currently, technologies for remote sensing image processing are actively developing, including both satellite images and aerial images obtained from video cameras of unmanned aerial vehicles. Often such images have artifacts such as low resolution, blurred image fragments, noise, etc. One way to overcome such limitations is to use modern technologies to restore super-resolution images based on deep learning methods. The specificity of aerial images is the presentation of texture and structural elements in a higher resolution than in satellite images, which objectively contributes to better results of restoration. The article provides a classification of super-resolution methods based on the main architectures of deep neural networks, namely convolutional neural networks, visual transformers and generative adversarial networks. The article proposes a method for reconstructing super-resolution aerial images SemESRGAN taking into account semantic features by using an additional deep network for semantic segmentation during the training stage. The total loss function, including adversarial losses, pixel-level losses, and perception losses (feature similarity), is minimized. Six annotated aerial and satellite image datasets CLCD, DOTA, LEVIR-CD, UAVid, AAD, and AID were used for the experiments. The results of image restoration using the proposed SemESRGAN method were compared with the basic architectures of convolutional neural networks, visual transformers and generative adversarial networks. Comparative results of image restoration were obtained using objective metrics PSNR and SSIM, which made it possible to evaluate the quality of restoration using various deep network models.

MDTGAN: Multi domain generative adversarial transfer learning network for traffic data imputation

Accurate, real-time, and efficient traffic data, crucial for intelligent transportation systems, which is often disrupted in the real world due to the influence of weather, interference, and equipment failures. Generative Adversarial Networks (GANs), have achieved significant results in image restoration, providing insights for traffic data imputation. Recent research has focused on developing more effective and reasonable model by learning transferable common knowledge from different cities or different scenarios, which is also an excellent idea for improving the data imputation performance. In light of, we contrive to design a GANs based transferable traffic data imputation model, namely Multi Domain Generative Adversarial Transfer Learning Network (MDTGAN). Firstly, the model consists of two stages. In pre-training stage, the model utilizes the source domain datasets (Similar cities road network datasets) to update parameters and transferred them. Then, the model parameters are optimized using the target domain dataset (Target city road network dataset) in fine-tuning stage to avoid the issue of insufficient samples caused by data missing. Secondly, to adapt the model to different dataset topologies, MDTGAN models the node-level data by taking node time series as input, enabling the model to autonomously learn the prevalent spatiotemporal correlations inherent in nodes. Additionally, we introduce a domain discriminator module that guides the spatial encoder in learning domain-invariant features of network node spatial information, enhancing the model generalization ability. The experiments conducted on three publicly available datasets, in which one dataset is regarded as the target dataset, while the others serve as source datasets. The experimental results demonstrate that the MDTGAN model consistently outperforms the baseline models. Specifically, the MDTGAN model can transfer valuable knowledge from the source domain datasets to improve data imputation performance on the target domain dataset, providing practical significance for cities lacking historical traffic data.

SFMnet: Edge detection of HABs based on spatial feature mapping encoder-decoder network

Achieving high-accuracy real-time recognition of marine red tide algae images is crucial in implementing red tide algae early warning systems. However, due to uneven lighting and the deposition of unknown suspended particles in seawater, real-time collected harmful algal blooms (HABs) microscopic images in marine engineering suffer from poor clarity and blurred cell contours, leading to lower accuracy of current edge detection models for microscopic images. Firstly, to enhance the quality of degraded images in the original HABs dataset, we design an image enhancement algorithm based on underwater image restoration models, which involves precise depth estimation of seawater scenes and fusion of transmission maps. This algorithm effectively eliminates chromatic aberration and enhances image edge details and contrast. Secondly, we propose a deep-learning model based on the VGG16 network. This model integrates multi-scale features through the Spatial Feature Mapping module to improve edge detection accuracy. Experimental results using real-time HABs microscopic image data show that the proposed enhancement method achieves the best results regarding contrast, image enhancement metrics, and clarity gradient. Moreover, the edge detection method achieves an ODS of 0.636 and an OIS of 0.701. The results demonstrate that performing edge detection after the image enhancement preprocess significantly improves detection accuracy.

Research on a multi‐scale degradation fusion network in all‐in‐one image restoration

AbstractImage restoration aims to recover high‐quality clean images from degraded low‐quality ones. Deep learning‐based approaches have been a focal point in the field of image restoration. However, most methods focus solely on a single type of degradation and may not extend well to real‐world scenarios with unknown degradation. For this purpose, the present study introduces an all‐in‐one image restoration approach by designing a multi‐scale feature fusion UNet structure (MdfUNet). In summary, the proposed method exhibits two significant advantages. For starters, it implicitly fuses degradation information across multiple scales, enabling the network to extract rich hierarchical features and enhancing its generalization ability towards unknown degradations. Secondly, MdfUnet possesses strong image reconstruction capabilities. It utilizes a simple non‐linear feature optimizer to enhance skip connections, providing rich feature representations for the image reconstruction process, and ultimately generating high‐quality restored images. Extensive experimental results show the proposed method outperforms multiple baselines on deraining, dehazing, and denoising datasets.

A family of spectral conjugate gradient methods with strong convergence and its applications in image restoration and machine learning

In this paper, we propose a family of spectral conjugate gradient methods for solving unconstrained optimization problems. Specifically, we provide two classes of bounded spectral parameters to be chosen, design a new truncation scheme of the non-negative conjugate parameter and set a restart procedure in our search direction. Independently of the specific spectral parameter, conjugate parameter and line search criterion, we prove that our proposed family satisfies the sufficient descent condition. We also prove its strong convergence under mild assumptions and the weak Wolfe line search. Numerical comparisons with other methods demonstrate the outstanding performances of our algorithm for solving medium–large-scale unconstrained optimization, image restoration and machine learning problems.

A deep blur learning-based motion artifact reduction algorithm for dental cone-beam computed tomography images

We propose a motion artifact correction algorithm (DMBL) for reducing motion artifacts in reconstructed dental cone-beam computed tomography (CBCT) images based on deep blur learning. A blur encoder was used to extract motion-related degradation features to model the degradation process caused by motion, and the obtained motion degradation features were imported in the artifact correction module for artifact removal. The artifact correction module adopts a joint learning framework for image blur removal and image blur simulation for treatment of spatially varying and random motion patterns. Comparative experiments were conducted to verify the effectiveness of the proposed method using both simulated motion data sets and clinical data sets. The experimental results with the simulated dataset showed that compared with the existing methods, the PSNR of the proposed method increased by 2.88%, the SSIM increased by 0.89%, and the RMSE decreased by 10.58%. The results with the clinical dataset showed that the proposed method achieved the highest expert level with a subjective image quality score of 4.417 (in a 5-point scale), significantly higher than those of the comparison methods. The proposed DMBL algorithm with a deep blur joint learning network structure can effectively reduce motion artifacts in dental CBCT images and achieve high-quality image restoration.

Edge-preserving image deraining network using cumulative feature aggregation

This study proposes an edge-preserving image deraining network using a wavelet feature aggregation method. Wavelet subbands re correlated with each other, and the high-frequency subband in the horizontal direction is the least affected by rain streak contamination. On this basis, we introduce a single image deraining network that cumulatively aggregates wavelet subband features according to their importance. The network architecture primarily comprises a wavelet feature aggregation block and a residue channel guide block. The aggregation of features with the cumulative wavelet feature aggregation block moves downward and upward, and a long short-term memory-based multiscale attentive rain streak removal block is developed to serve as the backbone for rain streak removal. We use a residual channel map based on the low-frequency subband to construct guide features that assist in rain streak removal. A repetitive image restoration framework that incorporates two proposed blocks is used to iteratively improve rainy images. We test the proposed network on various image datasets and compare the deraining performance with those of existing methods The experimental results demonstrate that the performance of the proposed scheme is superior to that of other tested deraining methods. https://github.com/SYChoi98/CWFANet/

Innovative design of wood texture images for indoor furniture based on variable space

In the design of furniture wood texture images, image restoration is a key issue. This study proposes a Bregmanized operator splitting optimization algorithm based on variable space. This study combines variable spatial morphology to process texture images and effectively extract image features using different operators, thereby achieving image restoration. The results of comparing the proposed algorithm with other image processing algorithms showed that the research algorithm achieved a peak signal-to-noise ratio of 29.86 and a structural similarity index of 0.87 in image denoising, respectively, and had a good denoising effect. In terms of image deblurring, the research algorithm had the lowest root mean square error values on the France and Boat datasets, with values of 8.98 and 8.82, respectively, indicating that the image processed by the algorithm had a high similarity with the real image. In terms of image resolution reconstruction, the peak signal-to-noise ratio and root mean square error values of the research algorithm reached 29.74 and 12.67, respectively, indicating that the reconstructed image had the best fit with the original image and the smallest error. In summary, the proposed algorithm has shown good performance in image processing and can be effectively applied in fields such as image denoising, deblurring, and resolution reconstruction. It provides effective methods and technical support for innovative design of wood texture images in indoor furniture.

Frequency-Oriented Transformer for Remote Sensing Image Dehazing.

Remote sensing images are inevitably affected by the degradation of haze with complex appearance and non-uniform distribution, which remarkably affects the effectiveness of downstream remote sensing visual tasks. However, most current methods principally operate in the original pixel space of the image, which hinders the exploration of the frequency characteristics of remote sensing images, resulting in these models failing to fully exploit their representation ability to produce high-quality images. This paper proposes a frequency-oriented remote sensing dehazing Transformer named FOTformer, to explore information in the frequency domain to eliminate disturbances caused by haze in remote sensing images. It contains three components. Specifically, we developed a frequency-prompt attention evaluator to estimate the self-correlation of features in the frequency domain rather than the spatial domain, improving the image restoration performance. We propose a content reconstruction feed-forward network that captures information between different scales in features and integrates and processes global frequency domain information and local multi-scale spatial information in Fourier space to reconstruct the global content under the guidance of the amplitude spectrum. We designed a spatial-frequency aggregation block to exchange and fuse features from the frequency domain and spatial domain of the encoder and decoder to facilitate the propagation of features from the encoder stream to the decoder and alleviate the problem of information loss in the network. The experimental results show that the FOTformer achieved a more competitive performance against other remote sensing dehazing methods on commonly used benchmark datasets.

Dynamic and static fusion mechanisms of infrared and visible images

This paper propose a dynamic fusion mechanism of infrared and visible images, named DIFM, capable of solving the static fusion optimization problem. The DIFM correlates the image fusion quality with the image restoration quality to construct a unified optimization loss function. According to the DIFM, a dynamic image fusion network of infrared and visible images is constructed and is therefore denoted with DF-Net. Specifically, the DF-Net comprises two modules, i.e., the dynamic fusion module (DFM) and the self-learning dynamic restoration module (SLDRM). In order to solve the static fusion problem of existing methods, the DFM is proposed to learn the fusion weight dynamically. Specifically, the DFM comprises a classification module (CM) and an image fusion module (IFM), which determine whether and how to fuse source images. In addition, a unified fusion loss function is introduced to obtain more hidden features of infrared and visible images in complex environments. Therefore, the stumbling block of deep learning in image fusion, i.e., static fusion, is significantly mitigated. Extensive experiments demonstrate that the dynamic fusion optimization method neatly outperforms the state-of-the-art methods in most metrics.

A variable p[u] exponent reaction-diffusion PDE for image denoising

The paper presents a unique reaction-diffusion coupled system with p(u(t,x))-growth, employing the decomposition approach of H−1 norm. This method is designed to address image denoising by considering both texture and smooth components during the recovery process. The existence and uniqueness of weak solutions for the coupled system are established using Galerkin's method within a suitable space framework. Experimental analysis demonstrates the model's effectiveness in image restoration, and it is compared with other competitive models to showcase its performance.

Low-light wheat image enhancement using an explicit inter-channel sparse transformer

Low-light image enhancement poses a significant challenge in agricultural settings, particularly for time-series captured wheat images. The use of time-series captured image data can improve the accuracy of wheat yield prediction by analyzing the growth status and characteristics of wheat. Given the difficulty of capturing paired wheat images in various lighting conditions, a method based on the Global Wheat Head Detection (GWHD) dataset is presented to synthesize pairs of low-light/normal-light wheat images. This approach expands to create a new dataset, LN-GWHD, specifically for the task of enhancing wheat images in low-light conditions. Importantly, it enables a more extensive evaluation of wheat image restoration tasks and performance accuracy under more controlled and diverse conditions. It is worth noting that this method can be extended to a wide range of scenarios for the enhancement of low-light situations. Where low-light conditions are common due to various factors such as weather, existing methods based on the transformer architecture have achieved state-of-the-art (SOTA) enhancement performance by capturing relationships between low-light feature channels. However, these approaches perform the similarity computation on all tokens and may introduce redundant features, limiting their ability to focus on critical information for high-quality image reconstruction. To recover more high-quality wheat image information, the explicit inter-channel sparse transformer (EIST) network is proposed, which is specifically designed to recover wheat images under low-light conditions. EIST comprises multiple blocks, each featuring explicit sparse top-k attention (ESTA) and bilateral gated feed-forward network (BGFN). ESTA dynamically selects the most critical information by focusing on the top-k key labels in each query, while BGFN enhances valid information interaction through re-double filtering. The overall architecture employs transformers to learn correlations and features between different image channels, resulting in noise reduction, detail enhancement, and improved clarity. Additionally, the Fourier spectrum loss is introduced to constrain the image reconstruction in the frequency domain, allowing more detailed information to be recovered. Extensive experiments on the LN-GWHD datasets demonstrate that EIST surpasses SOTA methods. Furthermore, our approach achieves superior results in detecting low-light wheat ears during back-end evaluation.

Universal Image Restoration with Text Prompt Diffusion.

Universal image restoration (UIR) aims to accurately restore images with a variety of unknown degradation types and levels. Existing methods, including both learning-based and prior-based approaches, heavily rely on low-quality image features. However, it is challenging to extract degradation information from diverse low-quality images, which limits model performance. Furthermore, UIR necessitates the recovery of images with diverse and complex types of degradation. Inaccurate estimations further decrease restoration performance, resulting in suboptimal recovery outcomes. To enhance UIR performance, a viable approach is to introduce additional priors. The current UIR methods have problems such as poor enhancement effect and low universality. To address this issue, we propose an effective framework based on a diffusion model (DM) for universal image restoration, dubbed ETDiffIR. Inspired by the remarkable performance of text prompts in the field of image generation, we employ text prompts to improve the restoration of degraded images. This framework utilizes a text prompt corresponding to the low-quality image to assist the diffusion model in restoring the image. Specifically, a novel text-image fusion block is proposed by combining the CLIP text encoder and the DA-CLIP image controller, which integrates text prompt encoding and degradation type encoding into time step encoding. Moreover, to reduce the computational cost of the denoising UNet in the diffusion model, we develop an efficient restoration U-shaped network (ERUNet) to achieve favorable noise prediction performance via depthwise convolution and pointwise convolution. We evaluate the proposed method on image dehazing, deraining, and denoising tasks. The experimental results indicate the superiority of our proposed algorithm.

On-chip silicon photonic micro-ring processor lights up optical image encryption.

Optical image encryption has long been an important concept in the fields of photonic network processing and communication. Here, we propose a convolution-like operation-based optical image encryption algorithm exploiting a silicon photonic multiplexing architecture to achieve content security. Particularly, the encryption process is completed in a 3 × 3 cross-shaped photonic micro-ring resonator (MRR) array on chip. For the first time, to the best of our knowledge, this algorithm encodes information in an integrated intensity modulation, effectively reducing the encoding difficulty. Moreover, the high reliability and scalability of optical encryption are ensured using both linear and nonlinear operations on photonic chips according to characteristics of MRRs. As the encryption and decryption experiments show, the image restoration accuracy of our optical encryption algorithm exceeds 99% under real system noise at the pixel level, indicating its noise-robust property. Meanwhile, the peak signal-to-noise ratios of the restored and encrypted images are >60 and <15 dB, respectively, revealing both the high accuracy of the restored image and the small correlation between the encrypted and original images. This work adds to the rapidly expanding field of optical image encryption on photonic chips.

Deep learning-based restoration of nonlinear motion blurred images for plant classification using multi-spectral images

There have been various plant image-based studies for segmentation, deblurring, super-resolution reconstruction, and classification. However, nonlinear motion blur in thermal images was not considered in the existing studies on plant classification. Nonlinear motion blur occurs in images due to camera or plant movements, and it causes the degradation of plant classification accuracy. Moreover, nonlinear motion blur in images gets worse when both camera and plant movements occur simultaneously. In this case, it becomes difficult to recognize plants, and the performance of plant image classification becomes very low. Therefore, to reduce the nonlinear motion blur, a thermal and visible light plant images-based deblurring network (TVPD-Net) is proposed in this study. In addition, a thermal and visible light plant images-based classification network (TVPC-Net) is also proposed to improve the plant classification performance on deblurred images. Experimental results revealed that the proposed TVPD-Net achieved 21.21 and 22.53 of the peak signal-to-noise ratio (PSNR), and 0.726 and 0.737 of the structural similarity index measure (SSIM) on both visible light and thermal plant image datasets which were self-collected, respectively. Moreover, the proposed TVPC-Net with deblurred images by TVPD-Net achieved 92.52 % (top-1 accuracy) and 87.73 % (harmonic mean of precision and recall (F1-score)). In addition, the experimental results on an open dataset named Hyperspectral Flower Dataset (HFD100) revealed that the proposed plant classification method achieved 90.94 % of top-1 accuracy and 86.21 % of F1-score. The accuracies of the proposed methods are greater than those of the state-of-the-art methods.

Iterative algorithm for a generalized matrix equation with momentum acceleration approach and its convergence analysis

This article discusses a numerical gradient-based method for solving a generalized matrix equation. The iterative method mentioned includes two positive parameters, for which a range is determined to ensure the convergence of the introduced method. It has been shown that the optimal parameters of this method satisfy a constrained optimization problem. Then, specific solutions of this equation, such as the reflexive solution, are examined. Furthermore, to increase the rate of convergence of the proposed method, the idea of momentum is utilized, and a range for the momentum parameter is obtained to ensure the convergence. Finally, the efficiency of this method is investigated through numerical simulations. Additionally, in numerical results section, applications of mentioned matrix equation in image restoration and anti-linear systems are addressed.

Solving a system of complex matrix equations using a gradient-based method and its application in image restoration

Solving a system of complex matrix equations using a gradient-based method and its application in image restoration

Tikhonov regularization with conjugate gradient least squares method for large-scale discrete ill-posed problem in image restoration

Image restoration is a large-scale discrete ill-posed problem, which can be transformed into a Tikhonov regularization problem that can approximate the original image. Kronecker product approximation is introduced into the Tikhonov regularization problem to produce an alternative problem of solving the generalized Sylvester matrix equation, reducing the scale of the image restoration problem. This paper considers solving this alternative problem by applying the conjugate gradient least squares (CGLS) method which has been demonstrated to be efficient and concise. The convergence of the CGLS method is analyzed, and it is demonstrated that the CGLS method converges to the least squares solution within the finite number of iteration steps. The effectiveness and superiority of the CGLS method are verified by numerical tests.

Underwater image restoration based on progressive guidance

Underwater images often suffer from local distortions during the imaging and transmission process, which can negatively impact their quality. Fortunately, it is possible to improve image quality by removing local distortion without making any hardware or software adjustments to the transmission system. However, existing algorithms designed for global distortions are not suitable for addressing local distortions, while end-to-end restoration and inpainting algorithms do not perform satisfactorily on underwater images. To address this issue, this paper proposes a Joint distortion localization and restoration model based on Progressive Guidance (JPG) specifically tailored for underwater imaging and transmission. Our strategy employs a two-stage framework where the first stage focuses exclusively on accurately localizing distortions to obtain precise position. Subsequently, in the second stage, we utilize this position information for effective distortion restoration. To further enhance restoration performance, our approach progressively guides the restoration process by incorporating global, distortion-free as well as distortion-specific information into different components of the second-stage network. The work surpasses current state-of-the-art methods in restoring both mixed and individual distortions.