Image Restoration Research Articles

Elliptic Quaternion Matrices: A MATLAB Toolbox and Applications for Image Processing

In this study, we developed a MATLAB 2024a toolbox that performs advanced algebraic calculations in the algebra of elliptic numbers and elliptic quaternions. Additionally, we introduce color image processing methods, such as principal component analysis, image compression, image restoration, and watermarking, based on singular-value decomposition theory for elliptic quaternion matrices; we added these to the newly developed toolbox. The experimental results demonstrate that elliptic quaternionic methods yield better image analysis and processing performance compared to other hypercomplex number-based methods.

Image restoration through few-mode fiber using a new comprehensive attention model

Few-mode fibers (FMF) has unique advantages in single fiber imaging (SFI) in special applications. However, the limited number of transmission modes restricts the information carriers, hence the high-quality image restoration is a major challenge. A deep learning network with new comprehensive attention, called few-mode fiber Swin Transformer (FFST), is developed based on Swin Transformer V2 to realize successful image restoration with a FMF on handwriting letters and numbers. This network is proved outperforming nine widely used convolutional neural networks or Transformer-based models with four evaluation criteria. In addition, single-fiber-multi-location (SFML) experiment is conducted to investigate the effect of different view on imaging feasibility, the result of which shows that SFML can improve the restoration quality despite one object corresponds to multiple different speckle images, indicating that multi-mapping relationships has constructive effect in single FMF imaging. This paper provides a method for imaging in very narrow cavities.

Deep Variational Network Toward Blind Image Restoration.

Blind image restoration (IR) is a common yet challenging problem in computer vision. Classical model-based methods and recent deep learning (DL)-based methods represent two different methodologies for this problem, each with their own merits and drawbacks. In this paper, we propose a novel blind image restoration method, aiming to integrate both the advantages of them. Specifically, we construct a general Bayesian generative model for the blind IR, which explicitly depicts the degradation process. In this proposed model, a pixel-wise non-i.i.d. Gaussian distribution is employed to fit the image noise. It is with more flexibility than the simple i.i.d. Gaussian or Laplacian distributions as adopted in most of conventional methods, so as to handle more complicated noise types contained in the image degradation. To solve the model, we design a variational inference algorithm where all the expected posteriori distributions are parameterized as deep neural networks to increase their model capability. Notably, such an inference algorithm induces a unified framework to jointly deal with the tasks of degradation estimation and image restoration. Further, the degradation information estimated in the former task is utilized to guide the latter IR process. Experiments on two typical blind IR tasks, namely image denoising and super-resolution, demonstrate that the proposed method achieves superior performance over current state-of-the-arts.

Enhanced osmosis model with bilateral total variation for effective shadow removal

This research paper presents an innovative technique for shadow images removal. The method involves redefining a contemporary osmosis model by incorporating bilateral total variation (TV) operators. This integration allows to take advantage of robust anisotropic diffusion, resulting in improved image restoration. The paper also outlines new mathematical derivation of the bilateral TV and a combination with nonlinear anisotropic transport term. The experimental results substantiate the effectiveness of the anisotropic osmosis model, showcasing its superior qualitative and quantitative performance when compared to current state-of-the-art techniques.

ACMamba: A State Space Model-Based Approach for Multi-Weather Degraded Image Restoration

In computer vision, eliminating the effects of adverse weather conditions such as rain, snow, and fog on images is a key research challenge. Existing studies primarily focus on image restoration for single weather types, while methods addressing image restoration under multiple combined weather conditions remain relatively scarce. Furthermore, current mainstream restoration networks, mostly based on Transformer and CNN architectures, struggle to achieve an effective balance between global receptive field and computational efficiency, limiting their performance in practical applications. This study proposes ACMamba, an end-to-end lightweight network based on selective state space models, aimed at achieving image restoration under multiple weather conditions using a unified set of parameters. Specifically, we design a novel Visual State Space Module (VSSM) and a Spatially Aware Feed-Forward Network (SAFN), which organically combine the local feature extraction capabilities of convolutions with the long-range dependency modeling capabilities of selective state space models (SSMs). This combination significantly improves computational efficiency while maintaining a global receptive field, enabling effective application of the Mamba architecture to multi-weather image restoration tasks. Comprehensive experiments demonstrate that our proposed approach significantly outperforms existing methods for both specific and multi-weather tasks across multiple benchmark datasets, showcasing its efficient long-range modeling potential in multi-weather image restoration tasks.

MIMO-Uformer: A Transformer-Based Image Deblurring Network for Vehicle Surveillance Scenarios

Motion blur is a common problem in the field of surveillance scenarios, and it obstructs the acquisition of valuable information. Thanks to the success of deep learning, a sequence of CNN-based architecture has been designed for image deblurring and has made great progress. As another type of neural network, transformers have exhibited powerful deep representation learning and impressive performance based on high-level vision tasks. Transformer-based networks leverage self-attention to capture the long-range dependencies in the data, yet the computational complexity is quadratic to the spatial resolution, which makes transformers infeasible for the restoration of high-resolution images. In this article, we propose an efficient transformer-based deblurring network, named MIMO-Uformer, for vehicle-surveillance scenarios. The distinct feature of the MIMO-Uformer is that the basic-window-based multi-head self-attention (W-MSA) of the Swin transformer is employed to reduce the computational complexity and then incorporated into a multi-input and multi-output U-shaped network (MIMO-UNet). The performance can benefit from the operation of multi-scale images by MIMO-UNet. However, most deblurring networks are designed for global blur, while local blur is more common under vehicle-surveillance scenarios since the motion blur is primarily caused by local moving vehicles. Based on this observation, we further propose an Intersection over Patch (IoP) factor and a supervised morphological loss to improve the performance based on local blur. Extensive experiments on a public and a self-established dataset are carried out to verify the effectiveness. As a result, the deblurring behavior based on PSNR is improved at least 0.21 dB based on GOPRO and 0.74 dB based on the self-established datasets compared to the existing benchmarks.

Dual-path aggregation transformer network for super-resolution with images occlusions and variability

While Transformer-based approaches have recently achieved notable success in super-resolution, their extensive computational requirements impede widespread practical adoption. High-resolution meteorological satellite cloud imagery is essential for weather analysis and forecasting. Enhancing image resolution through super-resolution techniques facilitates the accurate identification and localization of geographic features by meteorological systems. However, current super-resolution methods fail to restore the intricacies of cloud formations and complex regions fully. This research introduces a novel dual-path aggregation Transformer network (DPAT) tailored to enhance the super-resolution of meteorological satellite cloud images. The DPAT network adeptly captures cloud imagery's subtle details and textures, effectively addressing occlusions and the variability inherent in satellite imagery. It bolsters the model's ability to manage the complex attributes of cloud images through the introduction of the Dual-path Aggregation Self-Attention (DASA) mechanism and the Multi-scale Feature Aggregation Block (MFAB), thereby enhancing performance in processing intricate cloud features. The DASA mechanism synthesizes features across spatial, depth, and channel dimensions via a dual-path approach, thoroughly exploiting feature correlations. The MFAB, designed to supplant the multilayer perceptron, incorporates shift convolution and a multi-scale interaction block to augment feature information, compensating for the deficiency in local information absorption due to fixed receptive fields. Experimental outcomes indicate that DPAT delivers superior super-resolution outcomes. With a parameter count of only 32% of the Enhanced Deep Residual Network (EDSR) or 77% of the Image Restoration using Shift Window Transformer (SwinIR), DPAT matches SwinIR's performance on the satellite cloud dataset. Moreover, DPAT balances accuracy and parameter economy across various datasets. This technology is expected to improve image super-resolution capabilities in multiple fields such as human action recognition and industrial recognition, and indirectly improve the accuracy of image perception tasks.

Research on the Laser Scattering Characteristics of Three-Dimensional Imaging Based on Electro–Optical Crystal Modulation

In this paper, we construct a laser 3D imaging simulation model based on the 3D imaging principle of electro–optical crystal modulation. Unlike the traditional 3D imaging simulation method, this paper focuses on the laser scattering characteristics of the target scene. To accurately analyze and simulate the scattering characteristic model of the target under laser irradiation, we propose a BRDF (Bidirectional Reflectance Distribution Function) model fitting algorithm based on the hybrid BBO–Firefly model, which can accurately simulate the laser scattering distribution of the target at different angles. Finally, according to the fitted scattering characteristic model, we inverted the target imaging gray map. We used the laser 3D imaging restoration principle to reconstruct the 3D point cloud of the target to realize the laser 3D imaging of the target.

Some combined techniques of spectral conjugate gradient methods with applications to robotic and image restoration models

Some combined techniques of spectral conjugate gradient methods with applications to robotic and image restoration models

Framework of fourth-order partial differential equation–based remote sensing image restoration

Framework of fourth-order partial differential equation–based remote sensing image restoration

Hybrid plug-and-play CT image restoration using nonconvex low-rank group sparsity and deep denoiser priors

Abstract Objective. Low-dose computed tomography (LDCT) is an imaging technique that can effectively help patients reduce radiation dose, which has attracted increasing interest from researchers in the field of medical imaging. Nevertheless, LDCT imaging is often affected by a large amount of noise, making it difficult to clearly display subtle abnormalities or lesions.
Therefore, this paper proposes a multiple complementary priors CT image reconstruction method by simultaneously considering both the internal prior and external image information of CT images, thereby enhancing the reconstruction quality of CT images.
Approach. Specifically, we propose a CT image reconstruction method based on weighted nonconvex low-rank regularized group sparse and deep image priors under hybrid plug-and-play framework by utilizing the weighted nonconvex low rankness and group sparsity of dictionary domain coefficients of each group of similar patches, and a convolutional neural network (CNN) denoiser.
To make the proposed reconstruction problem easier to tackle, we utilize the Alternate Direction Method of Multipliers (ADMM) for optimization.Main results. To verify the performance of the proposed method, we conduct detailed simulation experiments on the images of the abdominal, pelvic, and thoracic at projection views of 45, 65, and 85, and at noise levels of 1×10^5 and 1×10^6, respectively. A large number of qualitative and quantitative experimental results indicate that the proposed method has achieved better results in texture preservation and noise suppression compared to several existing iterative reconstruction methods.
Significance. The proposed method fully considers the internal nonlocal low rankness and sparsity, as well as the external local information of CT images, providing a more effective solution for CT image reconstruction. Consequently, this method enables doctors to diagnose and treat diseases more accurately by reconstructing high-quality CT images.

Parametric Analytical Modulation Transfer Function Model in Turbid Atmosphere with Application to Image Restoration

To address the issues of image blurring and color distortion in hazy conditions, an image restoration method based on a parametric analytical modulation transfer function model is proposed under turbid atmospheric conditions. A source database is established using a numerical radiative transfer method based on discrete ordinate. Through multivariate nonlinear fitting and linear interpolation, the quantitative relationships among critical spatial frequency, turbid atmospheric MTF, and key atmospheric optical parameters—such as optical thickness, single scattering albedo, and asymmetry factor—are examined. A fast and efficient parametric analytical MTF model for turbid atmospheres is developed and applied to restore images affected by fog. The results demonstrate that, within the applicable range of the model, the model’s maximum mean relative error and the root mean square error are 7.16% and 0.0454, respectively. The computational speed is nearly a thousand times faster than that of the numerical radiative transfer method, achieving high accuracy and ease of application. Images restored using this model exhibit enhanced clarity and quality, effectively compensating for the degradation in image quality caused by turbid atmospheres. This approach represents a novel solution to the challenges of image processing in complex atmospheric environments.

Deep learning approaches for image restoration in invasive coronary angiography

Abstract Background Despite being the gold standard for diagnosing coronary artery disease and guiding percutaneous coronary interventions (PCI), image restoration during invasive coronary angiography (ICA) has rarely been explored. Generative adversarial network (GAN), a deep learning model, has demonstrated its effectiveness in image restoration. Our hypothesis was that we could restore ICA images, damaged or deleted to varying degrees, using the inpainting GAN method, and we undertook efforts to test this hypothesis. Methods We extracted 22,659 ICA patch images (128 × 128 size) from 60 patients, focusing on sparse vessels for model training. To assess our inpainting GAN model, we degraded the extracted images to varying degrees: minimal, mild, moderate, severe, and completely blocked. The performance was quantitatively analyzed using metrics such as the peak signal-to-noise ratio (PSNR), structured similarity indexing methods (SSIM), and mean squared error (MSE). Results The inpainting GAN method significantly improved the image quality in all degraded images. For the blocked images, PSNR and SSIM were enhanced by 161.9% and 14.3%, respectively, with a 98% MSE reduction. In severely, moderately, mildly, and minimally damaged images, PSNR and SSIM increased by 14.9% and 14.8%, 15.7% and 14.7%, 19.3% and 15.6%, and 12.7% and 5.3%, respectively, with MSE reductions of 61.5%, 64.1%, 72.7%, and 33.9%, respectively. Conclusions Inpainting GAN can restore ICA images with varying damage levels, including deletions. Our deep-learning model can instantaneously improve ICA image quality, potentially aiding in emergency or chronic total occlusion PCI.Table 2.Quantitative Analysis of ImageFigure 1.Sample Cases Pre and Post-Rest

Parallel linearized ADMM with application to multichannel image restoration and reconstruction

Many large-scale regularized inverse problems in imaging such as image restoration and reconstruction can be modeled as a generic objective function involves sum of nonsmooth but proximable terms, which are usually linear-operator-coupled. For the solution of these problems, a parallel linearized alternating direction method of multipliers (PLADMM) is proposed in this paper. At each step of the proposed algorithm, the proximity operators of the nondifferential terms are called individually. This leads to a highly parallel algorithm structure, where most sub-steps can be simultaneously solved. Profiting from the linearization step, the linear inverse operation is excluded. The convergence property of the proposed method is analyzed. The image deblurring, inpainting, and pMRI reconstruction experiments show that the proposed method has vast applicable vistas. Compared with the state-of-the-art methods, such as PADMM [21], FTVD-v4 [22], PPDS [33], FUSL [8], LADM [36], and ALADMM [27], it gains competitive results both in terms of quantitative indicators, such as PSNR or SSIM, and in terms of visual impression.

Ablation Study of Diffusion Model with Transformer Backbone for Low-count PET Denoising.

Diffusion models (DM) built from a hierarchy of denoising autoencoders have achieved remarkable progress in image generation, and are increasingly influential in the field of image restoration (IR) tasks. In the meantime, its backbone of autoencoders also evolved from UNet to vision transformer, e.g. Restormer. Therefore, it is important to disentangle the contribution of backbone networks and the additional generative learning scheme. Notably, DM shows varied performance across IR tasks, and the performance of recent advanced transformer-based DM on PET denoising is under-explored. In this study, we further raise an intuitive question, "{if we have a sufficiently powerful backbone, whether DM can be a general add-on generative learning scheme to further boost PET denoising}". Specifically, we investigate one of the best-in-class IR models, i.e., DiffIR, which is a latent DM based on the Restormer backbone. We provide a qualitative and quantitative comparison with UNet, SR3 (UNet+pixel DM), and Restormer, on the 25% low dose 18F-FDG whole-body PET denoising task, aiming to identify the best practices. We trained and tested on 93 and 12 subjects, and each subject has 644 slices. It appears that Restormer outperforms UNet in terms of PSNR and MSE. However, additional latent DM over Restormer does not contribute to better MSE, SSIM, or PSNR in our task, which is even inferior to the conventional UNet. In addition, SR3 with pixel space DM is not stable to synthesize satisfactory results. The results are consistent with the natural image super-resolution tasks, which also suffer from limited spatial information. A possible reason would be the denoising iteration at latent feature space cannot well support detailed structure and texture restoration. This issue is more crucial in the IR tasks taking inputs with limited details, e.g., SR and PET denoising.

A modified non-convex Cauchy total variation regularization model for image restoration

A modified non-convex Cauchy total variation regularization model for image restoration

High-fidelity Image Restoration of Large 3D Electron Microscopy Volume.

Volume electron microscopy (VEM) is an essential tool for studying biological structures. Due to the challenges of sample preparation and continuous volumetric imaging, image artifacts are almost inevitable. Such image artifacts complicate further processing both for automated computer vision methods and human experts. Unfortunately, the widely used contrast limited adaptive histogram equalization (CLAHE) can alter the essential relative contrast information about some biological structures. We developed an image-processing pipeline to remove the artifacts and enhance the images without CLAHE. We apply our method to VEM datasets of a Microwasp head. We demonstrate that our method restores the images with high fidelity while preserving the original relative contrast. This pipeline is adaptable to other VEM datasets.

New hybrid conjugate gradient method for nonlinear optimization with application to image restoration problems

New hybrid conjugate gradient method for nonlinear optimization with application to image restoration problems

Systematic Analysis of Image Restoration Methods Based on Deep Learning

In today's digital age, people often use images to convey information. However, during the process of sending images, they can become damaged for various reasons, causing the images to lose their original meaning. In such cases, image repair is necessary to restore the damaged areas to their original, undamaged appearance. Traditional image repair methods often struggle to produce effective results for highly damaged images. However, image repair using deep learning techniques can address these challenges effectively. This paper will detail the impact of image repair on Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory networks (LSTM), Convolutional Recurrent Neural Networks (CRNN), and other models, as well as the advantages and disadvantages of each approach. From the data, it is evident that combining multiple models for image repair is the most effective method. This approach reduces the consumption of hardware resources and time required for image repair, thereby improving the overall efficiency of the process.

EFCformer: High-resolution image restoration network for optical synthetic aperture imaging system

EFCformer: High-resolution image restoration network for optical synthetic aperture imaging system