Image Inpainting Problems Research Articles

LAMA-WAVELET: IMAGE IMPAINTING WITH HIGH QUALITY OF FINE DETAILS AND OBJECT EDGES

Context. The problem of the image impainting in computer graphic and computer vision systems is considered. The subject of the research is deep learning convolutional neural networks for image inpainting.
 Objective. The objective of the research is to improve the image inpainting performance in computer vision and computer graphics systems by applying wavelet transform in the LaMa-Fourier network architecture.
 Method. The basic LaMa-Fourier network decomposes the image into global and local texture. Then it is proposed to improve the network block, processing the global context of the image, namely, the spectral transform block. To improve the block of spectral transform, instead of Fourier Unit Structure the Simple Wavelet Convolution Block elaborated by the authors is used. In this block, 3D wavelet transform of the image on two levels was initially performed using the Daubechies wavelet db4. The obtained coefficients of 3D wavelet transform are splitted so that each subband represents a separate feature of the image. Convolutional layer, batch normalization and ReLU activation function are sequentially applied to the results of splitting of coefficients on each level of wavelet transform. The obtained subbands of wavelet coefficients are concatenated and the inverse wavelet transform is applied to them, the result of which is the output of the block. Note that the wavelet coefficients at different levels were processed separately. This reduces the computational complexity of calculating the network outputs while preserving the influence of the context of each level on image inpainting. The obtained neural network is named LaMa-Wavelet. The FID, PSNR, SSIM indexes and visual analysis were used to estimate the quality of images inpainted with LaMa-Wavelet network.
 Results. The proposed LaMa-Wavelet network has been implemented in software and researched for solving the problem of image inpainting. The PSNR of images inpainted using the LaMa-Wavelet exceeds the results obtained using the LaMa-Fourier network for narrow and medium masks in average by 4.5%, for large masks in average by 6%. The LaMa-Wavelet applying can enhance SSIM by 2–4% depending on a mask size. But it takes 3 times longer to inpaint one image with LaMa-Wavelet than with LaMa-Fourier network. Analysis of specific images demonstrates that both networks show similar results of inpainting of a homogeneous background. On complex backgrounds with repeating elements the LaMa-Wavelet is often more effective in restoring textures.
 Conclusions. The obtained LaMa-Wavelet network allows to improve the image inpainting with large masks due to applying wavelet transform in the LaMa network architecture. Namely, the quality of reconstruction of image edges and fine details is increased.

A Designed Thresholding Operator for Low-Rank Matrix Completion

In this paper, a new thresholding operator, namely, designed thresholding operator, is designed to recover the low-rank matrices. With the change of parameter in designed thresholding operator, the designed thresholding operator can apply less bias to the larger singular values of a matrix compared with the classical soft thresholding operator. Based on the designed thresholding operator, an iterative thresholding algorithm for recovering the low-rank matrices is proposed. Numerical experiments on some image inpainting problems show that the proposed thresholding algorithm performs effectively in recovering the low-rank matrices.

Text Image Inpainting via Global Structure-Guided Diffusion Models

Real-world text can be damaged by corrosion issues caused by environmental or human factors, which hinder the preservation of the complete styles of texts, e.g., texture and structure. These corrosion issues, such as graffiti signs and incomplete signatures, bring difficulties in understanding the texts, thereby posing significant challenges to downstream applications, e.g., scene text recognition and signature identification. Notably, current inpainting techniques often fail to adequately address this problem and have difficulties restoring accurate text images along with reasonable and consistent styles. Formulating this as an open problem of text image inpainting, this paper aims to build a benchmark to facilitate its study. In doing so, we establish two specific text inpainting datasets which contain scene text images and handwritten text images, respectively. Each of them includes images revamped by real-life and synthetic datasets, featuring pairs of original images, corrupted images, and other assistant information. On top of the datasets, we further develop a novel neural framework, Global Structure-guided Diffusion Model (GSDM), as a potential solution. Leveraging the global structure of the text as a prior, the proposed GSDM develops an efficient diffusion model to recover clean texts. The efficacy of our approach is demonstrated by thorough empirical study, including a substantial boost in both recognition accuracy and image quality. These findings not only highlight the effectiveness of our method but also underscore its potential to enhance the broader field of text image understanding and processing. Code and datasets are available at: https://github.com/blackprotoss/GSDM.

Fixed-precision randomized quaternion singular value decomposition algorithm for low-rank quaternion matrix approximations

The fixed-precision randomized quaternion singular value decomposition algorithm (FPRQSVD) is presented to compute the low-rank quaternion matrix approximation. The FPRQSVD algorithm estimates the appropriate rank according to the given tolerance without a predetermined rank parameter, and computes the corresponding low-rank quaternion matrix approximation automatically. Error analysis indicates that the FPRQSVD algorithm is mathematically equivalent to the randomized quaternion singular value decomposition algorithm. Numerical experiments are given to demonstrate that the FPRQSVD algorithm is feasible and effective. Moreover, we use it to deal with the problem of color image inpainting.

A BLIND IMAGE INPAINTING MODEL INTEGRATED WITH RATIONAL FRACTAL INTERPOLATION INFORMATION

Aiming to solve the problem of blind image inpainting, this study proposed a blind image inpainting model integrated with rational fractal interpolation information. First, wavelet decomposition and closed operations were adopted to obtain masks and transform blind inpainting into non-blind inpainting. Then, on the basis of similar structural groups, rational fractal interpolation functions were introduced to complete the restoration. On the one hand, this model can sufficiently express the texture features of the image with high fidelity. On the other hand, it can better represent the structural features of the image, avoid serrated edges, enhance the restoration effect, and approximate the original image. The experimental results show that the restoration effect of this model can reserve texture details and ensure edges without distortion, possessing great practical application value.

Shock filter as the classifier for image inpainting problem using the Cahn-Hilliard equation

We consider a problem of digital image inpainting using the modified Cahn-Hilliard equation, where the standard double well potential is replaced by the shock filter. Using fixed point arguments and Aubin-Lions lemma we prove the existence and uniqueness of the solution. In addition, we introduce a numerical method based on the convexity splitting idea to approximate the solutions of the considered problem. We apply this method to several binary images and demonstrate that this approach naturally extends image features and preserves their edges.

Fine Tuning of Deep Contexts Toward Improved Perceptual Quality of In-Paintings.

Over the recent years, a number of deep learning approaches are successfully introduced to tackle the problem of image in-painting for achieving better perceptual effects. However, there still exist obvious hole-edge artifacts in these deep learning-based approaches, which need to be rectified before they become useful for practical applications. In this article, we propose an iteration-driven in-painting approach, which combines the deep context model with the backpropagation mechanism to fine-tune the learning-based in-painting process and hence, achieves further improvement over the existing state of the arts. Our iterative approach fine tunes the image generated by a pretrained deep context model via backpropagation using a weighted context loss. Extensive experiments on public available test sets, including the CelebA, Paris Streets, and PASCAL VOC 2012 dataset, show that our proposed method achieves better visual perceptual quality in terms of hole-edge artifacts compared with the state-of-the-art in-painting methods using various context models.

Blind Image Inpainting Using Low-Dimensional Manifold Regularization

In this paper, we present a novel method for blind image inpainting, which can restore images with missing or corrupted pixels, or images where the location of the damaged pixels is unknown. The method applies weighted nonlocal Laplacian to address the problem of blind image inpainting using low-dimensional manifold model (LDMM) regularization, and uses semi-local blocks instead of point integrals to implement constraints in LDMM. This solves the problem of low solution efficiency caused by the asymmetry of the linear equations solved by point integration, and the problem of the high iteration count to get good restoration effect. Experiments show that our method is competitive with latest methods in terms of both repairing images with large missing pixels rate and inpainting speed.

A Nonconvex Method to Low-Rank Matrix Completion

In recent years, the problem of recovering a low-rank matrix from partial entries, known as low-rank matrix completion problem, has attracted much attention in many applications. However, it is a NP-hard problem due to the nonconvexity nature of the matrix rank function. In this paper, a rank Laplace function is studied to recover the low-rank matrices. Firstly, we propose an iterative Laplace thresholding algorithm to solve the regularized Laplace low-rank matrix completion problem. Secondly, some other iterative thresholding algorithms are designed to recover the low-rank matrices. Finally, we provide a series of numerical simulations to test the proposed algorithms on some low-rank matrix completion and image inpainting problems, and the results show that our algorithms perform better than some state-of-art methods in recovering the low-rank matrices.

Deep Generative Inpainting with Comparative Sample Augmentation

Recent advances in deep learning techniques such as Convolutional Neural Networks (CNN) and Generative Adversarial Networks (GAN) have achieved breakthroughs in the problem of semantic image inpainting, the task of reconstructing missing pixels. While more effective than conventional approaches, deep learning models require large datasets and computational resources for training, and inpainting quality varies considerably when training data differs in size and diversity. To address these problems, we present an inpainting strategy called Comparative Sample Augmentation, which enhances the quality of the training set by filtering irrelevant images and constructing additional images using information about the surrounding regions of the target image. Experiments on multiple datasets demonstrate that our method extends the applicability of deep inpainting models to training sets with varying levels of diversity, while enhancing the inpainting quality as measured by qualitative and quantitative metrics for a large class of deep models, with little need for model-specific consideration.

Predicting the Future is Like Completing a Painting: Towards a Novel Method for Time-Series Forecasting

This article is an introductory work towards a larger research framework relative to Scientific Prediction. It is a mixed between science and philosophy of science, therefore we can talk about Experimental Philosophy of Science. As a first result, we introduce a new forecasting method based on image completion, named Forecasting Method by Image Inpainting (FM2I). In fact, time series forecasting is transformed into fully images- and signal-based processing procedures. After transforming a time series data into its corresponding image, the problem of data forecasting becomes essentially a problem of image inpainting, i.e., completing missing data in the image. Extensive experimental evaluation was conducted using the shortest series of the dataset proposed by the well-known M3-competition. Results show that FM2I, despite still being in its infancy, represents an efficient and robust tool for short-term time series forecasting. It has achieved prominent results in terms of accuracy and outperforms the best M3 methods. We have also investigated the effectiveness of the FM2I against the Smyl method, the winner of the M4 competition. Using the same category of shortest series, results show a close accuracy compared to the Smyl method. The FM2I is also able to generate ensemble data forecasts, which contain the best and more accurate forecast compared to existing and considered methods.

Incremental proximal gradient scheme with penalization for constrained composite convex optimization problems

We consider the problem of minimizing a finite sum of convex functions subject to the set of minimizers of a convex differentiable function. In order to solve the problem, an algorithm combining the incremental proximal gradient method with smooth penalization technique is proposed. We show the convergence of the generated sequence of iterates to an optimal solution of the optimization problems, provided that a condition expressed via the Fenchel conjugate of the constraint function is fulfilled. Finally, the functionality of the method is illustrated by some numerical experiments addressing image inpainting problems and generalized Heron problems with least squares constraints.

Design strategy of thresholding operator for low-rank matrix recovery problem

This paper proposes the band restricted thresholding (BRT) operator and studies theoretically the performance of the corresponding singular value thresholding algorithms for the affine matrix rank minimization problem. We first introduce a new notion named matrix-type generalized restricted isometry constant (MgRIC) as a generalization of restricted isometry constant (RIC). Then we show the stability results characterized via gRIC and RIC. Finally, we design a new thresholding operator and numerical demonstration on random low-rank matrix completion problems and image inpainting problems have been shown that the design strategy of thresholding operator is effective.

Rank-One Matrix Approximation With ℓ p -Norm for Image Inpainting

In the problem of image inpainting, one popular approach is based on low-rank matrix completion. Compared with other methods which need to convert the image into vectors or dividing the image into patches, matrix completion operates on the whole image directly. Therefore, it can preserve latent information of the two-dimensional image. An efficient method for low-rank matrix completion is to employ the matrix factorization technique. However, conventional low-rank matrix factorization-based methods often require a prespecified rank, which is challenging to determine in practice. The proposed method factorizes an image matrix as a sum of rank-one matrices so that it does not require rank information in advance as it can be automatically estimated by the algorithm itself when the algorithm has satisfactorily converged. In our study, matching pursuit is applied to search for the best rank-one matrix at each iteration. To be robust against impulsive noise, the residual error between the observed and estimated matrices is minimized by $\ell _p$ -norm with $0 . Then the resultant $\ell _p$ -norm minimization is solved by the iteratively reweighted least squares method. The proposed model is beneficial for the robustness against outliers, and does not require rank information. Experimental results verify the effectiveness and higher accuracy of the proposed method with comparison to several state-of-the-art matrix completion-based image inpainting approaches.

High-order anisotropic diffusion operators in spaces of variable exponents and application to image inpainting and restoration problems

We consider a class of nonstandard high-order PDEs models, based on the p(⋅)-Laplace operator with variable exponents, for image denoising and image inpainting problems. These models are obtained from the minimization of a family of anisotropic energies that range between the so called TV−H−1 and the biharmonic models. The PDEs system consists of quasi-linear equations that we solve with a fixed point iterative method and we prove the convergence of the iterative process. To solve the problems, we consider an algorithm which includes a practical and efficient adaptive strategy for the choice of the exponent 1<p≤2, which allows us to fit to the multi-scale nature of the images. We present several numerical examples to test our approach and to make some comparisons with existing methods.

Image Inpainting Using Nonlocal Texture Matching and Nonlinear Filtering.

Nonlocal texture similarity and local intensity smoothness are both essential for solving most image inpainting problems. In this paper, we propose a novel image inpainting algorithm that is capable of reproducing the underlying textural details using a nonlocal texture measure and also smoothing pixel intensity seamlessly in order to achieve natural-looking inpainted images. For matching texture, we propose a Gaussian-weighted nonlocal texture similarity measure to obtain multiple candidate patches for each target patch. To compute the pixel intensity, we apply the -trimmed mean filter to the candidate patches to inpaint the target patch pixel-by-pixel. The proposed algorithm is compared with four current image inpainting algorithms under different scenarios, including object removal, texture synthesis, and error concealment. Experimental results show that the proposed algorithm outperforms the existing algorithms when inpainting large missing regions in images with texture and geometric structures.

Fast and efficient algorithm for matrix completion via closed-form 2/3-thresholding operator

Matrix completion is arguably one of the most studied problems in machine learning and data analysis. Inspired by the closed-form formulas for L2/3 regularization, we employ the Schatten 2/3 quasi-norm to approximate the rank of a matrix, which can provide a better approximation than traditional ways. We also establish a necessary optimal condition and propose a fixed point iterative scheme for solving L2/3 regularization problem. Through analysing the monotonicity and the accumulation point of L2/3 regularization problem, the convergence of this iteration is analysed. By discussing the optimal selection of the regularization parameter together with a fast Monte Carlo algorithm and an approximate singular value decomposition (SVD) procedure, we build a fast and efficient algorithm that solves the induced optimization problem well. Extensive experiments have been conducted and the results show that the proposed algorithm is fast, efficient and robust. Specifically, we compare the proposed algorithm with state-of-the-art matrix completion algorithms on many synthetic data and large recommendation datasets. Our proposed algorithm is able to achieve similar or better prediction performance, while being faster and more efficient than alternatives. Furthermore, we demonstrate the effectiveness of our proposed algorithm by solving image inpainting problems.

Exact recovery low-rank matrix via transformed affine matrix rank minimization

Abstract The goal of affine matrix rank minimization problem is to reconstruct a low-rank or approximately low-rank matrix under linear constraints. In general, this problem is combinatorial and NP-hard. In this paper, a nonconvex fraction function is studied to approximate the rank of a matrix and translate this NP-hard problem into a transformed affine matrix rank minimization problem. The equivalence between these two problems is established, and we proved that the uniqueness of the global minimizer of transformed affine matrix rank minimization problem also solves affine matrix rank minimization problem if some conditions are satisfied. Moreover, we also proved that the optimal solution to the transformed affine matrix rank minimization problem can be approximately obtained by solving its regularization problem for some proper smaller λ > 0. Lastly, the DC algorithm is utilized to solve the regularization transformed affine matrix rank minimization problem and the numerical experiments on image inpainting problems show that our method performs effectively in recovering low-rank images compared with some state-of-art algorithms.

Improved structure tensor for fine-grained texture inpainting

Partial differential equations (PDEs) have emerged as a useful tool for recovering structures when solving image inpainting problems. In the research presented in this paper, we extend structure tensor (ST)-based PDE to fine-grained texture inpainting by improving the ST with three modifications to enhance its suitability to more accurately process fine-grained textures of which the small-scale patterns repeat non-locally. These modifications are then inserted into an anisotropic PDE to recover damaged textures. The result is a subtly-conducted anisotropic diffusion inpainting algorithm free of detail-blurring artifacts produced by the classic tensor diffusion model and dislocation deficiencies affecting patch-based completion approaches. This is made possible by the following strategies: (1) construct a fractional ST (FST11Fractional structure tensor. ) composed of the inner product of the fractional derivative vector and its transposition to more effectively address complex fractal-like texture details because of the characteristics of the fractional differential; (2) because FST is composed of the inner product of two vectors, it is necessary to represent the tensor at a higher resolution than for the original image to avoid spectrum aliasing, which is automatically implemented by the calculation of the twofold oversampling fractional derivative (i.e., double-sampling FST, DFST22Double-sampling fractional structure tensor. ); (3) apply a non-local regularizing method to all four channels of the resulting DFST based on the repeatability of textures, which integrates non-local geometric characteristics to effectively infer how damaged textures extend. Finally, we insert the modified non-locally regularized DFST (NDFST33Nonlocally-regularized double-sampling fractional structure tensor. ) into an anisotropic PDE to effectively guide the diffusion process and design its numerical implementation scheme. The experimental results show that, for fine-grained texture images, the proposed NDFST is equipped to accurately detect subtle and complex texture features, and the resulting NDFST-based PDE is particularly effective not only for recovering non-homogeneous structures, but also for the restitution of fine-grained textures.

Inpainting via High-dimensional Universal Shearlet Systems

Thresholding and compressed sensing in combination with both wavelet and shearlet transforms have been very successful in inpainting tasks. Recent results have demonstrated that shearlets outperform wavelets in the problem of image inpainting. In this paper, we provide a general framework for universal shearlet systems in high dimensions. This theoretical framework is used to analyze the recovery of missing data via $\ell^{1} $ minimization in an abstract model situation. In addition, we set up a particular model inspired by seismic data and a box mask to model missing data. Finally, the results of numerical experiments comparing various inpainting methods are presented.