Image Inpainting Methods Research Articles

An image inpainting method for object removal based on difference degree constraint

In the inpainting method for object removal, SSD (Sum of Squared Differences) is commonly used to measure the degree of similarity between the exemplar patch and the target patch, which has a very important impact on the restoration results. Although the matching rule is relatively simple, it is likely to lead to the occurrence of mismatch error. Even worse, the error may be accumulated along with the process continues. Finally some unexpected objects may be introduced into the target region, making the result unable to meet the requirements of visual consistency. In view of these problems, we propose an inpainting method for object removal based on difference degree constraint. Firstly, we define the MSD (Mean of Squared Differences) and use it to measure the degree of differences between corresponding pixels at known positions in the target patch and the exemplar patch. Secondly, we define the SMD (Square of Mean Differences) and use it to measure the degree of differences between the pixels at known positions in the target patch and the pixels at unknown positions in the exemplar patch. Thirdly, based on MSD and SMD, we define a new matching rule and use it to find the most similar exemplar patch in the source region. Finally, we use the exemplar patch to restore the target patch. Experimental results show that the proposed method can effectively prevent the occurrence of mismatch error and improve the restoration effect.

Study on Extraction Methods for Different Components in a Carbonate Digital Core

It is difficult to carry out petrophysical experiments because of the serious damage caused to cores in the development of fractures and pores in carbonate reservoirs. The development of a three-dimensional digital core in carbonate reservoirs has become a hot topic in rock physics research. Compared with the three-dimensional digital core, including basic rock skeletons and pores in sandstone reservoirs, carbonate reservoirs also include secondary structures such as microfractures. The carbonate contains different components, and extracting these components is a very difficult problem. The resolution on the electrical image log image is high, which can clearly reflect the macrocomponents in various reservoirs. There are some blank areas between electrodes on the electrical image log, which affects the extraction of components in a three-dimensional digital core. Aiming at the serious heterogeneities in the carbonate reservoirs and affecting image inpainting on the electrical image log image, a new method of image inpainting based on a combination of multipoint geostatistics and an interpolation method is put forward. The experimental results show that this method generates faster and better full-bore images than other methods. Due to the multipeak histogram, the maximum interclass variance in the two times method is proposed to extract macrocomponents such as basic rock skeletons, pores, and connected parts. The microfractures can be extracted from the CT scanned images by using image segmentation from the combination of the watershed and OTSU methods. The experimental results prove that using extraction methods for different components enables better results to be obtained.

Generative image inpainting for link prediction

Link prediction is a fundamental task that predicts whether a link exists between two nodes based on the currently observed network. Existing approaches such as heuristic-based algorithms assume that two nodes are likely to have a link in a network. In fact, they limit algorithm effectiveness when the assumptions are not correct. Moreover, these link prediction algorithms lack generalization ability, which indicates that they have different effects on different types of networks. For example, the common neighbours algorithm works well on social networks, but it shows poor performance on electric power networks. Inspired by the image inpainting technology of generative adversarial networks and the adjacency matrix representation of networks, we propose a new framework for link prediction based on the image inpainting method, named the Generative Image Inpainting for Link Prediction algorithm (GIILP), to address these problems. The key idea of the GIILP is that the network can be converted into an image (the image is a form of the adjacency matrix (two dimensions)). Pixel values represent the likelihood of two nodes. Thus, the problem of predicting a possible link between nodes is converted into filling missing pixels in an image. The link information of the network can be expressed by the image information, which means that our algorithm does not need assumptions such as heuristics, and works regardless of the dataset type. The experimental results on multiple link prediction public datasets demonstrate that our algorithm has an advantage over other algorithms, including heuristic-based and deep learning methods.

Image Inpainting Based on Multi-Patch Match with Adaptive Size

Patch-based image inpainting methods iteratively fill the missing region via searching the best sample patch from the source region. However, most of the existing approaches basically use the fixed size of patch regardless of content features nearby, which may lead to inpainting defects. Also, global match is needed for searching the best sample patch, but only to fill one target patch in each iteration, resulting in low efficiency. To handle the issues above, we first evaluate the nonuniformity in an image, by which the patch size is adaptively determined. Moreover, we divide the source region into multiple non-overlapping subregions with different nonuniformity levels, and the patch match proceeds in every subregion, respectively. This strategy not only saves the match time for single target patch, but also reduces the mismatch, and enables the simultaneous filling of multiple target patches in a single iteration. Experimental results show that in comparison to previous patch-based works, our method has achieved further improvement both in quality and efficiency. We believe our method could provide a new way for patch match with better accuracy and efficiency in image inpainting tasks.

Global and Local Attention-Based Free-Form Image Inpainting.

Deep-learning-based image inpainting methods have shown significant promise in both rectangular and irregular holes. However, the inpainting of irregular holes presents numerous challenges owing to uncertainties in their shapes and locations. When depending solely on convolutional neural network (CNN) or adversarial supervision, plausible inpainting results cannot be guaranteed because irregular holes need attention-based guidance for retrieving information for content generation. In this paper, we propose two new attention mechanisms, namely a mask pruning-based global attention module and a global and local attention module to obtain global dependency information and the local similarity information among the features for refined results. The proposed method is evaluated using state-of-the-art methods, and the experimental results show that our method outperforms the existing methods in both quantitative and qualitative measures.

Attentional coarse-and-fine generative adversarial networks for image inpainting

Deep learning based image inpainting methods have recently made significant improvements, which can now generate visually realistic and semantically plausible pixels for large missing regions. However, blurry textures and distorted edges are the two main problems of these methods. In nature, they are caused by the ineffectiveness of utilizing features. Thus, we propose a coarse-and-fine structure, which can extract coarse features with a larger receptive field to guarantee the accuracy of global semantic information, and can simultaneously extract fine features with a smaller receptive field at multiple levels to serve as a supplement in a parallel manner. To avoid the artifacts caused by the missing part of image, we flexibly use the gating mechanism and propose an interleaved gated residual block (IGRB) to encourage the useful information flow through the neural network. Moreover, we propose a channel and spatial attention block (CSAB) to alleviate the influence of redundant information and better model the long-range dependency between different regions in the image. Extensive experiments on faces, natural objects and scenes demonstrate that our method outperforms the existing state-of-the-art methods.

Multi-granularity generative adversarial nets with reconstructive sampling for image inpainting

Image inpainting based on deep neural network has drawn more attention with the development of deep learning, then Generative adversarial nets(GANs) which is a combination of multiple deep networks has been applied to image inpainting and achieved top-level performance. However, GANs-based inpainting method always suffers from unstable training and vanishing gradient. In this paper, we present an image inpainting method based on GANs with reconstructive sampling and multi-granularity generative adversarial strategy. The key idea is to solve unstable training and vanishing gradient by proposed reconstructive sampling, in which we sample from reconstructive distribution than low-dimension noise. It has been proved that reconstructive sampling is effective to avoid unstable training and gradient vanish. Then, multi-granularity generative adversarial strategy, which is decomposed into two steps, is adopted to make inpainted image more continuous and realistic in texture structure and vision, respectively. Extensive experiments show that ours brings substantial improvements over other state-of-the-art image inpainting algorithms in distortion and perception evaluation. Besides, comparisons on stability with baselines show that our method gains better stability during image inpainting.

Region Normalization for Image Inpainting

Feature Normalization (FN) is an important technique to help neural network training, which typically normalizes features across spatial dimensions. Most previous image inpainting methods apply FN in their networks without considering the impact of the corrupted regions of the input image on normalization, e.g. mean and variance shifts. In this work, we show that the mean and variance shifts caused by full-spatial FN limit the image inpainting network training and we propose a spatial region-wise normalization named Region Normalization (RN) to overcome the limitation. RN divides spatial pixels into different regions according to the input mask, and computes the mean and variance in each region for normalization. We develop two kinds of RN for our image inpainting network: (1) Basic RN (RN-B), which normalizes pixels from the corrupted and uncorrupted regions separately based on the original inpainting mask to solve the mean and variance shift problem; (2) Learnable RN (RN-L), which automatically detects potentially corrupted and uncorrupted regions for separate normalization, and performs global affine transformation to enhance their fusion. We apply RN-B in the early layers and RN-L in the latter layers of the network respectively. Experiments show that our method outperforms current state-of-the-art methods quantitatively and qualitatively. We further generalize RN to other inpainting networks and achieve consistent performance improvements.

Analysis of Artifacts in Shell-Based Image Inpainting: Why They Occur and How to Eliminate Them

In this paper we study a class of fast geometric image inpainting methods based on the idea of filling the inpainting domain in successive shells from its boundary inwards. Image pixels are filled by assigning them a color equal to a weighted average of their already filled neighbors. However, there is flexibility in terms of the order in which pixels are filled, the weights used for averaging, and the neighborhood that is averaged over. Varying these degrees of freedom leads to different algorithms, and indeed the literature contains several methods falling into this general class. All of them are very fast, but at the same time all of them leave undesirable artifacts such as “kinking” (bending) or blurring of extrapolated isophotes. Our objective in this paper is to build a theoretical model in order to understand why these artifacts occur and what, if anything, can be done about them. Our model is based on two distinct limits: a continuum limit in which the pixel width h rightarrow 0 and an asymptotic limit in which h > 0 but h ll 1. The former will allow us to explain “kinking” artifacts (and what to do about them) while the latter will allow us to understand blur. Both limits are derived based on a connection between the class of algorithms under consideration and stopped random walks. At the same time, we consider a semi-implicit extension in which pixels in a given shell are solved for simultaneously by solving a linear system. We prove (within the continuum limit) that this extension is able to completely eliminate kinking artifacts, which we also prove must always be present in the direct method. Finally, we show that although our results are derived in the context of inpainting, they are in fact abstract results that apply more generally. As an example, we show how our theory can also be applied to a problem in numerical linear algebra.

Group Sparsity Residual Constraint with Non-Local Priors for Image Restoration.

Group sparse representation (GSR) has made great strides in image restoration producing superior performance, realized through employing a powerful mechanism to integrate the local sparsity and nonlocal self-similarity of images. However, due to some form of degradation (e.g., noise, down-sampling or pixels missing), traditional GSR models may fail to faithfully estimate sparsity of each group in an image, thus resulting in a distorted reconstruction of the original image. This motivates us to design a simple yet effective model that aims to address the above mentioned problem. Specifically, we propose group sparsity residual constraint with nonlocal priors (GSRC-NLP) for image restoration. Through introducing the group sparsity residual constraint, the problem of image restoration is further defined and simplified through attempts at reducing the group sparsity residual. Towards this end, we first obtain a good estimation of the group sparse coefficient of each original image group by exploiting the image nonlocal self-similarity (NSS) prior along with self-supervised learning scheme, and then the group sparse coefficient of the corresponding degraded image group is enforced to approximate the estimation. To make the proposed scheme tractable and robust, two algorithms, i.e., iterative shrinkage/thresholding (IST) and alternating direction method of multipliers (ADMM), are employed to solve the proposed optimization problems for different image restoration tasks. Experimental results on image denoising, image inpainting and image compressive sensing (CS) recovery, demonstrate that the proposed GSRC-NLP based image restoration algorithm is comparable to state-of-the-art denoising methods and outperforms several state-of-the-art image inpainting and image CS recovery methods in terms of both objective and perceptual quality metrics.

基于信息熵和梯度因子的改进Criminisi图像修复方法

基于信息熵和梯度因子的改进Criminisi图像修复方法

Raindrop Removal With Light Field Image Using Image Inpainting

In this paper, we propose a method that removes raindrops with light field image using image inpainting. We first use the depth map generated from light field image to detect raindrop regions which are then expressed as a binary mask. The original image with raindrops is improved by refocusing on the far regions and filtering by a high-pass filter. With the binary mask and the enhanced image, image inpainting is then utilized to eliminate raindrops from the original image. We compare pre-trained models of several deep learning based image inpainting methods. A light field raindrop dataset is released to verify our method. Image quality analysis is performed to evaluate the proposed image restoration method. The recovered images are further applied to object detection and visual localization tasks.

Image Inpainting Based on Inside–Outside Attention and Wavelet Decomposition

Recently developed deep learning-based image inpainting methods have suggested their potential applications in filling large missing regions displaying plausible content. Nevertheless, most existing studies either merely considered the external information of the missing regions or relied on the region context to yield semantically coherent patches while overlooking the semantic relevance and feature continuity exhibited by missing regions; these demerits are likely to cause a discontinuous contextual structure and blurry texture details. In this work, a novel inside-outside attention layer (IOA) was proposed, capable of exploiting unmasked image features as references as well as learning the affinity between hole features to predict more consistent semantic information. The adversarial loss attributed exclusively on the natural image level cannot adequately generate a sharp texture detail. To address this problem, a texture component discriminator was introduced via wavelet decomposition to enhance the specific performance. Several experiments were performed on the CelebA and Places2 datasets. As revealed from the results, in contrast to the existing research, the proposed method is capable of restoring images with complex structures and significantly enhancing plausible structure and visual quality.

An Image Inpainting Method for Interleaved 3D Stacked Image Sensor

This paper presents an image inpainting method for 3D stacked image sensor. A typical structure for 3D stacked image sensor includes a pixel array, an analog-to-digital converter (ADC) array, and an image signal processor (ISP), which are connected vertically by through silicon vias (TSVs) or microbumps ( $\mu $ bumps). To improve the quality of the output images when existing one or more $\mu $ bump/TSV/ADC failures, it is proposed to interleave the connections from pixel array to ADC array and recover the inaccessible data in the ISP. The sub-array of pixels corresponding to the defective $\mu $ bump/TSV/ADC would be convenient to be inpainted. An inpainting method used for the damaged block of pixels is proposed, the image is progressively reduced through a process of the discrete wavelet transform (DWT) and down-sampling until the damaged block converts to one pixel only. The reduced image is inpainted and optimized, and then the original image size would be restored by up-sampling. Evaluations over a range of images demonstrate that the proposed method improves average peak-signal-to-noise-ratio (PSNR) by 1.93-3.25 dB compared with the exemplar-based method and the TV method in case single or multiple $\mu $ bump/TSV/ADC defects.

Research on texture image inpainting of jacquard fabric based on non-single vision

For the texture image inpainting of jacquard fabric with a large damaged region but complete data information around it, traditional exemplar-based image inpainting algorithms are not only have rapid falling traditional confidence value, but also the matching precision is greatly limited due to the lack of robustness, which results in a wrong guided direction and an unsatisfactory inpainting effect. To solve the above-mentioned problems, a novel digital image acquisition system for damaged fabric was designed and one set of texture image inpainting algorithms of jacquard fabric based on non-single vision was developed. In this image inpainting method, the images to be inpainted and target images were obtained through the non-single-vision imaging method. By using the image registration and Poisson image blending technology, the damaged region was filled by block matching first and then patches from the target image to the data missing region were copied. The experimental results clearly showed that the proposed method achieves better results in visual appearance and inpainting quality, compared with traditional approaches for inpainting. In addition, this method shows broad potential application prospects in the virtual restoration of ancient fabrics using image inpainting technology.

Hybrid Patch Propagation and Diffusion Technique for Image Inpainting

This paper deals with anovel hybrid technique based onpatch propagation and diffusion method for image inpainting. The presented technique is used to reconstruct a damaged image. Image inpainting is a method to fill the hole with the best plausible, which is created due to damage. The novel hybridization technique of diffusion-based and exemplar-based is presented to overcome the existing problem of inpainting. The method is tested on the image dataset of TUM-IID. The performance of present method is measured using quality factor (QF) analysis, peak signal-noise ratio (PSNR), and comparing similarity by structure similarity index (SSIM). Result demonstrates that the proposed calculation performed better contrast with the existing exemplar-based technique.

Image Inpainting: A Review

Although image inpainting, or the art of repairing the old and deteriorated images, has been around for many years, it has gained even more popularity because of the recent development in image processing techniques. With the improvement of image processing tools and the flexibility of digital image editing, automatic image inpainting has found important applications in computer vision and has also become an important and challenging topic of research in image processing. This paper is a brief review of the existing image inpainting approaches we first present a global vision on the existing methods for image inpainting. We attempt to collect most of the existing approaches and classify them into three categories, namely, sequential-based, CNN-based and GAN-based methods. In addition, for each category, a list of methods for the different types of distortion on the images is presented. Furthermore, collect a list of the available datasets and discuss these in our paper. This is a contribution for digital image inpainting researchers trying to look for the available datasets because there is a lack of datasets available for image inpainting. As the final step in this overview, we present the results of real evaluations of the three categories of image inpainting methods performed on the datasets used, for the different types of image distortion. In the end, we also present the evaluations metrics and discuss the performance of these methods in terms of these metrics. This overview can be used as a reference for image inpainting researchers, and it can also facilitate the comparison of the methods as well as the datasets used. The main contribution of this paper is the presentation of the three categories of image inpainting methods along with a list of available datasets that the researchers can use to evaluate their proposed methodology against.

A qualitative study of Exemplar based Image Inpainting

Image inpainting is exploring the way of filling the missing or left out regions in an image by only having knowledge of surrounding pixel values. One of the block matching algorithms use for image inpainting is exemplar based technique. Exemplar based image inpainting method search the best matching patch for the target region in the image and when it is obtained, the pixel values are copied from the source region and pasted to the target region. The selections of different patch matching criteria, patch size and filling order have a great affect on the results. This paper describes some of the variations in traditional exemplar method which include change in patch selection, priority, patch size and multi-resolution inpainting. The advantages and drawbacks for the modified methods are also discussed here.

Robust quaternion matrix completion with applications to image inpainting

SummaryIn this paper, we study robust quaternion matrix completion and provide a rigorous analysis for provable estimation of quaternion matrix from a random subset of their corrupted entries. In order to generalize the results from real matrix completion to quaternion matrix completion, we derive some new formulas to handle noncommutativity of quaternions. We solve a convex optimization problem, which minimizes a nuclear norm of quaternion matrix that is a convex surrogate for the quaternion matrix rank, and the ℓ1‐norm of sparse quaternion matrix entries. We show that, under incoherence conditions, a quaternion matrix can be recovered exactly with overwhelming probability, provided that its rank is sufficiently small and that the corrupted entries are sparsely located. The quaternion framework can be used to represent red, green, and blue channels of color images. The results of missing/noisy color image pixels as a robust quaternion matrix completion problem are given to show that the performance of the proposed approach is better than that of the testing methods, including image inpainting methods, the tensor‐based completion method, and the quaternion completion method using semidefinite programming.

Learning Deep CNN Denoiser Priors for Depth Image Inpainting

Due to the rapid development of RGB-D sensors, increasing attention is being paid to depth image applications. Depth images play an important role in computer vision research. In this paper, we address the problem of inpainting for single depth images without corresponding color images as a guide. Within the framework of model-based optimization methods for depth image inpainting, the split Bregman iteration algorithm was used to transform depth image inpainting into the corresponding denoising subproblem. Then, we trained a set of efficient convolutional neural network (CNN) denoisers to solve this subproblem. Experimental results demonstrate the effectiveness of the proposed algorithm in comparison with three traditional methods in terms of visual quality and objective metrics.