Image Inpainting Tasks Research Articles

A Novel Image Inpainting Method Used for Veneer Defects Based on Region Normalization.

The quality of the veneer directly affects the quality and grade of a blockboard made of veneer. To improve the quality and utilization of a defective veneer, a novel deep generative model-based method is proposed, which can generate higher-quality inpainting results. A two-phase network is proposed to stabilize the network training process. Then, region normalization is introduced to solve the inconsistency problem between the mean and standard deviation, improve the convergence speed of the model, and prevent the model gradient from exploding. Finally, a hybrid dilated convolution module is proposed to reconstruct the missing areas of the panels, which alleviates the gridding problem by changing the dilation rate. Experiments on our dataset prove the effectiveness of the improved approach in image inpainting tasks. The results show that the PSNR of the improved method reaches 33.11 and the SSIM reaches 0.93, which are superior to other methods.

SIFNet: Free-form image inpainting using color split-inpaint-fuse approach

Recent deep learning-based approaches have shown outstanding performance in generating visually plausible and refined contents for the missing regions in free-form image inpainting tasks. However, most of the existing methods employ a coarse-to-refine approach where the refinement process depends on a single coarse estimation, often leading to texture and structure inconsistencies. Though several existing methods focus on incorporating additional inputs to mitigate this problem, no learning-based studies have investigated the effects of decomposing input corrupted image into luma and chroma images and performing decoupled inpainting of the decomposed components. To this end, we propose a Split-Inpaint-Fuse Network (SIFNet), an end-to-end two-stage inpainting approach that uses a split-inpaint sub-network for separately inpainting the corrupted luma and chroma images using two decoupled branches in the coarse stage and a fusion sub-network for fusing the inpainted luma and chroma images into a refined image in the refinement stage. Additionally, we propose two attention mechanisms for the coarse stage – a progressive context module to find the patch-level feature similarity for the luma image reconstruction and a spatial-channel context module to find important spatial and channel features for the chroma image reconstruction. Experimental results reveal that our Split-Inpaint-Fuse approach outperforms the existing inpainting methods by comparative margins. In addition, extensive ablation studies confirm the effectiveness of the proposed approach, constituting modules and architectural choices.

AFFNet: Attention Mechanism Network Based on Fusion Feature for Image Cloud Removal

Clouds frequently affect optical remote sensing pictures throughout the gathering process, resulting in low-resolution images that affect judgment and subsequent use of ground data. Because of the thick cloud cover, the ground surface information below is entirely incorrect. This kind of end-to-end image problem should not be dismissed as a simple task of image inpainting or image translation. Therefore, this paper proposes a multi-head self-attention module based on the encoding–decoding generative adversarial network, considering the redundant information of the deep network, furthermore this paper introduces Ghost convolution to effectively solve the influence of redundant feature maps in the network on the increase of time consumption and parameters. The method in this paper can solve the problem of cloud occlusion. By considering spatial information, it can better complete the prediction of cloud removal. It can reduce the amount of network calculations and parameters while maintaining the effect. In addition, Feature Fusion Module is proposed to integrate high-level features with low-level features, so that the network can extract enough feature information and better supplement the details to complete the cloud removal. The method in this paper has achieved excellent results on the RICE1 and RICE2 datasets.

Partial Convolution for Padding, Inpainting, and Image Synthesis.

Partial convolution weights convolutions with binary masks and renormalizes on valid pixels. It was originally proposed for image inpainting task because a corrupted image processed by a standard convolutional often leads to artifacts. Therefore, binary masks are constructed that define the valid and corrupted pixels, so that partial convolution results are only calculated based on valid pixels. It has been also used for conditional image synthesis task, so that when a scene is generated, convolution results of an instance depend only on the feature values that belong to the same instance. One of the unexplored applications for partial convolution is padding which is a critical component of modern convolutional networks. Common padding schemes make strong assumptions about how the padded data should be extrapolated. We show that these padding schemes impair model accuracy, whereas partial convolution based padding provides consistent improvements across a range of tasks. In this article, we review partial convolution applications under one framework. We conduct a comprehensive study of the partial convolution based padding on a variety of computer vision tasks, including image classification, 3D-convolution-based action recognition, and semantic segmentation. Our results suggest that partial convolution-based padding shows promising improvements over strong baselines.

Image Inpainting With Local and Global Refinement.

Image inpainting has made remarkable progress with recent advances in deep learning. Popular networks mainly follow an encoder-decoder architecture (sometimes with skip connections) and possess sufficiently large receptive field, i.e., larger than the image resolution. The receptive field refers to the set of input pixels that are path-connected to a neuron. For image inpainting task, however, the size of surrounding areas needed to repair different kinds of missing regions are different, and the very large receptive field is not always optimal, especially for the local structures and textures. In addition, a large receptive field tends to involve more undesired completion results, which will disturb the inpainting process. Based on these insights, we rethink the process of image inpainting from a different perspective of receptive field, and propose a novel three-stage inpainting framework with local and global refinement. Specifically, we first utilize an encoder-decoder network with skip connection to achieve coarse initial results. Then, we introduce a shallow deep model with small receptive field to conduct the local refinement, which can also weaken the influence of distant undesired completion results. Finally, we propose an attention-based encoder-decoder network with large receptive field to conduct the global refinement. Experimental results demonstrate that our method outperforms the state of the arts on three popular publicly available datasets for image inpainting. Our local and global refinement network can be directly inserted into the end of any existing networks to further improve their inpainting performance. Code is available at https://github.com/weizequan/LGNet.git.

Progressively Inpainting Images Based on a Forked-Then-Fused Decoder Network.

Image inpainting aims to fill in corrupted regions with visually realistic and semantically plausible contents. In this paper, we propose a progressive image inpainting method, which is based on a forked-then-fused decoder network. A unit called PC-RN, which is the combination of partial convolution and region normalization, serves as the basic component to construct inpainting network. The PC-RN unit can extract useful features from the valid surroundings and can suppress incompleteness-caused interference at the same time. The forked-then-fused decoder network consists of a local reception branch, a long-range attention branch, and a squeeze-and-excitation-based fusing module. Two multi-scale contextual attention modules are deployed into the long-range attention branch for adaptively borrowing features from distant spatial positions. Progressive inpainting strategy allows the attention modules to use the previously filled region to reduce the risk of allocating wrong attention. We conduct extensive experiments on three benchmark databases: Places2, Paris StreetView, and CelebA. Qualitative and quantitative results show that the proposed inpainting model is superior to state-of-the-art works. Moreover, we perform ablation studies to reveal the functionality of each module for the image inpainting task.

Dynamic Selection Network for Image Inpainting.

Image inpainting is a challenging computer vision task that aims to fill in missing regions of corrupted images with realistic contents. With the development of convolutional neural networks, many deep learning models have been proposed to solve image inpainting issues by learning information from a large amount of data. In particular, existing algorithms usually follow an encoding and decoding network architecture in which some operations with standard schemes are employed, such as static convolution, which only considers pixels with fixed grids, and the monotonous normalization style (e.g., batch normalization). However, these techniques are not well-suited for the image inpainting task because the random corrupted regions in the input images tend to mislead the inpainting process and generate unreasonable content. In this paper, we propose a novel dynamic selection network (DSNet) to solve this problem in image inpainting tasks. The principal idea of the proposed DSNet is to distinguish the corrupted region from the valid ones throughout the entire network architecture, which may help make full use of the information in the known area. Specifically, the proposed DSNet has two novel dynamic selection modules, namely, the validness migratable convolution (VMC) and regional composite normalization (RCN) modules, which share a dynamic selection mechanism that helps utilize valid pixels better. By replacing vanilla convolution with the VMC module, spatial sampling locations are dynamically selected in the convolution phase, resulting in a more flexible feature extraction process. Besides, the RCN module not only combines several normalization methods but also normalizes the feature regions selectively. Therefore, the proposed DSNet can illustrate realistic and fine-detailed images by adaptively selecting features and normalization styles. Experimental results on three public datasets show that our proposed method outperforms state-of-the-art methods both quantitatively and qualitatively.

Stick-Breaking Dependent Beta Processes with Variational Inference

The beta processes (BP) is a powerful nonparametric tool in feature learning, which is often used as the prior of Bernoulli process for choosing features from a feature dictionary. However, it still shows a limitation in processing some real-world data, as the stood and BP is independent of data. In practice, the probabilities of selecting features in the latent space for different observed data are different, and they are usually dependent on some information from data, such as the location or time information. For example, data with closer distances usually have similar features. This kind of information (location or time) often called covariates, which are ignored in most BP-related literature. To account this problem, we propose a variational inference based dependent beta processes (VDBP), in which the dependent beta process is constructed using the stick-breaking representation and the dependency on the covariates is captured by a Gaussian process prior. An elegant representation of variational inference for with VDBP prior is obtained, which offers the efficient training method for the models using VDBP as priors. Through instantiating a Bayesian factor analysis model with VDBP, we verify the effectiveness of the proposed VDBP in image denoising and image inpainting tasks.

Single image HDR reconstruction using a CNN with masked features and perceptual loss

Digital cameras can only capture a limited range of real-world scenes' luminance, producing images with saturated pixels. Existing single image high dynamic range (HDR) reconstruction methods attempt to expand the range of luminance, but are not able to hallucinate plausible textures, producing results with artifacts in the saturated areas. In this paper, we present a novel learning-based approach to reconstruct an HDR image by recovering the saturated pixels of an input LDR image in a visually pleasing way. Previous deep learning-based methods apply the same convolutional filters on wellexposed and saturated pixels, creating ambiguity during training and leading to checkerboard and halo artifacts. To overcome this problem, we propose a feature masking mechanism that reduces the contribution of the features from the saturated areas. Moreover, we adapt the VGG-based perceptual loss function to our application to be able to synthesize visually pleasing textures. Since the number of HDR images for training is limited, we propose to train our system in two stages. Specifically, we first train our system on a large number of images for image inpainting task and then fine-tune it on HDR reconstruction. Since most of the HDR examples contain smooth regions that are simple to reconstruct, we propose a sampling strategy to select challenging training patches during the HDR fine-tuning stage. We demonstrate through experimental results that our approach can reconstruct visually pleasing HDR results, better than the current state of the art on a wide range of scenes.

Image Inpainting Based on Multi-Patch Match with Adaptive Size

Patch-based image inpainting methods iteratively fill the missing region via searching the best sample patch from the source region. However, most of the existing approaches basically use the fixed size of patch regardless of content features nearby, which may lead to inpainting defects. Also, global match is needed for searching the best sample patch, but only to fill one target patch in each iteration, resulting in low efficiency. To handle the issues above, we first evaluate the nonuniformity in an image, by which the patch size is adaptively determined. Moreover, we divide the source region into multiple non-overlapping subregions with different nonuniformity levels, and the patch match proceeds in every subregion, respectively. This strategy not only saves the match time for single target patch, but also reduces the mismatch, and enables the simultaneous filling of multiple target patches in a single iteration. Experimental results show that in comparison to previous patch-based works, our method has achieved further improvement both in quality and efficiency. We believe our method could provide a new way for patch match with better accuracy and efficiency in image inpainting tasks.

Generative image inpainting via edge structure and color aware fusion

Very recently, with the widespread research of deep learning, its achievements are increasingly evident in image inpainting tasks. However, many existing methods fail to effectively reconstruct vivid contents and refine structures. In order to solve this issue, in this paper, a novel two-stage generative adversarial network based on the fusion of edge structures and color aware maps is proposed. In the first-stage network, edges with missing regions are employed to train an edge structure generator. Meanwhile, the input image with missing regions is transformed into a global color feature map after the content aware fill algorithm and a large kernel size Gaussian filtering. In the second-stage network, the image fused from the edge map and the color map is used as a label to guide the network to reconstruct the refined image. Qualitative and quantitative experiments conducted on multiple public datasets demonstrate that the method proposed in this paper has superior performance.

E2I: Generative Inpainting From Edge to Image

Deep learning-based methods especially using convolutional neural network (CNN) and generative adversarial network (GAN) have achieved certain success for the task of image inpainting. The previous methods usually try to generate the content in the missing areas from scratch. However, these methods have difficulty in producing salient image structures that appear natural and consistent with the neighborhood, especially when the missing area is large. In this paper, we address the challenge by introducing edges into the convolutional GAN-based inpainting. We split the inpainting task into two steps: first edge generation, then edge-based image generation. We adopt CNN to accomplish the two steps and use GAN-based training, thus our method is named E2I: generative inpainting from edge to image. Specifically, we adopt a deep network-based edge detector to achieve an edgeness map of an incomplete image, then we fill-in the missing areas in the edgeness map, and finally generate the missing pixels with the assistance of the complete edgeness map. We verify the proposed method on three challenging image datasets: Places2, ImageNet, and CelebA. We also compare our method with the state-of-the-arts on the Places2 test set. Our experimental results demonstrate the superior performance of our method in producing more plausible inpainting results.

Multistage attention network for image inpainting

Image inpainting refers to the process of restoring the mask regions of damaged images. Existing inpainting algorithms have exhibited outstanding performance on certain inpainting tasks that are focused on recovering small masks or square masks. Tasks that attempt to reconstruct large proportion of damaged images can still be improved. Although many attention-related algorithms have been proposed to solve image inpainting tasks, most of them ignore the requirements to balancing the detail and style level. In this paper, we propose a novel image inpainting method for large-scale irregular masks. We introduce a special multistage attention module that considers structure consistency and detail fineness. The proposed multistage attention module operates in a coarse to-fine manner, where the early stage performs large feature patch swapping and ensures the global consistency in images, and the next stage swaps small patches to refine the texture. Then, we adopt a partial convolution strategy to avoid the misuse of invalid data during convolution. Several losses are combined as the training objective function to generate excellent results with global consistency and exquisite detail. Qualitative and quantitative experiments on the Paris StreetView, CelebA, and Places2 datasets demonstrate the superior performance of the proposed approach compared with state-of-the-art models.

Deep Fusion Local-Content and Global-Semantic for Image Inpainting

The upsampling layers are adopted in almost all the existing encoder-decoder based generative adversarial networks (GANs), which have shown promising results in the image inpainting field. However, existing upsampling layers (e.g. deconvolution and bilinear interpolation) suffer from two limitations: (1) they obtain few semantic information from the global structure. (2) upsampling layer could hardly capture the local content details. To eliminate the above issues, we propose a deep Fusion local-content and global-semantic (DFLG) model that is both effective and general. The DFLG model mainly consists of four components: the Local Content-Response (LCR) module, the pixel-shuffle operator, the Global Semantic-Aware (GSA) module and the reassembly module. Firstly, the LCR module divides the channel into several groups, then utilizes the squeeze-and-excitation mechanism within each group to capture the correlation between channels. Secondly, the pixel shuffle operator reshapes depth on the channel space into width and height on the spatial space, which transforms the correlation within groups on the channel space into correlation within patches on the spatial space. Next, the GSA module employs a patch-based spatial attention mechanism to calculate the correlation between different patches. Finally, the reassembly module refines the feature map. Furthermore, we propose a novel loss function called Attention Loss (ATLoss), which guides the network to concentrate on regions with obvious artifacts. The experiments on CelebA-HQ, Places2, and Paris StreetView datasets demonstrate the effectiveness of our proposed methods in image inpainting tasks and the capability of obtaining images with higher quality.

Low-Rank Tensor Completion and Total Variation Minimization for Color Image Inpainting

Low-rank (LR) and total variation (TV) are two most frequent priors that occur in image processing problems, and they have sparked a tremendous amount of researches, particularly for moving from scalar to vector, matrix or even high-order based functions. However, discretization schemes used for TV regularization often ignore the difference of the intrinsic properties, so it will lead to the problem that local smoothness cannot be effectively generated, let alone the problem of blurred edges. To address the image inpainting problem with corrupted data, in this paper, the color images are naturally considered as three-dimensional tensors, whose prior of smoothness can be measured by varietal TV norm along different dimensions. Specifically, we propose incorporating Shannon total variation (STV) and low-rank tensor completion (LRTC) into the construction of the final cost function, in which a new nonconvex low-rank constraint, namely truncated $\gamma $ -norm, is involved for closer rank approximation. Moreover, two methods are developed, i.e., LRRSTV and LRRSTV-T, due to the fact that LRTC can be represented by tensor unfolding and tensor decomposition. The final solution can be achieved by a practical variant of the augmented Lagrangian alternating direction method (ALADM). Experiments on color image inpainting tasks demonstrate that the proposed methods perform better then the state-of-the-art algorithms, both qualitatively and quantitatively.

Face completion with Hybrid Dilated Convolution

Image completion is a challenging task which aims to fill the missing or masked regions in images with plausibly synthesized contents. In this paper, we focus on face image inpainting tasks, aiming at reconstructing missing or damaged regions of an incomplete face image given the context information. We specially design the U-Net architecture to tackle the problem. The proposed U-Net based method combines Hybrid Dilated Convolution (HDC) and spectral normalization to fill in missing regions of any shape with sharp structures and fine-detailed textures. We perform both qualitative and quantitative evaluation on two challenging face datasets. Experimental results demonstrate that our method outperforms previous learning-based inpainting methods. The proposed method can generate realistic and semantically plausible images.

Promising Generative Adversarial Network Based Sinogram Inpainting Method for Ultra-Limited-Angle Computed Tomography Imaging.

Limited-angle computed tomography (CT) image reconstruction is a challenging problem in the field of CT imaging. In some special applications, limited by the geometric space and mechanical structure of the imaging system, projections can only be collected with a scanning range of less than 90°. We call this kind of serious limited-angle problem the ultra-limited-angle problem, which is difficult to effectively alleviate by traditional iterative reconstruction algorithms. With the development of deep learning, the generative adversarial network (GAN) performs well in image inpainting tasks and can add effective image information to restore missing parts of an image. In this study, given the characteristic of GAN to generate missing information, the sinogram-inpainting-GAN (SI-GAN) is proposed to restore missing sinogram data to suppress the singularity of the truncated sinogram for ultra-limited-angle reconstruction. We propose the U-Net generator and patch-design discriminator in SI-GAN to make the network suitable for standard medical CT images. Furthermore, we propose a joint projection domain and image domain loss function, in which the weighted image domain loss can be added by the back-projection operation. Then, by inputting a paired limited-angle/180° sinogram into the network for training, we can obtain the trained model, which has extracted the continuity feature of sinogram data. Finally, the classic CT reconstruction method is used to reconstruct the images after obtaining the estimated sinograms. The simulation studies and actual data experiments indicate that the proposed method performed well to reduce the serious artifacts caused by ultra-limited-angle scanning.

Inpainting-Based Virtual Try-on Network for Selective Garment Transfer

Image-based garment transfer systems aim to swap the desired clothes from a model to arbitrary users. However, existing works cannot provide the capacity for users to try on various fashion articles according to their wishes, i.e., users can decide which article (e.g., tops, pants or both) to be swapped. In this paper, we propose an Inpainting-based Virtual Try-On Network (I-VTON) which allows the user to try on arbitrary clothes from the model image in a selective manner. To realize the selectivity, we reshape the virtual try-on as a task of image inpainting. Firstly, the texture from the garment and the user are extracted respectively to form a coarse result. In this phase, users can decide which clothes they hope to try on via an interactive texture control mechanism. Secondly, the missing regions in the coarse result are recovered via a Texture Inpainting Network (TIN). We introduce a triplet training strategy to ensure the naturalness of the final result. Qualitative and quantitative experimental results demonstrate that I-VTON outperforms the state-of-the-art methods on both the garment details and the user identity. It is also confirmed our approach can flexibly transfer the clothes in a selective manner.

Face Inpainting via Nested Generative Adversarial Networks

Face inpainting aims to repaired damaged images caused by occlusion or cover. In recent years, deep learning based approaches have shown promising results for the challenging task of image inpainting. However, there are still limitation in reconstructing reasonable structures because of over-smoothed and/or blurred results. The distorted structures or blurred textures are inconsistent with surrounding areas and require further post-processing to blend the results. In this paper, we present a novel generative model-based approach, which consisted by nested two Generative Adversarial Networks (GAN), the sub-confrontation GAN in generator and parent-confrontation GAN. The sub-confrontation GAN, which is in the image generator of parent-confrontation GAN, can find the location of missing area and reduce mode collapse as a prior constraint. To avoid generating vague details, a novel residual structure is designed in the sub-confrontation GAN to deliver richer original image information to the deeper layers. The parent-confrontation GAN includes an image generation part and a discrimination part. The discrimination part of parent-confrontation GAN includes global and local discriminator, which benefits the reconstruction of overall coherency of the repaired image while obtaining local details. The experiments are executed over the publicly available dataset CelebA, and the results show that our method outperforms current state-of-the-art techniques quantitatively and qualitatively.

Image inpainting algorithm based on convolutional neural network structure and improved Deep Image Prior

Abstract: In previous studies, researchers believed that the reason for the excellent performance of convolutional neural networks was that they could learn hidden information from special-purpose datasets, and the emphasis was on learning. Recently, the authors of Deep Image Prior proved that the generator structure itself (using convolutional neural network) could extract image prior information and be used for the image inpainting task. In this paper, based on Deep Image Prior, four improvements (mix input, network noise, weight decay, and burning mean output) are proposed for preventing overfitting and improving output stability. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) in two stepwise comparative experiments showed that our image inpainting algorithm surpassed the original algorithm and state-of-the-art algorithms after adding the proposed improvements in sequence. In large hole inpainting, the PSNR of our algorithm was 3.23 dB higher than in the original Deep Image Prior. Then, in a binary Bernoulli inpainting experiment, our algorithm achieved better performance in most classical image inpainting, proving that the algorithm could use the same set of parameters for each image in the task. In addition, this experiment also illustrated the performance of burning mean output in stabilizing the output and reducing the influence of meaningless noise in the early stage of the iteration on subsequent image inpainting. Keywords: convolutional neural network; Deep Image Prior; image prior information; image inpainting; overfitting; large hole inpainting; binary Bernoulli inpainting DOI: 10.33440/j.ijpaa.20200304.135 Citation: Wang J S, Han Y H. An image inpainting algorithm based on convolutional neural network structure and improved Deep Image Prior. Int J Precis Agric Aviat, 2020; 3(4): 65–73.