Inpainting Model Research Articles

RECONSTRUCTED TEETH IMAGE FROM BRACES WITH GAN

Recent advances in deep learning models have shown promising potential in object removal, which is the task of replacing undesired objects with appropriate pixel values using known context. Object removal-based deep learning can commonly be solved by modeling it as the Img2Img (image to image) translation or Inpainting. Instead of dealing with a large context, this paper aims at a specific application of object removal, that is, erasing braces trace out of an image having teeth with braces (called braces2teeth problem). We solved the problem by three methods corresponding to different datasets. Firstly, we use the CycleGAN model to deal with the problem that paired training data is not available. In the second case, we try to create pseudo-paired data to train the Pix2Pix model. In the last case, we utilize GraphCut combining generative inpainting model to build a user-interactive tool that can improve the result in case the user is not satisfied with previous results. To our best knowledge, this study is one of the first attempts to take the braces2teeth problem into account by using deep learning techniques and it can be applied in various fields, from health care to entertainment.

Deep learning-based image inpainting with structure map

In the past few years, deep learning-based image inpainting has made significant progress. However, many existing methods do not take into account the rationality of the structure and the fineness of the texture, which leads to the scattered structure or excessive smoothness of the repaired image. To solve this problem, we propose a two-stage image inpainting model composed of structure generation network and texture generation network. The structure generation network focuses on the structure and color domain and uses the damaged structure map extracted from the mask image to reasonably fill the mask area to generate a complete structure map. The texture generation network uses the repaired structure map to guide the refinement process. We train the two-stage network on the public datasets Places2, CelebA, and Paris StreetView, and the experimental results show the superiority of our method over the previous methods.

Image Inpainting Using Channel Attention and Hierarchical Residual Networks

<p indent="0mm">Existing deep-learning-based inpainting methods may have some shortcomings in perceiving and presenting image information at multi-scales. For this problem, we proposed an image inpainting model based on multi-scale channel attention and a hierarchical residual backbone network. Firstly, we adopted a U-Net architecture as the generator backbone of our inpainting model to encode and decode the damaged image. Secondly, we built multi-scale hierarchical residual structures in the encoder and decoder respectively, which can improve the ability of the model to extract and express occluded image features. Finally, we designed a dilated multi-scale channel-attention block and inserted it into the skip-connection of the generator. This block can improve the utilization efficiency of low-level features in the encoder. Experimental results show that our model outperforms other classical inpainting approaches in the face, street-view inpainting tasks, both qualitatively and quantitatively.

Toward semantic image inpainting: where global context meets local geometry

Image inpainting attempts to fill the missing areas of an image with plausible content that is visually coherent with the image context. Semantic image inpainting has remained a challenging task even with the emergence of deep learning-based approaches. We propose a deep semantic inpainting model built upon a generative adversarial network and a dense U-Net network. Such a design helps achieve feature reuse while avoiding feature explosion along the upsampling path of the U-Net. The model also uses a composite loss function for the generator network to enforce a joint global and local content consistency constraint. More specifically, our new loss function combines the global reconstruction loss characterizing the semantic similarity between the missing and known image regions with the local total variation loss characterizing the natural transitions among adjacent regions. Experimental results on CelebA-HQ and Paris StreetView datasets have demonstrated encouraging performance when compared with other state-of-the-art methods in terms of both quantitative and qualitative metrics. For the CelebA-HQ dataset, the proposed method can more faithfully infer the semantics of human faces; for the StreetView dataset, our method achieves improved inpainting results in terms of more natural texture transitions, better structural consistency, and enriched textural details.

Deep learning‐based X‐ray inpainting for improving spinal 2D‐3D registration

Two-dimensional (2D)-3D registration is challenging in the presence of implant projections on intraoperative images, which can limit the registration capture range. Here, we investigate the use of deep-learning-based inpainting for removing implant projections from the X-rays to improve the registration performance. We trained deep-learning-based inpainting models that can fill in the implant projections on X-rays. Clinical datasets were collected to evaluate the inpainting based on six image similarity measures. The effect of X-ray inpainting on capture range of 2D-3D registration was also evaluated. The X-ray inpainting significantly improved the similarity between the inpainted images and the ground truth. When applying inpainting before the 2D-3D registration process, we demonstrated significant recovery of the capture range by up to 85%. Applying deep-learning-based inpainting on X-ray images masked by implants can markedly improve the capture range of the associated 2D-3D registration task.

See-Through Vision With Unsupervised Scene Occlusion Reconstruction.

Among the greatest of the challenges of minimally invasive surgery (MIS) is the inadequate visualisation of the surgical field through keyhole incisions. Moreover, occlusions caused by instruments or bleeding can completely obfuscate anatomical landmarks, reduce surgical vision and lead to iatrogenic injury. The aim of this paper is to propose an unsupervised end-to-end deep learning framework, based on fully convolutional neural networks to reconstruct the view of the surgical scene under occlusions and provide the surgeon with intraoperative see-through vision in these areas. A novel generative densely connected encoder-decoder architecture has been designed which enables the incorporation of temporal information by introducing a new type of 3D convolution, the so called 3D partial convolution, to enhance the learning capabilities of the network and fuse temporal and spatial information. To train the proposed framework, a unique loss function has been proposed which combines feature matching, reconstruction, style, temporal and adversarial loss terms, for generating high fidelity image reconstructions. Advancing the state-of-the-art, our method can reconstruct the underlying view obstructed by irregularly shaped occlusions of divergent size, location and orientation. The proposed method has been validated on in vivo MIS video data, as well as natural scenes on a range of occlusion-to-image (OIR) ratios. It has also been compared against the latest video inpainting models in terms of image reconstruction quality using different assessment metrics. The performance evaluation analysis verifies the superiority of our proposed method and its potential clinical value.

Face Image Inpainting With Evolutionary Generators

Recently, deep learning has become a mainstream method of image inpainting. It can not only restore the image texture, obtain high-level abstract features of images, but also restore semantic images such as human face images. Among these methods, generative adversarial networks (GANs) using autoencoder as the generator have become the promising model for image inpainting. These models implement the end-to-end image inpainting and also generate visually reasonable and clear image structures and textures. However, GANs often have problems with gradient vanishing and model collapse during training, so we propose a Generative Adversarial Network with Evolutionary Generators (EG-GAN) and apply it in face image inpainting. To stabilize the model training process, EG-GAN trains the generator network by evolution, combines two mutation functions as a training objective to update the parameter of generator networks, and produces offspring generators through crossover, using the matcher assists the discriminator to criticize the generated image. Experiments on various face image datasets such as CelebA-HQ and CelebA show that EG-GAN successfully overcomes the gradient vanishing problem, achieves stable and efficient training, and generates visually reasonable images.

Artificial Intelligence for Intraoperative Guidance: Using an Object Detection Model in Conjunction With Data Augmentation to Detect Parathyroid Glands During Thyroidectomy

Background: Postoperative hypoparathyroidism is a major complication of thyroidectomy that occurs when the parathyroid glands are inadvertently damaged during surgery. Although intraoperative images are rarely used to train AI because of its complex nature, our group recognized the potential of training AI to intraoperatively detect parathyroid glands using various augmentation methods. The aim of the present study was to train an effective artificial intelligence (AI) model to detect parathyroid glands during thyroidectomy and evaluate its accuracy. Methods: Video clips of the parathyroid gland were collected during thyroid lobectomy procedures. Confirmed parathyroid images were used to form three types of datasets according to augmentation status: baseline, geometric transformation, and generative adversarial network-based image inpainting. The average precision was set as the primary outcome regarding the performance of parathyroid detection performance. Findings: The performance of the AI model in detecting parathyroid glands based on baseline data was 77%. This performance was enhanced by applying both geometric transformation and image inpainting augmentation methods. Between the two, the geometric transformation data augmentation dataset demonstrated superior results compared to that of the image inpainting model. The average precision was 79% in the transformation dataset vs. 78.6% in the inpainting dataset. However, when the model was subjected to images of a completely different thyroidectomy approach, image inpainting methods proved to be more effective compared to both geometric transformation and baseline (46% vs. 37% vs. 33%). Conclusions: Our study demonstrated the utility of our AI model, which, when aided with appropriate augmentation methods, is potentially an effective and generalizable tool for intraoperative identification of parathyroid glands during thyroidectomy. Furthermore, this technology has significant potential to be easily extended to identifying important structures during other surgical procedures in general. Funding Information: Basic Science Research Program of the National Research Foundation of Korea [NRF-2019R1C1C1008384]. Declaration of Interests: We declare no competing interests. Ethics Approval Statement: Institutional review board approval was acquired (GDIRB2019-359), and research was conducted in accordance with the Declaration of Helsinki in its latest form.

Automatic Object Removal With Obstructed Façades Completion Using Semantic Segmentation and Generative Adversarial Inpainting

Automatic object removal with obstructed facades completion in the urban environment is essential for many applications such as scene restoration, environmental impact assessment, and urban mapping. However, the previous object removal typically requires a user to manually create a mask around unwanted objects and obtain background facade information in advance, which would be labor-intensive when implementing multitasking projects. Moreover, accurately detecting objects to be removed in the cityscape and inpainting the static obstructed building facade to obtain plausible images are the main challenges for this objective. To overcome these difficulties, this study addresses the object removal with the facade inpainting problem from the following two aspects. First, we proposed an image-based cityscape elimination method for automatic object removal and facade inpainting by applying semantic segmentation to detect several classes, including pedestrians, riders, vegetation, and cars, as well as using generative adversarial networks (GANs) for filling detected regions by background textures and patching information from street-level imagery. Second, we proposed a workflow to filter unoccluded building facades from street view images automatically and tailored a dataset for the GAN-based image inpainting model with original and mask images. Furthermore, several full-reference image quality assessment (IQA) metrics are introduced to evaluate the generated image quality. Validation results demonstrated the feasibility and effectiveness of our proposed method, and the synthetic image is visually realistic and semantically consistent.

Image Inpainting Using Double Discriminator Generative Adversarial Networks

The image inpainting algorithms based on GAN have problems such as color inconsistencies of the inpainted region with the surrounding part and the mode collapse during the training. This paper proposes an inpainting algorithm based on double discriminator generative adversarial networks (GANs). This algorithm introduces the loss function of WGAN-GP into a global discriminator and local discriminator independently. It combines the Mean-Square Error (MSE) and feature reconstruction loss function to train the inpainting model and repair the missing area. The algorithm uses more dilated convolutions instead of standard convolutions to obtain a larger receptive field and the skip connection to enhance the structural prediction ability of the generator. The experiment results on the labeled faces in the wild dataset show that our algorithm improves the quality of image inpainting and the stability of the training process.

Deep Two-Stage High-Resolution Image Inpainting

In recent years, the field of image inpainting has developed rapidly, learning-based approaches show impressive results in the task of filling missing parts in an image. But most deep methods are strongly tied to the resolution of the images on which they were trained. A slight resolution increase leads to serious artifacts and unsatisfactory filling quality. These methods are therefore unsuitable for interactive image processing. In this article, we propose a method that solves the problem of inpainting arbitrary-size images. We also describe a way to better restore texture fragments in the filled area. For this, we propose to use information from neighboring pixels by shifting the original image in four directions. Moreover, this approach can work with existing inpainting models, making them almost resolution-independent without the need for retraining. We also created a GIMP plugin that implements our technique. The plugin, code, and model weights are available at https://github.com/a-mos/High Resolution Image Inpainting.

Nonconvex Total Generalized Variation Model for Image Inpainting

It is a challenging task to prevent the staircase effect and simultaneously preserve sharp edges in image inpainting. For this purpose, we present a novel nonconvex extension model that closely incorporates the advantages of total generalized variation and edge-enhancing nonconvex penalties. This improvement contributes to achieve the more natural restoration that exhibits smooth transitions without penalizing fine details. To efficiently seek the optimal solution of the resulting variational model, we develop a fast primal-dual method by combining the iteratively reweighted algorithm. Several experimental results, with respect to visual effects and restoration accuracy, show the excellent image inpainting performance of our proposed strategy over the existing powerful competitors.

Deep Learning-Based Masonry Wall Image Analysis

In this paper we introduce a novel machine learning-based fully automatic approach for the semantic analysis and documentation of masonry wall images, performing in parallel automatic detection and virtual completion of occluded or damaged wall regions, and brick segmentation leading to an accurate model of the wall structure. For this purpose, we propose a four-stage algorithm which comprises three interacting deep neural networks and a watershed transform-based brick outline extraction step. At the beginning, a U-Net-based sub-network performs initial wall segmentation into brick, mortar and occluded regions, which is followed by a two-stage adversarial inpainting model. The first adversarial network predicts the schematic mortar-brick pattern of the occluded areas based on the observed wall structure, providing in itself valuable structural information for archeological and architectural applications. The second adversarial network predicts the pixels’ color values yielding a realistic visual experience for the observer. Finally, using the neural network outputs as markers in a watershed-based segmentation process, we generate the accurate contours of the individual bricks, both in the originally visible and in the artificially inpainted wall regions. Note that while the first three stages implement a sequential pipeline, they interact through dependencies of their loss functions admitting the consideration of hidden feature dependencies between the different network components. For training and testing the network a new dataset has been created, and an extensive qualitative and quantitative evaluation versus the state-of-the-art is given. The experiments confirmed that the proposed method outperforms the reference techniques both in terms of wall structure estimation and regarding the visual quality of the inpainting step, moreover it can be robustly used for various different masonry wall types.

An adaptive image inpainting method based on euler's elastica with adaptive parameters estimation and the discrete gradient method

Euler's Elastica is a common approach developed based on minimizing the elastica energy. It is one of the effective approaches to solve the image inpainting problem. Nevertheless, there are two major issues: the Euler's elastica variational image inpainting model itself is multiparameter, and the performance of methods for solving the model is not high. In the article, we propose an adaptive Euler's elastica image inpainting model by combining with adaptive parameter estimation based on the smoothed structure tensor. To implement the model, a numerical algorithm based on the discrete gradient method is developed. The experiments showed that the proposed image inpainting method outperforms other state-of-the-arts methods in terms of inpainted image quality.

An image inpainting model based on the mixture of Perona–Malik equation and Cahn–Hilliard equation

In this paper, a fourth-order PDE model is proposed for image inpainting. This model is based on the mixture of Perona–Malik equation and Cahn–Hilliard equation. Using the idea of energy splitting, a numerical scheme is introduced for solving the proposed PDE model. Numerical experiments for testing the proposed model are provided at the end of the paper. The numerical results indicate that the proposed model can provide better inpainting performance with less computational time comparing to the classical PDE based inpainting models.

Nonlocal patches based Gaussian mixture model for image inpainting

We consider the inpainting problem for noisy images. It is very challenge to suppress noise when image inpainting is processed. An image patches based nonlocal variational method is proposed to simultaneously inpainting and denoising in this paper. Our approach is developed on an assumption that the small image patches should be obeyed a distribution which can be described by a high dimension Gaussian Mixture Model. By a maximum a posteriori (MAP) estimation, we formulate a new regularization term according to the log-likelihood function of the mixture model. To optimize this regularization term efficiently, we adopt the idea of the Expectation Maximization (EM) algorithm. In which, the expectation step can give an adaptive weighting function which can be regarded as a nonlocal connections among pixels. Using this fact, we built a framework for non-local image inpainting under noise. Moreover, we mathematically prove the existence of minimizer for the proposed inpainting model. By using a splitting algorithm, the proposed model are able to realize image inpainting and denoising simultaneously. Numerical results show that the proposed method can produce impressive reconstructed results when the inpainting region is rather large.

Weak and stationary solutions to a Cahn–Hilliard–Brinkman model with singular potentials and source terms

Abstract We study a phase field model proposed recently in the context of tumour growth. The model couples a Cahn–Hilliard–Brinkman (CHB) system with an elliptic reaction-diffusion equation for a nutrient. The fluid velocity, governed by the Brinkman law, is not solenoidal, as its divergence is a function of the nutrient and the phase field variable, i.e., solution-dependent, and frictionless boundary conditions are prescribed for the velocity to avoid imposing unrealistic constraints on the divergence relation. In this paper we give a first result on the existence of weak and stationary solutions to the CHB model for tumour growth with singular potentials, specifically the double obstacle potential and the logarithmic potential, which ensures that the phase field variable stays in the physically relevant interval. New difficulties arise from the interplay between the singular potentials and the solution-dependent source terms, but can be overcome with several key estimates for the approximations of the singular potentials, which maybe of independent interest. As a consequence, included in our analysis is an existence result for a Darcy variant, and our work serves to generalise recent results on weak and stationary solutions to the Cahn–Hilliard inpainting model with singular potentials.

Multi-scale generative adversarial inpainting network based on cross-layer attention transfer mechanism

Deep learning-based methods have recently shown promising results in image inpainting. These methods generate patches with visually plausible image structures and textures, which are semantically coherent with the context of surrounding regions. However, existing methods tend to generate artifacts which are inconsistent with surrounding regions, especially when dealing with complex images. Aiming at the limitations current in deep learning-based methods, this paper proposes a multi-scale generative adversarial network model based on cross-layer attention transfer mechanism. Cross-Layer Attention Transfer Module (CL-ATM) is presented to guide the filling of the corresponding low-level semantic feature map by using the high-level semantic feature map, so as to ensure visual and semantic consistency of inpainting. Meanwhile, a multi-scale generator and the multi-scale discriminators are added into the network structure. Different scales of discriminators have different receptive fields, which enable the generator to produce images with better global consistency and more details. Qualitative and quantitative experiments show that our method has superior performance against state-of-art inpainting models.

Double-Constraint Inpainting Model of a Single-Depth Image.

In real applications, obtained depth images are incomplete; therefore, depth image inpainting is studied here. A novel model that is characterised by both a low-rank structure and nonlocal self-similarity is proposed. As a double constraint, the low-rank structure and nonlocal self-similarity can fully exploit the features of single-depth images to complete the inpainting task. First, according to the characteristics of pixel values, we divide the image into blocks, and similar block groups and three-dimensional arrangements are then formed. Then, the variable splitting technique is applied to effectively divide the inpainting problem into the sub-problems of the low-rank constraint and nonlocal self-similarity constraint. Finally, different strategies are used to solve different sub-problems, resulting in greater reliability. Experiments show that the proposed algorithm attains state-of-the-art performance.

Image inpainting model with consistent global and local attributes

Image inpainting model with consistent global and local attributes