Image Inpainting Methods Research Articles

Enhancing wildfire detection: a novel algorithm for controllable generation of wildfire smoke images

Background The lack of wildfire smoke image data is one of the most important factors hindering the development of image-based wildfire detection. Smoke image generation based on image inpainting techniques is a solution worthy of study. However, it is difficult to generate smoke texture with context consistency in complex backgrounds with current image inpainting methods. Aims This work aims to provide a wildfire smoke image database for specific scenarios. Methods We designed an algorithm based on generative adversarial networks (GANs) to generate smoke images. The algorithm includes a multi-scale fusion module to ensure consistency between the generated smoke and backgrounds. Additionally, a local feature-matching mechanism in the discriminator guides the generator to capture real smoke’s feature distribution. Key results We generated 13,400 wildfire smoke images based on forest background images and early fire simulation from the Fire Dynamics Simulator (FDS). Conclusions A variety of advanced object detection algorithms were trained based on the generated data. The experimental results confirmed that the addition of the generated data to the real datasets can effectively improve model performance. Implications This study paves a way for generating object datasets to enhance the reliability of watchtower or satellite wildfire monitoring.

Symmetric Connected U-Net with Multi-Head Self Attention (MHSA) and WGAN for Image Inpainting

This study presents a new image inpainting model based on U-Net and incorporating the Wasserstein Generative Adversarial Network (WGAN). The model uses skip connections to connect every encoder block to the corresponding decoder block, resulting in a strictly symmetrical architecture referred to as Symmetric Connected U-Net (SC-Unet). By combining SC-Unet with a GAN, the study aims to reconstruct images more effectively and seamlessly. The traditional discriminators only differentiate the entire image as true or false. In this study, the discriminator calculated the probability of each pixel belonging to the hole and non-hole regions, which provided the generator with more gradient loss information for image inpainting. Additionally, every block of SC-Unet incorporated a Dilated Convolutional Neural Network (DCNN) to increase the receptive field of the convolutional layers. Our model also integrated Multi-Head Self-Attention (MHSA) into selected blocks to enable it to efficiently search the entire image for suitable content to fill the missing areas. This study adopts the publicly available datasets CelebA-HQ and ImageNet for evaluation. Our proposed algorithm demonstrates a 10% improvement in PSNR and a 2.94% improvement in SSIM compared to existing representative image inpainting methods in the experiment.

Structure-Guided Image Inpainting Based on Multi-Scale Attention Pyramid Network

Current single-view image inpainting methods often suffer from low image information utilization and suboptimal repair outcomes. To address these challenges, this paper introduces a novel image inpainting framework that leverages a structure-guided multi-scale attention pyramid network. This network consists of a structural repair network and a multi-scale attention pyramid semantic repair network. The structural repair component utilizes a dual-branch U-Net network for robust structure prediction under strong constraints. The predicted structural view then serves as auxiliary information for the semantic repair network. This latter network exploits the pyramid structure to extract multi-scale features of the image, which are further refined through an attention feature fusion module. Additionally, a separable gated convolution strategy is employed during feature extraction to minimize the impact of invalid information from missing areas, thereby enhancing the restoration quality. Experiments conducted on standard datasets such as Paris Street View and CelebA demonstrate the superiority of our approach over existing methods through quantitative and qualitative comparisons. Further ablation studies, by incrementally integrating proposed mechanisms into a baseline model, substantiate the effectiveness of our multi-view restoration strategy, separable gated convolution, and multi-scale attention feature fusion.

Image Inpainting via Correlated Multi-Resolution Feature Projection.

With the advancement in image editing applications, image inpainting is gaining more attention due to its ability to recover corrupted images efficiently. Also, the existing methods for image inpainting either use two-stage coarse-to-fine architectures or single-stage architectures with a deeper network. On the other hand, shallow network architectures lack the quality of results and the methods with remarkable inpainting quality have high complexity in terms of number of parameters or average run time. Despite the improvement in the inpainting quality, these methods still lack the correlated local and global information. In this work, we propose a single-stage multi-resolution generator architecture for image inpainting with moderate complexity and superior outcomes. Here, a multi-kernel non-local (MKNL) attention block is proposed to merge the feature maps from all the resolutions. Further, a feature projection block is proposed to project features of MKNL to respective decoder for effective reconstruction of image. Also, a valid feature fusion block is proposed to merge encoder skip connection features at valid region and respective decoder features at hole region. This ensures that there will not be any redundant feature merging while reconstruction of image. Effectiveness of the proposed architecture is verified on CelebA-HQ Liu, et al. 2015, Karras et al. 2017, and Places2 Zhou et al. 2018 datasets corrupted with publicly available NVIDIA mask dataset Liu et al. 2018. The detailed ablation study, extensive result analysis, and application of object removal prove the robustness of the proposed method over existing state-of-the-art methods for image inpainting.

A Mumford-Shah Model-based Method for Inpainting Markers in Ultrasound Images

Abstract During ultrasound diagnosis in clinics, clinicians often mark the lesion area, resulting in the presence of unnecessary objects in the collected ultrasound images. These markers potentially affect subsequent image analysis. To address this problem, we proposed an image inpainting method that combines image processing with Mumford-Shah algorithm to remove markers from ultrasound images. The proposed method consists of two parts. First, the input ultrasound images were processed by several image processing algorithms, including contrast enhancement, image binarization, edge detection, hole filling, and connected domain marking. These algorithms were combined to extract the mask of the manual marker in the ultrasound images. Second, the original ultrasound images and mask images were put into the Mumford-Shah model to implement ultrasound image inpainting by minimizing the following energy functional. The continuous iteration of Euler-Lagrange equation was employed to update the iteration item. If the difference between the two output results was less than the pre-set threshold, the output result could be regarded as the repaired ultrasound image. Experimental results show that the designed image processing method accurately extracted the binary mask, whether they are colourful or binary, number or line in ultrasound images. Meanwhile, the improved Mumford-Shah model effectively implemented the ultrasound image inpainting, with the accuracy of 96.55%. Therefore, the proposed method can be used to remove unnecessary objects in ultrasound images, which is beneficial to improve the accuracy of the artificial intelligence model.

Image inpainting algorithm based on inference attention module and two-stage network

Current image inpainting techniques often yield distorted structures or blurred textures that clash with the surrounding context, particularly when addressing extensive missing regions or highly textured images. These methods often struggle to reconstruct realistic and coherent image structures. To address this challenge, we introduce a two-stage network image restoration approach that leverages an inferential attention mechanism. In the first stage, an edge generation network is employed to produce plausible phantom edge information. Subsequently, an image complementation network is utilized to complete the image restoration process. To enhance the visual realism and restoration accuracy of the generated images, we incorporate an inferential attention mechanism within the image complementation network. This mechanism effectively mitigates inconsistencies in the generated features, leading to the production of more meaningful and effective information. We evaluate our proposed method on benchmark datasets, including Places2 dataset, CelebFaces Attributes dataset, and Paris StreetView dataset. The experimental results can demonstrate that the proposed method achieves better Structural Similarity Index and Peak Signal-to-noise Ratio than others. These metrics indicate superior performance compared to existing image inpainting methods, delivering higher image inpainting accuracy and more realistic image reconstruction effect.

Ancient Painting Inpainting Based on Multi-Layer Feature Enhancement and Frequency Perception

Image inpainting aims to restore the damaged information in images, enhancing their readability and usability. Ancient paintings, as a vital component of traditional art, convey profound cultural and artistic value, yet often suffer from various forms of damage over time. Existing ancient painting inpainting methods are insufficient in extracting deep semantic information, resulting in the loss of high-frequency detail features of the reconstructed image and inconsistency between global and local semantic information. To address these issues, this paper proposes a Generative Adversarial Network (GAN)-based ancient painting inpainting method using multi-layer feature enhancement and frequency perception, named MFGAN. Firstly, we design a Residual Pyramid Encoder (RPE), which fully extracts the deep semantic features of ancient painting images and strengthens the processing of image details by effectively combining the deep feature extraction module and channel attention. Secondly, we propose a Frequency-Aware Mechanism (FAM) to obtain the high-frequency perceptual features by using the frequency attention module, which captures the high-frequency details and texture features of the ancient paintings by increasing the skip connections between the low-frequency and the high-frequency features, and provides more frequency perception information. Thirdly, a Dual Discriminator (DD) is designed to ensure the consistency of semantic information between global and local region images, while reducing the discontinuity and blurring differences at the boundary during image inpainting. Finally, extensive experiments on the proposed ancient painting and Huaniao datasets show that our proposed method outperforms competitive image inpainting methods and exhibits robust generalization capabilities.

Intraoperative detection of parathyroid glands using artificial intelligence: optimizing medical image training with data augmentation methods

BackgroundPostoperative hypoparathyroidism is a major complication of thyroidectomy, occurring when the parathyroid glands are inadvertently damaged during surgery. Although intraoperative images are rarely used to train artificial intelligence (AI) because of its complex nature, AI may be trained to intraoperatively detect parathyroid glands using various augmentation methods. The purpose of this study was to train an effective AI model to detect parathyroid glands during thyroidectomy.MethodsVideo clips of the parathyroid gland were collected during thyroid lobectomy procedures. Confirmed parathyroid images were used to train three types of datasets according to augmentation status: baseline, geometric transformation, and generative adversarial network-based image inpainting. The primary outcome was the average precision of the performance of AI in detecting parathyroid glands.Results152 Fine-needle aspiration-confirmed parathyroid gland images were acquired from 150 patients who underwent unilateral lobectomy. The average precision of the AI model in detecting parathyroid glands based on baseline data was 77%. This performance was enhanced by applying both geometric transformation and image inpainting augmentation methods, with the geometric transformation data augmentation dataset showing a higher average precision (79%) than the image inpainting model (78.6%). When this model was subjected to external validation using a completely different thyroidectomy approach, the image inpainting method was more effective (46%) than both the geometric transformation (37%) and baseline (33%) methods.ConclusionThis AI model was found to be an effective and generalizable tool in the intraoperative identification of parathyroid glands during thyroidectomy, especially when aided by appropriate augmentation methods. Additional studies comparing model performance and surgeon identification, however, are needed to assess the true clinical relevance of this AI model.Graphical abstract

Image inpainting based on GAN-driven structure- and texture-aware learning with application to object removal

In this paper, a novel deep learning-based image inpainting framework consisting of restoring image structure and reconstructing image details from corrupted images is proposed. Most image inpainting methods in the literature aim at restoring image details, outlines, and colors, simultaneously, which may suffer from blurring, deformation, and unreasonable content recovery due to interference among various information. To solve these problems, a two-stage image inpainting deep neural network based on GAN (generative adversarial network) architecture is proposed. The proposed inpainting framework consists of two modules: (1) the first stage, called the structure-aware learning stage, aims at learning a GAN-based structure restoration network, focusing on recovering the low-frequency image component, including colors and outlines of the missing regions of the input corrupted image; and (2) the second stage, called the texture-aware learning stage, aims at learning a GAN-based detail refinement network, focusing on rebuilding the high-frequency image details and texture information. In particular, we also propose to remove details from the training images to better train the structure restoration network to avoid inadequate image structure recovery induced by richer image textures, where the detail reconstruction task is left to the second stage. This strategy achieves to balance the workload between the two stages and the image quality can be progressively enhanced through the two stages. Experimental results have shown that the proposed deep inpainting framework quantitatively and qualitatively achieves state-of-the-art performance on the well-known datasets, including the CelebA, Places2, and ImageNet datasets, compared with existing deep learning-based image inpainting approaches. More specifically, in terms of the two well-known image quality assessment metrics, PSNR (peak signal-to-noise ratio) and SSIM (structural similarity), the improvement percentage of the proposed method, compared with the baseline approach, respectively, ranges from 3.23 % to 11.12 %, and 1.95 % to 13.39 %. The improvements have been shown to stably and significantly outperform the compared state-of-the-art methods in most types of inpainting mask. We also show that the proposed method is applicable to image editing in object removal from a single image.

Hierarchical Vector-Quantized Variational Autoencoder and Vector Credibility Mechanism for High-Quality Image Inpainting

Image inpainting infers the missing areas of a corrupted image according to the information of the undamaged part. Many existing image inpainting methods can generate plausible inpainted results from damaged images with the fast-developed deep-learning technology. However, they still suffer from over-smoothed textures or textural distortion in the cases of complex textural details or large damaged areas. To restore textures at a fine-grained level, we propose an image inpainting method based on a hierarchical VQ-VAE with a vector credibility mechanism. It first trains the hierarchical VQ-VAE with ground truth images to update two codebooks and to obtain two corresponding vector collections containing information on ground truth images. The two vector collections are fed to a decoder to generate the corresponding high-fidelity outputs. An encoder then is trained with the corresponding damaged image. It generates vector collections approximating the ground truth by the help of the prior knowledge provided by the codebooks. After that, the two vector collections pass through the decoder from the hierarchical VQ-VAE to produce the inpainted results. In addition, we apply a vector credibility mechanism to promote vector collections from damaged images and approximate vector collections from ground truth images. To further improve the inpainting result, we apply a refinement network, which uses residual blocks with different dilation rates to acquire both global information and local textural details. Extensive experiments conducted on several datasets demonstrate that our method outperforms the state-of-the-art ones.

A mechanism-based method for image inpainting

An inpainting method based on mechanism learning is proposed from the perspective of inverse problems. The underlying data mechanism, characterized by linear differential equations, is identified from data on the known area and then exploited to infer that on the missing part. Special attention is paid to incorporation of historical or prior information as higher order mechanism. Numerical examples show effectiveness, robustness and flexibility of the method and it performs particularly well over mechanism/scientific data.

Quaternion matrix completion using untrained quaternion convolutional neural network for color image inpainting

The use of quaternions as a novel tool for color image representation has yielded impressive results in color image processing. By considering the color image as a unified entity rather than separate color space components, quaternions can effectively exploit the strong correlation among the RGB channels, leading to enhanced performance. Especially, color image inpainting tasks are highly beneficial from the application of quaternion matrix completion techniques, in recent years. However, existing quaternion matrix completion methods suffer from two major drawbacks. First, it can be difficult to choose a regularizer that captures the common characteristics of natural images, and sometimes the regularizer that is chosen based on empirical evidence may not be the optimal or efficient option. Second, the optimization process of quaternion matrix completion models is quite challenging because of the non-commutativity of quaternion multiplication. To address the two drawbacks of the existing quaternion matrix completion approaches mentioned above, this paper tends to use an untrained quaternion convolutional neural network (QCNN) to directly generate the completed quaternion matrix. This approach replaces the explicit regularization term in the quaternion matrix completion model with an implicit prior that is learned by the QCNN. Extensive quantitative and qualitative evaluations demonstrate the superiority of the proposed method for color image inpainting compared with some existing quaternion-based and tensor-based methods.

Image inpainting method based on AU-GAN

Image inpainting refers to the process of filling in missing regions or removing objects, and has broad application prospects. The rapid development of deep learning has led to new technological breakthroughs in image repair technology, continuously improving the quality of image inpainting. However, when we inpaint large missing regions, the texture and structural features of the image cannot be comprehensively utilized. This leads to blurry images. To solve this problem, we propose an improved dual-stream U-Net algorithm that adds an attention mechanism to the two U-Net networks known as a dual AU-Net network to improve the texture details of the image. In addition, the location code (LC) of damaged regions is added to the network to guide network repair and accelerate the network convergence speed. Least squares GAN (LSGAN) loss is added to the generator’s adversarial network to capture more content details and enhance training stability. The PSNR is 33.93 and the SSIM is 0.931 in the CelebA and Paris datasets. This method has been proven effective when compared to other methods.

DNNAM: Image inpainting algorithm via deep neural networks and attention mechanism

Most image inpainting algorithms have problems such as fuzzy images, texture distortion and semantic inaccuracy, and the image inpainting effect is limited when processing photos with large missing sections and resolution levels. The paper proposes an effective image inpainting algorithm via partial multi-scale channel attention mechanism and deep neural networks to address the above phenomenon that existing image inpainting methods using deep learning modules have insufficient perception and representation capabilities for multi-scale features with high proportion of irregular defects. Initially, we used the Res-U-Net module as a generator. The U-Net-like backbone network topology can achieve the encoding and decoding stages of damaged images. Secondly, the residual network structure was built in the encoder and decoder to improve the ability of the proposed network to extract and display the features of the damaged images. Finally, the partial multi-scale channel attention module was inserted in the skip connection with the decoder to increase the efficiency of using the low-level features of the original images. The experimental results of the research can show that the proposed method outperforms state-of-the-art methods in terms of subjective visual perception and objective evaluation indicators on the CelebA, Places2 and Paris Street View datasets.

Retracted: Ancient Stone Inscription Image Denoising and Inpainting Methods Based on Deep Neural Networks

Retracted: Ancient Stone Inscription Image Denoising and Inpainting Methods Based on Deep Neural Networks

Deformable Dynamic Sampling and Dynamic Predictable Mask Mining for Image Inpainting.

Existing image inpainting methods often produce artifacts that are caused by using vanilla convolution layers as building blocks that treat all image regions equally and generate holes at random locations with equal probability. This design does not differentiate the missing regions and valid regions in inference and does not consider the predictability of missing regions in training. To address these issues, we propose a deformable dynamic sampling (DDS) mechanism which is built on deformable convolutions (DCs), and a constraint is proposed to avoid the deformably sampled elements falling into the corrupted regions. Furthermore, to select both valid sample locations and suitable kernels dynamically, we equip DCs with content-aware dynamic kernel selection (DKS). In addition, to further encourage the DDS mechanism to find meaningful sampling locations, we propose to train the inpainting model with mined predictable regions as holes. During training, we jointly train a mask generator with the inpainting network to generate hole masks dynamically for each training sample. Thus, the mask generator can find large yet predictable missing regions as a better alternative to random masks. Extensive experiments demonstrate the advantages of our method over state-of-the-art methods qualitatively and quantitatively.

Transformer-Based High-Fidelity Facial Displacement Completion for Detailed 3D Face Reconstruction

In this paper, we tackle a special face completion task, facial displacement completion, which can offer a key component for many single image 3D face reconstruction systems. To produce a detailed 3D face with ear-to-ear complete displacement UV map, we propose a novel Displacement Completion method based on Transformer (DCT). Current transformer based image inpainting methods usually follow a two-stage scheme, which firstly recovers the masked pixels in low resolution with transformer, and then replenishes the inpainting result in high resolution with GAN. Although these methods have achieved great success, they suffer from information loss from two aspects when applied in face completion: 1) The downsampling operation makes transformer only produce a coarse appearance prior for GAN, incurring middle and low level information loss. 2) Some meaningful facial semantics can be well captured by transformer and further benefit the completion, but it's has not yet been explored. Motivated by the above considerations, we come up with three key designs in the proposed DCT: PCA tokenization, BERT-style learning, and style modulation. Firstly, we use PCA tokenization to replace the downsampling in transformer to preserve more meaningful structures. Secondly, we make transformer simulate the two tasks in BERT, Masked Language Model (MLM) and Next Sentence Prediction (NSP), for both masked pixels and facial attributes recovery. Thirdly, we encode the outcome of transformer as the latent code to guide an image translation network in the StyleGAN2 modulation way. Experments on both FaceScape dataset and in-the-wild data demonstrate DCT's better performance compared with other transformer based or GAN based completion methods.

Image Inpainting based on Gated Convolution and spectral Normalization

Traditional image inpainting methods based on deep learning have the problem of insufficient discrimination between the missing area and the global area information in the feature extraction of image inpainting tasks because of the characteristics of the model constructed by the convolutional layer. At the same time, the traditional generative adversarial network often has problems such as training difficulties and model collapse in the training process. To solve the above problems and improve the repair effect of the model, this paper proposes a dual discriminator image inpainting model based on generative adversarial network combining gated convolution and spectral normalization. The model is mainly composed of an image inpainting module and an image recognition module. The traditional image inpainting model considers all input pixels as valid pixels when extracting the features of the image to be inpainted, which is unreasonable for the image inpainting task. In order to solve this problem, the gated convolution is designed to replace the role of traditional convolution in the image inpainting module. Gated convolutions address the irrationality of ordinary convolutions in image inpainting tasks by providing learnable dynamic feature selection mechanisms for each channel at each spatial location in all layers. At the same time, generative image inpainting models usually have problems such as training hard mode collapse during the training process. In this paper, we intend to introduce the spectral normalization mechanism in the convolutional layer design of the discriminator module. By introducing Lipschitz continuity constraints from the spectral norm of the parameter matrix of each layer of the neural network, the neural network is better insensitive to input perturbations, making the training process more stable and easier to converge. It solves the problems of mode collapse and model training difficulty in the training of image inpainting model based on generative adversarial network. Finally, qualitative and quantitative experiments show that the image inpainting model based on gated convolution and spectral normalization solves the above problems, and the inpainted image has reasonable texture structure and contextual semantic information.

An image inpainting method based on generative adversarial networks inversion and autoencoder

AbstractImage inpainting aims to repair the damaged region according to the known content in the damaged image. Recently, image inpainting methods have poor effects on high‐resolution damaged images, and the research on the inpainting of large‐area damaged images is limited. Therefore, this paper proposes an image inpainting method based on Generative Adversarial Networks (GAN) inversion and autoencoder. This work consists of two phases: first, the authors design an autoencoder‐based GAN, which learns the mapping from noise to low‐dimensional feature maps by training a generator, and then converts the generated feature maps into high‐resolution images. Thus, the difficulty of learning the mapping relationship is reduced. Second, the authors adopt the learning‐based GAN inversion to infer the closest latent code. The trained GAN is then used to reconstruct the complete image. Finally, the authors compare their method with other classical methods on the CelebAMask‐HQ, Flickr‐Faces‐HQ, and ImageNet datasets. According to the quantitative comparison, when the mask range is large, in other words, when the image has a large area of damage, the authors’ method is superior to the comparison methods. According to the qualitative comparison, the structure of the high‐resolution image inpainted by the authors’ method is more reasonable and the texture details are more realistic.

Structure-aware multi-view image inpainting using dual consistency attention

Image inpainting based on deep learning has made remarkable progress and is widely used in image editing, cultural relic preservation, etc. However, most image inpainting methods are implemented based on single-view images. This does not fully utilize the known information and leads to unsatisfactory inpainting results. Moreover, these methods usually ignore the importance of image consistency and the surrounding regions, leading to irrelevant contents and visual artifacts in the inpainting results. To solve these problems, a structure-aware multi-view image inpainting method using dual consistency attention (SM-DCA) is proposed in this paper. It consists of two parts. The first part is the structure-aware multi-view image inpainting. This part constructs structure views as additional views to assist image inpainting. It is implemented by two networks: a structure inpainting network with strong constraints (SSC) and an image inpainting network with dual consistency attention (IDCA). SSC is used to repair structure views and make them closely resemble the ground truth through strong constraints. IDCA improves the consistency between the generated content and the whole image, making the repaired image more reasonable. The second part is image refinement, implemented by an image local refinement network (ILR). It can focus on the surrounding regions, eliminating boundary artifacts and obtaining finer local details. In Paris StreetView, SM-DCA achieves 22.0194, 0.7457 and 0.0557 in terms of PSNR, SSIM and MAE at 50%–60% damage. The corresponding values in CelebA are 22.5526, 0.8623 and 0.0453, respectively. The extensive experimental results on the Paris StreetView and CelebA datasets demonstrate the superiority of SM-DCA.