Image Inpainting Methods Research Articles

3D Measurement Method for Saturated Highlight Characteristics on Surface of Fuel Nozzle.

Saturated highlights on metal surfaces reduce the detection accuracy of fuel nozzles. In this paper, we propose an image inpainting method with a saturated highlight based on the statistics of similar patches used in prior segmentation of the subregion. The sequence image acquisition is based on the shape from focus in the five-axis platform. By our method, the focus measure (FM) operator and the window size are evaluated using the sharpness evaluation curve and calculating time. We observe that the detection accuracy is improved when the highlight area is filled by the highlight-free area within the same segmentation region. There are fewer deviation points in the three-dimensional (3D) point cloud that are extracted from the sequence images. The inlet circle and the outlet circle of the fuel nozzle are both detected by the two-dimensional (2D) Hough Transform (HT) method. Our experiments show that the method yields better results in 3D detection of the key parameters of fuel nozzles with the saturated highlight characteristics.

A Cascade Defense Method for Multidomain Adversarial Attacks under Remote Sensing Detection

Deep neural networks have been widely used in detection tasks based on optical remote sensing images. However, in recent studies, deep neural networks have been shown to be vulnerable to adversarial examples. Adversarial examples are threatening in both the digital and physical domains. Specifically, they make it possible for adversarial examples to attack aerial remote sensing detection. To defend against adversarial attacks on aerial remote sensing detection, we propose a cascaded adversarial defense framework, which locates the adversarial patch according to its high frequency and saliency information in the gradient domain and removes it directly. The original image semantic and texture information is then restored by the image inpainting method. When combined with the random erasing algorithm, the robustness of detection is further improved. Our method is the first attempt to defend against adversarial examples in remote sensing detection. The experimental results show that our method is very effective in defending against real-world adversarial attacks. In particular, when using the YOLOv3 and YOLOv4 algorithms for robust detection of single-class targets, the AP60 of YOLOv3 and YOLOv4 only drop by 2.11% and 2.17%, respectively, under the adversarial example.

A novel stereo image self-inpainting network for autonomous robots

Recently, vision-based methods have exhibited promising prospects in assigning autonomous robots more capabilities for better environment perception. Since visual sensors are easily affected under extreme conditions, current image inpainting methods based on CNNs jointly with generative adversarial networks (GANs) usually generate patches quite different from the ground truth (GT), which is harmful to autonomous robots. In this paper, we propose a novel multiscale feature alignment module with an early fusion strategy to align the left and right feature maps to better capture the motion cues between them. Then, the aligned features are fused to fill holes in the left image. To aggregate the multiscale feature maps dynamically, we propose a multiscale feature aggregation module based on an attention mechanism, of which the fusion module is designed as a symmetrical architecture to adaptively incorporate the complementary contextual correlations from different feature branches. In addition, a spatial attention module able to capture the correlations among pixels is introduced into our network to enhance the inpainting capacity and generate more refined details. To evaluate the effectiveness of our proposed method, many experiments are conducted on a stereo image dataset. The quantitative and qualitative results show that our method significantly outperforms the recent state-of-the-art image inpainting methods while running over 22 fps on a single NVIDIA RTX2080Ti GPU.

An Image Inpainting Method Based on Whale-Integrated Monarch Butterfly Optimization-Based DCNN

This paper proposes an image inpainting method based on Whale integrated Monarch Butterfly Optimization-based Deep Convolutional Neural network (Whale-MBO-DCNN) model. Initially, the patch extraction and mapping are applied to the input image to extract the patches of the image followed by image reconstruction in order to map the patches. The patch with minimum distance is selected using the concept of Bhattacharya distance in patch extraction. On the other hand, the construction of the residual image form the input image is done using Deep CNN, which is trained with the proposed Whale-MBO algorithm. The proposed Whale-MBO algorithm is developed from the integration of Monarch Butterfly Optimization (MBO) and (WOA. Finally, the residual image and the reconstructed image are fused using Holoentropy to obtain the reconstructed image. The experimentation is performed using the evaluation metrics, such as PSNR, SDME, and SSIM. The effectiveness of the proposed image inpainting method is revealed through a higher PSNR, SDME, and SSIM of 33.0585, 74.4249, and 0.9479, respectively.

Brain MR Atlas Construction Using Symmetric Deep Neural Inpainting.

Modeling statistical properties of anatomical structures using magnetic resonance imaging is essential for revealing common information of a target population and unique properties of specific subjects. In brain imaging, a statistical brain atlas is often constructed using a number of healthy subjects. When tumors are present, however, it is difficult to either provide a common space for various subjects or align their imaging data due to the unpredictable distribution of lesions. Here we propose a deep learning-based image inpainting method to replace the tumor regions with normal tissue intensities using only a patient population. Our framework has three major innovations: 1) incompletely distributed datasets with random tumor locations can be used for training; 2) irregularly-shaped tumor regions are properly learned, identified, and corrected; and 3) a symmetry constraint between the two brain hemispheres is applied to regularize inpainted regions. Henceforth, regular atlas construction and image registration methods can be applied using inpainted data to obtain tissue deformation, thereby achieving group-specific statistical atlases and patient-to-atlas registration. Our framework was tested using the public database from the Multimodal Brain Tumor Segmentation challenge. Results showed increased similarity scores as well as reduced reconstruction errors compared with three existing image inpainting methods. Patient-to-atlas registration also yielded better results with improved normalized cross-correlation and mutual information and a reduced amount of deformation over the tumor regions.

Reconstruction of Photospheric Velocity Fields from Highly Corrupted Data

The analysis of the photospheric velocity field is essential for understanding plasma turbulence in the solar surface, which may be responsible for driving processes such as magnetic reconnection, flares, wave propagation, particle acceleration, and coronal heating. Currently, the only available methods to estimate velocities at the solar photosphere transverse to an observer’s line of sight infer flows from differences in image structure in successive observations. Due to data noise, algorithms such as local correlation tracking may lead to a vector field with wide gaps where no velocity vectors are provided. In this paper, a novel method for image inpainting of highly corrupted data is proposed and applied to the restoration of horizontal velocity fields in the solar photosphere. The restored velocity field preserves all the vector field components present in the original field. The method shows robustness when applied to both simulated and observational data.

Face Image Completion Based on GAN Prior

Face images are often used in social and entertainment activities to interact with information. However, during the transmission of digital images, there are factors that may destroy or obscure the key elements of the image, which may hinder the understanding of the image’s content. Therefore, the study of image completion of human faces has become an important research branch in the field of computer image processing. Compared with traditional image inpainting methods, deep-learning-based inpainting methods have significantly improved the results on face images, but in the case of complex semantic information and large missing areas, the completion results are still blurred, and the color of the boundary is inconsistent and does not match human visual perception. To solve this problem, this paper proposes a face completion method based on GAN priori to guide the network to complete face images by directly using the rich and diverse a priori information in the pre-trained GAN. The network model is a coarse-to-fine structure, where the damaged face images and the corresponding masks are first input to the coarse network to obtain the coarse results, and then the coarse results are input to the fine network with multi-resolution skip connections. The fine network uses the a priori information from the pre-trained GAN to guide the network to generate the face images, and finally uses the SN-PatchGAN discriminator to evaluate the completion results. The experiment is performed on the CelebA-HQ dataset. Compared with the latest three completion methods, the qualitative and quantitative experimental analysis shows that our method has obvious improvement in texture and fidelity.

Face Anonymity Based on Facial Pose Consistency

With the development of artificial intelligence, there are more and more applications related to face images. The recording of face information causes potential cyber security risks and personal privacy disclosure risks to the public. To solve this problem, we hope to protect face privacy through face anonymity. This paper designs a conditional autoencoder that uses the data preprocessing method of image inpainting. Based on the realistic generation ability of StyleGAN, our autoencoder model introduces facial pose information as conditional information. The input image only contains pre-processed face-removed images. Our method can generate high-resolution images and maintain the posture of the original face. It can be used for identity-independent computer vision tasks. Experiments further proves the effectiveness of our anonymization framework.

Image Inpainting via Context Discriminator and U-Net

Image inpainting is one of the research hotspots in the field of computer vision and image processing. The image inpainting methods based on deep learning models had made some achievements, but it is difficult to achieve ideal results when dealing with images with the relationship between global and local attributions. In particular, when repairing a large area of image defects, the semantic rationality, structural coherence, and detail accuracy of results need to be improved. In view of the existing shortcomings, this study has proposed an improved image inpainting model based on a fully convolutional neural network and generative countermeasure network. A novel image inpainting algorithm via network has been proposed as a generator to repair the defect image, and structural similarity had introduced as the reconstruction loss of image inpainting to supervise and guide model learning from the perspective of the human visual system to improve the effect of image inpainting. The improved global and local context discriminator networks had used as context discriminators to judge the authenticity of the repair results. At the same time, combined with the adversarial loss, a joint loss has proposed for the training of the supervision model, which makes the content of the real and natural repair area and has attribute consistency with the whole image. To verify the effectiveness of the proposed image inpainting model, the image inpainting effect is compared with the current mainstream image inpainting algorithm on the CelebA-HQ dataset based on subjective and objective indicators. The experimental results show that the proposed method has made progress in semantic rationality, structural coherence, and detail accuracy. The proposed model has a better understanding of the high-level semantics of image, and a more accurate grasp of context and detailed information.

Image Inpainting Based on Contextual Coherent Attention GAN

In order to address the problems of traditional inpainting algorithm models, such as the inability to automatically identify the specific location of the area to be restored, the cost of inpainting and the difficulty of inpainting, and the problems of structural and texture discontinuity and poor model stability in deep learning-based image inpainting, this paper proposes an image inpainting based on a contextual coherent attention. This paper designs a network model based on generative adversarial networks. First, to improve the global semantic continuity and local semantic continuity of images in image inpainting, a contextual coherent attention layer is added to the network; second, to solve the problems of slow convergence and insufficient training stability of the model, a cross-entropy loss function is used; finally, the trained generator is used to repair images. The experimental results are compared using PSNR and SSIM metrics, compared with the traditional GAN model, our model has a 3.782dB improvement in peak signal-to-noise ratio and a 0.025% improvement in structural similarity. The experimental results show that the image inpainting method in this paper has better performance in terms of image edge processing, pixel continuity and overall image structure.

Blind Image Inpainting Using Low-Dimensional Manifold Regularization

In this paper, we present a novel method for blind image inpainting, which can restore images with missing or corrupted pixels, or images where the location of the damaged pixels is unknown. The method applies weighted nonlocal Laplacian to address the problem of blind image inpainting using low-dimensional manifold model (LDMM) regularization, and uses semi-local blocks instead of point integrals to implement constraints in LDMM. This solves the problem of low solution efficiency caused by the asymmetry of the linear equations solved by point integration, and the problem of the high iteration count to get good restoration effect. Experiments show that our method is competitive with latest methods in terms of both repairing images with large missing pixels rate and inpainting speed.

Semantic Image Inpainting with Multi-Stage Feature Reasoning Generative Adversarial Network

Most existing image inpainting methods have achieved remarkable progress in small image defects. However, repairing large missing regions with insufficient context information is still an intractable problem. In this paper, a Multi-stage Feature Reasoning Generative Adversarial Network to gradually restore irregular holes is proposed. Specifically, dynamic partial convolution is used to adaptively adjust the restoration proportion during inpainting progress, which strengthens the correlation between valid and invalid pixels. In the decoding phase, the statistical natures of features in the masked areas differentiate from those of unmasked areas. To this end, a novel decoder is designed which not only dynamically assigns a scaling factor and bias on per feature point basis using point-wise normalization, but also utilizes skip connections to solve the problem of information loss between the codec network layers. Moreover, in order to eliminate gradient vanishing and increase the reasoning times, a hybrid weighted merging method consisting of a hard weight map and a soft weight map is proposed to ensemble the feature maps generated during the whole reconstruction process. Experiments on CelebA, Places2, and Paris StreetView show that the proposed model generates results with a PSNR improvement of 0.3 dB to 1.2 dB compared to other methods.

An improved Criminisi method for image inpainting

The Criminisi method is a classical image inpainting method, which has flaw of unreasonable priority calculation. To overcome this flaw, we propose an improved Criminisi method. To improve the accuracy of a repaired block’s priority, we propose a novel method by considering the continuity of structure and color information between the repaired block and its adjacent blocks. When searching the matching block for a repaired block, we compute the similar between two blocks as the sum of squared differences (SSD). Since different blocks within an image may contain different levels of information, we set block to different sizes to determine the best matching block. Experimental results demonstrate the effectiveness of the proposed method.

IID-Net: Image Inpainting Detection Network via Neural Architecture Search and Attention

Deep learning (DL) has demonstrated its powerful capabilities in the field of image inpainting, which could produce visually plausible results. Meanwhile, the malicious use of advanced image inpainting tools (e.g. removing key objects to report fake news, erasing visible copyright watermarks, etc.) has led to increasing threats to the reliability of image data. To fight against the inpainting forgeries (not only DL-based but also traditional ones), in this work, we propose a novel end-to-end Image Inpainting Detection Network (IID-Net), to detect the inpainted regions at pixel accuracy. The proposed IID-Net consists of three sub-blocks: the enhancement block, the extraction block and the decision block. Specifically, the enhancement block aims to enhance the inpainting traces by using hierarchically combined special layers. The extraction block, automatically designed by Neural Architecture Search (NAS) algorithm, is targeted to extract features for the actual inpainting detection tasks. To further optimize the extracted latent features, we integrate global and local attention modules in the decision block, where the global attention reduces the intra-class differences by measuring the similarity of global features, while the local attention strengthens the consistency of local features. Furthermore, we thoroughly study the generalizability of our IID-Net, and find that different training data could result in vastly different generalization capability. By carefully examining 10 popular inpainting methods, we identify that the IID-Net trained on only one specific deep inpainting method exhibits desirable generalizability; namely, the obtained IID-Net can accurately detect and localize inpainting manipulations for various unseen inpainting methods as well. Extensive experimental results are presented to validate the superiority of the proposed IID-Net, compared with the state-of-the-art competitors. Our results would suggest that common artifacts are shared across diverse image inpainting methods. Finally, we build a public inpainting dataset of 10K image pairs for future research in this area.

Semantic Residual Pyramid Network for Image Inpainting

Existing image inpainting methods based on deep learning have made great progress. These methods either generate contextually semantically consistent images or visually excellent images, ignoring that both semantic and visual effects should be appreciated. In this article, we propose a Semantic Residual Pyramid Network (SRPNet) based on a deep generative model for image inpainting at the image and feature levels. This method encodes a masked image by a residual semantic pyramid encoder and then decodes the encoded features into a inpainted image by a multi-layer decoder. At this stage, a multi-layer attention transfer network is used to gradually fill in the missing regions of the image. To generate semantically consistent and visually superior images, the multi-scale discriminators are added to the network structure. The discriminators are divided into global and local discriminators, where the global discriminator is used to identify the global consistency of the inpainted image, and the local discriminator is used to determine the consistency of the missing regions of the inpainted image. Finally, we conducted experiments on four different datasets. As a result, great performance was achieved for filling both the regular and irregular missing regions.

Structural Knowledge-Guided Feature Inference Network for Image Inpainting

Image inpainting is an essential task in image restoration field. Currently, most meth- ods for image inpainting employ the encoder- decoder framework to restore degraded areas, and this often results in synthesizing wrong se- mantic structure due to the lack of guiding from effective prior information. In this paper, we pro- pose a structural knowledge-guided framework for image inpainting, which predicts both the edge map and corrupted content at the same time. Our model captures structural knowledge in the structure estimation branch to guide the content inference in the latent feature space. By employing self-attention mechanism to aggre- gate known information and inferred structural knowledge, our model is able to synthesize more semantically reasonable content for the corrupted areas. Extensive experiments on three bench- mark datasets demonstrate that our method out- performs most state-of-the-art methods for image inpainting in terms of the evaluation of both vi- sual quality and quantitative metrics.

Collaborative Contrastive Learning-Based Generative Model for Image Inpainting

The critical challenge of image inpainting is to infer reasonable semantics and textures for a corrupted image. Typical methods for image inpainting are built upon some prior knowledge to synthesize the complete image. One potential limitation is that those methods often remain undesired blurriness or semantic mistakes in the synthesized image while handling images with large corrupted areas. In this paper, we propose a Collaborative Contrastive Learning-based Generative Model ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C2LGM</i> ), which learns the content consistency in the same image to ensure that the inferred content of corrupted areas is reasonable compared to the known content by pixel-level reconstruction and high-level semantic reasoning. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C2LGM</i> leverages the encoder-decoder based framework to directly learn the mapping from the corrupted image to the intact image and perform the pixel-level reconstruction. To perform semantic reasoning, our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C2LGM</i> introduces a Collaborative Contrastive Learning ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C2L</i> ) mechanism that learns high-level semantic consistency between inferred and known content. Specifically, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C2L</i> mechanism introduces the high-frequency edge maps to participate in the process of typical contrastive learning and enables the deep model to ensure the semantic reasonableness between high-frequency structures and pixel-level content by pushing the representations of inferred content and known content close and keeping unrelated semantic content away in the latent feature space. Moreover, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C2LGM</i> also directly absorbs the prior knowledge of structural information from the proposed structural spatial attention module, and leverages the texture distribution sampling to improve the quality of synthesized content. As a result, our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C2LGM</i> achieves a 0.42 dB improvement over competing methods in terms of the PSNR metric while coping with a 40~ 50% corruption ratio in the Places2 dataset. Extensive experiments on three benchmark datasets, including Paris Street View, CelebA-HQ, and Places2, demonstrate the advantages of our proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C2LGM</i> over other state-of-the-art methods for image inpainting both qualitatively and quantitatively.

FASNet: Feature Aggregation and Sharing Network for Image Inpainting

Image inpainting is a reconstruction method, where a corrupted image consisting of holes is filled with the most relevant contents from the valid region of an image. To inpaint an image, we have proposed a lightweight cascaded architecture with <i>2.5M parameters</i> consisting of encoder feature aggregation block (FAB) with decoder feature sharing (DFS) inpainting network followed by a refinement network. Initially, the FAB with DFS (inpainting) generator network is proposed which comprises of multi-level feature aggregation mechanism and feature sharing decoder. The FAB makes use of multi-scale spatial channel-wise attention to fuse weighted features from all the encoder levels. The DFS reconstructs the inpainted image with multi-scale and multi-receptive feature sharing in order to inpaint the image with smaller to larger hole regions effectively. Further, the refinement generator network is proposed for refining the inpainted image from the inpainting generator network. The effectiveness of proposed architecture is verified on CelebA-HQ [1], [2], Paris Street View (PARIS_SV) [3] and Places2 [4] datasets corrupted using publicly available NVIDIA mask dataset [5]. Extensive result analysis with detailed ablation study prove the robustness of the proposed architecture over state-of-the-art methods for image inpainting.

Multi-Reception and Multi-Gradient Discriminator for Image Inpainting

Many deep learning methods for image inpainting rely on the encoder-decoder architecture to estimate missing contents. When guidance information from uncorrupted regions could not be adequately represented or utilized, the encoder may have difficulty handling the rich surrounding or background pixels, and the decoder could not recover visually sophisticated or realistic contents. This paper proposes an effective multi-scale optimization network to alleviate these issues and generate coherent results with fine details. It encodes multi-receptive fields feature maps adaptively and puts multi-scale outputs into a discriminator to guide training. Specifically, we propose a Multi-Receptive feature maps & masks Selective Fusion (MRSF) operator that can adaptively extract features at different receptive fields to handle sophisticated destroyed images. Then a Multi-Gradients Discriminator (MGD) module uses intermediate features of the discriminator to guide the generator to produce results with natural textures and semantically real contents. Experiments on several benchmark datasets demonstrate that our proposed method can synthesize more realistic and coherent image contents.

Cloud-Free Sea-Surface-Temperature Image Reconstruction From Anomaly Inpainting Network

Sea-surface temperature (SST) images obtained by satellites contain noise and missing SSTs due to cloud covers. We propose a method for reconstructing denoised cloud-free SST images via deep-learning-based image inpainting. For denoizing, we use data-assimilation images to train a reconstruction network by considering the physical correctness of SSTs. For reconstruction stability, we introduce <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">anomaly inpainting network</i> , which does not directly complete missing SSTs but estimates the difference between the unobserved SSTs and the average SSTs. SSTs do not fluctuate much over a few days; thus, we can use recent average SSTs as a rough estimation of SSTs and can assume that the SST difference will be within a specific range. We conducted experiments to evaluate our method with satellite SST images and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in situ</i> SST data. The results indicate that our method with anomaly inpainting network qualitatively and quantitatively outperformed conventional SST image inpainting methods.