Image Inpainting Network Research Articles

Local and global mixture network for image inpainting

In general, CNN-based inpainting can recover local patterns effectively using convolutional filters, but it may not exploit global correlation fully. On the other hand, transformer-based inpainting can fill in large holes faithfully based on global correlation, rather than local one. In this paper, we propose a novel image inpainting algorithm, called local and global mixture (LGM), to take advantage of the strengths of both approaches and compensate for their weaknesses. The LGM network comprises the local inpainting network (LIN) and the global inpainting network (GIN) in parallel, which are based on convolutional layers and transformer blocks, respectively, and exchange their intermediate results with each other. Furthermore, we develop an error propagation model with a continuous error mask, updated in LIN but used in both LIN and GIN to provide more reliable inpainting results. Extensive experiments demonstrate that the proposed LGM algorithm provides excellent inpainting performance, which indicates the efficacy of the parallel combination of LIN and GIN and the effectiveness of the error propagation model.

Self-prior guided generative adversarial network for image inpainting

Self-prior guided generative adversarial network for image inpainting

STNet: Structure and texture-guided network for image inpainting

Generative Adversarial Network (GAN) has made great progress in image inpainting due to its strong generation capability. However, the previous methods based on GAN cannot understand the structure and texture of damaged images simultaneously, which leads to inconsistent structure and blurred details of inpainting results. To address this issue, we propose a Structure and Texture-guided Network for image inpainting (STNet), which consists of three networks, i.e. structure reconstruction network, texture reconstruction network, and image refinement network. STNet can restore the coherent structure and fine textures of damaged images in the first two networks, and fuse them to complete image inpainting in the last network. Experiments on three standard datasets CelebA, Places2, and Pairs StreetView show STNet is effective and outperforms other related methods. In particular, we also conducted experiments on the damaged images from tomb murals of Tang Dynasty in China, whose results also verify that our method can be applied to the restoration of historical relics. Our code is available at https://github.com/nwuAI/STNet.

Partial hard occluded target reconstruction of Fourier single pixel imaging guided through range slice

Fourier single pixel imaging utilizes pre-programmed patterns for laser spatial distribution modulation to reconstruct intensity image of the target through reconstruction algorithms. The approach features non-locality and high anti-interference performance. However, Poor image quality is induced when the target of interest is occluded in Fourier single pixel imaging. To address the problem, a deep learning-based image inpainting algorithm is employed within Fourier single pixel imaging to reconstruct partially obscured targets with high quality. It applies a distance-based segmentation method to segment obscured regions and the target of interest. Additionally, it utilizes an image inpainting network that combines multi-scale sparse convolution and transformer architecture, along with a reconstruction network that integrates Channel Attention Mechanism and Attention Gate modules to reconstruct complete and clear intensity images of the target of interest. The proposed method significantly expands the application scenarios and improves the imaging quality of Fourier single pixel imaging. Simulation and real-world experimental results demonstrate that the proposed method exhibits the high inpainting and reconstruction capacity in the conditions of hard occlusion and down-sampling.

A Novel Dual U-Net Generative Adversarial Network for Image Inpainting

A Novel Dual U-Net Generative Adversarial Network for Image Inpainting

Spatial and Contextual Path Network for Image Inpainting

Spatial and Contextual Path Network for Image Inpainting

SC_LPR: Semantically Consistent LiDAR Place Recognition Based on Chained Cascade Network in Long-Term Dynamic Environments.

In large-scale long-term dynamic environments, high-frequency dynamic objects inevitably lead to significant changes in the appearance of the scene at the same location at different times, which is catastrophic for place recognition (PR). Therefore, how to eliminate the influence of dynamic objects to achieve robust PR has universal practical value for mobile robots and autonomous vehicles. To this end, we suggest a novel semantically consistent LiDAR PR method based on chained cascade network, called SC_LPR, which mainly consists of a LiDAR semantic image inpainting network (LSI-Net) and a semantic pyramid Transformer-based PR network (SPT-Net). Specifically, LSI-Net is a coarse-to-fine generative adversarial network (GAN) with a gated convolutional autoencoder as the backbone. To effectively address the challenges posed by variable-scale dynamic object masks, we integrate the updated Transformer block with mask attention and gated trident block into LSI-Net. Sequentially, in order to generate a discriminative global descriptor representing the point cloud, we design an encoder with pyramid Transformer block to efficiently encode long-range dependencies and global contexts between different categories in the inpainted semantic image, followed by an augmented NetVALD, a generalized VLAD (Vector of Locally Aggregated Descriptors) layer that adaptively aggregates salient local features. Last but not least, we first attempt to create a LiDAR semantic inpainting dataset, called LSI-Dataset, to effectively validate the proposed method. Experimental comparisons show that our method not only improves semantic inpainting performance by about 6%, but also improves PR performance in dynamic environments by about 8% compared to the representative optimal baseline. LSI-Dataset will be publicly available at https://github.KD.LPR.com/.

Deep generative network for image inpainting with gradient semantics and spatial-smooth attention

As a powerful forgery operation in the image content security area, image inpainting based on deep generative networks can yield visually appealing outputs but often produces ambiguous artifacts, especially in boundary and high semantic areas. To address this issue, a novel end-to-end network with gradient semantics and spatial-smooth attention (GS-SSA) is proposed, which combines a gradient learning network and an image inpainting network. The gradient learning network is meant to properly anticipate the gradient semantics in the hole region and get a complete gradient semantic map. The image inpainting network utilizes the complete gradient semantic map to better repair the missing pixels and obtain the final inpainting results. Moreover, spatial-smooth attention is introduced into the image inpainting network, considering both the spatial structural relations and the surrounding information in the hole region. Experimental results on public datasets show the superiority of the proposed method, especially in large-hole region tasks.

Structure-aware multi-view image inpainting using dual consistency attention

Image inpainting based on deep learning has made remarkable progress and is widely used in image editing, cultural relic preservation, etc. However, most image inpainting methods are implemented based on single-view images. This does not fully utilize the known information and leads to unsatisfactory inpainting results. Moreover, these methods usually ignore the importance of image consistency and the surrounding regions, leading to irrelevant contents and visual artifacts in the inpainting results. To solve these problems, a structure-aware multi-view image inpainting method using dual consistency attention (SM-DCA) is proposed in this paper. It consists of two parts. The first part is the structure-aware multi-view image inpainting. This part constructs structure views as additional views to assist image inpainting. It is implemented by two networks: a structure inpainting network with strong constraints (SSC) and an image inpainting network with dual consistency attention (IDCA). SSC is used to repair structure views and make them closely resemble the ground truth through strong constraints. IDCA improves the consistency between the generated content and the whole image, making the repaired image more reasonable. The second part is image refinement, implemented by an image local refinement network (ILR). It can focus on the surrounding regions, eliminating boundary artifacts and obtaining finer local details. In Paris StreetView, SM-DCA achieves 22.0194, 0.7457 and 0.0557 in terms of PSNR, SSIM and MAE at 50%–60% damage. The corresponding values in CelebA are 22.5526, 0.8623 and 0.0453, respectively. The extensive experimental results on the Paris StreetView and CelebA datasets demonstrate the superiority of SM-DCA.

MFMAM: Image inpainting via multi-scale feature module with attention module

Currently, most image inpainting algorithms based on deep learning cause information loss when acquiring deep level features. It is not conducive to the image inpainting of texture details and often ignore the image restoration of semantic features, which generates image inpainting results with unreasonable structures. To address those problems, we have proposed improved image inpainting network using multi-scale feature module and improved attention module. Firstly, we propose the multi-scale fusion module based on dilated convolution to reduce information loss during the convolution process by fusing multi-scale features when acquiring image deep level features. Then, the attention module can strengthen the ability of image semantic inpainting and ensure that the proposed model can generate clear texture inpainting results. To ensure the consistency of image inpainting details and styles, the style loss and perceptual loss functions are introduced in the proposed network. The qualitative experimental results on CelebA-HQ, Places2 and Outdoor Scene datasets and common evaluation metrics such as PSNR, SSIM and FID. These metrics can validate the proposed method to be superior to the state-of-the-arts.

Image inpainting network based on multi‐level attention mechanism

AbstractImage inpainting networks based on deep learning techniques have been widely used in many important fields. However, most inpainting networks fail to generate desirable repaired images. This may be due to their failure to extract effective features and accurately assign high weights to the undamaged regions. To alleviate these problems, an image inpainting network based on gated convolution and multi‐level attention mechanism (IIN‐GCMAM) is proposed in this paper. This network follows encoder–decoder architecture, consisting of the gated convolution encoder (GC‐encoder) and the multi‐level attention mechanism decoder (MAM‐decoder). The GC‐encoder weighs the extracted features with gated convolutions, which reduces the interference caused by the damaged regions. The multi‐level attention mechanism employed in the MAM‐decoder uses multi‐scale feature maps spatially and channel‐wise to improve the consistency in global structure and the fineness of repaired results. Extensive experiments are conducted on the common datasets, Paris StreetView and CelebA. Experimental results indicate that the proposed IIN‐GCMAM can achieve a good performance on the common evaluation metrics and visual effects. It can achieve 0.0408, 0.720, and 22.27 in MAE, SSIM, and PSNR at the mask ratio of 50%–60%, respectively.

Cascading Blend Network for Image Inpainting

Image inpainting refers to filling in unknown regions with known knowledge, which is in full flourish accompanied by the popularity and prosperity of deep convolutional networks. Current inpainting methods have excelled in completing small-sized corruption or specifically masked images. However, for large-proportion corrupted images, most attention-based and structure-based approaches, though reported with state-of-the-art performance, fail to reconstruct high-quality results due to the short consideration of semantic relevance. To relieve the above problem, in this paper, we propose a novel image inpainting approach, namely cascading blend network (CBNet), to strengthen the capacity of feature representation. As a whole, we introduce an adjacent transfer attention (ATA) module in the decoder, which preserves contour structure reasonably from the deep layer and blends structure-texture information from the shadow layer. In a coarse to delicate manner, a multi-scale contextual blend (MCB) block is further designed to felicitously assemble the multi-stage feature information. In addition, to ensure a high qualified hybrid of the feature information, extra deep supervision is applied to the intermediate features through a cascaded loss. Qualitative and quantitative experiments on the Paris StreetView, CelebA, and Places2 datasets demonstrate the superior performance of our approach compared with most state-of-the-art algorithms.

Research of soil surface image occlusion removal and inpainting based on GAN used for estimation of farmland soil moisture content

In smart agriculture, soil images are frequently used to estimate soil moisture content (SMC). The soil image can’t fully reflect the real soil surface condition due to the interference of weeds and other obstructions. To solve the occlusion problem, a new occlusion removal idea is proposed: identifying the occluded object, generating corresponding masks to cover it, and restoring the area. Therefore, it involves three parts: image inpainting network research, occlusion recognition and removal system, and practical application in farmland. Soil Surface Occlusion Image Inpainting Network (SOI NET) was proposed for image inpainting based on Generative Adversarial Networks (GAN). A two-stage generation network was designed: Rough Network and Refinement Network which uses the coherent semantic attention (CSA) module to improve the network's context information utilization capability. A soil surface image occlusion removal system was designed, including vehicle-mounted detection system, occlusion target recognition, mask generation and occlusion removal. Indoor and field models for estimating SMC using image color parameters were established. To verify the performance of SOI NET, experiments were carried out to compare with the traditional method and other deep learning methods. The actual farmland occlusion removal experiment was carried out with weeds as the removal object which are the most common occlusion. The results revealed that the system can effectively remove the occlusion from image surface images. In order to verify the improvement effect of occlusion removal image on the detection accuracy of vehicle-mounted terminals, further farmland experiments were conducted to estimate SMC. R2 increased from 0.637 to 0.667 and Root Mean Square Error (RMSE) decreased from 1.916 % to 1.822 % after occlusion removal. SOI NET and soil occlusion removal system can make the processed images reflect the real soil conditions and provide help for soil image analysis.

Pyramid-VAE-GAN: Transferring hierarchical latent variables for image inpainting

Significant progress has been made in image inpainting methods in recent years. However, they are incapable of producing inpainting results with reasonable structures, rich detail, and sharpness at the same time. In this paper, we propose the Pyramid-VAE-GAN network for image inpainting to address this limitation. Our network is built on a variational autoencoder (VAE) backbone that encodes high-level latent variables to represent complicated high-dimensional prior distributions of images. The prior assists in reconstructing reasonable structures when inpainting. We also adopt a pyramid structure in our model to maintain rich detail in low-level latent variables. To avoid the usual incompatibility of requiring both reasonable structures and rich detail, we propose a novel cross-layer latent variable transfer module. This transfers information about long-range structures contained in high-level latent variables to low-level latent variables representing more detailed information. We further use adversarial training to select the most reasonable results and to improve the sharpness of the images. Extensive experimental results on multiple datasets demonstrate the superiority of our method. Our code is available at https://github.com/thy960112/Pyramid-VAE-GAN.

A Two-Stage Image Inpainting Technique for Old Photographs Based on Transfer Learning

To address the challenge of sparse old photo datasets, we apply transfer learning to image inpainting tasks. Specifically, we improve a two-stage image inpainting network that focuses on collaborative subtasks. We also design a transform module based on the cross-aggregation of windows to improve long-distance contextual information acquisition in image inpainting and enhance the integrity of images in terms of structure and texture. Our improved two-stage network has a significantly better repair performance compared to that of the current common inpainting methods. We further apply transfer learning techniques by utilizing the improved two-stage image inpainting network as the base network and decoupling the generator into a feature extractor and classifier, which consist of an encoder and a decoder, respectively. We obtain a domain-invariant feature extractor through minimax game training using source and target domain data. This feature extractor can be combined with the original encoder to restore old photo images. To verify the effectiveness of our approach, we conducted comparative experiments. Our results show that the PSNR, SSIM, and FID indexes of the model using transfer learning are 11.8%, 2.96%, and 44.4% higher than those without transfer learning, respectively. These findings suggest that applying transfer learning techniques can be an effective solution to address the challenge of sparse old photo datasets in image inpainting tasks.

Semantic prior-driven fused contextual transformation network for image inpainting

Recent advances in image inpainting have achieved impressive performance for generating plausible visual details on small regular image defects or simple backgrounds. However, current solution suffers from the lack of semantic priors for the image and the inability to deduce the image content from distant background, leading to distorted structures and artifacts in the results when inpainting large random irregular complicated images. To address these problems, a semantic prior-driven fused contextual transformation network for image inpainting is proposed as a promise solution. First, the semantic prior generator is put forward to map the semantic features of ground truth images and the low-level features of broken images to semantic priors. Subsequently, an image split-transform-aggregated strategy, named fusion context transformation block, is presented to infer rich multi-scale remote texture features and thus to improve the restored image finesse. Thereafter, an aggregated semantic attention-aware module, consisting of spatially adaptive normalization and enhanced spatial attention is designed to aggregate semantic priors and multi-scale texture features into the decoder to restore reasonable structure. Finally, the mask guided discriminator is developed to effectively discriminate between real and false pixels in the output image to improve the capability of the discriminator and hence to reduce the probability of artifacts containing in the output image. Comprehensive experimental results on CelebA-HQ, Paris Street View, and Places2 datasets demonstrate the superiority of the proposed network over the state-of-the-arts, whose PSNR, SSIM and MAE are improved about 20 %, 12.6 %, and 42 % gains, respectively.

Image Inpainting Based on Structural Constraint and Multi-Scale Feature Fusion

When repairing masked images based on deep learning, there is usually insufficient representation of multi-level information and inadequate utilization of long distance features. To solve the problems, this paper proposes a second-order generative image inpainting model based on Structural Constraints and Multi-scale Feature Fusion (SCMFF). The SCMFF model consists of two parts: edge repair network and image inpainting network. The edge repair network combines the auto-encoder with the Dilated Residual Feature Pyramid Fusion (DRFPF) module, which improves the representation of multi-level semantic information and structural details of images, thus achieves better edge repair. Then, the image inpainting network embeds the Dilated Multi-scale Attention Fusion (DMAF) module in the auto-encoder for texture synthesis with the real edge as the prior condition, and achieves fine-grained inpainting under the edge constraint by aggregating the long-distance features of different dimensions. Finally, the edge repair results are used to replace the real edge, and the two networks are fused and trained to achieve end-to-end repair from the masked image to the complete image. The model is compared with the advanced methods on datasets including Celeba, Facade and Places2. The quantitative results show that the four metrics of LPIPS, MAE, PSNR and SSIM are improved by 0.0124-0.0211, 3.787-6.829, 2.934dB-5.730dB and 0.034-0.132, respectively. The qualitative results show that the edge distribution in the center of the hole reconstructed by the SCMFF model is more uniform, and the texture synthesis effect is more in line with human visual perception.

Context Adaptive Network for Image Inpainting.

In a typical image inpainting task, the location and shape of the damaged or masked area is often random and irregular. The vanilla convolutions widely used in learning-based inpainting models treat all spatial features as valid and share parameters across regions, making it difficult for them to cope with those irregular damages, and models tend to produce inpainting results with color discrepancy and blurriness. In this paper, we propose a novel Context Adaptive Network (CANet) to address this issue. The main idea of the proposed CANet is able to generate different weights depending on the miscellaneous input, which may help to complement images with multiple broken forms in a flexible way. Specifically, the proposed CANet has two novel context adaptive modules, namely, the context adaptive block (CAB) and the cross-scale contextual attention (CSCA), which utilize attention mechanisms to cope with diverse content breakdowns. The proposed CAB, during the forward propagation, uses an adaptive term to determine the importance between adaptive term and convolution kernel, so as to dynamically balance features based on the degree of breakage (confidence level or soft mask), and the overall calculation is formulated as a classic convolution implementation with an additional attention term to describe local structure. Besides, the proposed CSCA, not only takes advantage of the contextual attention module, but also considers cross-scale information transfer to generate reasonable features for damaged areas, thus alleviating the inefficiency of the long-range modeling capability of convolutional neural networks. Qualitative and quantitative experiments show that our method performs better than state-of-the-arts, producing clearer, more coherent and visually plausible inpainting results. The code can be found at github.com/dengyecode/CANet_image_inpainting.

HDA-GAN：hybrid dual attention generative adversarial network for image inpainting

HDA-GAN：hybrid dual attention generative adversarial network for image inpainting

Contour–Context Joint Blind Image Inpainting Network for Molecular Sieve Particle Size Measurement of SEM Images

Scanning electron microscope (SEM) images generated in the experiment of high-throughput molecular sieves catalysts contain particle size information of various molecular sieves. However, a certain proportion of molecular sieves in SEM images are occluded due to their characteristics of cluster distribution, which cause difficulties to measure the particle size of these occluded molecular sieves. In this paper, we leverage the image inpainting technique to reconstruct missing regions, and then calculate the particle size. Aiming at the problem that the generated regions of recent image inpainting methods exist blurriness in contour and context, we propose a contour-context joint blind image inpainting network, which is specifically a two-stage model with full utilization of edge information to reconstruct the contour and context of these occluded molecular sieve images. In the first stage, the edge of occluded region is reconstructed. In the second stage, the reconstructed edge is introduced through down sampling and skip connection to generate images with complete contour. Meanwhile, a novel encoder loss is proposed to motivate the encoder to extract more reasonable features. Compared with the existing methods, our model achieved more integral results visually and higher quantitative metrics. Thus, our approach can accurately predict the particle size from reconstructed molecular sieve images.