Image Inpainting Network Research Articles

EBAT: Enhanced Bidirectional and Autoregressive Transformers for Removing Hairs in Hairy Dermoscopic Images

A great progress in deep learning technologies for skin cancer detection from dermoscopic images has been made for a decade. While its performance is vulnerable to a large amount of hairs densely covering the skin surface, the existing image processing methods frequently fail to remove hairs in hairy skin images. In this paper, we propose, as a deep learning approach to removing hairs, a generative image inpainting network where bidirectional autoregressive transformers (BATs) are employed to learn image features and are systematically integrated with convolutional neural networks (CNNs) in multiple spatial scales in order to reconstruct missing regions. Each patch split from a masked image is unfolded and processed through BATs, and re-folded to constitute diverse shapes of feature maps through kernel-based unfolding-folding operations. By introducing the multi-scale features extracted by collaborative learning of transformers and CNNs to the texture generator network, our method can effectively reconstruct minute details of local regions as well as global structure which might not be easily inferred from neighbor pixels in hairy skin images. Quantitative and qualitative evaluations show not only that our multi-scale dual-modality strategy is much robust to reconstruct hair-shaped missing regions compared to the existing transformer-based image inpainting method called BAT-Fill, but also that our framework outperforms the state-of-the-art image inpainting models in removing hairs from hairy dermoscopic images.

Multichannel Cross-Scale Semantic Coherent Attention Network for Image Inpainting

This paper investigates a cross-scale space semantic feature coherent image inpainting approach since it is challenging for the existing image inpainting methods to fuse the semantic feature information effectively. Firstly, we learn the feature semantic relevance step-by-step from the high-level semantic feature map’s attention mechanism and then we apply what we have learned to the preceding low-level feature map. In order to preserve the visual and semantic coherence of image repair, the missing content can be filled by changing attention from deep to shallow in a multiscale manner. A broader receptive field is generated by partial convolution, and semantic feature relevance is achieved using a multiscale cross feature space feature attention mechanism based on semantic attention. This technique improves the extensibility and continuity of the restored images by reconstructing the semantic information of different feature spaces, not only taking into account the reuse of existing semantic space features but also including across feature spaces. The experimental results demonstrated an improvement in PSNR, SSIM, and L1 performance by 10.50%, 0.13%, and 47.09%, respectively, with clear benefits.

Image Inpainting with Bilateral Convolution

Due to sensor malfunctions and poor atmospheric conditions, remote sensing images often miss important information/pixels, which affects downstream tasks, therefore requiring reconstruction. Current image reconstruction methods use deep convolutional neural networks to improve inpainting performances as they have a powerful modeling capability. However, deep convolutional networks learn different features with the same group of convolutional kernels, which restricts their ability to handle diverse image corruptions and often results in color discrepancy and blurriness in the recovered images. To mitigate this problem, in this paper, we propose an operator called Bilateral Convolution (BC) to adaptively preserve and propagate information from known regions to missing data regions. On the basis of vanilla convolution, the BC dynamically propagates more confident features, which weights the input features of a patch according to their spatial location and feature value. Furthermore, to capture different range dependencies, we designed a Multi-range Window Attention (MWA) module, in which the input feature is divided into multiple sizes of non-overlapped patches for several heads, and then these feature patches are processed by the window self-attention. With BC and MWA, we designed a bilateral convolution network for image inpainting. We conducted experiments on remote sensing datasets and several typical image inpainting datasets to verify the effectiveness and generalization of our network. The results show that our network adaptively captures features between known and unknown regions, generates appropriate content for various corrupted images, and has a competitive performance compared with state-of-the-art methods.

Geometric Constraint and Image Inpainting-Based Railway Track Fastener Sample Generation for Improving Defect Inspection

Defective fastener images detection is an essential task in the vision-based railway track safety inspection. Although existing methods have achieved some level of success, the detection accuracy in this field suffers from the defective fasteners being far less common than normal fasteners. One way to tackle this problem is to expand the defect sample. However, current state-of-the-art defective fastener generation methods mainly rely on generative adversarial networks or simply augment the defect data through traditional image processing. These methods may not be ideal as it is difficult to produce images with high quality and rich diversity at the same time. This paper proposes a new method for fastener sample generation that actively divides the sample generation into two independent parts: defective foregrounds generation and complete backgrounds generation. The key to this method is to generate foregrounds and backgrounds based on geometric constraint and image inpainting, respectively. Specifically, we adopt a skeleton mapping algorithm to directionally control the generated types of defective foregrounds. Meanwhile, an image inpainting network is employed to expand the background. The experiments show that this enables us to generate better-quality and richer-diversity images by combining deep learning and image processing advantages. To the best of our knowledge, our method is the first to achieve state-of-the-art performance, i.e., the classification accuracy reaches 97.97%, without using real defective fastener images during the defect classification network training process.

Deep Matrix Factorization Based on Convolutional Neural Networks for Image Inpainting

In this work, we formulate the image in-painting as a matrix completion problem. Traditional matrix completion methods are generally based on linear models, assuming that the matrix is low rank. When the original matrix is large scale and the observed elements are few, they will easily lead to over-fitting and their performance will also decrease significantly. Recently, researchers have tried to apply deep learning and nonlinear techniques to solve matrix completion. However, most of the existing deep learning-based methods restore each column or row of the matrix independently, which loses the global structure information of the matrix and therefore does not achieve the expected results in the image in-painting. In this paper, we propose a deep matrix factorization completion network (DMFCNet) for image in-painting by combining deep learning and a traditional matrix completion model. The main idea of DMFCNet is to map iterative updates of variables from a traditional matrix completion model into a fixed depth neural network. The potential relationships between observed matrix data are learned in a trainable end-to-end manner, which leads to a high-performance and easy-to-deploy nonlinear solution. Experimental results show that DMFCNet can provide higher matrix completion accuracy than the state-of-the-art matrix completion methods in a shorter running time.

A Generative Image Inpainting Model Based on Edge and Feature Self-Arrangement Constraints.

At present, the image inpainting method based on deep learning has achieved a better inpainting effect than traditional methods, but the inpainting results still have problems such as local structure disorder and blurred texture when the images involving a large defect area are processed. This paper proposes a second-order generative image inpainting model based on edge and feature self-arrangement constraints. The model consists of two parts: edge repair network and image repair network. Based on the self-encoder, the edge repair network generates the edges in the defect area according to the known information of the image and improves the edge repair effect by minimizing the adversarial loss and feature matching loss. The image inpainting network fills the defect area with the edge repair result as a priori condition. On the basis of U-Net, the feature self-arrangement module (FSM) is proposed to reconstruct the coding features of a specific scale, and the reconstructed feature skips to connect the decoding layer of the same scale, and it is fused with the upper layer underlying features for decoding. Meanwhile, guide loss, adversarial loss, and reconstruction loss are introduced to narrow the difference between the repaired image and the original image. The experimental results show that the inpainting results of the proposed model have stronger structural connectivity and clearer textures and the performance of PSNR, SSIM, and mean L1 loss in the Celeba, Facade, and Places2 is better than other inpainting methods, indicating that the algorithm can produce an inpainting effect with highly connected structure, reasonable semantics, and fine details.

Visual Loop Closure Detection With Instance Segmentation and Image Inpainting in Dynamic Scenes Using Wearable Camera

Dynamic interference is an important factor that affects the accuracy of loop closure detection in visual SLAM. Traditional methods that remove the feature points in dynamic regions are prone to insufficient image features, which will decrease the reliability of loop closure detection. To overcome this problem, a novel loop closure detection approach based on image inpainting and feature selection is proposed. A dynamic object instance segmentation guided image inpainting network is utilized to segment and repair dynamic regions in scene images. To avoid the negative influence of image inpainting, an image quality evaluation network is used to confine the extraction of Superpoint features only in the properly inpainted patches. Therefore, valid Superpoint features only in the areas with high inpainting qualities are selected as the input of the Bag of Words model for loop-closure detection. We tested the proposed method on both the KITTI dataset and a wearable camera system. Experimental results indicate that the proposed method improves the accuracy of loop closure detection in dynamic scenes.

Image Inpainting Using Multi-Scale Feature Joint Attention Model

Nowadays, image inpainting methods based on deep learning would lead to information loss when acquiring deep features, which is not conducive to the restoration of texture details and ignores the inpainting of semantic features. Besides, great majority of them generate inpainting results with unreasonable structures. In response to the above problems, an image inpainting network based on a multi-scale feature joint attention model is proposed. First of all, an image inpainting network using multi-scale feature joint attention model is advanced. When acquiring image depth features, multi-scale feature fusion is used to reduce the loss of information in the convolution process. Afterwards, a joint attention mechanism not only strengthens the model’s ability to repair image semantics, but also ensures the model could generate inpainting results with distinct texture. Last but not least, the style loss and perceptual loss are introduced into the network for the purpose of ensuring the consistency of the detail and style of the inpainting results. The qualitative experimental results on the CelebA-HQ and Places2 datasets and commonly used evaluation indicators such as PSNR and SSIM verify the method is superior to the existing image inpainting methods. Compared with the comparison methods, the proposed method improves PSNR and SSIM by 0.4%−6% and 0.4%−3% respectively.

Research into an Image Inpainting Algorithm via Multilevel Attention Progression Mechanism

Existing image inpainting schemes generally have the problems of structural disorder and blurred texture details. This is mainly because, in the reconstruction process of the damaged area of the image, it is difficult for the inpainting network to make full use of the information in the nondamaged area to accurately infer the content of the damaged area. Therefore, the paper has proposed an image inpainting network driven by multilevel attention progression mechanism. The proposed network has compressed the high-level features extracted from the full-resolution image into multiscale compact features and then drives the compact features to perform multilevel order according to the scale size. Attention feature progression is to achieve the goal of the full progression of high-level features including structure and details in the network. To further realize fine-grained image inpainting and reconstruction, the paper has also proposed a composite granular discriminator to achieve image inpainting process performing global semantic constraints and nonspecific local dense constraints. The related experimental results in the paper can show that the proposed method can achieve higher quality repair results than state-of-the-art ones.

Parallel adaptive guidance network for image inpainting

Motivated by human behavior, dividing inpainting tasks into structure reconstruction and texture generation helps to simplify restoration process and avoid distorted structures and blurry textures. However, most of tasks are ineffective for dealing with large continuous holes. In this paper, we devise a parallel adaptive guidance network(PAGN), which repairs structures and enriches textures through parallel branches, and several intermediate-level representations in different branches guide each other via the vertical skip connection and the guidance filter, ensuring that each branch only leverages the desirable features of another and outputs high-quality contents. Considering that the larger the missing regions are, less information is available. We promote the joint-contextual attention mechanism(Joint-CAM), which explores the connection between unknown and known patches by measuring their similarity at the same scale and at different scales, to utilize the existing messages fully. Since strong feature representation is essential for generating visually realistic and semantically reasonable contents in the missing regions, we further design attention-based multiscale perceptual res2blcok(AMPR) in the bottleneck that extracts features of various sizes at granular levels and obtains relatively precise object locations. Experiments on the public datasets CelebA-HQ, Places2, and Paris show that our proposed model is superior to state-of-the-art models, especially for filling large holes.

A multi-level feature integration network for image inpainting

A multi-level feature integration network for image inpainting

Semantic Residual Pyramid Network for Image Inpainting

Existing image inpainting methods based on deep learning have made great progress. These methods either generate contextually semantically consistent images or visually excellent images, ignoring that both semantic and visual effects should be appreciated. In this article, we propose a Semantic Residual Pyramid Network (SRPNet) based on a deep generative model for image inpainting at the image and feature levels. This method encodes a masked image by a residual semantic pyramid encoder and then decodes the encoded features into a inpainted image by a multi-layer decoder. At this stage, a multi-layer attention transfer network is used to gradually fill in the missing regions of the image. To generate semantically consistent and visually superior images, the multi-scale discriminators are added to the network structure. The discriminators are divided into global and local discriminators, where the global discriminator is used to identify the global consistency of the inpainted image, and the local discriminator is used to determine the consistency of the missing regions of the inpainted image. Finally, we conducted experiments on four different datasets. As a result, great performance was achieved for filling both the regular and irregular missing regions.

Structural Knowledge-Guided Feature Inference Network for Image Inpainting

Image inpainting is an essential task in image restoration field. Currently, most meth- ods for image inpainting employ the encoder- decoder framework to restore degraded areas, and this often results in synthesizing wrong se- mantic structure due to the lack of guiding from effective prior information. In this paper, we pro- pose a structural knowledge-guided framework for image inpainting, which predicts both the edge map and corrupted content at the same time. Our model captures structural knowledge in the structure estimation branch to guide the content inference in the latent feature space. By employing self-attention mechanism to aggre- gate known information and inferred structural knowledge, our model is able to synthesize more semantically reasonable content for the corrupted areas. Extensive experiments on three bench- mark datasets demonstrate that our method out- performs most state-of-the-art methods for image inpainting in terms of the evaluation of both vi- sual quality and quantitative metrics.

Deep Generative Model for Image Inpainting With Local Binary Pattern Learning and Spatial Attention

Deep learning (DL) has demonstrated its powerful capabilities in the field of image inpainting. The DL-based image inpainting approaches can produce visually plausible results, but often generate various unpleasant artifacts, especially in the boundary and highly textured regions. To tackle this challenge, in this work, we propose a new end-to-end, two-stage (coarse-to-fine) generative model through combining a local binary pattern (LBP) learning network with an actual inpainting network. Specifically, the first LBP learning network using U-Net architecture is designed to accurately predict the structural information of the missing region, which subsequently guides the second image inpainting network for better filling the missing pixels. Furthermore, an improved spatial attention mechanism is integrated into the image inpainting network, by considering the consistency not only between the known region with the generated one, but also within the generated region itself. Extensive experiments on public datasets including <monospace>CelebA-HQ</monospace>, <monospace>Places</monospace> and <monospace>Paris StreetView</monospace> demonstrate that our model generates better inpainting results than the state-of-the-art competing algorithms, both quantitatively and qualitatively. The source code and trained models are available at <uri>https://github.com/HighwayWu/ImageInpainting</uri>.

Lightweight Generative Network for Image Inpainting Using Feature Contrast Enhancement

Lightweight Generative Network for Image Inpainting Using Feature Contrast Enhancement

Cascading Blend Network for Image Inpainting

Cascading Blend Network for Image Inpainting

FASNet: Feature Aggregation and Sharing Network for Image Inpainting

Image inpainting is a reconstruction method, where a corrupted image consisting of holes is filled with the most relevant contents from the valid region of an image. To inpaint an image, we have proposed a lightweight cascaded architecture with <i>2.5M parameters</i> consisting of encoder feature aggregation block (FAB) with decoder feature sharing (DFS) inpainting network followed by a refinement network. Initially, the FAB with DFS (inpainting) generator network is proposed which comprises of multi-level feature aggregation mechanism and feature sharing decoder. The FAB makes use of multi-scale spatial channel-wise attention to fuse weighted features from all the encoder levels. The DFS reconstructs the inpainted image with multi-scale and multi-receptive feature sharing in order to inpaint the image with smaller to larger hole regions effectively. Further, the refinement generator network is proposed for refining the inpainted image from the inpainting generator network. The effectiveness of proposed architecture is verified on CelebA-HQ [1], [2], Paris Street View (PARIS_SV) [3] and Places2 [4] datasets corrupted using publicly available NVIDIA mask dataset [5]. Extensive result analysis with detailed ablation study prove the robustness of the proposed architecture over state-of-the-art methods for image inpainting.

A New Method of Image Restoration Technology Based on WGAN

With the development of image restoration technology based on deep learning, more complex problems are being solved, especially in image semantic inpainting based on context. Nowadays, image semantic inpainting techniques are becoming more mature. However, due to the limitations of memory, the instability of training, and the lack of sample diversity, the results of image restoration are still encountering difficult problems, such as repairing the content of glitches which cannot be well integrated with the original image. Therefore, we propose an image inpainting network based on Wasserstein generative adversarial network (WGAN) distance. With the corresponding technology having been adjusted and improved, we attempted to use the Adam algorithm to replace the traditional stochastic gradient descent, and another algorithm to optimize the training used in recent years. We evaluated our algorithm on the ImageNet dataset. We obtained high-quality restoration results, indicating that our algorithm improves the clarity and consistency of the image.

Image inpainting network for filling large missing regions using residual gather

Filling arbitrary missing regions with visually plausible contents is a challenging task. Existing methods have accomplished promising performance for inpainting regular or small defects. However, filling large continuous holes is still difficult due to out of step with the performance requirement for inferring large region contents. Considering that the model needs to predict more missing contents with less information when filling large holes, we first introduce a Residual Gather Module (RGM) for high prediction performance by fully taking advantage of residual learning and capturing feature information from a hierarchical distribution. To obtain both high-quality textures and deep model performance, we propose a novel image translation architecture. Different from U-Net, our skip method is semi-complete form. It eliminates losing details caused by downsampling and is capable of designing very deep generative models. Experiments demonstrate that the proposed method exhibits outstanding performance on filling large missing regions.

A generative image inpainting network based on the attention transfer network across layer mechanism

To solve the problems of blur, artifacts, and semantic inaccuracy of existing deep learning-based image inpainting algorithm when the large-area irregular defect area images are repaired, combining the U-NET architecture and the idea of a generative adversarial network, a generative image inpainting network based on the attention transfer cross layer mechanism is proposed. The network is divided into two parts: generator and discriminator; the generator proposes a attention transfer network cross layer based on the U-NET architecture; after the input image is encoded, the attention transfer network cross layer is used to reconstruct the encoding feature map of each layer. The decoding is performed by fusing the skip connection with the corresponding potential feature in the multi-scale decoder, solving the information loss between the codec network layers. Moreover, each decoding layer of the multi-scale decoder is prompted to generate consistent content, and finally the repaired image is obtained. After the decoding is completed, the decoding feature map of each layer is converted into a multi-scale RGB image, and the reconstruction L1 loss is applied on each scale; by minimizing the adversarial loss, the discriminator can determine the authenticity of the repaired image; the perceptual loss and style loss are introduced to judge the output image, so as to restrict the generator to generate more realistic and reasonable content. A qualitative and quantitative comparative analysis is performed on the Celeba, Façade, and Places2 with the current mainstream image inpainting algorithms. The results show that the repair results in this paper are better than others in terms of mean L1 loss, PSNR, and SSIM, and more in line with human visual characteristics. The attention transfer network cross layer proposed in this paper can effectively reconstruct the more detailed coding feature map, and it can play an active guiding role in the coding process by using skip connection; the image inpainting network can generate content that is highly consistent with the structure and semantics of the real image when repairing large-area irregular defect areas, which is more in line with the human visual experience.