Image Inpainting Approach Research Articles

Improved medical image inpainting using automatic multi-task learning driven deep learning approach

Distorted medical images can drastically reduce diagnosis accuracy using computer-aided diagnostic (CAD) systems. The objective of medical image classification is to improve diagnostic imaging precision and restore regions degraded by image inpainting. Deep Learning (DL) methods like Generative Adversarial Networks (GANs) are being used to solve these medical image inpainting issues. Recently, various improved GAN models were proposed for improved image inpainting, but challenges like computational inefficiency, unreliable restoration quality, and lack of correct analysis of discriminator block. To overcome these problems, a novel GAN model is proposed with modified Generator and Discriminator blocks for improved medical image inpainting. To improve image inpainting quality with minimum computational complexity, the Multi-Task Learning (MTL)-based generator block is designed. The MTL encoder is connected with three decoders for three tasks edge detection, organ boundary generation, and image completion. The MTL-based encoder and decoders are modified with an optimum number of layers and regularization functions to improve the outcomes. The output of the generator block is input into the discriminator block, where we designed the relevant feedback strategy to decrease generator loss by automatically changing the weights on wrong classifications. The simulation results revealed the efficiency of the proposed medical image inpainting approach and discriminator block performances over the state-of-the-art. Due to the modified MTL-based generator and discriminator block in the proposed approach, the PSNR is improved by 14.31 %, SSIM is improved by 8.56%, UQI is improved by 9.45%, and MSE is reduced by 22.78% using the proposed model compared to the state-of-the-arts.

SyFormer: Structure-Guided Synergism Transformer for Large-Portion Image Inpainting

Image inpainting is in full bloom accompanied by the progress of convolutional neural networks (CNNs) and transformers, revolutionizing the practical management of abnormity disposal, image editing, etc. However, due to the ever-mounting image resolutions and missing areas, the challenges of distorted long-range dependencies from cluttered background distributions and reduced reference information in image domain inevitably rise, which further cause severe performance degradation. To address the challenges, we propose a novel large-portion image inpainting approach, namely the Structure-Guided Synergism Transformer (SyFormer), to rectify the discrepancies in feature representation and enrich the structural cues from limited reference. Specifically, we devise a dual-routing filtering module that employs a progressive filtering strategy to eliminate invalid noise interference and establish global-level texture correlations. Simultaneously, the structurally compact perception module maps an affinity matrix within the introduced structural priors from a structure-aware generator, assisting in matching and filling the corresponding patches of large-proportionally damaged images. Moreover, we carefully assemble the aforementioned modules to achieve feature complementarity. Finally, a feature decoding alignment scheme is introduced in the decoding process, which meticulously achieves texture amalgamation across hierarchical features. Extensive experiments are conducted on two publicly available datasets, i.e., CelebA-HQ and Places2, to qualitatively and quantitatively demonstrate the superiority of our model over state-of-the-arts.

Cascading Blend Network for Image Inpainting

Image inpainting refers to filling in unknown regions with known knowledge, which is in full flourish accompanied by the popularity and prosperity of deep convolutional networks. Current inpainting methods have excelled in completing small-sized corruption or specifically masked images. However, for large-proportion corrupted images, most attention-based and structure-based approaches, though reported with state-of-the-art performance, fail to reconstruct high-quality results due to the short consideration of semantic relevance. To relieve the above problem, in this paper, we propose a novel image inpainting approach, namely cascading blend network (CBNet), to strengthen the capacity of feature representation. As a whole, we introduce an adjacent transfer attention (ATA) module in the decoder, which preserves contour structure reasonably from the deep layer and blends structure-texture information from the shadow layer. In a coarse to delicate manner, a multi-scale contextual blend (MCB) block is further designed to felicitously assemble the multi-stage feature information. In addition, to ensure a high qualified hybrid of the feature information, extra deep supervision is applied to the intermediate features through a cascaded loss. Qualitative and quantitative experiments on the Paris StreetView, CelebA, and Places2 datasets demonstrate the superior performance of our approach compared with most state-of-the-art algorithms.

Optimization enabled deep learning approach with probabilistic fusion for image inpainting

ABSTRACT Nowadays, various deep learning (DL) approaches have been devised for image inpainting, which provided a substantial improvement in image quality. However, these approaches have failed to reconstruct the accurate structure of the original image. Hence, this research devised a novel and effective image inpainting approach, namely Autoregressive Flower Pollination Student Psychology Optimization (ArFPSPO). The image inpainting is carried out based on a newly devised hybrid context DL with the hybrid optimization scheme. The designed hybrid context DL approach adopts three DL techniques, such as Context encoder (CE), Context-Conditional Generative Adversarial Networks (CC-GAN), and Partial convolutional layer to complete the image inpainting process so that the outcomes are fused using probabilistic fusion with maximum entropy, thereby the final inpainted image is attained. Each of these three DL techniques is separately trained using a developed hybrid optimization technique. The experimental outcome reveals that the devised model gives optimal performance.

An Image Inpainting Approach to Short-Term Load Forecasting

In current power systems, electrical energy is generated whenever there is a demand for it. Therefore, load forecasting, which estimates the active load in advance, is imperative for power system planning and operations. Based on the time horizon, load forecasting is classified as very short-term (below one day), short-term (a day to two weeks), medium-term (two weeks to three years) and long-term (over three years). This paper focuses on the short-term forecasting. The complex multi-level seasonality of load series (e.g., the load in a given hour is not only dependent on load in the previous hour, but also on the previous day.s load in the same hour, and on the previous week's load in the same day-of-the-week and hour) makes this task challenging, especially when the load data is represented in 1d numerical series. However, in multi-channel images, the patterns in spatial neighbourhood of one channel and the patterns in the neighbourhood along the channel dimension are able to be captured by 3d image processing operations. Hence, this study proposes to transform electrical load data from 1d series to 3d images and transform the problem from future series forecasting to missing patch inpainting. Furthermore, it proposes a recurrent neural network to model the temporal trends in the series by convolutional operations on the spatial neighbourhood in the images. Experimental results demonstrate the effectiveness of the proposed method on two benchmarks and show the capability of inferring the future load from related series if there is a lack of history.

Multichannel Cross-Scale Semantic Coherent Attention Network for Image Inpainting

This paper investigates a cross-scale space semantic feature coherent image inpainting approach since it is challenging for the existing image inpainting methods to fuse the semantic feature information effectively. Firstly, we learn the feature semantic relevance step-by-step from the high-level semantic feature map’s attention mechanism and then we apply what we have learned to the preceding low-level feature map. In order to preserve the visual and semantic coherence of image repair, the missing content can be filled by changing attention from deep to shallow in a multiscale manner. A broader receptive field is generated by partial convolution, and semantic feature relevance is achieved using a multiscale cross feature space feature attention mechanism based on semantic attention. This technique improves the extensibility and continuity of the restored images by reconstructing the semantic information of different feature spaces, not only taking into account the reuse of existing semantic space features but also including across feature spaces. The experimental results demonstrated an improvement in PSNR, SSIM, and L1 performance by 10.50%, 0.13%, and 47.09%, respectively, with clear benefits.

DKH: a hybridized approach for image inpainting using Bayes probabilistic-based image fusion

DKH: a hybridized approach for image inpainting using Bayes probabilistic-based image fusion

Reference-Guided Landmark Image Inpainting With Deep Feature Matching

Despite impressive progress made by recent image inpainting methods, they often fail to predict the original content when the corrupted region contains unique structures, especially for landmark images. Applying similar images as a reference is helpful but introduces a style gap of textures, resulting in color misalignment. To this end, we propose a style-robust approach for reference-guided landmark image inpainting, taking advantage of both the representation power of learned deep features and the structural prior from the reference image. By matching deep features, our approach builds style-robust nearest-neighbor mapping vector fields between the corrupted and reference images, in which the loss of information due to corruption leads to mismatched mapping vectors. To correct these mismatched mapping vectors based on the relationship between the uncorrupted and corrupted regions, we introduce mutual nearest neighbors as reliable anchors and interpolate around these anchors progressively. Finally, based on the corrected mapping vector fields, we propose a two-step warping strategy to complete the corrupted image, utilizing the reference image as a structural “blueprint”, avoiding the style misalignment problem. Extensive experiments show that our approach effectively and robustly assists image inpainting methods in restoring unique structures in the corrupted image.

Image inpainting via spatial projections

Image inpainting is now-a-days sought after due to its wide variety of applications in the reconstruction of the corrupted image, occlusion removal, reflection removal, etc. Existing image inpainting approaches utilize different types of attention mechanisms to inpaint the image and produce visibly admirable results. These methods are more concerned at weighing the feature maps of the hole region with some weight from the non-hole region. But, due to the lack of spatial contextual correlation in the attention maps, the inpainted image may suffer from the inconsistencies among hole and non-hole regions. Transformer-based inpainting methods give significant results by capturing the relationship between the patches with a compromise of high computational complexity. In this context, we propose a novel spatial projection layer (SPL) without any attention mechanism to project the spatial contextual information in the hole region from non-hole regions for producing a spatially plausible inpainted image. The SPL is proposed mainly to focus on the non-hole spatial information in the high-level feature maps for filling the hole regions efficiently. Also, while training the network, we propose the use of edge loss with a Canny edge operator for image inpainting to focus on the relevant edges instead of noise contents. Analysis with the extensive experiments, ablation, and user study on the proposed architecture demonstrates the superiority over existing state-of-the-art methods for image inpainting. The code is available at: https://github.com/shrutiphutke/spatial_projection_inpainting.

Blind Image Inpainting with Sparse Directional Filter Dictionaries for Lightweight CNNs

Blind inpainting algorithms based on deep learning architectures have shown a remarkable performance in recent years, typically outperforming model-based methods both in terms of image quality and run time. However, neural network strategies typically lack a theoretical explanation, which contrasts with the well-understood theory underlying model-based methods. In this work, we leverage the advantages of both approaches by integrating theoretically founded concepts from transform domain methods and sparse approximations into a CNN-based approach for blind image inpainting. To this end, we present a novel strategy to learn convolutional kernels that applies a specifically designed filter dictionary whose elements are linearly combined with trainable weights. Numerical experiments demonstrate the competitiveness of this approach. Our results show not only an improved inpainting quality compared to conventional CNNs but also significantly faster network convergence within a lightweight network design. Our code is available at https://github.com/cv-stuttgart/SDPF_Blind-Inpainting .

Exploration of Image Inpainting approaches and challenges: A Survey

Exploration of Image Inpainting approaches and challenges: A Survey

A new approach for image inpainting by modification on total variation model

In this paper, we present a new method for inpainting an image part of which is damaged or lost. Our approach contains a modification on the total variation model. It is based on a new insight into the neighborhood pixels of the damaged areas. Some numerical experiments are given to confirm our model.

LRGAN: A computational lightweight image inpainting neural network

<p indent="0mm">Currently, most image inpainting approaches focus on improving the quality of inpainted images while ignoring the requirement for a low-end platform. In this work, we propose a lightweight group convolution module in place of vanilla convolution. This module, named lightweight attention-based group convolution (LABGC), uses a depth-wise convolution to achieve multi-scale image inpainting. Meanwhile, to alleviate the information constraint between channels, we propose a cycle channel attention mechanism. Then in our proposed LRGAN inpainting network, we use LABGC. We use parallel discriminators in the overall network design to ensure local and non-local consistency. Experiments show that our model is as accurate as state-of-the-art approaches in terms of speed and FLOPs.

Feature learning and patch matching for diverse image inpainting

We present an image inpainting approach to generate diverse high-quality inpainting results. Recent advances in deep adversarial networks have led to significant improvements in the challenging task of filling large holes in natural images. Although deep generative models can generate visually plausible structures and textures, most of them are not interpretable, making it difficult to control the inpainting output. In addition, deep generative models do not have capacity to produce diverse results for each input. To address such limitations, we design a novel free-form image inpainting framework with two sequential steps: the first step formulates the inpainting process as a regression problem and utilizes a U-Net-like convolutional neural network to map an input to a coarse inpainting output, and the second step utilizes the nearest neighbor based pixel-wise matching to map the coarse output to diverse high-quality outputs. The second step allows our approach to compose novel high-quality content by copy-pasting high-frequency missing information from different training exemplars. Experiments on multiple datasets, i.e., CelebA-HQ, AFHQ, and Paris StreetView, show that our approach is able to offer multiple natural outputs with higher diversity in a controllable manner.

Edge-guided GAN: a depth image inpainting approach guided by edge information

目的目前大多数深度图像修复方法可分为两类：色彩图像引导的方法和单个深度图像修复方法。色彩图像引导的方法利用色彩图像真值，或其上一帧、下一帧提供的信息来修复深度图像。若缺少相应信息，这类方法是无效的。单个深度图像修复方法可以修复数据缺失较少的深度图像。但是，无法修复带有孔洞（数据缺失较大）的深度图像。为解决以上问题，本文将生成对抗网络（generative adversarial network，GAN）应用于深度图像修复领域，提出了一种基于GAN的单个深度图像修复方法，即Edge-guided GAN。 方法首先，通过Canny算法获得待修复深度图像的边界图像，并将此两个单通道图像（待修复深度图像和边界图像）合并成一个2通道数据；其次，设计Edge-guided GAN高性能的生成器、判别器和损失函数，将此2通道数据作为生成器的输入，训练生成器，以生成器生成的深度图像（假值）和深度图像真值为判别器的输入，训练判别器；最终得到深度图像修复模型，完成深度图像修复。 结果在Apollo scape数据集上与其他4种常用的GAN、不带边界信息的Edge-guided GAN进行实验分析。在输入尺寸为256×256像素，掩膜尺寸为32×32像素情况下，Edge-guided GAN的峰值信噪比（peak signal-to-noise ratio，PSN）比性能第2的模型提高了15.76%；在掩膜尺寸为64×64像素情况下，Edge-guided GAN的PSNR比性能第2的模型提高了18.64%。 结论Edge-guided GAN以待修复深度图像的边界信息为其修复的约束条件，有效地提取了待修复深度图像特征，大幅度地提高了深度图像修复的精度。

Image Inpainting Based on Structural Tensor Edge Intensity Model

In the exemplar-based image inpainting approach, there are usually two major problems: the unreasonable calculation of priority and only considering the color features in the patch lookup strategy. In this paper, we propose an image inpainting approach based on the structural tensor edge intensity model. First, we use the progressive scanning inpainting method to avoid the image filling order being affected by the priority function. Then, we use the edge intensity model to build the patches similarity function for correctly identifying the local image structure. Finally, the balance operator is used to restrict the excessive propagation of structural information to ensure the correct structural reconstruction. The experimental results show that the our approach is comparable and even superior to some state-of-the-art inpainting algorithms.

Image Inpainting Using Two-Stage Loss Function and Global and Local Markovian Discriminators.

Image inpainting networks can produce visually reasonable results in the damaged regions. However, existing inpainting networks may fail to reconstruct the proper structures or tend to generate the results with color discrepancy. To solve this issue, this paper proposes an image inpainting approach using the proposed two-stage loss function. The loss function consists of different Gaussian kernels, which are utilized in different stages of network. The use of our two-stage loss function in coarse network helps to focus on the image structure, while the use of it in refinement network is helpful to restore the image details. Moreover, we proposed a global and local PatchGANs (GAN means generative adversarial network), named GL-PatchGANs, in which the global and local markovian discriminators were used to control the final results. This is beneficial to focus on the regions of interest (ROI) on different scales and tends to produce more realistic structural and textural details. We trained our network on three popular datasets on image inpainting separately, both Peak Signal to Noise ratio (PSNR) and Structural Similarity (SSIM) between our results, and ground truths on test images show that our network can achieve better performance compared with the recent works in most cases. Besides, the visual results on three datasets also show that our network can produce visual plausible results compared with the recent works.

Adaptive Image Sampling Using Deep Learning and Its Application on X-Ray Fluorescence Image Reconstruction

This paper presents an adaptive image sampling algorithm based on Deep Learning (DL). The adaptive sampling mask generation network is jointly trained with an image inpainting network. The sampling rate is controlled in the mask generation network, and a binarization strategy is investigated to make the sampling mask binary. Besides the image sampling and reconstruction application, we show that the proposed adaptive sampling algorithm is able to speed up raster scan processes such as the X-Ray fluorescence (XRF) image scanning process. Recently XRF laboratory-based systems have evolved to lightweight and portable instruments thanks to technological advancements in both X-Ray generation and detection. However, the scanning time of an XRF image is usually long due to the long exposures requires (e.g., $100 \mu s-1ms$ per point). We propose an XRF image inpainting approach to address the issue of long scanning time, thus speeding up the scanning process while still maintaining the possibility to reconstruct a high quality XRF image. The proposed adaptive image sampling algorithm is applied to the RGB image of the scanning target to generate the sampling mask. The XRF scanner is then driven according to the sampling mask to scan a subset of the total image pixels. Finally, we inpaint the scanned XRF image by fusing the RGB image to reconstruct the full scan XRF image. The experiments show that the proposed adaptive sampling algorithm is able to effectively sample the image and achieve a better reconstruction accuracy than that of the existing methods.

Interactive deformation‐driven person silhouette image synthesis

AbstractThis paper addresses the problem of synthesizing person silhouette images. Previous methods on human‐centric image synthesis deal with normal images taken under consistent lighting conditions. However, a silhouette image has an inconsistent lighting appearance, in which the dark subject and the bright background form a strong contrast. Therefore, it brings great challenges to person silhouette image synthesis. We present a staged method to synthesize realistic person silhouette photos with various poses. The purpose of our method is to give ordinary users the opportunities to interactively adjust the pose of a person in a silhouette photo. The method consists of four main steps: person detection, person analysis, interactive deformation, and image synthesis. To obtain the shape of the target person, we propose a silhouette image segmentation algorithm combined with person detection. Moreover, we also present an effective image inpainting approach to complement a silhouette image with an irregular hole. Experimental results show that the proposed method can generate a set of realistic person silhouette images with interactively changed poses.

Review of Inpainting Algorithms for Wireless Communication Application

Digital image inpainting is a technique of restoring large removed /damaged regions of an image with the data from the surrounding pixels of the removed region. The issue of image restoration with inpainting techniques occurs commonly in computer vision/image processing when unwanted objects have to be removed from images, for filling cracks in photographs, etc. Digital image inpainting approach is an active field of research in two significant applications of wireless communication: image compression and image recovery from a damaged image due to errors in a wireless channel. This work presents a brief survey of different image inpainting techniques and their contributions to different wireless communication applications.