Image Inpainting Methods Research Articles

Structure-aware multi-view image inpainting using dual consistency attention

Image inpainting based on deep learning has made remarkable progress and is widely used in image editing, cultural relic preservation, etc. However, most image inpainting methods are implemented based on single-view images. This does not fully utilize the known information and leads to unsatisfactory inpainting results. Moreover, these methods usually ignore the importance of image consistency and the surrounding regions, leading to irrelevant contents and visual artifacts in the inpainting results. To solve these problems, a structure-aware multi-view image inpainting method using dual consistency attention (SM-DCA) is proposed in this paper. It consists of two parts. The first part is the structure-aware multi-view image inpainting. This part constructs structure views as additional views to assist image inpainting. It is implemented by two networks: a structure inpainting network with strong constraints (SSC) and an image inpainting network with dual consistency attention (IDCA). SSC is used to repair structure views and make them closely resemble the ground truth through strong constraints. IDCA improves the consistency between the generated content and the whole image, making the repaired image more reasonable. The second part is image refinement, implemented by an image local refinement network (ILR). It can focus on the surrounding regions, eliminating boundary artifacts and obtaining finer local details. In Paris StreetView, SM-DCA achieves 22.0194, 0.7457 and 0.0557 in terms of PSNR, SSIM and MAE at 50%–60% damage. The corresponding values in CelebA are 22.5526, 0.8623 and 0.0453, respectively. The extensive experimental results on the Paris StreetView and CelebA datasets demonstrate the superiority of SM-DCA.

A blind recovery technique with integer wavelet transforms in image watermarking

The development of internet technology has simplified the sharing and modification of digital image information. The aim of this study is to propose a new blind recovery technique based on integer wavelets transform (BRIWT) by utilizing their image content. The LSB adjustment technique on the integer wavelet transform is used to embed recovery data into the two least significant bits (LSB) of the image content. Authentication bits are embedded into the current locations of the LSB of the image content, while the recovery information is embedded into different block locations based on the proposed block mapping. The embedded recovery data is securely placed at random locations within the two LSBs using a secret key. A three-layer embedding of authentication bits is used to validate the integrity of the image contents, achieving high precision and accuracy. Tamper localization accuracy is employed to identify recovery bits from the image content. This research also investigates the image inpainting method to enhance recovery from tampered images. The proposed image inpainting is performed by identifying non-tampered pixels in the surrounding tamper localization. The results demonstrate that the proposed scheme can produce highly watermarked images with imperceptibility, with an average SSIM value of 0.9978 and a PSNR value of 46.20 dB. The proposed scheme significantly improves the accuracy of tamper localization, with a precision of 0.9943 and an accuracy of 0.9971. The proposed recovery technique using integer wavelet transforms achieves high-quality blind recovery with an SSIM value of 0.9934 under a tampering rate of 10%. The findings of this study reveal that the proposed scheme improves the quality of blind recovery by 14.2 % under a tampering rate of 80 %.

Metameric Inpainting for Image Warping.

Image-warping, a per-pixel deformation of one image into another, is an essential component in immersive visual experiences such as virtual reality or augmented reality. The primary issue with image warping is disocclusions, where occluded (and hence unknown) parts of the input image would be required to compose the output image. We introduce a new image warping method, Metameric image inpainting - an approach for hole-filling in real-time with foundations in human visual perception. Our method estimates image feature statistics of disoccluded regions from their neighbours. These statistics are inpainted and used to synthesise visuals in real-time that are less noticeable to study participants, particularly in peripheral vision. Our method offers speed improvements over the standard structured image inpainting methods while improving realism over colour-based inpainting such as push-pull. Hence, our work paves the way towards future applications such as depth image-based rendering, 6-DoF 360 rendering, and remote render-streaming.

A Motion Deblurring Network for Enhancing UAV Image Quality in Bridge Inspection

Unmanned aerial vehicles (UAVs) have been increasingly utilized for facility safety inspections due to their superior safety, cost effectiveness, and inspection accuracy compared to traditional manpower-based methods. High-resolution images captured by UAVs directly contribute to identifying and quantifying structural defects on facility exteriors, making image quality a critical factor in achieving accurate results. However, motion blur induced by external factors such as vibration, low light conditions, and wind during UAV operation significantly degrades image quality, leading to inaccurate defect detection and quantification. To address this issue, this research proposes a deblurring network using a Generative Adversarial Network (GAN) to eliminate the motion blur effect in UAV images. The GAN-based motion deblur network represents an image inpainting method that leverages generative models to correct blurry artifacts, thereby generating clear images. Unlike previous studies, this proposed approach incorporates deblur and blur learning modules to realistically generate blur images required for training the generative models. The UAV images processed using the motion deblur network are evaluated using a quality assessment method based on local blur map and other well-known image quality assessment (IQA) metrics. Moreover, in the experiment of crack detection utilizing the object detection system, improved detection results are observed when using enhanced images. Overall, this research contributes to improving the quality and accuracy of facility safety inspections conducted with UAV-based inspections by effectively addressing the challenges associated with motion blur effects in UAV-captured images.

Research on image inpainting methods based on machine learning

The technique of restoring sections of a picture that have been lost or damaged is known as "image inpainting." In light of recent developments in machine learning, academics have begun investigating the possibility of using deep learning methods to the process of picture inpainting. However, the current body of research does not include a comprehensive review of the many different inpainting methods that are based on machine learning, nor does it compare and contrast these methods. This article provides an overview of some of the most advanced and common machine learning based image restoration techniques that are currently available. These techniques include Multivariate inpainting technology and Unit inpainting technology, such as Context-Encoder Network, Generative Adversarial Network (GAN), and U-Net Network. We examine not just the benefits and drawbacks of each method, but also the ways in which it might be used in a variety of settings. At the conclusion of the piece, we predict that machine learning-based inpainting will continue to gain popularity and application in the years to come.

Forward stagewise regression with multilevel memristor for sparse coding

Sparse coding is a prevalent method for image inpainting and feature extraction, which can repair corrupted images or improve data processing efficiency, and has numerous applications in computer vision and signal processing. Recently, several memristor-based in-memory computing systems have been proposed to enhance the efficiency of sparse coding remarkably. However, the variations and low precision of the devices will deteriorate the dictionary, causing inevitable degradation in the accuracy and reliability of the application. In this work, a digital-analog hybrid memristive sparse coding system is proposed utilizing a multilevel Pt/Al2O3/AlOx/W memristor, which employs the forward stagewise regression algorithm: The approximate cosine distance calculation is conducted in the analog part to speed up the computation, followed by high-precision coefficient updates performed in the digital portion. We determine that four states of the aforementioned memristor are sufficient for the processing of natural images. Furthermore, through dynamic adjustment of the mapping ratio, the precision requirement for the digit-to-analog converters can be reduced to 4 bits. Compared to the previous system, our system achieves higher image reconstruction quality of the 38 dB peak-signal-to-noise ratio. Moreover, in the context of image inpainting, images containing 50% missing pixels can be restored with a reconstruction error of 0.0424 root-mean-squared error.

Image Inpainting Based on Multi-Level Feature Aggregation Network for Future Internet

(1) Background: In the future Internet era, clarity and structural rationality are important factors in image inpainting. Currently, image inpainting techniques based on generative adversarial networks have made great progress; however, in practical applications, there are still problems of unreasonable or blurred inpainting results for high-resolution images and images with complex structures. (2) Methods: In this work, we designed a lightweight multi-level feature aggregation network that extracts features from convolutions with different dilation rates, enabling the network to obtain more feature information and recover more reasonable missing image content. Fast Fourier convolution was designed and used in the generative network, enabling the generator to consider the global context at a shallow level, making it easier to perform high-resolution image inpainting tasks. (3) Results: The experiment shows that the method designed in this paper performs well in geometrically complex and high-resolution image inpainting tasks, providing a more reasonable and clearer inpainting image. Compared with the most advanced image inpainting methods, our method outperforms them in both subjective and objective evaluations. (4) Conclusions: The experimental results indicate that the method proposed in this paper has better clarity and more reasonable structural features.

Sparse self-attention transformer for image inpainting

Learning-based image inpainting methods have made remarkable progress in recent years. Nevertheless, these methods still suffer from issues such as blurring, artifacts, and inconsistent contents. The use of vanilla convolution kernels, which have limited perceptual fields and spatially invariant kernel coefficients, is one of the main causes for these problems. In contrast, the multi-headed attention in the transformer can effectively model non-local relations among input features by generating adaptive attention scores. Therefore, this paper explores the feasibility of employing the transformer model for the image inpainting task. However, the multi-headed attention transformer blocks pose a significant challenge due to their overwhelming computational cost. To address this issue, we propose a novel U-Net style transformer-based network for the inpainting task, called the sparse self-attention transformer (Spa-former). The Spa-former retains the long-range modeling capacity of transformer blocks while reducing the computational burden. It incorporates a new channel attention approximation algorithm that reduces attention calculation to linear complexity. Additionally, it replaces the canonical softmax function with the ReLU function to generate a sparse attention map that effectively excludes irrelevant features. As a result, the Spa-former achieves effective long-range feature modeling with fewer parameters and lower computational resources. Our empirical results on challenging benchmarks demonstrate the superior performance of our proposed Spa-former over state-of-the-art approaches.

GANMasker: A Two-Stage Generative Adversarial Network for High-Quality Face Mask Removal.

Deep-learning-based image inpainting methods have made remarkable advancements, particularly in object removal tasks. The removal of face masks has gained significant attention, especially in the wake of the COVID-19 pandemic, and while numerous methods have successfully addressed the removal of small objects, removing large and complex masks from faces remains demanding. This paper presents a novel two-stage network for unmasking faces considering the intricate facial features typically concealed by masks, such as noses, mouths, and chins. Additionally, the scarcity of paired datasets comprising masked and unmasked face images poses an additional challenge. In the first stage of our proposed model, we employ an autoencoder-based network for binary segmentation of the face mask. Subsequently, in the second stage, we introduce a generative adversarial network (GAN)-based network enhanced with attention and Masked-Unmasked Region Fusion (MURF) mechanisms to focus on the masked region. Our network generates realistic and accurate unmasked faces that resemble the original faces. We train our model on paired unmasked and masked face images sourced from CelebA, a large public dataset, and evaluate its performance on multi-scale masked faces. The experimental results illustrate that the proposed method surpasses the current state-of-the-art techniques in both qualitative and quantitative metrics. It achieves a Peak Signal-to-Noise Ratio (PSNR) improvement of 4.18 dB over the second-best method, with the PSNR reaching 30.96. Additionally, it exhibits a 1% increase in the Structural Similarity Index Measure (SSIM), achieving a value of 0.95.

A Novel Thanka Image Inpainting Method with Euler’s Elastica and Iterative Denoising and Backward Projections

This paper presents a brand-new Thanka picture inpainting technique based on Euler’s elastica, iterative denoising, and backward projections (EEIDBP). Specifically, a model of Euler’s elastica is introduced to estimate the original observation due to its lower staircasing effects and better approximation of natural images. A method for backward projection and iterative denoising is applied to achieve a more accurate estimate of the original signal by alternating iterations between the estimation of the original signal and the estimation of the original observation. The experimental findings demonstrate that, in terms of a subjective assessment, the quantitative peak signal-to-noise ratio (PSNR), and the structural similarity (SSIM), the proposed technique outperforms the state-of-the-art picture inpainting methods.

Pyramid-VAE-GAN: Transferring hierarchical latent variables for image inpainting

Significant progress has been made in image inpainting methods in recent years. However, they are incapable of producing inpainting results with reasonable structures, rich detail, and sharpness at the same time. In this paper, we propose the Pyramid-VAE-GAN network for image inpainting to address this limitation. Our network is built on a variational autoencoder (VAE) backbone that encodes high-level latent variables to represent complicated high-dimensional prior distributions of images. The prior assists in reconstructing reasonable structures when inpainting. We also adopt a pyramid structure in our model to maintain rich detail in low-level latent variables. To avoid the usual incompatibility of requiring both reasonable structures and rich detail, we propose a novel cross-layer latent variable transfer module. This transfers information about long-range structures contained in high-level latent variables to low-level latent variables representing more detailed information. We further use adversarial training to select the most reasonable results and to improve the sharpness of the images. Extensive experimental results on multiple datasets demonstrate the superiority of our method. Our code is available at https://github.com/thy960112/Pyramid-VAE-GAN.

Inpainting with Separable Mask Update Convolution Network.

Image inpainting is an active area of research in image processing that focuses on reconstructing damaged or missing parts of an image. The advent of deep learning has greatly advanced the field of image restoration in recent years. While there are many existing methods that can produce high-quality restoration results, they often struggle when dealing with images that have large missing areas, resulting in blurry and artifact-filled outcomes. This is primarily because of the presence of invalid information in the inpainting region, which interferes with the inpainting process. To tackle this challenge, the paper proposes a novel approach called separable mask update convolution. This technique automatically learns and updates the mask, which represents the missing area, to better control the influence of invalid information within the mask area on the restoration results. Furthermore, this convolution method reduces the number of network parameters and the size of the model. The paper also introduces a regional normalization technique that collaborates with separable mask update convolution layers for improved feature extraction, thereby enhancing the quality of the restored image. Experimental results demonstrate that the proposed method performs well in restoring images with large missing areas and outperforms state-of-the-art image inpainting methods significantly in terms of image quality.

Combining Deep Image Prior and Second-Order Generalized Total Variance for Image Inpainting

Image inpainting is a crucial task in computer vision that aims to restore missing and occluded parts of damaged images. Deep-learning-based image inpainting methods have gained popularity in recent research. One such method is the deep image prior, which is unsupervised and does not require a large number of training samples. However, the deep image prior method often encounters overfitting problems, resulting in blurred image edges. In contrast, the second-order total generalized variation can effectively protect the image edge information. In this paper, we propose a novel image restoration model that combines the strengths of both the deep image prior and the second-order total generalized variation. Our model aims to better preserve the edges of the image structure. To effectively solve the optimization problem, we employ the augmented Lagrangian method and the alternating direction method of the multiplier. Numerical experiments show that the proposed method can repair images more effectively, retain more image details, and achieve higher performance than some recent methods in terms of peak signal-to-noise ratio and structural similarity.

Generative adversarial network based image inpainting

Image inpainting, which is the repair of pixels in damaged areas of an image to make it look as much like the original image as possible. Deep learning-based image inpainting technology is a prominent area of current research interest. This paper focuses on a systematic and comprehensive study of GAN-based image inpainting and presents an analytical summary. Firstly, this paper introduces GAN, which includes the principle of GAN and its mathematical expression. Secondly, the recent GAN-based image inpainting algorithms are summarized, and the advantages and disadvantages of each algorithm are listed. After that, the evaluation metrics, and common datasets of deep learning-based image inpainting are listed. Finally, the existing image inpainting methods are summarized and summarized, and the ideas for future key research directions are presented and prospected.

A Generate Adversarial Network with Structural Branch Assistance for Image Inpainting

In digital image inpainting tasks, existing deep-learning-based image inpainting methods have achieved remarkable staged results by introducing structural prior information into the network. However, the corresponding relationship between texture and structure is not fully considered, and the inconsistency between texture and structure appears in the results of the current method. In this paper, we propose a dual-branch network with structural branch assistance, which decouples the inpainting of low-frequency and high-frequency information utilizing parallel branches. The feature fusion (FF) module is introduced to integrate the feature information from the two branches, which effectively ensures the consistency of structure and texture in the image. The feature attention (FA) module is introduced to extract long-distance feature information, which enhances the consistency between the local features of the image and the overall image and gives the image a more detailed texture. Experiments on the Paris StreetView and CelebA-HQ datasets prove the effectiveness and superiority of our method.

Generator pyramid for high-resolution image inpainting

Inpainting high-resolution images with large holes challenges existing deep learning-based image inpainting methods. We present a novel framework—PyramidFill for high-resolution image inpainting, which explicitly disentangles the task into two sub-tasks: content completion and texture synthesis. PyramidFill attempts to complete the content of unknown regions in a lower-resolution image, and synthesize the textures of unknown regions in a higher-resolution image, progressively. Thus, our model consists of a pyramid of fully convolutional GANs, wherein the content GAN is responsible for completing contents in the lowest-resolution masked image, and each texture GAN is responsible for synthesizing textures in a higher-resolution image. Since completing contents and synthesizing textures demand different abilities from generators, we customize different architectures for the content GAN and texture GAN. Experiments on multiple datasets including CelebA-HQ, Places2 and a new natural scenery dataset (NSHQ) with different resolutions demonstrate that PyramidFill generates higher-quality inpainting results than the state-of-the-art methods.

DARGS: Image inpainting algorithm via deep attention residuals group and semantics

To solving the problems that the existing image inpainting methods lack authenticity, do not deal with the information of missing and non-missing regions flexibly, and do not deal with the image feature information on different stages effectively, we propose an image restoration method combining Semantic Priors and Deep Attention Residual Group. The image restoration method mainly consists of Semantic Priors Network, Deep Attention Residual Group, and Full-scale Skip Connection. The Semantic Priors Network learns the complete semantic prior information of the visual elements in the missing regions and completes the missing regions with the learned semantic information. The deep attention residual set allows the generator to focus more on not only the missing regions of the image but also to learn the features of each channel adaptive. The full-scale jump joins can combine the low-level feature maps containing image boundaries with the high-level feature maps containing image textures and details to repair the missing regions. In this paper, a full comparison experiment is conducted on the CelebA-HQ and the Paris StreetView datasets, and the experimental results show that proposed method outperforms the current representative state-of-the-art image restoration methods.

A Survey on Improved GAN based Image Inpainting for Different Aims

Computer Vision (CV) is an important field of Artificial Intelligence (AI) that focuses on processing and analyzing the visual data input like images or videos. And image inpainting technique is a task of reconstructing image with missing parts by using the information of the existing part, which becomes a crucial part in CV. It has various image processing applications such as digital cultural heritage preservation, old photograph restoration and object removal. The researchers found that using generative adversarial network can extract the feature from image without huge amount of computation which are two main challenges of image inpainting, therefore GAN-based inpainting method has recently been a hub for research. In this paper, we review most of the latest improved image inpainting methods and divide them into three categories according to their different aims: for semantic image inpainting, for generating higher diversity and for irregular or free-form image input. Firstly, we review the methods from structure and algorithm. Then we analyze the advantages and disadvantages of each method. Finally, we conclude the challenges or limitations raised from our findings and some potential development trend.

RNON: image inpainting via repair network and optimization network.

In the last few years, image inpainting methods based on deep learning models had shown obvious advantages compared with existing traditional methods. The former can better generate visually reasonable image structure and texture information. However, the existing premier convolutional neural networks methods usually causes the problems of excessive color difference and image texture loss and distortion phenomenon. The paper has proposed an effective image inpainting method using generative adversarial networks, which is composed of two mutually independent generative confrontation networks. Among them, the image repair network module aims to solve the problem of repairing the irregular missing areas of the image, and its generator is based on a partial convolutional network. The image optimization network module aims to solve the problem of local chromatic aberration in the repaired images, and its generator has based on deep residual networks. Through the synergy of the two network modules, the visual effect and image quality of the images has improved. The experimental results can show that the proposed method (RNON) performs better from comparisons of qualitative and quantitative evaluations with state-of-the-arts in image inpainting quality field.

FFTI: Image inpainting algorithm via features fusion and two-steps inpainting

In view of the faultiness that the existing image inpainting methods fail to make full use of the complete region to predict the missing region features when the object features are seriously missing, resulting in discontinuous features and fuzzy detail texture of the inpainting results, a fine inpainting method of incomplete image based on features fusion and two-steps inpainting (FFTI) is proposed in this paper. Firstly, the dynamic memory networks (DMN+) are used to fuse the external features and internal features of the incomplete image to generate the incomplete image optimization map. Secondly, a generation countermeasure generative network with gradient penalty constraints is constructed to guide the generator to rough repair the optimized incomplete image and obtain the rough repair map of the target to be repaired. Finally, the coarse repair graph is further optimized by the idea of coherence of relevant features to obtain the final fine repair graph. It is verified by simulation on three image data sets with different complexity, and compared with the existing dominant repair model for visual effect and objective data. The experimental results show that the results of the model repair in this paper are more reasonable in texture structure, better than other models in visual effect and objective data, and the Peak Signal-to-Noise Ratio of the proposed algorithm in the most challenging underwater targe dataset is 27.01, the highest Structural Similarity Index is 0.949.