Inpainting Algorithm Research Articles

CGD-Based Inpainting Algorithm for Time-Varying Signals on Strong Product Graph

CGD-Based Inpainting Algorithm for Time-Varying Signals on Strong Product Graph

Transformer-Guided Video Inpainting Algorithm Based on Local Spatial-Temporal joint

INTRODUCTION: Video inpainting is a very important task in computer vision, and it’s a key component of various practical applications. It also plays an important role in video occlusion removal, traffic monitoring and old movie restoration technology. Video inpainting is to obtain reasonable content from the video sequence to fill the missing region, and maintain time continuity and spatial consistency.OBJECTIVES: In previous studies, due to the complexity of the scene of video inpainting, there are often cases of fast motion of objects in the video or motion of background objects, which will lead to optical flow failure. So the current video inpainting algorithm hasn’t met the requirements of practical applications. In order to avoid the problem of optical flow failure, this paper proposes a transformer-guided video inpainting model based on local Spatial-temporal joint.METHODS: First, considering the rich Spatial-temporal relationship between local flows, a Local Spatial-Temporal Joint Network (LSTN) including encoder, decoder and transformer module is designed to roughly inpaint the local corrupted frames, and the Deep Flow Network is used to calculate the local bidirectional corrupted flows. Then, the local corrupted optical flow map is input into the Local Flow Completion Network (LFCN) with pseudo 3D convolution and attention mechanism to obtain a complete set of bidirectional local optical flow maps. Finally, the roughly inpainted local frame and the complete bidirectional local optical flow map are sent to the Spatial-temporal transformer and the inpainted video frame is output.RESULTS: Experiments show that the algorithm achieves high quality results in the video target removal task, and has a certain improvement in indicators compared with advanced technologies.CONCLUSION: Transformer-Guided Video Inpainting Algorithm Based on Local Spatial-Temporal joint can obtain high-quality optical flow information and inpainted result video.

Infrared Dim and Small Target Detection Based on Background Prediction

Infrared dim and small target detection is a key technology for various detection tasks. However, due to the lack of shape, texture, and other information, it is a challenging task to detect dim and small targets. Recently, since many traditional algorithms ignore the global information of infrared images, they generate some false alarms in complicated environments. To address this problem, in this paper, a coarse-to-fine deep learning-based method was proposed to detect dim and small targets. Firstly, a coarse-to-fine detection framework integrating deep learning and background prediction was applied for detecting targets. The framework contains a coarse detection module and a fine detection module. In the coarse detection stage, Region Proposal Network (RPN) is employed to generate masks in target candidate regions. Then, to further optimize the result, inpainting is utilized to predict the background using the global semantics of images. In this paper, an inpainting algorithm with a mask-aware dynamic filtering module was incorporated into the fine detection stage to estimate the background of the candidate targets. Finally, compared with existing algorithms, the experimental results indicate that the proposed framework has effective detection capability and robustness for complex surroundings.

A Ginzburg-Landau-$${H}^{-1}$$ Model and Its SAV Algorithm for Image Inpainting

A Ginzburg-Landau-$${H}^{-1}$$ Model and Its SAV Algorithm for Image Inpainting

Hybrid attention generative adversarial network: texture inpainting algorithm for iris defects with excellent repair performance and generalization

Hybrid attention generative adversarial network: texture inpainting algorithm for iris defects with excellent repair performance and generalization

MRS-Transformer: Texture Splicing Method to Remove Defects in Solid Wood Board

Defects in wood growth affect the product’s quality and grade. At present, the research on texture defects of wood mainly focuses on defect localization, ignoring the splicing problem of maintaining texture consistency. In this paper, we designed the MRS-Transformer network and introduced image inpainting to the field of solid wood board splicing. First, we proposed an asymmetric encoder-decoder based on Vision Transformer, where the encoder uses a fixed mask(M) strategy, discarding the masked patches and using only the unmasked visual patches as input to reduce model calculations. Second, we designed a reverse Swin (RS) module with multi-scale characteristics as the decoder to adjust the divided image patches’ size and complete the restoration from coarse to fine. Finally, we proposed a weighted L2 loss (MSE, mean square error), which assigns different weights to the unmasked region according to the distance from the defective region, allowing the model to make full use of the effective pixels to repair the masked region. To demonstrate the effectiveness of the designed modules, we used MSE (mean square error), LPIPS (learned perceptual image patch similarity), PSNR (peak signal to noise ratio), SSIM (structural similarity), and FLOPs (floating point operations) to measure the quality of the model generated wood texture images and the model computational complexity, we designed relevant ablation experiments. The results show that the MSE, LPIPS, PSNR, and SSIM of the wood images restored by the MRS-Transformer reached 0.0003, 0.154, 40.12, 0.9173, and the GFLOPs is 20.18. Compared with images generated by the Vision Transformer, the MSE and LPIPS were reduced by 51.7% and 30%, PSNR and SSIM were improved by 12.2% and 7.5%, and the GFLOPs were reduced by 38%. To verify the superiority of MRS-Transformer, we compared the image inpainting algorithms with Deepfill v2 and TG-Net, respectively, in which the MSE was 47.0% and 66.9% lower; the LPIPS was 60.6% and 42.5% lower; the FLOPs was 70.6% and 53.5% lower; the PSNR was 16.1% and 26.2% higher; and the SSIM was 7.3% and 5.8% higher. MRS-Transformer repairs a single image in 0.05 s, nearly five times faster than Deepfill v2 and TG-Net. The experimental results demonstrate that the RSwin module effectively alleviates the sense of fragmentation caused by the division of images into patches, the proposed weighted L2 loss improves the semantic consistency of the edges of the missing regions and makes the generated wood texture more detailed and coherent, and the adopted asymmetric encoder-decoder effectively reduces the computational effort of the model and speeds up the training.

Localization of Inpainting Forgery With Feature Enhancement Network

Inpainting the given region of an image is a typical requirement in computer vision. Conventional inpainting, through exemplar-based or diffusion-based strategies, can create realistic inpainted images at a very low cost. Also, such easy-to-use manipulation poses new security threats. Therefore, the detection of inpainting has attracted considerable attention from researchers. However, the existing methods are typically not suitable for the general detection of various inpainting algorithms. Motivated by this, in this work, an efficient feature enhancement network is proposed to locate the inpainted regions in the digital image. Firstly, we design an artifact enhancement block to effectively capture the traces left by diffusion or exemplar-based inpainting. Then, the VGGNet is used as a feature extractor to describe advanced and low-resolution features. Finally, to take full advantage of enhanced features, we concatenate the features obtained by the feature extractor and the up-sampling operations. Extensive experimental evaluations, covering benchmarking, ablation, robustness, generalization, and efficiency studies, confirm the usefulness of the proposed method. This is especially true on the conventional inpainting dataset, our method obtains an average F1 score 7.63% higher than the second-best method. Theoretical and numerical analyses support the effectiveness of our feature enhancement network in representing the artifacts in inpainted images, exhibiting better potential for real-world forensics than various state-of-the-art strategies.

Structure-guided virtual restoration for defective silk cultural relics

The digital restoration of damaged silk relics is a type of blind restoration problem that relies on limited prior information. Existing image restoration techniques face significant difficulties in repainting missing patterns of both structural and content. Silk cultural relics are prone to decay, mold, moth, fading, pollution, and adhesion due to long-term exposure to external factors such as temperature, humidity, and microorganisms. Thus, restoring cultural relic patterns is crucial for archeological and design industries. Virtual restoration, based on image inpainting technology, can tackle this problem. However, existing inpainting algorithms are not effective in defective areas lacking a structural trend and only relying on prior information. Moreover, the unique and irregular patterns of silk cultural relics pose a further challenge. To address these issues, we propose a structure-guided virtual restoration method. Concretely, to maintain global image consistency, we introduce an adaptive curve fitting algorithm to reconstruct missing structural lines. We then establish a new priority function to improve the filling order of patches to be repaired, and an adaptive selection of the sample block size is adopted based on the sparsity of the structure. Our method outperforms existing techniques by reconstructing the structural lines according to the pattern design, thus guiding image inpainting to avoid erroneous fillings and block effects. Results show that the restored images achieved an average SSIM of 0.977 and an average PSNR of 39.16. The proposed method for virtual restoration, utilizing curve fitting and image inpainting guided by the artifact's structure, has demonstrated significant improvements in the preservation of structure continuity and faithful reproduction of original patterns on the silk cultural relics.

DARGS: Image inpainting algorithm via deep attention residuals group and semantics

To solving the problems that the existing image inpainting methods lack authenticity, do not deal with the information of missing and non-missing regions flexibly, and do not deal with the image feature information on different stages effectively, we propose an image restoration method combining Semantic Priors and Deep Attention Residual Group. The image restoration method mainly consists of Semantic Priors Network, Deep Attention Residual Group, and Full-scale Skip Connection. The Semantic Priors Network learns the complete semantic prior information of the visual elements in the missing regions and completes the missing regions with the learned semantic information. The deep attention residual set allows the generator to focus more on not only the missing regions of the image but also to learn the features of each channel adaptive. The full-scale jump joins can combine the low-level feature maps containing image boundaries with the high-level feature maps containing image textures and details to repair the missing regions. In this paper, a full comparison experiment is conducted on the CelebA-HQ and the Paris StreetView datasets, and the experimental results show that proposed method outperforms the current representative state-of-the-art image restoration methods.

Image inpainting using non-convex low rank decomposition and multidirectional search

Low-rank (LR) and nonlocal self-similarity (NSS) are two important priors for image inpainting as a typical inverse problem. Nuclear norm minimization (NNM) is a widely used convex relaxation for relevant rank minimization problems. However, NNM regularizes each singular value equally and ignores the significance of bigger singular values. In this paper, we propose a non-convex low-rank decomposition (NC-LRD) model that is based on robust principal component analysis (RPCA) with a weighted L1 norm. Utilizing NSS prior for image inpainting we search similar patches by using a newly designed multidirectional search (MS) method, and apply the NC-LRD model to complete each corrupted patch matrix (low-rank decomposition with multidirectional search, MS-LRD). We focus on the spatial distribution of similar patches by restricting matched N patches to locate at N different directions relative to a target patch, while previous state-of-the-art methods do not consider the spatial distribution in similarity criterion. The MS method solves the problem that many patch-based inpainting algorithms fail to complete missing lines. Experimental results on line missing demonstrate that the proposed NC-LRD method has lower reconstruction error in matrix completion, and it converges faster than several state-of-the-art matrix completion algorithms. At the same time, the effectiveness and superiority of MS-LRD over other competitive inpainting algorithms are also verified.

Precise Human Removal and Inpainting Using Mask RCNN and LaMa

Sometimes people are not supposed to be in a photo for various purposes, but this is usually unavoidable. Therefore, in the postprocessing of the image, it can be solved by removing people from the picture without affecting the coherence and naturalness of the object and background in the photo. We propose a human removal method based on image instance segmentation and image inpainting. Firstly, we send an image into the image instance segmentation algorithm to obtain a mask covering the unwanted parts of the picture. Then we do dilatation on this mask to expand the mask region. Finally, the inpainting algorithm will take the image and the processed mask and produce an inpainted image with no human and feel natural.

Recovering Microscopic Images in Material Science Documents by Image Inpainting

Microscopic images in material science documents have increased in number due to the growth and common use of electron microscopy instruments. Through the use of data mining techniques, they are easily accessible and can be obtained from documents published online. As data-driven approaches are becoming increasingly common in the material science field, massively acquired experimental images through microscopy play important roles in terms of developing an artificial intelligence (AI) model for the purposes of automatically diagnosing crucial material structures. However, irrelevant objects (e.g., letters, scale bars, and arrows) that are often present inside original microscopic photos should be removed for the purposes of improving the AI models. To avoid the issue above, we applied four image inpainting algorithms (i.e., shift-net, global and local, contextual attention, and gated convolution) to a learning approach, with the aim of recovering microscopic images in journal papers. We estimated the structural similarity index measure (SSIM) and ℓ1/ℓ2 errors, which are often used as measures of image quality. Lastly, we observed that gated convolution possessed the best performance for inpainting the microscopic images.

Research on Image Inpainting Algorithms Based on Attention Guidance

In recent years, the use of deep learning in image inpainting has yielded positive results. However, existing image inpainting algorithms do not pay sufficient attention to the structural and textural features of the image when inpainting, which leads to issues in the inpainting results such as blurring and distortion. To solve the above problems, a channel attention mechanism was introduced to emphasize the importance of structure and texture after extraction by the convolutional network. A bidirectional gated feature fusion module was employed to exchange and fuse the structural and textural features, ensuring the overall consistency of the image. In addition, the features of the image were better captured by selecting a deformable convolution that can adapt the receptive field to replace the ordinary convolution in the contextual feature aggregation module. This resulted in highly vivid and realistic restoration results with more reasonable details. The experiments showed that, compared with the current mainstream network, the repair results of this algorithm were more realistic, and the superiority of this algorithm was proved by qualitative and quantitative experiments.

FFTI: Image inpainting algorithm via features fusion and two-steps inpainting

In view of the faultiness that the existing image inpainting methods fail to make full use of the complete region to predict the missing region features when the object features are seriously missing, resulting in discontinuous features and fuzzy detail texture of the inpainting results, a fine inpainting method of incomplete image based on features fusion and two-steps inpainting (FFTI) is proposed in this paper. Firstly, the dynamic memory networks (DMN+) are used to fuse the external features and internal features of the incomplete image to generate the incomplete image optimization map. Secondly, a generation countermeasure generative network with gradient penalty constraints is constructed to guide the generator to rough repair the optimized incomplete image and obtain the rough repair map of the target to be repaired. Finally, the coarse repair graph is further optimized by the idea of coherence of relevant features to obtain the final fine repair graph. It is verified by simulation on three image data sets with different complexity, and compared with the existing dominant repair model for visual effect and objective data. The experimental results show that the results of the model repair in this paper are more reasonable in texture structure, better than other models in visual effect and objective data, and the Peak Signal-to-Noise Ratio of the proposed algorithm in the most challenging underwater targe dataset is 27.01, the highest Structural Similarity Index is 0.949.

The advantages of sub-sampling and Inpainting for scanning transmission electron microscopy

Images and spectra obtained from aberration corrected scanning transmission electron microscopes (STEM) are now used routinely to quantify the morphology, structure, composition, chemistry, bonding, and optical/electronic properties of nanostructures, interfaces, and defects in many materials/biological systems. However, obtaining quantitative and reproducible atomic resolution observations from some experiments is actually harder with these ground-breaking instrumental capabilities, as the increase in beam current from using the correctors brings with it the potential for electron beam modification of the specimen during image acquisition. This beam effect is even more acute for in situ STEM observations, where the desired outcome being investigated is a result of a series of complicated transients, all of which can be modified in unknown ways by the electron beam. The aim in developing and applying new methods in STEM is, therefore, to focus on more efficient use of the dose that is supplied to the sample and to extract the most information from each image (or set of images). For STEM (and for that matter, all electron/ion/photon scanning systems), one way to achieve this is by sub-sampling the image and using Inpainting algorithms to reconstruct it. By separating final image quality from overall dose in this way and manipulating the dose distribution to be best for the stability of the sample, images can be acquired both faster and with less beam effects. In this paper, the methodology behind sub-sampling and Inpainting is described, and the potential for Inpainting to be applied to novel real time dynamic experiments will be discussed.

Melanoma classification from dermatoscopy images using knowledge distillation for highly imbalanced data

Melanoma is a deadly malignant skin cancer that generally grows and spreads rapidly. Early detection of melanoma can improve the prognosis of a patient. However, large-scale screening for melanoma is arduous due to human error and the unavailability of trained experts. Accurate automatic melanoma classification from dermoscopy images can help mitigate such issues. However, the classification task is challenging due to class-imbalance, high inter-class, and low intra-class similarity problems. It results in poor sensitivity scores when it comes to the disease classification task. The work proposes a novel knowledge-distilled lightweight Deep-CNN-based framework for melanoma classification to tackle the high inter-class and low intra-class similarity problems. To handle the high class-imbalance problem, the work proposes using Cost-Sensitive Learning with Focal Loss, to achieve better sensitivity scores. As a pre-processing step, an in-painting algorithm is used to remove artifacts from dermoscopy images. New CutOut variants, namely, Sprinkled and microscopic Cutout augmentations, have been employed as regularizers to avoid over-fitting. The robustness of the model has been studied through stratified K-fold cross-validation. Ablation studies with test time augmentation (TTA) and the addition of various noises like salt & pepper, pepper-only, and Gaussian noises have been studied. All the models trained in the work have been evaluated on the SIIM-ISIC Melanoma Classification Challenge - ISIC-2020 dataset. With our EfficientNet-B5 (FL) teacher model, the EfficientNet-B2 student model achieved an Area under the Curve (AUC) of 0.9295, and a sensitivity of 0.8087 on the ISIC-2020 test data. The sensitivity value of 0.8087 for melanoma classification is the current state-of-the-art result in the literature for the ISIC-2020 dataset which is a significant 49.48% increase from the best non-distilled standalone model, EfficientNet B5 (FL) teacher with 0.5410.

Application of hyperspectral imaging technology to digitally protect murals in the Qutan temple

Hyperspectral imaging technology is a research hotspot in the field of cultural heritage protection. It can be used to quickly and noninvasively obtain detailed spectral information from the surfaces of cultural relics of different categories. We can intuitively analyse pigment compositions, line characteristics, painting skills and patterns using spectral information. Hyperspectral imaging has high scientific significance and application value for the protection, restoration and research of ancient murals and other cultural relics. In this study, a mural from Daheitian hall in the Qutan temple, Qinghai Province, China, was used as a sample. The hyperspectral data were acquired and analysed for several purposes. Pigment spectral matching and abundance inversion were carried out to obtain the pigment distribution. These data were enhanced by continuum removal and histogram stretching to obtain hidden information. The dark channel prior, Criminisi and Retinex methods were used to virtually restore the image of the mural. The results indicated that by using hyperspectral imaging data, the constructed pure pigment spectrum library and suitable approaches, the types and distributions of mural pigments can be quantitatively analysed, and the lines in murals can be extracted. Hyperspectral images are helpful for identifying information hidden by pigments or surface materials. Mural images can be enhanced, and hidden information can be highlighted using enhancement methods, such as continuum removal and histogram linear stretching. In addition, hyperspectral imaging data have unique advantages in the restoration of mural images, and the combination of defogging methods and image inpainting algorithms can realize the virtual restoration of mural images. In brief, hyperspectral imaging technology was found to have a highly favourable effect on pigment analysis, line extraction, information enhancement, hidden information extraction and the virtual restoration of ancient murals.

Progressive With Purpose: Guiding Progressive Inpainting DNNs Through Context and Structure

The advent of deep learning in the past decade has significantly helped advance image inpainting. Although achieving promising performance, deep learning-based inpainting algorithms still struggle from the distortion caused by the fusion of structural and contextual features, which are commonly obtained from, respectively, deep and shallow layers of a convolutional encoder. Motivated by this observation, we propose a novel progressive inpainting network that maintains the structural and contextual integrity of a processed image. More specifically, inspired by the Gaussian and Laplacian pyramids, the core of the proposed network is a feature extraction module named GLE. Stacking GLE modules enables the network to extract image features from different image frequency components. This ability is important to maintain structural and contextual integrity, for high frequency components correspond to structural information while low frequency components correspond to contextual information. The proposed network utilizes the GLE features to progressively fill in missing regions in a corrupted image in an iterative manner. Our benchmarking experiments demonstrate that the proposed method achieves clear improvement in performance over many state-of-the-art inpainting algorithms.

Algorithms for audio inpainting based on probabilistic nonnegative matrix factorization

Audio inpainting, i.e., the task of restoring missing or occluded audio signal samples, usually relies on sparse representations or autoregressive modeling. In this paper, we propose to structure the spectrogram with nonnegative matrix factorization (NMF) in a probabilistic framework. First, we treat the missing samples as latent variables, and derive two expectation–maximization algorithms for estimating the parameters of the model, depending on whether we formulate the problem in the time- or time-frequency domain. Then, we treat the missing samples as parameters, and we address this novel problem by deriving an alternating minimization scheme. We assess the potential of these algorithms for the task of restoring short- to middle-length gaps in music signals. Experiments reveal great convergence properties of the proposed methods, as well as competitive performance when compared to state-of-the-art audio inpainting techniques.

A comparison of deep-learning-based inpainting techniques for experimental X-ray scattering.

The implementation is proposed of image inpainting techniques for the reconstruction of gaps in experimental X-ray scattering data. The proposed methods use deep learning neural network architectures, such as convolutional autoencoders, tunable U-Nets, partial convolution neural networks and mixed-scale dense networks, to reconstruct the missing information in experimental scattering images. In particular, the recovered pixel intensities are evaluated against their corresponding ground-truth values using the mean absolute error and the correlation coefficient metrics. The results demonstrate that the proposed methods achieve better performance than traditional inpainting algorithms such as biharmonic functions. Overall, tunable U-Net and mixed-scale dense network architectures achieved the best reconstruction performance among all the tested algorithms, with correlation coefficient scores greater than 0.9980.