Inpainting Algorithm Research Articles

CMB-PAInT: An inpainting tool for the cosmic microwave background

The presence of astrophysical emissions in microwave observations forces us to perform component separation to extract the Cosmic Microwave Background (CMB) signal. However, even in the most optimistic cases, there are still strongly contaminated regions, such as the Galactic plane or those with emission from extragalactic point sources, which require the use of a mask. Since many CMB analyses, especially the ones working in harmonic space, need the whole sky map, it is crucial to develop a reliable inpainting algorithm that replaces the values of the excluded pixels by others statistically compatible with the rest of the sky. This is especially important when working with Q and U sky maps in order to obtain E- and B-mode maps which are free from E-to-B leakage. In this work we study a method based on Gaussian Constrained Realizations (GCR), that can deal with both intensity and polarization. Several tests have been performed to asses the validation of the method, including the study of the one-dimensional probability distribution function (1-PDF), E- and B-mode map reconstruction, and power spectra estimation. We have considered two scenarios for the input simulation: one case with only CMB signal and a second one including also Planck PR4 semi-realistic noise. Even if we are limited to low resolution maps, Nside = 64 if T, Q and U are considered, we believe that this is a useful approach to be applied to future missions such as LiteBIRD, where the target are the largest scales.

A text guided cross modal joint inpainting algorithm for ancient murals

Existing deep learning algorithms for ancient mural inpainting only consider prior information of mural images to guide the inpainting of damaged areas, lacking guidance from textual semantic information, which often leads to problems such as inconsistent semantics and disordered textures in the inpainting results. In this work, we propose a text guided cross modal joint inpainting algorithm for ancient murals. Firstly, a cross modal joint inpainting network for ancient murals was constructed, which introduced text semantic information as the control guidance for mural inpainting, providing contextual information for mural image inpainting and guiding the inpainting of damaged areas. Then, design a text encoder and an image encoder to extract text and image features respectively, including word and sentence features. Input the sentence features into the conditional enhancement module, resample the text sentences, and solve the problem of text sparsity. Secondly, design a text decoupling module that utilizes a multimodal attention mechanism to extract the semantic information of the missing parts of damaged murals, and design a text semantic enhancement module to improve the semantic representation ability of the text, achieve the repair of local details of murals, and solve the problem of texture disorder. Finally, the reconstruction and repair of murals are completed through a residual discriminator game and a spectral normalized Markov dual discriminator against the mural generation network. Through inpainting experiments on real Dunhuang murals, it has been shown that the proposed method is more effective than comparative methods in repairing damaged murals, achieving better visual perception and coordination in reconstruction, and outperforming comparative algorithms in both subjective and objective evaluations.

Context-Encoder-Based Image Inpainting for Ancient Chinese Silk

The rapid advancement of deep learning technologies presents novel opportunities for restoring damaged patterns in ancient silk, which is pivotal for the preservation and propagation of ancient silk culture. This study systematically scrutinizes the evolutionary trajectory of image inpainting algorithms, with a particular emphasis on those firmly rooted in the Context-Encoder structure. To achieve this study’s objectives, a meticulously curated dataset comprising 6996 samples of ancient Chinese silk (256 × 256 pixels) was employed. Context-Encoder-based image inpainting models—LISK, MADF, and MEDFE—were employed to inpaint damaged patterns. The ensuing restoration effects underwent rigorous evaluation, providing a comprehensive analysis of the inherent strengths and limitations of each model. This study not only provides a theoretical foundation for adopting image restoration algorithms grounded in the Context-Encoder structure but also offers ample scope for exploration in achieving more effective restorations of ancient damaged silk.

Underwater image restoration based on progressive guidance

Underwater images often suffer from local distortions during the imaging and transmission process, which can negatively impact their quality. Fortunately, it is possible to improve image quality by removing local distortion without making any hardware or software adjustments to the transmission system. However, existing algorithms designed for global distortions are not suitable for addressing local distortions, while end-to-end restoration and inpainting algorithms do not perform satisfactorily on underwater images. To address this issue, this paper proposes a Joint distortion localization and restoration model based on Progressive Guidance (JPG) specifically tailored for underwater imaging and transmission. Our strategy employs a two-stage framework where the first stage focuses exclusively on accurately localizing distortions to obtain precise position. Subsequently, in the second stage, we utilize this position information for effective distortion restoration. To further enhance restoration performance, our approach progressively guides the restoration process by incorporating global, distortion-free as well as distortion-specific information into different components of the second-stage network. The work surpasses current state-of-the-art methods in restoring both mixed and individual distortions.

DSOMF: A Dynamic Environment Simultaneous Localization and Mapping Technique Based on Machine Learning.

To address the challenges of reduced localization accuracy and incomplete map construction demonstrated using classical semantic simultaneous localization and mapping (SLAM) algorithms in dynamic environments, this study introduces a dynamic scene SLAM technique that builds upon direct sparse odometry (DSO) and incorporates instance segmentation and video completion algorithms. While prioritizing the algorithm's real-time performance, we leverage the rapid matching capabilities of Direct Sparse Odometry (DSO) to link identical dynamic objects in consecutive frames. This association is achieved through merging semantic and geometric data, thereby enhancing the matching accuracy during image tracking through the inclusion of semantic probability. Furthermore, we incorporate a loop closure module based on video inpainting algorithms into our mapping thread. This allows our algorithm to rely on the completed static background for loop closure detection, further enhancing the localization accuracy of our algorithm. The efficacy of this approach is validated using the TUM and KITTI public datasets and the unmanned platform experiment. Experimental results show that, in various dynamic scenes, our method achieves an improvement exceeding 85% in terms of localization accuracy compared with the DSO system.

Terrain point cloud inpainting via signal decomposition

The rapid development of 3D acquisition technology has made it possible to obtain point clouds of real-world terrains. However, due to limitations in sensor acquisition technology or specific requirements, point clouds often contain defects such as holes with missing data. Inpainting algorithms are widely used to patch these holes. However, existing traditional inpainting algorithms rely on precise hole boundaries, which limits their ability to handle cases where the boundaries are not well-defined. On the other hand, learning-based completion methods often prioritize reconstructing the entire point cloud instead of solely focusing on hole filling. Based on the fact that real-world terrain exhibits both global smoothness and rich local detail, we propose a novel representation for terrain point clouds. This representation can help to repair the holes without clear boundaries. Specifically, it decomposes terrains into low-frequency and high-frequency components, which are represented by B-spline surfaces and relative height maps respectively. In this way, the terrain point cloud inpainting problem is transformed into a B-spline surface fitting and 2D image inpainting problem. By solving the two problems, the highly complex and irregular holes on the terrain point clouds can be well-filled, which not only satisfies the global terrain undulation but also exhibits rich geometric details. The experimental results also demonstrate the effectiveness of our method.

DNNAM: Image inpainting algorithm via deep neural networks and attention mechanism

Most image inpainting algorithms have problems such as fuzzy images, texture distortion and semantic inaccuracy, and the image inpainting effect is limited when processing photos with large missing sections and resolution levels. The paper proposes an effective image inpainting algorithm via partial multi-scale channel attention mechanism and deep neural networks to address the above phenomenon that existing image inpainting methods using deep learning modules have insufficient perception and representation capabilities for multi-scale features with high proportion of irregular defects. Initially, we used the Res-U-Net module as a generator. The U-Net-like backbone network topology can achieve the encoding and decoding stages of damaged images. Secondly, the residual network structure was built in the encoder and decoder to improve the ability of the proposed network to extract and display the features of the damaged images. Finally, the partial multi-scale channel attention module was inserted in the skip connection with the decoder to increase the efficiency of using the low-level features of the original images. The experimental results of the research can show that the proposed method outperforms state-of-the-art methods in terms of subjective visual perception and objective evaluation indicators on the CelebA, Places2 and Paris Street View datasets.

The Potential of Subsampling and Inpainting for Fast Low-Dose Cryo FIB-SEM Imaging.

Traditional image acquisition for cryo focused ion-beam scanning electron microscopy (FIB-SEM) tomography often sees thousands of images being captured over a period of many hours, with immense data sets being produced. When imaging beam sensitive materials, these images are often compromised by additional constraints related to beam damage and the devitrification of the material during imaging, which renders data acquisition both costly and unreliable. Subsampling and inpainting are proposed as solutions for both of these aspects, allowing fast and low-dose imaging to take place in the Focused ion-beam scanning electron microscopy FIB-SEM without an appreciable loss in image quality. In this work, experimental data are presented which validate subsampling and inpainting as a useful tool for convenient and reliable data acquisition in a FIB-SEM, with new methods of handling three-dimensional data being employed in the context of dictionary learning and inpainting algorithms using a newly developed microscope control software and data recovery algorithm.

Image Inpainting Forgery Detection: A Review.

In recent years, significant advancements in the field of machine learning have influenced the domain of image restoration. While these technological advancements present prospects for improving the quality of images, they also present difficulties, particularly the proliferation of manipulated or counterfeit multimedia information on the internet. The objective of this paper is to provide a comprehensive review of existing inpainting algorithms and forgery detections, with a specific emphasis on techniques that are designed for the purpose of removing objects from digital images. In this study, we will examine various techniques encompassing conventional texture synthesis methods as well as those based on neural networks. Furthermore, we will present the artifacts frequently introduced by the inpainting procedure and assess the state-of-the-art technology for detecting such modifications. Lastly, we shall look at the available datasets and how the methods compare with each other. Having covered all the above, the outcome of this study is to provide a comprehensive perspective on the abilities and constraints of detecting object removal via the inpainting procedure in images.

Deep video inpainting detection and localization based on ConvNeXt dual-stream network

Currently, deep learning-based video inpainting algorithms can fill in a specified video region with visually plausible content, usually leaving imperceptible traces. Since deep video inpainting methods can be used to maliciously manipulate video content, there is an urgent need for an effective method to detect and localize deep video inpainting. In this paper, we propose a dual-stream video inpainting detection network, which includes a ConvNeXt dual-stream encoder and a multi-scale feature cross-fusion decoder. To further explore the spatial and temporal traces left by deep inpainting, we extract motion residuals and enhance them using 3D convolution and SRM filtering. Furthermore, we extract filtered residuals using LoG and Laplacian filtering. These residuals are then entered into ConvNeXt, thereby learning discriminative inpainting features. To enhance detection accuracy, we design a top-down pyramid decoder that aims at deep fusion of multi-dimensional multi-scale features to fully exploit the information of different dimensions and levels in detail. We created two datasets containing state-of-the-art video inpainting algorithms and conducted various experiments to evaluate our approach. The experimental results demonstrate that our approach outperforms existing methods and attains a competitive performance despite encountering unseen inpainting algorithms.

VEHICLE OCCLUSION REMOVAL FROM SINGLE AERIAL IMAGES USING GENERATIVE ADVERSARIAL NETWORKS

Abstract. Removing occluding objects such as vehicles from drivable areas allows precise extraction of road boundaries and related semantic objects such as lane-markings, which is crucial for several applications such as generating high-definition maps for autonomous driving. Conventionally, multiple images of the same area taken at different times or from various perspectives are used to remove occlusions and to reconstruct the occluded areas. Nevertheless, these approaches require large amounts of data, which are not always available. Furthermore, they do not work for static occlusions caused by, among others, parked vehicles. In this paper, we address occlusion removal based on single aerial images using generative adversarial networks (GANs), which are able to deal with the mentioned challenges. To this end, we adapt several state-of-the-art GAN-based image inpainting algorithms to reconstruct the missing information. Results indicate that the StructureFlow algorithm outperforms the competitors and the restorations obtained are robust, with high visual fidelity in real-world applications. Furthermore, due to the lack of annotated aerial vehicle removal datasets, we generate a new dataset for training and validating the algorithms, the Aerial Vehicle Occlusion Removal (AVOR) dataset. To the best of our knowledge, our work is the first to address vehicle removal using deep learning algorithms to enhance maps.

Coarse-to-Fine Task-Driven Inpainting for Geoscience Images

The processing and recognition of geoscience images have wide applications. Most of existing researches focus on understanding the high-quality geoscience images by assuming that all the images are clear. However, in many real-world cases, the geoscience images might contain occlusions during the image acquisition. This problem actually implies the image inpainting problem in computer vision and multimedia. As far as we know, all the existing image inpainting algorithms learn to repair the occluded regions for a better visualization quality, they are excellent for natural images but not good enough for geoscience images, and they never consider the following gescience task when developing inpainting methods. This paper aims to repair the occluded regions for a better geoscience task performance and advanced visualization quality simultaneously, without changing the current deployed deep learning based geoscience models. Because of the complex context of geoscience images, we propose a coarse-to-fine encoder-decoder network with the help of designed coarse-to-fine adversarial context discriminators to reconstruct the occluded image regions. Due to the limited data of geoscience images, we propose a MaskMix based data augmentation method, which augments inpainting masks instead of augmenting original images, to exploit the limited geoscience image data. The experimental results on three public geoscience datasets for remote sensing scene recognition, cross-view geolocation and semantic segmentation tasks respectively show the effectiveness and accuracy of the proposed method. The code is available at: https://github.com/HMS97/Task-driven-Inpainting.

Conditional generative adversarial networks for stripe artefact removal in high-resolution X-ray tomography

Tomographic imaging supports a great number of medical and material science applications. The collected projection data usually has different types of imaging artefacts and noise. Various image pre-processing and reconstruction methods are used to obtain volumetric datasets of high quality for further analysis. In order to minimise reconstruction artefacts, one can apply either filtering and/or data completion/inpainting techniques which can recover the data. Deep learning (DL) methods to remove artefacts and noise have been successfully applied in the past. In this paper, we present a novel approach based on conditional generative adversarial networks (cGANs) to remove stripe artefacts. The novelty of the presented technique is in how the training data for DL is extracted from the same tomographic dataset that needs recovery. We also provide new deterministic stripe detection and inpainting algorithms to support the development. The presented methods are compared with other stripe removal algorithms and applied to 3D and 4D high-resolution X-ray data collected at Diamond Light Source synchrotron, UK. The proposed DL method delivers reconstructed images with minimised ring artefacts while being a parameter-free approach. A similar DL strategy can also be applied to remove other types of artefacts in images.

MFMAM: Image inpainting via multi-scale feature module with attention module

Currently, most image inpainting algorithms based on deep learning cause information loss when acquiring deep level features. It is not conducive to the image inpainting of texture details and often ignore the image restoration of semantic features, which generates image inpainting results with unreasonable structures. To address those problems, we have proposed improved image inpainting network using multi-scale feature module and improved attention module. Firstly, we propose the multi-scale fusion module based on dilated convolution to reduce information loss during the convolution process by fusing multi-scale features when acquiring image deep level features. Then, the attention module can strengthen the ability of image semantic inpainting and ensure that the proposed model can generate clear texture inpainting results. To ensure the consistency of image inpainting details and styles, the style loss and perceptual loss functions are introduced in the proposed network. The qualitative experimental results on CelebA-HQ, Places2 and Outdoor Scene datasets and common evaluation metrics such as PSNR, SSIM and FID. These metrics can validate the proposed method to be superior to the state-of-the-arts.

Physics-informed neural networks in the recreation of hydrodynamic simulations from dark matter

ABSTRACT Physics-informed neural networks have emerged as a coherent framework for building predictive models that combine statistical patterns with domain knowledge. The underlying notion is to enrich the optimization loss function with known relationships to constrain the space of possible solutions. Hydrodynamic simulations are a core constituent of modern cosmology, while the required computations are both expensive and time-consuming. At the same time, the comparatively fast simulation of dark matter requires fewer resources, which has led to the emergence of machine learning algorithms for baryon inpainting as an active area of research; here, recreating the scatter found in hydrodynamic simulations is an ongoing challenge. This paper presents the first application of physics-informed neural networks to baryon inpainting by combining advances in neural network architectures with physical constraints, injecting theory on baryon conversion efficiency into the model loss function. We also introduce a punitive prediction comparison based on the Kullback–Leibler divergence, which enforces scatter reproduction. By simultaneously extracting the complete set of baryonic properties for the simba suite of cosmological simulations, our results demonstrate improved accuracy of baryonic predictions based on dark matter halo properties and successful recovery of the fundamental metallicity relation, and retrieve scatter that traces the target simulation’s distribution.

Exemplar-Based Inpainting for 6DOF Virtual Reality Photos.

Multi-layer images are currently the most prominent scene representation for viewing natural scenes under full-motion parallax in virtual reality. Layers ordered in diopter space contain color and transparency so that a complete image is formed when the layers are composited in a view-dependent manner. Once baked, the same limitations apply to multi-layer images as to conventional single-layer photography, making it challenging to remove obstructive objects or otherwise edit the content. Object removal before baking can benefit from filling disoccluded layers with pixels from background layers. However, if no such background pixels have been observed, an inpainting algorithm must fill the empty spots with fitting synthetic content. We present and study a multi-layer inpainting approach that addresses this problem in two stages: First, a volumetric area of interest specified by the user is classified with respect to whether the background pixels have been observed or not. Second, the unobserved pixels are filled with multi-layer inpainting. We report on experiments using multiple variants of multi-layer inpainting and compare our solution to conventional inpainting methods that consider each layer individually.

A novel color image tampering detection and self-recovery based on fragile watermarking

In industrial applications, sensing devices' image data must be accurately identified and recovered to prevent economic losses. Nevertheless, most industrial image watermarking methods focus on grayscale images with difficulty in achieving superior performance in three parameters: watermarked image quality, tamper detection and content recovery quality. To resolve this issue, a color image tampering detection and self-recovery method based fragile watermarking scheme is proposed in this paper. In the proposed scheme, block-based regular marker with pixel-based continuous marker is used for watermark embedding to enhance the quality of watermarked images. Then, a feature extraction-based tampering detection scheme for diagonal blocks combined with three-layer authentication is developed to resist tampering attacks of diverse shapes and sizes. Additionally, the block pixel-level recovery mechanism along with the Smoothing Inpainting algorithm is designed to implement the high-quality recovery of tampered images. Compared with recent methods, experimental results demonstrate the superiority of the scheme on tamper detection and image self-recovery, in terms of the image recovery robustness, tamper localization and attack resistance. Our scheme has a PSNR value of 25.4921 dB and an SSIM value of 0.7728 for the recovered image at 90% tampering rate.

A Review of Image Inpainting Methods Based on Deep Learning

Image Inpainting is an age-old image processing problem, with people from different eras attempting to solve it using various methods. Traditional image inpainting algorithms have the ability to repair minor damage such as scratches and wear. However, with the rapid development of deep learning in the field of computer vision in recent years, coupled with abundant computing resources, methods based on deep learning have increasingly highlighted their advantages in semantic feature extraction, image transformation, and image generation. As such, image inpainting algorithms based on deep learning have become the mainstream in this domain.In this article, we first provide a comprehensive review of some classic deep-learning-based methods in the image inpainting field. Then, we categorize these methods based on component optimization, network structure design optimization, and training method optimization, discussing the advantages and disadvantages of each approach. A comparison is also made based on public datasets and evaluation metrics in image inpainting. Furthermore, the article delves into the applications of current image inpainting technologies, categorizing them into three major scenarios: object removal, general image repair, and facial inpainting. Finally, current challenges and prospective developments in the field of image inpainting are discussed.

Group sparse optimization for inpainting of random fields on the sphere

Abstract We propose a group sparse optimization model for inpainting of a square-integrable isotropic random field on the unit sphere, where the field is represented by spherical harmonics with random complex coefficients. In the proposed optimization model, the variable is an infinite-dimensional complex vector and the objective function is a real-valued function defined by a hybrid of the $\ell _2$ norm and non-Lipschitz $\ell _p (0<p<1)$ norm that preserves rotational invariance property and group structure of the random complex coefficients. We show that the infinite-dimensional optimization problem is equivalent to a convexly-constrained finite-dimensional optimization problem. Moreover, we propose a smoothing penalty algorithm to solve the finite-dimensional problem via unconstrained optimization problems. We provide an approximation error bound of the inpainted random field defined by a scaled Karush–Kuhn–Tucker (KKT) point of the constrained optimization problem in the square-integrable space on the sphere with probability measure. Finally, we conduct numerical experiments on band-limited random fields on the sphere and images from Cosmic Microwave Background (CMB) data to show the promising performance of the smoothing penalty algorithm for inpainting of random fields on the sphere.

Inpainting surgical occlusion from laparoscopic video sequences for robot-assisted interventions.

Medical technology for minimally invasive surgery has undergone a paradigm shift with the introduction of robot-assisted surgery. However, it is very difficult to track the position of the surgical tools in a surgical scene, so it is crucial to accurately detect and identify surgical tools. This task can be aided by deep learning-based semantic segmentation of surgical video frames. Furthermore, due to the limited working and viewing areas of these surgical instruments, there is a higher chance of complications from tissue injuries (e.g., tissue scars and tears). With the aid of digital inpainting algorithms, we present an application that uses image segmentation to remove surgical instruments from laparoscopic/endoscopic video. We employ a modified U-Net architecture (U-NetPlus) to segment the surgical instruments. It consists of a redesigned decoder and a pre-trained VGG11 or VGG16 encoder. The decoder was modified by substituting an up-sampling operation based on nearest-neighbor interpolation for the transposed convolution operation. Furthermore, these interpolation weights do not need to be learned to perform upsampling, which eliminates the artifacts generated by the transposed convolution. In addition, we use a very fast and adaptable data augmentation technique to further enhance performance. The instrument segmentation mask is filled in (i.e., inpainted) by the tool removal algorithms using the previously acquired tool segmentation masks and either previous instrument-containing frames or instrument-free reference frames. We have shown the effectiveness of the proposed surgical tool segmentation/removal algorithms on a robotic instrument dataset from the MICCAI 2015 and 2017 EndoVis Challenge. We report a 90.20% DICE for binary segmentation, a 76.26% DICE for instrument part segmentation, and a 46.07% DICE for instrument type (i.e., all instruments) segmentation on the MICCAI 2017 challenge dataset using our U-NetPlus architecture, outperforming the results of earlier techniques used and tested on these data. In addition, we demonstrated the successful execution of the tool removal algorithm from surgical tool-free videos that contained moving surgical tools that were generated artificially. Our application successfully separates and eliminates the surgical tool to reveal a view of the background tissue that was otherwise hidden by the tool, producing results that are visually similar to the actual data.