Digital Inpainting Research Articles

Restoration of Automotive Radar Signals Under Mutual Interference by Digital In-Painting

Restoration of Automotive Radar Signals Under Mutual Interference by Digital In-Painting

Inpainting surgical occlusion from laparoscopic video sequences for robot-assisted interventions.

Medical technology for minimally invasive surgery has undergone a paradigm shift with the introduction of robot-assisted surgery. However, it is very difficult to track the position of the surgical tools in a surgical scene, so it is crucial to accurately detect and identify surgical tools. This task can be aided by deep learning-based semantic segmentation of surgical video frames. Furthermore, due to the limited working and viewing areas of these surgical instruments, there is a higher chance of complications from tissue injuries (e.g., tissue scars and tears). With the aid of digital inpainting algorithms, we present an application that uses image segmentation to remove surgical instruments from laparoscopic/endoscopic video. We employ a modified U-Net architecture (U-NetPlus) to segment the surgical instruments. It consists of a redesigned decoder and a pre-trained VGG11 or VGG16 encoder. The decoder was modified by substituting an up-sampling operation based on nearest-neighbor interpolation for the transposed convolution operation. Furthermore, these interpolation weights do not need to be learned to perform upsampling, which eliminates the artifacts generated by the transposed convolution. In addition, we use a very fast and adaptable data augmentation technique to further enhance performance. The instrument segmentation mask is filled in (i.e., inpainted) by the tool removal algorithms using the previously acquired tool segmentation masks and either previous instrument-containing frames or instrument-free reference frames. We have shown the effectiveness of the proposed surgical tool segmentation/removal algorithms on a robotic instrument dataset from the MICCAI 2015 and 2017 EndoVis Challenge. We report a 90.20% DICE for binary segmentation, a 76.26% DICE for instrument part segmentation, and a 46.07% DICE for instrument type (i.e., all instruments) segmentation on the MICCAI 2017 challenge dataset using our U-NetPlus architecture, outperforming the results of earlier techniques used and tested on these data. In addition, we demonstrated the successful execution of the tool removal algorithm from surgical tool-free videos that contained moving surgical tools that were generated artificially. Our application successfully separates and eliminates the surgical tool to reveal a view of the background tissue that was otherwise hidden by the tool, producing results that are visually similar to the actual data.

Digital inpainting of mural images based on DC-CycleGAN

Located in Dunhuang, northwest China, the Mogao Grottoes are a cultural treasure of China and the world. However, after more than 2000 years of weathering and destruction, many murals faded and were damaged. This treasure of human art is in danger. Mural inpainting through deep learning can permanently preserve mural information. Therefore, a digital restoration method combining the Deformable Convolution (DCN), ECANet, ResNet and Cycle Generative Adversarial Network (CycleGAN) is proposed. We name it DC-CycleGAN. Compared with other image digital inpainting methods, the proposed DC-CycleGAN based mural image color inpainting method has better inpainting effects and higher model performance. Compared with the current repair network, the Frechet Inception Distance (FID) value and the two-image structural similarity metric (SSIM) value are increased by 52.61% and 7.08%, respectively. Image color inpainting of Dunhuang murals can not only protect and inherit Chinese culture, but also promote academic research and development in related fields.

Fragments Inpainting for Tomb Murals Using a Dual-Attention Mechanism GAN with Improved Generators

As the only underground mural in the collection, the tomb murals are subject to damage due to temperature, humidity, and foundation settlement changes. Traditional mural inpainting takes a long time and requires experts to draw it manually. Therefore, the need for digital inpainting is increasing to save time and costs. Due to the scarcity of samples and the variety of damage, the image features are scattered and partially sparse, and the colors are less vivid than in other images. Traditional deep learning inpainting causes information loss and generates irrational structures. The generative adversarial network is, recently, a more effective method. Therefore, this paper presents an inpainting model based on dual-attention multiscale feature aggregation and an improved generator. Firstly, an improved residual prior and attention mechanism is added to the generator module to preserve the image structure. Secondly, the model combines spatial and channel attention with multiscale feature aggregation to change the mapping network structure and improve the inpainting accuracy. Finally, the segmental loss function and its training method are improved.The experimental results show that the results of using signal-to-noise ratio (PSNR), structural similarity (SSIM), and mean square error (MSE) on epitaxial mask, crack mask, random small mask, and random large mask are better than other methods. It demonstrates the performance of this paper in inpainting different diseases of murals. It can be used as a reference for experts in manual inpainting, saving the cost and time of manual inpainting.

Virtual restoration of paintings using adaptive adversarial neural network

Cracks (craquelure) and paint losses are the main types of deterioration of master paintings as they are ageing. We explore the potential of deep-learning-based methods for virtual restoration of paintings focusing on crack detection and their digital inpainting. For the crack detection stage, we develop a model that combines the benefits of multimodal convolutional (MCN) and autoencoder neural networks based on U-Net. The proposed model, dubbed U-Net multimodal convolutional, proves to outperform both MCN and U-Net architectures as well as the benchmark machine learning models for multimodal crack detection, both visually and in terms of objective performance measures. The second stage in our virtual restoration framework, the digital crack inpainting, is an adaptive adversarial network. The obtained virtual restoration results show clear improvement in comparison with the reference methods in this domain. Also, we propose an original way of training an adversarial neural network, which allows us to apply it more successfully in practice. A series of experiments shows encouraging results compared to the current state-of-the-art and confirms the huge potential of deep learning in crack detection and virtual restoration of master paintings.

A deep neural network based framework for restoring the damaged persian pottery via digital inpainting

Conservation of the historical artworks involves carefully planning interventions to prolong the material, historical, and design integrity of humanity's-built heritage. Among which, restoration of the destroyed parts of the artworks via inpainting is one of the most critical stages in the restorative conservation.Indeed, one of the significant assets of any art-historical work is its aesthetic aspects. However, usually due to the visual complexity and the diversity of the motifs, the number of patterns that a restorer artist can have in mind when repairing them, are limited. Also, the inpainting process of such artworks might be very slow and prone to errors; a slight error can cause severe mistakes for the aesthetic aspects of the artifact. Therefore, the need to develop an intelligent system to provide an image of the refurbished artifact before to the restoration operation, is undeniable. Motivated by the idea that the computer-vision-based image inpainting process could be an efficient solution for this purpose, in this paper, for the first time, we propose a framework that employs a deep artificial neural network-based approach to inpaint the images of Persian pottery for restoring their damaged parts. The framework can repair irregular structures that usually exist in Persian pottery and inpaints relatively large missing parts. Besides, the restored parts were seamlessly integrated with the rest of the image without the need for any further postprocessing. To evaluate the proposed framework, we have collected 677 images from different Kubachi ware (known as Persian pottery) to study the inpainting results. The provided dataset is available free for the research community. Quantitative and qualitative evaluations of the results show that the proposed framework performs well, where the inpainted parts are visually and semantically plausible, and the details of the predicted colors in the damaged areas are very satisfying.

Colour-Balanced Edge-Guided Digital Inpainting: Applications on Artworks.

The virtual inpainting of artworks provides a nondestructive mode of hypothesis visualization, and it is especially attractive when physical restoration raises too many methodological and ethical concerns. At the same time, in Cultural Heritage applications, the level of details in virtual reconstruction and their accuracy are crucial. We propose an inpainting algorithm that is based on generative adversarial network, with two generators: one for edges and another one for colors. The color generator rebalances chromatically the result by enforcing a loss in the discretized gamut space of the dataset. This way, our method follows the modus operandi of an artist: edges first, then color palette, and, at last, color tones. Moreover, we simulate the stochasticity of the lacunae in artworks with morphological variations of a random walk mask that recreate various degradations, including craquelure. We showcase the performance of our model on a dataset of digital images of wall paintings from the Dunhuang UNESCO heritage site. Our proposals of restored images are visually satisfactory and they are quantitatively comparable to state-of-the-art approaches.

Segmentation and Removal of Surgical Instruments for Background Scene Visualization from Endoscopic / Laparoscopic Video.

Surgical tool segmentation is becoming imperative to provide detailed information during intra-operative execution. These tools can obscure surgeons' dexterity control due to narrow working space and visual field-of-view, which increases the risk of complications resulting from tissue injuries (e.g. tissue scars and tears). This paper demonstrates a novel application of segmenting and removing surgical instruments from laparoscopic/endoscopic video using digital inpainting algorithms. To segment the surgical instruments, we use a modified U-Net architecture (U-NetPlus) composed of a pre-trained VGG11 or VGG16 encoder and redesigned decoder. The decoder is modified by replacing the transposed convolution operation with an up-sampling operation based on nearest-neighbor (NN) interpolation. This modification removes the artifacts generated by the transposed convolution, and, furthermore, these new interpolation weights require no learning for the upsampling operation. The tool removal algorithms use the tool segmentation mask and either the instrument-free reference frames or previous instrument-containing frames to fill-in (i.e., inpaint) the instrument segmentation mask with the background tissue underneath. We have demonstrated the performance of the proposed surgical tool segmentation/removal algorithms on a robotic instrument dataset from the MICCAI 2015 EndoVis Challenge. We also showed successful performance of the tool removal algorithm from synthetically generated surgical instruments-containing videos obtained by embedding a moving surgical tool into surgical tool-free videos. Our application successfully segments and removes the surgical tool to unveil the background tissue view otherwise obstructed by the tool, producing visually comparable results to the ground truth.

Study on Digital Image Inpainting Method Based on Multispectral Image Decomposition Synthesis

The paper analyzes the image inpainting problem of damaged Painting Arts for high fidelity images reproduction, and a digital image inpainting method based on multispectral image decomposition synthesis is proposed. Firstly, multi-channel images of Painting Arts are obtained by multispectral technology. Then, a polynomial regression method based on principal component is used to reconstruct the spectral image. The reconstructed image is decomposed by VO image decomposition model. During the inpainting process, the channel correlation of the structure image and the texture image of multispectral image is effectively removed. The digital image inpainting is performed respectively. Finally, the digital inpainted image is obtained by synthesis. The experimental results show that the digital image inpainting based on multispectral image decomposition synthesis reduces the problem of low image inpainting accuracy caused by the correlation between the color components in the traditional digital image inpainting process, and reduces the mismatch of the inpainting image. Appearance of pseudo color of inpainting image is reduced. MSE of multispectral images inpainting qualities is 2.7951 and PSNR of multispectral images inpainting qualities is 44.1681, so it is superior to traditional image inpainting algorithm. It provides a reliable basis for digital inpainting, digital archives and high fidelity replication of defective Painting Arts.

Using digital inpainting to estimate incident light intensity for the calculation of red blood cell oxygen saturation from microscopy images.

Red blood cell oxygen saturation (SO2 ) is an important indicator of oxygen supply to tissues in the body. SO2 can be measured by taking advantage of spectroscopic properties of hemoglobin. When this technique is applied to transmission microscopy, the calculation of saturation requires determination of incident light intensity at each pixel occupied by the red blood cell; this value is often approximated from a sequence of images as the maximum intensity over time. This method often fails when the red blood cells are moving too slowly, or if hematocrit is too large since there is not a large enough gap between the cells to accurately calculate the incident intensity value. A new method of approximating incident light intensity is proposed using digital inpainting. This novel approach estimates incident light intensity with an average percent error of approximately 3%, which exceeds the accuracy of the maximum intensity-based method in most cases. The error in incident light intensity corresponds to a maximum error of approximately 2% saturation. Therefore, though this new method is computationally more demanding than the traditional technique, it can be used in cases where the maximum intensity-based method fails (eg, stationary cells), or when higher accuracy is required.

Efficient image compression based on side match vector quantization and digital inpainting

In this paper, we propose two efficient compression schemes for digital images using an adaptive selection mechanism for vector quantization (VQ), side match vector quantization (SMVQ), and image inpainting. On the sender side, after the original image is divided into blocks, the compression is implemented block by block. In both schemes, blocks in pre-specified locations are first compressed by VQ. For each remaining block, the optimal compression method (for the first scheme, including VQ or inpainting, and for the second scheme, including VQ, SMVQ, and inpainting) is determined by computing the mean square error (MSE) between the original block and its inpainted result and then comparing it with a predefined threshold. If MSE is greater than the threshold, image inpainting continues to be used to compress the current block. Otherwise, the compression mode of VQ or SMVQ is selected to substitute image inpainting to maintain higher visual quality. With the assistance of transmitted indicator flags, the receiver side can execute the image inpainting and decompression successfully. Experimental results demonstrate the effectiveness and superiority of two proposed schemes.

Combination of single image super resolution and digital Inpainting algorithms based on GANS for robust image completion

Image inpainting, a technique of completing missing or corrupted image regions in undetected form, is an open problem in digital image processing. Inpainting of large regions using Deep Convolutional Generative Adversarial Nets (DCGAN) is a new and powerful approach. In described approaches the size of generated image and size of input image should be the same. In this paper we propose a new method where the size of input image with corrupted region can be up to 4 times larger than generated image.

Literature Review of Dual Data Hiding and Image Compression and Digital Inpainting

Literature Review of Dual Data Hiding and Image Compression and Digital Inpainting

Exemplar-based image inpainting using svd-based approximation matrix and multi-scale analysis

Reconstruction of images by digital inpainting is an active field of research and such algorithms are, in fact, now widely used. In conventional methods, a texture synthesis algorithm is used for f...

Digital inpainting with applications to forensic image processing

With the development of multimedia processing and applications, multiple types of media security and forensic have been broadly taken into consideration. The media data includes audio, video, graphic, image and etc. There are three main applications of the forensic image processing: hiding data, tampering detection and recovering of the digital information. Some artifacts and lost details on the image or video could lead to an erroneous interpretation. The objective of this paper is to ensure the production of quality forensic imagery using digital inpainting. Inpanting in images and videos aims at filling gaps (holes) occur due to instrumentation error, losses of image data during transmission. The holes may correspond to missing parts or removed objects from the scenes (logos, text, etc.). This paper includes brief descriptions of advantages, disadvantages, and potential of inpainting with applications to the forensic image processing. We present an image inpainting method based on the texture and structure propagation. It is shown that this approach allows restoring a fingerprint, missing blocks and removing a text from the scenes. Several examples considered in this paper show the effectiveness of the proposed approach for large objects removal as well as recovery of small regions on several test images and videos.

Special issue on real-time color image processing

Color perception plays an important role in object recognition and scene understanding both for humans and intelligent vision systems. Recent advances in digital color imaging and computer hardware technology have led to an explosion in the use of color images in a variety of applications including medical imaging, content-based image retrieval, biometrics, watermarking, digital inpainting, remote sensing, visual quality inspection, among many others. As a result, automated processing and analysis of color images has become an active area of research, which is witnessed by the large number of publications during the past two decades. The multivariate nature of color image data presents new challenges for researchers and practitioners as the numerous methods developed for singlechannel images are often not directly applicable to multichannel images. In addition, common scalar image processing operations often become computationally intractable when performed in high-dimensional spaces. As a result, the development of computationally practical methods for color image processing has become an important research area. The goals of this special issue are to summarize the state-of-the-art in real-time color image processing and to provide future directions for this exciting subfield of image processing. The intended audience includes researchers and professionals, who are increasingly dealing with the processing and analysis of color images and video. The special issue opens with ‘‘Design of a Shift-andAdd Based Hardware Accelerator for Color Space Conversion’’ by Li et al. The authors propose a hardware/software co-design architecture for Nios II, a 32-bit embedded processor. The architecture integrates a pipelined color space converter hardware accelerator based on a genetic algorithm and an LCD touch module. When implemented on a system-on-a-programmable-chip, the proposed architecture is capable of converting a 512 512 RGB image to YCbCr in 0:11 s. The special issue continues with seven articles on color image enhancement. In ‘‘A Simple Gray-Edge Automatic White Balance Method with FPGA Implementation,’’ Tan et al. propose a real-time automatic white balance method based on the gray-edge hypothesis. To reduce computational time, the method employs horizontal downsampling using the mean filter and gradient computation using horizontal first-order difference operator. The authors demonstrate the practicality of their approach using an FPGA implementation. In ‘‘A Full Linear 3 3 Color Correction between Images,’’ Lecca describes a fast color correction algorithm that is capable of equalizing the colors of two images of the same scene that are acquired under different illuminants and/or by different devices. The author demonstrates the performance of her approach against two state-of-the-art approaches on a variety of synthetic and real-world image databases. The method has linear time complexity in the M. E. Celebi (&) Department of Computer Science, Louisiana State University, Shreveport, LA, USA e-mail: ecelebi@lsus.edu

Image Inpainting on the Basis of Spectral Structure From 2-D Nonharmonic Analysis

The restoration of images by digital inpainting is an active field of research and such algorithms are, in fact, now widely used. Conventional methods generally apply textures that are most similar to the areas around the missing region or use a large image database. However, this produces discontinuous textures and thus unsatisfactory results. Here, we propose a new technique to overcome this limitation by using signal prediction based on the nonharmonic analysis (NHA) technique proposed by the authors. NHA can be used to extract accurate spectra, irrespective of the window function, and its frequency resolution is less than that of the discrete Fourier transform. The proposed method sequentially generates new textures on the basis of the spectrum obtained by NHA. Missing regions from the spectrum are repaired using an improved cost function for 2D NHA. The proposed method is evaluated using the standard images Lena, Barbara, Airplane, Pepper, and Mandrill. The results show an improvement in MSE of about 10-20 compared with the examplar-based method and good subjective quality.

An Automatic Framework for Segmentation and Digital Inpainting of 2D Frontal Face Images

Nowadays applications that use face images as input for people identification have been very common. In general, the input image must be preprocessed in order to fit some normalization and quality criteria. In this paper, we propose a computational framework composed of digital image quality computation, segmentation of the damaged regions on face images by statistical decision, morphological operators and image restoration by inpainting techniques. Additionally, in this work we propose a new method for digital inpainting that considers as relevant information for such restoration the neighboring pixel intensities as well as prior information extracted from an image database. To assess the efficiency of the computational framework proposed, we have used 2D face images from public databases. The results show that our inpainting method performs similarly to the traditional ones on quasi-homogeneous regions and gives better results when the damaged areas includes complex image patterns.

Total Variation Inpainting using Split Bregman

Given an image where a specified region is unknown, image inpainting or image completion is the problem of inferring the image content in this region. Traditional retouching or inpainting is the practice of restoring aged artwork, where damaged or missing portions are repainted based on the surrounding content to approximate the original appearance. In the context of digital images, inpainting is used to restore regions of an image that are corrupted by noise or where the data is missing. Inpainting is also used to solve disocclusion, to estimate the scene behind an obscuring foreground object. A popular use of digital inpainting is object removal, for example, to remove a trashcan that disrupts a scene of otherwise natural beauty. Inpainting is an interpolation problem, filling the unknown region with a condition to agree with the known image on the boundary. A classical solution for such an interpolation is to solve Laplace’s equation. However, Laplace’s equation is usually unsatisfactory for images since it is overly smooth. It cannot recover a step edge passing through the region. Total variation (TV) regularization is an effective inpainting technique which is capable of recovering sharp edges under some conditions (these conditions will be explained). The use of TV regularization was originally developed for image denoising by Rudin, Osher, and Fatemi [3] and then applied to inpainting by Chan and Shen [13]. TV-regularized inpainting does not create texture, the method is limited to inpainting the geometric structure.

Removing high contrast artifacts via digital inpainting in cryo-electron tomography: An application of compressed sensing

To cope with poor quality in cryo-electron tomography images, electron-dense markers, such as colloidal goldbeads, are often used to assist image registration and analysis algorithms. However, these markers can create artifacts that occlude a specimen due to their high contrast, which can also cause failure of some image processing algorithms. One way of reducing these artifacts is to replace high contrast objects with pixel densities that blend into the surroundings in the projection domain before volume reconstruction. In this paper, we propose digital inpainting via compressed sensing (CS) as a new method to achieve this goal. We show that cryo-ET projections are sparse in the discrete cosine transform (DCT) domain, and, by finding the sparsest DCT domain decompositions given uncorrupted pixels, we can fill in the missing pixel values that are occluded by high contrast objects without discontinuities. Our method reduces visual artifacts both in projections and in tomograms better than conventional algorithms, such as polynomial interpolation and random noise inpainting.