Inpainting Algorithm Research Articles

Texture Synthesis Based Thyroid Nodule Detection From Medical Ultrasound Images: Interpreting and Suppressing the Adversarial Effect of In-place Manual Annotation.

Deep learning method have been offering promising solutions for medical image processing, but failing to understand what features in the input image are captured and whether certain artifacts are mistakenly included in the model, thus create crucial problems in generalizability of the model. We targeted a common issue of this kind caused by manual annotations appeared in medical image. These annotations are usually made by the doctors at the spot of medical interest and have adversarial effect on many computer vision AI tasks. We developed an inpainting algorithm to remove the annotations and recover the original images. Besides we applied variational information bottleneck method in order to filter out the unwanted features and enhance the robustness of the model. Our impaiting algorithm is extensively tested in object detection in thyroid ultrasound image data. The mAP (mean average precision, with IoU = 0.3) is 27% without the annotation removal. The mAP is 83% if manually removed the annotations using Photoshop and is enhanced to 90% using our inpainting algorithm. Our work can be utilized in future development and evaluation of artificial intelligence models based on medical images with defects.

Improving Image Inpainting based on Structure and Texture Information Using Quadtree

In this paper, we present a novel and efficient algorithm for image inpainting based on the structure and texture components. In our method, after decomposing the image into its texture and structure components using PCA (Principal Component Analysis), these components are inpainted separately using the proposed algorithm. Finally, the inpainted image is simply acquired by adding the two inpainted images. For structure inpainting we used quadtree concept to identify the importance of each pixel located on the boundary of the target region. Subsequently, we detect the correct path for filling so that this path demonstrates an orientation for the better structure inpainting. It is noteworthy that structure inpainting is more important because human vision is sensitive to the coherence of structure. For texture inpainting, we use Euclidean distance in the texture component for patch selection. Also, the geometric feature is considered by LSK (Local Steering Kernel) in the original image to assist chooing a better patch candidate. The experimental results of our algorithm demonstrate the effectiveness of the proposed method.

Multistage attention network for image inpainting

Image inpainting refers to the process of restoring the mask regions of damaged images. Existing inpainting algorithms have exhibited outstanding performance on certain inpainting tasks that are focused on recovering small masks or square masks. Tasks that attempt to reconstruct large proportion of damaged images can still be improved. Although many attention-related algorithms have been proposed to solve image inpainting tasks, most of them ignore the requirements to balancing the detail and style level. In this paper, we propose a novel image inpainting method for large-scale irregular masks. We introduce a special multistage attention module that considers structure consistency and detail fineness. The proposed multistage attention module operates in a coarse to-fine manner, where the early stage performs large feature patch swapping and ensures the global consistency in images, and the next stage swaps small patches to refine the texture. Then, we adopt a partial convolution strategy to avoid the misuse of invalid data during convolution. Several losses are combined as the training objective function to generate excellent results with global consistency and exquisite detail. Qualitative and quantitative experiments on the Paris StreetView, CelebA, and Places2 datasets demonstrate the superior performance of the proposed approach compared with state-of-the-art models.

AutoRemover: Automatic Object Removal for Autonomous Driving Videos

Motivated by the need for photo-realistic simulation in autonomous driving, in this paper we present a video inpainting algorithm AutoRemover, designed specifically for generating street-view videos without any moving objects. In our setup we have two challenges: the first is the shadow, shadows are usually unlabeled but tightly coupled with the moving objects. The second is the large ego-motion in the videos. To deal with shadows, we build up an autonomous driving shadow dataset and design a deep neural network to detect shadows automatically. To deal with large ego-motion, we take advantage of the multi-source data, in particular the 3D data, in autonomous driving. More specifically, the geometric relationship between frames is incorporated into an inpainting deep neural network to produce high-quality structurally consistent video output. Experiments show that our method outperforms other state-of-the-art (SOTA) object removal algorithms, reducing the RMSE by over 19%.

Structural inpainting techniques using equations of engineering physics

A comprehensive survey on structure-based inpainting methods based on equations of engineering physics is provided in this work. Diffusion equations, describing the random motion of the micro-particles in physics, have been successfully applied in the image processing areas in the last decades. Nonlinear anisotropic diffusion-based structural inpainting techniques, using variational and non-variational PDE models, are discussed first. Influential variational interpolation approaches, such as those using the Mumford–Shah functional, total variation inpainting and its versions, and Euler’s Elastica inpainting, are addressed here. Second and fourth-order anisotropic diffusion-based interpolation schemes, some of them not following variational principles, are then presented. Our own contributions in this field are also discussed. The applications of the reaction-diffusion equations, like Ginzburg–Landau equation, describing a large variety of physics phenomena, in the inpainting domain, are described next. Structural reconstructions methods using partial differential equations for fluid dynamics, such as the Navier–Stokes equations describing the motion of the viscous fluids, are then considered here. Other state of the art third-order PDE-based structural inpainting algorithms surveyed here are based on the curvature-driven diffusion equations that use the thermal diffusion principle in physics. Fourth-order PDE-based interpolation solutions, such as Cahn–Hilliard inpainting that uses a modified mathematical physics equation describing the phase separation process, are also described.

A Novel Azimuth Ambiguity Suppression Method for Spaceborne Dual-Channel SAR-GMTI

Azimuth ambiguity degrades the quality of synthetic aperture radar (SAR) images and leads to the increase of false alarm rate in ground moving target indication (GMTI). Due to the existing azimuth ambiguity, suppression methods do not remove the first-order ambiguity completely and ignore the ambiguity above first order as well, moving target detection is affected by residual ambiguity. Hence, a novel method to suppress first-order and higher order azimuth ambiguities for the dual-channel SAR/GMTI is proposed in this letter. First, the displaced phase center antenna (DPCA) technique is applied to suppress clutter. Then, estimate the local azimuth ambiguity-to-signal ratio (LAASR) to find out the area affected by ambiguity. Finally, an inpainting algorithm is improved to patch the ambiguity area. The proposed method can remove the ambiguity almost completely. Moreover, the method is verified using the GaoFen-3 SAR dual-channel complex image data, and the result shows that the false alarm of moving target detection is degraded without reducing the detection rate.

Improving image inpainting quality by a new SVD-based decomposition

In this paper, we present a new algorithm for image inpainting using structure and texture information. Our image decomposition to texture and structure is accomplished by the SVD method in the primary step, and then an algorithm for texture inpainting is applied. At the next level, edge detection is used in target region related to inpainted texture component. The detected edges demonstrate border of different textures in the target region, and the boundary pixels are ignored from mask temporarily. The other target pixels should be primarily inpainted, and then border pixels would be filled subsequently. Experimental results of this algorithm show better consistency in comparison with state of the art methods.

A 360-degree Panoramic Image Inpainting Network Using a Cube Map

Inpainting has been continuously studied in the field of computer vision. As artificial intelligence technology developed, deep learning technology was introduced in inpainting research, helping to improve performance. Currently, the input target of an inpainting algorithm using deep learning has been studied from a single image to a video. However, deep learning-based inpainting technology for panoramic images has not been actively studied. We propose a 360-degree panoramic image inpainting method using generative adversarial networks (GANs). The proposed network inputs a 360-degree equirectangular format panoramic image converts it into a cube map format, which has relatively little distortion and uses it as a training network. Since the cube map format is used, the correlation of the six sides of the cube map should be considered. Therefore, all faces of the cube map are used as input for the whole discriminative network, and each face of the cube map is used as input for the slice discriminative network to determine the authenticity of the generated image. The proposed network performed qualitatively better than existing single-image inpainting algorithms and baseline algorithms.

Image Inpainting by Low-Rank Prior and Iterative Denoising

To reconstruct the missing or damaged parts of images from observed incomplete data, some traditional methods have been researched in recent years. The iterative denoising and backward projections(IDBP)algorithm with a simple parameter mechanism have been recently introduced, which solves the typical inverse problem by utilizing the existing 3D transform-domain collaborative filtering denoising algorithm(BM3D). While this algorithm has simple parameter tuning, the collaborative hard-thresholding applied to the 3D group is greatly restricted in the procedure of denoising. In this paper, we remedy this deficiency using an iteration reweighted shrinkage denoising method. First, the model is obtained by a Plug and Play(P&P) framework. Then, we solve the optimization problem by using a proposed denoising model based on low rank prior and reweighted shrinkage and obtain a closed-form solution. Finally, the closed-form solution is operated iteratively by using the adaptive backward projection technique. Utilizing this novel strategy, the proposed algorithm not only removes the image noise and effectively recovers the degraded image, but also preserves fine structure and texture information of the image. Experimental results indicate that the proposed algorithm is competitive with some state-of-the-art inpainting algorithms in terms of both numerical evaluation and visual quality.

基于Gabor变换和组稀疏表示的敦煌壁画修复算法

基于Gabor变换和组稀疏表示的敦煌壁画修复算法

Sequential Detection of Image Defects for Patterned Fabrics

Automatic fabric defect detection has been successfully applied to establish the quality quick response system for the automation of textile production. However, the image complexity and diversity of patterned fabrics have effects on the fabric defect detection, which makes it difficult for automatic quality inspection of textiles. In order to solve this problem, a novel method that sequential detection of image defects for patterned fabrics was proposed in this article. Firstly, the fabric image was segmented adaptively based on the periodic distance of the pattern, which can be used for the determination of repetitive pattern block element in fabrics, so that the defect image blocks could be identified according to the minimum principle of structure similarity; Then, the texture features of defect image blocks were extracted to match the constructed feature dictionary of defect-free image block, the defect position could be located by distance measurement and threshold segmentation. Our experiments show that the proposed method has the advantages of low computational complexity, high detection accuracy and strong applicability, by the comparison with existing approaches for defect detection of patterned fabric. Moreover, the new proposed automatic defect location method can solve the problem of deficiency caused by manual defect marking using existing image inpainting algorithms, it can be used for the automatic image inpainting of patterned fabrics without manual marking to improve the robustness of computer vision-based fabric re-engineering.

基于序贯相似性和布谷鸟寻优的敦煌壁画修复算法

基于序贯相似性和布谷鸟寻优的敦煌壁画修复算法

Spatiotemporal Modeling of Urban Growth Using Machine Learning

This paper presents a general framework for modeling the growth of three important variables for cities: population distribution, binary urban footprint, and urban footprint in color. The framework models the population distribution as a spatiotemporal regression problem using machine learning, and it obtains the binary urban footprint from the population distribution through a binary classifier plus a temporal correction for existing urban regions. The framework estimates the urban footprint in color from its previous value, as well as from past and current values of the binary urban footprint using a semantic inpainting algorithm. By combining this framework with free data from the Landsat archive and the Global Human Settlement Layer framework, interested users can get approximate growth predictions of any city in the world. These predictions can be improved with the inclusion in the framework of additional spatially distributed input variables over time subject to availability. Unlike widely used growth models based on cellular automata, there are two main advantages of using the proposed machine learning-based framework. Firstly, it does not require to define rules a priori because the model learns the dynamics of growth directly from the historical data. Secondly, it is very easy to train new machine learning models using different explanatory input variables to assess their impact. As a proof of concept, we tested the framework in Valledupar and Rionegro, two Latin American cities located in Colombia with different geomorphological characteristics, and found that the model predictions were in close agreement with the ground-truth based on performance metrics, such as the root-mean-square error, zero-mean normalized cross-correlation, Pearson’s correlation coefficient for continuous variables, and a few others for discrete variables such as the intersection over union, accuracy, and the f 1 metric. In summary, our framework for modeling urban growth is flexible, allows sensitivity analyses, and can help policymakers worldwide to assess different what-if scenarios during the planning cycle of sustainable and resilient cities.

An effective fragile watermarking scheme for color image tampering detection and self-recovery

In this paper, a fragile watermarking scheme for color-image authentication and self-recovery is proposed. Original image is divided into non-overlapping blocks, and for each i-th block, the watermarks used for recovery and authentication are generated, which are embedded into a different block according to an embedding sequence given by a permutation process. The designed scheme embeds the watermarks generated by each block within the 2-LSB, where a bit-adjustment phase is subsequently applied to increase the quality of the watermarked image. In order to increase the quality of the recovered image, we use in the post-processing stage the bilateral filter that efficiently suppresses noise preserving image edges. Additionally, in the tamper detection process high accuracy is achieved employing a hierarchical tamper detection algorithm. Finally, to solve tampering coincidence problem, three recovery watermarks are embedded in different positions to reconstruct a specific block, and a proposed inpainting algorithm is implemented to regenerate those regions affected by this problem. Simulation results demonstrate that the watermarked images appear to demonstrate higher quality, and the proposed novel scheme can reconstruct the alteration of extremely high rates (up to 80%), obtaining good quality for altered regions that are self-recovered with higher visual performance compared with a similar scheme from state of the-art methods.

An Investigation for Enhancing Registration Performance with Brain Atlas by Novel Image Inpainting Technique using Dice and Jaccard Score on Multiple Sclerosis (MS) Tissue

This paper presents a novel approach of investigating registration performance of different Multiple Sclerosis (MS) affected brain tissues with brain atlas by image inpainting technique. MS is an immune-mediated disorder that develops from an interaction of the individual's genetics and unidentified environmental causes. In order to carry out medical diagnosis in proper time image registration plays a vital role. In this work, different MS lesions are introduced in healthy brain tissue in order to analyse and characterize according to their characteristics (T1 weighted image, T2 weighted image) because MS lesions create a strong bias in image registration process. Thus, image inpainting technique appears to be a handy tool to reduce the bias efficiently. It is evident that the proposed inpainting algorithm performs satisfactorily with a view to reducing the bias in the registration step. The overall performance of the technique is evaluated by utilizing Dice and Jaccard scores. MATLAB and FSL software are used to perform the simulation.

Hyperspectral Image Inpainting Based on Robust Spectral Dictionary Learning

To address the problems of defective pixels and strips in hyperspectral images affecting subsequent processing and applications, we modeled the hyperspectral image (HSI) inpainting problem as a sparse signal reconstruction problem with incomplete observations using the theory of sparse representation, and proposed an HSI inpainting algorithm based on spectral dictionary learning. First, we studied the HSI observation model under the assumption of additive noise. We subsequently proposed a new algorithm for constructing a spectral dictionary directly from hyperspectral data by introducing an online learning optimization method and performing dictionary learning using a robust function. Afterwards, the image was sparsely encoded by applying the variable decomposition and augmented Lagrangian sparse regression method. Finally, the inpainted HSI was obtained by sparse reconstruction. The experimental results showed that compared with the existing algorithms, the algorithm proposed herein could effectively inpaint the defective HSI under different noise conditions with a shorter calculation time than those of existing methods and other dictionary learning inpainting algorithms.

Tomographic Collection of Block-Based Sparse STEM Images: Practical Implementation and Impact on the Quality of the 3D Reconstructed Volume.

The reduction of the electron dose in electron tomography of biological samples is of high significance to diminish radiation damages. Simulations have shown that sparse data collection can perform efficient electron dose reduction. Frameworks based on compressive-sensing or inpainting algorithms have been proposed to accurately reconstruct missing information in sparse data. The present work proposes a practical implementation to perform tomographic collection of block-based sparse images in scanning transmission electron microscopy. The method has been applied on sections of chemically-fixed and resin-embedded Trypanosoma brucei cells. There are 3D reconstructions obtained from various amounts of downsampling, which are compared and eventually the limits of electron dose reduction using this method are explored.

Inpainting of Dunhuang Murals by Sparsely Modeling the Texture Similarity and Structure Continuity

Ancient mural paintings often suffer from damage such as color degradation, pigment peeling, and even large-area shedding. Image inpainting techniques are widely used to virtually repair these damages. Generally, the inpainting task can be very challenging when structures are totally missing within a large area. In this article, we study mural image inpainting by incorporating structure information collected from line drawings, and propose a line-drawings-guided inpainting algorithm for repairing the damaged murals of Mogao Grottoes, Dunhuang. Unlike traditional methods that use one single patch to inpaint the target area, the proposed method constructs the target patch with a linear combination of multiple candidate patches. These candidate patches are selected by a sparse model, where two special constraints have been introduced to guarantee the texture similarity and structure continuity. Experimental results demonstrate the effectiveness of the proposed method.

Patch-Sparsity-Based Image Inpainting Through a Facet Deduced Directional Derivative

This paper presents a patch-sparsity-based image inpainting algorithm through a facet deduced directional derivative. The algorithm could ensure the continuity of boundaries of the inpainted region and achieve a better performance on restoring the missing structure of an image. In this paper, two improvements are proposed. First, the facet model is introduced to get direction features of the image, which could efficiently reduce the effect of noises. The first-order directional derivatives, along with pixel values, are used to measure the difference between patches. Consequently, a more reliable and accurate matching result is promised. At the same time, the local patch consistency constraint of sparse representation of the target patch is also rewritten in the form of the first-order directional derivative. Therefore, a more precise sparse linear combination could be obtained under constraints for both color and derivative information. Second, the value of patch confidence in the traditional exemplar-based inpainting algorithms drops sharply in the late stage so that the data term or structure sparsity has little influence on priority function. Aiming at this problem, the algorithm makes a modification to the calculating of priority. Thus, the filling order decided by priority function appears more reasonable as a result of a better balance between the values of modified confidence and structure sparsity. Experiments on different types of damages to images show the superiority of the algorithm.

A new algorithm for image inpainting in Fourier transform domain

One of the aims of image inpainting is recovering an image some of which Fourier transform coefficients are lost. In this paper, we present a new algorithm for image inpainting in Fourier transform domain. We consider the effect of spectrum and phase angle of the Fourier transform, separately. Hence, two regularization parameters are generated; therefore, we have two degree of freedom. Some numerical examples confirm our proposed method.