Exemplar-based Image Inpainting Research Articles

Exemplar-based image inpainting using adaptive two-stage structure-tensor based priority function and nonlocal filtering

For the exemplar-based image inpainting problem, the filling order and local intensity smoothness are two crucial factors that should be considered carefully. This work gives a new exemplar-based image inpainting method, preventing geometric structures from being destroyed and reconstructing textures well to obtain elegant-looking outputs. For a better filling order, we define a new adaptive two-stage structure-tensor based priority function. To promote the local intensity smoothness, we adopt a non-local way, and at the same time, propose a weighted filter based on a Gaussian-like function to generate the ideal filling patch by combining non-local patches. We compare the proposed method with some recent state-of-the-art image inpainting approaches on different tasks, such as texture and structure synthesis, object removal, and remote sensing images inpainting. Experimental results demonstrate the superiority of the proposed method, both visually and quantitatively.

An improved image quality algorithm for exemplar-based image inpainting

Image inpainting is a common technique for repairing image regions that are scratched or damaged. This process involves reconstructing damaged parts and filling-in regions in which data/colour information is missing. There are many potential applications for image inpainting, such as repairing old images, repairing scratched images, removing unwanted objects, and filling-in missing areas. This paper develops an exemplar-based algorithm, one of the most important and popular image inpainting techniques, to fill-in missing regions caused by removing unwanted objects, image compression, scratches, or image transformation via the Internet. The proposed algorithm includes two phases of searching to select the best-matching information. In the first phase, the searching mechanism uses the entire image to find and select the most similar patches using the Euclidean distance. The second phase measures the distance between the location of the selected patches and the location of the patch to be filled. The performance of the proposed approach is evaluated through comprehensive experiments on several well-known images used in this area of research. The experimental results demonstrate the superior performance of the proposed approach over some state-of-the-art approaches in terms of quality in terms of both objective (using the peak signal-to-noise ratio (PSNR) as well as the structural similarity index method (SSIM)) and subjective (i.e., visual) measures.

Research on image inpainting algorithm of improved total variation minimization method

In order to solve the issue mismatching and structure disconnecting in exemplar-based image inpainting, an image completion algorithm based on improved total variation minimization method had been proposed in the paper, refer as ETVM. The structure of image had been extracted using improved total variation minimization method, and the known information of image is sufficiently used by existing methods. The robust filling mechanism can be achieved according to the direction of image structure and it has less noise than original image. The priority term had been redefined to eliminate the product effect and ensure data term had always effective. The priority of repairing patch and the best matching patch are determined by the similarity of the known information and the consistency of the unknown information in the repairing patch. The comparisons with cognitive computing image algorithms had been shown that the proposed method can ensure better selection of candidate image pixel to fill with, and it is achieved better global coherence of image completion than others. The inpainting results of noisy images show that the proposed method has good robustness and can also get good inpainting results for noisy images.

Improved Exemplar based Image Inpainting for Partial Instance Occlusion Handling with K-means Clustering and YCbCr Color Space

The images acquired in real time outdoor environment are often subject to uneven illumination conditions, cloudy weather, lighting conditions. The instances of partial occlusions deteriorate the background modeling of such scenes. Varying illumination outdoor scene set of images with partial occlusions are addressed through this investigative work. There is a need for a restoration method that finds improved subjective perception and execution time. The proposed work focuses on novel amended exemplar model to improve the subjective perception. The exemplar inpainting method is improved through the color quantization with K-means clustering approach in YCbCr color space. Experimental validations and proposed method results show better improvement in qualitative and objective measures than existing methods. The average “Peak Signal to Noise Ratio (PSNR)” as 28.2869 and “Structural SIMilarity index (SSIM)” as 0.9759 of the proposed method has shown better results respectively both visually and with a tradeoff in time.

Image Inpainting Based on Structural Tensor Edge Intensity Model

In the exemplar-based image inpainting approach, there are usually two major problems: the unreasonable calculation of priority and only considering the color features in the patch lookup strategy. In this paper, we propose an image inpainting approach based on the structural tensor edge intensity model. First, we use the progressive scanning inpainting method to avoid the image filling order being affected by the priority function. Then, we use the edge intensity model to build the patches similarity function for correctly identifying the local image structure. Finally, the balance operator is used to restrict the excessive propagation of structural information to ensure the correct structural reconstruction. The experimental results show that the our approach is comparable and even superior to some state-of-the-art inpainting algorithms.

A Detection Approach Using LSTM-CNN for Object Removal Caused by Exemplar-Based Image Inpainting

Exemplar-based image inpainting technology is a “double-edged sword”. It can not only restore the integrity of image by inpainting damaged or removed regions, but can also tamper with the image by using the pixels around the object region to fill in the gaps left by object removal. Through the research and analysis, it is found that the existing exemplar-based image inpainting forensics methods generally have the following disadvantages: the abnormal similar patches are time-consuming and inaccurate to search, have a high false alarm rate and a lack of robustness to multiple post-processing combined operations. In view of the above shortcomings, a detection method based on long short-term memory (LSTM)-convolutional neural network (CNN) for image object removal is proposed. In this method, CNN is used to search for abnormal similar patches. Because of CNN’s strong learning ability, it improves the speed and accuracy of the search. The LSTM network is used to eliminate the influence of false alarm patches on detection results and reduce the false alarm rate. A filtering module is designed to eliminate the attack of post-processing operation. Experimental results show that the method has a high accuracy, and can resist the attack of post-processing combination operations. It can achieve a better performance than the state-of-the-art approaches.

Quality Improvement for Exemplar-based Image Inpainting using a Modified Searching Mechanism

Digital image processing has a significant impact in different research areas including medical image processing, biometrics, image inpainting, object detection, information hiding, and image compression. Image inpainting is a science of reconstructing damaged parts of digital images and filling-in regions in which information are missing which has many potential applications such as repairing scratched images, removing unwanted objects, filling missing area, and repairing old images. In this paper, an image inpainting algorithm is developed based on exemplar, which is one of the most important and popular images inpainting technique, to fill-in missing area that caused either by removing unwanted objects, by image compression, by scratching image, or by image transformation through internet. In general, image inpainting consists of two main steps: The first one is the priority function. In this step, the algorithm decides to select which patch has the highest priority to be filled at the first. The second step is the searching mechanism to find the most similar patch to the selected highest priority patch to be inpainted. This paper concerns the second step and an improved searching mechanism is proposed to select the most similar patch. The proposed approach entails three steps: (1) Euclidean distance is used to find the similarity between the highest priority patches which need to be inpainted with each patch of the input image, (2) the position/location distance between those two patches is calculated, and (3) the resulted value from the first step is summed with the resulted value obtained from the second step. These steps are repeated until the last patch from the input image is checked. Finally, the smallest distance value obtained in step 3 is selected as the most similar patch. Experimental results demonstrated that the proposed approach gained a higher quality in terms of both objectives and subjective compared to other existing algorithms.

A qualitative study of Exemplar based Image Inpainting

Image inpainting is exploring the way of filling the missing or left out regions in an image by only having knowledge of surrounding pixel values. One of the block matching algorithms use for image inpainting is exemplar based technique. Exemplar based image inpainting method search the best matching patch for the target region in the image and when it is obtained, the pixel values are copied from the source region and pasted to the target region. The selections of different patch matching criteria, patch size and filling order have a great affect on the results. This paper describes some of the variations in traditional exemplar method which include change in patch selection, priority, patch size and multi-resolution inpainting. The advantages and drawbacks for the modified methods are also discussed here.

Average-face-based virtual inpainting for severely damaged statues of Dazu Rock Carvings

Abstract Numerous image inpainting algorithms are guided by a basic assumption that the known region in the original image itself can provide sufficient prior information for the guess recovery of the unknown part, which is not often the case in actual art image inpainting. In order to solve the challenging inpainting case that there is little image prior in the remainder of the original image, we propose an average-face-based inpainting method based on a sample database with 3 steps: reference images selection, average image generation and exemplar-based image inpainting. In which, average image generation is crucial. In the inpainting framework, the average image can be directly viewed as an inpainting proposal for the severely damaged or absolutely lost image. Moreover, the average image can be applied to exemplar-based inpainting algorithm as a sample image to extend the searching region for match patch, so as to perform the restoration for images with large-scale or irregularly damaged holes. The inpainting experiments over some facial images of Dazu Rock Carvings demonstrate the validity and effectiveness of our method. It is first utilized for two extremely challenging inpainting tasks: reconstruction for the stolen head of Willow Avalokitesvara in Shimenshan No. 6 and the absolutely broken heads of two Avalokitesvaras in Beishan No. 180. Compared to the failure of the exemplar-based inpainting algorithm within the original image and the directly duplication of a similar image, the generated average image can be as a more reliable inpainting proposal. The comparative experiments also show the efficiency and advantage of the average face applied to exemplar-based inpainting framework. Compared with some related inpainting algorithms, our method is more competitive when there is little prior information in the original image. The efficient virtual inpainting results are valuable references for both cave art historians and conservators.

Hierarchical Guidance Strategy and Exemplar-Based Image Inpainting

To solve the issue that it is difficult to maintain the consistency of linear structures when filling large regions by the exemplar-based technique, a hierarchical guidance strategy and exemplar-based image inpainting technique is proposed. The inpainting process is as follows: (i) the multi-layer resolution images are firstly acquired through decomposing of the pyramid on the target image; (ii) the top-layer inpainted image, the beginning of the inpainting from the top layer, is generated by the exemplar-based technique; (iii) there is a combined result between the next layer of the top image and the up-sampling output on the top-layer inpainted image, and the target regions are filled with information as guidance data; (iv) this process is repeated until the inpainting of all layers have been completed. Our results were compared to those obtained by existing techniques, and our proposed technique maintained the consistency of linear structures in a visually plausible way. Objectively, we choose SSIM (structural similarity index measurement) and PSNR (peak signal-to-noise ratio) as the measurement indices. Since the values of SSIM are well reflected when compared with other techniques, our technique clearly demonstrated that our approach is better able to maintain the consistency of linear structures. The core of our algorithm is to fill large regions whether they are synthesis images or real-scene photographs. It is easy to apply in practice, with the goal of having plausible inpainted image.

A novel patch matching algorithm for exemplar-based image inpainting

In the existing exemplar-based image inpainting algorithms, the Sum of Squared Differences (SSD) method is employed to measure the similarities between patches in a fixed size, and then using the most similar one to inpaint the destroyed region. However, sometimes only calculating the SSD difference would produce a discontinuous structure and blur the texture. To solve this problem, we firstly optimize the inpainting priority function and proposed an adaptive patch method to obtain more significant patches. The adaptive patch method changes the size of the patch by computing the patch sparsity. Secondly the proposed method calculates the maximum similarity between patches in different rotation angles so that it obtains the most similar rotation invariant matching patch. From the experimental results, the proposed method can improve the accuracy of the patch selection process compared with the traditional methods, and the proposed method can keep a better global visual appearance, especially for the image which contains more structure contents and the images whose destroyed region has a large width.

Damaged region filling and evaluation by symmetrical exemplar-based image inpainting for Thangka

Exemplar-based image inpainting, as proposed by Criminisi et al. (IEEE Trans Image Process 13(9):1200–1212, 2004), fills missing regions by using a similar exemplar. However, when the missing region is a unique texture patch, an incorrect texture is filled in the missing region because a similar exemplar of damaged patch could not be found. A new image inpainting method based on an eight-direction or arbitrary direction symmetrical exemplar is proposed, suitable for damaged images containing local symmetry. The following three steps are the keys of this method. (1) According to certain similarity criteria, the symmetrical exemplars of damaged regions in eight directions or arbitrary directions are found. (2) The most similar symmetrical exemplar is selected from eight-direction or arbitrary-direction symmetrical exemplars. (3) Finally, the damaged region is filled using the most similar symmetrical exemplar. It is shown that the results of image inpainting are good when missing image regions have similar symmetry. Image inpainting is a single, efficient method. In addition, a new evaluation method of image restoration results based on similar exemplars is proposed for the inpainting effect, which is closely related to the repair algorithm. Therefore, the methods can more objectively measure the inpainting effect.

Exemplar-based image inpainting using structure consistent patch matching

Image inpainting restores lost or deteriorated parts of images according to the information of known regions. Criminisi has proposed an effective exemplar-based inpainting method, which has the advantages of both texture synthesis and diffusion-based inpainting. Yet, it has its own flaws of fast priority dropping and visual inconsistency. In this paper, we propose a space varying updating strategy for the confidence term and a matching confidence term to improve the filling priority estimation. We propose structure consistent patch matching to take the distribution of source and target patch differences into account. Fast Fourier transform is adapted for full image searching to achieve better and faster matching results. Experimental results are given to demonstrate the improvements made by our proposed method.

A robust forgery detection algorithm for object removal by exemplar-based image inpainting

Object removal is a malicious image forgery technique, which is usually achieved by exemplar-based image inpainting in a visually plausible way. Most existing forgery detection approaches utilize similar block pairs between inpainted area and the rest areas, but they invalidate when those inpainted images are further subjected to some post-processing operations such as JPEG compression, Gaussian noise addition and blurring. It is desirable to develop a forensic method which is robust to object removal with post-processing. From some preliminary experiments, we observe that post-processing destroys the similarity of block pairs and simultaneously disturbs the correlations among adjacent pixels to some extent. Inspired by the strong ability of joint probability density matrix (JPDM) in characterizing such correlation, we propose a hybrid forensics strategy. Firstly, our earlier method is employed to detect whether a candidate image is forged or not. Secondly, for those undetected images after the first step, JPDM is computed for each difference array to model the correlations among adjacent DCT coefficients, and the average of these matrixes are computed as feature vectors to further expose tampering traces. Experimental results show that the proposed approach can effectively detect object removal by exemplar-based inpainting either with or without post-processing.

Fast Exemplar-Based Image Inpainting Using a New Pruning Technique

This paper presents a new algorithm for fast exemplar-based image inpainting by using image pyramid and a novel pruning scheme. The proposed technique separates the inpainting process into three major steps. First the model determines which patch belonging to the inpainting region should be filled first. Then the pruning system analyzes the selected patch and weeds out regions having no chance to be the best matching patch. Finally, it performs a fast texture synthesis based on greedy search method in the remaining regions to fill the selected patch. Compared with previous methods, the proposed algorithm is much more efficient computationally. Experiments on synthetic and natural images show the advantages of the proposed algorithm, where it is shown to compare favorably with contemporary state-of-the-art schemes.

Image Inpainting Through Metric Labeling via Guided Patch Mixing.

In this paper, we present a novel formulation of exemplar-based image inpainting as a metric labeling problem, and solve it through the simulated annealing algorithm. Due to their greedy nature, exemplar-based methods sometimes produce inpainted images, which are visually inconsistent. These methods are highly dependent upon the initialization. To solve these problems, we generate five images with a different initialization. A suitable mixture of these five images produces a good inpainted image. The cost function of the proposed metric labeling problem consists of three components, namely, neighbor cost, total variation cost, and structure cost. A linear combination among these components is used to maintain better visual consistency in the inpainted region having smooth transition from the bordering regions of the source image. We use a quality measure to this end. Our experiments on a wide variety of images demonstrate that the proposed technique produces better inpainting images as compared with some other state-of-the-art techniques.

Exemplar-based image inpainting using svd-based approximation matrix and multi-scale analysis

Reconstruction of images by digital inpainting is an active field of research and such algorithms are, in fact, now widely used. In conventional methods, a texture synthesis algorithm is used for f...

Paper] A Study on Effect of Automatic Perspective Correction on Exemplar-based Image Inpainting

Paper] A Study on Effect of Automatic Perspective Correction on Exemplar-based Image Inpainting

영상 인페인팅을 이용한 틀린그림찾기 게임 컨텐츠 자동 생성 기법

In this paper, we propose a method to automatically create the contents for spot-the-difference games. A spot-the-difference game is the game that finds the differences such as removal of some objects in the image or changes of shapes and colors from the original image. The original purpose of the exemplar-based image inpainting technique is to remove unnecessary objects. We use the exemplar-based inpainting technique to make the spot-the-difference game contents. From our implementation and experiments, we showed the effectiveness of our automatic generation technique of spot-the-difference contents.

Improvement of Patch Selection in Exemplar-based Image Inpainting

In the existing exemplar based image inpainting algorithms, the most similar match patches are used to in-paint the destroyed region, and are searched in the whole source region in a fixed size. However, sometimes such an approach would decrease the connectivity of structure and clearness of texture while increases the time complexity. To solve these problems, firstly we propose an adaptive sample algorithm based on patch sparsity, by which divide the patch into three types ( smooth type, transition type and edge type ). Then the size of the sample patch can be adaptively changed according to the type. Secondly a candidate patch method is also proposed to improve the patch matching performance. Simulation results show that the proposed method obtains a more reasonable patches than the traditional one. Furthermore, it gives a better texture inpainting effect, especially when the processed image either has the complex and regular textures or has a large destroyed region.