Exemplar-based Inpainting Algorithm Research Articles

DEFORMABLE OBJECT REMOVAL VIA EXEMPLAR-BASED ALGORITHM FOR 3D MODEL OF HERITAGE STRUCTURES

Heritage structures are important for tourism and for learning about nations’ history. Historical sites around the world are in damaged conditions due to various factors from the environment, man-made activities or natural calamities. To ensure safety and avoid further deterioration, digital re- construction of heritage sites can be used to create 3D virtual models of these structures for archiving and for inspection purpose. This paper presents a technique to reconstruct a high-quality image-based 3D model of a heritage structure. An exemplar-based inpainting algorithm was used for removing deformable obstacles such as trees in 2D images since such obstacles are unwanted in a final 3D model. They also create noise in the 3D modeling process, which makes the final 3D model less accurate. In this paper, an inpainting algorithm is applied to generate implied texture to fill holes that occurred from removing objects so that the generated 3D model does not contain any hole and becomes watertight. The modified exemplar-based inpainting algorithm was implemented on 2D images to remove unwanted objects, and then the image-based 3D modeling technique was applied to create a 3D model that did not contain any unwanted objects. The results show that the final 3D model is more accurate when applied the object removal process and the model provides better visualization for heritage structures.

Patch-Sparsity-Based Image Inpainting Through a Facet Deduced Directional Derivative

This paper presents a patch-sparsity-based image inpainting algorithm through a facet deduced directional derivative. The algorithm could ensure the continuity of boundaries of the inpainted region and achieve a better performance on restoring the missing structure of an image. In this paper, two improvements are proposed. First, the facet model is introduced to get direction features of the image, which could efficiently reduce the effect of noises. The first-order directional derivatives, along with pixel values, are used to measure the difference between patches. Consequently, a more reliable and accurate matching result is promised. At the same time, the local patch consistency constraint of sparse representation of the target patch is also rewritten in the form of the first-order directional derivative. Therefore, a more precise sparse linear combination could be obtained under constraints for both color and derivative information. Second, the value of patch confidence in the traditional exemplar-based inpainting algorithms drops sharply in the late stage so that the data term or structure sparsity has little influence on priority function. Aiming at this problem, the algorithm makes a modification to the calculating of priority. Thus, the filling order decided by priority function appears more reasonable as a result of a better balance between the values of modified confidence and structure sparsity. Experiments on different types of damages to images show the superiority of the algorithm.

View synthesis using foreground object extraction for disparity control and image inpainting

Among the rapidly growing three-dimensional technologies, multiview displays have drawn great research interests in three-dimensional television due to their adaption to the motion parallax and wider viewing angles. However, multiview displays still suffer from dazzling discomfort on the border of viewing zones. Leveraging on the separability of scene via foreground segmentation, we propose a novel virtual view synthesis method for depth-image-based rendering to alleviate the discomfort. Foreground objects of interest are extracted to segment the whole image into multiple layers, which are further warped to the virtual viewpoint in order. To alleviate the visual discomfort, global disparity adjustments and local depth control are performed for specific objects in each layer. For the post-processing, we improve an exemplar-based inpainting algorithm to tackle the disoccluded areas. Experimental results demonstrate that our method achieves effective disparity control and generates high-quality virtual view images.

Average-face-based virtual inpainting for severely damaged statues of Dazu Rock Carvings

Abstract Numerous image inpainting algorithms are guided by a basic assumption that the known region in the original image itself can provide sufficient prior information for the guess recovery of the unknown part, which is not often the case in actual art image inpainting. In order to solve the challenging inpainting case that there is little image prior in the remainder of the original image, we propose an average-face-based inpainting method based on a sample database with 3 steps: reference images selection, average image generation and exemplar-based image inpainting. In which, average image generation is crucial. In the inpainting framework, the average image can be directly viewed as an inpainting proposal for the severely damaged or absolutely lost image. Moreover, the average image can be applied to exemplar-based inpainting algorithm as a sample image to extend the searching region for match patch, so as to perform the restoration for images with large-scale or irregularly damaged holes. The inpainting experiments over some facial images of Dazu Rock Carvings demonstrate the validity and effectiveness of our method. It is first utilized for two extremely challenging inpainting tasks: reconstruction for the stolen head of Willow Avalokitesvara in Shimenshan No. 6 and the absolutely broken heads of two Avalokitesvaras in Beishan No. 180. Compared to the failure of the exemplar-based inpainting algorithm within the original image and the directly duplication of a similar image, the generated average image can be as a more reliable inpainting proposal. The comparative experiments also show the efficiency and advantage of the average face applied to exemplar-based inpainting framework. Compared with some related inpainting algorithms, our method is more competitive when there is little prior information in the original image. The efficient virtual inpainting results are valuable references for both cave art historians and conservators.

Bodhisattva head images modeling style recognition of Dazu Rock Carvings based on deep convolutional network

Abstract As the only World Culture Heritage in Chongqing, the Dazu Rock Carvings have been suffering from natural environment erosion for almost one thousand years, so the inpainting of these carvings is desired. In order to reduce the inpainting risk and keep the carvings’ original appearance, it is necessary to introduce digital image processing techniques to perform virtual inpainting initially. The key step in exemplar-based inpainting algorithm is to search for the most similar patch. Efficient modeling style recognition is therefore the basis of the carvings’ inpainting. Focusing on Bodhisattva head images, this paper proposes a two-step recognition method for their modeling style: feature extraction utilizing VGGNet 1 and clustering with K-means algorithm. The proposed method obtains promising results compared with 5 classical feature extraction algorithms. According to experiment results, combining both art archaeology and image analysis, we conclude: (1) the modeling style is similar for the statues in the same cave or region; and the modeling style of the statues on the same subject is also similar, even though they are in different caves or regions; (2) the name of Beishan No 180 should be “Cave of Eleven Incarnations of Avalokitesvara” instead of the previous “Cave of Thirteen Incarnations of Avalokitesvara”. It is because the modeling style of two statues behind the major statue is quite different from the others. They were probably carved to substitute the two outmost damaged statues. Our work can be viewed as a reference to solve some art problems. Moreover, as it is efficient to search for similar images by our method, this paper can be as the basis for virtual exemplar-based inpainting of Dazu Rock Carvings in our future work.

Structure-aware image inpainting using patch scale optimization

Image inpainting is widely used in many image processing applications such as image stitching, image editing and object removal. The main challenge stems from producing visually plausible results after reconstruction. Most of the image inpainting algorithms cannot maintain structure continuity and texture consistency precisely. To address this problem, we propose a robust exemplar-based inpainting algorithm. Firstly, we present local structure multiplier to contain sufficient structure information in the priority function which ensures the structure continuity. Secondly, we combine color feature and space distance between two patches to search for the optimized patch to avoid texture inconsistency. At last, we calculate the average pixel difference between two patches under each candidate scale, we select the scale which the minimal average pixel difference is to be the optimized scale. We copy the target patch with the optimized patch. Extensive experiments show the effectiveness of the proposed method.

Hole-filling Based on Disparity Map and Inpainting for Depth-Image-Based Rendering

Due to the changes in viewpoint, holes may appear in the novel view synthesized by depth-image-based rendering (DIBR). A hole-filling method combining disparity-map-based hole-filling and inpainting is proposed. The method first eliminates matching errors in destination image according to the disparity map. Then, holes are classified into different types according to their sizes. Finally, a disparity-map-based approach and an improved exemplar-based inpainting algorithm are used to fill different types of holes according to the type of hole. Experimental results show that the artifacts at the edge of foreground objects can be reduced in synthesized view since the matching errors are eliminated before hole-filling. In addition, the proposed method can achieve more natural and satisfactory results in filled areas in comparison with the disparity-map-based approach and Gautier’s inpainting algorithm, though it may result in higher time complexity.

A color-gradient patch sparsity based image inpainting algorithm with structure coherence and neighborhood consistency

To better maintain structure coherence and neighborhood consistency, an exemplar-based inpainting algorithm is presented by introducing color-gradient patch sparsity (CGPS). Two concepts of CGPS including color-gradient structure sparsity (CGSS) and patch sparse representation are proposed to obtain the filling order, the search region size and the sparse representation of target patch, which are key steps in an exemplar-based inpainting algorithm. Firstly, the CGSS is designed based on weighted color-gradient distance (WCGD) to determine the filling order of all patches located at fill-front. Secondly, the WCGD is applied to search candidate patches and the CGSS is used to limit the search region size. Thirdly, the patch to be filled is sparsely represented under the local patch consistency constraints in color and gradient spaces. Differing from the exemplar-based inpainting approaches in which only color information is used, the proposed algorithm considers both color and gradient information, which ensures a better maintenance of structure coherence, texture clarity and neighborhood consistency. Moreover, the inpainting efficiency can be significantly improved by limiting the search region size via the CGSS. Experimental results on natural images are presented to demonstrate the advantages of the proposed approach for various tasks such as scratch removal, text removal, block removal and object removal.

On consistent inter-view synthesis for autostereoscopic displays

In this paper we present a novel stereo view synthesis algorithm that is highly accurate with respect to inter-view consistency, thus to enabling stereo contents to be viewed on the autostereoscopic displays. The algorithm finds identical occluded regions within each virtual view and aligns them together to extract a surrounding background layer. The background layer for each occluded region is then used with an exemplar based inpainting method to synthesize all virtual views simultaneously.Our algorithm requires the alignment and extraction of background layers for each occluded region; however, these two steps are done efficiently with lower computational complexity in comparison to previous approaches using the exemplar based inpainting algorithms. Thus, it is more efficient than existing algorithms that synthesize one virtual view at a time.This paper also describes the implementation of a simplified GPU accelerated version of the approach and its implementation in CUDA. Our CUDA method has sublinear complexity in terms of the number of views that need to be generated, which makes it especially useful for generating content for autostereoscopic displays that require many views to operate.An objective of our work is to allow the user to change depth and viewing perspective on the fly. Therefore, to further accelerate the CUDA variant of our approach, we present a modified version of our method to warp the background pixels from reference views to a middle view to recover background pixels. We then use an exemplar based inpainting method to fill in the occluded regions. We use warping of the foreground from the reference images and background from the filled regions to synthesize new virtual views on the fly. Our experimental results indicate that the simplified CUDA implementation decreases running time by orders of magnitude with negligible loss in quality.

An Image Completion Method Using Decomposition

This study presents a hybrid technique for simultaneously completing images by using geometry and texture components of input data. The approaches using inpainting methods based on partial differential equations (PDEs) to fill in large image regions usually fail if these regions contain textures. On the other hand, texture synthesis algorithms sometimes fail due to complex structures and textures in the image. However, this study, suggesting a hybrid method using both techniques, produces satisfactory results in completing the missing parts of images. In the proposed method, the given image is decomposed into two components. The geometry component, obtained by using the regularization PDE based on a trace operator, was inpainted by a tensor-driven PDE algorithm that takes curvatures of line integral curves into account, and the texture component, obtained by subtracting the given image from the geometry component, was reconstructed by the modified exemplar-based inpainting algorithm. Both of these methods work on color information. The main contribution of this paper is that it uses decomposition and montage stages together which provides superior results compared with the existing methods. Experimental results show that the proposed method efficiently fills in target regions, which is promising.

Improved exemplar-based inpainting algorithm for broken Thangka images

The algorithm of exemplar-based image inpainting was introduced to the digital protection of Thangka image due to the advantage that can repair the broken structure and texture effectively at the same time.This algorithm is good at repairing the specific broken images of Thangka,but is not appropriate to other kinds of broken images.Hence,this paper proposed two aspects of improvement for the deficiencies of exemplar-based broken Thangka image inpainting:the improvement of computing method of confidence and the improvement of computing method of isophote intensity.The problem that optimum exemplar block is not exclusive has been solved.The experimental results show that the improved algorithm can not only get a satisfied inpainting result but also improve repairing efficiency.