Exemplar Texture Research Articles

Arbitrary-Scale Texture Generation From Coarse-Grained Control.

Existing deep-network based texture synthesis approaches all focus on fine-grained control of texture generation by synthesizing images from exemplars. Since the networks employed by most of these methods are always tied to individual exemplar textures, a large number of individual networks have to be trained when modeling various textures. In this paper, we propose to generate textures directly from coarse-grained control or high-level guidance, such as texture categories, perceptual attributes and semantic descriptions. We fulfill the task by parsing the generation process of a texture into the three-level Bayesian hierarchical model. A coarse-grained signal first determines a distribution over Markov random fields. Then a Markov random field is used to model the distribution of the final output textures. Finally, an output texture is generated from the sampled Markov random field distribution. At the bottom level of the Bayesian hierarchy, the isotropic and ergodic characteristics of the textures favor a construction that consists of a fully convolutional network. The proposed method integrates texture creation and texture synthesis into one pipeline for real-time texture generation, and enables users to readily obtain diverse textures with arbitrary scales from high-level guidance only. Extensive experiments demonstrate that the proposed method is capable of generating plausible textures that are faithful to user-defined control, and achieving impressive texture metamorphosis by interpolation in the learned texture manifold.

Patch Erosion for Deformable Lapped Textures on 3D Fluids

AbstractWe propose an approach to synthesise a texture on an animated fluid free surface using a distortion metric combined with a feature map. Our approach is applied as a post‐process to a fluid simulation. We advect deformable patches to move the texture along the fluid flow. The patches are covering the whole surface every frame of the animation in an overlapping fashion. Using lapped textures combined with deformable patches, we successfully remove blending artifact and rigid artifact seen in previous methods. We remain faithful to the texture exemplar by removing distorted patch texels using a patch erosion process. The patch erosion is based on a feature map provided together with the exemplar as inputs to our approach. The erosion favors removing texels toward the boundary of the patch as well as texels corresponding to more distorted regions of the patch. Where texels are removed leaving a gap on the surface, we add new patches below existing ones. The result is an animated texture following the velocity field of the fluid. We compared our results with recent work and our results show that our approach removes ghosting and temporal fading artifacts.

Conditional Generative ConvNets for Exemplar-Based Texture Synthesis.

The goal of exemplar-based texture synthesis is to generate texture images that are visually similar to a given exemplar. Recently, promising results have been reported by methods relying on convolutional neural networks (ConvNets) pretrained on large-scale image datasets. However, these methods have difficulties in synthesizing image textures with non-local structures and extending to dynamic or sound textures. In this article, we present a conditional generative ConvNet (cgCNN) model which combines deep statistics and the probabilistic framework of generative ConvNet (gCNN) model. Given a texture exemplar, cgCNN defines a conditional distribution using deep statistics of a ConvNet, and synthesizes new textures by sampling from the conditional distribution. In contrast to previous deep texture models, the proposed cgCNN does not rely on pre-trained ConvNets but learns the weights of ConvNets for each input exemplar instead. As a result, cgCNN can synthesize high quality dynamic, sound and image textures in a unified manner. We also explore the theoretical connections between our model and other texture models. Further investigations show that the cgCNN model can be easily generalized to texture expansion and inpainting. Extensive experiments demonstrate that our model can achieve better or at least comparable results than the state-of-the-art methods.

Deep Texture Exemplar Extraction Based on Trimmed T-CNN

Texture exemplar has been widely used in synthesizing 3D movie scenes and appearances of virtual objects. Unfortunately, conventional texture synthesis methods usually only emphasized on generating optimal target textures with arbitrary sizes or diverse effects, and put little attention to automatic texture exemplar extraction. Obtaining texture exemplars is still a labor intensive task, which usually requires carefully cropping and post-processing. In this paper, we present an automatic texture exemplar extraction based on Trimmed Texture Convolutional Neural Network (Trimmed T-CNN). Specifically, our Trimmed T-CNN is filter banks for texture exemplar classification and recognition. Our Trimmed T-CNN is learned with a standard ideal exemplar dataset containing thousands of desired texture exemplars, which were collected and cropped by our invited artists. To efficiently identify the exemplar candidates from an input image, we employ a selective search algorithm to extract the potential texture exemplar patches. We then put all candidates into our Trimmed T-CNN for learning ideal texture exemplars based on our filter banks. Finally, optimal texture exemplars are identified with a scoring and ranking scheme. Our method is evaluated with various kinds of textures and user studies. Comparisons with different feature-based methods and different deep CNN architectures (AlexNet, VGG-M, Deep-TEN and FV-CNN) are also conducted to demonstrate its effectiveness.

A novel framework for inverse procedural texture modeling

Procedural textures are powerful tools that have been used in graphics for decades. In contrast to the alternative exemplar-based texture synthesis techniques, procedural textures provide user control and fast texture generation with low-storage cost and unlimited texture resolution. However, creating procedural models for complex textures requires a time-consuming process of selecting a combination of procedures and parameters. We present an example-based framework to automatically select procedural models and estimate parameters. In our framework, we consider textures categorized by commonly used high level classes. For each high level class we build a data-driven inverse modeling system based on an extensive collection of real-world textures and procedural texture models in the form of node graphs. We use unsupervised learning on collected real-world images in a texture class to learn sub-classes. We then classify the output of each of the collected procedural models into these sub-classes. For each of the collected models we train a convolutional neural network (CNN) to learn the parameters to produce a specific output texture. To use our framework, a user provides an exemplar texture image within a high level class. The system first classifies the texture into a sub-class, and selects the procedural models that produce output in that sub-class. The pre-trained CNNs of the selected models are used to estimate the parameters of the texture example. With the predicted parameters, the system can generate appropriate procedural textures for the user. The user can easily edit the textures by adjusting the node graph parameters. In a last optional step, style transfer augmentation can be applied to the fitted procedural textures to recover details lost in the procedural modeling process. We demonstrate our framework for four high level classes and show that our inverse modeling system can produce high-quality procedural textures for both structural and non-structural textures.

Distribution Update of Deformable Patches for Texture Synthesis on the Free Surface of Fluids

AbstractWe propose an approach for temporally coherent patch‐based texture synthesis on the free surface of fluids. Our approach is applied as a post‐process, using the surface and velocity field from any fluid simulator. We apply the texture from the exemplar through multiple local mesh patches fitted to the surface and mapped to the exemplar. Our patches are constructed from the fluid free surface by taking a subsection of the free surface mesh. As such, they are initially very well adapted to the fluid's surface, and can later deform according to the free surface velocity field, allowing a greater ability to represent surface motion than rigid or 2D grid‐based patches. From one frame to the next, the patch centers and surrounding patch vertices are advected according to the velocity field. We seek to maintain a Poisson disk distribution of patches, and following advection, the Poisson disk criterion determines where to add new patches and which patches should e flagged for removal. The removal considers the local number of patches: in regions containing too many patches, we accelerate the temporal removal. This reduces the number of patches while still meeting the Poisson disk criterion. Reducing areas with too many patches speeds up the computation and avoids patch‐blending artifacts. The final step of our approach creates the overall texture in an atlas where each texel is computed from the patches using a contrast‐preserving blending function. Our tests show that the approach works well on free surfaces undergoing significant deformation and topological changes. Furthermore, we show that our approach provides good results for many fluid simulation scenarios, and with many texture exemplars. We also confirm that the optical flow from the resulting texture matches the fluid velocity field. Overall, our approach compares favorably against recent work in this area.

Synthesis of exemplar textures by self-similarity matching

Texture synthesis is proposed to construct a large digital image from a small sample by taking advantage of its structural content. However, many approaches in texture synthesis are based on the presence of broken features at the overlap of adjacent patches. Due to inaccurate similarity measures, several optimization schemes for patch mergence may fail if these schemes cannot find satisfactory candidates from input samples. Self-similarity of candidate patches was proposed, and an algorithm was developed to perform distance error matching and alignment via the sum of self-similarity. First, when the matching window moves in the texture samples along the scan line, the sum of self-similarity replaces the sum-of-squared difference to decrease broken features or texels of outputs. Next, the synthesis speed was accelerated by eliminating redundancy calculation during matching window sliding. Finally, the proposed method enlarges the search range of the suture from one patch to all patches in the horizontal direction, and the broken features of overlaps are eliminated in the outputs. To demonstrate the performance of the proposed method, the obtained synthesis results were compared with other conventional synthesis methods. In all cases, experimental results demonstrate that the self-similarity matching-based method can decrease the number of feature discontinuities and accelerate synthesis time.

Learning the Synthesizability of Dynamic Texture Samples.

Exemplar-based dynamic texture synthesis (EDTS) is targeted to generate new samples of high quality that are perceptually similar to a given input dynamic texture exemplar. This paper addresses the issue of learning the synthesizability of dynamic texture samples. Given a dynamic texture sample, how is its possibility of being synthesized by EDTS methods estimated, and what is the most suitable EDTS algorithm to complete the task? To this end, we propose associating dynamic texture samples with synthesizability scores by learning regression models on a compiled dynamic texture dataset annotated in terms of synthesizability. More precisely, we first define the synthesizability of DT samples and characterize them by a set of spatiotemporal features. We then train regression models on the annotated dataset with feature representation to predict the synthesizability scores of the DT samples and learn classifiers to select the most suitable EDTS algorithm. We further complete the selection, partition and synthesizability prediction of the DT samples in a hierarchical scheme. The learned synthesizability is finally applied to detecting synthesizable regions in videos. Both quantitative and qualitative experiments demonstrate that our method can efficiently learn and predict the synthesizability of DT samples.

Non-Local Texture Optimization With Wasserstein Regularization Under Convolutional Neural Network

Example-based texture synthesis aims to generate a new texture from an exemplar texture and has long been drawing attention in the fields of computer graphics, computer vision, and image processing. Nevertheless, synthesizing structured textures remains a challenging task. Most previous methods rely on additional guidance channels, which encode the structured features of textures. However, estimating the guidance channel is very difficult, and often fails when a texture has unpronounced features. In this paper, we propose a novel texture synthesis method, based on non-local operators, which captures the long-range structure of a texture without the additional guidance channel. The synthesized texture is generated by minimizing non-local texture energy through an expectation–maximization like optimization algorithm. A statistical constraint based on the Wasserstein distance is also proposed to ensure that the synthesized texture preserves the global statistics of the exemplar texture. Extensive experiments show that the proposed method can stably handle textures with different scale structures.

Texture-guided generative structural designs under local control

A novel generative design approach is developed in this study, which produces a mechanically optimized topological design while simultaneously mimicking an input exemplar texture. The textures are believed embedding certain functional information intrinsic to these objects. Designing objects similar to these textures will not only maintain such functions within the design but also widely expand the design space to explore more design options. However, simple textural replications or reconstructions cannot produce expected designs as an ideal structure has to adapt to the variations of the complex stress distributions caused by external loadings. On the other hand, a simple topology optimization formulation under a single global similarity constraint may produce undesirable structures exhibiting geometric disconnections or boundary protrusions.Due to these considerations, the proposed approach carefully formulates the problem as a classical topological compliance minimization problem under block-wise similarity constraints between the target design and an input texture. In addition, a novel physics-adaptive regulator is also proposed, which fine-tunes the block similarity according to its per-element compliances. Ultimately, we can create a set of design options both physically optimized and geometrically smooth for generative design. Extensive numerical results were also tested to demonstrate the approach’s performance.

Light Field Coding With Field-of-View Scalability and Exemplar-Based Interlayer Prediction

Light field imaging based on microlens arrays-a.k.a. holoscopic, plenoptic, and integral imaging-has currently risen up as a feasible and prospective technology for future image and video applications. However, deploying actual light field applications will require identifying more powerful representations and coding solutions that support arising new manipulation and interaction functionalities. In this context, this paper proposes a novel scalable coding solution that supports a new type of scalability, referred to as field-of-view scalability. The proposed scalable coding solution comprises a base layer compliant with the High Efficiency Video Coding (HEVC) standard, complemented by one or more enhancement layers that progressively allow richer versions of the same light field content in terms of content manipulation and interaction possibilities. In addition, to achieve high-compression performance in the enhancement layers, novel exemplar-based interlayer coding tools are also proposed, namely: 1) a direct prediction based on exemplar texture samples from lower layers and 2) an interlayer compensated prediction using a reference picture that is built relying on an exemplar-based algorithm for texture synthesis. Experimental results demonstrate the advantages of the proposed scalable coding solution to cater to users with different preferences/requirements in terms of interaction functionalities, while providing better rate-distortion performance (independently of the optical setup used for acquisition) compared to HEVC and other scalable light field coding solutions in the literature.

Non-stationary texture synthesis by adversarial expansion

The real world exhibits an abundance of non-stationary textures. Examples include textures with large scale structures, as well as spatially variant and inhomogeneous textures. While existing example-based texture synthesis methods can cope well with stationary textures, non-stationary textures still pose a considerable challenge, which remains unresolved. In this paper, we propose a new approach for example-based non-stationary texture synthesis. Our approach uses a generative adversarial network (GAN), trained to double the spatial extent of texture blocks extracted from a specific texture exemplar. Once trained, the fully convolutional generator is able to expand the size of the entire exemplar, as well as of any of its sub-blocks. We demonstrate that this conceptually simple approach is highly effective for capturing large scale structures, as well as other non-stationary attributes of the input exemplar. As a result, it can cope with challenging textures, which, to our knowledge, no other existing method can handle.

Automatic texture exemplar extraction based on global and local textureness measures

Texture synthesis is widely used for modeling the appearance of virtual objects. However, traditional texture synthesis techniques emphasize creation of optimal target textures, and pay insufficient attention to choice of suitable input texture exemplars. Currently, obtaining texture exemplars from natural images is a labor intensive task for the artists, requiring careful photography and significant postprocessing. In this paper, we present an automatic texture exemplar extraction method based on global and local textureness measures. To improve the efficiency of dominant texture identification, we first perform Poisson disk sampling to randomly and uniformly crop patches from a natural image. For global textureness assessment, we use a GIST descriptor to distinguish textured patches from non-textured patches, in conjunction with SVM prediction. To identify real texture exemplars consisting solely of the dominant texture, we further measure the local textureness of a patch by extracting and matching the local structure (using binary Gabor pattern (BGP)) and dominant color features (using color histograms) between a patch and its sub-regions. Finally, we obtain optimal texture exemplars by scoring and ranking extracted patches using these global and local textureness measures. We evaluate our method on a variety of images with different kinds of textures. A convincing visual comparison with textures manually selected by an artist and a statistical study demonstrate its effectiveness.

A Texture Synthesis Model Based on Semi-Discrete Optimal Transport in Patch Space

Exemplar-based texture synthesis consists in producing new synthetic images which have the same perceptual characteristics as a given texture sample while exhibiting sufficient innovation (to avoid verbatim copy). In this paper, we propose to address this problem with a model obtained as local transformations of Gaussian random fields. The local transformations operate on $3 \times 3$ patches and are designed to solve a semi-discrete optimal transport problem in order to reimpose the patch distribution of the exemplar texture. The semi-discrete optimal transport problem is solved with a stochastic gradient algorithm, whose convergence speed is evaluated on several practical transport cases. After studying the properties of such transformed Gaussian random fields, we propose a multiscale extension of the model which aims at preserving the patch distribution of the exemplar texture at multiple scales. Experiments demonstrate that this multiscale model is able to synthesize structured textures while keeping several mathematical guarantees, and with low requirements in synthesis time and memory storage. In particular, a single patch optimal transport map is shown to be better than iterated nearest neighbor assignments in terms of statistical guarantees. Besides, once the model is estimated, the resulting synthesis algorithm is fast and highly parallel since it amounts to performing weighted nearest neighbor patch assignments at each scale.

Analysis and Controlled Synthesis of Inhomogeneous Textures

Many interesting real-world textures are inhomogeneous and/or anisotropic. An inhomogeneous texture is one where various visual properties exhibit significant changes across the texture's spatial domain. Examples include perceptible changes in surface color, lighting, local texture pattern and/or its apparent scale, and weathering effects, which may vary abruptly, or in a continuous fashion. An anisotropic texture is one where the local patterns exhibit a preferred orientation, which also may vary across the spatial domain. While many example-based texture synthesis methods can be highly effective when synthesizing uniform stationary isotropic textures, synthesizing highly non-uniform textures, or ones with spatially varying orientation, is a considerably more challenging task, which so far has remained underexplored. In this paper, we propose a new method for automatic analysis and controlled synthesis of such textures. Given an input texture exemplar, our method generates a source guidance map comprising: i a scalar progression channel that attempts to capture the low frequency spatial changes in color, lighting, and local pattern combined, and ii a direction field that captures the local dominant orientation of the texture. Having augmented the texture exemplar with this guidance map, users can exercise better control over the synthesized result by providing easily specified target guidance maps, which are used to constrain the synthesis process.

By-example synthesis of structurally sound patterns

Several techniques exist to automatically synthesize a 2D image resembling an input exemplar texture. Most of the approaches optimize a new image so that the color neighborhoods in the output closely match those in the input, across all scales. In this paper we revisit by-example texture synthesis in the context of additive manufacturing. Our goal is to generate not only colors, but also structure along output surfaces: given an exemplar indicating 'solid' and 'empty' pixels, we generate a similar pattern along the output surface. The core challenge is to guarantee that the pattern is not only fully connected, but also structurally sound. To achieve this goal we propose a novel formulation for on-surface by-example texture synthesis that directly works in a voxel shell around the surface. It enables efficient local updates to the pattern, letting our structural optimizer perform changes that improve the overall rigidity of the pattern. We use this technique in an iterative scheme that jointly optimizes for appearance and structural soundness. We consider fabricability constraints and a user-provided description of a force profile that the object has to resist. Our results fully exploit the capabilities of additive manufacturing by letting users design intricate structures along surfaces. The structures are complex, yet they resemble input exemplars, resulting in a modeling tool accessible to casual users.

PatchTable

This paper presents a data structure that reduces approximate nearest neighbor query times for image patches in large datasets. Previous work in texture synthesis has demonstrated real-time synthesis from small exemplar textures. However, high performance has proved elusive for modern patch-based optimization techniques which frequently use many exemplar images in the tens of megapixels or above. Our new algorithm, PatchTable, offloads as much of the computation as possible to a pre-computation stage that takes modest time, so patch queries can be as efficient as possible. There are three key insights behind our algorithm: (1) a lookup table similar to locality sensitive hashing can be precomputed, and used to seed sufficiently good initial patch correspondences during querying, (2) missing entries in the table can be filled during pre-computation with our fast Voronoi transform, and (3) the initially seeded correspondences can be improved with a precomputed k-nearest neighbors mapping. We show experimentally that this accelerates the patch query operation by up to 9× over k-coherence, up to 12× over TreeCANN, and up to 200× over PatchMatch. Our fast algorithm allows us to explore efficient and practical imaging and computational photography applications. We show results for artistic video stylization, light field super-resolution, and multi-image editing.

Unsupervised Texture Flow Estimation Using Appearance-Space Clustering and Correspondence.

This paper presents a texture flow estimation method that uses an appearance-space clustering and a correspondence search in the space of deformed exemplars. To estimate the underlying texture flow, such as scale, orientation, and texture label, most existing approaches require a certain amount of user interactions. Strict assumptions on a geometric model further limit the flow estimation to such a near-regular texture as a gradient-like pattern. We address these problems by extracting distinct texture exemplars in an unsupervised way and using an efficient search strategy on a deformation parameter space. This enables estimating a coherent flow in a fully automatic manner, even when an input image contains multiple textures of different categories. A set of texture exemplars that describes the input texture image is first extracted via a medoid-based clustering in appearance space. The texture exemplars are then matched with the input image to infer deformation parameters. In particular, we define a distance function for measuring a similarity between the texture exemplar and a deformed target patch centered at each pixel from the input image, and then propose to use a randomized search strategy to estimate these parameters efficiently. The deformation flow field is further refined by adaptively smoothing the flow field under guidance of a matching confidence score. We show that a local visual similarity, directly measured from appearance space, explains local behaviors of the flow very well, and the flow field can be estimated very efficiently when the matching criterion meets the randomized search strategy. Experimental results on synthetic and natural images show that the proposed method outperforms existing methods.

On Neighbourhood Matching for Texture-by-Numbers

Texture-by-Numbers is an attractive texture synthesis framework, because it is able to cope with non-homogeneous texture exemplars, and provides the user with intuitive creative control over the outcome of the synthesis process. Like many other exemplar-based texture synthesis methods, its basic underlying mechanism is neighbourhood matching. In this paper we review a number of commonly used neighbourhood matching acceleration techniques, compare and analyse their performance in the specific context of Texture-by-Numbers (as opposed to ordinary unconstrained texture synthesis). Our study indicates that the standard approaches are not optimally suited for the Texture-by-Numbers framework, often producing visually inferior results compared to searching for the exact L2nearest neighbour. We then show that performing Texture-by-Number using the Texture Optimization framework in conjunction with an efficient FFT-based search is able to produce good results in reasonable running times and with a minimal memory overhead.

Local resampling for patch-based texture synthesis in vector fields

We develop a direct and accurate approach for local resampling in vector fields, and then use the approach to synthesise textures on 2D manifold surfaces directly from a texture exemplar. Regular-grid patches produced by the local resampling are used as building blocks for texture synthesis. Then, texture optimisation and patch-based sampling are generalised to synthesise texture directly in vector fields. Users can control the vector field on the mesh to generate textures with local variations including the orientation and scale. Many experimental results are presented to demonstrate the ease of use and reliable results provided by our system.