Texture Synthesis Algorithm Research Articles

Interactive texture replacement of cartoon characters based on deep learning model

Abstract To understand the deep learning model, the author proposed the research of interactive texture replacement of cartoon characters. For image segmentation, if you want to fill a cartoon without any texture in detail, or replace the unsatisfied texture area, first, we need to separate the filled or replaced area from the cartoon. For this reason, the traditional image segmentation algorithm has been carefully studied and analyzed, and the author chooses the Graphcut texture synthesis algorithm, the algorithm is parallelized and improved, and the innovative point of lighting customization is proposed based on the original algorithm, which can affect the synthesis effect according to the input lighting image. In terms of timeliness and synthesis effect, the Graphcut algorithm has been improved. Experimental results show that the algorithm can maintain the brightness distribution of the original cartoon and the practicability and efficiency of the algorithm proposed by the author.

Development of Texture Mapping Approaches for Additively Manufacturable Surfaces

Additive manufacturing (AM) technologies have been recognized for their capability to build complex components and hence have offered more freedom to designers for a long time. The ability to directly use a computer-aided design (CAD) model has allowed for fabricating and realizing complicated components, monolithic design, reducing the number of components in an assembly, decreasing time to market, and adding performance or comfort-enhancing functionalities. One of the features that can be introduced for boosting a component functionality using AM is the inclusion of surface texture on a given component. This inclusion is usually a difficult task as creating a CAD model resolving fine details of a given texture is difficult even using commercial software packages. This paper develops a methodology to include texture directly on the CAD model of a target surface using a patch-based sampling texture synthesis algorithm, which can be manufactured using AM. Input for the texture generation algorithm can be either a physical sample or an image with heightmap information. The heightmap information from a physical sample can be obtained by 3D scanning the sample and using the information from the acquired point cloud. After obtaining the required inputs, the patches are sampled for texture generation according to non-parametric estimation of the local conditional Markov random field (MRF) density function, which helps avoid mismatched features across the patch boundaries. While generating the texture, a design constraint to ensure AM producibility is considered, which is essential when manufacturing a component using, e.g., Fused Deposition Melting (FDM) or Laser Powder Bed Fusion (LPBF). The generated texture is then mapped onto the surface using the developed distance and angle preserving mapping algorithms. The implemented algorithms can be used to map the generated texture onto a mathematically defined surface. This paper maps the textures onto flat, curved, and sinusoidal surfaces for illustration. After the texture mapping, a stereolithography (STL) model is generated with the desired texture on the target surface. The generated STL model is printed using FDM technology as a final step.

Variety decorative bas‐relief generation based on normal prediction and transfer

AbstractAs the generation of realistic bas‐relief models from 2D images suffers from insufficient 3D depth information and severe under‐constraint, in this article, we propose a new framework for bas‐relief modeling based on 2D decorative images, which adopts conditional generative adversarial network to infer the normal information of the decorative bas‐reliefs from the greyscale information extracted from the images. For the variety of models, we extract the internal structure information through the saliency detection method based on scene perception, and use the transfer process based on the optimized texture synthesis algorithm to complete the normal editing from the source normal map to the new one, which can diversify and control the structure and detailed information of existed normal map. Finally, we adopt a bas‐relief reconstruction approach based on domain transfer recursive filter and surface from gradient to recover 2.5D information from predicted and transferred normal maps. Experiments on various model examples demonstrate the efficiency and diversity of the proposed method in reconstructing bas‐relief models from a single decorative image.

Deep Learning in Computer Real-time Graphics and Image Using the Visual Effects of Non-photorealistic Rendering of Ink Painting

This study is to explore the effect of deep learning algorithms applied to the non-photorealistic rendering (NPR) of ink painting. The powerful processing capabilities of NPR technology and convolutional neural networks (CNN) in computer vision are analyzed. Aiming at the features of ink texture images, a new sample-based ink texture synthesis method is proposed. The feature representation of ink texture is calculated using the CNN model, and then an ink texture with the same feature representation is generated on a random image. In addition, a synthesis method of ink painting picture is proposed based on the CNN. The results show that the ink texture generated by the ink texture synthesis algorithm is very good, and the Visual Geometry Group Network (VGG-Net) model shows the best generation effect. For images generated by high-level networks lack a lot of pixel information, but retain the main feature information of the image. When the learning rate is 10, the effect of the ink image synthesized based on the content image is slightly worse; and the image generated by the fusion of multiple styles confirms the feasibility of synthesizing ink images of different "sizes and styles". Therefore, this ink image synthesized on the basis of the CNN model shows a better effect, which extends the possibility of computer creation of richer ink painting works.

Calcium functional imaging with high-resolution CT in the inner ear

Although the otolith and otolith organs correlate with vertigo and instability, there is no method to investigate them without harmful procedures. We will create the technique for 3D microanatomical images of them, and investigate the in vivo internal state and metabolisms. The otolith and otolith organs images were reconstructed from a texture synthesis algorithm under the skull volume rendering algorithm using a cutting-plane method. The utricular macula was elongated pea-shaped. The saccular macula was almost bud-shaped. The changes in the amount of CaCO3 in the maculae and the endolymphatic sac showed various morphologies, reflecting the balance status of each subject. Both shapes and volumes were not always constant depending on time. In Meniere’s disease (MD), the saccular macula was larger and the utricular macula was smaller. In benign paroxysmal positional vertigo (BPPV), the otolith increased in the utricular macula but did not change much in the saccular macula. The saccule, utricle, and endolymphatic sac were not constantly shaped according to their conditions. These created 3D microanatomical images can allow detailed observations of changes in physiological and biological information. This imaging technique will contribute to our understanding of pathology and calcium metabolism in the in vivo vestibulum.

The Portilla-Simoncelli Texture Model: towards Understanding the Early Visual Cortex

Texture synthesis is a prolific subarea in computer vision where statistical methods are often successful. The Portilla and Simoncelli (PS) texture algorithm is one of such methods that became very popular and has influenced visual perception studies. For many reasons it can still be considered as a state-of-the art texture synthesis algorithm: (i) it generates textures that are often indistinguishable from the original without scrutiny; (ii) it relies on few parameters compared to recent deep learning methods; (iii) recent algorithms often compare to it. Here, we review the scientific impact of this algorithm and give a detailed explanation. Briefly, the PS algorithm synthesizes a new texture by iteratively imposing to a Gaussian white noise image a set of high-order statistics of wavelet coefficients precomputed on a texture example. After few iterations the initial white noise image is transformed into a texture that is similar to the texture example. We provide a fast C++ implementation, evaluate the effect of the algorithm parameters and illustrate its capabilities with many synthesis examples. In addition, we propose two notable new features to the original implementation: (i) the possibility to interpolate between two textures; (ii) the possibility to handle non-periodicity using the 'periodic+smooth' decomposition.

The roles of lower- and higher-order surface statistics in tactile texture perception.

Humans can haptically discriminate surface textures when there is a significant difference in the statistics of the surface profile. Previous studies on tactile texture discrimination have emphasized the perceptual effects of lower-order statistical features such as carving depth, inter-ridge distance, and anisotropy, which can be characterized by local amplitude spectra or spatial-frequency/orientation subband histograms. However, the real-world surfaces we encounter in everyday life also differ in the higher-order statistics, such as statistics about correlations of nearby spatial-frequencies/orientations. For another modality, vision, the human brain has the ability to use the textural differences in both higher- and lower-order image statistics. In this work, we examined whether the haptic texture perception can use higher-order surface statistics as visual texture perception does, by three-dimensional (3-D)-printing textured surfaces transcribed from different "photos" of natural scenes such as stones and leaves. Even though the maximum carving depth was well above the haptic detection threshold, some texture pairs were hard to discriminate. Specifically, those texture pairs with similar amplitude spectra were difficult to discriminate, which suggests that the lower-order statistics have the dominant effect on tactile texture discrimination. To directly test the poor sensitivity of the tactile texture perception to higher-order surface statistics, we matched the lower-order statistics across different textures using a texture synthesis algorithm and found that haptic discrimination of the matched textures was nearly impossible unless the stimuli contained salient local features. We found no evidence for the ability of the human tactile system to use higher-order surface statistics for texture discrimination.NEW & NOTEWORTHY Humans can discriminate subtle spatial patterns differences in the surrounding world through their hands, but the underlying computation remains poorly understood. Here, we 3-D-printed textured surfaces and analyzed the tactile discrimination performance regarding the sensitivity to surface statistics. The results suggest that observers have sensitivity to lower-order statistics whereas not to higher-order statistics. That is, touch differs from vision not only in spatiotemporal resolution but also in (in)sensitivity to high-level surface statistics.

Image reconstruction using the modified texture synthesis algorithm

The article discusses a method for image reconstruction based on the search for similar blocks using a texture synthesis algorithm. The effectiveness of the new approach is shown using several examples for various areas with lost pixels. The subject of the research is methods and algorithms for processing space-time reconstruction of two-dimensional signals based on a geometric model of images. The object of research is a set of test static images. The result of the study is a modification of the image reconstruction method based on the search for similar blocks in order to reduce the error in image reconstruction. The novelty of the work is an algorithm that improves the quality of image restoration. The results obtained make it possible to reduce the root mean square error.

Texture Synthesis Repair of RealSense D435i Depth Images with Object-Oriented RGB Image Segmentation.

A depth camera is a kind of sensor that can directly collect distance information between an object and the camera. The RealSense D435i is a low-cost depth camera that is currently in widespread use. When collecting data, an RGB image and a depth image are acquired simultaneously. The quality of the RGB image is good, whereas the depth image typically has many holes. In a lot of applications using depth images, these holes can lead to serious problems. In this study, a repair method of depth images was proposed. The depth image is repaired using the texture synthesis algorithm with the RGB image, which is segmented through a multi-scale object-oriented method. The object difference parameter is added to the process of selecting the best sample block. In contrast with previous methods, the experimental results show that the proposed method avoids the error filling of holes, the edge of the filled holes is consistent with the edge of RGB images, and the repair accuracy is better. The root mean square error, peak signal-to-noise ratio, and structural similarity index measure from the repaired depth images and ground-truth image were better than those obtained by two other methods. We believe that the repair of the depth image can improve the effects of depth image applications.

Ca functional imaging from the cerebellum to the vestibular end‐organs

ObjectivesThe calcium ion plays a major role in the inner ear and the cerebellum. But in vivo calcium metabolism is still unknown problems in these areas. Though, it is said that the endolymphatic sac and the cerebellum have the functions of calcium metabolism, no one directly detects them in vivo. The purpose of this study is to create in vivo functional Ca imaging of calcium metabolism in the cerebellum and the vestibular end‐organs.Subjects and MethodHigh‐resolution temporal bone CT data were used as follows; 5 normal subjects, 5 Meniere’s diseases (MD), and 5 woozy subjects. The 3D imaging technique was employed in the specific volume rendering based on the bone algorithms combining thresholding with a texture synthesis algorithm. This created algorithm can reconstruct images of calcium carbonate and calcium phosphate in the temporal bone region and the cerebellum.ResultsThe cerebellum showed two kinds of calcification spots. One was the same color as calcium carbonate of the otolith organs, and the other main component showed calcium phosphate like the color of the temporal bone. In the normal inner ear, both the otolith organs were created to normal size and shape from calcium carbonate images. The calcium carbonate spots in the cerebellum were increased in the normal aging subjects and the patients with MD and woozy. The endolymphatic sac function was activated in the woozy period. In MD, during the attack period, the calcium carbonate spots were increased in the cerebellum. In the recovery period, they were decreased. During the attack period, the saccular otolith organ was increased in volume, and the utricular otolith organ decreased. In the recovery period, the volume and shape of the saccular otolith organ decreased, and that of utricle was increased. However, in the age‐related changes, the calcium phosphate spots were increased in the cerebellum.ConclusionsThe 3D reconstructed images of calcium compounds in which the internal state of an in vivo subject can be observed in detail using bioinformation associated with CaCO3 and CaP. The in vivo internal state of the cerebellum and the vestibular end‐organs can be observed in detail using bioinformation related to calcium carbonate and calcium phosphate. To maintain balance, the cerebellar function seems to be active to compensate for the deterioration of the vestibular function. The researchers will be able to utilize this tool to analyze calcium metabolism at the temporal bone region and the cerebellum to enhance the understanding of the role they play in disease.Support or Funding InformationConflict of InterestAll authors of this study declare no relationship with any companies, whose products or services may be related to the subject matter of the article. Funding: All authors state that this work has not received any funding.The affected cerebellum shows multiple gray and white spots. However, the non‐affected side represents fewer than the affected side.Figure 1The both cerebellar hemispheres show a decreased in spots, and the affected utricular otolith organ increases in volume and shape.Figure 2

Haptic discrimination of 3D-printed patterns based on natural visual textures

We perceive spatial textures mainly through eyes and hands. In contrast to visual texture perception, computational mechanisms of haptic texture perception remain poorly understood. Here we measured haptic texture discrimination ability of 3D printed surfaces transcribed by visual images of five natural textures (stones, leaves etc.). For each texture image, the intensity map was converted into the 3D surface height modulation of a 40×40mm plate. The textures look sufficiently different and the maximum modulation depth (2mm) was well above the haptic detection threshold. Nevertheless, observers (n=10) could not accurately discriminate some texture pairs. In the main experiment, the observer passively touched the plate swept on their stabilized index finger (passive scan mode), but even when we tested other touching modes, the performance was improved only slightly (active scan), or not at all (static touch, vibration only). Since the amplitude spectra of natural visual textures are similar to each other (fall by a factor of f-a), we hypothesized that haptic texture discrimination may rely solely on the difference in amplitude spectra, or on the spatial-frequency/orientation subband histograms. In agreement with this, the discrimination performance we obtained could be explained by a multivariate linear regression based on the amplitude difference between the paired textures. In an additional experiment, we directly tested this hypothesis by matching the subband histogram of each texture using a texture synthesis algorithm (Heeger & Bergen, 1995). Haptic discrimination of these textures was found to be nearly impossible, although visual discrimination remains feasible due to differences in higher-order statistics (joint subband statistics to which V2 neurons are sensitive, or more complex phase information detectable by attentive foveal vision). These findings suggest that haptic texture processing may be qualitatively different from visual texture processing in that it simply relies on lower-order image statistics.

基于卷积神经网络和边缘检测的自然纹理合成算法

基于卷积神经网络和边缘检测的自然纹理合成算法

Light Field Coding With Field-of-View Scalability and Exemplar-Based Interlayer Prediction

Light field imaging based on microlens arrays-a.k.a. holoscopic, plenoptic, and integral imaging-has currently risen up as a feasible and prospective technology for future image and video applications. However, deploying actual light field applications will require identifying more powerful representations and coding solutions that support arising new manipulation and interaction functionalities. In this context, this paper proposes a novel scalable coding solution that supports a new type of scalability, referred to as field-of-view scalability. The proposed scalable coding solution comprises a base layer compliant with the High Efficiency Video Coding (HEVC) standard, complemented by one or more enhancement layers that progressively allow richer versions of the same light field content in terms of content manipulation and interaction possibilities. In addition, to achieve high-compression performance in the enhancement layers, novel exemplar-based interlayer coding tools are also proposed, namely: 1) a direct prediction based on exemplar texture samples from lower layers and 2) an interlayer compensated prediction using a reference picture that is built relying on an exemplar-based algorithm for texture synthesis. Experimental results demonstrate the advantages of the proposed scalable coding solution to cater to users with different preferences/requirements in terms of interaction functionalities, while providing better rate-distortion performance (independently of the optical setup used for acquisition) compared to HEVC and other scalable light field coding solutions in the literature.

Mid-level visual features underlie the high-level categorical organization of the ventral stream

Human object-selective cortex shows a large-scale organization characterized by the high-level properties of both animacy and object size. To what extent are these neural responses explained by primitive perceptual features that distinguish animals from objects and big objects from small objects? To address this question, we used a texture synthesis algorithm to create a class of stimuli-texforms-which preserve some mid-level texture and form information from objects while rendering them unrecognizable. We found that unrecognizable texforms were sufficient to elicit the large-scale organizations of object-selective cortex along the entire ventral pathway. Further, the structure in the neural patterns elicited by texforms was well predicted by curvature features and by intermediate layers of a deep convolutional neural network, supporting the mid-level nature of the representations. These results provide clear evidence that a substantial portion of ventral stream organization can be accounted for by coarse texture and form information without requiring explicit recognition of intact objects.

Modeling "Textured" Bones in Virtual Human Phantoms.

The purpose of this study was to develop detailed and realistic models of the cortical and trabecular bones in the spine, ribs, and sternum and incorporate them into the library of virtual human phantoms (XCAT). Cortical bone was modeled by 3D morphological erosion of XCAT homogenously defined bones with an average thickness measured from the CT dataset upon which each individual XCAT phantom was based. The trabecular texture was modeled using a power law synthesis algorithm where the parameters were tuned using high-resolution anatomical images of the Human Visible Female. The synthesized bone textures were added into the XCAT phantoms. To qualitatively evaluate the improved realism of the bone modeling, CT simulations of the XCAT phantoms were acquired with and without the textured bone modeling. The 3D power spectrum of the anatomical images exhibited a power law behavior (R2 = 0.84), as expected in fractal and porous textures. The proposed texture synthesis algorithm was able to synthesize textures emulating real anatomical images, with the simulated CT images with the prototyped bones were more realistic than those simulated with the original XCAT models. Incorporating intra-organ structures, the "textured" phantoms are envisioned to be used to conduct virtual clinical trials in the context of medical imaging in cases where the actual trials are infeasible due to the lack of ground truth, cost, or potential risks to the patients.

GPU-based interactive near-regular texture synthesis for digital human models.

Near-regular texture is a common feature for both nature scene and 3D human models. However, traditional texture synthesis has only been able to produce a single result. This study proposes a parallel, and interactive, method for the texture synthesis of irregular appearance in 3D human models. The new method includes two major procedures: (1) iterative edge extraction and processing, and (2) parallel texture synthesis, which generates a texture with higher quality. In this paper, the effectiveness of near-regular texture synthesis algorithm is demonstrated experimentally.

A Subjective Evaluation of Texture Synthesis Methods

This paper presents the results of a user study which quantifies the relative and absolute quality of example-based texture synthesis algorithms. In order to allow such evaluation, a list of texture properties is compiled, and a minimal representative set of textures is selected to cover these. Six texture synthesis methods are compared against each other and a reference on a selection of twelve textures by non-expert participants (N = 67). Results demonstrate certain algorithms successfully solve the problem of texture synthesis for certain textures, but there are no satisfactory results for other types of texture properties. The presented textures and results make it possible for future work to be subjectively compared, thus facilitating the development of future texture synthesis methods.

Style Transfer Via Texture Synthesis.

Style transfer is a process of migrating a style from a given image to the content of another, synthesizing a new image, which is an artistic mixture of the two. Recent work on this problem adopting convolutional neural-networks (CNN) ignited a renewed interest in this field, due to the very impressive results obtained. There exists an alternative path toward handling the style transfer task, via the generalization of texture synthesis algorithms. This approach has been proposed over the years, but its results are typically less impressive compared with the CNN ones. In this paper, we propose a novel style transfer algorithm that extends the texture synthesis work of Kwatra et al. (2005), while aiming to get stylized images that are closer in quality to the CNN ones. We modify Kwatra's algorithm in several key ways in order to achieve the desired transfer, with emphasis on a consistent way for keeping the content intact in selected regions, while producing hallucinated and rich style in others. The results obtained are visually pleasing and diverse, shown to be competitive with the recent CNN style transfer algorithms. The proposed algorithm is fast and flexible, being able to process any pair of content + style images.

Non-Parametric Texture Synthesis Using Texture Classification

In this paper we present a method of texture synthesis which removes the need for users to set, or even understand, parameters which have an impact on the synthesized output. We accomplish this by first classifying each input texture sample into one of three texture types: regular, irregular and stochastic. We found that textures within a class were synthesized well with similar parameters. If we know the input texture class, we can provide a good starting set of parameters for the synthesis algorithm. Instead of requiring a user to manually select a set of parameters, we simply ask that the user tell us whether the synthesized texture is satisfactory or not. If the output is not satisfactory, we adjust parameters and try again until the user is happy with the output. In this implementation we use the image quilting method in [1], a texture synthesis algorithm, as well as texture classification. With small adjustments our method can be applied to other texture synthesis methods.

Refining raw pixel values using a value error model to drive texture synthesis

The goal in photography is generally the construction of a model of scene appearance. Unfortunately, statistical variations introduced by photon shot and other noise introduce errors in the raw value reported for each pixel sample. Rather than simply accepting those values as the best raw representation, the current work treats them as initial estimates of the correct values, and computes an error model for each pixel's value. The value error models for all pixels in an image are then used to drive a type of texture synthesis which refines the pixel value estimates, potentially increasing both accuracy and precision of each value. Each refined raw pixel value is synthesized from the value estimates of a plurality of pixels with overlapping error bounds and similar context within the same image. The error modeling and texture synthesis algorithms are implemented in and evaluated using KREMY (KentuckY Raw Error Modeler, pronounced "creamy"), a free software tool created for this purpose.