Example-based Synthesis Research Articles

Flexible Example-Based Program Synthesis on Tree-Structured Function Compositions

We introduce a flexible program synthesis system whose task is to predict function compositions that transform given inputs to their corresponding given outputs. We process input lists in a sequential manner, allowing the system to generalize to a wide range of input lengths. We separate the operator and the operand in the lambda functions of the used higher order functions to achieve significantly wider numeric parameter ranges compared to the previous works. The evaluations show that this approach is competitive with state-of-the-art systems, while it is much more flexible in terms of the input length, the parameters of the lambda functions, and the integer range of the inputs and outputs. We extend the system to handle tree-structured function compositions by introducing two additional functions (zip_with, copy) and the ability to represent unfinished function compositions during the synthesis process. The extended system achieves state-of-the-art results while synthesizing complex function compositions with multiple forks. We believe that flexibility in these aspects is an important step towards solving real-world problems with example-based program synthesis.

Texture Synthesizability Assessment via Deep Siamese-Type Network

Example-based texture synthesis plays a significant role in many fields, including computer graphics, computer vision, multimedia, and image and video editing and processing. However, it is not easy for all textures to synthesize high-quality outputs of any size from a small input example. Hence, the assessment of the synthesizability of the example textures deserves more attention. Inspired by the broad studies in image quality assessment, we propose a texture synthesizability assessment approach based on a deep Siamese-type network. To our best knowledge, this is the first attempt to evaluate the synthesizability of sample textures through end-to-end training. We first train a Siamese-type network to compare the example texture and the synthesized texture in terms of their similarity and then transfer the experience knowledge obtained in the Siamese-type network to a traditional CNN by fine-tuning, so that to give an absolute score to a single example texture, representing its synthesizability. Not relying on laborious human selection and annotation, these synthesized textures can be generated automatically by example-based synthesis algorithms. We demonstrate that our approach is completely data-driven without hand-crafted features and/or prior knowledge in the field of expertise. Experiments show that our approach improves the accuracy of texture synthesizability assessment qualitatively and quantitatively and outperforms the manual feature-based method.

Audio-Visual Model for Generating Eating Sounds Using Food ASMR Videos

We present an audio-visual model for generating food texture sounds from silent eating videos. We designed a deep network-based model that takes the visual features of the detected faces as input and outputs a magnitude spectrogram that aligns with the visual streams. Generating raw waveform samples directly from a given input visual stream is challenging; in this study, we used the Griffin-Lim algorithm for phase recovery from the predicted magnitude to generate raw waveform samples using inverse short-time Fourier transform. Additionally, we produced waveforms from these magnitude spectrograms using an example-based synthesis procedure. To train the model, we created a dataset containing several food autonomous sensory meridian response videos. We evaluated our model on this dataset and found that the predicted sound features exhibit appropriate temporal synchronization with the visual inputs. Our subjective evaluation experiments demonstrated that the predicted sounds are considerably realistic to fool participants in a “real” or “fake” psychophysical experiment.

Program sketching with live bidirectional evaluation

We present a system called Smyth for program sketching in a typed functional language whereby the concrete evaluation of ordinary assertions gives rise to input-output examples, which are then used to guide the search to complete the holes. The key innovation, called live bidirectional evaluation, propagates examples "backward" through partially evaluated sketches. Live bidirectional evaluation enables Smyth to (a) synthesize recursive functions without trace-complete sets of examples and (b) specify and solve interdependent synthesis goals. Eliminating the trace-completeness requirement resolves a significant limitation faced by prior synthesis techniques when given partial specifications in the form of input-output examples. To assess the practical implications of our techniques, we ran several experiments on benchmarks used to evaluate Myth, a state-of-the-art example-based synthesis tool. First, given expert examples (and no partial implementations), we find that Smyth requires on average 66% of the number of expert examples required by Myth. Second, we find that Smyth is robust to randomly-generated examples, synthesizing many tasks with relatively few more random examples than those provided by an expert. Third, we create a suite of small sketching tasks by systematically employing a simple sketching strategy to the Myth benchmarks; we find that user-provided sketches in Smyth often further reduce the total specification burden (i.e. the combination of partial implementations and examples). Lastly, we find that Leon and Synquid, two state-of-the-art logic-based synthesis tools, fail to complete several tasks on which Smyth succeeds.

Augmented example-based synthesis using relational perturbation properties

Example-based specifications for program synthesis are inherently ambiguous and may cause synthesizers to generate programs that do not exhibit intended behavior on unseen inputs. Existing synthesis techniques attempt to address this problem by either placing a domain-specific syntactic bias on the hypothesis space or heavily relying on user feedback to help resolve ambiguity. We present a new framework to address the ambiguity/generalizability problem in example-based synthesis. The key feature of our framework is that it places a semantic bias on the hypothesis space using "relational perturbation properties" that relate the perturbation/change in a program output to the perturbation/change in a program input. An example of such a property is permutation invariance: the program output does not change when the elements of the program input (array) are permuted. The framework is portable across multiple domains and synthesizers and is based on two core steps: (1) automatically augment the set of user-provided examples by "applying" relational perturbation properties and (2) use a generic example-based synthesizer to generate a program consistent with the augmented set of examples. Our framework can be instantiated with three different user interfaces, with varying degrees of user engagement to help infer relevant relational perturbation properties. This includes an interface in which the user only provides examples and our framework automatically infers relevant properties. We implement our framework in a tool SKETCHAX specialized to the SKETCH synthesizer and demonstrate that SKETCHAX is effective in significantly boosting the performance of SKETCH for all three user interfaces.

Example-based synthesis of stylized facial animations

We introduce a novel approach to example-based stylization of portrait videos that preserves both the subject's identity and the visual richness of the input style exemplar. Unlike the current state-of-the-art based on neural style transfer [Selim et al. 2016], our method performs non-parametric texture synthesis that retains more of the local textural details of the artistic exemplar and does not suffer from image warping artifacts caused by aligning the style exemplar with the target face. Our method allows the creation of videos with less than full temporal coherence [Ruder et al. 2016]. By introducing a controllable amount of temporal dynamics, it more closely approximates the appearance of real hand-painted animation in which every frame was created independently. We demonstrate the practical utility of the proposed solution on a variety of style exemplars and target videos.

Content-Adaptive Sketch Portrait Generation by Decompositional Representation Learning.

Sketch portrait generation benefits a wide range of applications such as digital entertainment and law enforcement. Although plenty of efforts have been dedicated to this task, several issues still remain unsolved for generating vivid and detail-preserving personal sketch portraits. For example, quite a few artifacts may exist in synthesizing hairpins and glasses, and textural details may be lost in the regions of hair or mustache. Moreover, the generalization ability of current systems is somewhat limited since they usually require elaborately collecting a dictionary of examples or carefully tuning features/components. In this paper, we present a novel representation learning framework that generates an end-to-end photo-sketch mapping through structure and texture decomposition. In the training stage, we first decompose the input face photo into different components according to their representational contents (i.e., structural and textural parts) by using a pre-trained convolutional neural network (CNN). Then, we utilize a branched fully CNN for learning structural and textural representations, respectively. In addition, we design a sorted matching mean square error metric to measure texture patterns in the loss function. In the stage of sketch rendering, our approach automatically generates structural and textural representations for the input photo and produces the final result via a probabilistic fusion scheme. Extensive experiments on several challenging benchmarks suggest that our approach outperforms example-based synthesis algorithms in terms of both perceptual and objective metrics. In addition, the proposed method also has better generalization ability across data set without additional training.

Image Retargeting by Texture-Aware Synthesis.

Real-world images usually contain vivid contents and rich textural details, which will complicate the manipulation on them. In this paper, we design a new framework based on exampled-based texture synthesis to enhance content-aware image retargeting. By detecting the textural regions in an image, the textural image content can be synthesized rather than simply distorted or cropped. This method enables the manipulation of textural & non-textural regions with different strategies since they have different natures. We propose to retarget the textural regions by example-based synthesis and non-textural regions by fast multi-operator. To achieve practical retargeting applications for general images, we develop an automatic and fast texture detection method that can detect multiple disjoint textural regions. We adjust the saliency of the image according to the features of the textural regions. To validate the proposed method, comparisons with state-of-the-art image retargeting techniques and a user study were conducted. Convincing visual results are shown to demonstrate the effectiveness of the proposed method.

Antialiased super-resolution with parallel high-frequency synthesis

Image super-resolution (SR) increases the resolution of the target image, and has become a fundamental image-editing operation for real-world applications. Traditional methods often cause jaggies and blurring artifacts because natural images generally contain a lot of discrete continuities and edges. This paper proposes a new synthesis-based method for image super-resolution at a pixel level that takes advantages of convolution-based edge anti-aliasing. The target images are divided into two components representing, respectively, the high- and low-frequency contents of the images. We perform bicubic interpolation to reconstruct the missing information in the low-frequency component. A patch-based texture synthesis is subsequently adopted to synthesize the high-frequency patches with the final upscaled images. In particular, we also use the efficient edge-based anti-aliasing for correcting the quantization error, restore the high-frequency details damaged by nonlinear example-based synthesis. Our proposed approach generates super-resolution images dynamically and can be fully implemented in GPU parallelization. Experiments confirm the visual superiority of our proposed approach in comparison with competing state-of-the-art techniques.

WorldBrush

We present a novel approach for the interactive synthesis and editing of virtual worlds. Our method is inspired by painting operations and uses methods for statistical example-based synthesis to automate content synthesis and deformation. Our real-time approach takes a form of local inverse procedural modeling based on intermediate statistical models: selected regions of procedurally and manually constructed example scenes are analyzed, and their parameters are stored as distributions in a palette, similar to colors on a painter's palette. These distributions can then be interactively applied with brushes and combined in various ways, like in painting systems. Selected regions can also be moved or stretched while maintaining the consistency of their content. Our method captures both distributions of elements and structured objects, and models their interactions. Results range from the interactive editing of 2D artwork maps to the design of 3D virtual worlds, where constraints set by the terrain's slope are also taken into account.

Efficient multi-constrained optimization for example-based synthesis

Digital media content comes in a wide variety of modalities and representations. Although they have obvious semantic and structural difference, many of them can be unwrapped into a one-dimensional ...

Lightweight wrinkle synthesis for 3D facial modeling and animation

We present a lightweight non-parametric method to generate wrinkles for 3D facial modeling and animation. The key lightweight feature of the method is that it can generate plausible wrinkles using a single low-cost Kinect camera and one high quality 3D face model with details as the example. Our method works in two stages: (1) offline personalized wrinkled blendshape construction. User-specific expressions are recorded using the RGB-Depth camera, and the wrinkles are generated through example-based synthesis of geometric details. (2) Online 3D facial performance capturing. These reconstructed expressions are used as blendshapes to capture facial animations in real-time. Experiments on a variety of facial performance videos show that our method can produce plausible results, approximating the wrinkles in an accurate way. Furthermore, our technique is low-cost and convenient for common users.

Video-Based Crowd Synthesis

As a controllable medium, video-realistic crowds are important for creating the illusion of a populated reality in special effects, games, and architectural visualization. While recent progress in simulation and motion captured-based techniques for crowd synthesis has focused on natural macroscale behavior, this paper addresses the complementary problem of synthesizing crowds with realistic microscale behavior and appearance. Example-based synthesis methods such as video textures are an appealing alternative to conventional model-based methods, but current techniques are unable to represent and satisfy constraints between video sprites and the scene. This paper describes how to synthesize crowds by segmenting pedestrians from input videos of natural crowds and optimally placing them into an output video while satisfying environmental constraints imposed by the scene. We introduce crowd tubes, a representation of video objects designed to compose a crowd of video billboards while avoiding collisions between static and dynamic obstacles. The approach consists of representing crowd tube samples and constraint violations with a conflict graph. The maximal independent set yields a dense constraint-satisfying crowd composition. We present a prototype system for the capture, analysis, synthesis, and control of video-based crowds. Several results demonstrate the system's ability to generate videos of crowds which exhibit a variety of natural behaviors.

Example-based synthesis of 3D object arrangements

We present a method for synthesizing 3D object arrangements from examples. Given a few user-provided examples, our system can synthesize a diverse set of plausible new scenes by learning from a larger scene database. We rely on three novel contributions. First, we introduce a probabilistic model for scenes based on Bayesian networks and Gaussian mixtures that can be trained from a small number of input examples. Second, we develop a clustering algorithm that groups objects occurring in a database of scenes according to their local scene neighborhoods. These contextual categories allow the synthesis process to treat a wider variety of objects as interchangeable. Third, we train our probabilistic model on a mix of user-provided examples and relevant scenes retrieved from the database. This mixed model learning process can be controlled to introduce additional variety into the synthesized scenes. We evaluate our algorithm through qualitative results and a perceptual study in which participants judged synthesized scenes to be highly plausible, as compared to hand-created scenes.

Adaptive Synthesis of Distance Fields

We address the computational resource requirements of 3D example-based synthesis with an adaptive synthesis technique that uses a tree-based synthesis map. A signed-distance field (SDF) is determined for the 3D exemplars, and then new models can be synthesized as SDFs by neighborhood matching. Unlike voxel synthesis approach, our input is posed in the real domain to preserve maximum detail. In comparison to straightforward extensions to the existing volume texture synthesis approach, we made several improvements in terms of memory requirements, computation times, and synthesis quality. The inherent parallelism in this method makes it suitable for a multicore CPU. Results show that computation times and memory requirements are very much reduced, and large synthesized scenes exhibit fine details which mimic the exemplars.

A Texture Synthesis Algorithm Based on SFLA-PSO and Block Increasing

A texture synthesis algorithm based on SFLA-PSO and block increasing is presented for example-based synthesis. It can be used to speed up texture synthesizing through using the texture block of which size is increased by times of two during the process of texture synthesis; As the searching tactics of texture block matching, SFLA and PSO are combined to accelerate and improve the global searching performance. The experimental results prove that the algorithm can evidently accelerate texture synthesis on the premise of fine synthesis quality, and overcome the defect of easily falling into local optimal solution of PSO algorithm.

Block increasing texture synthesis algorithm based on shuffled frog leaping algorithm and particle swarm optimization

A texture synthesis algorithm based on Shuffled Frog Leaping Algorithm and Particle Swarm Optimization(SFLA-PSO) and block increasing was proposed for example-based synthesis.It can be used to speed up texture synthesizing through using the texture block of which size was increased two times during the process of texture synthesis.As texture block matching optimum strategy,SFLA and PSO were combined to accelerate and improve the global searching performance.The experimental results prove that the algorithm can significantly accelerate the speed of example-based synthesis on the premise of fine synthesis quality,and also can overcome the defect of easily falling into local optimal solution of PSO algorithm.

Example-based wrinkle synthesis for clothing animation

Example-based wrinkle synthesis for clothing animation

Layered shape synthesis

Many inhomogeneous real-world textures are non-stationary and exhibit various large scale patterns that are easily perceived by a human observer. Such textures violate the assumptions underlying most state-of-the-art example-based synthesis methods. Consequently, they cannot be properly reproduced by these methods, unless a suitable control map is provided to guide the synthesis process. Such control maps are typically either user specified or generated by a simulation. In this paper, we present an alternative: a method for automatic example-based generation of control maps, geared at synthesis of natural, highly inhomogeneous textures, such as those resulting from natural aging or weathering processes. Our method is based on the observation that an appropriate control map for many of these textures may be modeled as a superposition of several layers, where the visible parts of each layer are occupied by a more homogeneous texture. Thus, given a decomposition of a texture exemplar into a small number of such layers, we employ a novel example-based shape synthesis algorithm to automatically generate a new set of layers. Our shape synthesis algorithm is designed to preserve both local and global characteristics of the exemplar's layer map. This process results in a new control map, which then may be used to guide the subsequent texture synthesis process.

Texture optimization for example-based synthesis

We present a novel technique for texture synthesis using optimization. We define a Markov Random Field (MRF)-based similarity metric for measuring the quality of synthesized texture with respect to a given input sample. This allows us to formulate the synthesis problem as minimization of an energy function, which is optimized using an Expectation Maximization (EM)-like algorithm. In contrast to most example-based techniques that do region-growing, ours is a joint optimization approach that progressively refines the entire texture. Additionally, our approach is ideally suited to allow for controllable synthesis of textures. Specifically, we demonstrate controllability by animating image textures using flow fields. We allow for general two-dimensional flow fields that may dynamically change over time. Applications of this technique include dynamic texturing of fluid animations and texture-based flow visualization.