Texture Mixing by Interpolating Deep Statistics via Gaussian Models

Zhucun Xue,Ziming Wang

doi:10.1109/access.2020.2981410

Abstract

Recently, enthusiastic studies have devoted to texture synthesis using deep neural networks, because these networks excel at handling complex patterns in images. In these models, second-order statistics, such as Gram matrix, are used to describe textures. Despite the fact that these model have achieved promising results, the structure of their parametric space is still unclear, consequently, it is difficult to use them to mix textures. This paper addresses the texture mixing problem by using a Gaussian scheme to interpolate deep statistics computed from deep neural networks. More precisely, we first reveal that the statistics used in existing deep models can be unified using a stationary Gaussian scheme. We then present a novel algorithm to mix these statistics by interpolating between Gaussian models using optimal transport. We further apply our scheme to Neural Style Transfer, where we can create mixed styles. The experiments demonstrate that our method can achieve state-of-the-art results. Because all the computations are implemented in closed forms, our mixing algorithm adds only negligible time to the original texture synthesis procedure.

Highlights

Texture mixing is the process of generating new texture images that possess averaged visual characteristics of a given set of exemplars [1]–[5]
The ability to create smoothly morphing textures is regarded as a criterion for ‘‘good’’ texture synthesis algorithms [7] [8]
After studying existing deep texture models [18]–[22], we discover that the second-order statistics (e.g. Gram matrix, correlation matrix and their variations) used in these methods can be represented as continuous functions of a stationary Gaussian model, so the mixing of these statistics is reduced to the interpolation of Gaussian models, which is known to have a closed form solution

Summary

Introduction

Texture mixing is the process of generating new texture images that possess averaged visual characteristics of a given set of exemplars [1]–[5]. It can provide visually pleasing interpolations of difference textures, has numerous applications in computer vision and graphics [4], [6]. For copy-based texture synthesis methods [12], [13], textures can be mixed by combining pixels from multiple inputs using well-designed procedures such as in [4] or the patch match scheme [14]. These methods handle complex and geometric textures satisfactorily, but they tend to produce verbatim patterns and it is difficult to understand the mixing process. With parametric texture models, the mixing of textures can

Objectives

Methods

Results