Learning GANs in Simultaneous Game Using Sinkhorn With Positive Features

Abstract

Entropy regularized optimal transport (EOT) distance and its symmetric normalization, known as the Sinkhorn divergence, offer smooth and continuous metrized weak-convergence distance metrics. They have excellent geometric properties and are useful to compare probability distributions in some generative adversarial network (GAN) models. Computing them using the original Sinkhorn matrix scaling algorithm is still expensive. The running time is quadratic at $\mathcal {O}(n^{2})$ in the size $n$ of the training dataset. This work investigates the problem of accelerating the GAN training when Sinkhorn divergence is used as a minimax objective. Let $\mathcal {G}$ be a Gaussian map from the ground space onto the positive orthant $\mathbb {R}_{+}^{r}$ with $r \ll n $ . To speed up the divergence computation, we propose the use of $c(x,y)= - \varepsilon \log \left \langle{ \mathcal {G}(x),\mathcal {G}(y) }\right \rangle $ as the ground cost. This approximation, known as Sinkhorn with positive features, brings down the running time of the Sinkhorn matrix scaling algorithm to $\mathcal {O}(r \, n)$ , which is linear in $n$ . To solve the minimax optimization in GAN, we put forward a more efficient simultaneous stochastic gradient descent-ascent (SimSGDA) algorithm in place of the standard sequential gradient techniques. Empirical evidence shows that our model, trained using SimSGDA on the DCGAN neural architecture on tiny-coloured Cats and CelebA datasets, converges to stationary points. These are the local Nash equilibrium points. We carried out numerical experiments to confirm that our model is computationally stable. It generates samples of comparable quality to those produced by prior Sinkhorn and Wasserstein GANs. Further simulations, assessed on the similarity index measures (SSIM), show that our model’s empirical convergence rate is comparable to that of WGAN-GP.