Improving the Visual Quality of Video Frame Prediction Models Using the Perceptual Straightening Hypothesis

Abstract

We present a simple and effective method to improve the visual quality of the predicted frames in a frame prediction model. A recent neuroscience study hypothesizes that the perceptual representations of a sequence of frames extracted from a natural video follow a straight temporal trajectory. The perceptual representations of a sequence of video frames are found using a computational model of the LGN and V1 areas of the human visual system. In this work, we leverage the strength of this perceptual straightening model to formulate a novel objective function for video frame prediction. In general, a frame prediction model takes past frames as input and predicts the future frame. We enforce the perceptual straightness constraint through adversarial training by introducing the proposed novel quality aware discriminator loss. Our quality aware discriminator imposes the linear relationship between the perceptual representation of the predicted frame and the perceptual representations of the past frames. Specifically, we claim that imposing a perceptual straightness constraint through the discriminator helps in predicting (i.e., generating) video frames that look more natural and therefore, having a higher perceptual quality. We demonstrate the effectiveness of our proposed objective function on two popular video datasets using two different frame prediction models. These experiments show that our solution is both consistent and stable, thereby allowing it to be integrated with other frame prediction models as well.