Spatial-Temporal Wavelet Analysis Video Prediction Based on DifferentialAttention Mechanism

Abstract

Inspired by the visual process of human, a video prediction algorithm based on spatial-temporal wavelet analysis and differential attention is proposed to solve the problem that it is difficult to accurately predict the details of spatial structure information and the dependence of temporal motion. Firstly, the spatial-temporal wavelet analysis module is used to decompose the video in multiple frequencies, so as to enhance the model’s ability to understand high-frequency details and procedural motion. Then, the differential attention mechanism guides the model to allocate attention resources more efficiently and reasonably, and improves the expression ability of instantaneous motion. Experimental results on the KTH, Cityscapes, BAIR, KITTI, Caltech Pedestrian datasets show the proposed algorithm achieves better results than the existing algorithms in the quantitative evaluation metrics of PSNR, SSIM and LPIPS. Meanwhile, the visualization results also show that the prediction of the proposed algorithm is clearer.