Aligned variational autoencoder for matching danmaku and video storylines

Abstract

We study a task of aligning time-sync video comments (danmaku) to narrative video storylines, which is helpful for finding semantic segmentation of videos and conducting fine-grained user feedback analyses. Due to the mismatch of text styles and the shift of topics, it is hard to apply previous semantic matching models directly for the alignment. To tackle the problem, we propose to utilize variational auto-encoders to map both user comments and storylines into latent spaces. By posing a matching loss on their latent codes, we reduce their mismatches in the latent space and make the alignment easier to learn. To handle constraints in the alignment, we also apply dynamic programming for finding global optimal outputs. According to experiments on a TV series dataset (consisting of about 10 K pairs of storylines and danmaku sent by users), the proposed model can achieve competitive performances.