Screenplay alignment for closed-system speaker identification and analysis of feature films

Abstract

Existing methods for audiovisual and text analysis of videos perform blind recovery of metadata from the audiovisual signal. The film production process however, is based on the original screenplay and its versions. Using this information is like using the recipe book for the movie. High-level semantic information that is otherwise very difficult to derive from the audiovisual content can be extracted automatically by enhancing feature extraction with screenplay processing and analysis. As a test-bed of our approach, we investigated the use of screenplay as a source of information for speaker/character identification. Our speaker identification method consists of screenplay parsing, extraction of time-stamped transcript, alignment of the screenplay with the time-stamped transcript, audio segmentation and audio speaker identification. As the screenplay alignment cannot identify all dialogue sections within any film, we use the segments found by alignment as labels to train a statistical model in order to identify unaligned pieces of dialogue. We find that the screenplay alignment is able to identify the speaker correctly in 30% of lines of dialogue on average. However, with additional automatic statistical labeling for audio speaker ID on the soundtrack, our recognition rate improves significantly.