Synchro-Sub, an adaptive multi-algorithm framework for real-time subtitling synchronisation of multi-type TV programmes

Abstract

Subtitles are critical elements in making TV as accessible as possible for people with hearing impairment, elderly people and other end-users that could need this kind of service. Moreover, delay in live-originated TV captions or subtitles is a significant factor in the audience’s perception of the absence of quality for the deaf and hard of hearing. This lack of synchronisation leads to poor quality broadcasts because of the inconsistency between images and subtitles presented to the viewer. In some cases, this situation leads to a misunderstanding of the message. Subtitling is a reactive process that starts once a minimum of audio information has been generated and ends when an insertion time, deletion time and text are allocated. Therefore, there will always be a delay associated with this process, related to the audio duration being subtitled and the subtitle production time. Moreover, in live TV programmes, unsynchronised audio and subtitles are inherent to the subtitling process because the broadcasting is performed in real-time. In this paper, the authors present the Synchro-Sub framework for synchronising subtitles with the audio-visual contents for live-captioned TV programs. For doing this, and considering the knowledge obtained in previous approaches defined by the authors, several improvements have been included in the transcription and chronization phases of the framework that increase the percentage of correctly timed subtitles.