Digital Storytelling Book Generator with MIDI-to-Singing

Abstract

Creating a digital storytelling book is an important knowledge source for the blinds, but it usually takes a lot of time and efforts. In order to read the books from electronic contents, automatic procedures could be incorporated into a speech synthesis system. In this paper, we give a practical description using a free software Text-to-speech (TTS) program with a MIDI-to-Singing toolkit as a digital storytelling book generator. In this case, a certain amount of emotional TTS customization can be derived by using time-pitch manipulation of the synthesized acoustic waveform. MIDI-to-Singing voices can be generated automatically with special emphasis on lyrical or storytelling-styled contents that are usually discouraged by uninteresting natures of voices synthesized from traditional Text-to-speech (TTS) programs. Rule-based approaches rely on rules that describe the behavior of the pitch frequency along time to generate time-pitch values. Pitch values fluctuate within a certain range depending on the intended emotion. This MIDI-to-Singing voice synthesis relies on mapping the pitch frequency values to the 12 semi-tonal melodic scales and extracting semi-tonic intervals for each emotional state. In the current version of the system, a user can style the synthesized voice by selecting either male or female standard voice in combination with one of the predefined 12 expressive styles: Neutral, Monotonic, Lowly-pitched, Highly-pitched, Rising-pitched, Falling-pitched, Happy, Sad, Fear, Anger, Randomly-pitched, and Melody-aligning (singing) styles using a small set of musical notes. A subjective test shows that synthetic conversations based on MIDI-to-Singing with customized styles are more preferable, natural, intelligible and enjoyable than the traditional ones. Finally, the result of digital talking recordings can be heard on the web-site for the comparisons between human speech and MIDI-to-Singing synthesized speech.