Linguistic interpretation of human behavior by using motion symbol and corpus

Abstract

This paper proposes the model that learns the relation between human motion and language, and generates sentences that describe human motions. Previous researches presented the framework that can symbolize human motions by using Hidden Markov Model. Our proposed model, ”Motion Description Model”, learns the relation between the symbolized motions and the sentences labeled on motions. It uses Long Short-Term Memory for the memory and generator of the sentences. Furthermore, we propose the method to enrich the label sentences by using the corpus. In detail, before training the Motion Description Model, original label sentences are replaced with new label sentences by the sentence generation model trained with the corpus. This method realizes generating more detail and context-sensitive descriptions of motion. We confirmed the validity of our proposed approach by generating sentences from symbolized motions, and with the environment or object words.