Models of Cross-Situational and Crossmodal Word Learning in Task-Oriented Scenarios

Abstract

We present two related but different cross-situational and crossmodal models of incremental word learning. Model 1 is a Bayesian approach for co-learning object-word mappings and referential intention which allows for incremental learning from only a few situations where the display of referents to the learning system is systematically varied. We demonstrate the robustness of the model with respect to sensory noise, including errors in the visual (object recognition) and auditory (recognition of words) systems. The model is then integrated with a cognitive robotic architecture in order to realize cross-situational word learning on a robot. A different approach to word learning is demonstrated with Model 2, an information-theoretic model for the object- and action-word learning from modality rich input data based on pointwise mutual information. The approach is inspired by insights from language development and learning where the caregiver/teacher typically shows objects and performs actions to the infant while naming what the teacher is doing. We demonstrate the word learning capabilities of the model, feeding it with crossmodal input data from two German multimodal corpora which comprise visual scenes of performed actions and related utterances.