Statistics Analysis of Multi-Modality and Text-Derived Representations on Concept Learning

Abstract

When learning concepts, cognitive psychology research has revealed that there are two types of concept representation in the human brain: language-derived codes and sensory-derived codes. Thus, in terms of computing, do the multimodal concept vectors and linguistic concept vectors have differences and similarities as human beings? Four experiments are presented in this work, all focused on multi-modality representations labeled by psychologists and text-derived representations generated by computer scientists for concept learning, and the results demonstrate that 1)for the same concept, both forms of representations can properly reflect the concept, but 2) the representational similarity analysis findings reveal that the two types of representations are significantly different, 3) as the concreteness of a concept grows larger, the multi-modality representation of the concept becomes closer to human beings than the text-derived representation, and 4) we verified that combining the two improves the concept representation.