Multi-modal actuation with the activation bit vector machine

Abstract

Research towards a new approach to the abstract symbol grounding problem showed that through model counting there is a correspondence between logical/linguistic and coordinate representation in the visuospatial domain. The logical/verbal description of a spatial layout directly gives rise to a coordinate representation that can be drawn, with the drawing reflecting what is described. The main characteristic of this logical property is that it does not need any semantic information or ontology apart from a separation into symbols/words referring to relations and symbols/words referring to objects. Moreover, the complete mechanism can be implemented efficiently on a brain inspired cognitive architecture, the Activation Bit Vector Machine (ABVM), an architecture that belongs to the Vector Symbolic Architectures. However, the natural language fragment captured previously was restricted to simple predication sentences, with the corresponding logical fragment being atomic Context Logic (CLA), and the only actuation modality leveraged was visualization. This article extends the approach on all three aspects: adding a third category of action verbs we move to a fragment of first-order Context Logic (CL1), with modalities requiring a temporal dimension, such as film and music, becoming available. The article presents an ABVM generating sequences of images from texts.