Building a world model with structure-sensitive sparse binary distributed representations

Abstract

We present a new cognitive architecture named Associative-Projective Neural Networks (APNNs). APNNs have a multi-module, multi-level, and multi-modal design that works with an original scheme of sparse binary distributed representations to construct world models of varied complexity required for both task-specific and more general cognitive modeling. APNNs provide scalability and flexibility due to a number of design features. Internal representations of APNNs are sparse binary vectors of fixed dimensionality for items of various complexity and generality. Representations of input scalars, vectors, or compositional relational structures are constructed on-the-fly, so that similar items produce representations similar in terms of vector dot-products. Thus, for example, similarity of relational structures (taking into account similarity of their components, their grouping and order) can be estimated by dot-products of their representations, without the need to follow edges or to match vertices of underlying graphs. Decoding distributed representations through the input representations is also possible. Storage, retrieval, and decoding of distributed representations are implemented by efficient auto-associative memories; using distributed memories based on the idea of Hebb’s cell assemblies additionally provides a natural tool for emergence of generalization hierarchies. In addition, we consider how APNNs account for representation grounding, deal with recent challenges for distributed representations, and present some open problems.